SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PLATO drastically reduces the number of trainable weights compared to a standard MLP. The sharing of  $\mathcal{B}$  and  $\Phi$  across all input features drastically reduces the number of trainable weights compared to a standard MLP. For a high-dimensional tabular dataset (i.e.  $d \gg n$ ), a standard MLP  $\mathcal{T}$  with  $h$  hidden units has a large number of trainable weights in the first layer since  $\Theta^{[1]} \in \mathbb{R}^{d \times h}$ . A standard MLP  $\mathcal{T}$  must learn all  $dh$  of these trainable weights by backpropagation. By contrast,  $\mathcal{B}$  uses a shared set of trainable weights  $\Phi$  to infer  $\hat{\Theta}_j$  from  $\mathbf{Q}_j$  for every  $j \in \{1, \dots, d\}$ . The number of trainable weights in  $\Phi$  is small compared to  $dh$  since  $\mathcal{B}$  need only transform every  $\mathbf{Q}_j \in \mathbb{R}^c$  to  $\hat{\Theta}^{[1]} \in \mathbb{R}^h$ . Thus,  $|\Phi| = ch$  (assuming  $\mathcal{B}$  is a single layer neural network).  $c$ , the dimensionality of the feature embedding, is much less than  $d$  the number of input features. As a result,  $|\Phi| = ch \ll dh$  and PLATO drastically reduces the number of trainable weights in the first layer of a MLP.
3.7 The PLATO algorithm
PLATO is outlined in Algorithm 1.
Table 1: PLATO outperforms statistical and deep baselines when  $d \gg n$ . For every dataset, the best overall model is in bold and the second best model is underlined.
Dataset	MNSCLC	CM	PDAC	BRCA	CRC	CH
# of features d	15,390	13,183	12,932	12,693	18,206	19,902
# of samples n	295	286	321	476	562	924
d/n	52.2	46.1	40.3	28.2	22.6	19.7
Classic Stat ML	Ridge	0.153±0.000	0.390±0.000	0.344±0.000	0.538±0.000	0.376±0.000	0.546±0.000
Dim. Reduct.	PCA	0.156±0.113	0.070±0.000	0.232±0.121	0.452±0.000	0.193±0.163	0.237±0.232
Feat. Select.	LASSO	0.168±0.000	0.431±0.000	0.346±0.000	0.470±0.000	0.400±0.000	0.547±0.000
STG	0.132±0.130	0.366±0.043	0.258±0.055	0.485±0.037	0.301±0.010	0.262±0.076
Decision Tree	XGBoost	-0.02±0.000	0.225±0.000	0.363±0.000	0.347±0.000	0.354±0.000	0.728±0.000
Graph Reg.	GraphNet	0.169±0.030	0.277±0.099	0.249±0.018	0.350±0.069	0.125±0.061	0.646±0.051
NC LASSO	0.210±0.014	0.339±0.044	0.327±0.053	0.458±0.083	0.220±0.030	0.415±0.083
Network LASSO	0.212±0.046	0.243±0.058	0.136±0.027	0.348±0.033	0.171±0.040	0.212±0.091
Param. Infer.	Diet	-0.04±0.205	0.054±0.149	0.309±0.096	0.213±0.036	0.087±0.112	0.148±0.008
Tabular DL	MLP	0.128±0.126	0.322±0.043	0.289±0.047	0.240±0.067	0.355±0.022	0.044±0.039
NODE	0.003±0.000	0.150±0.000	0.190±0.000	0.512±0.000	0.344±0.000	0.181±0.000
TabTransformer	0.265±0.000	0.072±0.000	0.029±0.000	0.202±0.000	0.238±0.000	0.020±0.000
TabNet	0.085±0.028	0.010±0.068	0.088±0.037	0.055±0.037	0.018±0.016	0.039±0.026
Ours	PLATO	0.272±0.130	0.435±0.022	0.400±0.021	0.583±0.019	0.401±0.019	0.770±0.003
Table 2: PLATO's performance depends on updat- Table 3: PLATO's performance depends on ing feature embeddings with a trainable message- both feature nodes in  $G$  and other nodes passing (MP) function. representing broader domain information.
Weight Infer. B Input	Feature Info.	Trainable MP	PearsonR	Auxiliary KG	Feature Info.	Broader Info.	PearsonR
Updated feat. embed. Q	✓	✓	0.583±0.019	Full KG	✓	✓	0.583±0.019
General feat. embed M	✓	X	0.522±0.030	Feature-only KG	✓	X	0.539±0.038
None	X	X	0.240±0.067	No KG	X	X	0.240±0.067
We evaluate PLATO against 13 baselines on 10 tabular datasets (6 with  $d \gg n$ , 4 with  $d \sim n$ ).
Datasets. We use 6 tabular  $d \gg n$  datasets,  $4d \sim n$  datasets [16, 17, 30, 79], and a KG from prior studies [44, 35, 38, 56, 63, 74, 75] (Appendix G, H). The KG contains 108,447 nodes, 3,066,156 edges, and 99 relation types. All datasets include features which map to a subset of knowledge graph nodes. Code, data, and the KG are available at https://github.com/snap-stanford/plato.
Baselines. We compare PLATO to 13 state-of-the-art statistical and deep baselines. We consider regularization with Ridge Regression [45], dimensionality reduction with PCA [1] followed by linear regression, feature selection with LASSO [64], deep feature selection with Stochastic Gates [77], and gradient boosted decision trees with XGBoost [7]. We consider tabular deep learning with a standard MLP, self-attention-based methods with TabTransformer [29] and TabNet [3], differentiable decision trees with NODE [50], and weight inference with Diet Networks [54]. We also attempted FT-Transformer [18], but it experienced out of memory issues on all datasets due to the large number of features. Finally, we consider graph regularization methods which also have access to the knowledge graph including GraphNet [20], NC LASSO [40], and Network LASSO [23] (Appendix E).
Fair Comparison of PLATO with Baselines. To ensure a fair comparison with baselines, we follow evaluation protocols in recent tabular benchmarks [19, 18]. We conduct a random search with 500 configurations of every model (including PLATO) on every dataset across a broad range of hyperparameters (Appendix A). We split data with a 60/20/20 training, validation, test split. All results are computed across 3 data splits and 3 runs of each model in each data split. We report the mean and standard deviation of the Pearson correlation (PearsonR) between  $\mathbf{y}$  and  $\hat{\mathbf{y}}$  across runs and splits on the test set. Each model is run on a GeForce RTX 2080 TI GPU.
Table 4: PLATO's performance with an incom- Table 5: PLATO's MLP layers  $2,\ldots ,L$  with plete knowledge graph. trainable weights are useful for performance.
Fraction of edges in KG	PearsonR
100%	0.583 ± 0.019
90%	0.570 ± 0.017
70%	0.537 ± 0.044
50%	0.412 ± 0.011
Model	Description	PearsonR
PLATO	MLP with first layer weights inferred	0.583 ± 0.019
PLATO-LR	Linear regression with weights inferred	0.550 ± 0.020
Table 6: PLATO's performance is competitive with baselines when  $d \sim n$ . For every dataset, the best overall model is in bold and the second best model is underlined.
Dataset	ME	BC	SCLC	NSCLC
# of features d	19,902	18,261	18,437	18,308
# of samples n	10,064	10,101	10,712	16,730
d/n	2.0	1.8	1.7	1.1
Classic Stat ML	Ridge	0.566±0.008	0.483±0.008	0.604±0.057	0.679±0.008
Dim. Reduct.	PCA	0.239±0.310	0.233±0.294	0.284±0.274	0.645±0.000
Feat. Select.	LASSO	0.667±0.000	0.633±0.000	0.669±0.000	0.637±0.000
STG	0.676±0.000	0.643±0.000	0.668±0.000	0.646±0.000
Decision Tree	XGBoost	0.875±0.000	0.826±0.000	0.878±0.000	0.843±0.000
Graph Reg.	GraphNet	0.675±0.047	0.723±0.026	0.742±0.039	0.627±0.042
NC LASSO	0.733±0.016	0.730±0.027	0.793±0.009	0.746±0.023
Network LASSO	0.401±0.034	0.451±0.022	0.417±0.074	0.465±0.034
Param. Infer.	Diet	0.105±0.000	0.037±0.000	-0.050±0.000	0.002±0.000
Tabular DL	MLP	0.487±0.131	0.508±0.061	0.537±0.061	0.573±0.005
NODE	0.870±0.000	0.420±0.169	0.801±0.102	0.487±0.197
TabTransformer	0.305±0.028	0.010±0.000	0.288±0.203	0.503±0.187
TabNet	0.667±0.002	0.624±0.001	0.657±0.004	0.647±0.000
Ours	PLATO	0.875±0.004	0.844±0.003	0.883±0.002	0.839±0.000
PLATO outperforms statistical and deep baselines when  $d \gg n$ . PLATO outperforms all baselines across all 6 datasets with  $d \gg n$  (Table 1). PLATO achieves the largest improvement on the PDAC dataset, improving by  $10.19\%$  vs. XGBoost, the best baseline for PDAC (0.400 vs. 0.363). While PLATO achieves the strongest performance across all 6 datasets, the best performing baseline varies across datasets. Ridge Regression is the strongest baseline for BRCA, LASSO for CM and CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC. The remaining baselines are not the strongest baseline for any dataset. We also find that the performance of a specific baseline depends largely on the dataset. TabTransformer, for example, is the best baseline for the MNSCLC dataset but the worst baseline for the CH dataset. The rank order of all models on all datasets is Appendix D.
PLATO's performance depends on updating feature embeddings with a trainable message-passing function. PLATO infers the weights  $\hat{\Theta}^{[1]}$  in the first layer of a MLP  $\mathcal{F}$  by using feature embeddings which contain prior information about the input features. PLATO first pretrains general feature embeddings  $\mathbf{M} \in \mathbb{R}^{d \times c}$ . PLATO then updates the feature embeddings to  $\mathbf{Q} \in \mathbb{R}^{d \times c}$  with a trainable message-passing function. We test whether updating the feature embeddings based on the trainable message-passing function is necessary by evaluating PLATO's performance on the BRCA dataset in three configurations (Table 2). The default configuration uses the updated feature embeddings  $\mathbf{Q}$  generated by the message-passing function to infer  $\hat{\Theta}^{[1]}$  according to  $\hat{\Theta}_{j}^{[1]} = \mathcal{B}(\mathbf{Q}_{j}|\mathbf{X}_{i})$ . The second configuration uses the general feature embeddings  $\mathbf{M}$  instead of  $\mathbf{Q}$  to infer  $\hat{\Theta}^{[1]}$  according to  $\hat{\Theta}_{j}^{[1]} = \mathcal{B}(\mathbf{M}_{j})$ . The third configuration does not use feature embeddings and thus ablates to a standard MLP. Using general feature embeddings  $\mathbf{M}$  improves over not using feature embeddings at all (0.522 vs. 0.240). Using feature embeddings  $\mathbf{Q}$  that are generated by the trainable message-passing function further improves performance (0.583 vs. 0.522). Thus, updating the feature embeddings to  $\mathbf{Q}$  based on the trainable message-passing function is key to PLATO's performance.
PLATO's performance depends on both feature nodes and broader knowledge nodes in the auxiliary KG. PLATO relies on an auxiliary KG  $G$  which contains information describing input features and the broader domain. Information describing input features is represented as feature nodes while information describing the broader domain is represented as other nodes in  $G$  (Methods 3.1). To test the relative importance of the feature information in  $G$  vs. the broader domain information, we measured the performance of PLATO on the BRCA dataset in two KG configurations: PLATO with the full KG (i.e. both the feature nodes and the broader domain nodes) and PLATO with a "feature-only KG" (i.e. an induced subgraph on only the feature nodes) (Table 3). We also compare to a "No KG" configuration in which PLATO does not have access to the KG. Without auxiliary information describing the input features or the broader domain, PLATO is ablated to a standard MLP.
We find that both the feature nodes and the broader knowledge nodes are important for PLATO's performance. Using the "feature-only KG" configuration of PLATO improves performance vs the "no KG" configuration (0.539 vs 0.240). Using the "full KG" configuration further improves performance vs the "feature-only KG" configuration (0.583 vs 0.539). PLATO's performance thus relies on both the feature information and the broader domain information in the KG.
PLATO's performance with an incomplete knowledge graph. All KGs are incomplete since there is undiscovered knowledge. PLATO thus uses low-dimensional embeddings from KG embedding approaches [72, 78, 65] which are designed to account for missing information, enabling predictive performance even with missing edges. We conduct an ablation study to assess PLATO's robustness to missing edges in the KG. We randomly remove edges from the KG and measure PLATO's performance on the BRCA dataset. We observe that with only  $50\%$  of the KG's edges, PLATO still has  $71\%$  of the performance as PLATO with  $100\%$  of the KG's edges (0.412 vs. 0.583) (Table 4).
The importance of MLP layers  $2, \ldots, L$ , the layers with trainable weights, for PLATO. PLATO is a MLP in which the weights in the first layer are inferred from the knowledge graph (KG) but the weights in the remaining layers  $2, \ldots, L$  are trained normally. We conduct an ablation study to determine whether MLP layers  $2, \ldots, L$  are necessary for PLATO's performance or whether the first layer of inferred weights are sufficient. Note that a single layer of inferred weights in PLATO is equivalent to a linear regression in which the weights are inferred from the KG. We thus compare PLATO to PLATO-LR, a linear regression in which the weights are inferred from the KG (Table 5). PLATO's standard configuration outperforms PLATO-LR on the BRCA dataset (0.583 vs. 0.550). Therefore, layers  $2, \ldots, L$  of the MLP are important for PLATO's performance.
For datasets with  $d \sim n$ , PLATO is competitive with baselines. Finally, we test PLATO's performance for datasets with  $d \sim n$ . We test 4 datasets with  $d \sim n$  ranging from  $\frac{d}{n} = 1.1$  to 2.0 (Table 6). We find that on 4 datasets with  $d \sim n$ , PLATO is competitive with the best baseline, XGBoost, but does not improve performance substantially. PLATO's stronger performance for datasets with  $d \gg n$  than for datasets with  $d \sim n$  is justified. PLATO's key idea is to include auxiliary information describing the input features. Auxiliary information is likely to help performance the most in settings with the least labeled data (i.e.  $d \gg n$ ). When  $d \sim n$ , auxiliary information is less helpful since the tabular dataset may already have enough information to train a strong predictive model. We further find that XGBoost is the strongest baseline for all datasets with  $d \sim n$ , in contrast to XGBoost's varied performance on the datasets with  $d \gg n$  (Table 1).
PLATO achieves strong performance on tabular data when  $d \gg n$  by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). Across 6 datasets, PLATO outperforms 13 state-of-the-art baselines by up to  $10.19\%$ . Ablations demonstrate the importance of PLATO's trainable message-passing function, of including nodes in the KG that don't represent input features but instead represent domain information, and of the layers in the MLP whose weights are trained directly rather than inferred. We also test PLATO's robustness to missing information in the KG. PLATO has several limitations. First, PLATO matches but does not improve the performance of baselines for high-dimensional datasets with more samples (i.e.  $d \sim n$ ). Second, PLATO depends on the existence of an auxiliary KG of domain information. Overall, PLATO enables tabular deep learning when  $d \gg n$  by using an auxiliary KG of domain information describing input features.
We thank Maria Brbic, Michael Moor, Kaidi Cao, Weihua Hu, Rajas Bansal, Qian Huang, and Hamed Nilforoshan for discussions and for providing feedback on our manuscript. C.R. is supposed by a Siebel Fellowship, a National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518, and a Stanford Enhancing Diversity in Graduate Education (EDGE) Fellowship. H.R. is supposed by the Masason Foundation, an Apple Fellowship, and a Baidu Scholarship. We also gratefully acknowledge the support of DARPA under Nos. HR00112190039 (TAMI), N660011924033 (MCS); ARO under Nos. W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
[1] Hervé Abdi and Lynne J Williams. Principal Component Analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4):433-459, 2010.
[2] Rie Ando and Tong Zhang. Learning on graph with Laplacian regularization. In Advances in Neural Information Processing Systems, volume 19, 2006.
[3] Sercan Ö Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
[4] Muhammed Fatih Balin, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning, pages 444-453. Proceedings of Machine Learning Research, 2019.
[5] Anna Bauer-Mehren, Michael Rautschka, Ferran Sanz, and Laura I Furlong. DisGeNET: a cytoscape plugin to visualize, integrate, search and analyze gene-disease networks. Bioinformatics, 26(22):2924-2926, 2010.
[6] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. In International Joint Conference on Neural Networks, volume 2, pages 969-975. IEEE, 1991.
[7] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[8] Héctor Climente-González, Chloé-Agathe Azencott, Samuel Kaski, and Makoto Yamada. Block HSIC lasso: model-free biomarker detection for ultra-high dimensional data. Bioinformatics, 35(14):i427-i435, 2019.
[9] Gene Ontology Consortium. The Gene Ontology resource: 20 years and still GOing strong. Nucleic Acids Research, 47(D1):D330–D338, 2018.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
[11] Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, volume 26, pages 2148-2156, 2013.
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[13] Claudia d'Amato, Nicola Flavio Quatraro, and Nicola Fanizzi. Injecting background knowledge into embedding models for predictive tasks on knowledge graphs. In European Semantic Web Conference, pages 441-457. Springer, 2021.
[14] Tobias Freidling, Benjamin Poignard, Héctor Climente-González, and Makoto Yamada. Postselection inference with HSIC-Lasso. In International Conference on Machine Learning, pages 3439-3448. PMLR, 2021.
A Evaluation protocol and hyperparameter ranges
To ensure a fair comparison with baselines, we follow evaluation protocols outlined in tabular benchmarks [19, 18]. We conduct a random search with 500 configurations of every model (including PLATO) on every dataset across a broad range of hyperparameters. We base the hyperparameter ranges on the ranges used in prior tabular learning benchmarks [19, 18] and the ranges mentioned in the original papers of the methods. Hyperparameter ranges for PLATO are given in Table 7. Hyperparameter ranges for baseline methods are given in Table 8.
Module in PLATO	Hyperparameter	Range
General	Learning rate	LogUniform(1e-4, 5e-3)
Batch size	[16, 32, 64]
L2	0, LogUniform(1e-5, 1e-2)
KG H	Embedding dimension c	200
Embedding model	ComplEx
Message Passing (MP) Q	# Rounds R	2
β	LogUniform(1e-4, 1e-1)
Hidden dimension in A	UniformInt(16, 512)
Weight Inference B	# Layers	UniformInt(2, 6)
Hidden dimension	UniformInt(16, 512)
Layers 2, ..., L in MLP F	# Layers L	UniformInt(2, 6)
Hidden dimension	UniformInt(16, 512)
Table 7: Hyperparameter ranges used for PLATO.
Model	Hyperparameter	Range
LASSO	L1	LogUniform(1E-4, 10)
Ridge	L2	LogUniform(1E-4, 10)
XGBoost	n-estimators	UniformInt(1,2000)
Max depth	UniformInt(3, 10)
Min weight	LogUniform(1E-8,1E5)
Subsample	Uniform(0.5, 1)
Learning rate	LogUniform(1E-5,1)
Col sample by level	Uniform(0.5, 1)
Col sample by tree	Uniform(0.5, 1)
Gamma	0, LogUniform(1E-8, 1E2)
Lambda	0, LogUniform(1E-8, 1E2)
Alpha	0, LogUniform(1E-8, 1E2)
Booster	"gbtree"
Early-stopping-rounds	50
Iterations	100
PCA	Number of PCA Components	UniformInt(2,1000)
STG	Hidden dimension	UniformInt(10, 500)
Number of layers	UniformInt(1, 5)
Activation	[Tanh, Relu, Sigmoid]
Learning rate	LogUniform(1e-4, 1e-1)
Sigma	Uniform(0.001, 2)
Lambda	LogUniform(1e-3, 10)
MLP	Number of layers	UniformInt(1, 8)
Hidden dimension	UniformInt(1, 512)
Dropout	0, Uniform([0,0.5])
Learning rate	LogUniform(1e-5, 1e-2)
L2	0, LogUniform(1e-6, 1e-3)
TabNet	Decision Steps	UniformInt(3, 10)
Layer size	2, 4, 8, 16, 32, 64
Relaxation factor	Uniform[1, 2]
Sparsity loss weight	LogUniform[1e-6, 1e-1]
Decay rate	Uniform[0.4, 0.95]
Decay steps	100, 500, 2000
Learning rate	Uniform(1e-3, 1e-2)
Iterations	100
TabTransformer	Embedding dimension	4, 8, 16, 32, 64, 128
Number of heads	UniformInt(1, 10)
Number of attention blocks	UniformInt(1, 12)
Attention dropout rate	Uniform(0, 0.5)
Add norm dropout	Uniform(0, 0.5)
Transformation activation	[Tanh, Relu, LeakyReLU]
L2	LogUniform(1e-6, 1e-1)
Learning rate	LogUniform(1e-6, 1e-3)
FF dropout	Uniform(0, 0.5)
FF hidden multiplier	1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Out FF activation	[Tanh, Relu, LeakyReLU]
Out FF dropout	Uniform(0, 0.5)
NODE	Learning rate	LogUniform(1e-5, 1)
Number of layers	UniformInt(1, 10)
Number of trees	UniformInt(2, 2048)
Depth	UniformInt(1, 10)
Embedding choice	XT, random
Number of layers	UniformInt(1, 8)
Hidden dimension	UniformInt(1, 512)
Dropout	0, Uniform([0,0.5])
Learning rate	LogUniform(1e-5, 1e-2)
L2	0, LogUniform(1e-6, 1e-3)
GraphNet	Hidden dimension	UniformInt(1, 512)
Learning rate	LogUniform(1e-5, 1e-2)
λ	0, LogUniform(1e-5, 1e2)
L1 coefficient	0, LogUniform(1e-5, 1e2)
NC Lasso	Hidden dimension	UniformInt(1, 512)
Learning rate	LogUniform(1e-5, 1e-2)
λ	0, LogUniform(1e-5, 1e2)
L1 coefficient	0, LogUniform(1e-5, 1e2)
Network Lasso	Hidden dimension	UniformInt(1, 512)
Learning rate	LogUniform(1e-5, 1e-2)
λ	0, LogUniform(1e-5, 1e2)
L1 coefficient	0, LogUniform(1e-5, 1e2)
Diet Network
Table 8: Hyperparameter range for all baselines.
B Graph classification approaches
Graph classification models are not appropriate for PLATO's setting. In graph classification models, every input sample is a graph with node attributes, and a model must make a prediction for that graph. The PLATO problem setting breaks key assumptions made by typical graph classification models. First, graph classification models assume that different samples correspond to different graphs [81, 28, 27]. However, in PLATO every sample corresponds to the exact same graph. There is a single background knowledge graph for all samples, and every sample has input features that correspond to the exact same nodes within the knowledge graph. Second, graph classification approaches typically assume that every node in an input graph has a node attribute [81, 28, 27]. However, in PLATO only a small subset of the nodes in the knowledge graph have measured feature values. Finally, graph classification approaches typically assume small graphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes [27]. However in PLATO, the knowledge graph contains 108,447 and the smallest dataset has 12,932 features corresponding to nodes.
C PLATO's performance across node embedding methods for pre-training the feature embeddings
We conduct an ablation study to assess how PLATO's performance depends on the node embedding method used to pre-train the feature embeddings (Methods 3.4). We test three shallow node embedding methods for knowledge graphs which are scalable and prominent: TransE [72], DistMult [78], and ComplEx [65]. We find that PLATO's performance is similar across TransE, DistMult, and ComplEx (Table 9). More generally, PLATO makes no assumption about what type of self-supervised node embedding method is used to pre-train the feature embeddings. The self-supervised embedding step is simply a module that pre-trains feature embeddings which are then passed to the message passing and weight inference modules of PLATO.
KG Node Embedding Method	PearsonR (Test) on BRCA Dataset
TransE	0.582 ± 0.025
DistMult	0.575 ± 0.011
ComplEx	0.583 ± 0.019
Table 9: PLATO's performance is consistent across knowledge graph node embedding methods.
D Rank ordering of methods for datasets with  $d \gg n$