GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data
Authors: Andrei Margeloiu
Nikola Simidjievski
Pietro Lio
Mateja Jamnik
Venue: Accepted by TMLR ()

Abstract: Neural networks often struggle with high-dimensional but small sample-size tabular datasets. One reason is that current weight initialisation methods assume independence between weights, which can be problematic when there are insufficient samples to estimate the model's parameters accurately. In such small data scenarios, leveraging additional structures can improve the model's performance and training stability. To address this, we propose GCondNet, a general approach to enhance neural networks by leveraging implicit structures present in tabular data. We create a graph between samples for each data dimension, and utilise Graph Neural Networks (GNNs) to extract this implicit structure, and for conditioning the parameters of the first layer of an underlying predictor network. By creating many small graphs, GCondNet exploits the data's high-dimensionality, and thus improves the performance of an underlying predictor network. We demonstrate GCondNet's effectiveness on 12 real-world datasets, where it outperforms 14 standard and state-of-the-art methods. The results show that GCondNet is a versatile framework for injecting graph-regularisation into various types of neural networks, including MLPs and tabular Transformers. The code is available at https://github.com/andreimargeloiu/GCondNet.

Published in Transactions on Machine Learning Research (08/2024)
GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data
Andrei Margeloiu
am27700commco uk
Deportment of Computer Science and Technology University of Cambridge, UK
Nikola Simidjievski
TAS ns7770camm.cc
Precision Breast Cancer Institute, Department of Oncology, University of Cambridge, UK Deportment of Computer Science and Technology, University of Combridge, UK
Pietro Liò
pl2190cam.ccm uk
Department of Computer Science and Technology University of Cambridge, UK
Mateja Jamnik
mj201@comm.
Deportment of Computer Science and Technology University of Cambridge, UK
Reviewed on OpenReview: https: // openreview. net/ form?? id= y0000H1ndd
Abstract
Neural networks often struggle with nigh-dimensional but small sample-size tabular datasets. One reason that current weight initialisation methods assume independence between weights, which can be problematic when there are insufficient samples estimate the model's parameters accurately. In such small data scenarios, leveraging additional structures can improve the model's performance and training stability. To address this, propose GCondNet, a general approach to enhance neural networks by leveraging implicit structures present in tabular data. We create a graph between samples for each data dimension, and utilise Graph Neural Networks (GNNs) to extract this implicit structure, and for conditioning the parameters of the first layer of an underlying predictor network. By creating many small graphs, GCondNet exploits the data's high-dimmesiionality; and thus improves the performance of an underlying predictor network. We demonstrate GCondNet's effectiveness on 12 real-world datasets, where it outperforms 14 standard and state-of-the-an methods. The results show that GCondNet is a versatile framework for injecting graph-rogularisation into various lypes of neural networks, including MLPs and tabular Transformers. Code is available at https: /ttps:///ithbb com/ cctp///mnrriimarrelooooo/CCllll
1
Introduction
Tabular datasets are ubiquitous in scientific fields such as medicine (Meira 2001 Balendra & Isaacs
2018
Kelly & Semsarian 2009), physics (Baldi et al.) 2014 Kasieczka et al.) 2021 and chemistry (Zhai et al. 2021 Keith et al. 2021). These datasets often have limited number of samples but large number of features for each sample. This is because collecting many samples is often costly or infcasible, but collecting
many features for each sample is relatively easy. For example, in medicine (Schaefer el. al., 2020 Yang el al,
2012 Gao et al. 20155 Iorio et al. 2016 Garnett et al.] 2012 Bajwa et al.) 2016 Curtis et al. 2012 Tomezaki ct al. 2015), clinical trials targeting rare discases often enrol only few hundred patients most. Despite
the small number of participants, common gather extensive data on each individual, such measuring thousands of gene expression patterns. This practice results in small-size datasets that high-dimensional,

---

Published in Transactions on Machine Learning Research (08/2024)
with the number of features (D) greatly exceeding the number of samples (N). Making effective inferences from such datasets is vital for advancing research in scientific fields.
When faced with high-dimensional tabular data, neural network models struggle to achieve strong performance (Lu et al. 2017 Feng & Simon, 2017), partly because they encounter increased degrees of freedom, which results in overfitting, particularly scenarios involving small datasets. Despite transfer learning's success in image and language tasks (Tan et al. 2018); general transfer learning protocol lacking for tabular data [Borisov et al. 2022). , and current methods assume shared features (Levin et al.; 2023) or large upstream datasets (Wang & Sun 2022 Nam et al. 2022), which is unsuitable for our scenarios. Consequently, we focus on improving training neural networks from scratch.
Previous approaches for training models on small sample-size and high-dimensional data constrained the model's parameters to ensure that similar features have similar coefficients, as initially proposed in (Li & Li 2008) for linear regression and later extended to neural networks (Ruiz et al. (023). For applications in biomed- ical domains, such constraints can lead to more interpretable identilication of genes (leatures) that are biologi- cally relevant (Li & Li 2008). However, these methods require access to external application-speciin knowledge graphs (e.g., gene regulatory networks) to obtain feature similarities, which provide "explicit relationships" be- tween features. But numerous tasks do not have access to such application-specif graphs. We aim to integrate a similar inductive bias, posing that performance is enhanced when similar features have similar coefficients. We accomplish this without relying on "explicit relationships" defined in external application-speciffe graphs.
We propose a novel method GCondNet (Graph-Conditns neural network predictors, such as Multi-layer Perceptrons (MLPs). The key innovation of GCondNet lies in leveraging the "implicit relationships" between samples by performing "soft parameter-sharing" to constrain the model's parameters in principled manner, thereby reducing overfitting. Prior work has shown that such relationships between samples can be beneficial (Fatemi et al., 2021 Kazi et al., 2022; Zhou et al. 2022). These methods, however, typically generate and operate with one graph between samples while relying on additional dataset-specific assumptions such as the smoothness assumption (for extended discussion see Section 4 In contrast, we leverage sample-wise mulliplex graphs, a novel and general approach to identify and use these potential relationships between samples by constructing many graphs between samples, one for each feature. We then use Graph Neural Networks (GNNs) to extract any implicit structure and condition the parameters of the first layer of an underlying predictor MLP network. Note that GCondNet still considers the samples as independent and identically distributed (IID) at both train-time and test-time because the information from the graphs is encapsulated within the model parameters and is not used directly for prediction (see Section 2.2
Networks) enhance the performance of various
We introduce two similarity-based approaches for constructing the sample-wise multiplex graphs from any tabular dataset. Both approaches generate graph for each feature in the dataset (resulting in graphs), with each node representing sample (totalling N nodes per graph). For instance, in a gene expression dataset, we create a unique graph of patients for each gene. Unlike other methods (Ruiz et al.) 2023 Li & Li 2008 Scherer et al., 2022) that require external knowledge for constructing the graphs, our graphs can be constructed from any tabular datasct. We also proposc a decaying mechanism which improves the model's robustness when incorrect relationships between samples are specified.
The inductive bias of GCondNet lies in constraining the model's parameters to ensure similar features have similar coefficients at the beginning of training, and we show that our approach yields improved downstream performance and enhanced model robustness. One reason is that creating many small graphs elfectively "transposes" the problem and makes neural network optimisation more effective because we leverage the digh- of the data to our advantage by gencrating many small graphs. These graphs serve as a large training set for the GNN, which in turn computes the parameters of the MLP predictor. In addition, our approach also models a different aspect of the problem - the structure extracted from the implicit relationships between samples which we show serves as a regularisation mechanism for reducing overfitting.
Our contributions are summarised as follows:
1. We propose a novel method, GCondNet, for leveraging implicit relationships between samples into neural networks to improve predictive performance on small sample-size and high- dimensional tabular data. Our
2

---

Published in Transactions on Machine Learning Research (08/2024)
D features
Weight matrix
W [1]
IR Kx
(2)
MLP
X
(M)
Graphs
Construction
i
2
gD
Predicted label
N
Pu
K
Graph Neural Network
WNNN& E RXD
B
w (1)
(2)
(8)
(D)
Figure 1: GCondNet is general method for leveraging implicit relationships between samples to improve the performance of any predictor network with a linear first layer, such as a standard MLP, on tabular data. (A) Given a tabular dataset X E NND, we generate a graph gj for each feature in the dataset (results in D graphs), with each node representing sample (totalling N nodes per graph). (B) The resulting graphs are passed through a shared Graph Neural Network (GNN), which extracts graph embeddings w(l) E RK from each graph Gj. We concatenate the graph embeddings into matrix WGNN (w("), ..... (P)).
(C) We use WONN to parameterise the first layer WM
MLP of the MLP predictor as convex combination will MLP aWNNN + (1 - Wesattth where W scratth initialised to zero.
method is general and can be applied to any such tabular dataset, unlike other methods that rely on external application-spoccfic knowledge graphs.
We validate GCondNet's effectiveness across real-world biomedical datasets. We show that for such datasets, our method consistently outperforms an MLP with the same architecture and, in fact, outperforms all 14 stale-of-the-arl methods evaluate.
3. We analyse GCondNet's inductive bias, showing that our proposed sample-wise multiplez graphs improve performance and serve as an additional regularisation mechanism. Lastly, we demonstrate that GCondNet is robust lo various graph construction methods, which might also include incorrect relationships.
2 Method
Problem formulation. We study tabular classification problems (although the method can be directly applied to regression too), where the data matrix X = (x. [NN)T E NDD comprises N samples x00 E R2 of dimension D, and the labels are y = "N]
Method overview. Our method applies to any network with linear layer connected the input features, and for illustration, assume MLP predictor network. two components: (i) predictor network (c.g., MLP) that takes input sample (+()) RRD and outputs the predicted label ys; and (ii) Graph Neural Network (GNN) that takes input fixed graphs (D of them) and generates the parameters l ML. for the first input layer of the prodictor MLP network. Note that the GNN is shared across graphs. Since these graphs are fixed for all inputs 2(1, GCondNet maintains the dimensionality of the input space (which remains D).
presents our proposed method, which has
3

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1 Computing WANN.
1: for each feature j = 1, 1....... D do node-embeddings GNN(V);&)) &g)
D V3 represents the nodes, and & represents the edges of the j-th graph 3: w6S) = fagg( (ode--ebeebd)d b Aggregate all node embeddings to obtain the graph embedding w E RR*
end for
5: WONN [w("), (
D Concatenate the graph embeddings
In particular, we parameterise the MLP's first layer MLP as a convex combination of a weight matrix WGNN generated by the GNN (by extracting the implicit structure between samples), and a weight matrix W scratch initialised to zero:
will MLP aWaNN + (1 W scratch
W
(1)
The mixing coefficient determines how much the model should be conditioned on the relationships between samples learnt by the GNN. We schedule a to linearly decay + 0 over na training steps, as further motivated in Section 2.2 We found that GCondNet is robust to fax as supported by the statistical tests in Appendix F.1
Computing WGNN. We train the GNN model concurrently with the MLP predictor compute the weight matrix WGNN. To do this, we use the training split of X to generate a graph Gj = (V,. E,) for each feature in the dataset (resulting in D graphs), with cach node representing sample (totalling N nodes per graph). For example, in gene expression dataset, one graph of patients is created for each gene. All graphs are simultaneously passed to the GNN and are fixed during the training process. This way, we take advantage of high-dimensional data by creating many graphs to train the GNN. We describe the graph construction in Section 2.1 and investigate the impact of this choice in Section 3.2
At each training iteration, we use the GNN to extract graph embeddings from these graphs, as presented in Algorithm [ For each of the D graphs, we first apply the G.NN to obtain node embeddings of size K for all nodes. We then compute graph embeddings w(1) R by using permutation invariant function fags to aggregate the node embeddings. Thus, the graph embeddings are also of size K, which is independent of the number of nodes in the graphs. These embeddings are then concatenated horizontally to form the weight
matrix WONN = [w()), (2() (2) ...., w (D) Finally, we use the resulting matrix WaNN to parameterise the first. layer of the underlying MLP predictor network that outputs the final prediction, as shown in Equation I Appendix A presents the complete pscudocode for training GCondNet.
Test-time inference. We emphasise that the GNN and the associated graphs are employed exclusively during the training phase, becoming obsolete once the mixing coefficient reaches zero after na training steps. The predictor MLP retains its final weights upon training completion, rendering the GNN and graphs unnecessary for test inference. Test input samples are exclusively processed by the predictor MLP resulting in a model size and inference speed identical to a standard MLP.
2.1 Sample-wise Multiplex Graphs
We propose a novel and general graph construction method from labular data, creating multiplex graph g = (G), where each graph layer Gj = Ez),E) represents the relations across feature j and is constructed using only the values X.j of that feature. This enables the use of simple distance metrics between nodes and eliminates the need to work in a high-dimensionall space where distances can be inaccurate. Note that, the graph construction phase is typically fast, performed once per dataset and adds a negligible computational overhead.
The nodes V; of each graph represent the N training samples. The node features are one-hot encoded vectors, with the feature value for a sample located in the corresponding position in the onc-hot encoding. For instance, if the values of feature j of three training samples x(!), ( (3() are
Node features.
X. [z)) (1) (2) (3) then the first node's features would be (ar)] (1) 0, 0), the second node's features would
x xr
be [0, ; 0), and the third node's features would be [0, (0))
(2)
4

---

Published in Transactions on Machine Learning Research (08/2024)
Edges. We propose two similarity- methods for constructing the edges between samples from tabular data, which assume that similar samples should be connected. To measure sample similarity, we calculate the & distances between the feature values X.j- Using the earlier example, if the feature values used to create one graph are X = [m]] .... z. ) then the distances between nodes (based on which we create edges) are ||~]"" my I1,, || I - ar (3) |1, II, and |ar) - my 1l. While in this paper we use & distance, other l11
suitable distance functions can also be applied.
The two types of edges are: (i) KNN graphs connect each node with the closest & neighbours (we set k = 5 in this paper). The memory complexity is O(D-N N K), enabling GCondNet to scale linearly memory-wise w.r.t. both the sample size N and the number of features D. The time complexity for KNN graph construction is O(D . N log N + D.N. - K), with the first term for sorting features and the second for creating edges. (ii) Sparse Relative Distance (SRD) graphs connect a sample to all samples with a feature value within a specified distancc. This process creates a network topology where nodes with common feature values have more connections, and we use an accept-reject step to sparsify the graph (all details are included in Appendix B). The time complexity mirrors that of the KNN graphs. For smaller datasets, the mumber of edges in the SRD graphs is comparable to those of KNN graphs. In larger datasets, KNN graphs are likely more scalable due to direct control over the number of edges via K.
2.2 Rationale for Model Architecture
The inductive bias of GCondNet ensures that similar features have simslar weights in the first layer of the NN at the beginning of training. Uniquely to GCondNet, the feature similarity is uncovered by training GNNs end-to-end on graphs defining the implicit relationships between samples across each feature. Thus, our approach ultimately learns the feature similarity by looking at the relationships between samples. For example, if features i and j have similar values across samples, they define similar graphs, leading to similar graph em- beddings ((()) and wf) These embeddings correspond to the first layer weights will| MLP the neural work.
GCondNet is appropriate when D>> N because it introduces suitable inductive bias that enhances model optimisation, conventional neural approaches (such as an MLP) tend to exhibit unstable training behaviour and/or converge poorly one reason is a large degree of freedom for the small datasets. This happens because; (i) the number of parameters in the first layer is proportional to the number of features; and (ii) modern weight initialisation techniques GGlorot & Bengio 2010 He et al., 2013) assume independence between the parameters within a layer. Although the independence assumption may work well with large datascts, as it allows for flexibility, it can be problematic when there are too few samples to estimate the model's parameters accurately (as we show in Section 3.2). GCondNet is designed to mitigate these training instabilities: (i) by constraining the model's degrees of freedom via an additional GNN that outputs the model's first layer, which includes most of its lcarning parameters; and (ii) by providing more principled weight initialisation on the model's first layer (because at the start we have [i
demonstrate in Section
On small sample-size and tigh-dimensional data,
MLP WGNN).
We parameterise the first layer due to its large number of parameters and propensity to overfit. On high-dimensional tabular data, an MLP's first layer holds most parameters; for instance, on dataset of 20,000 features, the first layer has 98% of the parameters of an MLP with a hidden size 100.
GCondNet still consider the samples as IID at both train-time and test-time. Recall that sam- ples are ITD if they are independent, conditioned on the model's parameters 0 (Murphy) 2022), , so that p (y1, U2 1.(1), (2))) 0) = (yr I (1))) 8) p (yz xx2), 8). Note that unlike distanco-based models (c.g., KNN), our graphs are not used directly to make predictions. In GCondNet, to make prediction, input samples are exclusively processed by the predictor MLP, which uses the same model parameters 0 across all samples. Because all information extracted from our sample-wise graphs is encapsulated within the model parameters 0, the above IID equation holds for GCondNet.
In terms of graph construction, the conventional approach would be to have one large graph where nodes are features and 3) are node embeddings. In contrast, we generate many small graphs and compute un!)) as graph embeddings. Our approach offers several advantages: (i) Having multiple graphs "transposes" the problem and uses the high-dimeesionnality of the data to our advantage by generating many small graphs
5

---

Published in Transactions on Machine Learning Research (08/2024)
which serve as large training set for the GNN. (fi) It allows using simple distance metrics because the nodes contain only scalar values; in contrast, taking distances between features would require working in a high-dimensional space and encountering the curse of dimensionality. (iii) The computation is efficient because the graphs are small due to small-size datascts. (iv) Flexibility, as it can incorporate external knowledge graphs, if available, by forming hyper-graphs between similar feature graphs.
Decaying the mixing coefficient a introduces flexibility in the learning process by enabling the model to start training initialised with the GNN-extracted structure and later adjust the weights more autonomously as training advances. Since the true relationships between samples are unknown, the GNN-extracted structure may be noisy or suboptimal for parameterising the model. At the start, a 1 and the first layer is entirely determined by the GNN (W MLP WGNN). After a becomes O, the model trains as standard MLP (the GNN is disabled), but its parameters have been impacted by proposed method and will reach distinct minimum (evidenced in Scction 3.1 by GCondNet consistently outperforming an equivalent MLP). In contrast to our decaying of the mixing coefficient a, maintaining fixed (similar to PLATO's (Ruiz et al., 2023) inductive bias) leads to unstable training (see our experiments in Section 3.2). Moreover, if a was fixed, it would need optimisation like other hyperparameters, while by decaying a we avoid this tine-coonuming tuning.
3 Experiments
Our central hypothesis is that exploiting the implicit sample-wise relationships by performing soft parameter-sharing improves the performance of neural network predictors. First, we evaluate our model against 14 benchmark models (Section 3.1 We then analyse the inductive bias of our method (Section 3.2)
its effect optimisation, and GCondNet's robustness to different graph construction methods.
Datasets. We focus on classification tasks using small-samplc and high-dimensional datasets and consider 12 real-world tabular biomedical datasets ranging from 72 - 200 samples, and 3312 - 22223 features. We specifically keep the datasets small to mimic practical scenarios where data is limited. See Appendix C for details on the datasets.
Evaluation. We evaluate all models using a 5-fold cross-validation repeated 5 times, resulting in 25 runs per model. We report the mcan + std of the test balanced accuracy averaged across all 25 runs. To summarise the results in the manuscript, we rank the methods by their predictive performance. For each dataset, methods are ranked from 1 (the best) to 12 (the worst) based on their mean accuracy. If two methods have the same accuracy (rounded to two decimals), they obtain the same per-dataset rank. The final rank for each method is the average of their per-dataset ranks, which may be a non-integer.
GCondNet architecture and settings. GCondNet uses an MLP predictor model (as its backbone) with three layers with 100, 100, 10 neurons. The GNN within GCondNet is a two-layer Graph Convolutional Network (GCN) (Kipf & Welling 2017 The permutation invariant function fagy for computing graph embeddings is global average pooling We decay the mixing coefficient. a over n. = 200 training steps, although we found that GCondNet is robust to the number of steps Ra. as supported by the statistical tests in Appendix F.] We present the results of GCondNet with both KNN and SRD graphs. We provide complete reproducibility details for all methods in Appendix D and the training times for GCondNet and other methods are in Appendix
Benchmark methods. We evaluate 14 benchmark models, encompassing a standard MLP and modern methods typically employed for small sample-size and high-dimensional datasets, such as Nettoss [Romero et al. 2017), FsNet (Singh & Yamada 2023), SPINN (Feng & Simon 2017), DNP (Liu et al. 2017). and WPFS (Margeloiu et al. 2023), all of which use the same architecture as GCondNet for a fair comparison.
We also include contemporary neural architectures for tabular data, like TabNet (Anik & Pfister 2021 Tab- Transformer (Huang et al. 2020 Concrete Autoencoders (CAE) (Balin et al. 2019), and LassoNel" Lemhadri
et al. 2021), and standard methods such Random Forest (Breiman 2001) and LightGBM (Ke et al.) 2017).
We also compare the performance of GCondNet with GNNs on tabular data where relationships between samples are not explicitly provided. In particular, we evaluate Graph Convolutional Network (GCN) (Kipf &
Usiin hierarchical pooling (Ying et al. 2018 Ranjan et al. 2020 led unstable training and significantly poorer performance.
discuss LassoNet training instabllites Appendix

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Overall, GCondNet outperforms other benchmark models. We show the classification performance of GCondNet with KNN and SRD graphs and 14 benchmark models on 12 real-world datasets. N/D represents the per-dataset ratio of samples to features. We report the mean + std of the test balanced accuracy averaged over the 25 cross pross-validation runs. We highlight the First, Second and Third ranking accuracy for each dataset. To aggregate the results, we also compute each method's 's average rank across datasets, where a higher rank implies higher accuracy in general. Overall, GCondNet ranks best and generally outperforms all other benchmark methods.
Daraaet
sauk
alamd
ell
gloma 0.111
prostate 0.177
todictt
tego survival
tegp tumnc
zadta dE
macta-p00
AvR.
A/D
0.004
0.009
0.01
0.01
0.00
0.0
0.046
0.048
0.045
0.059
Bank
DictNerwocks
76.4 78.42
62.7
12 10
08.86
D8. 00
SL.laama
82 17
50.0229a
46, 46.07em:
26.98
15,122
50. 200.
8.92
FaNNe
74.52.117
78.00 18.00.19>
60.38
58.17.109
64.74go
60.20 20.20
58.58gma
45.5 45.94.9.
56.92.11
83.86
91. 61.76a30
11:17
DNP
83.1 83.172191
85.13
10. 14.50
5.38
SPINN
83.39.9x
65.91114
00.78
65.05155
75.00
50.02.44
98.50 03.50.41
57.7016.
06.14.12
03.06.13
94.77 94.76.
WPFS
00.8
70.14
80.10
55.00 244
61.541109
61.10g9a
60.57am1
50.00
66.78
11.38516
30.57
20.27
18.68
Th-Traanformer
Th. 1
61 0
70
4
07
56.91 20
10. 00.70km
8.85
CAE
74.18.11.7
30.501120
85.0 a3
71.541244
07.83_114
57.00
60:361:13
00.54..4
40.600
57.35 154
05.78 13.
85 00.00
9.17
LassoNet
50, 53.01gms
51.08
50 00.00
20.17-114
54 54.78gma
26 GT
16.08am1
31. 31.4
49 88.7
L8, al
25 11
15.00
MLP
77.22.16.
66.42
00.30
78.30gs0
73.00:14
68.76g5
08:21gr1
56.36567
48.19
59.56g51
04 94:20249
6.00
Rondom Rureet
81. 81.15
60.82
06.80
76.45gs11
74.17-104
00.35
17-17g7
52.98.2
99-99a7>
88
6.42
LIM
80.79.24
70.07.5*
00.36.53
74.221166
16.50111
01.91114
81,26:12
50.46154
44.99153
67.00
11, 01.42.13
89.79.44
GCN
84.09
65.6
7200
06,21_114
62.00.123
76.18 26.18
58.51
51.01aa2
91.13
30,40
2.75
CATV22
78.67g104
66.00xx
67.74g141
07.07g11
53.23gm05
76.00g/15
50.00g6
45.45gn
64.71gm
98.06
08.33
10.92
GoondNet (KNN)
85.02
65.02,xz
16.18.11
80.70,as
70.67.22
90.38...
.4.
58.62.va
51.70
59.94.ax
95.98,12
95.20...x
GCoudNet (SRD)
86.38geo
68.08g7a
07.56g41
79.52
77.07
50.33gr4
05.25g4
56,36gn
50.82gm
08:24
08.18g4o
96.64gx1
Welling 2017 and Graph Attention Network v2 (GATv2) (Brody et al. 2022 To ensure fairness, we employ a similar setup to GCondNet, constructing a KNN-based graph (k 5) in which each node represents a sample connected to its five nearest samples based cosine similarity, which is well-suited for high-dimensional data. Both GCN and GATv2 are trained in transductive setting, incorporating test sample edges during training while masking nodes to prevent data leakage.
3.1 Overall Classification Performance
Our experiments in Table I show that GCondNet outperforms all 14 benchmark models on average, achieving a better overall rank across 12 real-world datasets suggesting its effectiveness across diverse datasets. GCondNet is followed by WPFS, specialised method for small sample-size and high-ddmensional datasets, although GCondNet consistently putperforms il on 9 out of 12 lasks, providing improvements of up to 7%. Standard methods like LightGBM and Random Forest are competitive and perform well on some datasets, but their relative performance is somotimes highly dataset-dependent. For instance, in the case of the "tega-survival" dataset, GCoudNet (with an MLP backbone) enhances MLP's performance by 2%, but it still lags behind top methods like Random Forest, WPFS, and CAF, which exhibit. an additional 1-7% better performance. This suggests that the additional feature selection capabilities of these approaches can lead to further improvements. While in this work we did not analyse this behaviour in detail, GCondNet, in principle, can readily handle this.
GCondNet consistently outperforms an MLP with the same architecture, showing substantial improvements. The most notable increases of 3-8% are observed in the five most extreme datasets, which have the smallest N/D ratios. On those extreme datasets, GCondNet also improves stability compared to the standalone MLP, reducing the average standard deviation by over 3.5% when using the SRD version and 2.5% when using the KNN version. As the N/D ratio increascs, GCondNet continues to deliver superior performance to the baseline MLP, albeit with similar stability. These results underscore the role of GCondNet's 's inductive bias to mitigating overfitting and enhancing stability, particularly in datasets with extreme N/ D ratios.
We compare against GNNs on tabular data where relationships between samples are not explicitly provided. Despite the advantage of GCN and GATv2 of being trained in a transductive setting, GCondNet outperforms both methods across tasks. The performance gap ranges between 19-25% on three tasks and more than 5% on four other tasks. This indicates that models heavily reliant on latent structure present in tabular data, such as GNNs, are particularly sensitive to misspecifications during model construction. In contrast, GCondNet demonstrates resilience against such misspecifications, which we analyse in the following section.
7

---

Published in Transactions on Machine Learning Research (08/2024)
The results also show that GCondNet outperforms other methods specialised for this data scenario by large margin, such as DietNetworks, FsNet, SPINN, DNP, and more complex neural architectures for tabular data such as TabNet, TabTransformer, CAE and LassoNet. This finding aligns with recent research (Kadra ct al. 2021). suggesting that well-rogularised MLPs are more effective at handling tabular data than intricate architectures. Because our MLP baseline was already well-regularised, we altribule GCondNet's performance improvement to its inductive bias, which serves as an additional regularisation effect, as we further investigate in the next section.
Finally, find that GCondNet consistently performs well with both KNN and SRD graphs, ranking high across various datasets. However, no clear distinction emerges between the two graph construction methods, and in the next section, we further analyse GCondNet's robustness to this choice.
3.2 Analysing the Inductive Bias of GCondNet
Having found that GCondNet excels on small-size and high-dimensional tasks, we analyse its inductive bias and robustness different construction methods.
GCondNet outperforms other initialisation
schemes that do not use GNNs. To under- stand the effect of leveraging the latent relation- ships between samples to parameterise neural net- works (as GCondNet does), we investigate if other weight initialisation methods can imbue a sim- ilar inductive bias. To the best of our knowl-
Normalized balanced accuracy (%)
17
Improvement over a baseline MLP
edge, all such existing methods necessitate ex- ternal knowledge, like in (Li & Li, 2008 Ruiz
et al. 2023). Consequently, we propose three novel weight initialisation schemes incorporating a similar inductive bias as GCondNet, making similar features having similar weights. These schemes generate feature embeddings e(i) which
XMF
PCA
Weisfeiler
RamiLEdge graphs
SRD graphs
KNN graphs
Lehman
are then utilised to initialise the MLP's first layer will
with initialisations
GCondNet MLP)
MLP = (e()), e e!.,,...e (P))) The feature embed- dings are computed using Non-negative matrix factorisation (NMF), Principal Component Anal- ysis (PCA), and Weismnn (WL. algo- rithm (Weisfeiler & Leman 1965). The latter is parameter-fie free method to compute graph embed- dings, often used to check whether two graphs are isomorphic. We apply the WL algorithm to the same SRD graphs as used for GCondNet and use the j-th graph embedding as the feature embed- ding eWL. See Appendix F.3 for more details on these initialisation methods. We follow (Grinsz. tajn et al. 2022) and compute the normalised test balanced accuracy across all 12 datasets and 25 runs. We include the absolute accuracy numbers in Appendix
graph constraction methods
Figure 2: The inductive bias of GCondNet robustly improves performance and cannot be replicated without GNNs. We compute the normalised test bal- anced accuracy across all 12 datasets and 25 runs and report the relative improvement over baseline ML.P. First, we find that GCondNet is robust across various graph construction methods and provides consistent im- provement over an equivalent MLP. Second, to assess the usefulness of the GNNs, we propose three weight initialisa- tion methods designed to emulate GCondNet's inductive biases but without employing GNNs. The results show that GCondNet outperforms such methods, highlighting the effectiveness of the GNN-extracted latent structure.
Figure 2 shows that the specialised initialisation schemes (NMF, PCA, WL) outperform a standard MLP, and we observe that GCondNet with both SRD and KNN graphs further improves over these initialisations. These results suggest that initialisation methods that incorporate appropriate inductive biases, as in GCondNet, can outperform popular initialisation methods, which assume the weights should be independent at initialisation (Gloro & Bengio 2010 He et al.) 2015), which can lead to overfitting on small datasets.
8

---

Published in Transactions on Machine Learning Research (08/2024)
(MLP)
decaying a (GCondNet)
a= 0.2
a= 0.6
a=
gli
Training Loss
Validation Loss
sikk
0.4
0.4
allaml
glioma
0.2
0.2
prostate
0.1
toxicity
0.0
tuga-survival
2000 4000
6000
8000
10000
2000
4000
6000
8000
10000
Training iteration
Training iteration
tega-tumor
meta-dr
GCondNet reduces overfitting. The impact of varying the mixing coefficient a illustrated through the training and validation loss curves (averaged over 25 runs) on 'toxicity.. We train GCondNet with linearly decaying a, along with modified versions with fized Two observations are notable: (i) GCondNet exhibits less overfitting (evident from the converging validation loss) compared to an MLP (a=0), which overfits at the 4,000th iteration; (ii) decaying a enhances the training stability while improving the test-time accuracy by at least 2%.
Figure 3:
meta-p50
lung
0
2
4
6
8
10 12 14
Improvement Test Accuracy (%)
Figure 4: GCondNet versatile and can en- hance various models beyond MLPs. When applied to TabTransformer, GCondNet consis- tently improves performance by up to 14%.
The advantage of using GNNs becomes more evident when comparing the performance of GCondNet, which incorporates learnable GNN, to the Weisfeiler-Lehman method, which is parameter-free method. Although both methods are applied on the same SRD graphs, GCondNet's higher performance by 7% underscores the crucial role training GNNs to distil structure from the sample-wise relationships, exceeding the capabilities of other methods.
GeondNet robust to different graph construction methods. As GCondNet is general and does not rely on knowing the ground-truth relationship between samples, analyse its robustness to the user-defined method for defining such relationships. In addition to the KNN and SRD graphs from Scction 2.1 we also consider graphs with random edges between samples (called "RandEdge" graphs). Specifically, the RandEdge graphs with similar graph statistics to the SRD graphs but random graph edges (full details for creating RandEdge graphs are in Appendix F.3 We train GCondNet on 25 different data splits, and for each split, we sample RandEdge graphs five times resulting in 125 trained models on RandEdge graphs.
generate
We analyse two distinct facets of robustness. First, the relative performonce compared other benchmark modcls. We find that, generally, GCondNet, when using any of the KNN, SRD RandEdge graphs, outperforms all 14 benchmark baselines from Table I and achieves a higher average rank. This suggests that GCondNet is robust to different graph-creation methods and maintains stable relative performance compared to other benchmark methods, even when the graphs are possibly misspecified.
Next, we analyse the absolute performance denoted by the numerical value of test accuracy. As expected, we find that RandEdge graphs are suboptimal (sce Figure 2 and GCondNet performs better with more principled KNN or SRD graphs, which define similarity- based edges between samples. Nonetheless, the three graphs exhibit similar absolute performance with statistical significance (see Appendix F.3). making GCondNet resilient to misspecifications during graph construction. Moreover, the optimal graph construction method is task-dependent: SRD excels in seven datascts, KNN in another four, and RandEdge graphs in one. For instance, even with identical input data X (as in 'Lega-survival" or "tega-lunor') and thus identical graphs the optimal graph construction method can differ based on the prediction task. We believe limitation of this work the lack of an optimal graph construction method. However, having an optimal graph construction method is noiviaivial, as (i) different tasks rely on exploiting different modeling assumptions and structures in the data; and (ii) the GNN's oversmoothing issue (Chen et al. 2020)
9

---

Published in Transactions on Machine Learning Research (08/2024)
lead to computing just an "average" of the feature values. Lastly, we highlight that GCondNet is robust to the number of steps nax as supported by the statistical tests in Appendix F.
GCondNet's inductive bias serves as a regularisation mechanism. To isolate GCondNet's inductive bias which ensures that similar features have similar weights at the beginning of training we train two versions of GCondNet: (i) with decaying a, and (ii) a modified version with a fized mixing coefficient a E {0, 0.2, 0.4, 0.6, 0.8, 1} throughout the training process. As + O, the model becomes equivalent to an 1, the first layer is conditioned on the GNN-extracted structure.
MIP, and
We find that incorporating structure into the model serves a regularisation mechanism and can help prevent overfitting. Figure D shows the training and validation loss curves for different a values. An MLP (equivalent to a 0) begins overfitting at around the 4,000 iteration, as shown by the inflexion point in the validation loss. contrast, all models incorporating the structure between samples (i.e., a > 0) avoid this issue and attain better validation loss.
The optimisation perspective further motivates decaying a rather than using fixed value. Firstly, using fixed during training can lead to instabilities, such as high variance in the training loss. Secondly, fixed results in test-time performance drop compared to the decaying version. In the case of "toxicity" (presented in Figure B) this leads to at least 2% loss in performance compared to the decaying variant with 95.25%, ranging between 91.33% - 93.08% (with no particular trend) for different values of a E {0, 0.2, 0.4, 0.6, 0.8). Training with a fixed a = 1 results a lower accuracy of 84.24%. We posit this occurs because the model is overly constrained on potentially incorrect graphs without sufficient learning capacity for other weights. The model gains flexibility by decaying from 1 0, achieving better generalisation and increased stability.
Extension to Transformers. We highlight that GCondNet is general framework for injecting grapb-regularisation into various types of neural networks, and it can readily be applied to other architectures beyond an MLP. As a proof-of ofrrinncpee,,, we apply GCondNet to TabTransformer (Huang et al. 2020), and the results in Figure 4 show consistent performance improvements by up to 14% (averaged over 25 runs). This highlights that GCondNet is general and can be applied various downstream models.
GCondNet is effective across various sample- to-feature ratios N/D. We compare the per- formance of GCondNet (with MLP backbone
GCondNet relative improvement
10-
and KNN graphs) to an identical MLP baseline. Both models were trained under the same condi- tions. We selected four datasets ("allaml", "cll", "gli", "glioma") cover broad range of N/D ra-
6
tios, keeping the per-dataset number of samples N constant. We adjust the number of features D cordingly for instance, dataset with N 100 samples had D reduced from 10,000 to 20. We split the feature sets by sequentially removing features, re- peating the process five times, and training the MLP and GCondNet with identical features. Full details of the sctup and numerical results are in Appendix G.2
2
0
0.01
0.05
0.2
1.0
Ratio
Figure 5: Improvement of GCondNet (with an MLP backbone) over equivalent MLP baseline. This figure shows the relative increase in test balanced accuracy of GCondNet compared to the MLP baseline, averaged across datasets. GCondNet consistently enhances per- formance across various N/D ratios, demonstrating its advantages even on mimes-iimansoonal data.
Figure 5 shows that GCondNet outperforms baseline MLP by up to 10% (in relative terms) across various sample-to-feature ratios N/D. This indicates that GCondNet can also be effective for small-dimensional data. Moreover, GCondNet demonstrates greater stability and lower variance in performance, even with reduced feature sets.
10

---

Published in Transactions on Machine Learning Research (08/2024)
4 Related Work
Our focus is learning problems on higl-dimensional tabular datasets with a limited number of samples. As such, our method takes inspiration from "diet" methods such DietNetworks (Romero et al. 2017), FsNet (Singh & Yamada 2023) and WPFS (Margeloiu et al. 2023), which rely on auxiliary networks to predict (and in turn reduce) the number of learnable parameters in the first layer of an underlying leed-forward network. However, GCondNet differs from "diet" methods in two important ways: (i) "diet" methods require a well-defined strategy for computing feature embeddings, and their performance is highly sensitive to this choice. In contrast, GCondNet defines graphs between samples and uses a GNN to learn the feature embeddings; (ii) GCondNet provides a different inductive bias which leverages the implicit relationships between samples (via the learned graph embeddings w())).
"Diet" networks:
Out of all "dict" methods, GCondNet is most closely related to PLATO (Ruiz et al. 2023), as both methods employ GNNs as auxiliary networks to parameterise the predictor network. However, the similarities end there, and we highlight two key differences: (i) PLATO relies on domain knowledge, making it inapplicable when such information is unavailable, which is common. In contrast, GCondNet is more general, as it can be applied to any tabular dataset without requiring domain knowledge but can still utilise it when available. (ii) PLATO constructs a single graph between features, whereas GCondNet creates multiple graphs wetween samples. This distinction is crucial, PLATO leverages the relationships among features, while our method focuses on leveraging the relationships between samples (in addition to the relationships between features leartt by the MLP predictor itself).
Graph-based approaches for tabular data, including semi-supervised approaches, construct a graph between samples to capture the underlying relationships. The graphs are created using either a user-defined metric or by learning a latent graph between samples. Recent methods apply GNNs to these graphs, and our work distinguishes itself from such tabular data approaches in three ways.
1. We use GNNs indirectly and only during training to improve an underlying MLP predictor. Once trained, we store the MLP predictor's s final weights, eliminating the need for GNNs during inference. Test input samples are subsequently processed exclusively through the predictor MLP. In contrast, GNN approaches
to tabular data (You et al.) 2020 Wu et al.) 2021 Du et al. 2022 Fatemi et al., 2021 Satorras & Bruna
2018) directly employ GNNs for inference new inputs, including making predictions (You et al.) 2020
Du el al. 2022 Falemi et al. 2021 Salorras & Bruna 2018) and performinng feature imputalion You
et al.) 2020 Wu et al. 2021).
2. Our graph structure is different. GCondNet generates many graphs between samples (one for each feature) and then extracts graph embeddings w(i to parameterise predictor network. This approach
is novel and clearly distinguishes it from other work such (You et al. 2020 Wu et al. 2021 Du et al.
2022), which generate graphs connecting features and samples. Both You et al. 2020 Wu et al. 2021
construct bipartite graph between samples and features, while (Du et al. 2022 creates a hyper-graph
where each sample is a node linked to corresponding feature nodes (specifically for discrete data).
3. Graph-based approaches for tabular data often introduce additional assumptions that may be suboptimal or inapplicable. For example, (Kazi et al., 2022 Zhou et al. 2022 Falemi et al., 2021) create a graph between samples and rely on the smoothness assumption, which posits that neighbouring instances share the same labels. As demonstrated in Section 3.1 such assumptions can be suboptimal for high-dimensional data. Concerning dataset assumptions, (Satorras & Bruna 2018) addresses few-shot learning, which requires a substantial meta-training set comprising similar tasks. In contrast, our approach focuses on learning from small datasets without assuming the presence of an external meta-training set. The work of (Fatemi et al. 2021] infers a latent graph, focusing on either images or tabular data with a maximum of 30 features. In comparison, our research explores tabular datasets containing up to 20000 features.
Feature selection. When faced with high-dimensional data, machine learning models are presented with increased degrees of freedom, making prediction tasks more challenging, especially on small sample-size tasks, To address this issue, various feature selection methods have been proposed to reduce the dimensionality of the data [Tibshiran] 1996. Feng & Simon 2017 Liu et al., 2017 Singh & Yamada 2023) Balm et al. 2019 Margeloiu et al. 2023: Lemhadri et al.] 2021). All these methods aim to model the relationships between
11

---

Published in Transactions on Machine Learning Research (08/2024)
features (i.e., determining which features are similar irrelevant to the task), but they do not consider the relationships wetween samples. In contrast, GCondNet uses a GNN to extract the relationships between samples, while the MLP predictor learns the relationships between features.
Neural networks for tabular data. More broadly, our work is related to neural network methods for tabu- lar data. Recent methods include various inductive biases, such as taking inspiration from tree-based methods (Katzir et al., 2020; Hazimeh et al.] 2020: Popov et al. 2020; Yang et al., 2018), including attention modules (Arık & Pfister 2021 Huang et al. 2020), modelling multiplicative feature interactions IQin et al., 2021). For a recent review neural networks for tabular data, refer to (Borisov et al, 2022). However, these methods are generally designed for large sample size datascts, and their performance can vary on different datasets (Gorishniy et al., 2021), making them unsuitable for small-size and high-dimensional tasks. In contrast, our method is specifically designed for small-size high-dimensional tasks. Lastly, TabPFN (Hollmann) ct al.] 2022) recent pre-trained Transformer using in-context learning for prediction, which can scale only up to 100 features, making it inapplicable for our high-dimensional datasots (of up to 22,000 features).
5 Conclusion
We introduce GCondNet, a general method to improve neural network predictors on small and high-dimensional tabular datasets. The key innovation of GCondNet lies in exploiting the "implicit relationships" between samples by performing "soft parametor-sharing to constrain the model's parameters. We also propose sample-wise mullipplex gruphs, a novel and general approach to identify and use these potential relationships between samples by constructing many graphs between samples, one for each feature. We then use Graph Neural Networks (GNNs) to extract any implicit structure and condition the parameters of the first layer of an underlying predictor network. Unlike other methods, which require external application-specis knowledge graphs our method is general and can be applied to any tabular dataset.
We evaluate 12 classification tasks on biomedical datasets in real applications, this could mean identifying biomarkers for different discases and show that GCondNet outperforms 14 benchmark methods and is robust to different graph construction methods. We also show that the GNN-extracted structure serves as a regularisation mechanism for reducing overfitting. Future work can investigate using the learned structures to obtain insights into the dataset, such as detecting mislabeled data points or outliers.
Broader Impact Statement
This paper presents novel method that aims to advance the field of machine learning by offering new direction for leveraging the implicit sample relationships in machine learning, which is particularly beneficial for data-scarce tasks. This work can also serve as a basis for more interpretable approaches using the learned data structures. For critical domains such as medicine, GCondNet can provide patient/ patient/cohort-wine /cohort-wise insights through post-hoe mechanisms such as graph concept-based explanations (Magister et al., 2021 2022). From a machine learning perspective, GCondNet may also provide valuable insights into the dataset, such as identifying difficult training samples in the context of curriculum learning (Bengio et al. 2009).
Our work's impact is to advance machine learning capabilities in critical fields such medicine and scientific research, particularly in contexts where data availability is limited. By improving model performance in settings with scarce data, our approach supports essential research in early-phase clinical trials (Weissler ct al.] 2021 Zame et al., 2020) where typically only a small number of patients are enrolled and it can help in identilying subtle patterns and relationshipe from small dalasets. By handling complex, high- dimensional datasets, GCondNet can benefit genomics research, enabling better analysis of genetic variations and interactions with limited experimental data, thus supporting the discovery of genetic markers and pathways
Allarbi & Vakanski
2023 Way & Greene 2019 We do not foresee harmful applications for our method.
Acknowledgements
The authors thank Matco Espinosa Zarlenga and Ramon Viñas Torné for their feedback on carlier versions of the manuscript. We also thank Iulia Duta and Pietro Barbiero for their carly discussions on graph neural networks. NS acknowledges the support of the U.S. Army Medical Research and Development Command
12

---

Published in Transactions on Machine Learning Research (08/2024)
of the Department of Defense; through the FY22 Breast Cancer Research Program of the Congressionally Directed Medical Research Programs, Clinical Research Extension Award GRANT13769713, Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.
References
Ludmil B Alexandrov, Serena Nik-Zainal, David C Wedge, Peter J Campbell, and Michael R Stratton. Deciphering signat ures of mutational processes operative human cancer. Cell reports, 3(1):246- 3(1):246-259, 2013.
Fadiyah Ahmed Alharbi and Aleksandar Vakanski. Machine learning methods for cancer classification using gene expression data: A review. Bioengineering, 10, 2023.
Sercan O Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAT Conference on Artificial Intelligence, volume 35, pp. 6679 6687, 2021.
Gagan Bajwa, Ralph DeBerardinis, Baomei Shao, Brian Hall, J David Farrar, and Michelle A Gill. Cutting edge: Critical role of glycolysis in human plasmacyloid dendritic cell antiviral responses. The Journal of Immunology. 196655:200442000) 2016.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physies with deep learning. Nature communications, 5(1):4308, 2014.
Rubika Balendra and Adrian M Isaacs. C9or172-mediaeed ALS and FTD: multiple pathways to disease. Nature Reviews Neurology, 14(9):544-558, 2018.
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference Machine Learning, pp. 444-453. PMLR, 2019.
Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Leurning Research, 17(1):152-166 161, 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Internalional Conference on Machine Learning, 2009.
Arindam Bhattacharjee, William Richards, Jane Staunton, Cheng Li, Stefano Monti, Priya. Vasa, Christine Ladd, Javad Beheshti, Raphael Bueno, Michael Gillette, al. Classification of human lung carcinomas by mrna expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences, 98(24):13790-133 -37999 2001.
Vadim Borisov, Tobias Leemann, Kathrin Sessler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Lco Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Represenlations, 2022.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 3438 3445, 2020.
Christina Curtis, Sohrab P Shah, Suet-Feung Chin, Gulisa Turashvili, Oscar M Rueda, Mark J Dunning, Doug Speed, Andy G Lynch, Shamith Samarajiwa, Yinyin Yuan, et al. The genomic and transcriptomic archilecture of 2,000 breast tumours reveals novel subgroups. Nature, 486(7403):3446 352, 2012.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, and David Paul Wipf. Tearning enhanced representations for tabular data via neighborhood propagation. Advances in Neural Information Processing Systems, 35, 2022.
Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. Advances in Neural Information Processing Systems, 34:22667 22681, 2021.
Jean Feng and Noah Simon. Sparse-input neural networks for high-dimensional nonparametric regression and classification, arXiv preprint
2017.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. RIGM Workshop at the International Conference on Learning Representations, 2019.
William Freije, F Edmundo Castro- Castro-Vargas, Zixing Fang, Steve Horvath, Timothy Cloughesy, Linda Liau, Paul S Mischel, and Stanley F Nelson. Gene expression profiling of gliomas strongly predicts survival. Cancer research, 64(18):6503 6510, 2004.
Hui Gao, Joshua M Korn, Stéphane Ferretti, John E Monahan, Youzhen Wang, Mallika Singh, Chao Zhang, Christian Schnell, Guizhi Yang, Yun Zhang, et al. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nalure medicine, 21(11):1318 1325, 2015.
Mathew J Garnett, Elena Edelman, Sonja Heidorn, Chris D Greenman, Anahita Dastur, King Wai Lau, Patricia. Greninger, Richard Thompson, Xi Luo, Jorge Soares, al. Systematic identification of genomic markers of drug sensitivity in cancer cells. Nature, 483(7391):5700 575, 2012.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedingss of the thirleenth internalional conference artificial intelligence and statistics, pp. 249 256. JMLR. Workshop and Conference Proceedings, 2010.
Yu. V. Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems, 2021.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In Newral Information Processing Systems, Track on Datasets and Benchmarks, 2022.
Christian Haslinger, Norbert Schweifer, Stephan Stilgenbauer, Hartmut Dohner, Peter Lichter, Norbert Kraut, Christian Stratowa, and Roger Abschor. Microarray gene expression profiling of b-cell chronic lymphocytic leukemia subgroups defined by genomic aberrations and vh mulation status. Journal of Clivical Oncology, 22(19):3937 3949, 2004.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pp. 4138 4148. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Noah Hollmann, Samuel G. Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. In International Conference Learning Representations, 2022.
Xin Huang, Ashish Khetan, Milan W. Cvitkovic, and Zohar S. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. Xrv, abs//201.006678, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Francesco Iorio, Theo A Knijnenburg, Daniel J Vis, Graham R. Bignell, Michael P Menden, Michael Schubert, Nanne Aben, Emanuel Gonçalves, Syd Barthorpe, Howard fooh, et al. A landscape of pharmacogenomic interactions in cancer. Cell, 166(3):740 754, 2016.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datascts. Advances in Neural Information Processing Systems, 34, 2021.
Gregor Kasieczka, Benjamin Nachman, David Shih, Oz Amram, Anders Andreassen, Kees Benkendorfer, Blaz Bortolato, Gustaaf Brooijmans, Florencia Canelli, Jack Collins, et al. The Ihe olympics 2020 a community challenge for anomaly detection in high energy physics. Reports on progress in physics, 84(12): 124201, 2021.
Liran Katzir, Gal Elidan, and Ran El- Yaniv. Net-dnf: Effective deep modeling of tabular data. In International Conference on Learning Representations, 2020.
Ances Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael M Bronstein. Differentiable graph module (dgm) for graph convolutional networks. IEEE Transactions on Pattern Analysis ond Machine Intelligence, 45(2):1606-164) 1617, 2022.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 2017.
John A Keith, Valentin Vassilor-Galindo, Bingqing Cheng, Stefan Chmiela, Michael Gastegger, Klaus-Robert Muller, and Alexandre Tkatchenko. Combining machine learning and computational chemistry for predictive insights into chemical systems. Chemicul reviews, 121 (16):9816 9872, 2021.
Matthew Kelly and Christopher Semsarian. Multiple mutations in genetic cardiovascular discase: a marker of disease severity? Circulation: Candiovascular Genelies, 2(2):182-11) 190, 2009.
Hyunsoo Kim and Haesun Park. Sparse non-negative matrix factorizations via alternating non-negativity- constrained least squares for microarray data analysis. Bioinformatics, 23(1)):1495 1502, 2007.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Internation Conference on Learning Representations, 2015.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Internation Conference on Learning Representations, 2017.
Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. Lassonet: Neural networks with fcature sparsity. In International Conference on Arlificial Intelligence and Statistics, pp. 10 18. PMLR, 2021.
Roman Levin, Valeriaa Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. International Conference on Learning Representations, 2023.
Caiyan Li and Hongzhe Li. Network--onsstrained regularization and variable selection for analysis of genomic data. Bioinformatics, 24(9):1155-1182 2008.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):94, 2018.
Arthur Liberzon, Chet Birger, Helga Thorvaldsdóttir, Mahmoud Ghandi, Jill Mesirov, and Pablo Tamayo. The molecular signatures database hallmark gene set collection. Cell systems, 16)::417-425, 2015.
Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample size data. In International Joint Conference on Artificial Intelligence, pp. 2287-2293, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, and Pictro Lio'. Geexplainer: Chuman-in-the-oon concept-based explanations for graph neural networks. TCML. Workshop on Human in the Loop Learning, 2021.
Lucie Charlotte Magister, Pietro Barbiero, Dmitry Kazhdan, Federico Siciliano, Gabriele Ciravegna, Fabrizio Silvestri, Mateja Jamnik, and Pietro Lio. Encoding concepts in graph neural networks. arXiv preprint arXiu:2207.13586, 2022.
Andrei Margeloiu, Nikola Simidjievski, Pietro Lio, and Mateja Jamnik. Weight predictor network with feature selection for small sample tabular biomedical data. AAAI Conference Artificial Intelligence, 2023.
Lisiane B Meira, Autonio MC Reis, David L Cheo, Dorit Nahari, Dentis K Burns, and Errol C Friedberg. Cancer predisposition in mutant mice defective in multiple genetic pathways: uncovering important genetic interactions. Mutation Research/Fundee Funddmmena and Molecular Mechanisms of Mutagenesis, 477(1-2):51-55)
2001.
Kevin P Murphy. Probabilistic machine learning: an infroduction. MIT press, 2022.
Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tabular learning with senf-gene tasks from unlaboled tables. In International Conference on Learning Representations, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference Learning Represenlations, 2020.
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc-Alexander Najork. Are neural rankers still outperformed by gradient boosted decision trees? In International Conference on Learning Representations, 2021.
Ekagra Ranjan, Soumya Sanyal, and Partha Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. Proceedings of the AAAI Conference on Artificial Intelligence, 34: 5470-5477, 2020.
Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc- André Legault, Marie-Pierre Dubé, Julie G. Hussin, and Yoshua Bengio. Diet networks: Thin parameters for fat genomics. In International Conference on Learning Representations, 2017.
Camilo Ruiz, Hongyu Ren, Kexin Huang, and Jure Leskovec. Tabular deep learning when >> n by using an auxiliary knowledge graph. In Admances Neural Information Processing Systems, 2023.
Victor Garcia Satorras and Joan Bruna. Few-shot learning with graph neural networks. International Conference on Learning Representations, 2018.
Julia Schaefer, Moritz Lehne, Josef Schepers, Fabian Prasser, and Sylvia Thun. The use of machine learning in rare discases: scoping review. Orphanet journal of rare diseases, 15:1-10, 2020.
Paul Scherer, Maja Trebacz, Nikola Simidjievski, Ramon Viñas, Zohreh Shams, Helena Andres Terre, Mateja Jamnik, and Pictro Liò. Unsupervised construction of computational graphs for gene expression data with explicit structural inductive biases. Bioinformallics, 38(5):1320 38(5):7330-1327, 2022.
Dinesh Singh and Makoto Yamada. Fsnet: Feature selection network on high digh-dimensional biological data. International Joint Conference Neural Networks (IJCNN), 2023.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Dinesh Singh, Phillip G Febbo, Kenneth Ross, Donald G Jackson, Judith Manola, Christine Ladd, Pablo Tamayo, Andrew A Renshaw, Anthony V D'Amico, Jerome P Richie, et al. Gene expression correlates of clinical prostate cancer behavior. Cancer cell, 1(2):203 209, 2002.
Avrum Spira, Jennifer E Beane, Vishal Shah, Katrina Steiling, Gang Liu, Frank Schembri, Sean Gilman, Yves-Martine Dunas, Paul Calner, Paola Sebastiani, al. Airway epithelial gene expression in the diagnostic evaluation of smokers with suspect lung cancer. Nature medicine, 13(3):361-366, 2007.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhuldinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey deep transfer learning. In Artificial Newral Networks and Machine Learning-ICANN ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Port III 27, pp. 270-279. Springer, 2018.
Leo Taslaman and Björn Nilsson. A framework for regularized non-negative matrix factorization, with application to the analysis of gene expression data. PloS one, 7(11):c46331, 2012.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journel of the Royal Statistical Sociely: Series B (Methodological), 58(1):267-28) 1996.
Katarzyna Tomezak, Patrycja Czerwiíska, and Maciej Wiznerowiez. The cancer genome atlas (Loga): an immeasurable source of knowledge. Contemporary oncology, A961), 2015.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902 2915, 2022.
Gregory P Way and Casey S Greene. Discovering pathway and cell type signatures in transcriptomic compendia with machine learning. Annual Review of Biomedicaal Dala Science, 2:1 17, 2019.
Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. nli, Series, 2(9):12-10 16, 1968.
E Hope Weissler, Tristan Naumann, Tomas Andersson, Rajesh Ranganath, Olivier Elemento, Yuan Luo, Daniel F Freitag, James Benoit, Michael C Hughes, Faisal Khan, et al. The role of machine learning in clinical research: transforming the future of evidence generation. Trials, 22:1 15, 2021.
Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards open-world fcature extrapolation: An inductive graph learning approach. Advances in Neural Information Processing Systems, 34:19435 19447, 2021.
Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally sparse neural networks for tabular biomedical data. In International Conference on Machine Learning, pp. 25123 25153. PMLR, 2022.
Wanjuan Yang, Jorge Soares, Patricia Greninger, Elena Edelman, Howard Lightfoot, Simon Forbes, Nidhi Bindal, Dave Beare, James A Smith, I Richard Thompson, et al. Genomics of drug sensitivity in cancer (gdse): a resource for therapeutic biomarker discovery in cancer cells. Nucleic acids research, 41(D1): 2012.
Yongxin Yang, Trene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. International Conference Machine Learning Workshop on Human Interpretability in Machine Learning (WHT), 2018.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Newral Information Processing Systems, 190877 2020.
17

---

Published in Transactions on Machine Learning Research (08/2024)
William R Zame, Ioana Bica, Cong Shen, Alicia Curth, Hyun-Suk Lee, Stuart Bailey, James Weatherall, David Wright, Frank Bretz, and Mihaela van der Schaar. Machine learning for clinical trials in the era of covid-19. Statistics in biopharmaceutical research, 12(4):506 517, 2020.
Zenan Zhai, Christian Druckenbrodt, Camilo Thorne, Saber A Akhondi, Dat Quoc Nguyen, Trevor Cohn, and Karin Verspoor. Chemtables: datasct for semantic classification tables in chemical patents. Journal of Cheminformalics, 13(1):1-22 20, 2021.
Kaixiong Zhou, Zirui Liu, Rui Chen, Li Li, Soo- Choi, and Xia Hu. Table2graph: Transforming tabular data to unified weighted graph. In International Joinl Conference on Arlificial Intelligence, 2022.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix: GeondNet
Table of Contents
A GCondNet Training Pseudocode
20
Sparse Relative Distance (SRD) Graph Construction
20
Datasets
21
D Reproducibility: Benchmarks methods, Training Details and
22
Computational Cost
E.1 Training time
25
E.2 Number of steps for convergence
25
Ablations on GCondNet
Ablation number of steps for decaying the mixing coefficient
26
Vary the size of backbone's first layer
26
.3
Ablation GCondNet's graph construction method and the MLP weight initialisation
27
GCondNet beyond small-sample and high-dimensional data
29
Varying the dataset size N.
29
Vary the sample-to-feature ratio N/D
31
19

---

Published in Transactions on Machine Learning Research (08/2024)
A GCondNet Training Pseudocode
Algorithm Training GCondNet
Input: training data X E RNXD, training labels E RR classification network fo graph neural network 96NNN node aggregation function Sage, graph creation method h(-), steps for linear decay Ra
1: for each feature j = 1,2, 2..., D do
G ,(x,,)
Generate the sample-wise multiplex graphs
3: end for
Wscratch =
Initialise auxiliary weight matrix
5: for each mini- batch B = ((x(1)) Mi bi)) do
6:
each feature j = 1,,.... . do
7:
node-embeddingg J0NNN (Gz)
8: w fanu (nn(enddeeembeeddngs)) Aggregate node embeddings obtain graph embedding w E RRK
9: end for
10:
WGNN = (1) w(() (,.,),..., w.., (D)
Horizontally concatenate the graph embeddings
11:
max(0, (i/na))
b Compute mixing coefficient. b. Compute the weight matrix of the first layer
12:
MILP +- aWGNN + (1 a scratch
Make Will| MLP the weight matrix of the first layer of feuu
13:
14:
for each sample i = 1,2,... do
D. = (x )
16:
end for
17:
ÿ (21.82...... [6]
Concatenate all predictions
18:
Compute training loss L CrossEntrropyLosss 9)
19:
Compute the gradient the loss w.r.t.
20:
PMPP, AGNN, W scratch using backpropagation
21:
Update the parameters:
OMPP OMIP VASLEL BGNN + fCNNNN VOGNN L
23:
Wecratch Wecratch VWaccates L
25: end for
Return: Trained models feaar 96GNN and W [2] MLP
B Sparse Relative Distance (SRD) Graph Construction
We propose novel similarity-based method, Sparse Relative Distance (SRD), for creating edges between node (representing samples) in graph. It assumes that similar samples should be connected and use this principle to create edges Ej in the jth graph. The method also includes an accept-reject step to sparsify the graph. Specifically, SRD work as follows:
For each node create a set of candidate edges C₁ by identifying all samples 1 with a feature value within a certain distance, dist, of the corresponding feature value of sample i. Specifically, we include all samples I such that Xx, Xt,I S dist, where dist is defined as 5% of the absolute difference between the 5th and thth percentiles of all values of feature (to eliminate the effect of outlier feature values).
2. Perform Bernoulli trial with probability size(C,)/ Nerain for each node If the trial outcome is positive, create undirected edges between node i and all nodes within the candidate set C. If the outcome is negative, new edges created. This sampling procedure results in sparser graphs, which helps alleviate the issue of oversmoothing (Chen et al. 2020 commonly encountered in GNNs. Oversmoothing occurs when the model produces similar embeddings for all nodes in the graph, effectively 'smoothing out' any differences in the graph structure. It is worth noting that a node can also acquire new edges as part of the candidate set of other nodes.
This process results a network topology in which nodes with larger candidate sets are more likely have more connections. Intuitively, this means that 'representative' samples become the centres of node clusters in the network.
20

---

Published in Transactions on Machine Learning Research (08/2024)
3. To further prevent oversmoothing, if node has more than 25 connections, we randomly prine some of its edges until it has exactly 25 connections. This is because samples with highly frequent values can have an excessive number of connections, which can result in oversmoothing.
Table B.1: Key statistics of the graphs created using the Sparse Relative Distance (SRD) with dist = 5%.
Dalaset
Node degree
Edges of full graph (%)
ell
3.88 + 6.81
9.96
lung
5.16 9.46
7.37
meta-p50
3.97 + 6.73
5.56
meta-dr
3.92 6.69
5.48
prostate
11.28 + 17.96
31.77
smnk
3.94
5.93
tcga-survival
12.03
10.77
toga-tumor
10.42
loxicity
3.42
5.12
C
Datasets
Table C.2: Details of the 12 real-world biomedical datasets used for experiments. The datasets contain between 72-200 samples, and the number of features is 17 262 times larger than the number of samples.
# samples (N)
# features (D)
D/N
# classes
# samples per class
Dataset allaml
72
7129
99
2
25, 47
cll
111
11340
102
3
11, 49, 51
gli
85
22283
262
2
26, 59
glioma
50
4434
89
4
7, 14,
14, 15
lung
197
3312
17
4
17, 20,
21, 139
meta-dr
200
4160
2
61, 139
meta-p50
200
4160
21
2
33, 167
prostate
102
5966
2
50, 52
smkk
187
19993
107
2
90, 97
tega-survival
200
4381
22
2
78, 122
tega-tumor
200
4381
25, 51,
toxicity
171
5748
34
4
39, 42, 45, 45
All datasets are publicly available and summarised in Table C.2 Eight datasets open-source Li et al. htmlonline: CLL-SUB- ccaleed 'cll') (Haslinger et al. 20044), GLI 85 (called 'gli') (Freije et al.) 2004). lung (Bhattacharjee
2018
and available https::/jundong::: http://jjunnnngl...hhh.. githubt.bbb......
111
ct al.
2001) Prostate_ GE (called 'prostate") (Singh et al. 2002). SMK-CAN-187 (called 'smk') (Spira
et al.
2007). ) TOX-171 (called 'toxicity') (Bajwa et al. 2016 as well as allaml and glioma, for which the
original reference was not available.
We created four additional datasets following the methodology presented in Margeloiu al.
2023
Two datasets from the METABRIC (Curtis al. 2012) dataset. We combined the molecular data with the clinical label 'DR' to create the 'meta-dr' dataset, and we combined the molecular data with the clinical label PPam50Subtype' to create the 'meta-p50' dataset. Because the label 'Pam50Subtype' was very imbalanced, we transformed the task into a binary task of basal vs non-basal by combining the classes 'LumA', LLumB', 'Her2', 'Normal' into one class and using the remaining class 'Basal' as the second class. For both 'meta-dr' and 'meta-p50" we selected the Hallmark gene set (Liberzon et al. 2015 associated with breast cancer, and the new datascts contain 4160 expressions (features) for each patient. We created the final datasets by randomly sampling 200 patients stratified because we interested in studying datasets with small sample-size.
21

---

Published in Transactions on Machine Learning Research (08/2024)
Two datasets from the TCGA (omezak et al. 2015 dataset. We combined the molecular data and the label B22r.RF..uree to create the "tega-survival dataset, and we combined the molecular data and the label 'tumor Borrreggdeeeee to create the 'tcga-tumor' dataset. For both "toga-survival" and "tega-tumor" we selected the Hallmark gene set (Liberzon et al., 2015) associated with breast cancer, leaving 4381 expressions (features) for each patient. We created the final datasets by randomly sampling 200 patients stratified because we are interested in studying small sample-size datascts.
Dataset processing. Before training the models, we apply Z-score normalisation to each dataset. Specifically, on the training split, we learn a simple transformation to make each column of Xtrain E R- x D have zero mean and unit variance. We apply this transformation to the validation and test splits during valsaaanii
D Reproducibility: Benchmarks methods, Training Details and Hyper-parameters
Software implementation. We implemented GCondNet using PyTorch 1.12 Passke et al. 2019p, an open- source deep learning library with a BSD licence. We implemented the GNN within GCondNet, and the GCN and GATv2 benchmarks using Py
(Fey & Lenssen 2019), an open-source library for implementing Graph Neural Networks with an MIT licence. We train using a library stttp:///githe: www wightiing AI/ lightning settorch-lgghhnngg built on top of PyTorch and released under an Apache Licence 2.0. All numerical plots and graphics have been generated using Matplotlib 3.6, a Python-based plotting library with a BSD licence. The model architecture Figure [ was generated using poctps://gitain www.aau...oommaaaaaa coom/jgrap///ae drawio draw.io, a free drawing software under Apache License 2.0.
As for the other benchmarks, we implement MLP, CAE and Nettoos using PyTorrc 1.12 Paaske et al.) 2019), Random Forest using scikit-learn (Pedregosa et al.) 2011) (BSD license), LightGBM using the lightgbm library (Ke et al. 2017) (MTT licence) and TabNet. (Arik & Pfister 2021 using the https: //githu.. http//rreauuurrraaee wwwaaaaaaaaaaaataaeaaaaaaaaaaeee implementttmmeee (MITT licence) from Dreamquark AI. We use the MIT- licensed
implementation of WPFS made public by (Margeloiu et al. 2023). We re-implement FaNet (Singh & Yamada 2023) in PyTorrh 1.12 (Paszke et al. 2019) because the official code implementation contains differences from the paper, and they used a different evaluation setup from ours (they evaluate using unbalanced accuracy, while we run multiple data splits and evaluate using balanced accuracy). We use the of LassoNet (MIT licence), and the (no licence).
official implementation official implementation of SPINN
Computing Resources. All our experiments are run on a single machine from an internal cluster with a GPU Nvidia Quadro RTX 8000 with 48GB memory and an Intel(R) Xeon(R) Gold 5218 CPU with 16 cores (at 2.30GHz). The operating system was Ubuntu 20.4.4 LTS. We estimate that to carry out the full range of experiments, comprising both prototyping and initial experimental phases, we needed to train around 11000 distinct models, which we estimate required 1000 to 1200 GPU hours.
GeondNet architecture and settings. GCondNet is leveraging an MLP backbone model of three layers with 100, 100, 10 neurons. After each linear layer we add Leaky ReLU non-linearity with slope 0.01, batch normalisation (Toffe & Szegedy) 2015) and dropout (Srivastava et al. 2014): we tune the dropout probability p P {0.2, 0.4] on the validation accuracy. The last layer has softmax activation. The layers following the first one are initialized using a standard Kaiming method (He et al.] (015)), which considers the activation. The GNN within GCondNet is Graph Convolutional Network (GCN) Kipp & Welling 2017) with two layers of size 200 and 100. After the first GCN layer, we ReLU non-linearity and dropout with p 0.5. The permutation invariant function Jaee, for computing graph embeddings is global average pooling?
We intend GCondNet's computation overhead to be minimal, thus our GCondNet retains the same training arper-parameters as the optimised backbone MLP model. We initially performed hyperparameter tuning on an MLP. A After selecting the optimal hyperparameters, we retrained the GCondNet with the same ML. backbone and training settings, using two graph construction methods, resulting in two variants: GCondNet (KNN) and GCondNet (SRD), as shown in Table 1 Consequently, the time needed to tune GCondNet is
Using hierarchical pooling methods YYing et poor performance.
2018
Ranjan et al. 2020 resulted unstable training and significantly
22

---

Published in Transactions on Machine Learning Research (08/2024)
essentially equivalent to tuning the backbone model. Specifically, we train GCondNet for 10000 steps with batch size of 8 and optimise using AdamW (Loshchilov & Hutter 2019) with a fixed learning rate of le - 4. We decay the mixing coefficient a over na = 200 training steps, although we found that GCondNet is robust to the number of steps na: as supported by the statistical tests in Appendix F.I We use early stopping with patience 200 steps on the validation loss across all experiments.
Training details for all benchmark methods. Here, we present the training settings for all benchmark models, and we discuss paper-parameter tuning in the next paragraph. We train using 5-fold cross-validation with 5 repeats (training 25 models each run). For each rum, we select 10% of the training data for validation. We perform a fair comparison whenever possible: for instance, we train all models using a weighted loss (e.g,, weighted cross-entropy loss for neural networks), evaluate using balanced accuracy, and use the same classification network architecture for GCondNet, MLP, WPFS, FsNet, CAE and DietNetworks.
WPFS, DietNetworks, CAE, FsNet have three hidden layers of size 100, 100, 10. The Weight Predictor Network and the Sparsity Network have four hidden layers 100, 100, 100, 100. They are trained for 10,000 steps using early stopping with patience 200 steps on the validation cross-entropy and gradient clipping at 2.5. For CAE and FsNet we use the suggested anncaling schodule for the concrete nodes: exponential annealing from lemperature 10 to 0.01. On all datasets, Diet.et performed best. with not decoder. For WPFS we use the NMF embeddings suggested in Margeloiu et al. 2023
For GCN, we used two graph convolutional layers with 200 and 100 neurons, followed by a linear layer with softmax activation for class label computation. ReLU and dropout with p = 0.5 follow each convolutional layer. We train using Adamn (Loshchilov & Hutter, 2019), tune the learning rate in [lc 3.3e 3, le 4], and select the best model on the validation accuracy.
For GATv2, we used two graph attentional layers with dropout p = 0.5. The first attention layer used attention heads of size 100, and the second used one head of size 400. ReLU and dropout with p= 0.5 follow each attention layer. A linear layer with softmax activation for class label computation follows the attention layers. We train using AdamW (Loshchilov & Hutter 2019), tune the lcarning rate in [le [ 3, 3, 4), and select the best model on the validation accuracy.
LassoNet has three hidden layers of size 100, 100, 10. We use dropout 0.2, and train using AdamW (with betas 0.9, 0.98) and a batch size of 8. We train using weighted loss. We perform early stopping on the validation set.
For Random Forest, we used 500 estimators, feature bagging with the square root of the mumber of features, and used balanced weights associated with the classes.
For LightGBM we used 200200 estimators, feature bagging with 30
For TabNet, we use width 8 for the decision prediction layer and the attention embedding for each mask (larger values lead to severe overfitting) and 1.5 for the feature re-usage coefficient in the masks. We use three steps in the architecture, with two independent and two shared Gated Linear Units layers at each step. We train using Adam (Kingina & Ba, 2015) with momentum 0.3 and gradient clipping al 2.
For Tranfaorfrr,rr, we use only the head for continuous features, as our datasets do not contain categorical features. For a fair comparison, we use the same architecture as the MLP following the initial layer normalization, and train the model with the optimal settings of the MLP.
For SPINN, we followed the results from the ablations in the original paper and tuned the sparse group lasso hyper-parameter X E {0, 0.001, 0.0032, 0.1), the group lasso haper-parameter a E [0.9, 0.99, 0.999} in the sparse group lasso, the ridge-param do {0, 0.0001}, and train for at most 1,000 steps.
For DNP we did not find any suitable implementattinn, and used SPINN with different settings as a proxy for DNP (because DNP is a greedy approximation to optimizing the group lasso, and SPINN optimises directly group lasso). Specifically, our proxy for DNP results is SPINN trained for at most 1000 iterations, with a = 1 for the group lasso in the sparse group lasso, ridge-param An = 0.0001, and we tuned the sparse group lasso hyper-paranen parameter X e {0, 0.001. 0.0032, 0.1}.
dyper-parameter tuning. For each model, we use random search and previous experience to find good range of hyper-parameter values that we can investigate in detail. We then performed a grid search and
23

---

Published in Transactions on Machine Learning Research (08/2024)
ran 25 runs for each hyper-parameter configuration. We selected the best arper-parameter based the average validation accuracy across the 25 runs.
For the MLP and DittNesworks we individually grid-searched learning rate E {0.003, 0.001, 0.0003, 0.0001 } batch size E {8, 12, 16, 20, 24, 32), dropout rate E {0, (0,0.1,0.2 0.4, 0.5). We found that learning rate 0.003, batch size and dropout rate 0.2 work well across datasets for both models, and we used them the presented experiments. In addition, for Diet.Ntworks we also tuned the reconstruction hypee-parameter À {0, 0.01, 0.03, 0.1, 0.3, 3, 10, 30} and found that across dataset having À O performed best. For WPFS we used the best papemmeerseetess for the MLP and tuned only the sparsity hyper-paramete X 3e we grid-search the reconstruction parameter in À E {0, 0.2, 1, 1,5} and learning rate in {0.001, 0.003). For LightGBM we performed grid-search for the learning rate in {0.1, 0.01) and maximum depth in {1,2]. For Random Forest, we performed grid search for the maximum depth in (3,5,7) and the minimum number of samples in a leaf in (2.3). For TabNet we scarched the learning rate in f0.01, 0.02, 0.03} and the À sparsity motivated by (Yang et. al. 2022 For GCN and GATv2 searched the learning rate in {le- 3, 3e - 1e 4}. We selected the best prper-parametess (Tabll D.3 on the weighted cross-cntropy (exeept Random Forest, for which we used weighted balanced accuracy).
3e 6, 3c - 5. 4, 3, le 2} and the size of the feature embedding E {20, 50, 70). For FsNet
hyper-parametter in {0.1, 0.01, 0.001, 0.0001],
Table D.3: Best performing ppper-parameters for each benchmark model across datasets.
Random Forest
LightGBM
TabNet
max depth
min samples leaf
learning
maa depth
learning
spansity
leaziigg
learning
leurning
A
anocaling iterations
spansity
embedding
recnuteaction
size
alland
3
3
0.1
2
0.03
0.0001
0.001
0.003
0.003
0
1000
0
50
cll
3
2
0.03
0.001
0.003
0.003
0.003
1000
70
gli
3
0.1
1
0.03
0.1
0.001
0.003
0.003
300
30
glioma
3
[101
2
1.03
(.001
0,313
0.003
0,003
0
1000
Je-
30
lung
0.001
mcta-p30
2
0.01
2
0.02
0.001
0.303
0.003
0,003
0
1000
de-
30
meta di
0.1
0.03
0.1
0.003
0.003
0.003
300
50
poostate
2
0.01
0.0011
0.0001
0.003
1000
smk
0.1
2
0.03
0.001
0.301
0.0001
0.003
0
1000
3e 5
30
tege-survival
0.01
0.003
0.003
300
tega-tumor
0.1
1
0.02
0.01
0.003
0.001
0,003
0
300
de 5
30
texicity
3
0.1
2
0.03
0.1
0.303
6.0001
0.001
0.2
1000
36
50
LassoNet unstable training. We used the official implementation of LassoNet /https://githiii com/ lasso-net/Ise netllasonttaa))) and we successfully replicated some of the results in the LassoNet paper (Lemhadri et al. 2021]. However, LassoNet was unstable on all datasets we trained on. In our experiments, we grid-searched the In penalty coefficient E (0.001, 0.01, 1, 10, 'auto') and the hierarchy coefficient. M 1, 3, 10). These values are suggested in the paper and used in the examples from the official codebase. For all pyperrparammter combinations, LassoNet's performance was equivalent to random classifier (e.g., 25% balanced accuracy for a 4-class problem).
24

---

Published in Transactions on Machine Learning Research (08/2024)
E
Computational Cost
We present the time and number of steps required to train GCondNet. Note that we designed GCondNet to use the same training typerparameters as the optimised backbone MLP model. Therefore, when implementing Net one should first perform hyperparameter tuning on the backbone model, which is quicker than training GeondNet. After selecting the optimal hyperparameters, retrain GCondNet using the same backbone and training settings, but choose graph construction method (e.g., KNN or SRD). Consequently, the time for the backbone model, with the only difference being
needed to lune GCondNet is essentially the the final one-off GeondNet training, for which show the computational overhead here.
E.1 Training time
Table E.4: Average training time (in minutes) of a model with optimal hyper- typer-parameters. In general, GCondNet trains slower than the benchmarks: GCondNet takes 8.5 minutes train across datasets, other competitive "diet" networks, such as WPFS, take 7.7 minules, and an MLP takes 5.4 minutes.
Dataset
GCondNet
MLP
WPFS
DietNetworks
FsNet
12
6.1
7.7
8.1
5.4
lung
11
6.1
9.9
10.6
6.1
meta-p50
9.8
6.6
8.9
8.7
4.3
meta-dr
3.6
4.6
4.9
4.8
prostate
9.6
7.2
9.9
7.9
3.5
smk
6.2
8.9
5.9
4.3
tega-survival
3.8
3
4.2
4.7
5.3
toga-tumor
4
3.5
5
4.5
toxicity
13.7
7.3
10
8.2
5.9
Average minutes
8.5
5.4
7.7
4.9
E.2 Number of steps for convergence
Table E.5: Number of steps for convergence.
MLP
GCondNet with different graphs
KNN graphs
SRD graphs
RandEdge graphs
ell
1697
4020
4526
1298
lung
2836
6927
6777
6935
meta-p50
2952
5504
5365
5242
meta-dr
1331
1657
1577
1598
prostate
4307
4540
4535
smk
1379
2175
2102
2218
tega-survival
1320
15411
1606
1588
tega-tumor
1316
1805
1790
1805
toxicity
2807
6522
6809
6772
Average
1998
3829
3899
3887
25

---

Published in Transactions on Machine Learning Research (08/2024)
F Ablations on GCondNet
F.1 Ablation number of steps for decaying the mixing coefficient
Table F.6: We evaluate the impact of decaying the mixing coefficient for varying number of steps na- We present the mean=std balancod validation accuracy averaged over 25 runs. We find that GCondNet is robust to the decay length Ray and we choose 200 steps for all experiments of this paper unless otherwise specified.
Steps of linear decay for 200
Dataset
100
100
ell
88.88 + 8.47
88.09 + 9.34
88.33 + 8.28
lung
96.77 + 5.07
97.27 + 4.97
97.36 = 4.76
meta-p50
98.56 1 3.70
98.56 + 3.70
97.74 +: 5.02
mela-dr
71.20 8.80
68.36 -1 10.63
71.92 1 8.39
prostate
95.56 + 7.35
98.97 +
95.11 + 7.99
smk
76.39 + 11.59
76.89
75.78 = 11.64
toga-survival
69.20 + 11.38
9.35
68.53 = 10.01
tega-tumor
62.06 13.64
62.46
66.13 = 15.17
toxicity
98.75 + 3.12
98.25 + 3.83
98.5 = 3.73
Table F.7: Statistical analysis of the number of iterations from Table F.6 We report the p-values of
Wilcoxon signed-rank test (Demšar 2006 Benavoli et al. 2016 The results support our observation, namely that GCondNet is robust to the hyper-pane parameetrreer fa.
100 200
na 100 vs. na 400
200 vs. 400
4.00E-01
9.44E-01
4.96E-01
F.2 Vary the size of backbone's first layer
The GNN component of GCondNet outputs a matrix WGNN E RRDD where K is the size of the first hidden layer of the underlying backbone network (e.g., an MLP), and is the number of features. The graph embeddings w E R described in Algorithm (line 3) match the size K of the backbone MLP model; of GCondNet.
hence, K is determined by the backbone network, not
We assoss the effect of GCondNet when the backbone network varies in width. We vary the size K of the backbone MLP's first layer and apply GCondNet (with KNN graphs), which computed graph embeddings of size K accordingly. Figure F. and Table F.8 demonstrate that GCondNet consistently enhances performance across different values of K. Notably, GCondNet greatly improves stability, yielding comparable outcomes across various network widths, unlike the MLP, which exhibits substantial performance decrease in narrower networks.
Table F.8: Numerical results for Figure F.1 comparing the performance when varying the size of the first. hidden layer of an MLP, and GCondNet with an identical MLP backbone.
Dataset
gli
allaml
ell
glioma
Size first layer
MLPP
GCondNet
MLP
GCondNet
MLP
GCondNet
MLP
GCondNet
20
75.38
84.43
78.18415.09
96.80
67.97=7.48
81.21
66.00
78.00
50
79.98
84.02_9.58
95.07
97. 97.58:4.11
73. 4..49
81. 81.2946.98
56.50416.92
74.50
100
77.72+15.30
85.02=9.00
91.30+a.m
96. 96.18.4900
78. 78.30=9.00
80.70+5.30
73.00
76.67
150
19.88
84.92
91.64
95.1626
81.0627.22
80.6018.32
71.50
67.33
200
81.03
85.6641n.04
91.89+10.50
96.58
80. 26.94
81.2846.57
74.33 88
80.83 11.60
26

---

Published in Transactions on Machine Learning Research (08/2024)
MLP
GCondNet
gli
allaml
cll
g glloma
Test ballaced accuracy (%)
90
70
60
50
40
20
100 150
200
50
100
150
200
20 50 100
150
200
20
50
100
150
200
Size first hidden layer (K)
Size first hidden layer (K)
Size first hidden layer (K)
Size first hidden layer (K)
Figure F.1: Comparing the performance when varying the size of the first hidden layer of an MLP, and GCondNet with an identical MLP backbone. GCondNet consistently enhances performance across different values of K and it greatly improves stability, yielding comparable outcomes across various network widths.
F.3 Ablation on GCondNet's graph construction method and the MLP weight initialisation
RandEdge Graphs from Section 3.2 The proposed SRD method creates graphs that contain, on average,
8% of the edges of a fully connected graph (as shown in Appendix By. To create RandEdge graphs with similar statistics, we sample a proportion p from all possible edges in the graph, where p ~ NN(s 0.08, 0.03) is sampled for each of the D graphs. We used the same initial node embeddings from Section 2.1 We sample each graph five times and train each of the 25 models all graphs resulting in 125 trained models on RandFdge graphs.
MLP using any of the three initialisation methods that propose below. To mitigate the risk of exploding gradients, adopt the method proposed by He et al. (He et al., 2015) to rescale the weights. After computing w!ll MLPs then perform zero-centring on each row of will MI.P and subsequently rescale it to match the standard deviation of the Kaiming initialisation (He et al. 2015). The resulting matrix is used to initialise the first layer of the predictor MLP.
Specialised initialisations from Section 3.2 We first compute w
Principal Component Analysis (PCA) initialisation. We use PCA to compute feature embeddings (j)
ePCa for all features j- These embeddings are then concatenated horizontally to form the weight matrix of the first layer of the MLP predictor w-ll| MLP = jePCA SPCA .... epcAl-
Non-negative matrix factorisation (NMF) initialisation. NMF has been applied in bioinformatics to cluster gene expression (Kim & Park 2007 Taslaman & Nilsson 2012) and identify common cancer mutations (Alexandrov et al. 2013). Tt approximates X R WH, with the intuition that the column space of W represents "cigengenos", and the column H.j represents coordinates of gene j in the space spanned by the eigengenes. The feature embedding is e()) = H. These embeddings are then concatenated horizontally to form the weight matrix of the first layer of the MLP predictor will
(D) MLP = [eMM:; eNMP: [NMP]
3. Seisfeiler-Lehmarr (WL) initialisation. The WL algorithm (Weisfeiler & Leman 1968 often used in graph theory, is method to check whether two given graphs are isomorphic, i.c., identical up to a renaming of the vertices. The algorithm creates a graph embedding of size N. For our use-case, the embeddings must have size K, the size of the first hidden layer of the predictor MI.P; thus, we obtain the (f)
feature embeddings ewr. by computing the histogram with K bins of the WL-computed graph embedding, which is then normalised to be a probability density. We apply the WL algorithm on the SRD graphs
described in Scction 2.1 and finally obtain WL [e GWL e12
(1) (2)
27

---

Published in Transactions on Machine Learning Research (08/2024)
Table F.9: We evaluate the robustness of GCondNet on various graph construction methods and compare it to identically structured and trained MIP initialised with three weight initialisation methods emmlating the inductive biases of GCondNet, but without training GNN. Here, all versions of MLPs and GCondNet use a fixed dropout p = 0.2. We report the mean = std of the test balanced accuracy averaged over 25 runs and include statistical tests in Table F.10] Results show that GCondNet robust to the graphs and consistently outperforms a standard MLP across all graph construction methods. Further, GCondNet often outperforms other initialisation methods, highlighting the effectiveness of the GNN-extracted latent structure.
MLP
MLP with specialised initialisations
GCondNet with different graphs
PCA
NMF
WL
RandEdge
KNN
SRD
allaml
91.30 26.4
95.4945.97
95. 95.8446.4
15.33
96 96.3914.90
96.18
46.777
ell
78.30
79.9226.18
78.5926.64
79.98
81.36
80.7025.47
21.54
gli
#1..33
12.305
85.58 49.38
83.79 10.77
85 85.94
85.51 #s.96
86.36
glioma
73.00 14.88
75.00 15.
74.67 133.55
20.50g1...m
75.05 112.00
76.67 12.90
77.50
lung
94.201.44 4.95
96, 04.14.00
95. 95.05:: 05.14.00
94. 5.7.077
94. 16..8614.50 58
94.68 4.45
95.34414.49
meta-dr
55.56
65.75+68.27
59.36
68.69:27.0
57 .8.99 #8.79
59.34
58. 28.24
meta-p50
94.31
94. 7.
95.0944.sM
95.81
95 86...643
96.37
96.2613.79
prostate
88.7645.55
91.04+5.08
80.36-2644
89.97
80 59.56626.77
90.38
80.96=a.14
smk
64.42:8.44
66.79
65.87
64.47
66. 13.11388.12.
65.92+s.08
68.08=7.51
tega-survival
366.28+6.78
55.22+7.a9
60.08
34..79.s.28
58.31
58.66 38.61+#7.01
56.36:99.4
toga-tumor
48.19
50.95
51.49.1977
49.67 1..86
51.57 19
31.70.s.ee
52.43327.57
toxicity
93.21 16.14
92.58 #5.46
80.1 80.11
92.49
95 05.06
95.22+s3.9
95.25
Average rank
6.08
4.42
4.33
5.17
3.17
2.75
2.08
Table F.10: Statistical analysis of Table F.9 We compare both GCondNet w/ SRD and KNN variants to each other, to GCondNet with RandomEdge graphs, and to the three weight. inilialisations methods we proposed. We perform statistical testing using the Wilcoxon test (Demšar 2006; Benavoli et all 2016). The results support our discussion that GCondNet is robust to the graph construction method, as the performance differences between different graph constructing methods are not significant at a = 0.05. Furthermore, we find that both GCondNet variants (SRD and KNN) perform significantly better than the MLP baseline and the PCA and WL initialisations.
Model A
Model B
Wilcoxon p-vauue
GCondNet (KNN)
MLP
9.7656e-04
GCoudNet (KNN)
MLP (NMF)
1.0934o-01
Net (KNN)
MLP (PCA)
3.418e-012
GCondNet (KNN)
MLP (WL)
4.88286-04
GCondNet (SRD)
MLP
3.418e-03
GCoudNet (SRD)
MLP (NMF)
1.29396-01
Get (SRD)
MLP (PCA)
1.6113e-12
CondNet (SRD)
MLP (WL)
3.418c-03
GCondNet (SRD)
Net (KNN)
6.8894e-01
GCoudNet (SRD)
GCondNet (RandEdge)
1.29396-01
GCondNet (KNN)
GCondNet (RandEdge)
5.9335e-01
28

---

Published in Transactions on Machine Learning Research (08/2024)
G GCondNet beyond small-sample and high-dimensional data
G.1 Varying the dataset size N
We assess the scalability of GCondNet in comparison to its backbone model. For this evaluation, we use an MLP backbone and train a GCondNet with KNN graphs. Both GCondNet and the baseline MLP are trained under the same conditions using the same dataset splits.
Here, we adopt different evaluation setup from the main paper facilitate a fair and robust comparison across various dataset splits. Specifically, we examine two datasets from Metabric, which contains 2,000 samples, and two from TCGA, which contains 500 samples. We reserve 20% of the total data for testing, creating test set of 400 samples for Metabric and 100 samples for TCGA. We use this fixed test set to evaluate the models across training subsets of varying sizes. We use the remaining 80% of the data to create training sets of different sizes. Specifically, we reduce the training set size by sequentially decreasing the number of samples from the larger preceding subsets without introducing new samples. We conduct five- fold cross-validation for each resulting training subset, training on four folds and using the other fold for validation.
Sctup:
Results: In terms of accuracy, Figure G.2 (numerical results in Table G.11 illustrates the test accuracy when increasing the training dataset size N. GCondNet consistently outperforms the MLP on datasets prone to high overfitting, such as 'lega-survival', 'Loga-tumor', and 'meta-dr', with average performance improvements of 5.7%. On 'meta-pam50", the MLP occasionally outperforms GCondNet; however, the predictive performance on this dataset is already high and nearly saturated. Overall, GCondNet proves effective and can enhance performance beyond very small datasets.
In terms of training time, Figure G.3 (numerical results in Table G.1l indicates that GCondNet sometimes exhibits slightly longer convergence time, primarily due to the complexities involved in training the auxiliary GNN component and generally slower convergence. However, the increase in runtime does not disproportionately escalate with larger datasets. The overhead remains relatively constant across varying dataset sizes, averaging just 20% more training time, which indicates that GCondNet scales effectively without. significant size-dependent overhead, while providing performance improvements.
MLP
GCondNet
tcga-survival
tcga- (cga-tumor
meta-dr
meta-pam50
65
65
Test balanced accuracy (%)
60
55
60
60
55
55
90
50
88
100
200
300
400
100
200
300
400
100
500
1000
1500
100
500
1000
1500
Training set size
Training set size
Training set size
Training set size
Figure G.2: Test accuracy when increasing the training dataset size N, showing the med across four datascts. GCondNet outperforms the MLP on datascts prone to high overfitting, such as 'tega-survival; 'toga-tumor', and 'meta-dr'. On 'meta-pan50', the MLP sometimes outperforms GCondNet; however, note that the predictive performance this dataset is already high and nearly saturated. Overall, GCondNet proves effective and can enhance performance beyond very small datasets.
29

---

Published in Transactions on Machine Learning Research (08/2024)
MLP
GCondNet
tcga-survival
tcga-tumor
meta-dr
meta-pam50
250
400
800
300
Runtime (s)
200
300
600
150
200
200
100
100
100
200
50
100
200
300
400
100
200
300
400
100
500
1000
1500
100
500
1000
1500
Training set size
Training set size
Training set size
Training set size
Figure G.3: Training time with increasing dataset size, showing the meanistd across four datasets. GCondNet generally requires slightly more time to converge, but the overhead is relatively constant across different dataset sizes, indicating that GCondNet scales well and does not incur significant size-dependent overhead.
Comparison of MLP and GCondNet (with an MLP backbone) on datasets of varying sizes (N): We report the meantsstd of test. balanced accuracy (%) and the average training time over five runs. GCondNet generally improves the baseline MLP's performance, though it introduces some computational overhead which is generally constant independent of the dataset size.
Table G.11:
Dataset
Dataset (N)
Model
Test Accuracy Runtime (m)
MLP
50,93
100
GoondNet
55.20
2.3
MI.P
50.36
2.0
togn-survival
200
GCondNet
56.0224.5
MLP 50.41
2.8
300
GooudNet
57.88
4.0
MLP 56.38
2.9
100
GCoudNet
57.55
MLP 42.06
1.9
100
Get
51.32+4.84
3.0
MLP 47.50
3.3
togu-tumne
200
GCoadNet
14.20
MLP
52.33
2.8
GCondNet
57.48
4.7
MLP
56.32 15:11
3.1
400
GCondNNet
54.57 24.45
4.5
MLP
51.07
3.1
GCondNet
19.91
3.7
MLP
54.2612.48
2.8
250
GCondNet
59.35
3.6
metu-dr
ML.P
53. 23.29
2.5
GCondNet
60.73+2.5
3.9
MLP
56.95
1000
GetadNNt
61.20+3.a0
5.1
MLP 56.5443.05
4.9
1500
GotudNet
61.18+3.15
5.6
MLP
89. 20.8642.7
8.0
100
89.49+2.00
6.0
ML.P
90.11
9.0
GCondNet
90.8611.58
8.3
mell--pam
ML.P
91.33+100
500
GeoadNet
90.831212
8.2
MLP
92.17 1os1
9.5
1000
GoondNet
91.7840ur
8.7
MLP
11.8
1500
GCondNet
92.59
11.4
30

---

Published in Transactions on Machine Learning Research (08/2024)
G.2 Vary the comple-to-featuree ratio N/D
Assessing the impact of the sample-to-feature ratio N/D: Table G.12 presents the complete numerical results, while Figure G.4 visually represents these results, highlighting that GCondNet improves both accuracy and stability across various N/D sizes.
To isolate the impacts of GCondNet's 's inductive bias and regularisation, we compare the performance of GCondNet (with an MLP backbone and KNN graphs) to an identical MLP baseline. Both models were trained under the same conditions, as presented in the paper. We select four datasets ("allaml", "ell", "gli", "gliomaa) to examine broad range of N/D ratios (0.01, 0.05, 0.2, 1, 5), keeping the per-dataset number of samples N constant (presented in Appendix C We adjust the number of features D accordingly- instance, a dataset with N = 100 samples had D reduced from 10,000 to 20. Note that this reduction led to a performance loss for both models, which is expected, as reducing the feature set might also remove important features. However, here we are interested in the relative performance of GCondNet w.r.t to the baseline MLP.
We took several measures ensure robust results:
1. We vary the N/D ratio by changing the size of D. Particularly, increasing the ratio is done by reducing the feature subset D, where smaller feature subsets were derived by removing features from the proceding larger subsets (e.g., (d1, d2, d3, da] - [d1, d22 da] - [d1, - (d2]); ensuring that no new feat were introduced during the reduction process.
2. We created feature subsets for each dataset separately, repeating the process five times.
3. Both GCondNet and the MLP were trained using the same feature sets. For cach configuration encompassing model, dataset, feature-subsel, and Saature-subset-rppeaa we trained 25 distinct models (via 5-fold vross-validation, repeated times), resulting in 125 runs per entry in our results table. Overall, this setup led to the training and evaluation of 5,000 models in total.
Notably, GCondNet demonstrates greater stability and lower performance variability, even with reduced feature sets (Figure G.4.. For instance, in the "cll" dataset at N/D ratios of 0.05 and 0.2, GCondNet exhibits smooth performance transitions, contrasting with the fluctuating trends observed the MLP baseline. These findings underscore the effectiveness of GCondNet's inductive bias across diverse datasets and feature configurations.
Table G.12:
Assessing the impact of the sample-to-featee balanced accuracy (%) of an MLP and a GCondNet with an equivalent MLP backbone, averaged over 125 runs. We adjust the N/D ratio by holding the number of samples N constant and varying the number of features D. The results show that GCondNet's inductive bias predominantly enhances performance across different N/D ratios by up to 10, except for "glioma" when N/D E {1,5}, where it slightly reduces accuracy.
ratio N/D. We presented the mcans: std test
Dataset
gli
allaml
glioma
N/D ratio
MLP
GCondNet
MLP
GCondNet
MLP
GCondNet
MLP
GCondNet
0.01
82.30
84.46s8.11
91.67
96.61
78.38
21.03
73. 13.13412.77
76. 13
0.05
10.52212.80
83.91
85.28 15.28
94.652a.12
72.23
80.76
72.67 14.23
75.60
0.20
78.57:12.56
81.77
84.99
90.05
72.95
78, 78.4916.52
69. 69.40
72.57
1.00
74.14418.43
75.97+13.30
74.85+13.55
81.22
68.02247.00 88
68. 68.02
58. 58.37+17.85
57.67=17.05
5.00
39.70+13.40
64.98
61.89
66.13_14488
56.59 16.19
60.93
46.83
11.17
31

---

Published in Transactions on Machine Learning Research (08/2024)
MLP
GCondNet
gli
allaml
cli
glioma
Test balanced accuracy (%)
100
60
0.01
0.05
0.2
1
0.01
0.05
0.2
5 0.01
0.05
0.2
5 0.01
0.05
0.2
1
N/D ratio
N/D ratio
N/D ratio
N/D ratio
Figure G.4: Assessing the impact of the samplo-to-feature ratio N/D. We presented the meant: std test balanced accuracy (%) of an MLP and GCondNet with an equivalent MLP backbone, averaged over 125 runs. We adjust the N/D ratio by holding the number of samples N constant and varying the number of features D. The results show that GCondNet's inductive bias predominantly enhances performance across different N/D ratios by up to 10,, except for "glioma" when N/D E {1,5}, where it slightly reduces accuracy.
32


Reviews:
[{"review_id": "bbNWqq7RLS", "reviewer": "Reviewer_mcza", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think this work is solid and interesting. The experiments are convincing and well-designed, with a good discussion over the different hyper parameters. Moreover, the paper is very clear and tackles a frequent and difficult problem in data science, namely the difficulty to handle high-dimensional datasets with neural networks. The weaknesses I identified were only about a few unclear points in the exposition, that I would like the authors to adress for the final version (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this article, the authors propose a new method to deal with high-dimensional datasets using graph neural networks (GNNs). When the number of dimensions, or features, exceeds the number of instances, the authors make use of this large dimensionality to train a GNN architecture that can generate feature-specific vectors on top of nearest neighbors graphs computed directly from the features. These vectors are then concatenated into a matrix, which itself serves as an initialization of some multilayer perceptron (MLP) architecture. At training time, these MLP parameters are computed as a linear combination of the GNN-based matrix and some standard, learnable parameter matrix. Moreover, the linear combination coefficient corresponding to the GNN-based matrix decreases to zero over iterations, so that the GNN-based matrix is not optimized and used after a while. The authors then demonstrate the usefulness of their approach by showing improvement (in terms of accuracy) over standard baselines on several high-dimensional datasets. They also show that such GNN-based vectors can also be incorporated to more general achitectures (such as transformers), that decaying the linear combination coefficient is indeed more efficient than keeping it fixed as it allows to avoid overfitting, and that GNN-based vectors are more efficient than other types of feature-specific vectors obtained with, e.g., NMF and PCA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Section 2.1: the construction of the kNN and SRD graphs is a bit unclear, as the distances used to build the graphs are not specified. I figured that these distances were simply obtained from the absolute differences between the feature values on the instances, but it is not clear from the text as it looks that the distances were computed out of the graph node features (which are one-hot encoding vectors), which does not make a lot of sense. This part should be more explicit.\n\n2. The claim that decaying alpha reduces overfitting is well supported by Figure 3, but does not show so clearly from Table 1. I expected that the overfitting reduction would also induce smaller standard deviations on the accuracies but it seems that it is not the case (like on the glioma dataset). It would be good to add a discussion / explanation of this observation, as the results are currently a bit confusing w.r.t. that aspect.\n\n3. About the results presented in Figure 3, the text mentions a test performance drop of 2% when fixing alpha, but I could not find the numbers in the figure (which only presents train and validation losses), nor in Table 1 (in which alpha is decayed). It would be nice to include the performance values in the text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tNK1D7hnWt", "reviewer": "Reviewer_nQny", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n1. The novel use of implicit sample-wise relationships and graph-based regularization provides a clear improvement over existing methods.\n\n2. The method’s effectiveness is well-demonstrated with extensive experiments on multiple real-world datasets, showing significant gains in performance.\n\n3. The paper is clearly written, with a thorough explanation of the methodological innovations and the underlying theoretical considerations.\n\n\n\n[Weaknesses]\n\n1. The training time for GCondNet is longer than for benchmark methods, which could limit its practicality in some scenarios.\n\n2. The definitions of “small” and “large” in terms of sample size and feature number are vague and need clarification to better assess the method’s applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework called GCondNet for enhancing neural networks by leveraging implicit relationships in high-dimensional tabular data. It constructs multiple graphs corresponding to each data dimension and utilizes a Graph Neural Network to condition the initial layer parameters of the base predictor model. This strategy improves performance and robustness across various real-world datasets, particularly in scenarios with limited samples but numerous features. The approach is generic, not requiring external knowledge graphs, which broadens its applicability across different domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The claim at the end of the first paragraph in Section 2.1, stating that graph construction for each task takes five seconds, appears to be based on a specific experimental setup not yet introduced. Please clarify whether this timing is general or pertains to a particular dataset described later, and if so, detail this setup earlier to avoid confusion.\n\t\n2. The paper notes increased training times for GCondNet compared to benchmark methods. Please explain the reasons for this slower performance. Additionally, it would be helpful to discuss how the method scales with larger datasets—does the increase in the number of samples or features exacerbate the training time disproportionately?\n\t\n3. Given the method’s outlined suitability for datasets with many features but fewer samples, could you provide examples or scenarios where GCondNet underperforms? Specifically, insights into its performance on datasets with a large number of samples or fewer features would be valuable for understanding its limitations.\n\n4. The determination of the parameter $K$ in the graph neural network setup remains unclear. Could you elaborate on how $K$ is chosen and discuss whether the model’s performance is sensitive to variations in $K$? This information would aid in assessing the robustness and applicability of GCondNet across different settings.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f27ORdlcOB", "reviewer": "Reviewer_x2re", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and the presented ideas are easy to follow\n2. The idea presented in the paper is quite novel and shown to effective empirically.\n3. The evaluation is extensive and shows improvements over several baselines. The study on inductive bias compared to other initialization methods such as NMF and PCA is quite interesting. The proposed approach is shown to be effective when used in combination with multiple architectures KNN, SRD, Transformers, etc.   \n\nQuestions:\n1. I’m curious to see at what value of N/D the method will become less effective. A study on increasing the value N/D until GCondNet is not useful will be quite interesting.\n2. Training time for GCondNet is longer than other baselines considered. Would training a existing baselines  longer or baselines  with larger number of parameters be better in practice if given equal training time?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents GCondNet an approach for using implicit structure in the tabular data for improving neural network training. This proposed approach is designed for scenarios where small number tabular data is available with large number of features. The key idea is to introduce the inductive bias in the model’s training by ensuring similar features have similar coefficients early in the training. Many small graphs are built as a GNN initially and the similarity information from the features flows through the GNN to the model being trained. The evaluation shows that this approach  is quite effective on several benchmarks compared to large number of baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see questions above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Both the reviewers and the editors agree that the paper is interesting and relevant. The only request I have is that the authors carefully re-read their text and correct a few lingering typos and more carefully edit the text answered in response to the reviewer's comments.
Decision: Accept (poster)