The Role of Feature Interactions in Graph-based Tabular Deep Learning
Authors: Elias Dubbeldam
Reza Mohammadi
Marit Schoonhoven
Ilker Birbil
Venue: Accepted by TMLR ()

Abstract: Accurate predictions on tabular data rely on capturing complex, dataset-specific feature interactions. Attention-based methods and graph neural networks, referred to as graph-based tabular deep learning (GTDL), aim to improve predictions by modeling these interactions as a graph. In this work, we analyze how these methods model the feature interactions. Current GTDL approaches primarily focus on optimizing predictive accuracy, often neglecting the accurate modeling of the underlying graph structure. Using synthetic datasets with known ground-truth graph structures, we find that current GTDL methods fail to recover meaningful feature interactions, as their edge recovery is close to random. This suggests that the attention mechanism and message-passing schemes used in GTDL do not effectively capture feature interactions. Furthermore, when we impose the true interaction structure, we find that the predictive accuracy improves. This highlights the need for GTDL methods to prioritize accurate modeling of the graph structure, as it leads to better predictions

Published in Transactions on Machine Learning Research (03/2026)
The Role of Feature Interactions in Graph-based Tabular Deep Learning
Elias Dubbeldam University of Amsterdam
e.f.dubbeldam@@wo..t
Reza Mohammadi University of Amsterdam
a.mohamaadi@nuvv.l
Marit Schoonhoven
m.schoookooven@hwv...
University of Amsterdam
Ilker Birbil University of Amsterdam
s. birbbl@@@uu...
Reviewed on OpenReview: https: //
// openreview. net/ forum? id- o dud ZHZ
Abstract
Accurate predictions tabular data rely capturing complex, datasct-specific feature interactions. Attent ion-based methods and graph neural networks, referred to graph-based tabular deep learning (GTDL), aim to improve predictions by modeling these interactions as a graph. In this work, we analyze how these methods model the feature interactions. Current GTDL approaches primarily focus on optimizing predictive accuracy, often neglecting the accurate modeling of the underlying graph structure. Using synthetic datasets with known ground-truth graph structures, we find that current GTDL methods fail to recover meaningful feature interactions, as their edge recovery is close to random. This suggests that the attention mechanism and message-passing schemes used in GTDL do not effectively capture feature interactions. Furthermore, when we impose the true interaction structure, we find that the predictive accuracy improves. This highlights the need for GTDL methods to prioritize accurate modeling of the graph structure, as it leads to better predictions.
1 Introduction
Deep learning has achieved remarkable success in domains such as natural language processing and computer vision. On tabular data, however, deep learning methods still struggle to compete against traditional,
tree-based machine learning methods (Grinsztajn et al. 2022 McElfresh et al. 2024 Although recent advances in tabular deep learning occasionally surpass these baselines on select benchmarks (e.g., Gorishniy et al. 2021;; ; Hollmann et al. (2025)), no deep learning method has yet demonstrated consistent superiority
across datasets and evaluation settings (Grinsztajn et al.) 2022 McElfresh et al. 2024).
Tabular data is characterized by the heterogencous nature of its features: each feature often encodes distinct. semantics, and relationships among features (or feature interactions) can be complex, indirect, and dataset- specific. By modeling the feature interactions, one incorporates the inductive bias (domain-specific principles embedded into the model's architecture (Goyal & Bengio, 2022 Battaglia et al., 2018 Romero Guzman 2024g) that features interact with each other differently. With 'modeling' we mcan that the network has, by design, separate parameters for each feature interaction. Using inductive biases has proven to be important for success in other fields of deep learning. For example, convolutional neural networks (CNNs) achieve sample officiency and robustness in computer vision by encoding translational invariance (LeCun al. 1989. Fukushima, 1980), and transformers excel in natural language processing using attention- attention-based mechanisms to capture sequential and contextual relationships (Vaswani al. 2017

---

Published in Transactions on Machine Learning Research (03/2026)
Modeling this inductive bias of feature interactions comes naturally in the form of graph, where the nodes represent. features and the edges their interactions. Probabilistic graphical models (PGMs) have rich history in statistics, providing framework to model multivariate dependencies (Lauritzen, 1996). These methods excel at robustly describing the graph structure while enabling predictions, yet they lack the ability to model complex nonlinear relationships that deep learning can provide. Graph-based tabular deep learning (GTDL) methods aim to merge the expressive power of deep learning with graph-structured feature representations. Feature graph neural networks (GNNs), reviewed by Li et al. (2024), are GNNs focused on tabular data, having the features as nodes and the feature interactions edges However, how these feature interactions are modeled within these networks has not been extensively studied or evaluated. These methods do not evaluate explicitly whether their learned feature interactions accurately correspond to meaningful relationships in the data.
Existing GTDL mothods (e.g., Li et al. (019a Yan et al. (0023 Zheng et al. (023); Villaizán- Vallelado et al. (2024); Ye et al. [2024]: Zhouu et al. (2022)) typically evaluate the learned graph structure only qualitatively,
datasets rarely include ground-truth ura interaction graphs. Their training loss tied to predictive performance, providing no incentive to ensure accuracy meaningfulness in the learned graph structure. As result, the adjacency matrix may reflect optimization artifacts rather than genuine feature interactions, as sketched in Figure [ This emphasis on predictive metrics over structural fidelity constrains interpretability. Overall, there remains a lack of systematic techniques for both validating learned feature interactions and guiding learning with prior knowledge.
Girue
Iuput features
GTDL model
Target feature
Tabular data
b
U
A Lrue
A Aprod
X
A Apred # Atrue
Figure 1: (a) A true underlying graph structure generates tabular data. (b, c) Only the data is used to train GTDL model (using fully connected graph) to predict the target feature. (d) After training, the learned graph structure is extracted from the model. (e) When this predicted graph structure is compared to the true graph structure, we find that they are not similar for existing GTDL methods.
In this work, we analyze whether existing GTDL methods learn meaningful feature interactions. Do GTDL methods learn accurate graph structure when being trained to predict target feature? Conversely, does an exclusive focus on predictive accuracy lead to spurious interactions rather than genuine feature dependencies, thereby undermining robustness, gencralization, and explainability? When the graph structure is modeled correctly, does the predictive performance of GTDL methods improve?
To support our analysis, we take the following steps. In Section we review the existing literature on GTDL methods and identify their limitations in the evaluation and validation of learned feature interactions. In Section we introduce framework to systematically evaluate how well predictive GTDL methods learn the graph structure. The two key parts of this framework are (i) synthetic datasots with known ground-truth graphs, and (ii) a metric to quantitatively assess the accuracy of learned graphs. In Section we discuss the results of controlled experiments to showcase the framework. Specifically, we show that existing GTDL methods fail to recover meaningful feature interactions, and that enforcing the true interaction structure improves predictive performance. In Scction 5 we give a concluding discussion and propose future rescarch directions.
3
This categorization of feature graphs contrasts with instance graphs, where nodes represent instances (rows) and edges capture relationships between those instances.
2

---

Published in Transactions on Machine Learning Research (03/2026)
2 GTDL methods and their feature interactions
After formalizing the problem context, we organize this section around methods that model feature interactions in tabular data through feature graphs. First, we review attention-based methods and GNNs for tabular data, and discuss how both are interpreted as GTDL methods. Next, we relate GTDL, to PGMs, which serve as principled baseline for learning conditional independence structure. Lastly, we investigate methods that may appear related at first glance but, upon closer examination, are not directly pertinent to this study,
Problem setting and notation. A full tabular dataset D = [ar ll y] E Raxp consists in a traditional supervised setting of imput features x E Rax(x-1) and a target feature E R=x1, with n the number of samples, p the number of features and || indicating concatenation. When we refer to 'features', we mean both input and target features. Features could be either numerical or categorical.
The features and their interactions can be represented as an undirected graph G = (V, where V is the set of nodes and E the set of edges, such that the number of nodes [V] = p. A binary symmetric adjacency matrix Atrue describes the true graph structure. The absence of an edge between two nodes indicates the conditional independence of these two nodes conditioned on all other nodes (this is different from direeted causal graphs, where the presence of an edge indicates a direct causal effect between two nodes.)
From trained GTDL methods, we can extract a weighted adjacency matrix E 188P, where 0 S A. S indicates the strength of the interaction between features and j, with i,j E (1.....p). As the interaction from feature to j could be different. from the interaction from feature j to the weighted adjaceney matrix is not necessarily symmetrie: While A; can differ from Ajs: they should be symmetric in support (i.e., they share the same sparsity pattern). Specifically, if Atrueii O, then it should be that Aij, Aji = 0. Conversely, if Auru, = 1, then it should be that Agj Agi > 0.
The main task is to predict the target feature y given the input features x. Correct modeling of the feature interactions could improve this prediction, including both target-input interactions (relationships between y and each ny)) and input-input interactions (relationships between pairs of input features x, and 2,). The strength of feature interactions between two nodes should be (close to) zero when these nodes are conditionally independent.
2.1
Attention-based methods
Due to the success of the transformer architccture (Vaswani et al. 2017 y most recent tabular deep learning
methods are altention-based (Arik & Pfister 2020: Huang et al. 2020 Somepalli et al. 2021 Kossen el al.,
2021
Gorishnly et al.) 2021), from which TTT-Tranffomer (Gorishniy et al. 2021) ) has been established as a popular bascline. All of these methods are based on multi-head self-attention (Vaswani ct al. 2017).
In most works, the attention map is of size p x p if a larget token (or CLS, 'classification' or 'oulput" token (Devlin et al. 2019)) is appended. If the attention map is equal to the size of the total number of features (or the number of imput features), it can be used for interpretation and explaining the feature interactions. This approach is popular in natural language, with Bert (Vig) 2019) being used to interpret how the model assigns weights to different tokens. While in natural language the size of attention map differs per input due to varying sequence lengths, in tabular data the number of features is fixed, making the attention map the same size as the adjacency matrix. This allows for an interpretation of the attention map as a proxy for the weighted adjacency matrix, which is further discussed in Subsection 3.2 Therefore, we refer to such attention-based methods as implicit GTDL methods. They do not explicitly model the feature interactions, but due to the nature of the attention map, they do model the graph structure implicitly.
This notion, that the attention map be interpreted as the learned graph structure of tabular data, has not been thoroughly discussed in the literature. However, most methods, as listed in Table I use the attention map for interpretability. TabNet (Arik & Pfister 2020) reports the attention map of synthetic datasets as visualizations and notes how irrelevant features are ignored in the attention map. SAINT (Somepalli ctt al. 2021), although designed for tabular data, reports the attention map of MNIST and discusses that the visualization of the attention map is similar to the ground-truth image. FT- Transformer (Gorishniy et al., 2021 interprets the attention map as feature importances, and shows for some real-world datasets that the
3

---

Published in Transactions on Machine Learning Research (03/2026)
attention map has high rank correlation with integrated gradients (Sundararajan et al. 2017 measure feature importance.
method to
Table 1: GTDL methods evaluate the feature interaction only qualitatively, typically with a visualization of the attention map or adjacency matrix. Size denotes the length of square attention map or adjacency matrix A (e.g, size means A E RPXP).
Model
Reference
Size
Feature interaction evaluation
Attention-based
FT-Transformer
Gorishniy et al.
(2021
Correlation with feature importance
TabNet
Arik & Plister
2020)
Visual of syuthetic dataset.
SAINT
Somepalli et al.
2021
P
Visual of MNIST
Graph neural network
FiGNN
Li et. al.
2019a
Visual of real-world dataset. p
12G-Former
Yan et al.
2023
p
Visual of real-world datasets
DRSA-Net
Zheng a..
(2023)
1
Visual of real-world dataset
INCE
Villaizán Vallelado et al. (2024)
Visual of real-world dataset
MPCFIN
Ye et al.
2024)
Visual of real-world datasets
Table2Graph
Zhou et al.
2022
p-1 1
Visual synthetic dataset.
2.2
Graph neural networks
GNNs operate directly on graph-structured data by propagating information between connected nodes (Zhou et al.) 2021). Feature GNNs apply this paradigm to tabular data, modeling cach feature as node and explicitly learning feature interactions through message passing (L. al. 2024).
Generally, the message-passing mechanism can be summarized
ne-1) .n)-". An)
h2 ( = 00)
1-11 h
(6l1) JEN(()
+
(1(1-2) nn
(1)
with h the representation of node at layer I, Wii) the neighbors of node i ()(() message function, + aggregation operator, and g(!) an update function. The details of these differ between GNN architectures. Attention-bant based methods can be seen as a special case of GNNs (Joshi 2025). In the context of tabular data, a key advantage of GNNs is that they generalize attention-based methods by using a trainable weighted adjacency matrix, A, to explicitly propagate information between nodes rather than relying on implicitly learned attention maps. Furthermore, GNNs for tabular data typically have trainable parameters that represent individual features or interactions, allowing for more flexible modeling of feature interactions compared to attention- based methods that rely shared parameters across features. Therefore, refer to them explicil GTDL. methods, contrary to attention-based methods that model the graph structure implicitly.
Different GTDL methods implement Equation I in different ways. FiGNN (L et al. 2019a) uses a feature graph to explicitly model the separate feature interactions. T2G-Former Yan el al. 2023 adapts the transformer architecture (Vaswani al.) (2017) for tabular data and learns feature graph that focuses on learning meaningful interaction between different features. DRSA-Net (Zheng et al. 2023 uses dual-route structure GNNs to learn adaptively the sparse graph structure. INCE (Villaizán- Valeelalle et al. 2024 has similar approach as T2G-Former, but uses an Interaction Network (Battaglia et al. 2016) instead of a Transformer. MPCFIN YYe et al.) 2024) uses cross-feature embeddings and multiplex GNNs to model the interaction and dependencies between features.
II GTDL, the neighbors typically all other features, Le., A((1) = (I.....p) pp (0) fully connected graph used due the absence of known graph structure.
4

---

Published in Transactions on Machine Learning Research (03/2026)
The feature GNN literature [Li et al. 2019a Yan et al. 2023 Zheng et al. 2023 Villaizán- Vallelado et al. 2024 Ye et al. 2024 Zhou al., 2022) suggests that the learned adjacency matrix can be used to interpret and explain the feature interactions. However, there are two reasons to be careful with this interpretation.
The evaluation of the graph structure is only heuristic to the best of our knowledge. The afore- mentioned explicit GTDL, methods (FiGNN, INCE, T2G-Former, MPCFIN, and DRSA-Net) report the learned adjaceney matrix for one or few real-world datasets. They argue that the learned feature interactions meaningful by post-hoc explaining the feature interactions. They justify the connections by referencing the semantic meaning of the feature names, suggesting that connected features are intuitively related. The issue with this approach is that, in the absence of a ground-truth graph structure, it becomes impossible to quantitatively evaluate the learned adjacency matrix.
2. The GTDL methods do not explicitly instruct the model to learn the true underlying graph structure. The loss is computed exclusively on the error between predicted and true target values. This means the model is only incentivized to improve predictive accuracy, not to accurately model the underlying feature interactions. As a result, the learned graph structure may not reliably reflect the true relationships between features. This limits its usefulness for interpretability and potentially constrains predictive performance.
Table2Graph (Zhou et al. 2022) addresses the first problem, that real-world datasets do not have a ground truth graph structure, by using a synthetic dataset. However, the learned graph structure is still evaluated heuristically against the ground-truth interactions, by visually comparing the learned weighted adjacency matrix with the ground-trun -anntttruthhho interactions. The second problem, that the model only prediction-centiiy is addressed by introducing reinforcement learning term to the loss function to explore the adjacency matrix. This encourages the model to also focus on learning the graph structure, rather than just the predictive performance.
Not all GNN methods treat predicting the target feature in the same way. Some of these models treat predicting the target feature as a node-level task, while others treat it as a graph-level task (Prince 2023 Node-level approaches (T2G-Former, INCF, MPCFIN) include a target node in the graph structure, and pass the embedding of the target node to an output layer. With this approach, the model learns a weighted adjacency matrix of size p p. Graph-level approachos (FiGNN, DSRA-Net, Table2Graph) do not include larget node, resulting in an adjacency matrix of size (v (p-11 1) (p 1). The embeddings of all nodes are aggregated and passed to an output layer.
2.3 Probabilistic graphical model as a baseline for GTDL
Feature graphs, as studied by GTDL methods, share similarities with PGMs (Lauritzen 1996 Koller &
Friedman 2009). PGMs provide a principled framework for modeling multivariate dependencies by encoding conditional independence relationships among random variables using graphs. Widely applied in Bayesian statistics, PGMs represent the structure of a probability distribution, often Gaussian, through a compact graph encoding the conditional independencies among variables. Bayesian techniques in PGMs, like BDgraph (Mohammadi & Wit 2019). have demonstrated strong empirical performance in recovering interaction structures (Vogels et al., 2024). The ability to quantify uncertainty in the learned graph structure makes them useful sanity check for GTDL methods.
2.4 Related approaches
There are other related approaches that after closer inspection not relevant to our discussion GTDL. In the literature on recommender systems and thick- rate, there has been longer interest in different notion of feature interactions, that of cross features; c.g., (Cheng et al.] 2016 Guo et al. 2017 Lian et al., 2018 Wang el al. 2017 Cai et al. 2021 Wang el al. 2020 Song el al. 2019). These models focus on learning multiple weighted products of features to improve the prediction of the target feature. As noted by Li et al. (2019a), this limits the capability to model interactions across different features flexibly and explicitly. We are interested in how to model the feature interactions explicitly on a graph. Therefore, we do not discuss the feature interactions of these methods in further detail.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Tabular foundation models, TabPFN (Holmaan et al. 2025 TabICL QQu et al. 2025) and LimiX (Zhang et al. 2025) have recently gained attention due to their high predictive performance on tabular data. Two key aspects of these models are their alternating instance-wise and feature- wise attention layers, and their ensembling predictions over multiple feature permutations. The feature-wise attention layers work similar as the allention-based methods discussed in Subsection 2.1 and therefore could have been interpreted as implicit, GTDL methods. However, these tabular foundation models contain techniques that prevent a straightforward interpretation of the feature-wise attention map as a weighted adjacency matrix. TabPFN and LimiX encode groups of features collectively rather than individually. TabICL incorporates rotary positional embedding (RoPE) {Su et al. 2024) independent of the feature permutation, which alters the attention map. Therefore, we do not consider the feature interactions of these models in this work.
Tree-based models (e.g., XGBoost (Chen & Guestrin 2016), LightGBM (Ke et al. 2017 and CatBoost (Prokhorenkova et al. 2019) remain a popular choice for tabular data. Nevertheless, these approaches do not explicitly represent feature interactions in a graphical format. Due the nature of tree architectures, the learned feature interactions can not easily be extracted from the model.
3 Evaluating feature interactions in GTDL
The development of GTDL is hindered by the fact that the learned graph structure is only evaluated heuristically. To solve this, we introduce a framework to evaluate the learned graph structure of GTDL methods with synthetic datasets and quantitative metrics.
3.1 Synthetic data with graph structure
Most existing GTDL methods lack rigorous evaluation of the learned graph structure. Typically, the learned graph structure is evaluated heuristically, by reporting a visualization of the learned weighted adjacency mairix of real-world datasets. Feature interactions are post-hoe explained based on the semantic meaning of the feature names. Evaluating only on real-world datasets is problematic, as the true graph structure is not known. Therefore, we propose using synthetic datasets. Using synthetic data enables GTDL methods to compare the learned graph structure with the ground-truth underlying graph structure in a controlled environment.
We adapt two existing data generation methods from the literature. The three-step process is sketched in Figure 2 and the details are given in Appendix B
Multivariate normals (MVNs) are typically studied by PGM methods. We follow the default procedure
of generating conditional multivariate data (as described (Mohamuadi & Wil 2015), for instance).
short, we employ the following: (i) Sample graph structure Girue E <PPx from the Bernoulli distribution.
(ii) Sample covariance matrix Σg RRxp from the G-Wishart distribution (Roverato) 2002 Letac &
Massam 2007) to describe the feature interactions. (fii) Obtain samples D Raxp from NO. Eg).
Structural causal model (SCM) PPaal 2021) are used to generate tabular data in tabular foundation
models (Hollmann et al. 2025 Qu et al. 2025 Zhang et al. 2025 We follow simplified version of the data generation process in these tabular foundation models, that is: (i) Generate a directed acyelic graph (DAG) to define the graph structure of the SCM. To obtain the undirected graph Gizue from the DAG, we moralize (connect all parents of child node, and drop the direction of the edges because the child of a parent can also be used to prodict the parent) and marginalize (drop the root nodes and connect its children) the DAG (Cowell et al., 1999). More details on moralization and marginalization are discussed in Appendix B (ii) Sample computational maps fa for each child node in the DAG. The computational maps (fi) are smooth nonlinear functions that take all the values of the incoming edges as input, and the output is the value of the child node i. (iii) Traverse random data Froots in topological order through the DAG to obtain n samples D E Rexp.
For both approaches, we randomly select a target fcature y E R from D, with the remaining columns serving as input features x E (p-1) The target feature is directly influenced by its neighbors and only indirectly by non-neighbors. This setup lends itself well for evaluating the graph structure learned by GTDL.
6

---

Published in Transactions on Machine Learning Research (03/2026)
Multivariate normal
Structural causal model
(i) Girue
Bernoulli
()) DAG
DAG generator
moralized and marginalized DAG
Gerue
computational map generator
(ii) Eg
G-Wishart
(ii) Jireatp
(ii)) D
E Rexp ~ N0, Ea)
(ui) Troots
N(0,0) SCM
D
E Ruxp
Figure 2: Two synthetic data generation pipelines. Both pipelines can roughly be divided into three steps. (i) Sample graph structure. (ii) Sample feature interaction. (iii) Sample data given the graph and feature interactions. Nodes are colored cyan input features z, orange target feature y, and green root nodes Zroots-
methods. Consider the simple example: 20 20-21 21 T2. The model could learn use zo to predict 22 directly by learning an edge wetn them. However, this is suboptimal because 22 is conditionally independent of 20 given T1. That is, once 21 is known, To provides When observations are noisy, using To lo predict 22 accumulates uncertainty across multiple dependencies, leading to worse predictions than those obtained by using the immediate neighbor The model should instead recognize In as a more reliable predictor for 22, which results in learning the correct graph structure.
additional information for predicting T2 (Lauritzen 1996).
The proposed synthetic data generation methods are different from the synthetic datasets used by TabNet. (Ari & Pfister 2020) or Table2Graph (Zhou et al. (002),, previously discussed in Subsections 2.1 and 2.2 All input features in these datasets conditionally independent of each other, and have direet interaction with the target feature. Therefore, there is underlying graph structure to be learned, all features are connected only to the target leature. In contrast, the MVN and SCM data generation methods create datasets with more complex underlying graph structures, making them more suitable to evaluate GTDL methods.
We acknowledge that our data generation processes may not be fully representative of real-world tabular data. For instance, the graphs evaluated in this work (Subsection 3.4 and Section 4 are relatively small (p = 10), the MVN does only have linear feature interactions, and the SCM does not have missing nodes within the DAG. However, key is that the datasets have clear underlying ground-truth graph structure that GTDL methods should be able to learn. If models can not model the feature interactions of these synthetic datasots well, it is unlikely to expect that, on larger (p 2 100) and real-world datasets, these models will learn meaningful feature interactions.
3.2 Interpreting and evaluating the graph structure
The GTDL. methods introduced in Subsections 2.1 and 2.2 model the graph structure inside their architecture. In this section, we discuss how we extract the learned weighted adjacency matrix A E R from these methods, and how we propose to evaluate the quality of the learned graph structure. For the GNN-based GTDL methods, extracting the adjacency matrix is straight.forward, as these methods explicitly use a weighted adjacency matrix within their architocture. For the attention-based GTDL methods, we interpret. the altention maps as proxy for the learned weighted adjacency matrix.
To justify this interpretation, consider the following example: Let be p p matrix representing the attention map, where each element aij. denotes the attention weight from feature i to feature j. as used by Vaswani et al. (2017). If feature i is conditionally independent of feature j given all other fealures, we have Atrue, = O. The attention mechanism should learn during training to assign low attention weights agj, azz R 0, as feature does not rely on information from fcature j (and vice versa) for its reprosentation. Contrary, if feature and feature j are dependent, we have Arrue. 1. The attention mechanism should learn to assign non-zero attention weights aij, aji > 0, as feature i relies information from feature (and
7

---

Published in Transactions on Machine Learning Research (03/2026)
vice versa) for its representation. In summary, with Atrue,i we expect Oig, ajs 0, and with Airu,,t 1, we expect (ij, Ojj > 0.
After training, the attention maps of the test samples are extracted for all heads and layers. To obtain the adjacency matrix we perform two steps. First, we average the attention maps over the test samples, heads and layers obtain and obtain a single attention map of size p x p. Second, we account for the diagonal of the attention map and for the softmax normalization from the original attention equation, which further explained in Appendix A to obtain the weighted adjacency matrix A.
Current GTDL methods only report the predictive performance of the target feature, and do not evaluate the learned feature interactions quantitatively. We propose to evaluate the quality of the graph structure by comparing edge-wise (ignoring the diagonal) the true binary adjacency matrix Atrue with the learned weighted adjacency matrix Apred = with the receiver operating characteristic area under curve (ROC AUC) Braddey 1997). This metric reflects to what degree the feature interaction strengths of the truc edges are higher than those of the true non-edges. Ranging from zero to one, a value of 0.5 equals random guess. A high ROC AUC indicates that all true edges have higher feature interaction strengths than all true non-edges. The ROC AUC is 'relative measure', meaning that it is not sensitive to the scale of the feature interaction strengths. This way, we are forgiving in the evaluation of the feature interactions, as we only mcasure if the model can distinguish between true edges and true non-edges, and not the absolute values of the feature interaction strengths.
3.3 Pruning the feature interactions
To understand the effect of learning the correct graph structure, we model the GTDL methods in two different. settings, which are sketched in Figure 3 First, we train the GTDL with a fully connected graph. This the default setting in GTDL mothods, as the true graph in real-world datasots is not known. Sccond, we limit feature interactions to only those present in the synthetic data, effectively pruning the gruph the true edges. This means that the model is only allowed to learn feature interactions that are present in the true graph. Practically, this is done by masking the attention map or the graph structure within the network architecture. This is only possible if the true graph is known, which is the case for synthetic data. By comparing the results from the fully connected and the pruned graphs, we can see how much the GTDL, methods benefit. from using only the true edges.
Existing GTDL
Fully connected graph
Gtree
Tabular data
Apred
ROC AUC
Atruo
Pruning the feature interactions
True graph
pred
Figure 3: Upper branch (similar to Figure Existing GTDL methods use a fully connected graph. The learned adjacency matrix Apred, that we extract out of GTDL methods, is compared to the true adjacency matrix Arme with the ROC AUC to evaluate the learned feature interactions. Lower branch: When the graph is pruned to the true edges, GTDL methods can only model feature interactions that are present in the true graph. By doing this, we can analyze the effect when the correct graph structure is used in GTDL methods.
8

---

Published in Transactions on Machine Learning Research (03/2026)
3.4 Setup of the experiments
We conduct a standard deep learning experiment to evaluate existing GTDL methods. That is, we optimize the prediction of the target feature y given the input features x. This is done with the default GTDL setting of fully connected graph, and with the pruned setting where the feature interactions are limited to the true edges only. We tune the hyperparameters of the model and the learning rate, we use cross validation, and optimize the mean squared error (MSE) with Adam (Kingma & Ba 2017). Further details on the splitting, training, ovaluation, and hyperparameter tuning and cross validations can be found in Appendix C The code is publicly available 3
We report the quality of the learned graph structure (Subsection 3.2 only for the fully connected setting, as the pruned setting trivially has perfect graph quality. The pruned graph only contains the true edges, so the learned graph structure is equal to the true graph structure. For the predictive performance, we use the R2 score to compare the regression models. To average over the datasets, we use the normalized R2 score. This normalization is introduced by Wistuba et al. (2015) and used by Feurer et al. (2022); Grinsttajn et al. (2022) for instance. Per dataset, the R2 score is normalized between zero and one, using the worst-performing and the best-performing model on that dataset as the lower and upper bound, respectively.
We run the experiments for three datasets belonging to both dataset types (MVN and SCM), and their graph structures are shown in Appendix [ We compare all explicit GTDL methods that have publicly published code. That is, compare FIGNN (Li et al. 2019a), T2G-Former Yan et al., 2023) and INCE VVillazzán Vallelado et al. 2024). The remainder of the explicit GTDL methods, DRSA-Net (Zheng et al., 2023), MPCFIN (Ye et al. 2024) and Table2Graph (Zhou et al. 2022). have not published their code methods, we take FT-TTansformer (Gorishniy et al.) 2021) as an
repositories. For implicit, attention- illustrative example. In Appendix A we discuss how these methods use and interpret the learned weighted adjacency matrix, and how we adapt the implementations to compare them. We use the PGM method BDgraph (Mohammadi & Wit 2019) as a baseline to understand how well GTDL methods should be able to learn the feature interactions. Finally, we include TabPFN and XGBoost as additional baselines given their overall strong performance, to understand the difficulty of the predictive task.
4 Results of structure-aware learning in GTDL
To substantiate our claim that GTDL methods should focus on learning the graph structure, we demonstrate that GTDL methods do not accurately learn the feature interactions and that the predictive performance improves when the graph is pruned to its true edges. Results are aggregated per dataset type, see Appendix D for the results per dataset.
4.1 Feature interactions
The ROC AUC of the feature interactions is shown in Figure For all GTDL methods, across both datasets, the ROC AUC approximately 0.5, which equal to random chance. There is no difference in the values of the adjacency matrix whether there exists true edge or not. This shows that GTDL methods do not learn accurate graph structure. Therefore, the learned feature interactions should not be used for interpretability or explainability. Increasing the number of training samples does not change the ROC AUC, indicating that the poor performance of the GTDL methods is not due to insufficient data. This observation holds when increasing the number of training samples up to 10. samples as discussed in Appendix D.2
PGMs, which focus on learning the graph structure, can learn the feature interactions, while GTDL, methods cannot. The PGM method BDgraph has an ROC AUC very close to one for the MVN datascts. Even for the SCM datasets, which have nonlinear feature interactions, BDgraph can achieve reasonable ROC AUC values. The fact that this PGM method, non-deep learning method, can learn the feature interactions, while these advanced GTDL methods cannot, suggests that GTDL. methods have room for improvement in learning the graph structure.
htttps//gitthu. http://hhwww.ccce//a/////a/ee//e
9

---

Published in Transactions on Machine Learning Research (03/2026)
FTTrassommer
FIGNN
INCB
T2GFomer
BDymph
1.0
Maltivaraa
Normals
ROC AUC (Ame.Ayn Ayne)
01.5
Canaal Model
Stmoouna
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
Aleziz
Kandom guess
Available edges: Fully connected
Figure 4: Graph quality in the form of the ROC AUC comparing the learned weighted adjacency matrix with the true binary one, for two different dataset types. All GTDL models have ROC AUC x 0.5, which is random chance, indicating that they are not able to learn the feature intcractions in any meaningful way. The PGM method BDgraph does learn the correct feature interactions better. Error bars show standard deviation across seeds, cross validations and datasets.
4.2 Predictive performance
The R2 score of the prediction of the target feature shown in Figure The key lakeaway is that, in general, pruning the graph to the true edges improves the predictive performance of GTDL methods. This result indicates the importance of incorporating accurate structural information into GTDL models. When the graph pruned to only include true edges, the models are less likely to overfit. to spurious or irrelevant feature interactions. Restricting the model to only the true interactions simplifies the optimization landscape, allowing the learning algorithm to focus meaningful relationships rather than being distracted by false edges. This leads to better generalization and higher predictive accuracy. In contrast, fully-connected models must learn to ignore many false edges, which can introduce noise and make optimization more difficult, especially when data limited. This finding suggests that the inability of current GTDL methods to recover the true graph structure (as shown by the ROC AUC results) is not just a theoretical issuc, but has practical consequences for predictive performance. If the true graph known can be estimated reliably, enforcing this structure provide boost in performance.
We use the Linear Mixed-Effects Model (Lindstrom & Bates 1988 Pinheiro & Bales 2000) to assess whether the pruned graph outperforms the fully connected graph. We elaborate on the statistical testing procedure in Appendix C.I From Figure 5 we observe that, except for FIGNN, the prunod graph results are both visually (first and third row) and statistically significantly (second and fourth row) better than the fully connected graph results for low number of training samples. Tn Appendix [.3 we discuss results that indicate that this is because FiGNN treats the task of predicting the target feature as a graph-level task, while all other models treat it node-level task.
Regarding baselines, the PGM method BDgraph performs, as expected, well on the MVN datasets, and poorly on the SCM datasets. The SCM datasets have nonlinear feature interactions, while BDgraph can only model linear feature interactions. In most cases, GTDL methods outperform XGBoost, while TabPEN outperforms GTDL, methods. A possible explanation for the performance gap is that GTDL, and TabPFN may better exploit the tabular structure of data through learned representations, whereas XGBoost relies an ensemble of weak learners that may not fully capture feature interactions inherent in such structures. This underlines the need for deep learning methods due to their flexibility to learn nonlinear relationships.
Furthermore, the benefit of incorporating the truc graph increases by reducing the number of training samples. When ample data available, models benefit. less from incorporating the graph structure correctly, but when data is scarce, leveraging the graph structure improves the predictions. This is line with the general notion of geometric deep learning, where symmetrics in the data are used to improve the learning process (Bronstein) et al. 2021). When data is scarce, explicitly incorporating the symmetries from the data into the model is beneficial. However, abundance of data facilitates implicit learning of these symmetries (Marchettii
10

---

Published in Transactions on Machine Learning Research (03/2026)
et al.
2024 reducing the benefit of explicitly incorporating them. This observation holds when increasing the number of training samples up to 10 samples, discussed in Appendix D.2
FIGNN
GTDL buselines
Availabl
Mulication
Nomot
1.00
Pully comernal
Trie odges oaly
0.25
Liner Mised Effects
0.90
Sigrif, drraaedd (0,05)
10
Noa-GTDL
10~1
TakeNN
nomallzed
Strcore
Canual CanulMMkka Medih
10
BDGruph
XGiloout
10
1000
2000
5000
4000
1000
2000
9000
4000
2000
2000
9000
4000
2000
2000
5000
4000
Abahs
Figure 5: Predictive performance for two different dataset types. When the graph pruned to its true edges, the predictive performance is, in most cases, better compared to the fully connected graph. The difference reduces as the number of training samples increases. This confirmed by the p-values of the Linear Mixed-Effects Model, where lower p-values indicate stronger evidence that the pruned graph results are better than the fully connected graph results. Error bars show standard deviation across seeds, cross validations and datasets.
5 Conclusion, limitations, and future work
In this work, we have analyzed the capability of graph-based tabular deep learning (GTDL) methods to learn feature interactions in tabular data. Inspired by the principles of probabilistic graphical models (PGMs), we proposed to use synthetic tabular datasets with known ground-truth graph structures, enabling the GTDL community to quantitatively assess whether models accurately capture the intended graph structure. Current GTDL approaches often produce graph structures used for interpretation, yet our analysis shows that these structures fail to reflect the true interactions among featurcs. This indicates that the mechanisms of message- passsgng in GNNs, and altention in transformers, does not work as intended for labular data. Our empirical findings demonstrate that when models operate on accurate interaction structures, predictive performance improves, highlighting that structural fidelity is not merely a matter explainability, but a core driver performance.
A potential risk could be the interpretation of the attention map as a weighted adjacency matrix. First, the attention mechanism is not designed to specifically model feature interactions for explainability. If attention can be used for explainability is ongoing debate (Bibal al. 2022 Lopardo et al. 2024). although this debate in the literature mainly focuses around natural language tasks. However, our interpretation for tabular data that non-interacting features will exhibit low attention values while interacting features will manifest higher attention deliberately modest and pragmatic, rather than relying on attention as a fully explanatory tool. Second, while we acknowledge that aggregating attention across layers and heads might. obscure certain signals, we find no strong evidence that alternative aggregation strategics would significantly improve graph recovery alter the overall conclusions.
We highlight three directions how the analysis of this work can be extended in future work. First, future work should move beyond evaluating the learned graph structure (i.c., the presence of edges), but also consider the functional form of the feature interactions (i.e., the type of edges). Learning how features interact, and in what way, allows for more nuanced, robust, and interpretable modeling of feature relationships. Second, the evaluated datasets and graphs could be more expressive and challenging. Examples include larger graphs with richer topology, missing nodes, and more complex feature interactions involving categorical features, well as wollworld datasets with known ground-truth structure (e.g., knowledge graphs). However, our
11

---

Published in Transactions on Machine Learning Research (03/2026)
results showed that GTDL methods struggle to learn relatively simple, small graph structures, which suggests that improving robustness and structure induction on basic cases remains priority before scaling to such settings. Finally, structure-aware modeling should be extended beyond flat tables to richer data modalities.
Timo-scries data (Padhi et al., 2021 and relational databases (Fey et al.) 2023 Robinson et al. 2024 Cong
et al. 2024 Dwivedi et a 2025) present new challenges for learning and validating feature interactions over time or across relational contexts. Relational deep learning (RDL) (Fey et al. 2023) could ask the same question as we propose GTDL should do: how do columns / within a table relate? Currently, table-row embeddings in RDL lack structural bias from the graph structure of columns. Furthermore, our approach could be extended from table-level to relational database-level. When doing node-level prediction, does RDL rely on meaningful primary-foreign relationships?
Future work should build on the insights of this work to develop GTDL methods that more effectively learn and leverage feature interactions in tabular data. By prioritizing structural fidelity alongside predictive accuracy, future GTDL models can unlock their full potential.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next- generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD Inter- national Conference Knowledge Discovery & Dato Mining, pp. 2623 2631, Anchorage AK USA, July 2019. ACM. ISBN 978-1-4508-6201-6,
doi:
URL https: /ttpp::/ddl.a.....o org//oi/11
1145/3292500.33300
Sercan 0. Arik and Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning, December 2020. URL
http:// /taxx///rxiv.. http://xxvvvoooo/aaaaa aataaaaaa...00000
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics, December 2016. URL http://arxiv.ort
abs 11s2/161200022
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez,, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matl Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks, October 2018. URL http://arxiv wttp://.orgbbbb//oooeee 1800. 1/////000000.00
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper- Parameter Optimization. In Advances in Newral Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://papprrs .nness.ii nips. cttpppprrrppee..pppppppppppp tiless/aperr/211///aaee 86617778820411131722666666666 Abse:aate..-Atttttt..... html
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Patrick Watrin. Is Attention Explanation? An Introduction to the Debate. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Pupers), pp. 3889 3900, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/vv 12021/22222cccc acl- long. URL https://acclanthologyy org/
2022. acl.cc1-10 acl-long....... 1292
Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145 1159, July 1997. ISSN 00313203. doi: 10.1016 (0000//800111223300000001223 elsevier hwtp:///eriieeeeee////0010000000
URL
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, May 2021. URL httt:://aaxxi.org/abbaa
13478
Shaofeng Cai, Kaiping Zheng, Gang Chen, H. V. Jagadish, Beng Chin Ooi, and Meihui Zhang, ARM-Net: Adaptive Relation Modeling Network for Structured Data. In Proceedings of the 2021 International
12

---

Published in Transactions on Machine Learning Research (03/2026)
Conference on Management of Data, pp. 207-220, Virtual Event China, June 2021. ACM ISBN 978-1- 4503-8343-1. doi: 10.145//344806634573211 URL http:://dpl........ httpp::/ddl.acc.orrrr. wcm.oorg/oii/10// 1145/114481441 3457321
Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledgee Discovery and Data Minóng, pp. 785 794, August 2016. doi: 10.1145/ 00.114/2939672..0 2939672.29397888 URL http://arxiv. .rgg/
Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, Mu Li, Junyuan Xie, Min Lin, Yifeng Geng, Yutian Li, Jiaming Yuan, and David Cortes. Xyboost: Exlreme Gradient Boosting, 2026. URL Inttps://githutt com http:/dlll/xxl// c/gbooost
Heng- Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & Deep Learning for Recommender Systems, June 2016. URL
http://arxiv wwwpxii..rg///baaa prgg/bs//6000 http:/1666.07700 077992
Tianji Cong, Madelon Hulsebos, Zhenjie Sun, Paul Groth, and H. V. Jagadish. Observatory: Characterizing Embeddings of Relational Tables, January 2024. URL
Robert G. Cowell, Steffen L. Lauritzen, A. Philip David, David J. Spiegelhalter, V. Nair, J. Lawless, and M. Jordan. Probabilistic Networks and Expert Systems. Springer-Verlag, Berlin, Heidelberg, 1 edition, 1999. ISBN 0-387-98767-3.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding, May 2019. URL http:///aaxii..rr/abb////// 04805
Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, and Jure Leskovec. Relational Graph Transformer, May 2025. URL httt:://arxv..rrg
abs/2505. 10960
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter, Auto-Sklearn 2.0: Hands-free AutoML via Mcta-Lcarning. Journal of Machine Learning Research, (321))1616 2022. ISSN 1533-7928. URL
http://wwwro.rroppp///////////// html
Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Relational Deep Learning: Graph Representation Learning on Relational Databases, December 2023. URL
04615
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36 (6)33443333383 202, April 1980. ISSN 03400-2000, 1432-0770. doi: 10. LL007/BF00344251.. URL /ttp://tt////.......r.oooo//////
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting Deep Learning Models for Tabular Data. In Advances in Nearal Information Processing Systems, volume 34, pp. 18932-18943. Cur- ran Associates, Inc., 2021. URL https::/prroceeding
wwwpiiiininnn..uunrreeee neurips httpp//pppp...ilee///p///////aaa html
Anirudh Goyal and Yoshua Bengio. Inductive Biases for Deep Learning of Higher-Level Cognition, August. 2022. URL
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do brse- models still outperform deep learning on tabular data?, July 2022. URL
08815
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Lí, and Xiuqiang He. DeepFM: A Ractorization-Machine based Neural Network for CTR Prediction, March 2017. URL http::/aaaxi..rg//abb//////a////
04247
Noah Hollmann, Samuel Muller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Ti- bor Schirrmeister, and Frank Hutter. Accurate predictions on small data with tabular foundation model. Nature, 63745)))33319 126,,,6,,,,,.. January 2025. ISSN 0028-0836, 1476-4687. doi: 0..1088/s41556-022 024-0888-6.
URL https:/ //www//wwwwww./. nature. wtt.://at...ccaees//1111-000-000
13

---

Published in Transactions on Machine Learning Research (03/2026)
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings, December 2020. URL http://arxiv .rgg ab/g/abb//011.06778
Chaitanya K. Joshi. Transformers are Graph Neural Networks, June 2025. URL http://arxiv. org/abs/////aaaooooooooooeee /bs/
2506. 22084
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu.
LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceeddnng hewpeisenrrrpp... cc/ paper http:///........./././////////ta Abstraaaae Abstract. .ttll
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL
http://arxii: org/ absaaas/aaaa 111//1111... 6980
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive Computation and Machine Learning. MIT Press, Cambridge, 2009. ISBN 978-0-262-22833--7
Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self- Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learn- ing. In Advances in Neural Information Processing Systems, volume 34, pp. 28742 28756. Cur- ran Associates, Inc., 2021. URL 11072008fcC311177777733888111111 hww.r/efaeaat-sttaaattt.o html
Steffen L. Lauritzen. Graphical Models. Number 17 Oxford Statistical Science Series. Clarendon Press Oxford University Press, Oxford New York, 1996. ISBN 978-0-19-85229955
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backprop- agation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541 551, December 1989. ISSN 0899-7667. doi: 10.1162/neco (6662//cco.100 198..1....54.8. URL https://ieeexploreee ieee. org/abstract/ ort.p//ppppp..r.....oooooooaaac/ document/6795724
Gérard Letac and Hélène Massam. Wishart distributions for decomposable graphs. The Annals of Statis-
tics, 35(3):1278 1323, July 2007. ISSN 0090-5364, 2168-8966. doi: 10. 10.121440090336600000113 URL
Cheng- Te Li, Yu-Che Tsai, Chih-Yao Chen, and Jay Chichen Liao. Graph Neural Networks for Tabular Data
Learning: A Survey with Taxonomy and Directions, January 2024. URL http://arxiv.org/
02143
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. Fi-GNN: Modeling feature interactions via graph neural networks for CTR prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Cikm '19, pp. 539 548, Beijing, China and New York, NY, USA, 2019a. Association for Computing Machinery. ISBN 978 9788--6500. doi: 08.1445/337388438555991. URL https://doi.org::
1144/33553884 33579511
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction, October 2019b. URL httts://arxiv.. wwwpstvwwwrr/a////////obb /9900
05552v1
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction, July 2020. URL http:///aaxxiv..og/aa/////a//// 05552
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. XDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1754-1763, London United Kingdom, July 2018. ACM. ISBN 078-1-4503-5552-0, doi: 00.1145/321989..22 acm. wwwp/oo//////oo/oo/ooo http:://llaaacmrrr//////....... 1tt.://///11111////111111191 32200033
URL
14

---

Published in Transactions on Machine Learning Research (03/2026)
Mary J. Lindstrom and Douglas M. Bates, Raphson and EM Algorithms for Linear Mixed-
Models for RepeateddMeaasres Data. Journal of the American Statistionl Associalion, 83(404)10014112022 December 1988. ISSN 0162-1459. doi: 10.1080/01621459. 1988..888888..88888 URL http:tt:///oi..og//00..000 01621459. 1988. 10478693
Gianluigi Lopardo, Frederic Precioso, and Damien Garreau. Attention Meets Post-hoc Interpretability; A Mathematical Perspective, June 2024. URL http:://axiv.. http::///xxvvoor/////// forg/abi abs////////////////////////// 03485
H. B. Mann and D. R. Whitney. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Armals of Mathematical Slatistics, 18(1):50 60, 1947. ISSN 0003-4851. URL https://wwwz jstt://
Giovanni Luca Marchetti, Christopher J. Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks. In Proceedings of Thirty Seventh Conference on Learning Theory, pp. 3775-3791 PMLR, June 2024. URL jattps://proceedingss mlr. www.mlrreeet....... press/v247/
marchett124a.html
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Benjamin Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data?, July 2024. URL
02997
A. Mohammadi and E. C. Wit. Bayesian Structure Learning in Sparse Gaussian Graphical Models. Bayesian Analysis, 10(1), March 2015. ISSN 1936-0975. doi: 14.111414-BAA88 URL http://arxiv.on org org/abs/ 1210. 5371
abs/
Reza Mohammadi and Ernst C. Wit. BDgraph: An R Package for Bayesian Structure Learning in Graphical Models. Journal of Statistical Software, 89:1-30, May 2019. ISSN 1548-7660. doi: 00.18637/jss.v0S5 v08..003. URL :tttpa//doi... org 10. 18637 v089.
Claude Nadeau and Yoshua Bengio. Inference for the Generalization Error. In Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https:// www.eniips........ cc/ppper.ppppppoo..e wwwes/lllaaaeeeee /aper/ Abstractt.t......... html
Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mrouch, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Allman. Tabular Transformers for Modeling Multivariate Time Series. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3565-3569, June 2021. doi: N0.1109/CCASSP:30 242499411... URL https:// leee. arg/abstract/ document/9414142.
Judea Pearl. Causality; Models, Reasoning and Inference. Cambridge University Press, Cambridge, second edition, reprinted with corrections 2021 edition, 2021. ISBN 978-0-511-80316-1.
J. Pinheiro and D. Bates. Mised-Effects Models in S and s-PLUS. Statistics and Computing, Springer New York, 2000. ISBN 978-0-387-98957-0. URL httpps:/bookss. wow.booooooooooo..ooo.ooooooooo. hitr/oooktttddn-iiiifffffoll
Simon J.D. Prince. Understanding Deep Learning. The MIT Press, 2023. URL http://udlbook. com
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika. Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features, January 2019. URL http:////arxxt www.aaiwii.oooo//bbaaaaa arg/.bs/1706a/
09516
Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data, May 2025. URL http:p//aaxxxv.oorr///////0..00 05564
Joshua Robinson, Rishabh Ranjan, Weihua HU, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. RelBench: A benchmark for deep learning on relational databases. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomozak, and C. Zhang (cds.), Advances in Neural Information Processing Systems, volume 37, pp. 21330- 21341. Curran Associates, Inc., 2024. URL http//ts////rooeeiingg
222441/00aaa4e02aacecceeeeeeeee2 _nceeettss_____________aee Benchmarks
hat.eeeeaaaaaddeeeennneeeerssss Track. pdf
15

---

Published in Transactions on Machine Learning Research (03/2026)
David Wilson Romero Guzman. The Good, the Efficient and the Inductive Biases:: Exploring Efficiency in Deep Learning Through the Use of Inductive Bioses. PhD-Thesis Research and graduation internal, Vrije Universiteit Amsterdam, September 2024.
Alberto Roverato. Hyper Inverse Wishart Distribution for Non-decomposable Graphs and its Application to Bayesian Inference for Gaussian Graphical Models. Scandinavian Journal of Statistics, 29(3):391-411 2002. ISSN 1467-9469. doi: 10.1111/1467-4469.000 00297. URL aattps://onliie hatpa///oonnnnnbraaaie wiley. Maclee..com//e ctm/commooooooc /
als/10..111/117779999.00002 00297.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre- Pre-Training, June 2021.
URL httt:://ax/xii org/ aataaabb//200/ 01342
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic Feature Interaction Learning via Delf-Attentive Neural Networks, August 2019. URL http:
11921
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568:127063, February 2024. ISSN 0925- 2312. doi: 10. 10.1016/j.meucom www.:euuwooo.......222 127663 URL httpppp/wwww...cooototccccoooooa http://ccccccc//aaticc/lll..///a
pi1/50925251223011884
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Doep Networks, June 2017. URL http:://aarw...rr//abbb/////////
01365
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need, December 2017. URL http::/arxiv..o/ wwwpaivv..rra//////a//a/aaob
03762
Jesse Vig. A Multiscale Visualization of Attention in the Transformer Model. In Marta R. Costa-jussà and Farique Alfonseca (eds.), Proceedings of the 57th Annual Meeting of the Associalion for Compulational Linguistics: System Demonstrutions, pp. 37 42, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/11/P19-3007 URL
Mario Vallelado, Matteo Salvatori, Belén Carro, and Antonio Javier Banchez-Esgueville Graph Neural Network contextual embedding for Deep Learning on tabular data. Newral Networks, 173:106180, May 2024. ISSN 0893-6080. doi: 10. http::....eueee2222.........
URL https:// http://wwwcceeceerretttttt
Lucas Vogels, Reza Mohammadi, Marit Schoonhoven, and S. îlker Birbil. Bayesian Structure Learning Undirected Gaussian Graphical Models: Literature Review with Empirical Comparison. Journal of the American Statistical Association, 119(548):3164-311 October 2024. ISSN 0162-1459. doi: 10.1080/
01155.2022.2395000 URL https: 10. 10..00000062115500 2024. 2396504
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & Cross Network for Ad Click Predictions, In Proceedings of the ADKDD'17, pp. 1-7, Halifax NS Canada, August 2017. ACM. ISBN 978- 978-1-450.-.19422. doi: 10.1145/3124449.31225
URL httpps:/d/. acm. org//oi/// 114....111111311441111 3124754
Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems, October 2020. URL httt::///arrrt/e wwtp:///////ab//////oosss 13535
Frank Wilcoxon. Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1(6):80, December 1945. ISSN 00994987. doi: 10.2307 022307/300096... URL hatttps:/wwww.jjstt wtt.ooo.rr//ttabeeeeeee 2307 2307//001922
originscrossref
Martin Wistuba, Nicolas Schilling, and Lars Schmidt- Thiiddt- Learning hyperparameter optimization initializations. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), (DSAAA) pp. 1-10, October 2015. doi: N0.1109/DSSAA.21155
URL https::/ieeexpporre lece. wtt..////...o/.///cc/cc////
document/7344817
16

---

Published in Transactions on Machine Learning Research (03/2026)
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Z. Chen, and Jian Wu. T2G-Former: Organizing tabular features into relation graphs promotes het erogrngens feature interaction. In Proceedings of the Teerth-Seventh AA Conference on Artificial Intelligence and Thirty-Fifth Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, volume 37 of 2/AA23//AAA EAAT'23, pp. 10720 10728. AAAT Press, February 2023. ISBN 978-1-57735-880-0 doi: 10.1609/ v..ii.v377..222.. URL https: org/10 1009/ v3719. 26272
Mang Ye, Yi Yu, Ziqin Shen, Wei Yu, and Qingyan Zeng. Peass-e Interactive Tabular Data Modeling With Multiplex Graph Neural Networks. IEEE Transactions on Knowledge and Data Engincering, pp. 15, 2024. ISSN 2191. doi:
URL https: batps://ieeees /ttp:///eeeeeeeeeeee//eee ieee. org/ org/
Xingxuan Zhang, Gang Ren, Han Yu, Hao Yuan, Hui Wang, Jiansheng Li, Jiayun Wu, Lang Mo, Li Mao, Mingchao Hao, Ningbo Dai, Renzhe Xu, Shuyang Li, Tianyang Zhang, Yue He, Yuanrui Wang, Yunjia Zhang, Zijing Xu, Dongzhe Li, Fang Gao, Hao Zou, Jiandong Liu, Jiashuo Liu, Jiawei Xu, Kaijic Cheng, Kehan Li, Linjun Zhou, Qing Li, Shaohua Fan, Xiaoyu Lin, Xinyan Han, Xuanyue Li, Yan Lu, Yuan Xue, Yuanyuan Jiang, Zimu Wang, Zhenlei Wang, and Peng Cui. LimiX: Unleashing Structured- Data Modeling Capability for Generalist Intelligence, September 2025. URL http://arxiv..r
abs/2509 03505
Qinghua Zheng, Zhen Peng, Zhuohang Dang, Linchao Zhu, Ziqi Liu, Zhiqiang Zhang, and Jun Zhou. Deep Tabular Data Modeling With Dual-Route Structure-
Graph Networks. IEEE Transactions on Knowledge and Data Engineering, (5) 9727, September 2023. ISSN 1558-2191, doi: 10.1109/TKDE. URL htttps:///eeepppleettt:
leee. org/ /ocument/10054100
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Lin, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph Neural Networks: A Review of Methods and Applications, October 2021.
URL /ttp:///axiv..ao org/ abs
Kaixiong Zhou, Zirui Liu, Rui Chen, Li Li, Soo-Hyun Choi, and Xia Hu Table2Graph: Transforming Tabular Data to Unified Weighted Graph. In Proceedings of the Thirly Firsl Internalional Joinl Conference on Artificial Intelligence, pp. 2420 2426, Vienna, Austria, July 2022. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-956792-00-3, doi: 10.24963/1jeee /022/3.. URL
fttps://www.iz http://wwwww..aaa..ooooooooeee// wwwpproceeddngg///022...
A Implementations from literature
This section contains implementation details of the GTDL models that are evaluated in Section 4 In the following Appendices A.
A.4 each first paragraph explains what kind of graph is learned, and where and how this is used by the original authors. In the subsections, discuss the adaptations made to make the learned graph structures compatible with our discussion and implementation.
Additional notation. We use N for the number of samples, L as the nunber of layers in the network, H as the mumber of transformer heads, and p as the number of features. To indicate values that are ranging between zero and one, we denote the corresponding set as Ro..1-
Interpreting the attention map as a weighted adjacency matrix. For most methods G'T-Transformer, T2G-Former and FiGNN), the learned graph comes from averaging the attention map a E [j,11]
N xpxp over
the samples, layers, and heads. This allention map is normalized with softmax across the last dimension. We note individual values from the attention map as dakjks where is the sample index, / is the layer index, h is the head index, and j, k are the feature indices. So the average attention map is az4 = NxLxH Ean aaiig E IRPXP [o.
We want to interpret the average attention map azk as the weighted adjacency matrix Ayk. For this, we have to 'denormalize' the average attention map. As the attention map normalized with softmax, the last dimension (the rows in the attention map per individual layer and head) sum to one, such that Σg Galik for all i, j. This gives problem, as the maximum value the attention map daiik can have cannot have
17

---

Published in Transactions on Machine Learning Research (03/2026)
two values close to 1 in the same row, while the weighted adjacency matrix Ays should be able to have multiple values close to in the same row.
To 'denormalize" the attention map, we add two steps. First, we set the diagonal of the attention map to zero, as the self-interactions should not be taken into account during evaluation of the feature interactions:
Guijk 0 Vj k.
Second, we divide the attention maps by the maximum value across the row to obtain the adjacency matrices:
Aunsk = Gangsh/ (ax(()))k))
By doing this, all the values with the highest attention across that row now have a value of 1 in the weighted adjacency matrix. Ignoring the diagonal the aftention map key step this procedure: If the model learns that it should not give high attention to the non-diagonal values (as those features are not related), the model should learn to give high attention to the diagonal values. The diagonal values are excluded in the denormalization and do not affect the adjacency matrix.
A.1 Tranffosformer
FT-Transformer
Gorishniy
al.
2021
http://gggttbbb.oooaadeeeeeeeeee XXLXHXpX
ttdl-rovisitting-mddeld learns the attention map a E
R 10.1
GGorishni et al. 2021 in-
terpret in Section 5.3 the average allention map 0zk = NxLxH Ean aihiik E R [0.1]
as feature importance.
For a few real-world datascts, they compare it to Integrated Gradients (Snndaaaraaan et al. 2017 using rank correlation and find that it performs similarly.
As the attention map normalized with the softmax, we denormalize to obtain the weighted adjacency matrix as described above. This the only post-hoc adaptation we made the original implementation.
A.2 T2G-Former
learn a feature-relation graph (Equation 6 in (Yan et al. 2023 ). The strength of the graph can be interpreted as the strength of the relations between the features. Section 5.3 and Figure 3 in (Yan al. 2023 show the
T2G-Former (Yan el al.
2023
(FR-Graple E R. NXLX xpxp
0.1]
FR-Graph for two real-world datasets.
Instead of the Hadamard product in equation 6 in Yan et al. 2023) to construct the FR-Graph, we use sum, consistent with their code implementation of the FR- "R-Graph Yan et al. 2023) present the FR- GR-Graph from the first and the last layer of the network. We assume that these are averaged over the samples and the heads. Instead, we average over all the layers, following the approach of PT--Tansformer. As the FR-Graph is normalized with the softmax, we denormalize the FR-Graph to obtain the weighted adjacency matrix A as described above.
A.3 INCE
learn edge embed- dings (xxx(p(--))xxam with demb the embedding dimension and (p(p 1)) the number of edges
INCE
(Villaizán Vallelado et al. 2024) http:://ghthbbbcooaaaattttsatttt
fully connected graph excluding self-loops. Section 6.2 of (Villaizán- Vallelado al.) 2024) presents an algorithm to caleulate the feature-feature interaction Pmat [ from the edge embeddings. Figure 11 in
Vallelado et al. 2024) shows the feature-feature interaction Pint on a real-worid dataset.
A lower value of Pint implics more significance. Therefore, we apply one additional step obtain the weightod ad adjacency matrix A Pint
18

---

Published in Transactions on Machine Learning Research (03/2026)
A.4 FiGNN
FIGNN
et a1 2019a
wttp:/hhwwbccco/ccaAA----////// CttCC-DDDGGGGGGG/teeeeeeeeee GNN/tree/
learn a feature graph in the form of attentional [=Xx for the input features (equations and 5 Li et al. (2019a)) The odge weights are interpreted as the importance of the interactions. Therefore, they are used to providing explanations on the relationship between different features. Section 4.5 and Figure 5, the edge weights are presented as a heat map, and are used explain the relations between features on real-world dataset.
edge weights E [0.1]
As the attention edge weights normalized with the softmax, we denormalize them to obtain the weighted adjaceney malrix A as described above. The code implementation of FiGNN is published with Tensor Flow. As our implementation is in PyTorch, we have adapted the code to PyTorch. The learned adjaceney matrix is of size (p - 1) x (p- 1), we impute an additional row and column for the target feature with values of zero.
There are different implementation versions of FIGNN. The first version of FIGNN has been presented at CTKM in November 2019, which is identical version on ArXiv (Li et al. 2019b). We use the implementation of this version. In July 2020, version 2 on ArXiv (Li et al. 2020 was published, and the main repository /https://github cow /RI.//RIIIAAAGIII.... GNN/) some additional attention layers. Furthermore, when inspecting the published code. We observed that this second version does not have trainable feature graph in its code implementation. Therefore, we stick to the original code implementation of version 1 (https://githi) htt/g/ttub.co////II... T2007222144455522000777777000600
updated accordingly. Version 2 has
cww/ Con/CRIPPACDIG/FF. DEA.IACCDDIG/F/GGG//teeeee CNN/tree//
B Data generation
In this section, we describe the graph and data generation process of the two synthetic dataset approaches introduced in Subsection 3.1 and their hyperparameters used in Section
Multivariate normals. We follow the default procedure of generating conditional multivariate data, Mohammadi & Wit 2015
(i) Sample true graph structure Gtrue RPxp from the Bernoulli distribution with an edge inclusion probability Podge-
(ii) Sample covariance matrix Eg E RRxx from the G-Wishart distribution (Roveraoo 2002 Letac &
Massam 2007), which is conditioned on the graph structure Gtrue
5
(iii Obtain n samples D E R2 from a multivariate normal distribution N'0 Eg).
In our experiments, we have p = 10 nodes, and an edge inclusion probability of Pedge 0.267. This results in the graph structures as depicted Figure
Figure 6: Graphs used in experiments for the MVN1, MVN2 and MVN3 datasets.
"This purpose specific commit, there are different implementations.
In fact, we sample the precision matrix Ke Egd form the G-Wishart distribution, and invert obtain the covariance matrix Eeg. The underscore "G indicates that the matrix conditioned the graph structure Grree-
19

---

Published in Transactions on Machine Learning Research (03/2026)
Structural causal models. We follow similar setup as (Hollmann et al. 2025 to generate an SCM and sample data conditional on the graph. They show that with their setup, the synthesized data is similar to real-world tabular data.
(i) Randomly sample a DAG, with nroot root nodes and p child nodes with a probability Pedge of an incoming edge. The undirected graph structure Girue obtained by moralizing and marginalizing the DAG (Cowell et al. 1999). Moralizing makes the graph undirected by dropping the direction of the existing edges and connecting all parents of child node. Marginalizing removes the root nodes from the graph, as they are not part of the datasel D. When marginalizing node, we connect all neighbors of the removed node to each other. In this work, only the root nodes are marginalized. This makes the order of moralization and marginalization irrelevant. The red lincs in Figure show the new edges that are added by moralization and marginalization.
(ii) Randomly sample deterministic computational mappings fi for each child node in the graph, where the mappings are smooth nonlinear functions, randomly picked from the set of maps listed in Table 2 computational map defines how a child node i is computed from its parents. They take all the values of the incoming edges as imput, and the output is the value of the child node i.
(ii) Randomly sample root nodes Tooo ~ (O, 1) Rox and traverse the DAG in topological order, = fi fi( R=x1 Each output 2; is normalized, clipped between (-3, 3), and Gaussian noise (0, 0.5) is added. This is summarized by
clip(normalize(f (FF..aaess) + (0,00,,, 33.3).
(2)
We consider all the traversed outputs as the datasct D RRxp.
Each DAG has p 10 nodes. These p nodes are evenly distributed over #DAG layers =3 3 layers, where each layer has minimum each layer has or 4 nodes. Each node has a Pedge 0.5 probability of having an edge the nodes in the next layer. With these hyperparameters, the three DAGs that are used in Section 4 are shown in Figure
nodes. The DAG has a 'zeroth' layer Thoont = 3 root nodes. This means that
Figure 7: Top: DAGs used in experiments for the SCM1, SCM2 and SCM3 datasets. Bottom: Corresponding undirected graph structures Giree after moralization and marginalization. Moralized and marginalized edges are depicted in red.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Computational maps
parents
[[Pprronts)
r//3
0,5 + 331
111 + 4
2
(x1 + x1)/2
s1 + uÿ T1
-(11 + x2) 2122
3
(r1 + 23)/3
-z 1233
(x) + 12 13) 11
C
Experiment details
Data splitting. We adapt our train, validation, and test splitting and our tuning strategy to balance between a fair comparison between different dataset sizes and efficient hyperparameter tuning.
Following (Grinsztajn et al. 2022 we differentiate between validation set used for carly stopping Dval, early stop and a validation set used for hyperparaameter luning D hparam, such that we have four disjoint sets: Dirans Dval, carly stops vaal, hparam: and Dtest. We vary the number of training samples Rerain in our experiments between 1000 and 4000, and set both Thest = Avval, hparam 2500 and Tvval, early stop 0.25m
In our experiments, do not change Dval, hparam and Drest to limit the number of cross-validation and iterations have to do. We randomly sample Durain and D early stop for each fold. This strategy is visualized in Figure 8 For Terain 1000 samples evaluate over folds, Buaii 2000 over fold, for Ttrain 3000 over 2 folds and for Mraai 4000 over fold.
ignored
Cross-validation
train
val es
val hparam
20
60
100
Data samples
Figure 8: Splitting strategy for the example N = 100 and Ptrain = 0.3N. The training set and the validation set for early stopping are randomly sampled for each fold. The test set and validation set for hyperparameter tuning are fixed.
Training and evaluation. We minimize the MSE loss function and optimize using Adam (Kingma & Ba 2017) with a fixed batch size of 256 and tune the learning rate together with the other model hyperparameters. We continue training until the validation loss does not improve for 10 epochs. There is a theoretical upper bound of 400 epochs, which rarely reached in practice. We select the best hyperparameters that minimize the MSE on the separate hyperparameter validation
After tuning, we run 10 runs per cross-validess crossvvalddttinn fold. We report the R2 score to evaluate the predictive performance of the target feature, and the ROC AUC to evaluate the learned feature interactions.
Hyperparameter tuning. For every combination of network, dataset, and Thrain we tune the model's hyperparameters and the learning rate. For all models, we use tree-structured Parzen estimator (TPE)
(Bergstra al.] 2011), a Bayesian optimization technique within the Optuna library (Akiba et al.] 2019 We run total of 50 trials for each setting, where the first trial has the default hyperparameters of the implementation. We keep the default setting of the Optuna implementation, where the first. 10 trials
21

---

Published in Transactions on Machine Learning Research (03/2026)
done with random search. The hyperparameters of TabPFN are not tuned, as it is pretrained model that can get accurate predictions without fine-tuning (Hollmann et al. 2025
The hyperparameter distribution and the default hyperparameters of all models are listed in Tables 3 to For all models, the search space and default values are taken from the original implementations if not specified otherwise in the caption the tables. The search space of layer count and embedding size is set the same for fairer comparison across models. The distribution space of the learning rate is Log LogUnform[[10- 1003] with default value of 10-3 for all models.
Table 3: FT-Transformer
Gorishniy et al. 2021 hyperparameter space.
Paramoter
Distribution
Default
Layer count Embedding size
UniformInt[L4
{8, 16, 32, 64, 128, 264)
128
Attention head count
Autention dropout FFN size factor
Uniform 0.5]
0.2
Uniform P/s, 7/s]
4/3
FFN dropout
Uniform (0.0, 0.5
0.1
Residual dropout
Uniform (0.0, 0.2
Table 4: T2G-Formes from FT-Transformer.
Yan et al.
2023
hyperparameter space.
Default values are taken from the same
Parameter
Distribution
Default
Layer count Embedding size
UniformInt(1, 6]
{8, 16, 32, 64, 128, 264)
128
Attention head count
Attention dropout
Uniform (0.0, 0.5]
0.2
FFN size factor
Uniform 1//s, 7/3]
4/3
FFN dropout
Uniform(0.. 0.5)
0.1
Table 5: INCE (Villaizán-Vallee et al.
2024
hyperparameter space.
Parameter
Distribution
Default
Layer count
UniformInt[1,0
4
Embedding size
(8, 16, 32, 64, 128, 264) {1,2,3,4 4}
128
MLP layer count
Dropout
Uniform (0.0. 0.5]
Table 6: FIGNN Li et al. 2019a hyperparameter space. The distribution space was not shared by the
original implementation.
Parameter
Distribution
Default
Layer count
UniformInt 6]
Embodding size
{8, 16, 32, 64, 128, 264) Uniform[0.0, 0.5)
16
Dropout
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: XGooost (Chen & Guestrin 2016 hyperparameter space, Default values are taken from the official
implementation (Chen al. 2026), distribution space the same (Grinsztajn
2022
Parameter
Distribution
Default
Max depth
UniformInt 33, 10]
6
Min child weight
LogUniform[10-11 10 1
Subsample
Uniform(0.5. 1.0]
1.0
Learning rate (eta)
LogUniforn[10-9, 1.0]
0.3
Colsample by level
(0.formm0.5,00 1.0
1.0
Colsample by tree
Uniform (0.5, 1.0
1.0
Gamma
LogUniform(100 100)
10
Lambda
LogUniform(110 100
1.0
Alpha
MogUniform[10-1 100
10-8
C.1
Statistical testing
To assess the statistical significance of the predictive performance improvement when using the pruned graph compared the fully connected graph, we used Linear Mixed Effects Model (Lindstrom & Bates 1988 Pinheiro & Bates 2000). We selected this approach over aggregating results per fold (e.g., for a Wilcoxon signed-rank test [Wilcoxon] 1945 to avoid the loss of statistical power, given the limited number of cross-validation folds (4, 3, 2, 1) available for 7ltrain (1000, 2000, 3000, 4000), respectively. Since evaluate 10 randoom seeds per fold, the data exhibits a hierarchical structure. Treating these seeds as independent samples would violate the independence assumption of standard tests and lead to pseudoreplication (Nadeau & BBngiio 1999). Therefore, we modeled the cross-validation fold as a random effect to account for the correlation belween seeds, and the graph type (pruned or fully connected) as a fixed effect. Although not. documented in this work, we compared the Linear Mixed Effects Model with the one-sided Mann-Whitee Whitney U test (Mann & Whitney 1947 and found comparable p-values.
D Additional results
D.1 Results per dataset
In Section 4 have discussed the results of the learned graph structure and the predictive performance of
the target feature while aggregating over three datasets per the dataset type MVN and SCM. In Figure 9
and Table we show the results per individual datasct for the learned graph structure, in Figure 10 and Table we show them for the predictive performance. The results are consistent with the results shown in Figure and Figure 5 no new insights are gained.
23

---

Published in Transactions on Machine Learning Research (03/2026)
TTTTrasformer
FIGNN
INCE
T2GFormer
BDgraph
MVN MVND
1.0
0.5
1.0
ROC AUC (Armm Ayoa)
MVN2
1.0
0.5
SCMO
1.0
SCM
1.0
0.5
SCM
1.0
0.5
2000
4000
2000
4000
2000 4000
2000
4000
2000
4000
Random guess
Available edges: Fully connected
Figure 9: Graph quality in the form of the ROC AUC comparing the learned weighted adjacency matrix with the true binary one, for six different datascts. Most GTDL models have ROC AUC ~ 0.5, which is random chance, indicating that they are not able to learn the feature interactions in any meaningful way. The PGM method BDgraph can learn the feature interactions. Error bars show standard deviation across seeds and cross validations. See Figure for the results aggregated over the two dataset types.
24

---

Table Graph quality in the form of the ROC AUC comparing the learned weighted adjacency matrix with the true binary one, for six diffcrent datasels and five different models. The statisties j and represent the mean and standard deviation aggregated over seeds and validations. See Figure for the same results.
Published in Transaction tons on Machine Learning Research (03 (03 (22))
Model
FTTraasformmr Fully connected
FIGNN Fully connected μ
INCE
T2GFormer
BDgraph Fully connected
Graph Statistic
Fully connected
Fully connected
/
/
Dataset
Ttrain
MVN
1000
4.548o-01
3.508c-02
4.121e-01
4.071c-02
5.057e-01
5.581c-02
5.344c-01
9.255e-02
9.639c-01
1.910c-02
2000
4.658e-01
4.352e-02
4.121e-01
3.872e-02
5.179e-01
5.41le-02
5.441e-01
5.511e-02
9.790e-01
2.515e-02
3000
4.560e-01
2.904c-02
4.122e-01
4.682c-02
5.249c-01
3.699c-02
5.592e-01
7.778e-02
9.976e-01
1.733c-03
4000
4.883e-01
5.100c-02
4.116e-01
4.091e-02
5.295e-01
4.251e-02
5.595c-01
7.586c-02
9.975e-01
0.000e+000
MVN
1000
5.203e-01
4.563o-02
5.226c-01
4.434c-02
4.704c-01
6.147c-02
5.096c-01
6.267c-02
9.594c-01
2.092c-02
2000
5.108e-01
2.726e-02
5.184e-01
5.335e-02
4.658e-01
6.090e-02
5.025e-01
6.883e-02
9.697e-01
1.518e-02
3000
5,178e-01
4.747e-02
5.216e-01
4.772e-02
4.989e-01
4.354e-02
5.327e-01
6.719e-02
9.753e-01
1.896e-02
4000
5.200c-01
3.1070-02
5.003c-01
5.355c-02
4.793c-01
3.654c-02
5.360o-01
5.847c-02
9.773c-01
7.434c-03
MVN
1000
4.768e-01
5.439e-02
4.049e-01
3.134c-02
5.429e-01
5.289e-02
5.431e-01
6.307e-02
9.460e-01
5.934e-02
2000
4.763e-01
4.671e-02
4.069e-01
3.329c-02
5.032e-01
6.248c-02
5.410e-01
7.662e-02
9.822e-01
5.646e-03
3000
4.899e-01
6.302e-02
4.041e-01
3.894e-02
5. 5.236e-01
7.065e-02
5.398e-01
6.247e-02
9.861e-01
8.534e-03
25
4000
5.127e-01
7.981c-02
4.029c-01
3.976c-02
5.364c-01
5.616c-02
5.097e-01
7.822c-02
9.934c-01
2.715c-03
SCM O
1000
7.471e-01
3.716e002
6.218e-01
7.268e-02
5.805e-01
8.022e-02
4.739e-01
6.207e-02
6.113e-01
2.409e-02
2000
7.250e-01
3.665e-02
6.024e-01
8.207e-02
6.618e-01
5.272e-02
5.03Se-01
6.840e-02
5.945e-01
1.699e-02
3000
7.387e-01
3.545e-02
5.639e-01
7.701e-02
6.381e-01
6.953e-02
5.201e-01
7.332e-02
5.973e-01
8.906e-03
4000
7.071e-01
5.140e-02
6.099e-01
6.310e-02
6.683e-01
3.808e-02
5.694e-01
6.003e-02
5.996e-01
1.497e-02
SCM
1000
3.775c-01
8.564e-02
3.476e-01
4.6426-02
5.406e-01
1.419c-01
5.655e-01
7.674e-02
8.255e-01
2.569c-02
2000
3.938e-01
7.435e-02
3.567e-01
5.796e-02
5.463e-01
1.173e-01
5.564e-01
7.292e-02
8.420e-01
2.206e-02
3000
3.908e-01
5.154e-02
3.409e-01
3.738e-02
4.957e-01
8.539c-02
5.550e-01
5.998e-02
8.288e-01
1.437e-02
4000
3.671e-01
4.565e-02
3.280e-01
6.537e-02
4.369e-01
5.861e-02
5.946e-01
8,459e-02
7.921e-01
1.123e-02
SCM 2
1000
4.575c-01
5.500c-02
4.058c-01
6.178c-02
5.050c-01
8.119c-02
5.489c-01
6.800c-02
6.656c-01
2.108c-02
2000
4.364e-01
4.861e-02
4.013e-01
5.555e-02
4.706e-01
6.578e-02
5.433e-01
6.028e-02
6.486e-01
1.562e-02
3000
4.714e-01
4.357e-02
4.139e-01
6.014e-02
4.938e-01
8.293e-02
5.456e-01
6.480e-02
6.692e-01
1.296e-02
4000
4.762e-01
4.352e-02
4.077e-01
7.953e-02
5.308e-01
5.504e-02
5.507e-01
5.860e-02
6.641e-01
1.610e-02

---

Published in Transactions on Machine Learning Research (03/2026)
FTT minfommer
FIGNN
INCE
T2GFormer
Not-GTDL beselines
A edges
0.00
Pully cormonul
MVND
0.28
Lisse Mixed
paslls
Signt simcrrdd (8.5)
Non-TDD howlinn
TdeeN
10-10
BDXinpi
0.300
0.224
0.50
10-10
MVN2
10-11
64
10-11
0.800
0.825
10-41
0.92
0.90
0.28
30 19
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
Reah
Figure 10: Predictive performance for six different datasets. When the graph is pruned to its true edges, the predictive performance is, most cases, better compared to the fully connected graph. The difference reduces as the number of training samples increases. This confirmed by the p-values the Linear Mixed-Flffen Model, where lower p-values indicate stronger evidence that the pruned graph results are better than the fully connected graph results. When p-values not visible, they below the lower limit of 10-13 Error bars show standard deviation across seeds and cross validations. See Figure D for the results aggregated over the two dataset types.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Predictive performance in the form of Ri, for six different datasets and five different models. We compare the performance when the graph is fully connected versus when is pruned its true edges only. The statistics μ and a represent the mean and standard deviation aggregated over seeds and cross validations. With the Linear Mixed Effects (LME) Model we test if the R² results obtained with the pruned graph greater than those with the fully connected graph. See Figure 10 for the same results.
Graph Statistic
Fully connected
True edges only μ
LME
p-value
Dataset
Model
Thrain
MVN 0 FTTransformer
1000
7.928c-01
2.000c-03
7.950c-01
1. 1.782c-03
1.221c-07
2000
7.910c-01
7.379c-03
7.935e-01
3.681c-03
9.239c-03
3000
7.957e-01
1.114e-03
7.965e-01
1.3746-03
1.568e-02
4000
7.975e-01
7.178e-04
7.983e-01
1.299e-03
5.861e-02
FIGNN
1000
7.883c-01
3.955e-03
7.879c-01
3.121c-03
6.958c-01
2000
7.922e-01
2.499e-03
7.908e-01
2.436e-03
9.934e-01
3000
7.948c-01
2.658c-03
7.938c-01
1.819-03
9. 1.1222-01
4000
7.952e-01
1.693e-03
7.951e-01
1.575e-03
5.715e-01
INCE
1000
7.830e-01
4.252e-03
7.857e-01
9.369e-03
1.698e-02
2000
7.863c-01
5.282c-03
7.916e001
3.636o-03
1.403e-07
3000
7.911e-01
2.362e-03
7.936e-01
1.393e-03
1.593e-05
4000
7.918e-01
1.664e-03
7.929c-01
2.931e-03
1.519e-01
T2GFormer
1000
7.61le-01
2.067e-02
7.810e-01
1.160e-02
3.7688-08
2000
7.741c-01
1,252c-02
7.888c-01
5.172c-03
1.383c-09
3000
7.830e-01
9.613e-03
7.894e-01
4.455e-03
9.891e-04
4000
7.851e-01
8.265e-03
7.897e-01
5.989e-03
8.067e-02
TabPFN
1000
7.970e-01
1.230e-03
2000
7.985c-01
2.419c-04
3000
7.991e-01
1.497e-04
4000
7.989e-01
6.862e-05
XGBoost
1000
7.772e-01
7.654e-03
2000
7.867e-01
1.211e-03
3000
7.896c-01
9.442c-04
4000
7.843e-01
1.433e-03
BDgraph
1000
7.990e-01
2.730e-04
2000
7.994e-01
9.546e-05
3000
7.993c-01
3.010c-04
4000
7.994e-01
6.052e-06
MVN Fraanssforerr
1000
2.873e-01
1.157e-02
3.008e-01
7.322e-03
1.013e-12
2000
3.001c-01
4.982c-03
3.045c-01
4.561c-03
9.286c-05
3000
3.008c-01
6.024c-03
3.097c-01
3.192c-03
6.228c-10
4000
3.072c-01
3.170e-03
3.059e-01
3.2576-03
8.191e-01
FIGNN
1000
2.914e-01
9.587e-03
2.884e-01
1.186e-02
8,939e-01
2000
2.932c-01
1.712e-02
2.984c011
6.712c-03
4.367c-02
3000
3.031e-01
7.763e-03
2.973x-01
2.127e-02
8.810e-01
4000
3.053c-01
3.281c-03
3.071c-01
2.695c-03
8.243c-02
INCE
1000
2.688e-01
1.387e-02
2.953e-01
6.880e-03
1.989e-35
2000
2.823e-01
6.577e-03
3.009e-01
5.433e-03
1.105e-43
3000
2.952c-01
7.278e-03
3.073c-01
5.1256-03
2.980c-17
4000
2.991e-01
3.093e-03
3.080e-01
1.606e-03
2.957e-16
F2GFormer
1000
2.626e-01
2.122e-02
3.002+011
9.836e-03
2.980e-26
2000
2.767e-01
1.393e-02
2.959e-01
1.383e-02
1.249e-09
Continued on next page
27

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Predictive performance in the form R², for six different datasets and five different models. We compare the performance when the graph fully connected versus when it pruned to its true odges only, The statistics μ and represent the mean and standard deviation aggregated over seeds and cross validations. With the Linear Mixed Effects (LME) Model test if the R² results obtained with the pruned graph greater than those with the fully connected graph. See Figure 10 for the same results.
Graph Statistic
Fully connected
True edges only
LME p-value
Dataset
Model
Thrain
3000
2.927c-01
1.595c-02
3.082c-01
3.218c-03 5.987c-03
1.576c-06
4000
3.023e-01
7.366e-03
3.090e-01
1.329c-02
TabPFN
1000
3.103e-01
2.905e-03
2000
3.146e-01
1.909e-03
3000
3.151c-01
3.399e-04
4000
3.149e-01
2.604e-04
XGBoost
1000
2.788c-01
8.729c-03
2000
2.894e-01
3.750e-03
3000
2.996e-01
4.644e-03
4000
2.949c-01
3.129c-03
BDgraph
1000
3.145e-01
1.158e-03
2000
3.150e-01
1.218e-03
3000
3.151e-01
8.791e-04
4000
3.155c-01
2.023c-05
MVN 2
FTTransformer
1000
3.11le-01
1.77le-02
3.218e-01
1.356e-02
4.767e-06
2000
3.265e-01
5.374e-03
3.196e-01
1.063e-02
9.999e-01
3000
3.271e-01
2.848e-03
3.273e-01
3.762e-03
4.482e-01
4000
2.952c-01
1.051c-01
3.331c-01
2.340c-03
1.272e-01
FIGNN
1000
3.110e-01
8.880e-03
3.069c001
1.322e-02
9.635e-01
2000
3.207e-01
5.036e-03
3.198e-01
4.937e-03
7.556e-01
3000
3.280e-01
4.337e-03
3.240e-01
3.670e-03
9.993e-01
4000
3.290e-01
2.607e-03
3.273e-01
2.265e-03
9.404e-01
INCE
1000
2.751c-01
2.681c-02
2.985c-01
2.614-002
1.103c-08
2000
2.977e-01
1.265e-02
3.040e-01
1.372e-02
8.489e-03
3000
3.159e-01
5.979e-03
3.211e-01
5.549e-03
8.908e-04
4000
3.177e-01
6.735e-03
3.283e-01
2.787e-03
2.291e-06
T2GFormer
1000
2.725c-01
3.057c-02
2.964c001
1.609c-02
9.802c-07
2000
2.904e-01
2.975e-02
3.150e-01
1.239e-02
3.412e-08
3000
3.140e-01
1.346e-02
3.203e-01
7.734e-03
1.832e-02
4000
3.049c-01
1.888c-02
3.193c-01
1.188c-02
2.071c-02
TabPEN
1000
3.035c-01
1.660e-03
2000
3.046e-01
5.855e-04
3000
3.048e-01
3.652e-04
4000
3.061c-01
3.168e-04
GGBoott
1000
2.641e-01
2.197e-02
2000
2.789c-01
3.089c-03
3000
2.890e-01
3.739e-03
4000
2.889e-01
2.763e-03
BDgraph
1000
3.376e-01
1.335c-03
2000
3.392e-01
2.941e-04
3000
3.380e-01
1.755e-04
4000
3.380e-01
1.424e-05
SCM 0
FTTransformor
1000
6.852c-01
1.038c-02
7.012c-01
8.936c-03
3.191c-14
Continued on next page
28

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Predictive performance in the form R², for six different datasets and five different models. We compare the performance when the graph fully connected versus when it pruned to its true odges only, The statistics μ and represent the mean and standard deviation aggregated over seeds and cross validations. With the Linear Mixed Effects (LME) Model test if the R² results obtained with the pruned graph greater than those with the fully connected graph. See Figure 10 for the same results.
Graph Statistic
Fully connected
True edges only
LME p-value
Dataset
Model
Thrain
2000
6.996c-01
1.954c-02
7.195c-01
7.664c-03
5.864c-12
3000
7.166e-01
5.022c-03
7.186c-01
3.711c-03
6.297e-02
4000
7.176e-01
4.316e-03
7.191e-01
3. 3.1960-03
1.897e-01
FIGNN
1000
6.857e-01
8.771e-03
6.770e-01
1.356e-02
9.999e-01
2000
6.987e-01
5.186e-03
6.896c-01
1.378c-02
9.998c-01
3000
6.985e-01
1.175e-02
7.051e-01
4.950e-03
7.274e-03
4000
7.122c-01
3.105c-03
7.129c-01
4.739c-03
3.559c-01
INCE
1000
6.169e-01
1.583e-02
6.934e-01
1.276e-02
7.188e-139
2000
6.769e-01
1.258e-02
7.155e-01
3.774e-03
8.700e-
3000
6.881c-01
8.860e-03
7.149e-01
5.249c-03
4.991c-32
4000
6.993e-01
1.378e-02
7.200e-01
3.107e-03
1.913e-06
T2GFormer
1000
6.064e-01
6.247e-02
6.952e-01
1.588e-02
1.506e-18
2000
6.792e-01
1.816e-02
7.134e-01
9.392e-03
4.987e-22
3000
7.027c-01
9.075c-03
7.054c-01
1.035c-02
1.934c-01
4000
7.079e-01
5.211e-03
7.218e-01
3.525e-03
1.184e-12
TabPFN
1000
7.199e-01
3.512e-03
2000
7.260e-01
1.418e-03
3000
7.278c-01
4.063e-04
4000
7.288e-01
1.264e-04
XGBoost
1000
6.038e-01
2.008e-02
2000
6.360e-01
4.754e-03
3000
6.582e-01
1.142e-02
4000
6.790c-01
3.240c-03
BDgraph
1000
2.865e-01
4.755e-03
2000
2.895e-01
4.179e-03
3000
2.903e-01
1.251e-03
4000
2.944c-01
9.584c-05
SCM FTTransformer
1000
8.421e-01
4.800e-03
8.476e-01
6. 03.6-3-3
3.358e-07
2000
8.530e-01
3.025e-03
8.578e-01
3. 103e-00
1.220e-12
3000
8.536c-01
8.851c-03
8.614c-01
3.758c-03
2.128c-07
4000
8.655c-01
1.603e-03
8.629c-01
3.397c-03
9.864c-01
FIGNN
1000
8.353e-01
1.802e-02
8.456e-01
4.649c-03
6.706e-05
2000
8.544e-01
3.347e-03
8.549e-01
2.867e-03
2.557e-01
3000
8.575c-01
4.599e-03
8.579c-01
5.205c-03
3.690c-01
4000
8.601e-01
1.891e-03
8.618
2.076e-03
3.270e-02
INCE
1000
8.407c-01
4.241c-03
8.4836-01
-.
1.477c-19
2000
8.489e-01
3.068e-03
8.532e-01
3.350e-03
5.220e-08
3000
8.549e-01
2.966e-03
8.593e-01
2.800e-03
7.280e-08
4000
8.572c-01
3.749c-03
8.629e-01
1.749c-03
6.160e-06
T2GForrmer
1000
8.302e-01
6.270e-03
8.286e-01
1.756e-02
7.166e-01
2000
8.402e-01
6.429e-03
8.444e001
7.657e-03
9.535e-03
3000
8.475e-01
5.003e-03
8.444e-01
1.11le-02
8.923e-01
4000
8.508c-01
5.679c-03
8.540c-01
3.901c-03
7.649c-02
Continued on next page
29

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Predictive performance in the form R², for six different datasets and five different models. We compare the performance when the graph fully connected versus when it pruned to its true odges only. The statistics μ and represent the mean and standard deviation aggregated over seeds and cross validations. With the Linear Mixed Effects (LME) Model test if the R² results obtained with the pruned graph greater than those with the fully connected graph. See Figure 10 for the same results.
Graph Statistic
Fully connected
True edges only
LME p-value
Dataset
Model
Thrain
TabPEN
1000
8.721c-01
1.974o-03
2000
8.756e-01
9.270e-04
3000
8.780e-01
3.718c-04
4000
8.783e-01
2.596e-04
XGBoost
1000
8.495c-01
4.047e-03
2000
8.598e-01
1.057e-03
3000
8.618c-01
7.716c-04
4000
8.642e-01
7.016e-04
BDgraph
1000
7.968e-01
1.242e-03
2000
7.973c-01
5.478c-04
3000
7.972e-01
1.416e-04
4000
7.974e-01
1.145e-05
SCM 2
FTTransformer
1000
7.988e-01
6.280e-03
8.068e-01
5.799e-03
2.428e-11
2000
8.069c-01
5.623c-03
8.118c-01
3.220c-03
2.246c-07
3000
8.122e-01
2.971e-03
8.158e-01
1.102e-03
2.046e-08
4000
8.137e-01
9.271e-04
8.171e-01
1.481c-03
4.620e-10
FIGNN
1000
7.913e-01
1.540e-02
7.994e-01
1.198e-02
1.783e-04
2000
8.093e-01
5.427e-03
8.112c-01
6.110c-03
5.826e-02
3000
8.152e-01
3.812e-03
8.073e-01
1.318e-02
9.948e-01
4000
8.162e-01
1.393e-03
8.166e-01
1.864e-03
2.944e-01
INCE
1000
7.922e-01
6.950e-03
8.021e-01
5.126e-03
7.559e-20
2000
8.033e-01
3.749e-03
8.054e-01
5.855e-03
4.815e-02
3000
8.068c-01
4.149c-03
8.137c-01
3.154e-03
2.383c-18
4000
8.069e-01
3.399e-03
8.132e-01
2.133e-03
2.872e-07
T2GFormer
1000
7.863e-01
1.251e-02
7.923e-01
1.657e-02
2.163e-02
2000
7.974e-01
1.011e-02
8.084e-01
4.063e-03
1.483e-09
3000
7.993c-01
9.041c-03
8.115c-01
4.340c-03
2.233c-09
4000
8.020e-01
8.168e-03
8.125e-01
4.053e-03
1.372e-04
TabPFN
1000
7.972e-01
1.963e-03
2000
8.024c-01
1.394c-03
3000
8.035c-01
1.564e-03
4000
8.042c-01
1.431e-03
XGBoost
1000
7.725e-01
5.500e-03
2000
7.858c-01
1.684e-03
3000
7.908e-01
1.714e-03
4000
7.896c-01
1.285c-03
BDgraph
1000
7.575e-01
1.307e-03
2000
7.587e-01
3.103e-04
3000
7.606c-01
4.838c-05
4000
7.607e-01
1.319e-05
30

---

Published in Transactions on Machine Learning Research (03/2026)
D.2 Higher number of training samples
We repeat the experiments from Section 4 while varying the number of training samples Rurain up to 10ah samples. Note that 10 samples is quite high, considering the low number of features p 10 and the relatively simple feature interactions in the datasets. To reduce computational cost, we do not tune the hyperparameters but use the default hyperparameters as previously described in Appendix [ and we only evaluate two of the six datasets. We do use the same cross-validation strategy with 10 seeds as before. The results for the Icarned graph structure are shown in Figure results for the predictive performance are shown in Figure [2 The results are consistent with the main experiment (tuned hyperparameters, up Arrain 4000 samples), shown in Figure 9 and Figure 10 However, the results have larger variety because the hyperparameters are not tuned.
FI Frannsooeer
RONN
INCE
TZGFomer
0.8
S030
0.5
ROC (AC Aved)
04
0.6
SCM
0.3
10'
10
10
10'
guess
edges: connected
Figure 11: Graph quality in the form of the ROC AUC with higher number of training samples Thtran than the main experiment, for two different datasets. Error bars show standard deviation across seeds and cross validations.
FTirantomer
RGNN
INCE
T2GPumer
Avaiable conaccal
SC3O
0.6
0.5
Sc000
1
SCNT
0.80
Figure 12: Predictive performance with a higher number of training samples Purain than the main experiment, for two different datasets. Error bars show standard deviation across seeds and cross validations.
D.3 Node-level versus graph-level models
In Figure 5) not all models benefit from pruning the graph. For FIGNN, the pruned and fully connected graphs have similar performance. This could be because FiGNN treats the task of predicting the target feature on a graph-level task, while the other models have target token and treat as node-level task. As an example, we adapt the architecture from T2G-Former, which is by default node-level model, to a graph-level model. Results the predictive performance shown in Figure 13 The graph-level model
31

---

Published in Transactions on Machine Learning Research (03/2026)
benefits less from pruming the graph than the node-level model. This indicates that the node-level models are more sensitive to the graph structure than the graph-level models.
MVN2
SCMO
SCM2
0.30
0.54
RR-score
0.82
0.28
0.5
0.78
0.76
0.20
0.34
0.80
Node
Graph
Node
Graph
Node
Graph
Nod:
Graph
Nork
Graphh
Nisl
Criph
olges
odaes only
Figure 13: Node-level (default) versus graph-level adaptions of F2r-Frrmer for multiple MVN and SCM datasets. The node-level adaptation benefits more from pruning the graph to the true edges, Error bars show standard deviation across sceds and cross validations.
32


Reviews:
[{"review_id": "HHx6v77WOh", "reviewer": "Reviewer_bWh1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates whether Graph-based Tabular Deep Learning (GTDL) methods—including both explicit feature-level GNNs and implicit attention-based architectures—actually learn meaningful feature-interaction graphs.\nThe authors' contributions include:\n• A Systematic Evaluation Framework: A pipeline to generate synthetic datasets with known ground-truth interaction structures using multivariate normal (MVN) models and structural causal models (SCM).\n• Quantitative Benchmarking: An evaluation using ROC-AUC to compare learned adjacency matrices against the ground-truth graph.\n• Empirical Analysis: An assessment of several GTDL models, revealing that they currently fail to recover true interactions (performing near random guessing).\n• Performance Insight: Evidence showing that manually imposing the true graph structure via pruning improves predictive performance, suggesting that current models suffer from an inability to learn the correct structural inductive bias.", "claims_and_evidence_yn": "No", "claims_explanation": "No, the evidence is currently only partially convincing. While the empirical findings clearly show that the tested models fail to recover ground-truth graphs, the conclusion that these models are inherently unable to learn such interactions is undermined by the experimental setup:\n• Sample Size Limitations: The datasets used are relatively small (1,000 to 4,000 samples). Complex GTDL models may require significantly more data to infer intricate interaction patterns. This is corroborated by Figure 10 in the Appendix, which shows a performance trend upward as sample size increases. Without exploring larger data regimes, it is unclear if the failure is architectural or simply a result of data scarcity.\n• Lack of Baselines: The absence of traditional ML baselines (e.g., XGBoost, Random Forest) makes it difficult to assess the difficulty of the task. If standard models also fail to capture interactions in these specific regimes, the issue might lie in the dataset complexity rather than the GTDL architectures themselves.\n• Interpretability of Attention: The mapping of attention matrices to weighted adjacency graphs is a strong assumption. The paper lacks a rigorous justification for this mapping, which is critical given the ongoing academic debate regarding the interpretability of attention.", "audience_yn": "Yes", "audience_explanation": "Yes. The TMLR audience, particularly those working on Tabular Deep Learning and Graph Neural Networks, would find this work highly relevant.\nThe paper challenges a core assumption in the GTDL literature: that these models \"learn\" the underlying structure of tabular data. By providing a reproducible framework for verifying these claims, the authors encourage the community to move toward more rigorous evaluation standards and to develop models with more explicit structure-aware objectives.", "requested_changes": "To meet the threshold for acceptance, the following changes are requested:\n1. Large-Scale Experiments: Conduct experiments on significantly larger synthetic datasets to determine if the failure to learn interactions persists or if it is a function of sample size.\n2. Baselines Comparison: Include traditional machine learning models (XGBoost, LightGBM, or Random Forests). Use their internal feature importance or interaction scores to provide a benchmark for the \"learnability\" of the ground-truth graphs.\n3. Justification of Metrics: Elaborate on the rationale for treating attention weights as a proxy for adjacency, addressing potential limitations of this interpretation.", "broader_impact_concerns": "There are no significant ethical or broader impact concerns for this work. The paper focuses on the fundamental evaluation of machine learning architectures using synthetic data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fBPRkkqkQd", "reviewer": "Reviewer_gzJV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts an empirical study to determine whether established graph-based tabular models (GTDL methods) are able to discover the true feature interaction graph. To this end, the authors construct two graph generation mechanisms which model linear and non-linear interactions, respectively. Data generated from these mechanisms is then used to train GTDL methods and to determine how well these methods are able to recover the original graph structure. The authors also perform an ablation on substituting the randomly initialized complete graph with the true graph to determine the effect of discovering the true feature interactions on predictive performance.\n\n### Weakness\nThis could be due to a misunderstanding on my part, but from what I understand, in GTDL, the interaction graph weights are learned from e.g., an attention matrix like computation before being fed into a GNN, or directly processed as part of a transformer. Why do the authors expect the models to learn the feature interaction graph in the first place? For example, the inverse of the interaction graph could give equally strong signal to a GNN (especially, if it can learn or make changes to the edge embeddings). In a transformer, I understand the intuition more clearly, since lower attention weights literally mean that less information is exchanged between features. However, GNNs could learn to pay more \"attention\" to low edge weights, and still have strong predictive power.", "claims_and_evidence_yn": "Yes", "claims_explanation": "To me the main claim is that the presented GTDL methods are unable to accurately determine the true underlying feature interactions. This claim has been convincingly supported by the main experiments. I agree with the authors that if the methods cannot determine the feature interactions in simple synthetic data, they are unlikely to be broadly applicable in the real-world. Results in Figure 4 are very convincing, essentially establishing that discovering the true structure is possible (at least in the linear case), but not by the GTDL methods.\n\nAnother claim is that providing the ground-truth feature interactions helps in terms of predictive performance, which is not as convincing, but still convincing enough. Specifically, the R2 scores with and without pruned edges are often within each other's standard deviation. And even if not, they are only marginally outside. I would have expected the gap to be much clearer, especially given the simple nature of these tasks. I'd appreciate any comments on this by the authors.", "audience_yn": "Yes", "audience_explanation": "Given the strong interest in tabular deep learning (TDL), I am very certain that the findings in this paper are highly relevant, in particular to researchers aiming to improve graph-based TDL methods.", "requested_changes": "None of these requests are critical to my recommendation:\n\n* Please formally introduce the GNN based methods for GTDL. You do not have to introduce them all, but it would be very useful to at least describe the overall strategy more formally. From Section 2.2 alone, it is not fully clear exactly why GNNs are necessary here, or how they can be distinguished from Transformer-based methods.\n* The attention equation does not really serve a real purpose in the paper. The construction of Q, K, and V matrices in the context of GTDL isn't even specified. Instead of this equation, I would recommend to add more information to the main text on how the attention map is extracted.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BrCL9pf05l", "reviewer": "Reviewer_JABv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the capability of Graph-based Tabular Deep Learning (GTDL) methods to learn feature interactions. The authors introduce a framework using synthetic datasets where the ground-truth feature graph is known. Through this, they demonstrate that existing GTDL methods (such as FiGNN and FT-Transformer) fail to recover meaningful feature interactions, yielding ROC AUC scores near random chance. Furthermore, the authors show that pruning the graph to include only true edges generally improves predictive performance, validating the hypothesis that structural fidelity drives accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The problem outlined is significant, and the motivation is scientifically plausible. The introduction flows well, and the proposed approach of using synthetic data to isolate structural learning is well-designed. The results are convincing, and the provision of code and a clear evaluation setup is commendable.", "audience_yn": "Yes", "audience_explanation": "The findings of this paper are likely to be of significant interest to the TMLR audience, particularly those working within the Graph Learning and Tabular Learning communities. The work addresses a notable gap in the existing literature by systematically benchmarking GTDL methods using synthetic data with known ground-truth feature graphs. This methodological contribution, combined with the paper's clear structure and the availability of code for reproducibility, makes it a valuable resource.", "requested_changes": "Major Comments\n- **Claims regarding Data Symmetry** The statement in Section 4.2—\"When data is scarce, symmetries in the data are useful. However, when data is abundant, the model does not have to rely on symmetries in the data\"—is too bold and incorrect. This contradicts recent literature on data symmetry discovery (e.g., Marchetti et al., \"Harmonics of Learning\"). Symmetries remain important in both regimes; however, in large-scale data regimes, these symmetries are often learned implicitly rather than requiring explicit inductive bias. The authors should revise this to reflect that the abundance of data facilitates implicit learning, rather than rendering symmetries useless.\n\n- **Exclusion of State-of-the-Art Baselines** The authors explicitly exclude tree-based models (XGBoost, LightGBM) and recent foundation models (TabPFN, TabICL, etc.). While I understand these methods do not explicitly represent graphs, excluding them contradicts the work's motivation. The paper aims to validate the impact of true feature interactions on downstream performance. To prove that explicit structure is beneficial, the proposed technique (GTDL with true graphs) must be benchmarked and discussed against SOTA tabular methods that model interactions implicitly.\n\n- **Statistical Verification** Figure 5 outlines the main results regarding predictive performance. While the improvement from pruning the graph is visible, the paper lacks statistical rigor here. I strongly suggest adding statistical tests (e.g., t-tests or Wilcoxon signed-rank tests) to verify the significance of the performance gap between the \"Fully connected\" and \"True edges only\" settings in Section 4.2. Also, the inclusion of explicit tables would be beneficial for the community.\n\n- **Clarity of the Motivating Example** The example provided in Section 3.1 ($x_0 - x_1 - x_2$) is confusing. The text argues that the model should instead recognize $x_1$ as a better predictor for $x_2$. It is unclear why $x_1$ is inherently better than $x_0$ for predicting $x_2$ without more context. I would ask the authors to clarify this example to make the argument clear and potentially provide other examples. \n\n\nMinor Comments & Clarifications\n- In Section 3.3/4.2, the authors state: “By restricting the model to only the true interactions, the learning process becomes more efficient and focused…”. What specifically is meant by a \"more focused\" learning procedure? \n- While Figure 4 accurately outlines the claims, the visual presentation could be improved for better readability. (Suggestion only).", "broader_impact_concerns": "NA", "additional_comments": "I have enjoyed reviewing this work; I find it interesting and scientifically sound. I am willing to recommend this work for acceptance after the mentioned concerns are tackled.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)