FEAT-KD: Learning Concise Representations for Single and Multi-Target Regression via TabNet Knowledge Distillation
Authors: Kei Sen Fong
Mehul Motani
Venue: ICML 2025 poster (2025)
Keywords: FEAT
Knowledge Distillation
Symbolic Regression
Primary area: general_machine_learning->representation_learning

Abstract: In this work, we propose a novel approach that combines the strengths of FEAT and TabNet through knowledge distillation (KD), which we term FEAT-KD. FEAT is an intrinsically interpretable machine learning (ML) algorithm that constructs a weighted linear combination of concisely-represented features discovered via genetic programming optimization, which can often be inefficient. FEAT-KD leverages TabNet's deep-learning-based optimization and feature selection mechanisms instead. FEAT-KD finds a weighted linear combination of concisely-represented, symbolic features that are derived from piece-wise distillation of a trained TabNet model. We analyze FEAT-KD on regression tasks from two perspectives: 
(i) compared to TabNet, FEAT-KD significantly reduces model complexity while retaining competitive predictive performance, effectively converting a black-box deep learning model into a more interpretable white-box representation, (ii) compared to FEAT, our method consistently outperforms in prediction accuracy, produces more compact models, and reduces the complexity of learned symbolic expressions. In addition, we demonstrate that FEAT-KD easily supports multi-target regression, in which the shared features contribute to the interpretability of the system. Our results suggest that FEAT-KD is a promising direction for interpretable ML, bridging the gap between deep learning's predictive power and the intrinsic transparency of symbolic models.

FEAT-KD: Learning Concise Representations for Single and Multi-Targed Regression via TabNet Knowledge Distillation
Kei Sen Fong'
Mehul Motani
2.2
Abstract
1. Introduction
The representation of data is an important factor for the pre- dictive performance of machine learning (ML. algorithms (Bengio et al., 2013) and is key reason why deep learning- based approaches, which possess large representational ca- pacity are among the top-performing algorithms in many applications (Hinton, 2023). In practice, certain application areas, such as healthcare, demand more than just predictive performance the explainability of the learned represen- tation becomes precondition for assessing rithm's 's suitability for the problem (Di Martino & Delmastro, 2023). This is why low-complexity ML algorithms are still commonly used in various fields where it is important to understand and explain a model's predictions (Habehh & Gohel, 2021). While some researchers have attempted to create post-hoe explanations to black-box models, these are often not sufficient, and pale in comparison to white-box models with concise representations that provide intrinsic explainability (La Cava et al., 2023; Bordt et al., 2022). In these applications, intrinsic explainability is superior be- cause it provides immediate, transparent insights, fostering accountability in high-stakes domain. This is unlike post- hoc explainability which typically relies on approximations that may misrepresent model's true decision process. Ad- ditionally, intrinsically explainable models are not limited in their ability to leverage post-hoc techniques well.
In this work, propose novel approach that combines the strengths of FEAT and Tab- Net through knowledge distillation (KD), which we term FEAT-KD. FEAT an intrinsically interpretable machine learning (ML) algorithm that constructs weighted linear combination of concisely-represeented features discovered via ge- netic programming optimization, which can often be inefficient. FEAT-KD leverages TabNet's deep- learning-basse based optimization and feature selection mechanisms instead. FEAT-KD finds a weighted linear combination of voncisely-represemne bolic features that are derived from piece-wing wise dis- tillation of trained TabNet model. We analyze FEAT-KD regression tasks from two perspec- tives: (i) compared to TabNet, FEAT-KD signifi- cantly reduces model complexity while retaining competitive predictive performance, effectively converting a black-box deep learning model into more interpretable white-box representation, (ii) compared to FEAT, our method consistently out- performs in prediction accuracy, produces more compact models, and reduces the complexity of learned symbolic expressions. In addition, we demonstrate that FEAT-KD casily supports multi- target regression, which the shared features contribute to the interpretability of the system. Our results suggest that FEAT-KD a promising direction for interpretable ML, bridging the gap between deep learning's 's predictive power and the intrinsic transparency of symbolic models.
ML algo-
sym-
FEAT (Feature Engineering Automation Tool) is an ML algorithm that provides a unique white-box model structure with concise representations that provides intrinsic explain- ability (La Cava al., 2019). FEAT finds weighted linear combination of concisely-reprresented features that are dis- covered via genetic programming. To illustrate, consider i(x) 0.3(zz x2) + 0.5(x3 + =x4²), where f((x is the output and x, are the features. Here, the two concisely- represented features are 21 x 22 and I3 + xxa which has similar structure to popular indicators, such as the body mass index, mass + heigh? (Keys et al., 1972). FEAT pos- sesses similar interpretability that linear regression models have, in terms of decomposing the output predictions into in- dependent contributions from features, but allows for more complex features to better fit the dataset. The performance gain more than compensates for the slight decrease in inter-
Department of Electrical and Computer Engineering, National University of Singapore, Singapore. "N.1 Institute for Health, Institute for Digital Medicine (WisDM). Institute of Data Sci- ence, National University of Singapore, Singapore. Correspon- dence to: Kei Sen Fong <ffngkeiisnn @ www.wuu.uu..ooommmmmm edu>, Mchul Motani <motani edu..ed..ss>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FEAT-KD: Learning Concise Representations for Single and Multi-Tarna Target Regression via TabNet Knowledge Distillation
TabNet (already trained
Fully Connected
7()
the dataset):
ReLU
ReLU
Split
Split
Split
Feature transformer
Feature transformer
Feature transformer
Attentive transformer
Attentive transformer
Mask
Mask
Batch Normalization
Step
Step
Features,
Knowledge
N test samples
distillation
Brighter colors indicate features are unmasked (l.e., selected features)
process:
23 X5
Knowledge Distillation into a concise equation (via Exhaustive Search SR algorithm, l.e. DistilSR)
Knowledge Distillation into concise equation (vla Exhaustive Search SR algorithm, i.e., DistilSR)
Connisely-Repressene Feature (e.g., X2 X xz)
Feature (e.g., X3 + xf)
FEAT-KD:
Fully Connected
9(x)
Concisely- Prprissly-Represeente Feature 1 (e.g., x, x2)
Koncisely-Represeee Feature (e.g., Xg xii)
Step
Step 2
Features,
Alternatively, can be expressed single line, 9.g. 0.3(x, x xx2) + 0.5 xi)
Figure 1: FEAT-KD performs knowledge distillation For each 'Step' in TabNet (see dotted boxed 'Step 1', 'Step 2'), the knowledge distillation process takes the top original features selected by the masks in TabNet (e.g., I3, 244) and uses these subset of original features to discover a concisely- represented feature (e.g., 23 + x44 This is done via a non-genetic con--enetic-prograaee programming exhaustive search symbolic regression algorithm. In notations we will introduce later in Section 2.1, the output of FEAT-KD here f(x) = 3101 + 8202, where
tranned TabNet to obtain mancisely-reprseennne features.
3, = 0.3, Φ1 = σι x 22, 32 = 0.5, de I3 + 242
pretability compared to vanilla linear regression since the resultant model is still intrinsically explainable (La Cava al., 2023). Thus, FEAT has become first-class algorithm for real-world applications that values intrinsic 'white-box' explainability over post-hoc explanations.
2023), which motivates research into Seap-learning--oee based optimization. In this work, we propose PEAT-Knowleede. Distillation (FEAT-KD), which finds weighted linear com- bination of concisely-represene are derived from piece-wise distillation of a trained TabNet model. In other words, we train TabNet models for regres- sion tasks and perform knowledge distillation on pieces of the TabNet model to convert it into the same model struc- ture that FEAT produces: a weighted linear combination of ameisely-represeee features (see Figure I for general overview and Table for example of TabNet's output).
symbolic features that
To enhance FEAT, we are motivated to couple it with deep- learning-based optimization (such as TabNet) which is well- known and has been successful for a wide range learning problems. Recent evidence has also shown that genetic pro- gramming, which FEAT uses, be grossly inefficient in symbolic tasks (Kronberger et al., 2024; Fong & Motani,
2

---

FEAT-KD; Learning Concise Representations for Single and Multi-Targer Regression via TabNet Knowledge Distillation
Table I: FEAT-KD Example for Multi- Multi-Target Regression: Each target in the regression problem (e.g., atpld dataset, that features, Φ1 to ΦG.
has 6 targets) is predicted using a weighted linear combination of the same 6 shared The weights associated with each feature, 81 to 26 are shown here as illustration.
Concisely-rppresenn
Weight values 3 for the 7ithh target:
feature
71
11 ===
n =3 3 n
n = 5
7 =
Φ1 (2297) 2 X 2119
81 0.00329
-0.0153
-0.00938
-0.00958
0.00499
-0.01
< (1'297 + 1.03) x #119
82 = 0.0378
-0,0496
0.0308
0.00413
0.0146
-0.032
@ = (11172 =200 1.49
Ag = 1.17E-05
3.72F-05
1,56E-08
5.69E-066
1.06F-05
4,98E-06
Φ4 ( [-1.33 + #269) 00.0489
B4 0.0818
0.938
0.166
0.0231
0.238
-0.0254
6 (#1822 0.809) 0.106
3o 0.0482
0.302
0.554
0.52
0.175
0.55
OG (0. (0.856 + 1252] X
3G = 0.248
0.194
0.159
0.198
0.198
0.232
The main contributions of the paper are as follows:
Table 2: Primitive functions and terminals used to develop the concise representations in FEAT.
1. We propose HEAT--nnwleede Distillation (FEAT-KD), which finds the same model structure as FEAT (i.c.. a weighted linear combination of concisely-representee, sym- bolic features), but instead of discovering these features via genetic programming, these features are derived from piece- wise knowledge distillation of a trained TabNet model.
Parameter
Value
Continuous functions
+. /.2, v. sin, cos, exp, log, exponent, logit, tanh, gauss, relu and, or, not, xor, =, S, 31, Hd, constants
Boolean functions Terminals
2. We demonstrate across large variety of datasets (inclu- sive of the regression datasets used in the original TabNet work), that in comparison to TabNet (a deep learning model), FEAT-KD (a symbolic model) reduces TabNet into concise model which achieves competitive performance.
population of candidate solutions for ÿ. which then undergo selection process, and the selected solutions undergo vari- ations (e.g., crossover, mutation) and the varied solutions compete to survive. This process of selection, variation and survival repeats until terminal condition is met, most fre- quently determined by the maximum number of generations of evolution defined by the user.
3. We demonstrate across large variety of datasets (inclu- sive of the regression datasets used in the original FEAT work), that FEAT-KD consistently outperforms FEAT and learns models with better prediction performance, smaller size and less complex primitive symbols. In addition, we demonstrate that FEAT-KD easily supports multi-target re- gression, in which the shared features contribute to the in- terpretability of the system.
In FEAT, the selection and survival of candidate solutions for ý are done via multi-objectivv selection and survival techniques which considers two key measurements of each candidate solution: 1). mean squared error of ý against the true labels, y. 2). complexity, which is computed by taking weighted summation of user-defined complexity scores to each primitive symbol (La Cava et al., 2019).
2. Related Works
2.1. FEAT (Feature Engineering Automation Tool)
Two other variants of FEAT exist (La Cava et al., 2019). Both aim to increase the disentanglement between the concisely features by adding a third objective.
FEAT (La Cava et al., 2019) is interested in the task of re- gression, where the goal is to build a predictive model j(x): :
Rd + R using N paired examples T = {(x,, }))
x E Rdd, y E RR. Note that x is scaled to zero mean, unit- variance. FEAT uses the fixed form
1. FEAT-Corr, which additionally minimizes the average squared Pearson's correlation among features of Φ:
(xx) B
Corr(o) m(m
1
Σ Ed.ii
cov (0i, 24)) a (0:) (0))
where ø- = [bb. )m)) 1 are m conciselyereprrseented fea- tures with their respective coefficients 3 = [B1,..., .....,,m' IT. Each amncssely-represennee feature, Φ) (x) Rdd + RR
2. FEAT-CN, which additionally minimizes the condition number (CN) of the solutions:
where i E 21. m), is constructed using a fixed pre- defined primitive symbol set, given in Table 2. FEAT dis- covers ğ (more specifically, σ) via genetic programming. The genetic programming approach creates a random initial
(N
Pmax
umin (b)
3

---

FEAT-KD: Learning Concise Representations for Single and Multi- Target Regression via TabNet Knowledge Distillation
where A is the N x m representation matrix and Mmax and Mmiin are the largest and smallest singular values of Tb
that discover equations from raw data, on this dataset. How- ever, the search space of possible equations is large, thus, SR algorithms perform poorly on such a naive approach.
In this work, we keep the model structure in (I), but propose to move away from genetic programming and utilize deep- learning- baatingg-aeddddeo optimization, which is well-known and has been successful for a wide range of learning problems.
In prior SR work, there has been success in distilling a graph neural network for interacting particle systems (Cranmer et al., 2020; Lemos et al., 2023). In those works, trained graph neural networks were separated into smaller pieces which are then approximated by concise equations. The approximation is done by collecting input-output data from these pieces, then utilizing SR algorithms to find equations that best fit these data. In our work, we modify the idea and perform knowledge distillation on pieces of TabNet via an exhaustive search of short, and hence interpretable, equa- tions to obtain knowledge distilled TabNet. To perform the exhaustive search, we utilize DistiISR (Fong & Motani, 2023), an exhaustive search algorithm that demonstrates the best performance for recovering short equations and outperformed state-of-the-art SR algorithms. Note that Dis- tiSRR does not use genetic programming or evolutions, it is a deterministic brute-force search with computational cost of approximately (dd), where is the number of unique terminals and symbols, and is the expression length.
2.2. TabNet
TabNet (Arik & Pfister, 2021) is a deep tabular data learn- ing model that can be used for regression. In regression tasks, TabNet demonstrates state-of-thee art performance (Borisov et al., 2022). TabNet uses sequential attention mechanism that allows the model to focus on different sub- sets of features at each decision 'Step', mimicking decision trees' while benefiting from neural networks' representa- tional power. By utilizing sparse feature masks that guide the network to attend to specific features, TabNet simultane- ously performs feature selection and learning.
In the context of this paper (i.e., knowledge distillation ob- tain model structure as shown in (1)), there are three main reasons for selecting TabNet among other deep learning methods for tabular data:
2.4. Multi-Target Regression
1. TabNet has a similar model structure as (1): (x)
px,)TT, where ip [fI,...,, PP Natepa] 1 are the tracted transformed features with their respective coeffi- cients 9 = (f..... . ÂNZ2 Nategol I", and No is the dimension of the output of the ReLU unit in each 'Step' and Nateps are the number of 'Step' (see an example of 'Step I' and 'Step 2' in Figure 1). If we convert yo into concisely-representt symbolic features, we can obtain the same interpretable structure that FEAT has, which we discuss later.
Multi-target regression (MTR) extends the traditional single- target regression approach by predicting multiple continu- ous outcomes simultaneously, addressing the limitations of independently modeling each target. Unlike running single- target regression multiple times, MTR leverages the shared features and underlying correlations between targets. This is particularly important in complex real-world applications such as finance (Santana et al., 2019; da Silva et al., 2018), and healthcare (Jain et al., 2024), where the relationships between outputs can help develop more accurate models. By using shared features, MTR not only improves predic- tion accuracy but also enhances the interpretability of the entire predictive system, since the size of the model can be reduced by using shared features (see Table 1).
2. TabNet has sparse learnable feature masks, which courages disentanglement between the transformed features discovered in each 'Step' (see Figure 1). The features se- lected by the masks in each 'Stee'' inform us of the main features utilized by the ReLU unit at each 'Step'.
3. TabNet demonstrates the top prediction performance among state-of-the-art deep learning methods for tabular data. (Borisov et al., 2022).
3. Methodology
2.3. Knowledge Distillation
FEAT-KD finds weighted linear combination of concisely- represented, symbolic features that are derived from piece- wise distillation of trained TabNet model. We train TabNet models for regression tasks and perform knowledge distil- lation on pieces of the TabNet model to convert the entire TabNet into the same model structure that FEAT produces: a weighted linear combination of concisely-representee fea- tures. The FEAT-KD algorithm has 5 main phases:
Knowledge distillation refers to model compression tech- niques where smaller student model is trained to replica- tion the behavior of larger, more complex teacher model (Hinton et al., 2015). In this work, we consider specific case of knowledge distillation with the objective of com- pressing pieces of a deep learning model into symbolic concisely-representeed features to form a symbolic model. A naive approach would be to collect onput-output data from the entire trained deep learning model and run symbolic regression (SR) algorithms, which are class of algorithms
Phase 1, Training TabNet: Fit a TabNet regressor to the dataset, that learns a function g' (x) = (((x)" gy: gd + R
using N paired examples T = [xx, 3))),-
4

---

FEAT-KD: Learning Concise Representations for Single and Multi- Martet Regression via TabNet Knowledge Distillation
Phase 2, Extracting TabNet Transformed Features: From the trained TabNet model, extract Na x Nsteps transformed- features-datasets (each of the N Nateps 'Step' in Tabnet pro- duces Na-dimension output). We denote these datasets as T = [(x, (x,))) i= Vj E N₂ X N Nsteps-
Table 3: Primitive functions and terminals used to develop concise representations in FEAT-KD.
Parameter
Value
Continuous functions +. . exponent
Terminals
21, 22 ................................ Ld, constants
Phase 3, Extracting TabNet Masks for Feature Selection: For each 'Step" in TabNet (see Figure 1), extract the mask matrix and select the 3 features that is masked the least on average (across datapoints), i.e., most significant 3 features. Modify J, Vj E Na x Nsteps to only include the top 3 features of the mask that corresponds to the 'Step' that the transformed- featuures were obtained from.
Finally, based on recent criticism and insights on explain- ability and interpretability in SR, we modified the set of primitive functions and terminals to exclude complex sym- bols that reduces explainability (Petersen al., 2019) and restricted the length of the equation to be less than equal to 5 symbols, in-line with the empirically researched limits of the number of cogmitive concepts (Trazzi & Yampolskiy, 2020) and symbols (Matricciani et al., 2019) the mind can retain.
Phase 4, Knowledge Distillation of TabNet 'Step': Train DistilSR (Fong& & Motani, 2023), non-evolutionary-ber SR algorithm that performs exhaustive search of equations, on Ty Vj E X Nateps to generate Na X N Neteps concisely- represented features, o- - (.,,..., INa that here, though the mean squared error (MSE) of the candi- date equation against the transformed- sersforred-featuress is reasonable, we chose to use the invariant MSE (AFI-MSE) in anticipation of Phase where we know the distilled features obtained here will undergo natural affine transformation in the linear regressor. AFI-MSE is a metric designed to measure the discrepancy between two vectors, y and j. after optimally aligning ý with y through an affine transforma- tion. Specifically, AFI-MSE minimizes the traditional mean squared error by determining the best possible linear scaling and translation of ý in the form a + x y. where a and computed using least squares optimization minimize:
Note
Phase 5, Refitting of Knowledge Distilled Features: Per- form simple linear regression on the conciselyrrepresented features obtained from the previous 'Step' and the output, ((((s)) (.)))) ):)) to obtain 33 in j(() = +
Referring to Figure 1, Phase is illustrated by the trained TabNet. Phase 2 is illustrated by extracting the transformed features from the ReI.U units in the dotted boxed, e.g., 'Step I. Phase 3 illustrated by the mask obtained from 'Step I', in which the features æ1, Z2 are selected (least masked) from the mask. This means that in the next phase, the knowledge distillation of 'Step 1' will only be done with Ii, 22 exclusively. Note that two features are selected in the diagram, but in our experiments (as stated above), we select the top three features instead. Phase utilizes DistilSR (minimizing the objective in (2)) to produce the equation, 21 X 22, via exhaustive search of all possible equations with maximum of 5 symbols built from 21.22 and the other primitive symbols in Table 3. Phase 5 combines all the concisely-represented features through linear regression, to produce the model presented by FEAT-KD.
1 N (y:- bx+bx 2))
AFI-MSE(y, 0)
AFI-MSE is especially suitable for our purpose as it evalu- ates the similarity of two vectors, providing a more robust and invariant measure of fit. Consider a simple example with true outputs y = [0, 12, 28] and inputs x₁ = [1,2,3] and Ø2 = [4, 5, 6]. A candidate equation 2 x T2. produces predictions [4, 10, 18), which under traditional MSE yield relatively high value that would lead to the equation not being picked. However, when using AFI-MSE, which first optimally scales and shifts the predictions to best match the true values, we can find parameters and b (specifi- cally, a= = 8, b = 2) such that the adjusted prediction -8 + (I) x x22 exactly equals y, resulting in a zero error. Thus, AFI-MSE recognizes that while the candidate's predictions differ in scale and shifts, their underlying pat- tern aligns perfectly with the true values, highlighting the candidate's 's potential as a good fit despite the initial magni- tude mismatch. This is suitable particularly for FEAT-KD because in Phase 5', this equation will be scaled and shifted anyway because it is being used as a feature in linear regres- sion. This also effectively simplifies the search space.
4. Hy Hyperparameter Details and Tuning
In the experiments, create and distill models which are more interpretable, we chose to distill TabNet models with Na N. 3, which creates small set of N₂ X Nateps = concisely-represenned features. We obtained these values via preliminary experiments on a small subset of the data to prevent data leakage (e.g., target of atpld). The criterion for tuning is based on the test set H² score performance. We also enabled early-stopping, with 1000 max epochs and a patience of 50.
For DistilSR, which was used to convert 'Step' in TabNet into symbolic equations, we considered recent criticism and insights on explainability and interpretability in SR and modified the set of primitive functions (see Table 2)
5

---

FEAT-KD; Learning Concise Representations for Single and Multi- Marttt Regression via TabNet Knowledge Distillation
and terminals to exclude complex symbols that reduces explainability (Petersen et al., 2019). We also restricted the length of the equation to be less than or equal to 5 symbols, in-line with the empirically researched limits of the number of cognitive concepts (Trazzi & Yampolskiy, 2020) and symbols (Matricciani et al., 2019) the mind can retain. Thus, the hyperparameters for DistiISR were to allow for a max length of 5 and primitive symbol set as shown in Table 2.
5. Results and Discussion
To evaluate FEAT-KD, we performed a large variety of ex- periments taking datasets used in (i) the original TabNet paper (Arik & Pfister, 2021), (ii) the original FEAT paper (La Cava et al., 2019), (iii) multi-target regression bench- marks (Spyromitro--ioufis et al., 2016).
5.1. Datasets and Experiment Details
In this work, we further group the datasets into two broad categories: single-target regression and multi-target regression. For single-target regression: from the original TabNet paper (Arik & Pfister, 2021), we used 6 datasets, Synl to Syn6 (Chen et al., 2018), and the Rossmann store sales dataset (Kaggle, 2019). From the original FEAT paper (La Cava al., 2019), we used PMLB datasets (bodyfat, cpu_act pou_act_197, 197, epu_act_ ac3__73,, epu_small, house_8L, houses, pm10, puma8NH) (Romano et al., 2021).
Table 4: Varying Na. keeping Nsteps = 3.
Na
R²
score
2 0.652 (0.074)
3
0.637 (0.053)
0.528 (0.215)
In Phase 1 of FEAT-KD, by varying Na and keeping Nsteps = 3, we observed that increasing Na led to over- fitting, which reduced the R2 score on the test set despite increasing the learning capacity of the model (see Table 4).
For multi-target regression: from the original TabNet paper (Arik & Pfister, 2021), we used the SARCOS dataset (Vi- jayakumar & Schaal, 2000). From aulti-target regression benchmarks (Spyromitros-Xioue) Xioufis et al., 2016), we used benchmark datasets {atpld, enb, oes_10, rf1, scmld].
Table 5: Varying Nsteps: keeping Na- = 2.
All results were averaged across 100 randomly seeded 60-20-20 train-validation-eest splits. We evaluate FEAT- KD (ours), FEAT, FEAT-Corr, FEAT-CN and TabNet, and in addition to the standard deviation, we perform the Wilcoxon signed-rank test (Conover, 1999) with Bonferroni- adjustment account for the increased probability of ob- serving rare events from multiple hypotheses (Dunn, 1961). The simplicity of the symbolic expressions has been ar- gued to be interpretable via disentanglement (La Cava et al., 2019) and also validated by clinicians in studies which uses FEAT structure (i.e., see (1)) (La Cava et al., 2023). In the experiments, to create and distill models which are more interpretable, we chose to distill TabNet models with No = 2, N steps 3, which creates small concisely-representee features. In-
N Nateps
R² score
1
0.619 (0.062)
2
0.647 (0.047)
3 0.652 (0.074)
4 0.509 (0.19)
In Phase 1 of FEAT-KD, by varying Nsteps and keeping Na 2, we observed that increasing N Nateps led over- fitting well, which reduced the R² score the test set despite increasing the learning capacity of the model (see Table 5). However, decreasing Natepe created models that are too simple and that may not be able to capture the com- plexity of the dataset as well as models with higher Nateps-
set of Na x Nsteps terpretability is measured in proxy by model size, which is supported by works such as those by Lage et al. (2019); Abdul et al. (2020). Specifically, we define the model size to be total number of functions and terminals. For example, 2i x 22 has a model size of 3, 21/4 + 0.62 has a model size of
Table 6: Varying both No. N Nateps, while Na x Nateps 6.
N steps
Na
R2
score
2
3
0.634 (0.062)
3
2
0.652 (0.074)
Additional Details in Appendix. More information on datasets, evaluation metric, computational resources and timing are provided in the Appendix.
6
1
0.623 (0.079)
Since we decided on having total of 6 concisely- represented, symbolic features, we also experimented vary- ing both N₂, Nstepss keeping Na x Noteps = 6. Ultimately, Na = 2, Nateps = still had the best empirical perfor- mance on the test set (see Table 6).
5.2. Single- Tagge-Tarrett Regression
FEAT-KD has the best average prediction performance among symbolic model evaluated. We tabulate the results
6

---

FEAT-KD: Learning Concise Representations for Single and Multi- Target Regression via TabNet Knowledge Distillation
Table 7:
Single-Target Regression: 1t² score (Standard Deviation [SD] in brackets) of FEAT-KD, FEAT variants and TabNet, averaged across 100 randomly seeded 60-20-20 train- sein--alidatioo-tess splits on 6 synthetic datasets from Chen et al. (2018) and the Rossmann store sales dataset (Kaggle, 2019) used in the original TabNet paper (Arik & Pfister, 2021), and PMLB datasets (Romano et al., 2021) used by La Cava et al. (2019). Higher is better, best white-box performance in bold.
White-box
Black-box
Dataset
FEAT-KD (Ours)
FEAT
FEAT-Com
FEAT-CN
TabNet
Synl
0.941 (0.012)
0.937 (0.0031)
0.935 (0.0042)
0.936 (0.0036)
0.993 (0.0014)
Syn2
0.953 (0.0037)
0.936 (0.0042)
0.934 (0.0038)
0.935 (0.0041)
0.996 (0.0017)
Syn3
0.949 (0.0063)
0.938 (0.0037)
0.938 (0.0037)
0.937 (0.0039)
0.991 (0.0018)
Syn4
0.939 (0.0089)
0.93 (0.0045)
0.928 (0.0048)
0.929 (0.0042)
0.987 (0.0034)
Syn5
0.951 (0.0036)
0.933 (0.0042)
0.932 (0.0041)
0.932 (0.0043)
0.991 (0.0027)
Syn6
0.943 (0.0095)
0.931 (0.0032)
0.93 (0.0031)
0.929 (0.0029)
0.991 (0.0021)
Rossmann Store Sales
0.68 (0.0096)
0.679 (0.0096)
0.681 (0.01)
0.679 (0.009)
0.685 (0.0096)
bodyfat
0.987 (0.0058)
0.973 (0.023)
0.965 (0.0032)
0.974 (0.024)
0.968 (0.014)
cpu act 197
0.976 (0.0014)
0.963 (0.004)
0.962 (0.0039)
0.961 (0.0038)
0.982 (0.0013)
cpu_act_573
0.976 (0.0014)
0.963 (0.0041)
0.955 (0.0032)
0.961 (0.0034)
0.982 (0.0013)
cpu-small
0.968 (0.0015)
0.956 (0.0032)
0.955 (0.0032)
0.955 (0.0028)
0.973 (0.0014)
Mo
0.585 (0.01)
0.492 (0.019)
0.463 (0.037)
0.465 (0.037)
0.659 (0.01)
houses
0.664 (0.0078)
0.579 (0.015)
0.566 (0.012)
0.567 (0.013)
0.793 (0.0047)
pm10
0.181 (0.026)
0.215 (0.039)
0.226 (0.034)
0.221 (0.039)
0.135 (0.043)
puma8NH
0.618 (0.026)
0.628 (0.038)
0.56 (0.046)
0.585 (0.042)
0.684 (0.0064)
Table 8: Single-Targee Regression:
Bonferroni-adjusteed p-values using a Wilcoxon signed-rank test, with the one- sided alternative hypothesis that the distribution of R2 score outperformance of FEAT-KD over the 3 FEAT variants is stochastically greater than a distribution symmetric about zero. * indicates FEAT-KD outperformance is statistically significant (i.e.,) p < 0.05).
given in (1)), which are FEAT-KD, FEAT, FEAT-Cor and FEAT-CN, the algorithm which shows the best average R² score is FEAT-KD, with small, non-overlapping standard deviation on some datasets. Even the few datasets which FEAT-KD does not top, the performance is within 3% of the best average R2 score, with pm10 being the only exception.
FEAT-KD has consistent statistically significant outper- formance over the other symbolic models evaluated. Be- sides evaluating the average performance, we also compute and tabulate (see Table 8) the Bonferroni-adjustee p-values from the one-sided Wilcoxon signed-rank test. 38 out of the 45 comparisons are significant, with FEAT-KD produc- ing statistically significant outperformance over all 3 FEAT variants for 11 of the 15 datasets.
FEAT-KD
FEAT-KD FEAT-CCor
FEAT-KD FEAT-CN
Dataset
FEAT
Synl
181c-01
4.20c-03*
3.63e-02*
Syn2
2.66e-15*
2.66e-15*
2.66e-15*
Syn3
6.5
1.15e-11*
3.9
Syn4
3.37c-08*
7.98c-10"
2.64c-09*
Syn5
2.66c-15*
15"
2.66c-15*
Syn6
2.90c-09*
1.05c-09*
3.03e-10*
Why does FEAT-KD outperform FEAT in prediction? The performance of FEAT-KD is close to that of TabNet (see Table 7), tracking the good performance of TabNet. Though numerical parameters in symbolic models can be optimized via gradient based methods, the symbols themselves are difficult to optimize. Genetic programming for symbolic models, which FEAT uses, has no theoretical guarantees to date (the proof for NP-hardness for a simplified SR prob- lem was only done recently (Virgolin & Pissis, 2022)). and has been shown recently to be inefficient in terms of ne- glecting large search spaces of short, interpretable equations (Kronberger et al., 2024; Biggio et al., 2021). FEAT-KD addresses this function search by using TabNet, which has deep-learning-basee based optimization with a disentanglement mechanism (i.e., sparse learnable masks) to reduce the prob-
Rossmann
1.33e-05*
1.00c+00
1.99c-01
bodyfat
2.97e-03*
2.81e-05*
1.54e-02*
epu_act_197
2.66c-15*
2.66c-15*
2.66c-15*
cpu,act act 573
16666-15 15*
2.66c-15*
2.66e-15*
cpu small
2.66e- 15
1.5
2.66c-15*
house 8L
2.66e-
2.66e-15*
2.66e-15*
houses
2.66e-15*
2.66e-15*
2.66e-15*
pm10
1.00e+00
1.00c+00
1.00e+00
puma8NH
1.00e+00
9.51c-11*
3.35e-06*
of the various methods in terms of R² score on the single- target regression datasets in Table 7. Among the algorithms which produce symbolic model (i.e., models with the form
7

---

FEAT-KD: Learning Concise Representations for Single and Multi- Target Regression via TabNet Knowledge Distillation
Table 9: Single- Target Regression: Model Size (SD in brackets) of the models shown in Table 7. Lower is better, Also tabulated are the Bonferroni-adjusted p-values using a Wilcoxon signed-rank test, with the one-sided alternative hypothesis that the distribution of model size difference between FEAT--K and the 3 FEAT variants stochastically smaller than a distribution symmetric about zero. indicates FEAT-KD's smaller size statistically significant (i.e., p < 0.05).
Model Size
cnnterroni-adjustee p-values
FEAT-KD (Ours)
FEAT-KD FEAT
FEAT-KD FEAT-Corr
FEAT-KD
Dataset
FEAT
FEAT-Corr
FEAT-CN
TabNet
FEAT-CN
Synl
49.0 (0.0)
81.8 (32)
78.7 (32)
84.1 (36)
5.7k
1.47e-14*
1.25e-13#
6.67e-15*
Syn2
48.9(0.62). (0.62)
81.1 (30)
77.6 (30)
76,9 (28)
5.8k
3,56c-154
3.13e-13*
1.25c-13*
Syn3
49 (0.28)
86.4 (35)
78.3 (34)
77.8 (33)
6.0k
2.00c-14*
8.31c-12*
3.59c-124
Syn4
49 (0.00)
80.3 (29)
78.9 (24)
84.2 (31)
6.1k
4.04c-15*
8.23c-16*
2.03c-15*
Syn5
49 (0.28)
84 (37)
84.7 (33)
77.5 (30)
6.2k
14the114
2.99e-15*
1.43e-33*
Syn6
48.9 (0.56)
79.5 (33)
75 (30)
81.8 (33)
6.3k
6.64e- 13*
7.
7.69e-14*
Rossmann
49 (0.00)
57.3 (29)
62.3 (23)
64.7 (28)
5.5k
2.49e-01
5.41e-06*
3.12e-06*
bodyfat
49 (0.00)
49.8 (32)
64.1 (43)
61.2 (28)
6.4k
1.00++00
1.30c-02*
7.99c-04*
cpu act 197
49 (0.28)
59.9 (29)
70.5 (33)
65.3 (30)
7.2k
1.17e-02*
5.77e-08*
1.02-055*
cpu_act_573
48.9 (0.47)
62.1 (30)
72.5 (30)
63.1 (29)
7.2k
1.46e-03*
2.03e-100 10
5.69e-05*
cpu-small
49 (0.00)
65.3 (29)
68.6 (28)
69.3 (28)
6.2k
7.46e-06*
9.38c-09*
4.1
house_8I.
(0.00)
82.2 (34)
(40)
69.7 (41)
5.7k
4.47e-13*
2.69c-06*
3.24e-05*
houses
48.9 (0.39)
40.6 (24)
41.4 (27)
42.6 (27)
5.7k
1.00e+00
I. 1.00e+00
1.00c+00
pm10
48.9 (0.47)
64.4 (33)
83.2 (45)
70.4 (29)
5.6k
4.94c-05*
3.17e-10"
7.44e-10*
puma8NH
49 (0.00)
(31)
104 (31)
103 (34)
5.7k
2.05c-17*
6,98e-18*
7.10c-18*
lems into smaller equation discovery subtasks that can be tackled via exhaustive search. These knowledge distillation subtasks are easy to fit, with the R²2 score being 0.707. 0.795 and 0.663 for synthetic, Rossmann and PMLB datasets re- spectively.
Table 10: Multi-Target Regression: R² score (SD in brack- ets) of FEAT and TabNet the SARCOS dataset (Vijayaku- mar & Schaal, 2000) used in TabNet paper, averaged across 100 randomly seeded 60-20-20 train- siii--alidaiiin-test splits.
White-box
Black-box
FEAT-KD has the lowest average model size among all models evaluated and has statistically significant evi- dence that it is consistently smaller. We measure the model size (i.e., total number of functions and terminals in the model when represented in the form of (I)) and tabulate these Table 9. FEAT-KD is smaller, on average, com- pared to all other models evaluated, on all but one dataset. Although FEAT, FEAT-Com and FEAT-CN are unstable and produce models of largely varying sizes across the 100 random seeds, the statistical test reveals that FEAT-KD's smaller size is statistically significant, as seen by the right- most columns Table 9. 40 out of the 45 comparisons are significant, with FEAT-KD producing statistically sig- nificant smaller models over all other models for 12 of the 15 datasets. Note that TabNet is always of greater size than all symbolic models across all datasets and random seeds.
FEAT,
FEAT-KD
w-th target
FEAT-Com,, FEAT-CN
TabNet
(Ours)
n2
0.577 (0.061)
N.A.
0.449 (0.014)
n 2
0.612 (0.04)
N.A.
0.788 (0.0068)
n
0.675 (0.042)
N.A.
0.883 (0.0045)
n =
0.887 (0.049)
N.A.
0.898 (0.0029)
n 5
0.295 (0.11)
N.A.
0.279 (0.01)
0.264 (0.02)
N.A.
0.369 (0.014)
n
0.862 (0.053)
N.A.
0.906 (0.0022)
form given in (1) for interpretability reasons, like FEAT, also be positioned in the broader literature of SR al- gorithms. We evaluate FEAT-KD on all 88 datasets used by La Cava et al. (2019) against FEAT and other SR algo- rithms, and FEAT-KD is Pareto-optimal for 62.0% of them with respect to the other SR algorithms. More discussion and results are included in Appendix F.
Additionally, it should be noted that FEAT-KD uses a re- duced set of more interpretable primitive functions (see Table 3) (Petersen et al., 2019), compared to Table 2.
FEAT-KD frequently Pareto-optimal with respect to FEAT and other SR algorithms in SRBench single-target regression datasets. Although FEAT-KD uses the specific
5.3. Multi-Target Regression
FEAT-KD easily supports multi-target regression. Tab- Net supports multi-target regression by modifying the final
8

---

FEAT-KD: Learning Concise Representations for Single and Multi-Targel Regression via TabNet Knowledge Distillation
Table 11: Multi-Target Regression: R² score (SD in brack- ets) of FEAT and TabNet on other multi-target regression datasets (Spyromitros-Xinn -iiouus et al., 2016), averaged across 100 randomly seeded 60-20-20 seiin-vlidation-teer splits. FEAT-KD performs competitively with TabNet on average. Among total of 54 targets (including SARCOS from Table 10), FEAT-KD performs better on 32 of the targets.
layer accommodate multiple-dimension outputs. Since the architecture before the final layer remains the same, FEAT-KD's s 5 phases can still be used on TabNet that is trained for multi-targes -tagett regression dataset in Phase 1. This allows FEAT-KD to distill multi-target regression TabNet models. See Table for an example on the model found by FEAT-KD. Note that multi-target regression not sup- ported by FEAT, FEAT-Cor and FEAT-CN, as shown in Table 10. Though it may be possible to extend the origi- nal FEAT to multi-target, that is novel separate work that requires proposing new algorithm, beyond the scope of this paper. As a simple baseline, we ran the original FEAT separately and independently for each target, but the discov- ered models were much less interpretable because they do not share common transformed features across targets due to the independence during training. Additionally, genetic programming, which the original FEAT uses, does not scale as well with dimensions (Kronberger et al., 2024; Biggio et al., 2021).
FEAT-KD
Dataset
72-th target
TabNet
(Ours)
atpld
n
0.652 (0.074)
0.572 (0.045)
N
2
0.678 (0.064)
0,537 (0.064)
3
0.679 (0.041)
0.526 (0.048)
4
0.671 (0.046)
0.501 (0.038)
0.641 (0.067)
0.57 (0.054)
7 6
0.646 (0.05)
0.474 (0.039)
enb
71
0.85 (0.016)
0.434 (0.023)
72 = 2
0.85 (0.014)
0.435 (0.017)
10
71
0.176 (0.22)
0.401 (0.13)
1 2
0.498 (0.033)
0.609 (0.066)
3
0,378 (0.29)
0,479 (0.094)
FEAT-KD has some regularization effect over TabNet. We tabulate the results for the other multi-target regression datasets in Table 11. FEAT-KD demonstrates competitive performance to TabNet, performing better on average in 32 out of 54 targets. DistilSR is able to learn equation for the transformed features in TabNet (Phase 4), with the average R2 score for the knowledge distillation being 0.535, 0.838, 0.691, 0.938, 0.761, 0.578 for SARCOS, atpld, enb, oes 10, If1, scm ld datasets respectively. Unlike the single-target regression, where TabNet tends to predict better than any symbolic model, FEAT-KD outperforms TabNet in predic- tions for 32 (out of 52) targets in the multi-target regres- sion case. On inspection of the training R² score, TabNet over-fitted to certain targets and did not generalize well to perform well on the test set, whereas FEAT-KD, a simple model, has training score closer to its test score in Table 11.
4
0.678 (0.033)
0.628 (0.064)
n1 5
0.259 (0.21)
0.386 (0.11)
7 = 6
0.731 (0.37)
0.622 (0.086)
N 7
0,356 (0.3)
0.511 (0.082)
8
0,505 (0.29)
0.508 (0.1)
rfl
1
0,853 (0.037)
0,873 (0.0021)
2
0.178 (0.071)
0.051 (0.015)
3
0.890 (0.034)
0.91 (0.0023)
11
0.829 (0.035)
0.716 (0.029)
11 = 5
0.820 (0.041)
0.779 (0.034)
6
0.540 (0.065)
0,445 (0.013)
7
0.746 (0.11)
0.679 (0.14)
71 8
0.731 (0.087)
0.45 (0.02)
71 = 9
0.816 (0.027)
0.884 (0.0033)
7 = 10
0.172 (0.057)
0.036 (0.017)
11
0.795 (0.021)
0.888 (0.0032)
12
0.693 (0.039)
0.643 (0.018)
11 13
0.788 (0.043)
0.795 (0.04)
n = 14
0.494 (0.068)
0.427 (0.018)
=
15
0.770 (0.096)
0.683 (0.13)
6. Conclusion
16
0.688 (0.078)
0.494 (0.026)
semld
1
0.882 (0.0033)
0.702 (0.011) 1)
In this work, we propose FEAT-KD, an algorithm which finds weighted linear combination of symbolic, concisely- represented features. FEAT-KD converts TabNet from a 'black-box' model to an intrinsically explainable 'white-box' model, with competitive prediction performance. Compared to FEAT (including FEAT-Corr, FEAT-CN), FEAT-KD uses completely different model discovery approach: knowl- edge distillation of pieces of TabNet via exhaustive search SR instead of genetic programming. FEAT-KD demon- strates improvements over FEAT in terms of: (i) prediction performance, (ii) model size, (iii) complexity of primitive symbols, (iv) support for multi-targer regression.
2
0.874 (0.0042)
0.701 (0.015)
7 3
0.849 (0.0057)
0.718 (0.0072)
71 = 4
0.846 (0.0032)
0.718 (0.0053)
7 5
0.405 (0.019)
0.634 (0.013)
6
0.386 (0.016)
0.612 (0.013)
7
0.426 (0.027)
0.644 (0.022)
0.792 (0.0047)
0.632 (0.021)
N = 9
0.602 (0.045)
0.653 (0.01)
n = 10
0.592 (0.038)
0.654 (0.0094)
11
0.857 (0.003)
0.668 (0.0075)
n 12
0.833 (0.0042)
0.685 (0.0088)
n1 13
0.646 (0.049)
0.666 (0.0051)
n= = 14
0.636 (0.032)
0.674 (0.0058)
= 15
0.706 (0.05)
0.666 (0.0062)
16
0.697 (0.036)
0.666 (0.0064)
9

---

FEAT-KD; Learning Concise Representations for Single and Multi-Targer Regression via TabNet Knowledge Distillation
Acknowledgements
Chen, J., Song, L., Wainwright, M., and Jordan, M. Learn- ing explain: An information-theoreti perspective model interpretation. In International Conference Machine Learning, pp. 883-892. PMLR, 2018.
This research/project is supported by the National Re- search Foundation, Singapore under its AI Singapore Pro- gramme (AISG Award No: JISG3-PDD-2023-0--0527
and A*STAR, CISCO Systems (USA) Pte. Ltd and Na- tional University of Singapore under its Cisco-NUS Ac- celerated Digital Economy Corporate Laboratory (Award
Conover, W. J. Practical nonparametric statistics, volume 350. John Wiley & Sons, 1999.
Cranmer, M. Interpretable machine learning for science with PySR and SymbolicRegression jl. arXiv preprint 23X55:230..05582, 2023.
121001B0002).
Impact Statement
Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., and Ho, S. Discovering sym- bolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, 33: 17429-17443 2020.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
da Silva, A. R., Santana, E. J., Mastelini, S. M.. and Barbon Jr, S. Stock portfolio prediction by multi- target
References
Abdul, A., von der Weth, C., Kankanhalli, M., and Lim,
decision support. In Proceedings of the XIV Bruzilian Symposium on Information Systems, pp. 1 1-8, 2018.
B. Y. COGAM: Measuring and moderating cognitive load in machine learning model explanations. In Proceed- ings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2020.
Di Martino, E. and Delmastro, F. Explainable Al for clinical and remote health applications: A survey tabular and time series data. Artificial Intelligence Review, 56(6); 5261-5315, 2023.
Arik, S. O. and Pfister, T. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 6679-6687, 2021.
Dua, D. and Graff, C. UCI machine learning repository. School of Information and Computer Science, 25:27, 2019.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEFE Transac- tions on Puttern Analysis and Machine Intelligence, 35 (8):1798-1822 1828, 2013.
Dunn, J. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52-64 1961.
Fong, K. S. and Motani, M. DistilSR: A distilled version of gene expression programming symbolic regression. In Proceedings of the Companion Conference Genetic and Evolutionary Computation, pp. 567-570, 2023.
Biggio, I.,, Bendinelli, T., Neitz, A., Lucchi, A., and Paras- candolo, G. Neural symbolic regression that scales. In International Conference on Machine Learning, pp. 936- 945. PMLR, 2021.
Habchh, H. and Gohel, S. Machine learning in healtheare. Current Genomics, 22(4):291, 2021. 2021.
Bordt, S., Finck, M., Raidl, E., and von Luxburg, U. Post- hoc explanations fail to achieve their purpose in adversar- ial contexts. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 891- 905, 2022.
Hinton, G. How to represent part- whrle hierarchies in a neural network. Neural Computation, 35(3):413 4522, 2023.
Hinton, G., Vinyals, O., and Dean, Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503 S.Ci::5533..53,,, 2015.
Borisov, V., Leemann, T., Seßler, K., Haug. J., Pawelezyk, M., and Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Networks and Learning Systems, 2022.
Jain, K., Kaur, S., and Rani, G. Applications of multitarget regression models in healthcare. Machine Learning in Healthcare and Security, pp. 103-118 118. CRC Press, 2024.
Burlacu, B., Kronberger, G., and Kommenda, M. Operon C++: An efficient genetic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1562-1570, 2020.
Kaggle. le. Rossmann store sales, 2019. htths:::/// //t::////www.. /www. kaggle, com/ c/ //rroossaann satttrrreeees...... Ac- cessed: 2019-11-10.
10

---

FEAT-KD: Learning Concise Representations for Single and Multi- Target Regression via TabNet Knowledge Distillation
Keys, A., Fidanza, F., Karvonen, M. J., Kimura, and Tay- lor, H. L.. Indices of relative weight and obesity. Journal of Chronic Diseases, 25(6-7):329-344 1972.
Si, Cheng, W. Y., Cooper, M., and Krishnan, R. G. Inter- pretabnet: Distilling predictive signals from tabular data by salient feature interpretation. In Proceedings of the 41 International Conference on Machine Learning, PP- 2024.
Kronberger, G., Olivetti de Franca, F., Desmond, H., Bartlett, D. J., and Kammerer, L. The inefficiency of genetic programming for symbolic regression. In Inter- national Conference on Parallel Problem Solving from Nature, pp. 273-2892 289.. Springer, 2024.
Spyromitro-Xiouffs. E., Tsoumakas, G., Groves, W., and Vlahavas, I. Multi-target regression via input space ex- pansion: Treating targets as inputs. Machine Learning, 104:55-98, 2016.
La Cava, W., Singh, T. R., Taggart, J., Suri, S., and Moore, H. Learning concise representations for regression by evolving networks of trees. The International Conference on Learning Representations, 2019.
Trazzi, M. and Yampolskiy, R. V. Artificial stupidity: Data we need to make machines equals. Patterns, 1(2), 2020.
La Cava, W. G., Lee, P. C., Ajmal, I., Ding, X.., Solanki, P., Cohen, J. B., Moore. J. H., and Herman, D. S. A flexible symbolic regression method for constructing interpretable clinical prediction models. NPJ Digital Medicine, 6(1): 107, 2023.
Vijayakumar, S. and Schaal, Locally weighted projection regression: An O(n) algorithm for incremental real time learning in high dimensional space. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), volume 1, pp. 288-293, Morgan Kaufmann Burlington, MA, USA, 2000.
Lage. I., Chen, E., He. Narayanan, M., Kim, B., Ger- shman, S., and Doshi-Velez, An evaluation of the human-interpretabilityy of explanation. arXiv preprint arXiv:1922.00006, 2019.
Virgolin, M. and Pissis, S. P. Symbolic regression is NP- hard. Transactions Machine Leurning Research, 2022.
Lemos, P., Jeffrey, N., Cranmer, M., Ho. S., and Battaglia, Rediscovering orbital mechanies with machine learning. Machine Learning: Science and Technology, 4(4):045002, 2023.
Matricciani, E. et al. Deep language statistics of Italian throughout seven centuries of literature and empirical connections with Miller's 7+2 law and short-term mem- ory. Open Journal of Statistics, 9:373 2019.
Petersen, B. K., Larma, M. L., Mundhenk, T. N., Santi- ago, C. P., Kim, S. K., and Kim, J. T. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. The International Conference on Learning Representations, 2019.
Romano, J. D.. Le, T. T., La Cava, W., Gregg, T., Gold- berg, J., Chakraborty, P., Ray, N. L., Himmelstein. D., Fu, W., and Moore, J. H. PMLB v1.0: An open source dataset collection for benchmarking machine learning methods. arXiv preprint arXiv:2012.000588... 2021.
Santana, E. J., Mastelini, S. M., Barbon Jr, S., et al. Stock portfolio prediction by multi multi-target decision support. iSys- Brazilian Journal of Information Systems, 2019.
Scholl, P., Bieker, K., Hauger, H., and Kutyniok, G. ParFam- (neural guided) symbolic regression via continuous global optimization. In The Thirteenth International Conference on Learning Representations, 2025.

---

FEAT-KD; Learning Concise Representations for Single and Multi-Targe Regression via TabNet Knowledge Distillation
A. Dataset Details and Sources
KD over the 3 FEAT variants is stochastically greater than distribution symmetric about zero.
All datasets used are open-sourced and publicly available. We performed a large variety of experiments taking datasets used in (i) the original TabNet paper (Arik & Pfister, 2021), (ii) the original FEAT paper (La Cava et al., 2019), (iii) multi-target regression benchmarks (Spyromitro--iouuis et al., 2016).
For model size, we use the one-sided alternative hypothesis that the distribution of model size difference between FEAT- KD and the 3 FEAT variants is stochastically smaller than a distribution symmetric about zero.
C. Computational Resources and Timing
In this work, we can further group the datasets into two broad categories: single-target regression and multi-tare -taree regression. For single-target regression: from the original TabNet paper (Arik& & Pfister, 2021), we used datasets, Synl to Syn6 (Chen et al., 2018), and the Rossmann store sales dataset (Kaggle, 2019). From the original FEAT paper (La Cava et al., 2019), we used 8 PMLB datasets [bodyfat, cpu act_197, cpu 573, cpu small, house 8L, houses, pm10, puma8NH] (Romano et al., 2021).
The experiments were ran on Intel(R) Xeon(R) CPU E5- 2627 v4@2.30GHz with 128GB RAM, on 100 random seeds per experiment using a 60-20-20 train-validattion-test split. For all algorithms, each seeded run is given a max walltime of 3600 seconds.
D. InterpreTabNet Variant of FEAT-KD
For target regression: from the original TabNet paper (Arik& & Pfister, 2021), we used the SARCOS dataset (Vi- jayakumar& & Schaal, 2000). From multi-target regression benchmarks (Spyromitros-Xioute et al., 2016), we used 5 benchmark datasets (atpld, enb, oes_10, rf1, semld).
Table 12: Single-Targe Regression: R score (SD in brackets) of FEAT-KD KD distilling TabNet and InterpreTabNee Higher is better.
FEAT-KD (TabNet)
FEAT-KD
All results were averaged across 100 randomly seeded 60- 20-20 satn--alidatioo-tee splits.
Dataset
(InterpreTaabNe)
bodyfat
0.987 (0.0058)
0.990 (0.0051)
We also perform comparison with SRBench in which the dataset names and random seeds are provided in Appendix F. To be consistent with SRBench, the results are evaluated on 25% test set splits on the exact same random seeds.
cpu_act_197
0.976 (0.0014)
0.977 (0.0015)
553
0.976 (0.0014)
0.976 (0.0016)
cpu_small
0.968 (0.0015)
0,966 (0.0015)
house_ 88L
0.585 (0.01)
0.588 (0.0096)
houses
0.664 (0.0078)
0.660 (0.0053)
pm10
0.181 (0.026)
0.196 (0.042)
B. Additional Evaluation Metric Details
puma8NH
0.618 (0.026)
0.596 (0.0086)
We measure the metric, R²² score, also known as the coef- ficient of determination, which measures the proportion of the variance in the dependent variable that is predictable from the independent variables. It is defined as:
InterpreTabbNet is modification of TabNet which has the same structure as TabNet (Si et al., 2024). InterpreTab- Net has 2 main contributions, both of which are relevant to FEAT-KD: i). A regularization scheme that maximizes diversity between masks in the TabNet architecture, ii). cap- turing feature interdependencies by prompting LLMs with the learned masks.
yy - 312
y - 1
where y is the vector of true labels, y is vector of the ensemble-combined predictions and ÿ is the mean of the true labels. ||. [/ represents the Euclidean norm (or 2-norm).
For i)., Interpre' TabNee works very similarly to TabNet, just with extra regularization, so we could easily apply FEAT- KD techniques by replacing TabNet with InterpreTabbee in the code implementation, as done in Table 12. The results yield stronger performance on some datasets, but not on all, consistent with the results obtained by Si et al. (2024). Thus, we can say InterpreTabNet allows for variant of FEAT-KD that performs competitively.
We measure the metric, model size, which is total number of functions and terminals. For example, æ1 x 22 has a model size of 3, x1/ /4 0.62 has a model size of
For statistical comparison, we performed the Wilcoxon signed-rank test with Bonferroni-adjustmentt to account for the increased probability of observing rare events from mul- tiple hypotheses.
For ii). Interpre TabNet uses LLMs to generate linguistic interpretations of the masks obtained which is not mutually exclusive with FEAT-KD. In fact, InterpreTabbNet and FEAT- KD complement each other, in which InterpreTTabNe gener- ates simplified qualitative description, whereas FEAT-KD
For R² score, we use the one-sided alternative hypothesis that the distribution of R² score outperformance of FEAT-
12

---

FEAT-KD: Learning Concise Representations for Single and Multi-Targel Regression via TabNet Knowledge Distillation
Table 13: Classification: Accuracy (SD in brackets) of cFEAT-KD, cFEAT variants, TabNet and InterpreTabNNe averaged across 100 randomly seeded 60-20-20 sruin-valiattiontes splits on 8 PMLB datasets (Romano et al., 2021) and 3 UCI datasets (Dua & Graff, 2019). Higher is better, best white-box performance in bold.
White-box
Black-box
cFEAT-KD (TabNet)
cFEAT-KD (InterpreTaabNe)
Dataset
<FEAT
cFEAT-Cor
<FEAT-CN
TabNet
InterpreTabNee
chess
0.969 (0.0033)
0.969 (0.0034)
0.949 (0.0055)
0.943 (0.0042)
0.945 (0.0036)
0.985 (0.00086)
0.987 (0.00086)
hypothyroid
0.963 (0.0024)
0.963 (0.0024)
0,951 (0.0013)
0.949 (0.0029)
0.952 (0.0025)
0,956 (0.0018)
0.958 (0.0018)
ionosphere
0.914 (0.014)
0.909 (0.011)
0.885 (0.011)
0.868 (0.016)
0.876 (0.012)
0.873 (0.0061)
0.871 (0.0057)
kr_vs.kp
0,964 (0.0019)
0.966 (0.0027)
0,950 (0.0062)
0.945 (0.0068)
0.945 (0.0035)
0,983 (0,0022)
0.986 (0.0028)
sonar
0,815 (0.025)
0.806 (0.035)
0.705 (0.012)
0.702 (0.030)
0.746 (0.017)
0.635 (0.019)
0.682 (0.036)
spamhase
0,934 (0.0024)
0.935 (0.0024)
0.898 (0.0091)
0.895 (0.0039)
0,901 (0.0065)
0.934 (0.0016)
0.934 (0.0019)
spectf'
0.843 (0.018)
0.845 (0.019)
0.812 (0.021)
0.776 (0.011)
0.793 (0.017)
0.836 (0.022)
0.812 (0.025)
tokyol
0.918 (0.0030)
0.920 (0.0073)
0.907 (0.012)
0.906 (0.0033)
0.906 (0.0044)
0.908 (0.010)
0,914 (0.0052)
Diabetes
0.765 (0.0099)
0.770 (0.014)
0.738 (0.0079)
0.733 (0.0073)
0.737 (0.015)
0.775 (0.015)
0.777 (0.012)
Forest Cover Type
0.755 (0.0017)
0.753 (0.0017)
0.748 (0.0021)
0.748 (0.0012)
0.749 (0.0010)
0.753 (0.0022)
0.754 (0.0022)
Poker Hand
0.621 (0.0034)
0.621 (0.0026)
0.614 (0.0016)
0.613 (0.0021)
0.613 (0.0032)
0,618 (0.0019)
0.620 (0.0030)
generates a simplified quantitative description. Thus, from the trained mask, 2 interpretations can be generated, one from using an LLM to generate linguistic interpretation and the other from the learned equation from TE. Thus, the strengths of InterpreTabbee can be subsumed into FEAT-KD by replacing TabNet with Interpre TabNet.
SRBench, across the 88 datasets in PMLB used in FEAT, i.e. (ESL, SWD, LEV, ERA, USCrime, FacultySalaries, vineyard, auto price, cpu act 197, autoPrice, cloud, puma8NH, cpu-small, elusage, pwLinear, machine_cpu, satellite image, analcatdata vehicle, wind, vinnie, pm10, aaloaatdata_eeavote mmalaatdata_electionnn)
pollen, pollution, no2, analcatdata apnea2, analcatdata apneal,
E. Extension to Classification
bodyfat, cpu, cpu_small 5522, cpu_act_573, fri_c0.255555 250.5. fri.c3 1...2_500.25,,, fri.c11500.255 fri..c_500025,, fri. fri_c2_1000_22 fri.c1_1000.10 10, fri.c..1000.,, fri.c..000.00000.1... fri.c2.250.55
One possible way FEAT-KD can be adapted for classifica- tion is by using logistic regression in Phase 5 instead of lin- ear regression. Rather than using j(x) = d(x)² B, as shown in (1) for regression, we use a logistic regression model for classification. That is, for an input x and K classes, the model is defined as Pyy = le x) = exp(o(x)" 3x)/2, where = = Σ exp(o(x)² B,), ((x) is the feature vector, and 3x is the coefficient vector for class k. The predicted class is given by
fri_e1_100_10,, fri_c4_1000_255
fri c2 500 5, fri c2 1000.5, cl 250 5, fri ci.500000..11. 250.100 10,
fri_c0 250_502 fri_c4_1000,50, frt._i_3..100010,ffiiii fri_c1_1000_5, fri_c3.250.5. fricc..50.... fri.c4_500_50,
fri c3 500 f, c3 1000 50, fri c0 100 10, fri c2 1000 50,
fri.c4_1000_10, 5,., fri.33 fri.cl fri.co.50....00..... fri_c0_250_10, fri_e1_500005 f50_c1_000_10, fri_c2.500_25.
fri.c2.500.10.
fri.c2.100.10,
fri 250 25, fri 55.5500050, cri..._500_00 10, 2500_00,
fri_cl 250.50, rrl_c05500.3 fri 500000000 fri_c0_100 10._00025,
j = arg max &
Ply= = le lx).
fri_c0.550.25 tri_c05500_00 wtw.c11110000,0...0 fri.c2.250_100 and 10 random seeds, i.e., (11284, 11964, 15795, 21575, 22118, 23654, 29802, 5390, 6265, 860), FEAT-KD is Pareto-optimal for 62.0% of them with respect the other SR algorithms. The rate at which the other SR algorithms DSR, GP-GOMEA, Operon, gplearn, AFP, AFP_FE, AlFeynman, FEAT, EPLEX, ITEA, SBP-GP, BSR, MRGP, FFX, are 83.6%, 75.9%, 52.6%, 48.6%. 25.5%, 25.8% 8.3%, 23.0% 23. 23.2%, 4.9%, 8.8% 15.2%, 1.5%, 2.7%, respectively. In SRBench, Pareto-optimal means that FEAT-KD has the optimal trade-off with prediction performance and equation size, in which there are no other SR algorithms with smaller equation size with better prediction performance. Of note are only 2
We denote the methods with the prefix 'c' to differentiate them from the regression case. To evaluate the classification performance, we replaced R² score with both accuracy and Fl score instead. The results on accuracy and F1 score are presented in Tables 13 and 14 respectively.
F. Evaluating FEAT-KD against SR Algorithms
Although FEAT-KD uses the specific form given in (1) for interpretability reasons, it can also be counted as a type of SR algorithm and be positioned in the broader literature of SR algorithms. Using the results from
13

---

FEAT-KD: Learning Concise Representations for Single and Multi-Targel Regression via TabNet Knowledge Distillation
Table 14: Classification: Fl score (SD in brackets) of cFEAT-KD, cFEAT variants, TabNet and Interpre averaged across 100 randomly seeded 60-20-20 suuin-vlidatioontee splits on 8 PMLB datasets (Romano et al., 2021) and 3 UCI datasets (Dua & Graff, 2019). Higher better, best white-box performance bold.
White-box
Black-box
cFEAT-KD (TabNet)
cFEAT-KD (InterpeeTabNet)
Dataset
cFEAT
cFEAT-Corr
cFEAT-CN
TabNet
InterpreTabbNet
chess
0.970 (0.0041)
0.969 (0.0034)
0.952 (0.0051)
0.947 (0.0043)
0.948 (0.0039)
0,986 (0.00092)
0.987 (0.0015)
hypothyroid
0,955 (0.0036)
0.956 (0.0028)
0.945 (0.0015)
0,944 (0.0016)
0.946 (0.0013)
0.978 (0.00094)
0.978 (0.0016)
ionosphere
0.912 (0.015)
0.908 (0.011)
0.905 (0.011)
0.898 (0.016)
0.894 (0.015)
0.905 (0.0052)
0.902 (0.0035)
kr_vs.kp
0,964 (0.0019)
0.965 (0.0031)
0.952 (0.0056)
0,948 (0. 0058)
0.948 (0.0051)
0.984 (0.0022)
0,986 (0.0028)
sonar
0.814 (0.025)
0.805 (0.035)
0.678 (0.036)
0.702 (0,040)
0,721 (0.049)
0,591 (0.027)
0.623 (0.040)
spambase
0.934 (0.0025)
0.935 (0.0025)
0.865 (0.012)
0.866 (0.0059)
0.869 (0.0092)
0,918 (0.0031)
0,916 (0.003S)
speetf
0.836 (0.024)
0.844 (0.021)
0.863 (0.019)
0.850 (0.013)
0.856 (0.017)
0.887 (0.021)
0.876 (0.020)
wkyol
0.919 (0.0054)
0,920 (0.0073)
0.929 (0.0095)
0.926 (0.0066)
0.927 (0.0054)
0.930 (0.0085)
0,935 (0.0080)
Diabctes
0.728 (0.012)
0.734 (0.017)
0.699 (0.0088)
0.693 (0.0094)
0.697 (0.016)
0.740 (0.019)
0.742 (0.014)
Forest Cover Type
0.554 (0.0041)
0.550 (0.0039)
0.538 (0.0069)
0.536 (0.0045)
0.539 (0.0043)
0.552 (0.0052)
0.554 (0.0058)
Poker Hand
0.124 (0.0037)
0.125 (0.0047)
0.120 (0.00060)
0.120 (0.00070)
0.120 (0.0010)
0.122 (0.00090)
0.123 (0.0035)
Table 15: Comparison of FEAT-KD with FEAT and other SR algorithms using SRBench data aggregated across 88
G. Ablation of DistilSR
datasets and 10 random seeds each.
As an ablation study, we substituted DistiISR with Operon (Burlacu al., 2020), PySR (Cranmer, 2023), DSR (Pe- tersen et al., 2019) and ParFam (Scholl et al., 2025), and tested on the 88 datasets used in FEAT. Like in SR bench- marking, there is a tradc-off between model size and pre- diction performance. The frequency of being on the Pareto- front for FEAT-KD is 88.6%. Whereas substitution with DSR, PySR, ParFam and Operon yielded 54.4%, 25.3%, 19.0% and 1.3% frequency of being Pareto optimal, respec- tively. We found that DistilSR in FEAT-KD is best able to exploit the expressivity of the search space of short equa- tions to better model the learned representations by TabNet without overfitting. The results also show that the task of fitting the learned representations is different from normal regression and that SR algorithms that tend to generate shorter expressions perform better. Finally, an additional strength of using DistilSR over other algorithms is that the variance is the lowest because the equation structure search space is the same for every run.
SR Algorithm
model size
R2
training time (s)
AFP
35.4
0.688
2790
AFP_FE
36.0
0.700
2850
AlFeynman
2240
-4.03
82900
BSR
19.8
0.245
13100
DSR
8.88
0.598
35100
EPLEX
56.3
0,814
7530
FEAT
79.1
0,847
6750
FEAT-KD (Ours)
49.0
0.851
1550
FFX
1630
0.00612
210
GP-GOMEA
25.1
0.794
7610
ITEA
107
0.663
6240
MRGP
10800
0.521
201000
Operon
63.7
0.853
1350
SBP-GP
693
0.851
155000
gplearn
17.6
0.563
23300
algorithms which appear more frequently than FEAT-KD on the Pareto-optimal front: DSR, GP-GOMEA, of which FEAT-KD has a better Rt2 than DSR in 83.8% of the experiments and a better R² than GP-GOMEA in 70.7% of experiments. Including Operon which appears less frequently that FEAT-KD on the Pareto-optimal front but is the 4th most frequent, in terms of R², the trend is -DP GOMEA>DSR and in terms of model size, the trend is BORRGG-GOMMA<FFEE KD<Operon. Thus, FEAT-KD is frequently Pareto-optimal, providing a new point on the optimal Pareto front with unique trade-off between R²2 and model size. It is also noteworthy that FEAT-KD appears much more frequently on the optimal front than FEAT (62.0% vs 23.0%). We also report the aggregated performance in Table 15.
14


Reviews:
[{"review_id": "N8CBwqCaT3", "reviewer": "Reviewer_mwAW", "summary": "This paper introduces FEAT-KD, a method that transfers the strengths of TabNet and FEAT to create concise, interpretable models for both single-target and multi-target regression. The method distills pieces of a trained TabNet into short symbolic expressions using an exhaustive search algorithm (DistilSR). These symbolic expressions are then combined into a weighted linear model. Extensive experiments on various datasets show that FEAT-KD achieves competitive predictive performance with significantly smaller model sizes compared to both TabNet and previous FEAT variants.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> General\n\nR1: We thank the reviewers for the thorough reviews and suggestions. We address the comments and implement the suggestions in our individual responses to the reviewers. For convenient referencing to a response, we use the notation R{number} to label a response (e.g., this response is labelled as R1). We hope that if the responses address the reviewer’s concerns, the reviewer would consider increasing the score.\n\n> Other Strengths And Weaknesses\n\nR2: For weakness 1, given that the premise in which a user would want to obtain a FEAT-like model (i.e., Eq. 1) is to obtain interpretability, we do not foresee having a large feature subset because that would lead to a long expression and is hence no longer interpretable [1,2]. If the feature subsets are high-dimensional, then the initial benefit of choosing a FEAT-like structure rather than a black-box model is largely lost. \n\nAdditionally, in Reply R22 we show that FEAT-KD can use alternative SR algorithms that scale better. However, note their prediction performance is at the expense of loss of interpretability (measured by increased size) or may even predict worse. DistilSR exploits the high expressivity of the search space of short expressions instead of greedily searching longer expressions at the expense of interpretability.\n\nR3: For weakness 2, we thank the reviewer for the suggestion and will make changes to the organization of the paper. Currently, we describe in Appendix D how we obtained the hyperparameter values via preliminary experiments on a small subset of the data to prevent data leakage, e.g., target 1 of atp1d. For the selection of “top 3”, it was not tuned but rather a decision for interpretability which is consistent with other TabNet literature [3]. We will move Tables 6 & 7 to the Appendix and split Table 8 into 2 parts (one in the main text and the other half in the Appendix) in order to move the current discussion of hyperparameters in Appendix D to the main text.\n\nR4: For weakness 3, the practical interpretability is argued to be measured in proxy by model size in [1,2] and the simplicity of the symbolic expressions has been argued to be interpretable via disentanglement [4] and also validated by clinicians in studies which uses FEAT structure (i.e., Eq. 1) [5]. We will include this discussion in Section 4.\n\nR5: For weakness 4, we will clarify that in Phase 3, we will extract the multiple mask matrices learnt by TabNet. For each mask matrix, we average the mask across the datapoints to obtain a single averaged value per feature. The 3 features with the highest averaged value are deemed to be the least unmasked and will be the features that are used in building the equation obtained in Phase 4. The computational cost of the exhaustive search is approximately O($d^l$), where $d=9$ is the number of different terminals and symbols, and $l=5$ is the length of the expression. We will be clearer on these in the main text Section 3.\n\n[1] Lage, I., et al. (2019). An evaluation of the human-interpretability of explanation.\n\n[2] Abdul, A., et al. (2020). COGAM: measuring and moderating cognitive load in machine learning model explanations.\n\n[3] Si, J. Y. H., e al. InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation.\n\n[4] La Cava, W., et al. (2019). Learning concise representations for regression by evolving networks of trees.\n\n[5] La Cava, W., et al. (2023). A flexible symbolic regression method for constructing interpretable clinical prediction models.\n\n> Questions For Authors\n\nR6: For Q1 and Q2, please see response R5. In addition, response R20, where we use other SR algorithms may interest the reviewer as well.\n\nR7: For Q3, we thank the reviewer for the suggestion and will include these in the main text per the reorganization of tables outlined in response R3.\n\nR8: For Q4, consider a simple example with true outputs $y=[0,12,28]$ and inputs $x_1=[1,2,3]$ and $x_2=[4,5,6]$. A candidate equation $x_1 \\times x_2$, produces predictions $[4,10,18]$, which under traditional MSE yield a relatively high value that would lead to the equation not being picked. However, when using AFI-MSE, which first optimally scales and shifts the predictions to best match the true values, we can find parameters $a$ and $b$ (specifically, $a=−8, b=2$) such that the adjusted prediction $-8 + 2 \\times (x_1 \\times x_2)$ exactly equals $y$, resulting in a zero error. Thus, AFI-MSE recognizes that while the candidate's predictions differ in scale and shifts, their underlying pattern aligns perfectly with the true values, highlighting the candidate's potential as a good fit despite the initial magnitude mismatch. This is suitable particularly for FEAT-KD because in “Phase 5”, this equation will be scaled and shifted anyway because it is being used as a feature in linear regression. This also effectively simplifies the search space. We will include this discussion in the main text Section 3 under AFI-MSE.", "claims_and_evidence": "Yes, most of them are supported.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA, there is not theory", "experimental_designs_or_analyses": "Yes, I think they are valid.", "relation_to_broader_scientific_literature": "Good", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method effectively converts a deep learning model into a white-box model by producing symbolic, concise representations.\n2. FEAT-KD supports multi-target regression, which is not available in the original FEAT and its variants.\n3. The paper provides an extensive experimental evaluation on a variety of datasets, with statistical tests (e.g. Bonferroni-adjusted p-values) to support its claims.\n4. The five-phase approach is clearly laid out, showing how deep-learning based feature selection and symbolic regression are integrated (Sec 3).\n\nWeaknesses:\n1. The use of an exhaustive search algorithm (DistilSR) for symbolic regression may not scale well if more complex or higher-dimensional feature subsets are required.\n2. The paper does not fully address the sensitivity of the method to hyperparameters such as the number of features selected from the masks (top 3) or the choices for Nd and Nsteps. The current ablation studies (Tables 9–11) are informative but could be discussed more in the main text. (But in general, there are a bit too many tables in the main text...)\n3. While the paper claims improved interpretability via model size reduction, there is little discussion on how the simplicity of the symbolic expressions translates to practical interpretability for end users?\n4. Some methodological detailsare not fully detailed, esp on the feature selection using TabNet masks (Phase 3) and the computational cost of the exhaustive search.", "other_comments_or_suggestions": "See the questions for authors.", "questions_for_authors": "1. Please provide more details on the computational cost and scalability of the DistilSR step. A brief discussion or additional experimental data in sec 3 would be helpful.\n2. Clarify the exact procedure for selecting the top 3 features from the TabNet masks. Including a more detailed example or additional explanation near Section 3 could improve clarity.\n3. Expand the discussion on the sensitivity of the method to the hyperparameters Nd and Nsteps. It would be beneficial to integrate a short ablation study discussion from Tables 9–11 into the main text.\n4. Explain a bit further how the AFI-MSE metric (sec 3) compares to traditional MSE, perhaps with a simple illustrative example.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMboDH1pVY", "reviewer": "Reviewer_kPdZ", "summary": "The FEAT framework typically uses genetic algorithms to derive the concise-feature representations. The linear combination of these concise-representation features is then the predicted output of this model. TabNet is a deep-learning based model for tabular data that can be used for single/multi-target regression. In this work, the authors propose a novel framework called FEAT-KD wherein they combine the ideas of TabNet and FEAT along with knowledge distillation and an exhaustive-search symbolic regression instead of genetic algorithms to identify the concise-representations. More specifically, the authors combine the “steps” from the TabNet architecture, and then apply symbolic regression on the output of each “step” along with the extracted TabNet masks for feature selection to distill it into a concise mathematical equation. Finally, they perform a simple linear regression on the concise-representations to end up with a “white-box” model structure that is comparable to FEAT in terms of interpretability but without using genetic programming.\n\nThe main results show that FEAT-KD performs better than FEAT and other variants such as FEAT-Corr, and FEAT-CN across a range of single-target regression and multi-target regression datasets while still being similar to the TabNet (large sized teacher model) model(s). The authors also show that the resulting model size (in terms of the total number of functions and terminals in the model) is on-average the smallest across most of the model-dataset combinations evaluated upon, and they show that these improvements in model size/performance are statistically significant by using the one-sided Wilcoxon signed-rank test. Thus, the authors show that we can achieve the best of both worlds - the improvements shown by TabNet along with the interpretability of the FEAT framework, for tabular data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> General\n\nWe thank the reviewer for the thorough review and suggestions. We address the comments and implement the suggestions in our individual responses to the reviewers. For convenient referencing to a response, we use the notation R{number} to label a response. We hope that if the responses address the reviewer’s concerns, the reviewer would consider increasing the score.\n\n> Experimental Designs Or Analyses\n\nR9: We thank the reviewer for the suggestion and have included other comparisons with DSR and AIFeynman, methods that uses some elements of deep-learning to find interpretable equations, which we will include in Appendix G & H (please see responses R20 & R22).\n\n> Essential References Not Discussed\n\nR10: We thank the reviewer for the pointer, we will reference the influential work by Cranmer, M., et al. and discuss the differences. For evaluation purposes the method by Cranmer, M., et al. is designed for “interacting particle” systems, where the problem is modelled as individual particles, each with their own dataset, and the task is to find the interactions between them. It is thus not adaptable to common tabular regression done by FEAT and TabNet. We have in the revision, however, compared to DSR and AIFeynman, which uses elements of deep-learning to find interpretable equations, which we will include in Appendix G & H (please see responses R20 & R22). DSR and AIFeynman work very differently from FEAT-KD. While FEAT-KD uses deep-learning to model the distribution of the data directly and distill components piecewise, DSR uses deep-learning for modelling the distribution of candidate equations and AIFeynman uses a neural network fitting to remove noise from data. Another difference is the objectives of the algorithms are different, FEAT-KD tries to find a structure in the form of Eq. 1, which has been validated by clinicians to be interpretable, whereas DSR and AIFeynman uses different structures. We will add this discussion in the main text Section 4.\n\n> Other Strengths And Weaknesses\n\nR11: We thank the reviewer and will add more comparisons (we respond to this above in R9 & R10, which is in turn responded partially with responses R20 & R22). \n\n> Questions For Authors\n\nR12: For Q1, the use of symbolic regression on top of Graph Neural networks is not adaptable for tabular regression which TabNet and FEAT addresses. However, we recognize that although FEAT-KD uses the specific form given in Eq. 1 for interpretability reasons, it can also be counted as a type of SR algorithm and be positioned in the broader literature of SR algorithms. We respond to this above in R9 & R10, which is in turn responded partially with responses R20 & R22. \n\nR13: For Q2, we thank the reviewer for the reference. The current interpretable methods applied on TabNet look at feature attributions, similar to the trained masks in TabNet. FEAT-KD retains this exact property and these current methods are still applicable to FEAT-KD. However, FEAT-KD takes this one step further by finding a concise faithful equation that captures the behaviour of the learnt representation. Thus, in terms of interpretability, FEAT-KD subsumes the interpretability that TabNet possesses and further adds an additional aspect of interpretability via the learnt equations.", "claims_and_evidence": "The claims made by the authors are clearly discussed.\nClaims made - \n1. FEAT-KD results is close to the performance of the TabNet model while improving the interpretability from being a black-box model to a white-box model.\n2. FEAT-KD has the best average prediction performance that is consistent and statistically significant among symbolic models evaluated.\n3. FEAT-KD has the lowest average model size among all models evaluated and has statistically significant evidence that it is consistently smaller.\n4. FEAT-KD easily supports multi-target regression.\n5. FEAT-KD has some regularization effect over TabNet.\n\nIn general, the claims are well-substantiated with numerical comparisons and p-values for statistical significance of the results.", "methods_and_evaluation_criteria": "The proposed methods of using the TabNet architecture to distill the “step” outputs along with the masks to learn concise features with symbolic regression via an exhaustive search makes sense for the problem of tabular regression. Further, the use of the datasets to benchmark the performance is appropriate and consistent with the existing literature on the topic(s), considering that the datasets cover different types - i.e., synthetic datasets, single-target & multi-target regression datasets. The use of the evaluation metrics - R2 score, model size (in terms of functions and terminals) is also appropriate and relevant for the field.", "theoretical_claims": "The paper does not provide any proofs for theoretical claims, the claims made are empirical in nature and they are validated experimentally.", "experimental_designs_or_analyses": "Yes. Experimental design. An issue with the current setup is that the authors only considered other symbolic methods and the teacher model (TabNet) for comparisons but do not consider other similar approaches that use symbolic regression on top of deep-learning methods. Considering these baselines too would have made the evaluation section stronger that it currently is.", "relation_to_broader_scientific_literature": "1. Symbolic regression methods use genetic programming to identify the concise features. Feat-KD replaces this step by finding a weighted linear combination of concisely represented symbolic features via piece-wise knowledge distillation corresponding to each “step” of the trained TabNet model.\n\n2. Feat-KD is built on top of existing methods such as TabNet and FEAT. While TabNet is a deep-learning-based model and inherently a “black-box” model, FEAT, on the other hand, is a “white-box” algorithm. Feat-KD achieves competitive performance with TabNet on single and multi-target regression datasets while also making the resulting model more interpretable compared to the original TabNet.\n\n3. Across all the eval datasets considered Feat-KD consistently outperforms other symbolic methods such as FEAT and variants of FEAT. The authors show that FEAT-KD consistently outperforms other symbolic models such as FEAT and its variants both in performance (R2 score) and also in terms of model size (smaller size) compared to other symbolic methods.", "essential_references_not_discussed": "The following work(s) are not essential references required to understand the content of the paper, but they seem to be relevant to the ideas proposed here and the authors would be justified to have mentioned them in the related works section or done some comparisons against these methods, especially methods that also use symbolic regression on top of deep-learning based methods to find interpretable equations.\nCramer et al - Discovering Symbolic Models from Deep Learning with Inductive Biases (NeurIPS 2020)", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of combining existing methods such as TabNet and FEAT using symbolic regression with knowledge distillation instead of genetic programming to formulate FEAT-KD is novel.\n2. The straightforward extension from single-target regression to multi-target regression will have a big impact on critical industries such as healthcare.\n3. The resulting method ends up making the TabNet architecture “more interpretable”.\n\nWeakness:\n1. Even though the experiments and ablations presented are comprehensive they’d have been stronger by including comparisons against methods that also use symbolic regression methods (with genetic programming) along with Deep-Learning techniques to derive interpretable equations such as the use of symbolic regression on top of GNN networks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. In the evaluations section, is there a reason why other similar approaches as the use of symbolic regression on top of Graph Neural networks are not considered as another baseline?\n2. In terms of interpretability, how would the existing interpretable methods be applied on TabNet and fare vs the interpretable FEAT-KD in terms of interpretability? Ref - Fig.2 from Borisov et al., Deep Neural Networks and Tabular Data: A Survey", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "ln7n524iX4", "reviewer": "Reviewer_Dqc6", "summary": "The paper presents FEAT-KD, a method that distills knowledge from a TabNet deep neural network into interpretable symbolic regression models. It replaces traditional genetic programming (GP) used in FEAT with a neural-guided symbolic regression approach, producing concise mathematical expressions. Key contributions include demonstrating competitive performance compared to TabNet, significantly simpler symbolic models, and support for multi-target regression tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> General\n\nPlease see response R1 to reviewer mwAW.\n\n> Experimental Designs Or Analyses\n\nR14: We thank the reviewer for the suggestion, the original FEAT is designed for regression tasks [1] and since a large amount of evaluation was required to verify both single and multi-target regression, we did not focus on classification in this work and instead focused on existing datasets that TabNet and FEAT evaluates on. However, the reviewer is right FEAT-KD can be adapted for classification by changing using logistic regression in Phase 5, which we show is possible via the Diabetes dataset in the revision Appendix I, “Extension to Classification”, but the thorough evaluation and design would be left to future work since we would like the space to focus more thoroughly on single and multi-target regression tasks.\n\n[1] La Cava, W., et al. (2019). Learning concise representations for regression by evolving networks of trees.\n\n> Essential References Not Discussed\n\nR15: We thank the reviewer for the pointer to InterpreTabNet. InterpreTabNet has 2 main contributions, both of which are relevant and we will address: i). A regularization scheme that maximizes diversity between masks in the TabNet architecture, ii). capturing feature interdependencies by prompting LLMs with the learnt masks.\n\nFor i)., InterpreTabNet works very similarly to TabNet, just with extra regularization, so we could easily apply FEAT-KD techniques by replacing TabNet with InterpreTabNet in the code implementation. The results yield stronger performance on some datasets, but not on all, consistent with the results in [2] itself. Thus, InterpreTabNet allows for a variant of FEAT-KD which practitioners should try as it is dataset dependent (depends on whether reuse of features across attention masks in TabNet is desirable for the problem). We will include these results in Appendix J, “InterpreTabNet Variant of FEAT-KD” and show a subset of test $R^2$ below:\n\n|Dataset|FEAT-KD (TabNet)|FEAT-KD (InterpreTabNet)|\n|-|-|-|\n|bodyfat|0.987 (0.0058)|0.990 (0.0051)|\n|cpu_act_197|0.976 (0.0014)|0.977 (0.0015)|\n|cpu_act_573|0.976 (0.0014)|0.976 (0.0016)|\n|cpu_small|0.968 (0.0015)| 0.966 (0.0015)|\n|house_8L|0.585 (0.01)|0.588 (0.0096)|\n|houses|0.664 (0.0078)|0.660 (0.0053)|\n|pm10|0.181 (0.026)|0.196 (0.042)|\n|puma8NH|0.618 (0.026)|0.596 (0.0086)|\n\nFor ii). InterpreTabNet uses LLMs to generate linguistic interpretations of the masks obtained which is not mutually exclusive with FEAT-KD. In fact, InterpreTabNet and FEAT-KD complement each other, in which InterpreTabNet generates a simplified qualitative description, whereas FEAT-KD generates a simplified quantitative description, which we thank the reviewer for highlighting. Thus, from the trained mask, 2 interpretations can be generated, one from using an LLM to generate a linguistic interpretation and the other from the learnt equation from FEAT-KD. We will include these results in Appendix J, “InterpreTabNet Variant of FEAT-KD”.\n\nThus, the strengths of InterpreTabNet can be subsumed into FEAT-KD by replacing TabNet with InterpreTabNet, which we thank the reviewer for the recommendation and have included in the revision.\n\n[2] Si, J. Y. H., e al. InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation.\n\n> Other Strengths And Weaknesses\n\nR16: For clarity, we thank the reviewer for the suggestion and will reduce the size of Figure 1 by 70% and reduce the number of sentences in the abstract.\n\n> Questions For Authors\n\nR17: For interpretability aspect question, the practical interpretability is argued to be measured in proxy by model size in [3,4] and the simplicity of the symbolic expressions has been argued to be interpretable via disentanglement [5] and also validated by clinicians in studies which uses FEAT structure (i.e., Eq. 1) [6]. We will include this discussion in Section 4.\n\n[3] Lage, I., et al. (2019). An evaluation of the human-interpretability of explanation.\n\n[4] Abdul, A., et al. (2020). COGAM: measuring and moderating cognitive load in machine learning model explanations.\n\n[5] La Cava, W., et al. (2019). Learning concise representations for regression by evolving networks of trees.\n\n[6] La Cava, W., et al. (2023). A flexible symbolic regression method for constructing interpretable clinical prediction models.\n\nR18: We thank the reviewer for the suggestion and will tackle “interpretability” via a qualitative and quantitative approach, where we will make clear that we can leverage InterpreTabNet for qualitative interpretability via capturing feature interdependencies by prompting LLMs with the learnt masks and we can assess quantitative interpretability via computable metrics such as model size. We will also tackle the sacrifice and tradeoff via Pareto fronts, which we address in response R20 and R22 as well.", "claims_and_evidence": "Claims of competitive accuracy, improved interpretability, and successful multi-target support are supported by empirical results (15 single-target and 5 multi-target datasets). Statistical analyses confirm significant improvements over original FEAT variants.", "methods_and_evaluation_criteria": "TabNet's learned feature representations guide symbolic regression, greatly reducing the search space. Evaluation criteria are appropriate.", "theoretical_claims": "No complex theoretical proofs in the paper.", "experimental_designs_or_analyses": "The paper rigorously evaluates multiple datasets with repeated splits and statistical testing (Wilcoxon tests, Bonferroni correction). A limitation: evaluation on very high-dimensional or extremely large-scale data is not addressed. Additionally, FEAT-KD generates symbolic expressions as linear combinations of numeric features, which naturally aligns with continuous-valued predictions thus, scope of this method is limited to only regression tasks which is not ideal.", "relation_to_broader_scientific_literature": "Connects with symbolic regression (FEAT, DistilSR, DSR), knowledge distillation, interpretability literature, and multi-target regression research.", "essential_references_not_discussed": "The major benefit of this work is a form of interpretable machine learning for tabular data. Hence, InterpreTabNet (https://arxiv.org/abs/2406.00426) from ICML 2024 should be very relevant to this paper.", "other_strengths_and_weaknesses": "**Originality**\n\nWork is novel. As far as I know, I have not come across the use of symbolic regression within interpretable tabular learning.\n\n**Significance**\n\nGood significance. Addresses the critical need for interpretable ML models without substantial performance loss.\n\n**Clarity**\n\nGood. Although Figure 1 seems to be way too large filling 1 whole page and abstract is way too long. Per the guidelines, abstract should be 4-6 sentence long.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "As mentioned above, I understand that this work is a form of interpretable machine learning for tabular data. However, the interpretability aspect is still unclear to me. There are many nice experiments highlighting your quantitative performance over other symbolic methods as well as remaining somewhat competitive to TabNet barring in mind the interpretability vs. accuracy tradeoff. But I am unable to observe extensive analysis or examples into how in “interpretability” of FEAT can assist practitioners.\n\nI would highly recommend looking into [1] InterpreTabNet to rephrase how the “interpretability” of your method outweighs the sacrifice in accuracy.\n\n[1] Si, J., Cheng, W.Y., Cooper, M. and Krishnan, R.G., 2024. InterpreTabNet: distilling predictive signals from tabular data by salient feature interpretation. *arXiv preprint arXiv:2406.00426*.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZzJfSnGDFR", "reviewer": "Reviewer_9uvT", "summary": "The paper substitutes the feature search of the SR method FEAT by using the mask of a TabNet to select the important input variable and learn the feature using DistilSR, an exhaustive search method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> General\n\nPlease see response R1 to reviewer mwAW.\n> Claims And Evidence\n\nR19: We thank the reviewer and will add the results for extended datasets in PMLB used in FEAT in Appendix F, “Evaluation on More Datasets”. We pick a representative subset in the main pages because we want to show results on datasets from a diverse range of independent sources (synthetic and real-world data from both TabNet and FEAT, and multi-target regression datasets).\n\nR20: We thank the reviewer for the pointer. Although FEAT-KD uses the specific form given in Eq. 1 for interpretability reasons, we agree it can also be counted as a type of  SR algorithm and be positioned in the broader literature of SR algorithms. Using the results from SRBench, across the datasets and random seeds, FEAT-KD is Pareto-optimal for 60.0% of them with respect to the other SR algorithms. The rate at which the other SR algorithms DSR, GP-GOMEA, Operon, gplearn, AFP, AFP_FE, AIFeynman, FEAT, EPLEX, ITEA, SBP-GP, BSR, MRGP, FFX, are 88.8%, 68.8%, 61.3%, 53.8%, 48.8%, 40.0%, 27.5%, 23.8%, 21.3%, 15%, 11.3%, 8.8%, 5.0%, 2.5%, respectively. In SRBench, Pareto-optimal means that FEAT-KD has the optimal trade-off with prediction performance and equation size, in which there are no other SR algorithms with smaller equation size with better prediction performance. Of note are 3 algorithms which appear more frequently than FEAT-KD on the Pareto-optimal front: DSR, GP-GOMEA and Operon, of which FEAT-KD has a better test_R2 than DSR in 86.3% of the experiments, a better test_R2 than GP-GOMEA in 68.8% of experiments, and smaller size than Operon in 100% of experiments. A Pareto plot will be able to demonstrate this better, which we will include in the revision, but generally speaking, in terms of test_R2, the trend is Operon>FEAT-KD>GP-GOMEA>DSR and in terms of model size, the trend is DSR<GP-GOMEA<FEAT-KD<Operon. Thus, FEAT-KD is frequently Pareto-optimal, providing a new point on the optimal Pareto front with a unique trade-off between test_R2 and model size. We will include these results in Appendix G, “FEAT-KD as an SR Algorithm”.\n\nIt is also noteworthy that FEAT-KD appears much more frequently on the Pareto-optimal front than FEAT (60.0% vs 23.8%).\n> Methods And Evaluation\n\nR21: We think yes to a large extent, and the reasons to support this are kindly summarized succinctly from reviewer kPdZ: “the use of the datasets to benchmark the performance is appropriate and consistent with the existing literature on the topic(s), considering that the datasets cover different types - i.e., synthetic datasets, single-target & multi-target regression datasets.” And from reviewer Dqc6: “rigorously evaluates multiple datasets with repeated splits and statistical testing (Wilcoxon tests, Bonferroni correction).”\n> Experimental Designs\n\nR22: We address the datasets and baselines above. As for the ablation study, we thank the reviewer for the suggestion, and have substituted with Operon, PySR, DSR and ParFam. Similar to SR benchmarking, there is a trade-off between model size and prediction performance. The frequency of being on the Pareto-front for FEAT-KD is 88.6%. Whereas substitution with DSR, PySR, ParFam and Operon yielded 54.4%, 25.3%, 19.0% and 1.3% frequency of being Pareto optimal, respectively. We found that DistilSR in FEAT-KD is best able to exploit the expressivity of the search space of short equations to better model the learnt representations by TabNet without overfitting. The results also show the task of fitting the learnt representations is different from normal regression and that SR algorithms that tend to generate shorter expressions perform better. Finally an additional strength of using DistilSR over other algorithms is that the standard deviation over random seeds is the lowest because the equation structure search space is the same for every run. We will include this discussion in Appendix H, “Ablation: Substituting DistilSR”.\n> Other Strengths And Weaknesses\n\nR23: In cases where exhaustive search with too many base functions are expensive, non-exhaustive SR algorithms, such as those used in response R22, can be used. We also intentionally chose a smaller subset of function for FEAT-KD (see Table 3) for interpretability.\n> Comments\n\nR24: We will draw a vertical line to segregate models with FEAT-like structure (i.e., Eq. 1), and deep-learning models.\n\nR25: We thank the reviewer and will include this baseline for multi-target regression. Other than compute, it is also less interpretable because it does not reuse learnt features across targets.\n> Questions\n\nR26: FEAT was allowed to use all base functions in Table 2 in the main text and FEAT-KD was only allowed to use base functions in Table 3 in the main text for increased interpretability.\n\nR27: TabNet itself was run on the same single CPU which has 36 cores and a walltime of 3600 second (or equivalently 129600 core-seconds) counts into the time budget.\n\nR28: We will bold the winner in Table 7.", "claims_and_evidence": "The reviewer appreciates that the paper shows the improvement with respect to FEAT (and variants) and that the loss in performance comapred to TabNet is moderate, especially in the multi-task setting. However, for a fair and thorough comparison it is important to use the whole datasets, especially from the FEAT paper.\n\nWhile the most interesting comparison is definitely with respect to FEAT, it is also important to compare to other SR methods, with results readily available for the standard SR benchmark SRBench.", "methods_and_evaluation_criteria": "Do proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application at hand?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments itself are well designed, however, they are missing datasets and baselines (see Claims and Evidence). \n\nAn interesting ablation study would be to substitute DistilSR by a different SR algorithm to see if this idea is also able to improve SR in general. It would be beneficial to use a state-of-the-art algorithm (e.g., Operon [1], PySR [2], uDSR [3], RILS-ROLS [4], or ParFam [5]; one of these is enough, or a similar performing algorithm)  for this, since it would also directly show if this method helps to push the field of SR itself. \n\n[1] Kommenda, M., Burlacu, B., Kronberger, G. and Affenzeller, M., 2020. Parameter identification for symbolic regression using nonlinear least squares. _Genetic Programming and Evolvable Machines_, _21_(3), pp.471-501.\n\n[2] Cranmer, M., 2023. Interpretable machine learning for science with PySR and SymbolicRegression. jl. _arXiv preprint arXiv:2305.01582_.\n\n[3] Landajuela, M., Lee, C.S., Yang, J., Glatt, R., Santiago, C.P., Aravena, I., Mundhenk, T., Mulcahy, G. and Petersen, B.K., 2022. A unified framework for deep symbolic regression. _Advances in Neural Information Processing Systems_, _35_, pp.33985-33998.\n\n[4] Kartelj, A. and Djukanović, M., 2023. RILS-ROLS: robust symbolic regression via iterated local search and ordinary least squares. Journal of Big Data, 10(1), p.71.\n\n[5] Scholl, P., Bieker, K., Hauger, H. and Kutyniok, G., 2025. ParFam--(Neural Guided) Symbolic Regression via Continuous Global Optimization. In _The Thirteenth International Conference on Learning Representations_.", "relation_to_broader_scientific_literature": "The paper is well embedded in the literature.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- The method is very well motivated and shows that the idea worked. \n- I am sure that the idea itself is useful for the SR community, as partitioning a formula into smaller parts is an extremely usefull tool for SR. This is another reason for my interest in the ablation study.\n- The paper is written clearly.\n- The extension to multi-task is appreciated.\nWeaknesses:\n- See above\n- By relying on exhaustive search, it is hard to include more base functions like sin, exp, etc., which is why currently this method can only learn rational functions and potences.", "other_comments_or_suggestions": "- While I agree, that TabNet is not the competitor FEAT-KD has to be beat, Table 4 is slightly misleading as TabNet is included but not bold, even though it wins for most data sets.\n- The experiments for multi-task would benefit from a SR baseline. Possibly the authors could simply divide the compute budget by the number of dimensions and run standard FEAT separately. This should help to show the performance gain by FEAT-KD.", "questions_for_authors": "- Which base functions was FEAT allowed to use?\n- Is the TabNet itself also computed on a single CPU? And it counts into the time budget?\n- Why is the winner not bold in Table 7.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four reviewers submitted reviews for this paper. After the initial reviews, two of the reviewers updated their ratings.

During the reviewer-AC discussion period, all the reviewers shared their thoughts on this paper, considering the author rebuttal.
Through the internal discussions, one of the reviewers changed their position to leaning towards a reject as they consider applying symbolic regression to tabular learning is not novel. A few reviewers are satisfied with the author rebuttal and additional analysis.

It is an interesting study that applies an idea of knowledge distillation to TabNet with some perspective of symbolic regression (SR). Multi-target regression on top of that makes this study a little bit more unique. It may be worth discussing this study with people from the SR community.

Since some reviewers updated their ratings based on the author rebuttal and the amount of additional contents in their rebuttal is significant, I expect the authors to reflect the changes and additional results to the next revision. Specifically, I find the additional experimental results and analysis not detailed enough e.g., hyperparameter tuning, exactly what datasets they used. 

As one of the reviewers is now leaning towards rejection and some of the updated ratings are based on the rebuttal, I am concerned about the quality and soundness of the additional contents. While I suggest accepting this work, I would not fight for accepting this work if there is not room in the program.
Decision: Accept (poster)