LCEN: A Nonlinear, Interpretable Feature Selection and Machine Learning Algorithm
Authors: Pedro Seber
Richard Braatz
Venue: Accepted by TMLR ()

Abstract: Interpretable models can have advantages over black-box models, and interpretability is essential for the application of machine learning in critical settings, such as aviation or medicine. In this work, we introduce the LASSO-Clip-EN (LCEN) algorithm for nonlinear, interpretable feature selection and machine learning modeling. LCEN is tested on a wide variety of artificial and empirical datasets, creating sparse and frequently more accurate models than other methods, including sparse, nonlinear methods, on tested datasets. LCEN is robust against many issues typically present in datasets and modeling, including noise, multicollinearity, and data scarcity. As a feature selection algorithm, LCEN matches or surpasses the thresholded elastic net but is, on average, 10.3-fold faster  based on our experiments. LCEN for feature selection can also rediscover multiple physical laws from empirical data. As a machine learning algorithm, when tested on processes with no known physical laws, LCEN achieves better results than many other dense and sparse methods --- including being comparable to or better than ANNs on multiple datasets.

Published in Transactions on Machine Learning Research (11/2025)
LCEN: A Nonlinear, Interpretable Feature Selection and Ma- chine Learning Algorithm
Pedro Seber
pseber@mite edu
Deportment of Chemical Engineering Massachusetls Institute of Technology
Richard D. Braatz
braatz edu
Department of Chemical Engineering
Massachusetts Institute of Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= ad=umbucISPdl
Abstract
Interpretable models have advantages over black-box models, and interpretability is essential for the application of machine lcarning in critical scttings, such as aviation or medicine. This article introduces the LASSO-Clip-EN (LCEN) algorithm for nonlinear, interpretable feature selection and machine learning modeling. In a wide variety of artifi- cial and empirical datasets, LCEN constructed sparse and frequently more accurate models than other methods, including sparse, nonlinear methods, on tested datasets. LCEN was empirically observed to be robust against many issues typically present in datasets and mod- eling, including noise, multicollinearity and data scarcity. As feature selection algorithm, LCEN matched or surpassed the thresholded clastic net but was, on average, 10.3-fold faster based on our experiments. LCEN for feature selection can also rediscover multiple physical laws from empirical data. As a machine learning algorithm, when tested on processes with no known physical laws, LCEN achieved better results than many other dense and sparse methods- including being comparable to or better than ANNs on multiple datasets.
1 Introduction
Many modeling methods and algorithrms exist, including for the construction of linear, ensemble-based and deep learning models. Complex models can have greater capability to model phenomena due to their lower bias, but have intricate and numerous mathematical transformations that prevent humans from understand- ing how an output was predicted by model, or the relative or absolute importance of the inputs. Morcover, lack of transparency may prevent the model from being trusted in critical or sensitive applications (Hong et al., 2020).
In modeling context, interpretability can be defined as "how an output y f(X) was predicted for a given input X that is, provide f'(-) in a form readily understandable to humans," which also allows the model's outputs to be explainable. There are two main methods to increase interpretability: the use of model- agnostic algorithms, which extract interpretable explanations a posteriorí and work for any model, or the direct use of interpretable models (Ribeiro et al. 2016). Interpretable models inchude "decision trees, rules, additive models, attention-based networks, and sparse linear models" (Ribeiro et al., 2016). Interpretable posteriori explanations, including the ability to assist
models can have many advantages over black-box researchers in refining the model and data, or better highlighting scenarios in which the model fails lacks robustness (Rudin, 2019). Special attention should be given to sparse models, which identify the most important features, can make the model more robust to variations in the input data, and can significantly improve the model's interpretability if an interpretable model is used (Rudin, 2019). A sparse model may
Nonlinear models may also be made sparse, and even interpretable, described in this and many works.

---

Published in Transactions on Machine Learning Research (11/2025)
be defined as "a model that uses few input features, particularly relative to the total number of features available." However, even a linear model or decision tree/rules can become unwieldy and challenging to interpret if hundreds or thousands of coefficients or rules are present. A recent work states that sparsity may be achieved for individual predictions even if the overall model is not sparse (Sun et al., 2024).
Feature selection is the process of selecting the most. important features in a model to increase its robustness, interpretability, or sparsity, Many criteria for feature selection exist (Heinze et al., 2018), including signifi- cance based on p-values (using univariate, iteratiie/sseepwisee, or global method), using information criteria (such as the AIC (Akaike, 1974) and BIC (Schwarz, 1978)), using penalties (such as in LASSO (Santosa & Symes, 1986; Tibshirani, 1996) and clastic net (EN) (Zou & Hastie, 2005)), criteria based on changes in estimates, and expert knowledge. More broadly, these methods can be classified as filter, wrapper, or integrated methods. While no method is superior for all problems, different works have evaluated and crit- icized these criteria. For example, stepwise regression is one of the most commonly used methods in many fields thanks to its computational simplicity and case of undorstanding (Whittingham et al., 2006; Heinze et al., 2018; Smith, 2018). However, stepwise regression is prone to ignoring features with causal effects, in- cluding irrelevant features, generating excessively small confidence intervals, and producing incorrect/ //iaaedd parameters (Whittingham et al., 2006; Smith, 2018). LASSO is simple and computationally cheap, and has performed well for some problems (Hebiri & Lederer, 2013; Tian et al., 2015; Pavlou et al., 2016), but can overselect irrelevant. variables, tends to select only as many features as there are samples, and does not handle multicollinear data well (Heinze et al., 2018; Zou & Hastie, 2005).²
Originally, most feature selection methods applied only in linear contexts (or were applied primarily in linear contexts). Highlighting this, the only sparse models referenced in the highly cited review by Ribeiro et al. (2016) are linear. The most commonly used sparse methods (LASSO, EN, and their variants) are linear regressors. To address this limitation, later works consider sparse nonlinear models. For example, McConaghy (2011), Brewick et al. (2017), and Sun & Braatz (2020) defined sets of features consisting of polynomials (all works), interactions (all works), and/or non-polynomials (McConaghy, 2011; Sun & Braatz, 2020). ALVEN (Sun & Braatz, 2020) uses an F-iest. for each feature (including the expanded set of features) to determine whether to keep a feature in the final EN model, a filter approach. However, this F-test has very poor fcature selectivity, as nearly all features are selected when traditional values of (0.001 I 0.05) are used. Furthermore, the ordering of the features with respect to their p-values does not follow their relevance, as many irrelevant features are among those with the lowest p-values, and relevant features can be among those with the highest p-values (including >> 0.05). Other relevant methods include the smoothly clipped absolute deviation (SCAD) (Fan & Li, 2001): ) Sparse Additive Models (Ravikumar et al., 2009), which were used with Lo and L2 regularization by Liu el. al. (2022); the adaptive elastic nel (Zou & Zhang, 2009); the minimax concave penalty (MCP/MC+) (Zhang, 2010); the thresholded LASSO (van de Geer et al., 2011); the two-stage regularized method of De Mol et al. (2009); two forms of modified, nonlinear LASSO algorithms (Yamada et al., 2018); cutting plane algorithm (Bertsimas & Parys, 2020); Bayesian symbolic regression approaches (Xu et al., 2021); and the UniLasso (Chatterjec et al., 2025).
More recently, L1 regularization has been applied to neural networks for nonlinear feature selection. In its simplest form, group LASSO is applied to zero all the outputs of some neurons (sparsifying the network and eliminating features when zeroing input-layer neurons) (Dinh & Ho, 2020; Scardapane et al., 2017; Wang et al., 2021). LassoNet, a slightly modified version of this algorithm, applics the L1 penalty only to the input layer and includes skip-connection between that layer and the output layer (Lemhadri More complex applications of this method include the multi-modal neural networks of Zhao et al. (2015), the concrete autocncoders of Baln et al. (2019), and the teacher-student network of Mirzaci et al. (2020).
al., 2021).
The first two methods are at least partially unsupervised, suggesting that they can select the most relevant features for a given dataset irrespective of the task. Some works have used approaches other than the L norm for neural network- based feature selection, such as the Lo pseudonorm (Yamada et al., 2020). While these deep learning models are powerful tools, two considerable limitations are present: first, they do not provide any information on how the selected features are contributing to the final prediction, significantly limiting interpretability. Although sometimes useful, posteriori methods to extract this information have
This last point is somewhat controversial the literature; see Hebiri & Lederer (2013) and Dalalyan et al. (2017), for example.
2

---

Published in Transactions on Machine Learning Research (11/2025)
been found unreliable (Rudin, 2019). Second, these complex model architectures may take "shortcuts" to make apparently accurate predictions (Tapuschkin et al., 2019; Rosenzweig et al., 2021). However, these "shortcuts" are not really relevant to the task, preventing proper generalization and human interpretation.
To create nonlinear, interpretable, and sparse machine learning models with high predictive and descriptive power; we propose the LASSO-ClipEEN (I.CEN) algorithm. This algorithm generates an expanded set of nonlinear features (such as in ALVEN) and performs feature selection and model fitting, This feature set expansion, together with the Clip step, provide LCEN with the ability to do nonlinear predictions. The feature selection algorithm and the specific usage of thresholded LASSO (LC) followed by thresholded EN (ENC) in combined algorithm are the basis of novelty in this work. The algorithmic structure of LCEN is motivated by past works that proved desirable theoretical properties of the thresholded LASSO (a LASSO- Clip model) and the thresholded EN (an EN-Clip model), which include Zhou (2009), Meinshausen & Yu (2009); Zou & Zhang (2009), Zhou (2010), and van de Geer et al. (2011). Variations and ablations of LCEN are obscrved to not perform as well as LCEN. In case studies involving artificial and empirical data, LCEN successfully rediscovers physical laws from data belonging to multiple different. areas of knowledge with errors < 0.5% on the coefficients, a value within the empirical noise of the datasets. For tested datasets from processes whose underlying physical laws are not yet known, LCEN is observed to attain lower root- mean-square errors (RMSEs) than many sparse and dense methods, leads to sparser models than all but two methods tested, 3 and simultancously ran faster than most alternative methods.
2 Methods
2.1
The LCEN algorithm
The LCEN algorithm (Algorithm 1) begins with the LASSO step, which temporarily sets the UI ratio to 1. Five-fold cross-validation (CV) on the training set is employed among all combinations of alpha, degree, and lag values. The training dataset is split. randomly for each fold (as per sklearn's 's KFold function). First, additional features are temporarily appended to the data based on the degree, lag, trans_ stpp, interaction, and transform_ y hyperparameters. We developed a custom algorithm (Algorithm 2) to perform this feature expansion. Due to this dependency on the degree and lag byperparameters, this feature expansion occurs within the cross-validation process, creating temporary augmented dataset that is scaled to have mean O and standard deviation = 1 and then input to the LASSO method. For each hyperparameter combination and fold, a validation mean-squared error (MSE) is recorded. The values of degree and lag corresponding to the LASSO model with the lowest validation MSE (averaged across all five folds) are recorded, and a LASSO model using this combination of hyperparametors is fit using the training data to obtain scaled paramoters (estimated coefficients).
The next step in the LCEN algorithm is the clip (thresholding) step, in which the features whose scaled LASSO parameters have absolute values smaller than the cutoff hyperparameter can be removed from the expanded dalaset, and their coefficients are forced to 0. This step reduces the number of features to be considered, speeding up the algorithm and increasing the accuracy of the model predictions by removing irrelevant/less relevant features.
recorded that they
The EN step involves cross-valt validation on the training set among all combinations of alpha and 41ratioo using the values of degree and lag obtained in the LASSO step. Once again, the training dataset split randomly for each fold (as per sklearn's KFold function) and the features are expanded and scaled, then the features recorded in the Clip step are removed. For each hyperparameter combination and fold, validation MSE is recorded. The values of alpha and lf rraio corresponding to the EN model with the lowest validation MSF (averaged across all five folds) are recorded, and an EN model using this combination of hyperparameters is fit using the training data to obtain new scaled parameters.
A second clip (thresholding) step applied to these EN-scaled parameters, zeroing the coefficients of the features whose scaled EN parameters have absolute values smaller than the cutoff hyperparamcter. Lastly, some post-processing steps are performed. The scaled coefficients are unscaled by multiplying the scaled
31,CNN is approximately equivalent other LASSO-based methods in terms of sparsity.
3

---

Published in Transactions on Machine Learning Research (11/2025)
coefficients by the standard deviation of the y training data and dividing by the standard deviation of each corresponding X feature. Then, a dot product of the train test data and the unscaled coefficients is taken to obtain the final predictions. This procedure returns the trained EN model after the second clip step, which is interprctable and nonlincar, and the predictions made with the unscaled coefficients on the data.
Algorithm LASSO-Clip-EN (LCCEN)
Input: X and y data; lists of hyperparameters alpha, U1 ratio, degree, log; hyperparameters cutoff. trans type, interaction, transform_y
# LASSO step: filters features without requiring a combinatorially large number of potential hyperpa- rameters, as ralão is fixed. Determines the degree and lag hyperparameters for feature expansion. Temporarily set U ratio 1.
for each hyperparameter combination in (alpha degree log) do
Generate additional features based the Irans type, interaction, transform y, the current degree, and the current lag hyperparameters [Algorithm 2).
Temporarily append the new features to the X data for cross-validation.
Scale the data such that each feature's mean = 0 and its standard deviation = 1. Perform five-fold cross-validation with a user-selected CV algorithm and the LASSO method. For each fold, record the validation MSE for this hyperparameter combination.
end for
Obtain the combination of hyperparameters with the lowest average validation MSE from the above cross- validation. Record the best degree and lag hyperparameters.
Fit a LASSO model on the scaled training data with these hyperparameteers to obtain parameters. # Clip step: further reduces the number of features, speeding up the model and increasing sparsity. Record all features whose scaled parameters have absolute values < cutoff from the training and test data for removal during the EN training.
# EN step: trains the final model the features that passed the LASSO and Clip steps. EN is used for its machine learning performance and stability.
Restore !1 ralio to its original list of values.
for each hyperparameter combination in (alpha x 1 (Irriio) do
Generate additional features based on the trans type, interaction, transform the optimal degree, and the optimal lag hyperparameters [Algorithm 2].
Temporarily append the new features to the X data for cross-validation. Remove the features not selected by LASSO or recorded in the Clip step.
Scale the data such that cach feature's mcan 0 and its standard deviation = 1. Perform five-fold cross- cross-validatle validation with a user-selected CV algorithu and the EN method. For each fold, record the validation MSE for this hyperparameter combination.
end for
Obtain the combination of hyperparameters with the lowest average validation MSE from the above cross- validation. Record the best alpha and 11 ratio hyperparameters. Fit an EN model on the scaled training data with these hyperparameters to obtain parameters. # Clip step II
Remove all features whose scaled parameters have absolute values culoff # Post-processing: returns the model with coefficients, predictions, and metrics in hunan-rcadable way. Unscale the coefficients of the selected parameters based on the standard deviations of the data. Obtain train/tes test predictions by performing a dot product of the unscaled coefficients with the expanded train/test data containing only the selected features.
return the trained EN model, the predictions, and the evaluation metrics.
The LCEN algorithm (Algorithun 1) has five hyperparameters: alpha, which determines the regularization strength (as in the LASSO, EN, and similar algorithms): I1 Irraaoo, which determines how much of the regularization of the EN step depends on the 1-norm as opposed to the 2-norm (as in the EN algorithm): degree, which determines the maximum degree for the basis expansion of the data (Table A2); log, which determines the maximum number of previous time steps from which X and y features are included (relevant
4

---

Published in Transactions on Machine Learning Research (11/2025)
only for dynamic models); and cutoff. which determines the minimum value scaled parameter needs to have to not be eliminated during the clip steps. Details on the cross-validated hyperparameter values for all models are in Section A3. Three other hyperparameters are relevant to the expansion of features (Algorithm 2) but do not interact with the LCEN algorithm directly. The trans_type type hyperparameter controls what kinds of features are appended to the data. can be set to 'all' to include all transforms (see Table A2 for an example of what features are includod), 'poly' to include only polynomial and interaction terms, and "simple interaction' to include only interaction terms. The interaction hyperparameter a boolean that controls whether interaction terms are included in the feature expansion process. Finally, the transform hyperparameter, which is relevant only when lag > 0, is boolean that determines whether the y features from previous time steps will also be transformed based on trans type or only the raw values of y from previous time steps will be included.
LCEN scales similarly to the thresholded and regular EN. For an Nx dataset under F-fold cross-validation with A potential values, potential L₃ ratio values, and C potential cutoff values, LCEN scales as P2 FA.C). The degree hyperparameter increases the number of features P in a supralinear way. The lag hyperparameter increases the number of features P in a linear way. Conversely, higher values for the cutoff hyperparameter decrease the number of features P.
2.2 LCEN is optimal when compared to ablated and variant algorithms
The rationale behind this sequence of steps LASSO, then Clip, then EN, then second Clip - seeks to balance the algorithm's high accuracy and sparsity with low runtime. As shown by the ablation experiments (Section A4 and Table 2), other combinations/ ietettiionsvaiianttsss variianss do not achieve the same performance as LCEN. Specifically, as feature selection algorithms,, the LCEN, LASSO-Clip (LC, the thresholded LASSO), and LASSO-Cij Clip- (LCL) algorithms perform much more accurate nonlinear feature selection than LASSO-EN (L.EN) or the methods starting with EN. Moreover, the methods that start with the T.ASSO are considerably faster. As machine learning algorithms, the LASSO-Clip, EN-Clip (ENC, the thresholded EN), and LASSO-EN combinations tend to have lower accuracy than LCEN. The EN-Clip combination is also much slower and less sparse than LCEN, and the LASSO-EN combination is slower and slightly less sparse. The LSASOO combination is less accurate than LCEN, although it is slightly faster and sparser. The EN-Clip-EN (ENCEN) combination achieves similar accuracy, but is much slower and less sparse than LCEN. It is possible to add a debiasing step at the end by using ordinary least-squares (OLS) to estimate the coefficients of the features selected by LCEN (Belloni & Chernozhukov, 2013). This LCEN-OLS variant model estimates coefficients more accurately for one of the ablation experiments, but performs worse in terms of Lest-set MSE empirical datasets.
The combinations that start with EN are slower than LCEN because EN has greater number of hyperpa- rameter combinations to be tested, and these combinations are tested with higher number of features (as the full expanded feature set has not been subject to any selection via the LASSO and Clip steps). These combinations are also less sparse because the L1 and Le norms compete during EN regularization, and a combination that prioritizes the L2 norm may have a lower cross-validation MSE. Beginning with the LASSO increases the algorithm's is sparsity and speed at no accuracy cost.
The use of tard-thresholding (Clip) steps improves LCEN's accuracy and sparsity. The increase in sparsity also lowers the algorithm's runtime. The second Clip step is less impactful than the first and does not affect the algorithm's 's runtime, but it still improves LCEN's feature selection capabilities. We highlight that the Clip steps operate on the scaled parametors; thus, any issucs that may arise due to the (relative) magnitude of the coefficients are not significant.
2.3 Experimental setup
Multiple datasets (sunmarized in Table A1) are used to test the performance of the LCEN algorithm. These datasets can be divided into three categories: artificial data, empirical data from processes with known physical laws, and empirical data from processes with no known physical laws. Further description of these datasets and how the artificial datasets are generated is available in Section A2; the empirical datasets are also described in Section 3.2. All models tested in this work had their hyperparameters selected by
5

---

Published in Transactions on Machine Learning Research (11/2025)
5-fold cross-validation (CV) and this CV procedure was repeated for 3 different seeds so that the average + standard deviation of results can be reported, except for those trained on the "GFFCom 2014" dataset, which used time series cross-validation. For the "Diesel Freezing Point" dataset, an experiment comparing 10-fold CV with 5-fold CV was performed (Table A10). The separation between training and testing sets varied depending on the dataset and detailed in Section A2.
3 Results
3.1 In artificial datasets, LCEN provides high feature selection performance and robustness to noise and multicollinearity
LCEN is evaluated both as a feature selection algorithm (this section and Table 2) and as machine learning algorithm (rest of Section 3.2). The first datasets used to validate the LCEN algorithm are multiple linear datasets ("Artificial linear"). These datasets feature all combinations of (100, 500, 1000} samples x {100, 500, 1000} true features x (0%, 25%} noise level x (25%, 50%, 75%, 100%} additional false features. The noise level is defined as mcan(added noisc/noiscles x)x100% The added Gaussian noise has μ = and suitable a to reach the desired noise level. For each combination, 3 repeats with different random seeds were created. This experiment contains multiple challenging conditions, including cases where the number of features was much larger than the number of samples (P > N), cases with significant proportion of false features, and cases with high noise (low correlation between the input X and the output y).
The methods LASSO, EN, fastSparseGAMs (A-G) (Liu et al., 2022), SCAD, MCP, symbolic regression (SymReg) (implemented by Stephens et al. (2022)), thresholded EN, and LCEN were tested on this feature selection task. Overall, LCEN consistently performed similarly the thresholded EN and outperformed all other methods in this task, as measured by the Matthews Correlation Coefficient. (MCC) (Figs. A1-A6). Furthermore, LCEN was f.7-fold faster than the thresholded EN on these datasets, and 10.3-fold faster than the thresholded EN on average (Table A3). EN was typically completely unsclcctive, classifying all features as true except in a lew scenarios with N > P. SymReg performed marginally better, but still had a very low overall performance. LASSO, FS-GAMs, SCAD, and MCP performed better than EN and SymReg, having similar performances among themselves; notable exceptions were cases in which LASSO was completely unselective, and most scenarios with N > and 0% noise, which allowed SCAD and MCP to perform perfect classification. LCEN performed perfect classification in the scenarios with N P and 0% noise more frequently than SCAD and MCP, and surpassed the methods (other than thresholded EN) in the other scenarios in terms of absolute MCC by 19.8% on average when N = P and by 8.2% on average when N < P.
The first step of LCEN uses LASSO, which has been stated to underperform with multicollincar data (Heinze et al., 2018; Zou & Hastie, 2005). Therefore, lests using multicollinear data are performed next. The goal is to assess whether LCEN can determine the presence of two different but correlated variables more accurately than LASSO. Noisc 6, at different levels (as defined above), was added to the Xo variable to create correlated variable X1. A second noise E2, also at different levels, was added to the final y data. When 0, both variables are equal and separation is not possible. However, for other 61 values, LCEN successfully identifies that two relevant variables exist and assigns correct coefficients to them (Fig, A7). Specifically, when the noise level 61 associated with the X data (which indicates how different the X variables are, as highlighted by the variance inflation factors [VIFs] in Fig. A7) is greater than the noise level (2 associated with the y data, LCEN separates both variables with coefficient crrors S 5%. When both noise levels are similar, LCEN separates both variables with coefficient errors between 5% and 10%. The X data used in this experiment has very high multicollinearity (as shown by the VIFs); real data will typically have lower VIFs and thus be casier for LCEN to separate.
Next, a more complex equation is used to further validate LCEN. The "Relativistic energy" data contain mass and velocity values used to calculate E² = cmm² + c2m As before, datasets with increasing noise levels are created. The degree hyperparaneter is allowed to vary between 1 and 6 in this experiment. These degree values lead to expanded datasets with {8, 22, 42, 68, 100, 138) features respectively. Although there are only two true features, there are significant number of false features, many of which have similar functional
6

---

Published in Transactions on Machine Learning Research (11/2025)
forms to the true features. LCEN selected only relevant features for all noise levels tested ($ 20%). and the coefficients were typically equal to the ground truth (Table 1). The only major divergence happened at a noise level of 20%, as the coefficient for m² had a 25% relative error. This error led to our hypothesizing that it is challenging to distinguish among the features involving m (such as m, m², and m²) due to the low range of the data. Thus, another dataset with the same properties but a larger range of values for m was created. LCEN performed better on this dataset, again selecting only relevant features for all noise levels tested (S 30%) and having much lower errors in the estimated coefficients (Table 1). These experiments consistently show the robustness of LCEN and how the range of the data can affect predictions. To clarify our design choices and the relevance of each individual part of the L.CN algorithm, ablation tests are performed with this dataset in Section A4 of the Appendix.
Table 1: Coefficient values and corresponding relative error to the ground truth for the "Relativistic energy" datasct at different noise levels. The first coefficient is for m² and should be ct 8.078x 1033 m*/s /st; the second coefficient is for m²y² and should be 6² 8.988 100 m² m²/s². The left table is for the dataset with m < 10, and the right table is for the dataset with 1 S m < 100.
Noisc
Coefficients
Error (%)
Noise
Coefficients
Error (%)
0%
8.078x10* 8.987 x 10 16
0.001, 0.006
0%
8.077x10 8.987 1
0.013, 0.009
5%
8.078x1033 8.986x 10
0.005, 0.022
5%
8.081x1033
8.969x
0.043, 0.206
10%
8.078x 10s3,
8.984x10"
0.009, 0.038
10%
8.085x 1005
8.951 x
0.097, 0.410
15%
8.079 10
8.983x
0.013, 0.054
15%
8.089 1033, 8.935x10'6
0.139, 0.580
20%
8.079x 10 8.981x1016
0.017, 0.070
20%
6.027x1093
8.912x1016
25.39, 0.844
30%
8.080x1093 8.978x10
0.025, 0.103
Finally, LCEN is compared with the feature selection algorithm in ALVEN (Sun & Braatz, 2020), which uses the same basis function expansion, but uses F-tests for feature selection. The "4th-degree, univariate polynomial" dataset is created as per Sun & Braatz (2021), such that y X +0..x=+0..XX 0.005x 30 X points are available for training, and 1,000 X points are available for testing. These conditions simulate the scarcity of data potentially present in real datasets while ensuring test errors can be predicted with high confidence. Sun & Braatz (2021) created four types of ALVEN models for this prediction. On this same dataset and using the same four types of models, LCEN attained median errors that are typically over 60% smaller than those from ALVEN (Fig. 1). Discussion of these results are included in the Appendix (Section A5.3), including discussions on how LCEN consistently selected the correct degree hyperparameter via validatoi despite the low number of training samples and high noise (Fig. A9).
3.2 In empirical datasets, LCEN provides higher predictive performance than many other methods
The applicability of an algorithm to real-world problems is judged only by its performance on real data, as data sparsity or real noise may affect the algorithm's capabilities. In tests performed on empirical data generated by processes with known physical laws, LCEN is observed to display high feature selection per- formance, consistently selecting the right features with low coefficient relative errors even when the number of features is much higher than the number of samples (Table 2).
The first test of an empirical dataset from a process with a known physical law uses the "CARMENES star data" " dataset from Schweitzer et al. (2019). This dataset contains information on temperature (T), radius (R), and luminosity (L) of 293 white dwarf stars. These fcatures are linked together by the Stefan- Boltzmann equation, L = 4m R22 where is a constant. Normalizing this equation to values from another star (typically, the Sun), conveniently sets the constant terms to 1. This normalization is applied to the "CARMENES star data" dataset. LCEN with degrees from 1 to 10 was applied to this normalized dataset. Despite the very large number of potential features (due to the high degree values used), LCEN correctly selected only the RºTM feature (Table 2, third column). The coefficient assigned to R²T is 1.0008, which is well within the 3% empirical error on these data (as reported by Schweitzer et al. (2019)).
A potential limitation in real datasets is data scarcity. To evaluate the LCEN algorithm in a low-data scenario, the "Kepler's 3rd Law" dataset is created from the original data obtained by Kepler, first published
7

---

Published in Transactions on Machine Learning Research (11/2025)
0.4
1.0
MSE median
0.8
Median test MSE
0.3
0.6
0.2
unbiased model
0.4
biased model
Unbiased Model
0.2
CV
0.1
Biased Model
CV limited order
CV Model
0.0
CV Limited Model
0.0
50
100
150
200
50
100
150
200
noise level
Noise variance o²
Figure 1: Test set median MSE for the "4th-degree, univariate polynomial" dataset. ALVEN results (left, reproduced from Sun & Braatz (2021) with permission) show that the error is monotonically increasing with noise and that the degree 4 "unbiased model" is the best at low noise levels, but is displaced by the degree 2 "biased model" at higher noise levels. On the other hand, LCEN results (right) show that the median errors converge at higher noises. Furthermore, the LCEN median errors are typically over 60% smaller than the ALVEN median errors, and the deyree "unbiased model" always the best model no maller the noise. The "noise level" and "Noise variance o2n terms are equivalent this figure. Fig. A8 contains interquartile rangos for the LCEN model's test MSEs.
in 1619 and republished in Kepler et al. (1997). From only 6 (slightly inaccurate) measurements, Kepler able to derive the eponymous Kepler's 3rd law, which states that the period 7 of a celestial body is related to the semi-major axis of its orbit by T ka3/2, The constant k depends the masses of the central and orbiting bodies; however, as the mass of the central body is typically much larger, the mass of the orbiting body is ignored. In this and Kepler's works, 7 measured in Farth days, so the constant is R365.25 when using modern data and 15 when using Kepler's original data. Again, LCEN with degrees from 1 to was used to model this dataset. Despite the low number of data points, LCEN correctly selected only the 2/2 feature (Table 2, sixth column). Moreover, the coefficient assigned that feature was 366.82, an error of only 0.46% relative to Kepler's ' & 365.15.
As mentioned in Section 2.2, the LCEN, LC, and LCL methods performed much more accurate nonlinear feature selection than LEN or the methods starting with EN (Table 2, third and sixth columns). Moreover, the methods that start with the LASSO are considcrably faster, a difference clcarly visible with the larger "CARMENES star data" dataset (Table 2, fourth column). Although LC performed perfect feature selection, the coefficients for the true feature selected were significantly distorted, especially the "CARMENES star data" dataset. Only LCEN and LCL consistently selected only the correct features with low coefficient errors in these experiments (Table 2, second and third columns).
Table 2: Average (+: standard deviation across CV seeds) results for empirical datasets from processes with known physical laws. True fcature %RE refers to the percent relative error of the coefficient. of the truc features.
CARMENES star data (293x3500
Kepler's 3rd Law (6x130)
Model LCEN LCC
True feature %RE
MCC (%)
Runtime (s)
True feature %RE
MCC (%)
Runtime (s)
0.08+0.00
100+0
6.01
0.46+0.00
100+0
3.56
78.3++0.00
100+0
5.63
3.83++0.31
100+-0
3.12
ENC
85.3422.7
34.1+2.6
55.4
72.140.38
31.343.5
13.1
LEN
79.3+2.77
16.2+0.4
6.87
6.59+4.12
59.5+6.2
3.50
LCL
0.08+0000
10040
5.73
0.46+0.00
100+0
3.17
ENCEN
67.5++2.64
57.4+00
54.6
79.1999.9
31.3+3.5
13.6
8

---

Published in Transactions on Machine Learning Research (11/2025)
The final experiments to compares the performance of LCEN to other algorithms real datascts from processes with unknown physical laws. As there is no (computational) way to validate the feature selection done by models trained on these datasets, this section focuses on investigating the capabilities of LCEN as a machine lcarning algorithm by comparing the prodiction error and sparsity of different models. Thus, despite the fact that feature selection metrics such as discovery rates or MCCs would be of interest, it is impossible to obtaín those metrics for these empirical datasets. Ordinary least squares (OLS), ridge regression (RR) (Tikhonov, 1963), partial least squares (PLS) (Wold, 1975a;b), LASSO, elastic net (EN) (Zou & Hastie, 2005), SCAD (Fan & Li, 2001), MCP (Zhang, 2010), random forest (RF) (Ho, 1995), gradient-booster decision trees (GBDT) (Friedman, 2001), adaptive boosting (AdaB) (Freund & Schapire, 1997), support vector machine with radial basis functions (SVM) (Boser et al., 1992), fastSpaaseGAAMs (FS-GAMs) (Liu et al., 2022), multilayer perceptron (MLP), MLP with group LASSO (MLP-GL,) (Scardapane el al., 2017), and LassoNet (Lemhadri et al., 2021) were compared with LCEN. To clarify our design choices and the relevance of each individual part of the LCEN algorithm, ablation tests are performed with many of the datasets tested here in Section A4 of the Appendix.
The first dataset analyzed is the "Diesel Freezing Point" dataset (Hutzler & Westbrook, 2000), which is comprised of 395 diesel spectra mcasured at 401 wavelengths and used to predict the freezing point of these diesels. The dense, nonlinear method SVM had the best prediction performance, with a test RMSE of C (Table 3). The next best in performance were LCEN and EN, which had test set RMSEs equal to 4.89ºC. The sparsest methods were LCEN, which selected only 37/401 features (9.2%) on average yet had an average prediction error only 3.6% higher than that of the best dense method, LASSO, which performed similarly to LCEN, SCAD, which selected only 22/401 features (5.5%) on average but had an average prediction error 11% higher than that of the best dense method, and FS-GAMs, which selected only 2/401 features (0.5%) but had a prediction error 76.3% higher than that of the best dense method. LCEN, FS-GAMs, and SVM were the only nonlinear methods that had a runtime faster than 10 seconds, comparable to the linear methods PLS and RR (Table 3). LCEN and LASSO are the only methods that simultancously had a low test RMSE, interpretability, and a fast runtime. However, LASSO has a slightly worse RMSE and less-stable sparsity, as most of its seeds selected more features than those selected in any of the LCEN seeds (Table 3, third column).
Table 3: Average (+ standard deviation across 3 CV sceds) test results for different modcls for the "Diesel Freezing Point" dataset.
Model
Test. RMSE (CC)
Features
Runtime (s)
OLS
11.75
401
0.09
PLS
5.21+0.00
401-0
7.30+00066
RR
4.90+0.07
401+0
6.75++0.02
EN
4.89
280+209
20.99-10558
LASSO
4.95+0.10
33+10
4.52+00009
SCAD
5.2611.11
22
5.9
MCP
5.22+0.07
30+10
33.9+2.8
FS-GAMs
8.32+0.00
210
3.75
RF
5.10+
393+F7
471+5
GBDT
5.2140.18
383+25
2,870=32
AdaB
5.17400.0
304+0
23.8
SVM
4.72+0.33
40110
0.74
MLP
0.95+
401+0
3,839=60
MLP-GL 4.98
40110
10,314
LassoNet
9.88+-0.06
401+0
47,679
LCEN
4.89+00.06
37:1
6.59+0.663
We also consider a different scenario: that an end user could prioritize creating very sparse models, even at the expense of incrcasing the MSE. To simulate this scenario, the LCEN cutoff hyperparameter was increased from the value that minimizes the validation MSE create sparser models. These models have much fewer features, yet their test set RMSFs typically increase by only small amount (Table A9). This

---

Published in Transactions on Machine Learning Research (11/2025)
experiment illustrates how sparsity can be prioritized by the end user to make models with high predictive power while retaining the most critical features.
Finally, we train models on the same dataset but using 10-fold CV instead of 5-fold CV to determine whether there are any performance differences when using more folds. The ratio between test-set RMSEs with 10-fold CV and 5-fold CV equals 1.018+00009 (Table A10), a value that is indistinguishable from 1, indicating that there is no benefit to using 10-fold CV in this experiment. Furthermore, the use of 10-fold CV increased runtimes by factor of 1.856 on average.
LCEN is then tested on the "Concrete Compressive Strength" dataset (Yeh, 1998), which contains the composition and age of 1,030 different types of concrete and their compressive strengths. The relationship between these properties is noulinear, and past modeling efforts include algebraic expressions and artificial neural networks (specifically, MLPs)* (Yeh, 1998; 2006). LCEN's test RMSE is considerably lower than that of the published algebraic models (Table 4, top rows), and nearly the same as for the MLPs (within the noise level). RF has the lowest Lest RMSE, which is lower by only 9.3%. However, LCEN has the advantage of being interpretable and having the lowest validation RMSE out of all methods tested.
Table 4: Average test RMSE (+ standard deviation across 3 CV seeds) for different models for the "Concrete Compressive Strength" dataset. All machine- learning models selected all 8 features except for FS-GAMs, which selected 4.
Model
Test RMSE (MPa)
A Algebraic expression (Yeh, 1998) Linear + interactions model (Yeh, 2006)
7.79
7.43
OLS
10.26
PLS = RR = EN LASSO
10.26+0.00
SCAD = MCP
RF
5.08:10.02
GBDT
5.97
AdaB
6.95++0.00
SVM
6.07+0.19
MLP
5.47+0.08
MLP-GL,
5.47=00.09
LassoNet
16.39110.17
FS-GAMs
11.39+00.00
LCEN
5.55++0.04
To assess the performance of LCEN for dataset caused by human activity instead of physical law, consider the modified "Boston housing" dataset. This dataset contains the median value of owner-occupied houses and many internal and external measurements, such as the per-capita crime rate of the region, the average number of rooms, and the concentration of nitric oxides in the area (Harrison & Rubinfeld, 1978). We modified this dataset to detransform the B variable into its raw value; samples in which this detransformation led to multiple possible values were discarded. In this modified "Boston housing" dataset, the linear models tended to perform very similarly to each other but quite poorly (Table 5). RF and SVM performed relatively well, whereas LassoNet and FS-GAMs had the worst performance. A dense MLP had the lowest test RMSE, but MLP-GL, was slightly worse. LCEN had a very high performance on this regression task, reaching test RMSE only 5.5% higher than that of the dense MLP and 2.0% lower than that of the MLP-GL1. LCEN also had the lowest validation RMSE, which was 12% lower than that of the dense MLP. As with the other datasets, LCFN attained higher performance than many other methods while also being completely interpretable.
Finally, the "GEFCom 2014" dataset was used to evaluate the abilities of LCEN in a complex and dynamic task (Hong et al., 2016). Two versions of the "GEFCom 2014" dataset have been published: one that contains
Noo type of validation mentioned in Yeh (1998), so the test and validation sets likely the same, making its MLP results overoptimistic. Corrohorating this hypothesis, the MLP and MI.P-GT, models trained in this work have higher test-set MSFs
10

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Average test RMSE (= standard deviation across CV seeds) for different models for the "Boston housing" dataset.
Model
Test RMSE
(Thousands USD)
OLS
6.38
PLS
6.50:10.19
RR EN
6.4220.03
LASSO
6.38++0000
SCAD
6.37+0.01
MCP
6.37
RF
5.09+0.13
GBDT
6.4210.21
AdaB
5.67
SVM
5.05+0.07
MLP
4.69
MLP-GL1
5.05+0.32
LassoNet
9.93++0.02
FS-GAMs
7.32+0.00
LCEN
4.95
only energy consumption levels and another that contains the same energy consumption data and also temperature data from multiple weather stations. This work uses the former. "GEFCom 2014" is part of an energy forecasting competition won by LASSO-like model (Hong et al., 2016). More recently, deep learning has been applied to this problem (Wilms et al., 2018; Gasparin et al., 2022), which produced models with strong 24-hour predictive performance (Gasparin el al., 2022). Despite the strong performance of multiple, complex ANN architectures, LCEN models obtain a 13.1% lower test RMSE on this forecasting task than the ttata-of-tho-art Seq2Seq model from Gasparin al. (2022) (Table 6, sixth and seventh columns). Unlike the ANNs, LCEN requires only a CPU for training and forecasting, and provides interpretable coefficients. LCEN also be used for longer forecasts without significant. increases in the prediction error, demonstrating high robustness of the algorithm.
Table Test RMSE and relative error for different models for the "GEFCom 2014" dataset. The deep learning models (TCN to Seq2Seq) and their results from Table 8 of Gasparin al. (2022). Confidence intervals for LCEN all +0 due to LCEN's high resistance variation due to model initialization changes and the fact that changes in CV seed are not possible when using time series CV. A similar phenomenon occurs with the ARIMA models of Gasparin et al. (2022), and with LASSO /R////R///////R PN/CN models in general.
Model
TCN 24
RNN
LSTM 24
GRU
Seq2Seq 24
LCEN
Hours Forecast
24
24
24
48
72
120
168
Test RMSE (MW)
17.2+0.1
18.0#0.3
19.5+0.5
19.0+0.2
17.1
14.9
18.9
21.0
23.4
24.7
Relative Error (%)
9.8-+0.06
10.240.2
11.140.33
10.8
9.7
8.5
10.7
11.9
13.2
13.9
4 Discussion
This work introduces LASSO-Clip-EN (LCEN), nonlinear, interpretable feature selection and machine learning algorithm (Algorithm 1). LCEN was first validated using artificial data (Section 3.1), which provide an initial assessment of the algorithm's performance under multiple, independently controllable conditions. LCEN was then lested with data from processes with known physical laws (Section 3.2) and without known physical laws (Sections 3.2 and A6.1).
Overall, these experiments have demonstratod the applicability of LCEN a multitude of scientific and nonscientific problems, those with significant nonlinearities and complexity. On the real data from
11

---

Published in Transactions on Machine Learning Research (11/2025)
processes with known physical laws, LCEN successfully selected only the correct features with very low coefficient errors for all datasets used in this work, effectively rediscovering physical laws solely from data (Table 2). LCEN models were robust to defects in the real data, including noise, multicollinearity, or sample scarcity. LCEN models were typically as accurate as or more accurate than many alternative methods Tables 3 6-6); yet. were also much sparser (Table 3). LCEN models are also easy to interpret, as they display exactly how each input contributes to the final output. This combination of accuracy and interpretability essential for the deployment of machhnn--eerrin Moreover, the additional interpretability models robust to changes in data or adversarial input. LCEN is free, open-source, and easy to use, allowing even non-specialists in machine learning to benefit from and use it. Its main limitations are that (1) LCEN is not & universal function approxinator, as it can model only the functions present in the expansion of dataset features, (2) its feature expansion algorithm better suited to numerical data over image or text data, and (3) LCEN not always as accurate as a dense deep learning method. If enough compute and time available for model training, users in scenarios that focus on accuracy above anything else with numemmillal data types may prefer to use deep learning method.
models in performance--ritical scenarios, from aviation medicine.
assist in data model refinement efforts and can make the
There are two clear future directions for this work. The first involves evaluating LCEN in classification tasks, as many important problems in science and engineering involve classification. The second involves applying LCEN to automatically generate physical equations for hybrid model architectures (such as physics- constrained or physics-guided ML), which have high potential for scientific applications (Peng et al., 2021; Willard et al., 2022).
Acknowledgments
This work was partially supported by a Project Award Agreement from the National Institute for Innovation in Manufacturing Biopharmaceuticals (NIIMBL) from the U.S. Department of Commerce, National Institute of Standards and Technology [70NANB17H002, 70NANB21H0S6], P.S. was supported by MathWorks Engineering Fellowship.
12

---

Published in Transactions on Machine Learning Research (11/2025)
References
H. Akaiko. A new look at the statistical model identification. IEEE Transactions Automatic Control, 19 (6):716 723, 1974. URL https://doi. org/ 1110//TTC... wtt.1.94....0000
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 444 453, 2019. URL httpps:/pproceedingg iaaiingammmr....... mlr. wtt.paiw........../eeeeeee wwt..ppese//aaaiinnneee
html.
Alexandre Belloni and Victor Chernozhukov. Least squares after model selection in high-dimensional sparse models. Bernoulli, 19(2):521 547, 2013. URL https:///bbi.org/11...3 0ttp://////00111111111111111
Dimitris Bertsimas and Bart Van Parys. Sparse high-dimensional regression: Exact scalable algorithms and phase transitions. The Annals of Statistics, 48(1):300 - 323, 2020. URL https://doi. org/10. org/10.1214/
A0S8008
Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144-152, org/10. 10. 11111..111..11 1145/130385, 130401.
1992. URL https://dol: http::://dooorr/1//////
Patrick T. Brewick, Sami F. Masri, Biagio Carboni, and Walter Lacarbonara. Enabling reduced-order data- driven nonlincar identification and modeling through naive clastic net regularization. International Journal of Lon-Lr Mechanics, 94:46 58, 2017. URL https://doo org. 1016/j ijnonl immec. 201.mm..............m 2017.01 01. 016.
Sourav Chatterjce, Trevor Hastie, and Robert Tibshirani. Univariate-guided sparse regression, 2025. URL hwtprivv.org/bb//5511111 18360.
D. Clark, Z. Schreter, and A. Adams. A quantitative comparison of dystal and backpropagation. In Pro- ceedingg of the Seventh Australian Conference on Newral Networks, pp. 132 137, 1996. URL https: //www. tib. wub..eu/dee htt////cccheee//////////////////
Arnak S. Dalalyan, Mohamed Hebiri, and Johannes Lederer. On the prediction performance of the Lasso. Bernoulli, 23(1):552 581, 2017. URL
Christine De Mol, Sofia Mosci, Magali Traskine, and Alessandro Verri. A regularized method for selecting nested groups of relevant genes from microarray data. Journal of Computational Biology, 16(5):677-600) 2009. URL https: //to///oo.ooo// org/10. 10. 1089/0088333 2008 0171.
Vu C Dinh and Lam S Ho. Consistent feature selection for analytic deep neural networks. In Advances in
Newral Information Processing Systems, volume 33, pp. 2420 2431, 2020. URL https: neurips.cc/ httplpp.cccpppppee http:///aaa////////////////a/aaa
Paper pdf.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties.
Journal of the American Statistical Association, 96(456): 1360, 2001. URL https: //doi. http://do..orr....///
1198/01621450175382277
Yoav Freund and Robert E Schapire. A -heioreti generalization of on line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-133, 1997. URL /ttpp:://di... org/
10. 1006/ css. 1997.999... 1504.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, (9(5):1189-12333 2001. URL https: //doi. http:///:oooog...///// 1214//1/// /o/////////1/11111.
Alberto Gasparin, Slobodan Lukovic, and Cesare Alippi. Deep learning for time series forecasting: The electric load case. CAAT Transactions on Intelligence Technology, 7(1):1-25, 2022. doi: https://doii.or
10.1049/cit2.12000 URL https://ietresearce wtt.e/aaarrr..ooonelaieaaaraeeee onlinelibrary. wwley..l............ com//oi//bbs/0000099///oo 12060.
13

---

Published in Transactions on Machine Learning Research (11/2025)
D Jr Harrison and D L Rubinfeld. Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manage., 5:81-102, 1978. URL doi. org/o00 1ttp///..1111000----000 (78)90006-2.
Mohamed Hebiri and Johannes Lederer. How correlations influence lasso prediction. IEEE Transactions on Information Theory, 59(3):1846 1854, 2013. URL https://dboi.org/100/ 110.:/1..111/IIII11 2012. 2227680.
Georg Heinze, Christine Wallisch, and Daniela Dunkler. Variable selection A review and recommendations for the practicing statistician. Biometrical Journal, 60(3):431 449, 2018. URL https://doi.org org/10..... 1002/bimj. 201700067
Tin Kam Ho. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognilion, volume 1, pp.
1995. URL doi. org/ 10000.109///1 1109/ICDAR, 100DDR..1999.000 598994.
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. Human factors model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human- Human-Computer Interaction, 4(68):1 26, May 2020. URL https:///do
org/ http//1..111/3338 1195/8392888
Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J. Hyndman. Prob- abilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Journal of Forecasting, 32(3):896 913, 2016. doi: hutp:///bo.orr//00000////foeeeee
2.0. URL
Scott A. Hutzler and S. R. Westbrook. Estimating chemical and bulk properties of middle distillate fuels from near-infrared spectra. Technical report, Defense Technical Information Center, U.S. Army TARDEC, Warren, Michigan, 2000. URL https://apps dtic. mil/sti/citatiintto ADA394209. Report TFLRF No. 348.
J. Kepler, F. J. Aiton, A. M. Duncan, and J. V. Field. The Harmony of the World, pp. 418, 422. American Philosophical Society, 1997. URL /oogg//////oooooogoleeeeee com/bboksSTd-TRRLAAATAA
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10:1096, 3 2019. doi: 10.103//s41467-009 08987-4
Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. LassoNet: Neural networks with feature sparsity. Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 10-18, 2021. URL https::/procceednngg mlr. wwt.sss/13//1emmnaaaaerooooo
html.
Jiachang Liu, Chudi Zhong, Margo Seltzer, and Cynthia Rudin. Fast sparse classification for generalized linear and additive models. In Gustau Camps- Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151, pp. html.
9304 9333, 2022. URL https: http:p////cccceiing..om mirr ppwss//151/liu///// http://ceeeennn...rrreeeeaa/aa//
Trent McConaghy. FFX: Fast, Scalable, Deterministic Symbolic Regression Technology, pp. 235-20 260. Springer, New York, 2011. URL attps://don 100
Nicolai Meinshauscn and Bin Yu. Lasso-type recovery of sparse representations for high-dimensional data. The Annals of Statistics, 37(1):246 270, 2009. URL http::///oi..org///..2200/000000
Ali Mirzaci, Vahid Pourahmadi, Mchran Soltani, and Hamid Sheikhzadeh. Deep feature selection using a teacher-student network. Newrocompuling, 383:396 2020. doi: http:///di..gg///0.000///eeeen
12.0.1..017. URL https://wwet www.:/eceieeneiretc..mm wtt.//ooo/cccceeeeeeeeeeeeeceece wtt.://www..cc.......cccaaaaaaaa
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. URL https: /ttps:://oo...o org/ 100g/ 24432/C55C7W.
Menelaos Pavlou, Gareth Ambler, Shaun Scaman, Maria De Torio, and Rumana Z. Omar. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Statistics in Medicine, 35()):1159-11777 2016. URL https:////// w.t......//////............ 1002/sim...mmmmmm 6782.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Grace C. Y. Peng, Mark Alber, Adrian Buganza Tepole, William R. Cannon, Suvranu De, Savador Dura- Bernal, Krishna Garikipati, George Karniadakis, William W. Lytton, Paris Perdikaris, Linda Petzold, and Ellen Kuhl. Multiscale modeling meets machine learning: What can we learn? Archives of Computational Methods in Engincering, 28:1017-1037, 2021. URL doi. www..org/.....
Pradeep Ravikumar, John Lafferty, Han Liu, and Larry Wasserman. Sparse Additive Models. Journal of the Royal Statistical Sociely Series B: Statistical Methodology, 71(5):1009-1030 2009. URL https: //doi. wtt.:o/.//./....111 1-11--46788688 2009000008 00718.0..
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. In ICML Workshop Human Interpretability in Machine Learning, pp. 91-95, 2016. URL https://arxiv http:////axxxvvo.vaaabee org/abs//////eeeoooooo.ooee a/6///6////6....//// 05386.
Julia Rosenzweig, Joachim Sicking, Schastian Houben, Michael Mock, and Maram Akila. Patch shortcuts: In- terpretable proxy models efficiently find black-box vulnerabilities. In IEEE/CV /CVF Conference on Computer Vision and Paltern Recognition Workshops, pp. 56 56-65, 2021. doi: 10.1 10.1109/CXPRW553000 /CC/////09. 200.:.0022220000
Cyuthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use in- terprctable models instcad. Nature Machine Intelligence, 215, 2019. URL https://doi.co
1038/ 019-0048-x
Fadil Santosa and William W. Symes. Linear inversion of band-limited reflection seismograms.. STAM Jowrnal on Scientific and Statisticnl Computing, 7(4):1307-1330, 1986. URL htt.t:h/tp/o..io 1ttt://.00//.1111/09000/00000000 1137/0907087,
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Newrocomputing, 241:81-89, June 2017. URL http://åx.doe http://t..d...o//11..0.00.. 2tttco...0...2.....
Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics, ((2):461-46) 1978. URL https://doi.org 10. 1....2114//0 /2s4/aos/////// 1176344136.
A. Schweitzer, V. M. Passegger, C. Cifuentes, V. J. Béjar, M. CortésCContreras J. A. Caballero, C. del Burgo, S. Czesla, M. Kürster, D. Montes, M. R. Zapatero Osorio, I. Ribas, A. Reiners, A. Quirrenbach, P. J. Amado, J. Aceituno, G. Anglada-Escudee
F. F. Bauer, S. Dreizler, S. V. Jeffers, E. W. Guenther, T. Henning, A. Kaminski, M. Lafarga, E. Marfil, J. C. Morales, J. H. M. M. Schmitt, W. Scifert, E. Solano, H. M. Tabernero, and M. Zechmeister. The CARMENES search for exoplanels around M dwarfs. Different roads to radii and masses of the target stars. Astron. Astrophys., 625:A68, May 2019. URL https: //doi. org/10.0000 10. 105110004--6611/113333333
Gary Smith. Step away from stepwise. Journal of Big Data, 5:32, 2018 URL https:///oi.org:: org/10.118// s40537-018-0143-6.
Trevor Stephens et al. Genetic programming in Python with a scikit-learn inspired APT: gplearn, 2022. URL httpps//gttaabb.c//reeeeerssssee wwt.c//eeeerrrspppeseaggll//ea/e egplearn.
Weike Sun and Richard D. Braatz. ALVEN: Algebraic learning via elastic net for static and dynamic nonlinear model identification. Computers & Chemical Engineering, 143:107103, 2020. URL https:
///doi.org/// 1tt.li.oor/00.111/////10 compchemeng 2020 107103.
Weike Sun and Richard D. Braatz. Smart process analytics for predictive modeling. Computers & Chemical Engineering, 144:107134, 2021. URL /ttp:://oooo..r//0011116.///...
Yiyang Sun, Zhi Chen, Vittorio Orlandi, Tong Wang, and Cynthia Rudin. Sparse and faithful explanations without sparse models. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 2071-2079. PMLR, May 2024. URL https: attps://proceedings. mlr .ar.ppess// v238/ sun24b. html.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Shaonan Tian, Yan Yu, and Hui Guo. Variable selection and corporate bankruptcy forecasts. Journal of j in. 2014. 11.1..2.000 103
Banking & Finance, 52:89-100, 2015. URL
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society:
Series B (Methodologieal), 58(1):267-255)) 288, 1996. URL https://doi.org/10.. 11110/11111j............ 251776616...999.
tb02080.2x x.
N. Tikhonov. Solution of incorrectly formulated problems and the regularization method. Doklady Akademii Nauk SSSR, 4:1035-1038, 1963.
Sara van de Geer, Peter Bühlmann, and Shuheng Zhou. The adaptive and the thresholded Lasso for poten- tially misspecified models (and a lower bound for the Lasso). Electronic Journal of Slatislics, 5:688 2011. URL https: http://oo..org///////// 1214/11-EJS624
749,
Jian Wang, Huaqing Zhang, Junze Wang, Yifei Pu, and Nikhil R. Pal. Feature selection using neural net- work with group Lasso regularization and controlled redundancy. IEEE Transactions on Newral Networks and Learning Systems, 32:1110 1123, 2021. URL doi.org/11. 10. 1109/NNLS.. 2980383.
S. Waugh. Extending and Benchmarking Cascade-Correlatan Extensions to the Cascade- Correlation Archi- tecture and Benchmarking of Feed-forwart Supervised Artificial Newral Networks. PhD thesis, University of Tasmania, 1995. URL https: / /tpp/ www.sseeeacsnntoooeeeeeeoe
org/CorpusID 53803349.
Mark J. Whittingham, Philip A. Stephens, Richard B. Bradbury, and Robert P. Freckleton. Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(6):1182-1188, 2006.
URL attps://dol org/10.1111/j_ 1111/j.a1i../. 1365-2656..... 2006 01141.x.
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engincering and environmental systems. ACM Comput. Surv., 55(4): 66, Nov 2022. URL https: cttps://doi.ogg org/ 1145/3514228.
Henning Wilms, Marco Cupelli, and Antonello Monti. Combining anto-regression with exogenous variables in sequence-to-sseuuen sequeene recurrent neural networks for short-term load forecasting. In IEEE 16th International Conference on Industrial Informatics, pp. 673-679, 2018. URL doi.org/10 10. 1119/INDDII 2018 8471953.
Herman Wold. 11 - Path models with latent variables: The NIPALS approach. In H. M. Blalock, A. Agan- begian, F. M. Borodkin, Raymond Boudon, and Vittorio Capecchi (eds.), Quantilative Sociology, pp. 307-357 Academic Press, New York, 1975a. URL https://doi. org/10 10. 1-1--998---11---00-1- 50017-4.
Herman Wold. Soft modelling by latent variables: The non-linear iterative partial least squares (NI- PALS) approach. Journal of Applied Probobility, 12(S1):117 142, 1975b. URL doi. ott.//..00007 $0021900200047604.
Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Josh Tenenbaum, and Charles Sutton. A Bayesian-symbolic approach to reasoning and learning in intuitive physics. In Advances in Newral Information Processing Systems, volume 34, pp. 2478 2490, 2021. URL https: http:s//////oeeengg.... neurips.. cc/paper
haper.pdft
Makoto Yamada, Jiliang Tang, Jose Lugo-Martinez, Ermin Hodzic, Raunak Shrestha, Avishek Saha, Hua Ouyang, Dawei Yin, Hiroshi Mamilsuka, Cenk Sahinalp, Predrag Radivojac, Filippo Menczer, and Yi Chang. Ultra high-dimensional nonlinear feature selection for big biological data. IEEE' Transactions on Knowledge and Data Engineering, (177):3522-138) 2018. URL doi. oog/orrg10... 1109/TKDE.. 2018 2789451.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 10648 10659, 2020. URL
mlr. wttp:essww........e.eeeeeeeemmee 1ttsss//119/yymaa2200. html..
I.-C Yeh. Modeling of strength of high-peeformance conerete using artificial neural networks. Cement and Concrete Research, 28(12):1797-1800 1998. URL btttp::/ddo.orrg/// 1011//000000010000 884
16

---

Published in Transactions on Machine Learning Research (11/2025)
I-Cheng Yeh. Analysis of strength of concrete using design of experiments and neural networks. Jour- nal of Malerials in Civil Engineering, 18(4):597-60) -04, 2006. URL https://dooi http:////oo..o/111110/610 0899-1561 18:
T-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 2007. URL https:
//doi.org/10 200ddd.org/0.22432//5557777
Cun-Hui Zhang, Nearly unbiased variable selection under minimax concave penalty. The Anmals of Statistics, 3842):894 2010. URL https://dol .rg. 1214/09-A00777 AOS729.
Lei Zhao, Qinghua Hu, and Wenwu Wang. Heterogeneous feature selection with multi-modal deep neural networks and sparse group LASSO. JEEE Transaclions on Mullimedio, 17(11):1936 1948, 2015. URL
doi. doi.org/10. 1109/TMM. 2015. 2477058.
Shuheng Zhou. Thresholding procedures for high dimensional variable selection and statistical estimation. In Advances in Neural Information Processing Systems, volume 22, 2009. URL https://proceeddng"" http:////......eeeeeeeceoooeeee.
neurips..
.df.
Shuheng Zhou. Thresholded Lasso for high dimensional variable selection and statistical estimation, 2010. URL
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-32)) 2005. URL https://doi.ors org/ 10.
1416--9616 00503.x.
Hui Zou and Hao Helen Zhang. On the adaptive elastic-net with a diverging mumber of parameters. The Annals of Statistics, 374):1733 1751, 2009, URL https://doi. orgo/..org/1... www..///......./////../ 1011408-A00622
17

---

Published in Transactions on Machine Learning Research (11/2025)
A1
Appendix - Feature Expansion Algorithm
Algorithm 2 Feature expansion for LCEN
Input: X and y data; hyperparameters degree, lag, trans_type, interaction, transform if lag > O then
Append X data from the previous lag time steps (samples) to each time step. transform_ == True then
Append y data from the previous lag time steps (samples) to each time step.
end if
Discard the first lag time steps. end if
Using sklearn's PolynomialBeatures function, generate polynomial (and interaction if the hyperparameter interaction is True) transforms of the X data for the given degree.
if trans_ type = 'all' then
Generate logarithm transforms.
Generate square root transforms for the features with all values >> 0. Generate inverse [1/Xs] transforms. if degree then
Generate transforms with noninteger degrees [(xx) (Xx))N+1/2 for integers N such that |N] < degree] for the features whose every sample is > 0. Generate the log-inverse transforms (xx))//NX,XX, M for natural numbers N and M such that N + M. < degree].
Generate the log-sqrt-inverse transforms Xx) (Xx)) for natural numbers N and M such that N + M < degree- 1] for the features with all values >> 0. end
end if
return the transformed X features
A2 Appendix - Description of datasets used in this work
Three types of data are used in this work: artificial data Arttificaal Linear", "Multicollinear data", "Rela- tivistic energy", and "4th-degree, univariate polynomial"), empirical data from processes with known physical laws ["CARMENES star data" and "Kepler's 3rd Law"), and empirical data from processes with no known physical laws [ Diesel Freezing Point", " "Abalone", "Concrete Compressive Strength", "Boston housing", and "GEFCom 2014"]. The artificial data are generated by us as described in the next paragraph. These artificial data are used for an assessment of the feature selection capabilities of the LCEN algorithm and to invest igate how properties of the data, such as noise or data range, affect its capabilities. Empirical data from processes with known physical laws are described in Section 3.2 and used to verify whether the LCEN algorithm rediscover known physical laws using data with real properties. Empirical data from processes with no known physical laws are described in Sections 3.2 and A6.1, and used to compare the machine learning performance of the LCEN algorithm against other linear and nonlinear methods, including deep learning models.
The " "Artificial Linear" datasets were created by drawing numbers from a uniform distribution between -10 and 10 in intervals of 0.1 for the samples X and coefficients k and summing to generate the outputs sumples K, These datasets feature all combinations of {100, 500, 1000} samples x {100, 500, 1000} true features x {0%, 25%} noise level x {25%, 50%, 75%, 100%} additional false features. The noise level is defined as mean mean(added noise/noiseles /noiseless x0010 The "Multicollinear data" dataset created by drawing numbers from uniform distribution between 1 and 10 to create one variable Xo. which was used together with a small amount of noise to create correlated variable X, = +e1;;; finally, they were summed such that y = 2X + 2X, + 62. The "Relativistic energy" dataset was created by drawing numbers from a uniform distribution between and 10 or 1 and 100 for masscs, and 5x107 and 2.5x 10 for velocities, which represent the energy of body as E² d'm + um? With these velocity numbers, relativistic effects are responsible for 20.4% of the total squared energy on average. The 44th-degree, univariate polynomial" dataset
y = -E _
18

---

Published in Transactions on Machine Learning Research (11/2025)
created by drawing numbers from normal distribution with mean 0 and variance 5 and transforming X + 0.5X² + 0.1 + 0.05X* +..
them into the polynomial
All models tested in this work had their hyperparameters selected by 5-fold cross-validation, except for those trained on the "GEFCom 2014" dataset, which used time series tross-validation. The separation between training and testing sets varied depending on the dataset. None of the artificial datasets or datasets containing empirical data from processes with known physical laws have a separate test set, as they are used to investigate the feature selection capabilities of LCEN (which depend only on the training set). For the "Diesel freezing point" dataset, 30% of the dataset "Abalone" datasct, the last 1,044 entries (25%) were used as the test set as per Wangh (1995) and Clark et al. (1996). For the "Concrete Compressive Strength" dataset, 25% of the dataset was randomly separated to form the test set as per Yeh (1998). For the "Boston housing" dataset, 20% of the dataset was randomly separated to form the test set. For the "GEFCom 2014" dataset, the data from task 1 were used as the training set and all data from tasks 2-15 were usod as the test set.
randomly separated to form the test set. For the
Table A1: Datasets used in this work and their sources. The artificial datasets are used in Scction 3.1, the real datasels from processes with known physical laws are used in Section 3.2, and the real dalasets from processes with unknown physical laws are used in Section 3.2.
Datasct Name Artificial Linear
Source
Artificial data generated by us Artificial data generated by us Artificial data generated by us Artificial data generated by us Schweitzer et al. (2019) [link to dataset) Kepler et al. (1997) (Original from 1619)
Multicollinear data
Relativistic energy
4th-degree, univariate polynomial
CARMENES star data
Kepler's 3rd Law
Diesel Freezing Point
Hutzler & Westbrook (2000) [link to dataset] Nash et al. (1995)
Abalone
Concrete Compressive Strength
Yeh (1998) [dataset: Yeh (2007)
Boston housing (modified by us)
Harrison & Rubinfeld (1978) [link to dataset)
GEFCom 2014
Hong et al. (2016) [link to dataset]
A3 Appendix
List of hyperparameters used in this work
All possible permutations of the hyperparameters below were cross-validated.
For the LASSO and ridge regression models: a and 20 log-spacod values between -4.3 and 0 (as per np- moglope (-4.3, (p..1gapacee-4.33..220 (-4.3,0,20),
2. For the clastic net (EN) models: a above and L1 ratios equal to [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.97, 0.99].
3. For the SCAD models: a as above and the a parameter (also written as 7) equal to 3.7, the default value. According to Fan & Li (2001), SCAD is invariant to changes in a.
For the MCP models: a as above and 7 equal to [1, 1.5, 2, 2.5, 3, 3.5, 4).
5. For the symbolic regression (SymReg) models: most hyperparameters were set to their default values as per Stephens et al. (2022), except. for the following. population size was increased to mutation 0.15), P_ hoist__t_____ mutation _ equal to [0.01, 0.025, 0.05, 0.1), and p-ponnt___ mutation equal to [0.01, 0.025, 0.05, 0.1, 0.15] were tested. Because the sum of these probabilities must be < 1, some combinations are not feasible.
2,000, p_ crossover equal to [0.7, 0.8, 0.9, 0.95], p pusttreee_____ subtree_mmtattonn equal to [0.01, 0.025, 0.05, 0.1,
For the partial least squares (PLS) models: a number of components equal all integers between 1 and limit were used. This limit is either the number of features or 80% of the number of samples, whichever is smaller.
19

---

Published in Transactions on Machine Learning Research (11/2025)
7. For the LCEN models: and L1 ratios above. degree values equal to [1, 2, 3] were typically used, except when otherwise indicated (such as in the "Relativistic energy" dataset). lag = 0 was used, except for the "GEFCom 2014" dataset, which used lag = 168. cutoff values between 1 10003 and 5.5x10-1 were used; higher values were used only when intentionally creating models with fewer selected features. A cutoff = 0 is used the ablation tests for the LASSO-EN model (Section A4).
8. For the random forest (RF) and mudieent-boosted decision tree (GBDT) models: [10, 25, 50, 100, 200, 300] trees, maximum tree depth equal to [2, 3, 5, 10, 15, 20, 40), minimum fraction of samples per leaf equal to [0.01, 0.02, 0.05, 0.1], and minimum fraction of samples per tree equal to [0.1, 0.25, 0.333, 0.5, 0.667, 0.75, 1.0]. For the GBDT models, learning rates equal to [0.01, 0.05, 0.1, 0.2] were also used.
9. For the AdaBoost (AdaB) models: [10, 25, 50, 100, 200, 300] equal to [0.01, 0.05, 0.1, 0.2] were used.
and learning rates
10. For the support vector machine (SVM) models: C values equal to [0.01, 0.1, 1, 10, 50, 100], epsilon values equal to [0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3), and gamma values equal to [1/50, 1/10, 1 1/5, 1/2, 1, 2, 5, 10, 50] divided by the number of features in dataset were used.
11. For the fast sparse GAMs (FS+GAMs): the LoLe penalty, num gammaa = 20, gamma min = 5x10-5, gamma max = 1, and a max support_ssip size equal to the larger of 20% of the features or 8 were used.
12. For the multilayer perceptron (MLP), MLP with group LASSO (MLP-GL1), and LassoNet models: the hidden layer sizes varied for each dataset. Representing an MLP with one hidden layer as [X], an MLP with two as [X, Y], and an MLP with three as [X, Y, 2], hidden layer sizes of
{[800], [400], [200], [100], [800, 800], [800, 400), [800, 200], [800, 100), [400, 400J, [400, 200], [400,
100), [200, 200), [200, 100), [100, 100), [800, 800, 800), (800, 800, 400), [800, 800, 200), [800, 800,
100), [800, 400, 400), [800, 400, 200), [800, 400, 100), [800, 200, 200]. [800, 200, 100), [400, 400,
400), [400, 400, 200], [400, 400, 100), [400, 200, 200], [400, 200, 100], [200, 200, 200), [200, 200,
100), [200, 100, 100), [100, 100, 100] were used with the "Diesel Freezing Point" dataset.
{[18), [9], [4], [18, 18), [18,
[9, 4), [9, 2). [4, 4]} were used with the "Abalone" dataset.
[[48], [40], (32), [24], (4), 48). [48, 40), [48, 32), [48, 24], [48, 16). [48, 8), [40, 40).
[40, 32), [40, 24), [40 16, (4, 32, 32), [32, 24], [32, 16), [32, 8), [24, 24]. [24, 16). [24, 8). [16,
16), [16, 8). 18, 8) 41, 140
(0, 40), [40, 40, 32), [40, 40, 24], [40, 32, 32). [40, 32, 24]. [40, 32,
16, [40, 24, 24, [40, 24, 16) [40. 16 16), [32, 32, 32), [32, 32, 24], [32, 32, 16), [32, 24, 24), [32,
24, 16), [32, 16, 16). [24, 24, 24, 224, 24, 16), [24, 16, 16), [16, 16, 16). [16, 16, 8), [16, 8, 8), [8,
8,8 were used with the "Concrete Compressive Strength" " dataset.
{[78), [65], [52], [39], [26]. [13]. [6). 778, 78), [78, 65], [78, 52]. [78, 39], [78, 26), [65, 65]. [65, 52].
[65, 39], (65, 26), [65, 13), [52, 52). [52, 39]. [52 26]. [52, 13), [39, 39), [39, 26 [39, 13]. [26, 26].
[26, 13). [13, 13), [13, 6]. [78, 78, 78), [78, 78, 65). [78, 78, 52). [78, 78, 39), [78, 78, 26), [78, 65.
65), [78, 65, 52). 178, 65, 39), [78, 65, 26). [78, 52, 52), [78, 52, 39). [78, 52, 26). [78, 39, 39), [78,
39, 26), [78, 26, 26), 65. 65. 65), [65, 65, 52]. [65, 65, 39), [65. 65. 26), [65. 52, 52]. [65, 52, 39].
[65. 52, 26]. [65. 39, 39). [65, 39, 26), [65, 26, 26]. [52, 52, 52]. [52. 52, 39]. [52, 52, 26), [32, 39,
39), [52, 39, 26), [52, 26, 26), [39, 39, 39), 33, 39, 26), [39, 26, 26). [26, 26, 26]} were used with
the "Boston housing" dataset.
Learning rales equal to [0.0005, 0.001, 0.005, 0.01, 0.05], the AdamW optimizer, the ReLU and tanhshrink activation functions, a batch size of 32, weight decay with À equal to [o, 0.01, 0.05, 0.08, 0.1], 100 epochs, and a cosine scheduler with a minimum lcarning rate equal to 1/16 of the original learning rate with 10 epochs of warm-up were also used. For the MLP-GL, and NessoNet, regularization parameters equal to [1x10-4, 1x10-3 1 1x10-2 2] were used.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Table A2: Additional features included for each value the degree hyperparameter for a dataset with three features labeled Xo: X1. and X2 when the lag hyperparameter set to 'all', and the interaction hyperparameter is set True. If the trans trans_type hyperparameter were set 'poly', only the features of the form (X&)" and the interaction terms (if interaction were still set to True) would be present. A degree of (any natural number) also includes all features from degrees to n 1.
to O, the trans_type hyperparameter is
Degree
Sample new Teatures included [for all k) intercept, XR, In Xs, (Xx))/2, 1/Xx
Features after expansion
13
2
(Xx)²; 2-way interactions, (In Xx) (Xx) /2/2
1 In
37
(X.)a
X
3
(Xx)²; 3-w interactions, (In Xx) (Xx)9/2, [x2]3
(In Xx) Xg (Xx)
75
129
5
201
A4 Appendix - Ablation tests
To better clarify the design choices of the L.ENN algorithm and highlight. the relevance of each individual part of the algorithm, ablation tests performed. Threc ablated algorithms LASSO-Clip (LC, the thresholded LASSO), EN-Clip (ENC, the thresholded EN), and LASSO-EN (LEN) are compared with the original LCEN algorithm. Three variant algorithmss, LASSO-Clip-LASE, (LCL), EN-Clip-EN, (ENCEN), and regular LCEN followed by OLS for debiasing (LCEN-OLS) are also compared. The "Relativistic energy", "Dicsel Freezing Point", "Abalone", "Concrete Compressive Strength", and "Boston Housing" datasets are used in the ablation tests in this section; Table 2 provides ablation test results with the "CARMENES star data" and "Kepler's 3rd Law" datasets.
Tests with the "Relativistic energy" dataset show that models with Clip step had some degree of success with selecting only relevant features (Table A4). However, the ablated algorithms (I.C, ENC, and LEN) had much higher prediction errors for the coefficients of the relevant features, even though LC and ENC were able to select only the relevant features. The variant algorithms (LCL, ENCEN, and LCEN-OLS) had performances closer to that of I.CN,, but T.CL, was slightly worse in terms of error. I.CEN-OLSS was able estimate the coefficients with lower error than LCEN, but this improvement cocfficient estimation performance comes with an increase in test-set MSEs in other tasks.
Globally, the models that begin with EN (ENC and ENCEN) are, average, order of magnitude slower than LCEN on all datascts (Tables A4-A7). In particular, LCEN is, on average, 10.3-fold faster the thresholded EN (ENC) in our experiments (Table A3). LCEN consistently had the lowest validation RMSE in all datasets, and had the lowest test-set RMSE in all but one dataset. As highlighted by Table A5, LCEN tied for the sparsest and most accurate model out of all ablated and variant algorithms trained with the "Diesel Freezing Point" dataset. Overall, these ablation experiments highlight how LCEN is the optimal algorithm to maximize accuracy and selectivity while maintaining low runtime.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Table A3:
Comparison of runtimes for LCEN and ENC, and overall LCEN fold improvement, for the experiments in this work.
Dataset Name
LCEN (s)
Thresholded EN (ENC) (s)
Fold Improvement
Artificial Linear
191
902
4.72
10000x1000, 100% additional false features)
Relativistic energy
4.79
37
7.75
CARMENES star data
6.01
55.4
9.22
Kepler's 3rd Law
3.56
13.1
3.68
Diesel Freezing Point
6.59
20.6
3.13
Abalone
19.2
297
15.47
Concrete Compressive Strength
39.7
800
20.15
Boston housing (modified by us)
167
3045
18.23
Average fold improvement:
10.29=6.77
Table A4: Relative error (%) to the ground truth for the "Relativistic energy" dataset with S m < 100 at different noise levels for ablated and variant LCEN algorithms. The first coefficient is for m and the second cocfficient is for m
Noise Level
LC
ENC
LEN
0%
36.62, 18.08
41.52, 20.91
43.75, 22.32
5%
37.68, 18.85
41.30,
21.16
43.93, 23.45
10%
18.92, 1.647
23.70 44.31,
1.137,
0.468
15%
39.71, 20.61
23.70 44.31,
46.65,
26.40
20%
39.65, 21.13
21.95 39.99,
45.87,
27.23
30%
22.35, 2.603
2.660 22.93,
7.649, 1.036
Runtime (s)
3.70
37.1
5.40
Noise Level
LCL
ENCEN
LCEN >OLS
LCEN
0%
0.007, 0.012
0.001, 0.006
0, 0
0.001, 0.006
5%
0.011, 0.029
0.005, 0.022
0.004, 0.016
0.005, 0.022
10%
0.015, 0.045
0.009, 0.038
0.008, 0.032
0.009, 0.038
15%
0.019, 0.061
0.013, 0.054
0.012, 0.048
0.013, 0.054
20%
0.023, 0.077
0.017, 0.070
0.016, 0.064
0.017, 0.070
30%
0.031, 0.109
0.025, 0.103
0.024, 0.096
0.025, 0.103
Runtime
3.86
38.5
4.79
4.79
Table A5: Results of different ablated and variant LCEN algorithms for the "Diesel Freezing Point" dataset. Compare with Table 3.
Algorithm
Test RMSE (CC)
Features
Runtime (s)
LC
4.86:10.03
37.310.6
4.39
ENC
7.26
152+:119
20.6
LEN
4.92+0.05
39.0+0.0
6.6410.64
LCL
5.03++0.08
35.7++3.1
4.71++0.08
ENCEN
5.04+:0.2
120186
29.245.77
LCEN-OLS
5.02
36
6.59
LCEN
4.89-10.06
37.0+=1.0
6.59
22

---

Published in Transactions on Machine Learning Research (11/2025)
Table A6: Results of different ablated and variant LCEN algorithms for the " "Abalone" dataset. Compare with Table A11.
Algorithm
Test RMSE (rings)
Features
Runtime (s)
LC ENC
2.1
11.8
2.1
297
LEN
2.1
26.3
LCL
2.0
12.9
ENCEN
2.1
308
LCEN
2.0
19.2
Table A7: Results of different ablated and variant LCEN algorithms for the "Concrete Compressive Strength" dataset. All models selected all 8 features, but varying number of transforms of these fea- tures. Compare with Table 4.
Algorithm
Test RMSE (MPa)
Runtime (s)
LC
5.44::0.12
24.6
ENC
8.29
800
LEN
5.63++0.15
44.9
LCL
5.77+0.31
26.4
ENCEN
5.73:10.15
863
LCEN
5.55++0.04
39.7
Table A8: Results of different ablated and variant LCEN algorithms for the "Boston Housing" dataset. Compare with Table 5.
Algorithm
Test RMSE (Thousands USD)
Runtime (s)
LC ENC LEN
5.25:10.07
148
5.32::0.1
3045
5.10:10.66
167
LCL ENCEN
5.31+#0110 10
147
5.23 14
3094
LCEN
4.95++0.10
167
A5 Appendix Additional results with artificial data
A5.1
"Artificial Linear" datasets
Figures A1-A4 provide plots for the "Artificial Linear" datasets with 0% noise and Figs. A5-A6 provide plots for the datasets with 25% noise. Other additional results with artificial data follow from Fig. A7 onwards.
23

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 0% noise and 25 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 500 true features at 0% noise and 125 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
015
100
500
1000
Number of samples
MCC for 1000 true features at 0% noise and 250 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
Figure A1: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 0% noise and 25% additional false features, as written in each subfigure's title.
24

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 0% noise and 50 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 500 true features at 0% noise and 250 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
100
500
1000
Number of samples
MCC for 1000 true features at 0% noise and 500 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
Figure A2: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 0% noise and 50% additional false features, as written in each subfigure's title.
25

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 0% noise and 75 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 500 true features at 0% noise and 375 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 1000 true features at 0% noise and 750 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
Figure A3: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 0% noise and 75% additional false features, as written in each subfigure's title.
26

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 0% noise and 100 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 500 true features at 0% noise and 500 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
MCC for 1000 true features at 0% noise and 1000 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
SymReg
40
30
20
10
0
100
500
1000
Number of samples
Figure A4: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 0% noise and 100% additional false features, as writlen in each subfigure's title.
27

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 25% noise and 25 false features
100
90
80
70
60
MCC
50
LCEN
40
LASSO
30
EN
FS-GAMs
20
SCAD
10
MCP
Thresholded EN
0
100
500
1000
Number of samples
MCC for 500 true features at 25% noise and 125 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
40
30
20
10
100
500 Number of samples
1000
MCC for 1000 true features at 25% noise and 250 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
40
30
20
10
0
100
500
1000
Number of samples
Figure A5: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 25% noise and 25% additional false features, as writlen in each subfigure's title.
28

---

Published in Transactions on Machine Learning Research (11/2025)
MCC for 100 true features at 25% noise and 50 false features
100
90
80
70
60
MCC
50
LCEN
40
LASSO
30
EN
FS-GAMs
20
SCAD
10
MCP
Thresholded EN
0
100
500
1000
Number of samples
MCC for 500 true features at 25% noise and 250 false features
100
LCEN
90
LASSO
80
FS-GAMs
70
SCAD
MCP
60
Thresholded EN
MCC
50
40
30
20
10
100
500 Number of samples
1000
MCC for 1000 true features at 25% noise and 500 false features
100
LCEN
90
LASSO
EN
80
FS-GAMs
SCAD
MCP
60
Thresholded EN
MCC
50
40
30
20
10
0
100
500
1000
Number of samples
Figure A6: Plots of the Matthews Correlation Coefficients (MCCs) for models tested on the "Artificial Linear" dataset with 25% noise and 50% additional false features, as writlen in each subfigure's title.
29

---

Published in Transactions on Machine Learning Research (11/2025)
A5.2
"Multicollinear data" dataset
10
10
y-data noise level E2 (%)
y-data noise level E2 (%)
5
5
2
2
0
0
5
10
2866
717
125
33.5
2866
717
125
33.5
X-data noise level E1 (%) and VIF
X-data noise level (%) and VIF
Figure A7: LASSO (left) and LCEN/SCAD/MRT and y-data noise levels 62 ("Multicollinear data" dataset). Bright red squares indicate both variables were selected and their coefficients had errors & 5%. Light red squares indicate that both variables were selected and their coefficients had 5% < errors < 10%. While squares indicate that both variables were selected and their coefficients had 10% errors < 20%. Light blue squares indicate that both variables were selected and their coefficients had errors 20%. Bright blue squares indicate that only one of the variables selected.
(right) model output at different X-data noise levels
A5.3 "4th-degree, univariate polynomial" dataset
For the "4th-degree, univariate polynomial" dataset, Sun & Braatz (2020) created four models: one that always uses degree = 4 ("unbiased model"), one that always uses degree = 2 ("biased model"), one that selects a degree between and 10 based on cross-validation ("ev"), and one that selects degree equal to 2 or based on cross-validation ("ev limited order"). Sun & Braatz (2021) noted that the degree "unbiasod model" was the best at low noise levels, but its error quickly increases, leading to the degree "biased model" becoming the best for noise levels 75 (Fig. of Sun & Braatz (2021); reproduced with permission here the left subfigure of Fig. 1). The model with degree equal to 2 or 4 "cv limited order" was typically very close in performance to the best model at all noise levels, whereas the model with degree between 1 and 10 "cv" had lower performance. Sun & Braatz (2021) explain these observations with the bias-variance tradeoff: at low noise levels, models should follow the ground truth as closely as possible; thus, the degree 4 "unbiased model" was the best. However, at sufficiently high noise levels, it becomes impossible to obtain enough signal to compensate for the additional degrees of freedom (variance) degree 2 "bissod modcl" becomes the best. The degree between 1 and 10 "cv" model had lower performance due its greater hyperparameter variance, and the degree equal to 4 "ev limited order" model struck balance between the "unbiased model" and the "biased model".
4th degree model; thus, the
Similarly to the models generated using ALVEN, the LCEN model with degree between and 10 "cv" had the lowest performance and the LCEN model with degree equal to or 4 "ev limited order" had a performance between the degree 4 "unbiased model" and the degree "biased model". However, the degree "unbiased model" was always the best model, no matter the noise level used (Fig. 1). We attribute this considerable reduction median MSEs and the superiority of the degree 4 "unbiased model" created by LCEN to the improved feature selection and coefficient estimation algorithm, which is able better resist
30

---

Published in Transactions on Machine Learning Research (11/2025)
variance due to noise and large number of hyperparameters. This is corroborated by how the model with degree between and 10 "cv" tended to select degree 4 at lower noise levels and degree = 2 at higher noise levels (Figure A9), showing how LCEN can automatically follow the bias- varianae tradeoff hypothesis.
1.2
25% or 75% quartile Test MSE
1.0
0.8
0.6
Unbiased Model
0.4
Biased Model
CV Model
CV Limited Model
0.2
0.0
0
25
50
75
100
125
150
175
200
Noise variance o²
Figure A8; 25% (squares) and 75% quartile (triangles) test set MSEs for the LCEN model trained for the 44th-degree, univariate polynomial" dataset. The trends tend to match those from Fig. 1.
31

---

Published in Transactions on Machine Learning Research (11/2025)
0
10
20
6000
30
40
50
5000
60
70
80
Noise variance σ²
90
4000
100
110
120
130
3000
140
150
160
170
2000
180
190
200
210
1000
220
230
240
250
0
1
2
3
4 5 6
8
9
10
Selected Degree
Figure A9: Degrees selected by the model with a degree between and 10 "cv" trained using the LCEN algorithm. At lower noise levels (noise variance x² 300), LCEN tends to primarily select degree 4. At higher noise levels, there shift to primarily select degree 2.
32

---

Published in Transactions on Machine Learning Research (11/2025)
A6 Appendix - Additional results with empirical data
A6.1 Datasets for which no physical law is available
The following tables include results for the "Diesel Froczing Point" dataset for two different scenarios: one in which the end-user seeks a sparser model even at the cost of a higher test RMSE (Table A9), and another that compares the results of 5-fold CV with those of 10-fold CV (Table A10). Both scenarios are discussed in the main text (Section 3.2).
Table A9: Results of LCEN models with forced higher cutoff values trained on the "Diesel Freezing Point" dataset. Compare with Table 3 and the surrounding section.
Model
Test RMSE (CC)
Features
Runtime (s)
4.89=0.06
37#1
6.59+0.63
4.91
29
6.27
LCEN
5.52
13
5.76
7.40
6
5.53
Table A10: Average (# standard deviation across 3 CV seeds) test results for different models for the "Diesel Freezing Point" dataset. 5-fold CV values are equal to those in Tables 3 and A5.
Test RMSE (?C)
Runtime (s)
Model
5-fold CV
10-fold CV
5-fold CV
10-fold CV
PLS
5.21
5.10:10.19
7.30
17.240.1
RR
4.90=0.07 4.87+0.07
6.75#0.00
11.1+00.3
EN
4.89-10.06 4.88++0.66
20.99
43.3++0.3
LASSO
4.95+:0.10 4.98:10.05
4.52:00.09
6.46-10.09
SCAD
5.261:011
5.13+0.12
29.345.9
49.3
MCP
5.2210.07 5.20+0.04
33.912.8
59.0+3.0
RF
5.1010.12
5.3410.13
471=5
755
GBDT
5.21+0.18
6.19+1.51
2,870+32
6,631+81
AdaB
5.17400.0
5.17+:0.00
23.8+00.5
49.1+:0.4
SVM
4.72+0033
4.50+0.17
6.74+0.05
12.4+0016
MLP
4.95
5.0911.05
3,839166
1,828+122
LCEN
4.89
4.88
6.59-
11.3+:1.2
LEN
4.92+0.05
4.9410.02
6.64
11.3#1.0
LCL
5.03110.8
5.0111008
4.71100.08
7.13::0.15
ENCEN
5.04+0.22
5.61
29.245.7
61.5+9..0
Abalone (Haliotis sp.) are snails whose age can be determined by cutting their shells, staining them, and counting the stained shell rings under microscope. This process is laborious and error-prone. An alternative is to estimate the number of rings based on readily available physical characteristics, such weight and size. As before, LCEN was compared with other dense and sparse machine learning models (Table A11). In this problem, OLS, PLS, RR, LASSO, and EN all converged to the OLS solution (that is, no regularization), selecting all 8 linear features and having an RMSE, of 2.1 rings. On the other hand, LCEN automatically detected that 2nd degree fcatures would be relevant. The LCEN algorithm model also selected all 8 features and had an RMSE of 2.0 rings, surpassing all linear models and tying with the best nonlinear models in this task. Nonlinear models had test RMSEs between 2.0 and 2.5 rings, but most all lack the interpretability of LCEN.
By increasing the culoff hyperparameter, sparser LCEN models may be generaled similarly to what was done in Table A9. An LCEN model with only features had an RMSE 2.1 rings, and another with only 2 features had an RMSE of 2.2 rings. This experiment further illustrates LCEN's robust feature selection, and how very sparse LCEN models retain significant performance. Furthermore, LCEN models with the same lower number of selected features had lower test set RMSE than FS-GAMs.
33

---

Published in Transactions on Machine Learning Research (11/2025)
Table A11: Results of different models the " "Abalone" dataset. The number of features that minimizes the cross-validation MSE is for FS-GAMs and for LCEN.
Model
Test RMSE (rings)
Features
OLS PLS RR
2.1
LASSO EN
SCAD
2.1
MCP
2.1
SymReg
2.3
RF
2.1
GBDT
2.2
AdaB
2.3
SVM
2.0
MLP
2.0
MLP-GL1
2.0
LassoNet
2.0
2.1
FS-GAMs
2.2
2.4
2.0
LCEN
2.1
2.2
A7
Appendix - Computational resources used
All experiments were done in a personal computer equipped with 13th Gen I CM i5- 1360KK CPU, 64 GB of DDR4 RAM, and an NVIDIA Geforce RTX 4090 GPU. Runtimes for the models trained on the "Diesel Freezing Point" dataset are provided in Table 3, and runtimes for LCEN and ablated algorithms are provided the tables of Section A4.
34


Reviews:
[{"review_id": "YNjwYacSYk", "reviewer": "Reviewer_F6ab", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes LCEN, a novel algorithm aiming to combine interpretability, sparsity, and robustness in nonlinear feature selection and modeling. The empirical results are promising, demonstrating competitive or superior performance across several datasets. However, the paper would benefit from stronger methodological justification and theoretical grounding. In particular, the rationale behind certain design choices (e.g., cross-validation setup, combination of Lasso and thresholding) remains unclear, and the claimed theoretical justification for LCEN’s superiority is not formally presented. Including comparisons with more advanced Lasso variants such as UniLasso would further strengthen the evaluation. Overall, the work is interesting and potentially impactful, but it currently falls short of the rigor expected for publication at a top-tier venue.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The numerical experiments are thorough and well-designed, covering diverse datasets and modeling scenarios that convincingly demonstrate LCEN’s empirical strengths. However, the experimental section could be further strengthened by including comparisons with more recent feature selection methods, such as UniLasso. On the theoretical side, the paper lacks sufficient justification and formal analysis to support some of its key claims. Overall, the empirical evidence is convincing, but the theoretical support remains incomplete.", "audience_yn": "Yes", "audience_explanation": "Feature selection remains a fundamental problem in machine learning with broad practical relevance. Methods that improve sparsity, interpretability, and predictive accuracy are of clear interest to researchers and practitioners alike. The proposed approach could be valuable for applications in domains such as e-commerce, life sciences, and economics, where interpretability and model efficiency are critical. Consequently, the findings are likely to attract attention from a portion of TMLR’s audience.", "requested_changes": "**1) Cross-validation choice not justified.**\n\nThe use of 5-fold cross-validation is not explained. It is unclear whether this choice was empirically validated or adopted as a convention. The optimal number of folds can depend on dataset size and dimensionality; a brief justification or sensitivity analysis would improve credibility.\n\n**2) Unclear rationale for combining Lasso and thresholding.**\n\nLCEN applies both Lasso regularization and thresholding, but the theoretical or empirical motivation for this combination is missing. Since Lasso inherently performs feature selection via the λ parameter, it is unclear why thresholding is necessary or advantageous.\n\n**3) Claimed theoretical justification is absent.**\n\nAlthough the abstract and introduction suggest that LCEN’s superiority over Lasso has a theoretical basis, no formal justification (e.g., theorem or proposition) is provided. The paper would benefit from a clear statement of assumptions and conditions under which LCEN is provably better.\n\n**4) Missing comparison with stronger baselines.**\n\nIt would be helpful to compare LCEN with UniLasso (arXiv:2501.18360), a more robust variant of Lasso that produces sparser solutions. Including this comparison would clarify whether LCEN’s reported improvements hold against more recent and competitive Lasso-based methods.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "znoz6fwQ1m", "reviewer": "Reviewer_oGDy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a combination of LASSO and elastic net to select features in regression problems. From what I could understand, although with difficulty, a feature expansion is first performed to process the data in a nonlinear space, similarly to the philosophy of applying polynomial expansion + LASSO regression. The main novelty comes from combining clipping strategies with Lasso and elastic net.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper is extremely hard to follow, to the point that it is difficult to understand what are the existing algorithms (such as LASSO) and the novel elements. It is very hard to understand the interplay between Algorithm1 and Algorithm 2 and as a result, it is hard to evaluate accurately the quality of the results.The evaluation involves diverse datasets, including rediscovery of known a-priori physical laws. However, only ablation results are presented and no benchmark is proposed to other existing approaches in symbolic regression and in interpretability work.", "audience_yn": "No", "audience_explanation": "The paper is unfortunately too difficult to follow at this stage. The exposition is often dense, and key ideas are obscured by excessive detail. For example, the methods section starts by describing hyperparameters before the algorithm itself has been properly introduced, which leaves readers confused about what LCEN actually does. Similarly, the introduction and related work are long but do not clearly highlight the novel contribution. As a result, the paper risks undercutting its own impact: while the algorithm is interesting, the writing fails to communicate it in a straightforward way.", "requested_changes": "To me this paper needs an important re-work of its content. The approach needs to be clearly stated and possibily even illustrated with a schematic structure about the interplay of the two algorithms. I suggest the authors making a structural distinction between the overall overview of the approach and details such as the hyper-parameter values and experimental setup. Moreover, since LASSO and Elastic Net are well-established algorithms, I would clearly state what is the main proposed novelty and how exactly it advances the state of the art. \nAs for the results, it would be beneficial to contextualise the values obtained to the existing state of the art not only in symbolic regression but also in other approaches such as physics inspired modelling and machine learning interpretability.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nMvxSj4kem", "reviewer": "Reviewer_Ghph", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents LCEN (LASSO-Clip-EN), a novel algorithm integrating LASSO, thresholding (Clip), and Elastic Net for nonlinear, interpretable feature selection and machine learning. Its key contributions are: (1) A hybrid approach that expands features with nonlinear transformations (e.g., polynomials, interactions) and uses a sequential LASSO-Clip-EN process to achieve sparse, interpretable models robust to noise, multicollinearity, and data scarcity. (2) Empirical validation showing LCEN's superiority in feature selection and prediction, outperforming methods like LASSO, EN, SCAD, MCP, and symbolic regression on artificial and empirical datasets, with a 10x speed advantage over thresholded EN and rediscovery of physical laws (e.g., relativistic energy) with <0.5% coefficient errors. (3) Detailed ablation studies confirming the effectiveness of the LASSO-Clip-EN sequence, highlighting its balance of accuracy, sparsity, and speed compared to variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission about the LCEN (LASSO-Clip-EN) algorithm are supported by accurate, convincing, and clear evidence. The paper outlines a detailed methodology in Section 2, describing the algorithm’s five-step process—LASSO, clip (thresholding), EN, second clip, and post-processing—supported by pseudocode (Algorithm 1), which ensures technical soundness is assessable.\nEvidence is robust, spanning artificial and empirical datasets. The claim of rediscovering physical laws with errors <0.5% is backed by tests on datasets where underlying physical laws are known, as noted in the abstract and Section 3, with results fitting within empirical noise levels. The assertion that LCEN creates more accurate and sparser models than alternatives is supported by comparisons with methods like LASSO, EN, and ANNs, showing lower root mean square errors (RMSEs) and higher sparsity across multiple datasets (implied in the abstract and Methods). The 10-fold speed advantage over thresholded EN is implied through the algorithm’s design, which reduces feature sets early via clipping, though specific runtime data would strengthen this further.\nClarity is evident in the structured approach: the abstract succinctly outlines claims, the introduction justifies the need for interpretability, and the Methods section details hyperparameter tuning (alpha, l1_ratio, etc.) with cross-validation.", "audience_yn": "Yes", "audience_explanation": "Some individuals in TMLR's audience would be interested in knowing the findings of this paper. The paper introduces LCEN (LASSO-Clip-EN), a novel algorithm that combines feature selection with machine learning to create nonlinear, interpretable, and sparse models. This is highly relevant to researchers in machine learning, particularly those working on interpretable models for critical applications like aviation or medicine, where transparency is crucial. The findings that LCEN outperforms traditional methods (e.g., LASSO, EN) in accuracy and sparsity, runs significantly faster (10-fold over thresholded EN), and can rediscover physical laws from data offer practical insights. Additionally, its ability to achieve results comparable to or better than ANNs on multiple datasets without relying on complex black-box architectures appeals to those seeking efficient, explainable alternatives.", "requested_changes": "1. Improve Readability of Algorithm Description\nAdjustment: Break down the dense Algorithm 1 description in Section 2 into bullet points or add inline comments (e.g., explaining the role of each hyperparameter step) for readability.\n2. Fix Typos and Formatting\nAdjustment: Correct minor issues such as the extra space in “l1_ratio= 1.” (Algorithm 1, p. 3)\n3. Qualify Overgeneralized Claims \nAdjustment: Modify the abstract’s statement “frequently creating more accurate, sparser models” to “frequently creating more accurate, sparser models on tested datasets” to avoid implying universal applicability.", "broader_impact_concerns": "There are no explicit ethical concerns that immediately necessitate adding a Broader Impact Statement or suggest that an existing one is insufficiently addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JoUSufa1fJ", "reviewer": "Reviewer_7mYD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the algorithm LASSO-Clip-EN (LCEN) for sparse, nonlinear feature selection and predictive modeling on arbitrary tabular prediction tasks. LCEN uses several precedents in the sparse, nonlinear modeling and feature selection literature. The method works in two stages where the first stage performs a LASSO-based filtering of the original dataset augmented with a programmatically-determined set of nonlinear features and the second stage fits an elastic net model on the selected features. There is hard thresholding following each stage’s modeling to remove low-importance features from the models. The authors study several aspects of LCEN including its predictive accuracy, sparsity, robustness to noise, multicollinearity, and data scarcity as well as its runtime in comparison to LCEN variants such as the thresholded elastic net as well as some common deep learning and machine learning baseline methods in addition to existing sparse, nonlinear modeling approaches.\n- Key strengths:\n    - LCEN is clearly described and easy to understand.\n    - The writing is clear and easy to understand.\n    - The ablations are thorough.\n    - LCEN’s performance is impressive in comparison to baselines and extant methods.\n- Key weaknesses:\n    - Broadly, I think this paper makes too many claims (I counted 8; discussed below) and does not provide enough evidence to cover all of them thoroughly. I recommend reducing the claims to the most compelling and supported ones (see my specific recommendations below). \n    - The paper does not include a link to its code so the experimental reproducibility cannot be assessed.\n    - See the requested changes marked ‘critical’ for more.", "claims_and_evidence_yn": "No", "claims_explanation": "Claim 1: LCEN creates more accurate models than other methods, including extant sparse, nonlinear methods.\n\n- Evidence:\n    - On the Diesel eval, the test RMSE of LCEN, vanilla ridge regression, elastic net, and LASSO lie within the same confidence interval. This test RMSE is near the performance of SVM and MLP, which have the lowest average test RMSE but the difference isn’t statistically significant.\n    - On the Concrete eval, vanilla RR, EN and LASSO weren’t included. Random Forest had the lowest test RMSE but LCEN was close (see Table 4).\n    - On the Boston Housing eval, LCEN achieves significantly lower test RMSE than RR and LASSO. LCEN’s accuracy is comparable to RF and MLP (see Table 5).\n    - On the GEFCom eval, LCEN achieves lowest test RMSE out of deep learning methods but no confidence intervals are presented and vanilla ridge, elastic net and LASSO are omitted (see Table 6).\n- Takeaway: the evidence for this claim is not strongly convincing. Including RR/EN/LASSO baselines where absent would strengthen the claim if LCEN outperforms them.\n\nClaim 2: LCEN creates sparser models than other methods, including extant sparse, nonlinear methods.\n\n- Evidence:\n    - Diesel eval: LCEN selects 37±1 out of 401 features, LASSO selects 33±10 with equivalent test RMSE (Table 3).\n    - Diesel ablations in Appendix Table A4: LCEN variants LASSO-Clip (”LC”), LASSO-Elastic Net (”LEN”) and LASSO-Clip-LASSO (”LCL”) achieve similar test RMSE as LCEN with 37.3 ± 0.6, 39.0 ± 0.0, and 35.7 ± 3.1 features selected, respectively, which is about the same sparsity as LCEN’s 37 ± 1 features.\n- Takeaway: the evidence for this claim is not convincing since it only involves 1 evaluation and vanilla LASSO achieves a similar result to LCEN on it.\n\nClaim 3: LCEN is robust to noise.\n\n- Evidence:\n    - Figure 1 synthetic dataset: LCEN median test MSE’s are ≥60% lower than ALVEN’s across noise levels.\n    - Table A3: ablations LASSO-Clip, Elastic Net-Clip, and Lasso-Elastic Net have much higher coefficient errors than LCEN.\n- Takeaway: sufficient evidence. Evidence would be stronger if this were tested on a noised real-world dataset, e.g. Diesel with noised labels.\n\nClaim 4: LCEN is robust to multicollinearity.\n\n- Evidence:\n    - Figure A7: LCEN separates two correlated variables and estimates their coefficients with greater accuracy under a wider range of X and y noise levels than LASSO.\n- Takeaway: accurate, convincing and clear. Just a bit scant on the volume of evidence.\n\nClaim 5: LCEN is robust to data scarcity.\n\n- Evidence:\n    - Figure 1: 60% lower median test MSE than ALVEN for just n=30 training samples.\n    - Table 2, Kepler’s 3rd Law: LCEN selected only the a^{3/2} feature from 130 features and just n=6 observations. 2 ablations succeeded as well but 3 failed.\n- Takeaway: accurate, convincing, and clear.\n\nClaim 6: LCEN is robust to hyperparameter variance.\n\n- Evidence: authors state Table 2 supports this claim but this isn’t clear.\n- Takeaway: unclear, qualitative. Lacking a dedicated sensitivity study, e.g. variance over different HP grids.\n\nClaim 7: LCEN matches or surpasses the thresholded elastic net method [in predictive accuracy] but is 10-fold faster.\n\n- Evidence:\n    - Table 2 CARMENES: LCEN had a runtime of 6.01s, thresholded elastic net (i.e. “ENC”) had a runtime of 55.4s.\n    - Table 2 Kepler’s 3rd Law: LCEN had a runtime of 3.56s, ENC had a runtime of 13.1s.\n    - Table 1 + Table A3 Relativistic Energy: LCEN had a runtime of 4.79s and ENC 37.1s.\n    - Table 3 + Table A4 Diesel: LCEN had a runtime of 6.54s, ENC 20.6s.\n    - Table A5 Abalone: LCEN had a runtime of 19.2s, ENC 297s.\n    - Table 4 + Table A6 Concrete: LCEN had a runtime of 39.7s, ENC 800s.\n- Takeaway: the “10-fold faster” claim is not rigorous so it is not convincing. The actual ratios of runtimes range from 3.1x to 20.1x. Intuitively, the LASSO-based first stage of LCEN eliminates features so the second stage is faster. However, the time complexities are not analyzed so it isn’t clear how much benefit this yields as the number of samples, HP grid size, or number of expanded features grows. Also, the runtimes are absent without explanation for the Boston Housing evaluation.\n\nClaim 8: LCEN can recover physical laws from empirical data better than many other dense and sparse methods & is comparable to or better than ANNs on multiple datasets.\n\n- Evidence:\n    - Table 2: LCEN achieves 100% ± 0 MCC with very low % relative error on the CARMENES star data and Kepler’s 3rd Law data. Thresholded elastic net (”ENC”) and 2 other variants struggle.\n    - Table 3 Diesel: LCEN achieves parity with ANNs in terms of test RMSE (within statistical significance).\n    - Table 4 Concrete: LCEN achieves parity with ANNs (even slightly outperforms vanilla MLP) in test RMSE (within statistical significance).\n    - Table 5 Boston Housing: LCEN achieves parity with ANNs (within statistical significance).\n    - Table 6 GEFCom: LCEN achieves lower test RMSE than all ANN methods.\n- Takeaway: accurate, clear, convincing evidence barring the fact that the ANN baselines are very basic. There is no mention of weight decay, dropout, layer/batch norm, or early stopping and the learning rate grid is small. Also the depth of the models is only 1 or 2 layers, which may limit their accuracy.", "audience_yn": "Yes", "audience_explanation": "I think the interpretability, sparse modeling and scientific ML communities would be interested in this paper. Although each individual element of LCEN has precedents - l1-based feature filtering, hard thresholding of small coefficients, nonlinear feature generation, two-stage procedures for sparse modeling - there is no prior work that combines all these components in the same sequence as LCEN. Also, this paper does a good job of characterizing LCEN and its variants in the ablation experiments showing that the LCEN sequence is best for sparse nonlinear modeling in terms of balancing speed, sparsity and accuracy.", "requested_changes": "- **Critical:**\n    - Release the code for the experiments in this paper so that readers may reproduce the results.\n    - Define what the +/- units are for Tables 2-5, e.g. standard deviation across CV folds.\n    - Include ridge regression, elastic net and LASSO in the Concrete eval (Table 4).\n    - Correct the ± interval for elastic net’s number of selected features on the Diesel evaluation. It currently says 280 ± 209, but there are only 401 features in this dataset so this is impossible.\n    - Perform a short time complexity analysis to substantiate claim 7 (”LCEN is 10x faster than ENC”) else change the language to something empirical like “LCEN is consistently faster than ENC, we observed 3x to 20x faster runtimes in our experiments.”\n- **Nice-to-have:**\n    - Please cite De Mol et al. 2009 in the related works section: https://www.liebertpub.com/doi/abs/10.1089/cmb.2008.0171\n        - They present a two-stage elastic-net → ridge pipeline for sparse feature selection.\n    - Please include LCEN results in Appendix Tables A3, A4, A5, A6, A7 so that readers do not have to flip back and forth to compare results.\n    - Add confidence intervals for the GEFCom evaluation, especially for LCEN.\n    - Include runtimes (esp. LCEN and ENC) for the Boston Housing evaluation.\n    - Include an appendix table that compares runtimes of LCEN and ENC across all evaluations. Use this to substantiate claim 7.\n    - Repeat the MLP baselines with L2/weight decay, dropout, layer norm, early stopping, a wider learning rate grid and architectures with depths of 3, 5, 10 layers in addition to the 1-2 layer models already tested.", "broader_impact_concerns": "N/A", "additional_comments": "Nice work! Looking forward to our discussion.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)