Beyond Deep Learning: An Evolutionary Feature Engineering Approach to Tabular Data Classification
Authors: Hengzhe Zhang
Qi Chen
Aimin Zhou
bing xue
Yan Wang
mengjie zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automated Feature Construction
Automated Machine Learning
Genetic Programming
Evolutionary Algorithm

Abstract: In recent years, deep learning has achieved impressive performance in the computer vision and natural language processing domains. In the tabular data classification scenario, with the emergence of the transformer architecture, a number of algorithms have been reported to yield better results than conventional tree-based models. Most of these methods attribute the success of deep learning methods to the expressive feature construction capability of neural networks. Nonetheless, in real practice, manually designed high-order features with traditional machine learning methods are still widely used because neural-network-based features can be easy to over-fitting. In this paper, we propose an evolution-based feature engineering algorithm to imitate the manual feature construction process through trial and improvement. Importantly, the evolutionary method provides an opportunity to optimize cross-validation loss, where gradient methods fail to do so. On a large-scale classification benchmark of 119 datasets, the experimental results demonstrate that the proposed method outperforms existing fine-tuned state-of-the-art tree-based and deep-learning-based classification algorithms.

Under review as a conference paper at ICLR 2023
BEYOND DEEP LEARNING: AN EVOLUTIONARY FEA- TURE ENGINEERING APPROACH TO TABULAR DATA CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
In recent years, deep learning has achieved impressive performance in the computer vision and natural language processing domains. In the tabular data classification scenario, with the emergence of the transformer architecture, number of al- gorithms have been reported to yield better results than conventional tree-based models. Most of these methods attribute the success of deep learning methods to the expressive feature construction capability of neural networks. Nonetheless, in real practice, manually designed high-orde features with traditional machine learning methods are still widely used because catrralnnetwork-bae features can be easy over-fitting. In this paper, we propose algorithm to imitate the manual feature construction process through trial and improvement. Importantly, the evolutionary method provides an opportunity to op- timize cross-validation loss, where gradient methods fail to do so. On large-scale classification benchmark of 119 datasets, the experimental results demonstrate that the proposed method outperforms existing fine-tuned state-of state-of-the-at of--the-rt tree-based and deep-lean seee-llerring-base based classification algorithms.
iallution-baeed feature engineering
INTRODUCTION
Tabular data is typical way of storing information in computer system, and the tabular data learning methodology has been widely employed in recommendation systems (Wang al.] 2021), advertising (Yang et al.) 2021), and the medical industry (Spooner et al.] 2020) to auto- mate the decision process. Formally, tabular data learning techniques capture the association be- tween some explanatory variables {zI..... Lm. and response variable y for given dataset [(f=)...., x y ,".,.... (fzj..... "". "")""" where n the number of data items.
There are various machine learning methods that could address the tabular data learning problem, such as decision trees, linear models, and support vector machines. However, these learning algorithms have limited learning capability to fully tackle this problem. For example, the linear model assumes that the relationship between explanatory variables and response variables is linear, whereas the decision tree posits that an axis-parallel decision boundary can divide distinct categories of samples. These assumptions are not always correct in the real world, and thus machine learning practitioners always employ feature engineering techniques to construct set of high-order features ,....., (Ør 2% } to improve the learning capability of existing algorithms.
In real-world applications, many feature engineering methods can be used by industrial practition- ers to construct high-order features before training machine learning model. The conventional feature engineering strategies include manually designed features, dimensionality reduction meth- ods (Colange et al. 2020). kernel methods (Allen-Zhu & Li 2019), and so on. However, these methods have drawbacks. Manual feature construction approaches require a large amount of human labor, kernel methods are hard to be combined with tree-based methods, and dimensionality reduction algorithms are insufficient to enhance state-of-the-an learning algorithms, especially for unsupervised
ones (Lensen al. 2022).
With the success of deep learning, there is a growing interest in using neural networks to build high-order features automatically, particularly in the field of recommendation systems (Song et al.) 2019 [Lin et al.) (2018). However, it is debatable whether deep learning techniques can learn high- order features effectively (Wang et al. 2021). Despite the fact that various neural network-hased

---

Under review as a conference paper at ICLR 2023
tabular learning algorithms have shown better predictive accuracy than some tree-based ensemble models (Ank & Phitter 2021), most of these methods only validate their algorithms on datasets with large number of training samples and homogeneous features. Nowadays, it is still controversial whether deep learning methods outperform tree-based methods on datasets with heterogeneous features and a limited amount of data (Gorishniy al. 202T), and we hope to further explore this problem in this paper.
In real-world cases, machine learning engineers typically start from some randomly designed features, and then evaluate these features to get cross-validation loss and feature importance value. The feedback information can be used by engineers to design better features. To imitate this workflow, we design a feature engineering method named evolutionary feature engineering automation tool (EvoFeat D which constructs nonlinear features for enhancing an ensemble of decision tree and logistic regression models, and thus achieving better results than conventional ensemble learning methods. The three objectives of our paper are summarized as follows:
Considering cross-validation losses and feature importance values are widely used by machine learning engineers in the feature engineering process, we propose an evolutionary feature construction method that uses both information to guide the search process.
Given that the performance of the constructed features is highly bonded to the base learner, we propose mechanism that allows the evolutionary algorithm automatically determine whether to use random decision trees or logistic regression models as base learners.
We conduct experiments 119 datasets and compare our method to six traditional machine learning algorithms, four deep learning algorithms and two automated feature construction methods. Experiment results show that deep learning methods not silver bullets for classification tasks.
The remainder of the paper is structured as follows. Section introduces related work of feature construction. Section discusses the motivation behind our work. Section 4arooo///hheerr then explains the details of the proposed method, followed by the experimental results provided in Section Finally, Section summarizes our paper and recommends some intriguing future avenues.
RELATED WORK
Evolutionary feature construction methods have received of attention in the evolutionary learning domain. Nonetheless, the majority of these methods focus on simple learners, such as single decision tree (Tran et al. (019)) or a single support vector machine (Nag Pal 2019). Research on how to apply evolutionary feature construction methods to improve sfate-of-the-art classification algorithms is still lacking. recent years, ensemble-base evolutionary feature construction methods have produced encouraging results on both the symbolic regression (Zhang et al.) 2021) and image classification [Bi et al. 2020 2021a) tasks. However, feature construction on these tasks are different than tabular data classification tasks because domain knowledge for designing primitive functions are available for these tasks, such as a function named analytical quotient Ni et al. (2012) for symbolic regression and the histogram of oriented gradients (HOG) Dalal & Triggs (2005) for image classification.
In addition to the evolution-based method, beam search could also be used to construct high-order features (Katz et al. 2016) Luo et al.) [2199] (Shi et al., (2020). The main idea behind these methods is to start with some low-order features and iteratively generate higher-order features based on important low-order features. A meta learner (Katz et al.] (2016). logistic regression accuracy (Luo et al. 2019). or XGBoost feature importance (Shi et al. 20200) can used to determine the feature importance of each generated feature. However, these methods lack an effective mechanism to prevent over-fitting and thus may have restricted feature construction capabilities.
In the recommendation system domain, deep learning methods have been shown to be effective way to construct high-order features for tabular data classification tasks (Guo et al., 2017). Even though the ford-forward neural network (FNN) is considered universal function approximator (Kidger Lyons 2020), its ability to model high-order feature interactions has been questioned (Mudgal al. 2018). In order to efficiently construct high-order features, a cross network is designed in Deep & Cross Network (DCN) to explicitly model the feature interaction (Mudgal et al. 2018).
Source code:
https://tinyor// com/ CI.22223 KVOFKAT
2

---

Under review as a conference paper at ICLR 2023
Cross-validation Loss
Feature Initialization
Feature Evaluation
Feature
Improvement
Feature Importance Value
Figure I: An example of the feature engineering process.
Following that, XDeepFM employs field-wise feature cross method effectively (Lian al. 2018 model the feature interaction a bit-wise manner. AutoInt (Song et al., 2019) goes a step further, learning feature interaction using the attention mechanism rather than simple outer product. Due to useless feature interaction can introduce noise, AutoFIS (Liu et al. 2020) employs a generalized
regularized dual averaging (GRDA) (Chao & Cheng 2019 method to remove unimportant feature interaction. With the increasing popularity of the neural network architecture search (NAS) technique, an algorithm known as AutoPI (Meng et al.) (021) employs the NAS technique to automatically discover better feature interaction components. This method is inspired by differentiable architecture search (DARTS) (Liu et al., 2018). and it uses the temperature anneal (Liu et al.) (208)) technique to find the best subnet from supernet using the gradient descent method. However, the effectiveness of these feature cross modules over FNN still debatable (Wang et al. 202T), let alone the lack of comprehensive study of whether these approaches can outperform XGBoost on classical classification problems (Gorishniy et al. 202I).
PRELIMINARIES
First of all, let analyze how machine learning engineer constructs features for given dataset [(z=).... Zyn }, y y'),..., (fz".... ................................ 2"h v.y"}} in the absence of domain expertise. Figure depicts an example of the feature engineering workflow. It divided into three phases:
Feature Initialization: First, a machine learning engineer without any domain knowl- edge randomly construct some features +- [ based on original features [.i...Im'
Second, using the cross-validation approach, the generated features } are evaluated on the training set and the target learning algorithm. The trained model produces cross- vross-validation losses and feature importance values to help an engineer determine useful features.
Feature Evaluation:
........2 hb) (Φ1...
Feature Improvement: Finally, the engineer removes some useless features and replaces them with new features derived from important features.
During the feature construction process, the second and third phases are repeated until the engineer is satisfied with the result. Formally, if F is search space of machine learning models and L(f(x), y) is the loss function of a model F, the feature construction process is to find set of features + that can minimize the objective function miny-er (*))))
In real-world applications, some machine learning engineers prefer to only use feature importance values to detect useful features (Shi et al. 2020). However, this is risky because the feature importance values do not reflect the generalization performance of the constructed features. other words, feature may play an important role in an over-fitted model, and we must have a strategy to avoid sticking this local optima. Constructing numerous sets of features the same time a plausible idea. The engineer can then detect the over-fitting issue based on the cross-validatio performance of multiple sets of features and make a more informed decision.
The feature engineering paradigm depicted in Figure I is general. Even for machine learning algorithm that does not provide an internal feature importance report, it can still be employed in this feature engineering method based on some black-box feature importance calculation approaches, such as SHAP (Lundberg & Lee 2017). Under this paradigm, the most critical problem is to determine how to leverage information based feature importance values and cross-validation loss to generate better features. Here, we employ tree-based genetic programming (TGP) to do so. TGP is an evolutionary algorithm that focuses evolving symbolic trees. Its gradient free nature and variable length representation that make it suitable to generate expressive features based on non-differentiable objective values.
3

---

Under review as a conference paper at ICLR 2023
4 THE PROPOSED ALGORITHM
4.1 FEATURE REPRESENTATION
First all, unlike the traditional evolutionary algorithm, which utilizes an array to denote a solution, we employ set of symbolic trees to represent a set of constructed features. Each symbolic tree consists of two parts: non-leaf nodes and leaf nodes. Non-leaf nodes are also known as functions, which transform combine basic features, such as {+, - {log. sin}. The leaf nodes are also known as terminals, representing original features, such as ( Each symbolic tree can be used construct feature o, and all symbolic trees in an individual & together could form new feature space S. In addition to a set of symbolic trees, each individual is also accompanied by a base learner, which is a decision tree linear regression model in this research. This base learner is used to evaluate the predictive performance of each feature space and also used to give a prediction of unseen data.
4.2 ALGORITHM FRAMEWORK
The proposed evolutionary algorithm consists of five parts: initialization, evaluation, selection, generation, and archive update. The pseudocode for our algorithm is shown in Algorithm
Initialization: In the initialization stage, the algorithm randomly initializes N individuals { [y to form the initial population. Each individual consists of k symbolic trees (Ø)... }. which could be used to construct a k-dimensional feature space.
Evaluation: At the evaluation stage, we construct training data based on each individual Φ, and evaluate them on their corresponding base learners f. The fitness value of each individual can be calculated using the five-fold cross-validation loss, and the fitness value of each feature can be obtained using the average feature importance values across five folds.
The selection operator selects promising individuals { } from the population P for crossover and mutation. In this paper, we use the lexicase selection operator (La Cava et al. 2019) to select parent individuals based on the valsdvaaio losses.
Selection:
The generation operator generates new individuals based on selected parent features. We use cof-coomeeei based crossover (Nguyen et al. (02T) and guided mutation operators to increase search efficiency by leveraging feature importance values.
Generation:
At the end of each iteration, we dynamically update archive A to store historically top-performing models, depending on the performance of archived models and newly generated models (/1 ... SN J. The archive update method employed in this research is greedy ensemble selection method named reduce-crror pruning (Caruana et al. 2004].
Archive Update:
The selection, generation, evaluation and archive update operators executed repeatedly until the maximum number of generations is reached. Finally, all base learners in archive A could form an ensemble model based on the weights assigned by the ensemble selection method.
4.3 FEATURE INITIALIZATION
The proposed evolutionary feature construction method is based on TGP, and thus we use ramped- half-and-half (Burke et al., 2003) symbolic tree initialization strategy from TGP to initialize each tree 6. In particular, half of the symbolic trees are built full symbolic trees, while the other half are initialized with a random height. In addition, each individual is randomly assigned a random decision tree (RDT) logistic regression model (LR) with equal probability. This allows the evolutionary algorithm to determine the optimal base learner to form an ensemble model.
4.4 FEATURE SELECTION
The main purpose of the feature selection operator is to select two individuals from the population based on the cross-validation loss. In the existing EA literature, numerous selection operators are available, such as age-fitness pareto (AFP) (Schmidt & Lipson 2011), double tournament selection (Luke & Panait 2002), lexicase selection (La Cava et al., 20199, y, and
In this research, we employ lexicase selection as the selection operator because it is helpful to conserve population
diversity, which is advantageous in the ensemble learning scenario (Zhang et al. (021)..
Lexicase Selection: For a given vector of validation losses, the lexicase selection operator constructs a set of filters based on each loss value across all individuals. Each filter randomly selects one dimension
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Evolutionary Feature Engineering Automation Tool (Evofeeat)
Input: Population Size N, Number of Features k, Number of Generations max .een Output: Archive
1: Randomly initialize N set of features P { &N
Feature Initialization Feature Evaluation
2: for = N.....V do
3: f; f- cross-validation(f fi, 42))
4: A <-
5: A + archive update (ffi fN A)
Archive Update
6: gen
7: while gen <, mar_genn do
Main loop
8:
p'= =0 0
9:
for i 1,... N do
10:
PLR: PDT - population division(P))
11:
if rand() < 0.25 then
Base Learner Selection
12:
Φ - lexicase selection( PLR)
13:
ΦB lexicase selection PDT)
14:
else
P' random selection( Pi.n. Ppr)
16:
① A. 2B + lexicase selection(P")
D Individual Selection
17:
for j 1..... 1 do
Crossover (Feature Generation)
18:
rand() < Pe then
19:
Øa Φ6 self-competitive selection( A)
Feature Selection
20:
Φe 0s - self-competitive selection(
6a crossover(@e:: Q4)
22:
6. + crossover(@e: 06)
23:
for j = 1, ... k do
Mutation (Feature Generation)
24:
rand() ()= < Pm then
25:
Φa f- random selection(
26:
Ø mutation
27:
P' P U 1 A, Φn)
28:
P= p'
29:
for = I N do fs cross-validation((/; 4b).) A archive update (..1...fv):
Feature Evaluation
30:
Archive Update
32:
gen + gen +
33:
return A
j E [1. to do filterin,, and the threshold Ty is dynamically set as T, = min msej + madj, where mad, = median([mss} median(msee) represents the median absolute deviation. Based on this selection operator, we can select suitable number of parent individuals for crossover and selection.
Base Learner Selection: employ either a decision tree or logistic regression model as the base learner to evaluate each group of features. However, decision trees and logistic regression models may not produce comparable predictive results. For example, because single decision tree cannot produce categorical distribution, misclassifying only one example will result in infinite cross- entropy loss. Thus, we borrow an idea from the multitask evolutionary optimization domain (Weil et al.]] 2002 to select two parent individuals from heterogeneous population. First, divide the entire population into two subgroups: one with individuals exclusively using decision trees, and the other with individuals only using logistic regression. Then define hyperparameter called random mating probability rmpp = 0.25, which signifies the probability of selecting two parents from two subgroups. In the remaining seventy five percent of cases, we select two parents from either subgroup.
4.5 FEATURE GENERATION
There are two operators for transforming two given parent individuals into new individuals, namely a self-competitive crossover operator and guided mutation operator. The self-competitive crossover

---

Under review as a conference paper at ICLR 2023
operator was originally developed to evolve multiple job scheduling rules (Nguyen et al. 2021 It keeps effective rules and modifies the less effective rules, which imitates the manual rule design process and is suitable for feature construction scenario. Guided mutation was proposed to solve maximum clique problems (Zhang et al. 2005) by eliminating less useful variables, which is also important for classification problem. Thus, decide use these two operators in our algorithm.
First, we utilize the self-competitive crossover operator to exchange beneficial materials of two parent individuals. In general, the goal of the self-competitive operator to move important materials from the best feature to the worst feature while keeping the best feature intact. For given set of feature importance values {01... 0&} provided by the base learner, we e"u/T
Coff-oompeiiiiee Crossover:
can use softmax function, P(O:) Σ e//77> where T represents temperature, to determine the likelihood of selecting each feature 0, the donor, and negative softmax function to determine the acceptor.
Given the preceding probability distribution, begin by sampling the acceptor individual Pacceptor negatively proportionate to P():), i.e., P.- Then we sample another individual Pdonor as the donor individual according the probability P(O.). Thereafter, the crossover operator selects subtree from & Pionor at random to replace a subtree of Pacceptor- This crossover operator ensures that the top- pop-performing features are rarely destroyed, while the worst features have a high probability of being improved.
Guided Mutation: In addition to the self-competitive operator, the guided mutation operator is utilized to generate new features. The guided mutation operator is similar to the random mutation operator in that replaces a subtree of parent individual with a randomly generated subtree. However, the guided mutation operator uses guided probability vector p to determine the probability of using each terminal variable in a newly generated subtree.
For each terminal variable x, the guided probability {pi {Pi... Pm is defined as the percentage of occurrences of variable all features in the archive, weighted by the feature importance of each feature. The feature importance values of decision tree are calculated by counting the total reduction of Gini impurity with each feature Φ, whereas the feature importance values a lincar model are obtained by calculating the absolute value of model coefficients. All features are standardized before constructing the linear model. This ensures that the magnitude of each feature will not have an impact on the model coefficient.
4.6 FEATURE EVALUATION
For each group of features, feature evaluation is conducted on its corresponding classification model. To avoid over-fitting, we validate the effectiveness of each set of features using a cross-validation approach. For any given training set, we partition into five subsets, i.e., {X1... ...... Xs). Then, for each round, we utilize four folds to train model and the remaining folds to validate the learned model.
Instead of summing all losses into scalar, we keep the original loss vector because it contains more detailed and useful information which the selection operator can exploit and utilize. For each training instance, we employ cross entropy as the fitness function. If the problem contains C' classes, the cross entropy loss can be defined as Leec Pe * log(4e), where refers to the probability of the model prediction belonging a certain class e as ge and the ground truth probability distribution is pe-
5 EXPERIMENTS
We conduct extensive experiments on synthetic and real-world datasets answer the following three questions:
Q1: How does the proposed method perform compared with traditional machine learning
(ML) methods (Section (.221), deep (earning-baseee (DL) methods (Section 5.2.2). and
open-source feature engineering (FE) tools (Section 5.2.3
Q2: How does dynamically selecting base learners during the evolutionary process perform
compared with determining them in advance (Section 5.3.1
Q3: Is it more efficient to use both cross-validan cross-valiatiinn losses and feature importance values to
guide the search than using either one alone (Section 5.3.2

---

Under review as a conference paper at ICLR 2023
Algorithm
Mean
Imp. 1
EvoFeat
0.831899
EvaFeat
LightGBM
0.823112
1.07%
LightGBM
XGBoost
0.819236
1.55%
XGBoost
RF
0.811038
2.57%
FEW
FEW
0.809664
2.75%
AutoFeat
AutoFeat
0.798939
4.13%
FTTransformer
FTTransformert
0.780356
6.60%
MLP
0.778617
6.84%
DT
0.775535
7.27%
ResNet
DCN V2
0.772064
7.75%
ResNet
0.770875
7.92%
0.725 0.775
0.850
LR
0.760355
9.41%
Score
KNN
0.735761
13.07%
Figure 2: Comparison of balanced testing ac- curacy distributions over different algorithms on 119 PMLB dataset.
Table Statistical results of balanced test- ing accuracy for different algorithms on 119 PMLB datasets. (*: Our algorithm, f: ML al- gorithms, 8: DL algorithms,, o: FE algorithms, Imp. 1: Relative improvement)
5.1 EXPERIMENTAL SETTINGS
The experimental datasets include all datasets in DIGEN (Orzechowski & Moore 2021 and all PMLB (Romano et al. 2021) classification datasets between 200 and 10000 instances. DIGEN is diverse set datasets synthesized by the genetic programming method, whereas PMLB includes real-world datasets collected from OpenML (Vanschoren al. (014). In large-scale experiments, we test the performance of our approach against other machine learning algorithms on all 119 datasets in PMLB with up to 10000 instances. In ablation studies, use all datasets in DIGEN and all datasets in PMLB ranging from 200 to 2000 instances due to limited computational resources.
Experimental Datasets:
Evaluation Protocol:For the evaluation protocol, we randomly sample 80% data as the training set to train model and the remaining 20% data as the test set calculate the testing balanced accuracy of the trained model. To obtain a reliable result, we repeat all experiments with ten random seeds and take the average score the final score. For all baseline algorithms, Evolutionary Bayesian Optimization (HEBO) (Cowen-Rivee Rivers et al. (0200) or heuristic methods. The details of the tuning methods and all hyperparameter search spaces are shown the supplementary material.
tune them by Heteroscedastic
5.2
LARGE- -CCLE EXPERIMENT
5.2.1
COMPARISON WITH TRADITIONAL MACHINE LEARNING METHODS
First of all, compare algorithm with traditional machine learning algorithms to demonstrate how feature construction improves the performance. We compare our algorithm to six traditional machine learning classification algorithms, including XGBoost (Chen & Guestrin 2016). LightGBM (Ke et 2017). Random Forest (RF), Decision Tree (DT), Logistic Regression (LR), and K-Nearest Neighbors (KNN). Among these methods, XGBoost and LightGBM are regarded as state-of-the-art algorithms (Gorishniy et al. 2021).
Baseline Algorithms:
Experimental Results: Figure Saaureeilllttttre illlstrates the testing accuracy distribution of all algorithms across 119 datasets. It is clear that gradient boosting methods, such as XGBoost and LightGBM, are nop-performing classification algorithms. However, our approach achieves a better average accuracy than these algorithms. The numerical results shown in Table confirmm this, as our method achieves an average accuracy of 83%, which has more than 1% improvement over the average aceuracies of XGBoost and LightGBM. A further analysis shows that EvoFeat ranks first on 51 out of 119 datasets,
httpos:/github.coo http://sstttstto stt.ssiilaabddaoooo
https://gi Kpi stasi ht..///la..pmmmmmm..m

---

Under review as a conference paper at ICLR 2023
while LightGBM only ranks first on 26 datasets. These results demonstrate that the vvolution-based feature construction method enables an ensemble learning model composed of simple base learners to outperform state-of-the-ee art machine learning models. It is worth mentioning that EvoFeat ranks first does not mean it performs the best on every dataset. Due to space constraints, we discuss this topic in the supplementary material using the instance space analysis technique.
5.2.2 COMPARISON WITH DEEP LEARNING METHODS
Baseline Algorithms: There are a lot of deep learning-basee basend methods that have been proposed in recent years for tabular data learning. We select four models from the existing literature for comparison, i.e., multilayer perceptron (MLP), ResNet, DCN V2 (Wang et al.) 2021 and FT- Transformer (Gorishmiy et al. 2021). The first two architectures are commonly used baselines in the existing literature. The third one is the state-of-the- model for click-through, rate (CTR) prediction, and the fourth model is the state-of-the-ar model for tabular data learning.
The experimental results in Table reveal that existing deep learning approaches cannot compete with XGBoost or LightGBM because the best one, FT-Transformer, only achieves 78% average accuracy. Surprisingly, all deep learning algorithms perform similarly to fine-tuned decision tree, which has an average accuracy of 77%. One probable explanation that these deep learning algorithms require large amount of training data and thus fail to learn robust feature representation on datasets with less than ten thousand items, which is however typical for tabular data.
Experimental Results:
5.2.3 COMPARISON WITH FEATURE CONSTRUCTION METHODS
Baseline Algorithms: In this paper, we compare our method to two open-source feature construction methods: Feature engineering wrapper (FEW) (La Cava & Moore 2017b) and AutoFeat (Horn et al. 2019 FEW is genetic programming based feature construction method, whereas AutoFeat is exhaustive search based feature enginecring method. They are used to construct features for the best model among Extra Trees (ET), RF, XGBoost and LightGBM. The details of experiment settings given the supplementary material.
Experiment results in Table I show that both FEW and AutoFeat cannot achieve top performance, with accuracy scores of 80% and 79%, respectively. FEW uses the coefficient of determination and feature importance values to search for high-order features, and uses the cross-validation loss to determine the historically best set of features. However, generating features without considering the cross-validand volsssvvliotionn loss may mislead the search process and diminish the likelihood of uncovering expressive features. AutoFeat employs LI ---reuularized LR to determine superfluous high-order features. However, features useful on LR may not work well on RF, and therefore its feature construction capability is also limited.
Experimental Results:
5.3
ABLATION STUDIES
5.3.1
BASE LEARNERS
Due to the difficulties of generating a diverse set of linear models, most ensemble learning systems employ decision trees as base learners rather than linear regression models. However, the construction of non-linear features may lead to different results. Therefore, we perform ablation study in which we only use linear regression models or decision trees during the evolutionary feature construction process to investigate the benefit of simultaneously constructing features for linear regression models and decision trees.
Figure 3 Billlustrats the balanced testing accuracy with various base learners over 20 datasets in PMLB. The experimental results reveal that there is no free lunch when it comes to addressing classification problems, and each method has its own of niching tasks. Nonetheless, the combination of random decision tree and logistic regression allows our system to automatically select the appropriate model on the fly, resulting improved average performance. Tablee[shows the mean accuracy of different base learners over 89 PMLB datasets and 40 DIGEN datasets. This table illustrates that the combination DT and LR achieves an accuracy score of 86%, which is more than 1.2% higher than dropping either of them, confirming the effectiveness of using heterogeneous base learners.
5.3.2 EVOLUTIONARY ALGORITHMS
In this paper, we propose to use an evolutionary approach to exploit the information provided by feature importance and cross-validation loss. It is worthwhile investigating whether the evolutionary
8

---

Under review as a conference paper at ICLR 2023
1.0
1.00
0.95
0.8
0.0
0.90
0.85
0.5
0.80
Operator
Base Learner Rendom=3T4
0.4
0.3
AL
pro.
hate
Wayee
wotle
kan
yeust
aieend Aiaard donnzz di
digngi digggie
digers
dige
dgand
Dataser
Figure 3: The balanced testing accuracy of algorithm with respect to different base learners.
Figure 4: The balanced testing accuracy of our algorithm with respect to different selec- tion operators.
Selection Operator
Mean
Imp. f
Base Learner
Mean
Imp.
AL+SC
0,860044
Random- -DT+LR
0.860044
AL
0.857332
0.32%
LR Random-DT 0.848104
0.849455
1.25% 1.41%
SC Random
0,855892
0.49%
0.841332
2.22%
Table 2: Comparison of balanced testing ac- curacy over different base learners.
Table 3: Comparison of balanced testing ac-
curacy over different selection operators.
algorithm effectively uses this information during the search process. Otherwise, random search would be preferable option because much simpler. Therefore, this section presents an ablation experiment demonstrate the efficacy of combined selection operators, which is the driving force of an evolutionary algorithm.
Four selection operators: random selection, only automatic lexicase selection (AL), only self- competitive selection (SC), and both AL and SC (AL+SC) are compared in this section. Figure 4 presents the distribution testing accuracy with respect three selection mechanisms on 20 DIGEN datasets. From this figure, it is clear that using two selection operators simultaneously can effectively improve the predictive accuracy of the final searched features. Table displays the mean testing classification accuracy across 129 datasets for different selection operators. These results confirm the effectiveness of using both selection operators the searching process, where the accuracy score of combining two operators is more than 0.32% higher than that of using only one these operators. Therefore, we can conclude that leveraging both gross-validation loss and feature-importance information is beneficial for obtaining better final results.
CONCLUSIONS
Automated feature engineering is critical component of an automated machine learning system. In this research, we develop and present an evolution-based feature construction method for augmenting standard machine learning algorithms. To make the technique applicable to a wide range of tasks, we simultaneously evolve features for decision trees and linear regression models and form an ensemble of hop-performing models. To increase the search capability of our algorithm, we use both feature importance values and cross-validation losses to identify promising features. Experimental results on 119 datasets show that the proposed method outperforms both state-of-the-ard tree-based ensemble learning methods and deep learning methods.
For future study, considering the current base learner is confined to decision trees and logistic regression models, is worth investigating whether other learners, such as neural networks or GBDTs, could be added to this framework. Furthermore, exploring how to adapt algorithm to large-scale datasets is an intriguing issue; instance selection methods (Bi et al. 2021b) may be required to reduce computational complexity.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Edesio Alcobaça, Felipe Siqueira, Adriano Rivolli, Luís Paulo F Garcia, Jefferson Tales Oliva. André CPLF de Carvalho, et al. Mfe: Towards reproducible meta-feature extraction. Journal of Machine Learning Research, 21(111):1-5, 2020.
Zeyuan Allen-Zht and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? Advances in Neural Information Processing Systems, 32:9015-9000 9025, 2019.
Sercan O Ank and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. AAAI, volume 35, pp. 6679-6687, 2021.
Ying Bi, Bing Xue, and Mengjie Zhang- Genetic programming with new representation to automatically Iearn features and evolve ensembles for image classification. IEEE Transactions on Cybernetics, 1769):1769-1783, 2020.
Ying Bi, Bing Xue, and Mengjie Zhang. divide-and-conquer genetic programming algorithm with ensembles for image classification. IEEE Transactions on Evolutionary Computation, 25(6): 1148 1162, 2021a.
Ying Bi, Bing Xue, and Mengjie Zhang. Instance selection-based surrogate-assisted genetic program- ming for feature learning in image classification. IEEE Transactions on Cybernetics, pp. 1-15, 2021b. doi: 10.1109/TCYB.200 202113. 1.156666...6
Edmund Burke, Steven Gustafson, and Graham Kendall. Ramped half-n-half initialisation bias in gp- In Genetic and Evolutionary Computation Conference, pp. 1800-1801. Springer, 2003.
Rich Caruana, Alexandru Niculescu-Mizii, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In Proceedings of the Twenty-First International Conference Machine Learning, pp. 18, 2004.
Khih-Kang Chao and Guang Cheng. A generalization of regularized dual averaging and its dynamics. arXiv preprint arXiv: 1909, 1007. 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 785-794. ACM, 2016.
Benoît Colange, Jaakko Peltonen, Michaël Aupetit, Denys Dutykh, and Sylvain Lespinats. Steering distortions to preserve classes and neighbors in supervised dimensionality reduction. Advances in 2020.
Neural Information Processing Systems, 33:13214-1222
Alexander Cowen- (Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. An empirical study of assumptions in bayesian optimisation. arXiv preprint 20Xiv:0012.03822 2020.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume 1, pp. 886-893. IEEE, 2005.
Yury Gorishniy. Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-1888 2021.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: factorization- machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 1725 1731, 2017.
Franziska Hom, Robert Pack, and Michael Rieger. The autofeat python library for automated feature engineering and selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 11 1--100. Springer, 2019.
Gilad Katz, Eui Chul Richard Shin. and Dawn Song. Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 979-984. IEFE, 2016.
10

---

Under review as a conference paper at ICLR 2023
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30:3 146-3154, 2017.
Patrick Kidger and Terry Lyons Universal approximation with deep narrow networks. In Conference PMLR, 2020.
on Learning Theory, pp.
William La Cava and Jason Moore. A general feature engineering wrapper for machine learning using c-lexicase survival. In European Conference on Genetic Programming, pp. 80-95. Springer, 2017a.
William La Cava and Jason H Moore. Ensemble representation learning: an analysis of fitness and survival for wrapper- based genetic programming methods. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 961-968, 2017b.
William La Cava, Thomas Helmuth, Lee Spector, and Jason H Moore. A probabilistic and multi- objective analysis of lexicase selection and e-lexicase selection. Evolutionary Computation, 27(3): 377-402, 2019.
Andrew Lensen, Bing Xue, and Mengjie Zhang. Genetic programming for manifold learning: Preserving local topology. IEEE Transactions on Evolutionary Computation, 26(4):661-67) 2022. doi: 10. 20.100//CVC...21.33166
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1754-1763, 2018.
Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2636 2645, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018
Sean Luke and Liviu Panait. Fighting bloat with nonparametric parsimony pressure. International conference on parallel problem solving from nature, pp. 411-421. Springer, 2002.
Scott M Lundberg and Su-In Lee. unified approach interpreting model predictions. Advances in Neural Information Processing Systems, 30:4765-4774, 2017.
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Autocross: Automatic feature crossing tabular data in real-world applications. In Proceedings the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1936-1945, 2019.
Ze Meng, Jinnian Zhang, Yumeng Li, Jiancheng Li, Tanchao Zhu, and Lifeng Sun. A general method for automatic discovery of powerful interactions in click through rate prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1298-12888 1307, 2021.
Sidharth Mudgal, Han Li. Theodoros Rekatsinas, AnHai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, and Vijay Raghavendra. Deep learning for entity matching: A design space exploration. Proceedings of the 2018 International Conference on Management of Data, pp. 19-34, 2018.
Mario Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles Instance spaces for machine learning classification. Machine Learning, 107(1): 107917799-144, 2018.
Mario Andrés Muñoz, Tao Yan, Matheus R Leal, Kate Smith-Miin Ana Carolina Lorena, Gisele L Pappa, and Rômulo Madureira Rodrigues. An instance space analysis of regression problems. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(2):1-25, 2021. 2021.
11

---

Under review as a conference paper at ICLR 2023
Kaustuv Nag and Nikhil R Pal. Feature extraction and selection for parsimonious classifiers with multiobjective genetic programming. IEEE Transactions on Evolutionary Computation, 24(3): 454-466, 2019.
Su Nguyen, Dhananjay Thiruvady, Mengjie Zhang, and Damminda Alahakoon. Automated design of multipass heuristics for resource-constraiie job scheduling with self-competitive genetic programming. IEEE Transactions on Cybernetics, pp. 1-14, 2021.
Ji Ni, Russ H Drieberg, and Peter I Rockett. The use of an analytic quotient operator in genetic programming. IEEE Transactions on Evolutionary Computation, 17(1):146-152, 2012.
Patryk Orzechowski and Jason H Moore. Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers. arXiv preprint arXiv:2107.06475. 2021.
Joseph D Romano, Trang Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v v1.0: an open source dataset collection for benchmarking machine learning methods. arXiv preprint arXiv:201220005882,, 2021.
Michael Schmidt and Hod Lipson. Age-fitnsss pareto optimization. In Genetic Programming Theory and Practice VIII. pp. 146. Springer, 2011.
Qitao Shi, Ya-Lin Zhang, Longfei Li, Xinxing Yang. Meng Li, and Jun Zhou. Safe: Scalable automatic feature engineering framework for industrial tasks. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 1645-1656. IEEE, 2020.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp- 1171,1170, 2019.
Annette Spooner, Emily Chen, Arcot Sowmya, Perminder Sachdev, Nicole Kochan, Julian Trollor, and Henry Brodaty. comparison of machine learning methods for survival analysis of high- dimensional clinical data for dementia prediction. Scientific reports, 10(1):1-10, 2020.
Binh Tran, Bing Xue, and Mengjie Zhang. Genetic programming for multiple-feature construction on high-dimensional classification. Pattern Recognition, 93:404-417, 2019.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, (5(2)499660, 2014.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021, pp. 1785-1797, 2021.
Tingyang Wei, Shibin Wang. Jinghui Zhong, Dong Liu, and Jun Zhang. review on evolutionary multi-task optimization: Trends and challenges. IEEE Transactions on Evolutionary Computation, pp. 1- 2021. doi: 10. 109//EVCC.221.3133943
Haizhi Yang, Tengyun Wang, Xiaoli Tang, Qianyu Li, Yueyue Shi, Siyu Jiang, Han Yu, and Hengjie Song. Multi-task learning for bias-free joint etr prediction and market price modeling in online ad- vertising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2291-2300 2021.
Wei Yang, Kuanquan Wang, and Wangmeng Zuo. Neighborhood component feature selection for high-dimensional data. Journal of Computers. 7(1):161-168, 2012.
Hengzhe Zhang, Aimin Zhou, and Hu Zhang. An evolutionary forest for regression. IEEE Transac- tions on Evolutionary Computation, pp. 1-1, 2021. doi: 10.1 10.1000TVV..222.3333366
Mengjie Zhang, Xiaoying Gao, and Weijun Lou. A new crossover operator in genetic programming for object classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybermetics), 3335)31332 2007.
12

---

Under review as a conference paper at ICLR 2023
Qingfu Zhang, Jianyong Sun, and Edward Tsang. An evolutionary algorithm with guided mutation for the maximum clique problem. IEEE Transactions on Evolutionary Computation, 9(2):192-200 2005.
13


Reviews:
[{"review_id": "zL5s6HCyr9", "reviewer": "Reviewer_8pAi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this research, for the task of tabular data learning, an algorithm that searches for features used and the base learner used jointly has been proposed. Within the research, it has been found that on a large number of different tasks, the method outperforms the existing baselines.", "summary_of_the_review": "Although there are some slight limitations in the paper, but the idea of “searching features and learning model jointly” is effective and the experiments are solid.", "strength_and_weaknesses": "Strengths:\n\n1- This is a pioneering work that performs a search on features and learning models jointly (in the field of tabular data learning).\n\n2- A solid number of experiments have been performed and the proposed algorithm outperforms various types of existing baseline models.\n\nWeaknesses:\n\n1- Some background information requires more detail (e.g. TGP, feature importance value). A more in-depth introduction to their design and their main characteristics will make the paper more throughout.\n\n2- The influence of some design choices has not been analyzed empirically. For example, the design decision of using “self-competitive crossover” has not been analyzed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fkkClKl6XdK", "reviewer": "Reviewer_bBog", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out that neural-network-based features can be easy to over-fitting and proposes a an evolution-based feature engineering algorithm to imitate the manual feature construction process through trial and improvement. ", "summary_of_the_review": "This paper focus on an interesting problem and gives some experimental analysis. However, The technical novelty of this paper is limited.", "strength_and_weaknesses": "Strength:\n\nS1. The author illustrates that in the tabular data classification, the selected feature based on deep learning is prone to overfitting, and the proposed method based on the evolutionary algorithm can obtain more robust and better results.\nS2. The proposed method demonstrates promising results compared with various baselines (e.g., ML-based, DL-based, ...).\n\nWeaknesses:\n\nW1. The technical novelty of this paper is limited. There is no novel technique has been proposed or discussed in this paper.\nW2. In the real world, tabular data is usually very large-scale, and as the authors say in Section 5.1, they use part of the data due to limited computational resources. Therefore, an efficiency analysis (e.g., execution time, memory) of the proposed method is necessary, and a comparison with other baselines can also be added to comprehensively evaluate the applicability of each method.\nW3.The insights for the proposed method are not clear. It would be better to provide a more explicit motivation and vital support for the proposed approach than to beat some approaches only.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B5Pk69WitRR", "reviewer": "Reviewer_LFDt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the feature engineering problem for tabular data classification. It presents a new evolutionary algorithm for performing feature engineering. Each feature is generated by a feature tree, whose leaves are features and whose internal nodes are operators. The number of features is a hyperparameter. The evolutionary search process, finding those trees, is guided by the feature importance and training losses, computed by two base learners: logistic regression or decision tree. The best models at each stage are stored, and the final answer is their ensemble. Ablation study shows that considering both logistic regression and decision-tree is better than using just one of them.\nThe new method is applied to 119 PMLB datasets (with up to 10K samples each), and compared to two previous feature engineering methods, six classical ML methods and 4 deep-learning methods. The proposed method out performs all previous methods, with an advantage of about 0.8% average accuracy over the next best model, LightGBM. Previous work considered evolutionary algorithms for regression or image classification, but not for tabular data classification.\n", "summary_of_the_review": "This is an interesting paper that suggests a new method for an important problem. My main concern is whether the accuracy improvement could be due to non-optimal selection of hyper-parameters for the baselines. I suggest to try the default hyper-parameters for the two top baselines (XGBoost and LightGBM), and also try a hyperparameter search with more estimators (up to 5000). I also would like to have a clarification about the run-time measurements, as I mentioned above, and about my question above about the code. I will update my recommendation based on the author responses.", "strength_and_weaknesses": "The paper presents a new method for an important problem, and it is reasonably clear despite the space constraints. I think it would be interesting to see how sensitive are the results to the hyperparameter selection of the baselines compared. Specifically, what would be the results of the two top baselines (LightGBM and XGBoost) when simply using their default hyper-parameters? What would happen if the maximal number of estimators in the hyper-parameter search is allowed to be 5000 instead of 1000?\nAlso, the runtime measurements are not a main aspect of the results, but they are not fully clear (e.g., did each model run on a separate server, did all the servers have identical RAM, etc.).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject