OpenFE: Automated Feature Generation beyond Expert-level Performance
Authors: Tianping Zhang
Zheyu Zhang
Haoyan Luo
Fengyuan Liu
Wei Cao
Jian Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: tabular data
feature generation

Abstract: The goal of automated feature generation is to liberate machine learning experts from the laborious task of manual feature generation, which is crucial for improving the learning performance of tabular data. The major challenge in automated feature generation is to efficiently and accurately identify useful features from a vast pool of candidate features. In this paper, we present OpenFE, an automated feature generation tool that provides competitive results against machine learning experts. OpenFE achieves efficiency and accuracy with two components: 1) a novel feature boosting method for accurately estimating the incremental performance of candidate features. 2) a feature-scoring framework for retrieving effective features from a large number of candidates through successive featurewise halving and feature importance attribution. Extensive experiments on seven benchmark datasets show that OpenFE outperforms existing baseline methods. We further evaluate OpenFE in two famous Kaggle competitions with thousands of data science teams participating. In one of the competitions, features generated by OpenFE with a simple baseline model can beat 99.3% data science teams, demonstrating for the first time that automated feature generation can outperform human experts. In addition to the empirical results, we provide a theoretical perspective to show that feature generation has benefit provably in a simple yet representative setting. Codes and datasets are available in the supplementary materials.

Under review as a conference paper at ICLR 2023
OPENFE: AUTOMATED FEATURE GENERATION BE- YOND EXPERT-LEVEL PERFORMANCE
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of automated feature generation is liberate machine learning experts from the laborious task of manual feature generation, which is crucial for improv- ing the learning performance of tabular data. The major challenge in automated feature generation is to efficiently and accurately identify useful features from a vast pool of candidate features. In this paper, we present OpenFE, an automated feature generation tool that provides competitive results against machine learn- ing experts. OpenFE achieves efficiency and accuracy with two components: I) novel feature boosting method for accurately estimating the incremental per- formance of candidate features. 2) a feature-scoring framework for retrieving effective features from large number of candidates through successive feature- wise halving and feature importance attribution. Extensive experiments on seven benchmark datasets show that OpenFE outperforms existing baseline methods. We further evaluate OpenFE in two famous Kaggle competitions with thousands one of the competitions, features generated
of data science teams participating. by OpenFE with a simple baseline model can beat 99.3% data science teams. In addition to the empirical results, we provide a theoretical perspective show that feature generation is beneficial in simple yet representative setting. Codes and datasets are in the supplementary materials.
INTRODUCTION
Feature generation is an important yet challenging task when applying machine learning methods to tabular data. Tabular data, where each row represents an instance and each column corresponds distinct feature, ubiquitous industrial applications and machine learning competitions. has been well recognized that the quality of features has significant impact on the learning per- formance of tabular data (Domingos, 2012). The goal of feature generation is transform the base features into more informative ones better describe the data and enhance the learning performance. For example, -oreningg ratio (P/E ratio), calculated as (share price)/ (earnings per share), is derived from the base features "share price" and "carnings per share" in financial statements and informs investors about the value of a company. In practice, data scientists typically use their do- main knowledge to find useful feature transformations in trial-and-error manner, which requires tremendous human labor and expertise.
Since manual feature generation is time-consuming and requires case-by-case domain knowledge, automated feature generation emerges as an important topic in automated machine learning (Er- ickson al., 2020; Lu, 2019). Expand-and-reduce automated feature generation, in which we first expand the candidate features and then eliminate re- dundant ones (Kanter & Veeramachaneni, 2015; Lam et al., 2021; Kaul al., 2017; Shi al., 2020; Katz et al.. 2016). There are two challenges in a typical expand aapand-and-redee -enuuceduce practice. First, the num- ber of candidate features usually huge in many industrial applications. Calculating all candidate features is not only computationally expensive but also infeasible due to the enormous amount of memory required. The second challenge is how to efficiently and accurately estimate the incremen- tal performance of new feature, i.e., how much performance improvement new candidate feature can offer when added to the base feature set. The majority existing methods rely on statistical tests to determine if 2021: Shi et al., 2020). However, statistically significant features do not always translate into good predictors (Lo al., 2015). Features may be significantly correlated with the target simply for
arguably the prevalent framework in
feature should be included (Kanter & Veeramachaneni, 2015; Lam al.,

---

Under review as a conference paper at ICLR 2023
small group of instances in the population, thereby leading poor prediction in the population (Lo et al., 2015; Ward et al., 2010; Welch & Goyal, 2008). Besides, the effectiveness of new feature may be encompassed by the base feature set, even if the new feature is significantly correlated with the target.
In this paper, propose OpenFE, powerful automated feature generation algorithm that can ef- fectively generate useful features to enhance learning performance. First, motivated by the gap between significant features and good predictors, we propose feature boosting method that di- rectly estimates the predictive power of new features in addition to the base feature set. Second, inspired by the crucial fact that effective features are usually sparse in the huge number of candidate features, we propose a two-stage evaluation framework. In the first stage, we propose successive featurewise pruning algorithm to quickly eliminate redundant candidate features by dynamically allocating computing resources to promising ones. In the second stage, we propose feature impor- tance attribution method to rank the remaining candidate features based on their contributions the improvement in the learning performance and further eliminate redundant candidate features. We validate OpenFE on various datasets and Kaggle competitions, where OpenFE outperforms existing baseline methods. In a famous Kaggle competition with thousands of data science teams partici- pating', the baseline model with features generated by OpenFE beats 99.3% of 6351 data science teams. More importantly, the features generated by OpenFE result in comparable performance improvement than those provided by the competition 's top winners, demonstrating for the first time that automated feature generation competitive against machine learning experts.
even larger
In addition to proposing a novel method, this paper intends to address two important problems that hinder the research process of automated feature generation. The first problem is that the majority of existing methods are evaluated on different datasets, and these studies do not open-source their codes and datasets, hindering new research from conducting fair comparisons. In order facilitate fair comparisons in future research, we reproduce the main methods for automated feature generation and validate our reproduction by comparing the reproduced results with those in the corresponding papers. We will open-source the codes and datasets (see supplementary materials).
The second problem is the lack of evidence regarding the necessity of feature generation the era of deep learning. Deep neural networks (DNNs) are widely recognized for their ability to extract fea- ture representations. In recent years, a variety of DNNs have been carefully developed for modeling tabular data (Anı & Pfister, 2021; Gorishniy al., 2021). and several of them have demonstrated their efficiency in feature interaction learning (Song et al., 2019; Wang et al., 2021). We exten- sively evaluate the effect of OpenFi: on variety of DNNs. We demonstrate that generating new transformed features with OpenFF can further enhance the learning performance of existing DNN architectures. In addition the empirical results, we provide theoretical justification of our fea- ture generation procedure by presenting a simple yet representative transductive learning setting in which feature generation has provable benefits.
We summarize the contributions of our paper as follows:
We propose a novel automated feature generation method that can effectively identify useful new features to enhance learning performance. Extensive experiments show that OpenFE achieves state-of-the-art on seven benchmark datasets. More importantly, we demonstrate for the first time that OpenFR is competitive against human experts in feature generation.
We facilitate future research in feature generation by: 1) reproducing main methods for tomated feature generation and releasing the codes and datasets, 2) providing empirical and theoretical evidence that feature generation is a crucial component in modeling tabular data, even in the era of deep learning.
2 PROBLEM DEFINITION
For a given training dataset D, we split it into sub-training set Der and a validation set Deld- Assume D consists of a feature set T + S, where T is the base feature set and S is the generated feature set. We use a learning algorithm L to learn model L(D T + S), and compute the evaluation metric (() T S), Data, T + S) to measure the model performance, with a larger
Phttps://www.td kaggle e. com/ coom/copetiiiioo i cns/ eee-
onvvvevveeww
2

---

Under review as a conference paper at ICLR 2023
value indicating better performance. Now, formally define the feature generation problem as: max (L((Der T + S), Detd, T S) ACA(T)
(1)
where A(T) is the set of all possible candidate features generated from the base feature set. The goal of feature generation is find a feature set S from A(T) that maximizes the evaluation metric. We determine A(T) by the set O of operators we use to transform the base features. The operators O include unary operators, such as log, sigmoid, square, and binary operators, such as x, + min, mar, GroupBy7 heenThhheeea them to transform numerical and categorical features. Our method can extended to include other operators, and users can also define their own operators.
Appendix D.2 contains the list of operators and how we use
3 OPENFE
3.1 OVERVIEW
We present an overview of OpenFE in Figure 1. OpenFE follows the expand- and-reduce framework for automated feature generation. For expansion, first, we classify all base features into numer- ical features and categorical features. Then we create a pool of candidate fea-
include top-ranked features
feature
predictions
evaluation
boosting
pruning
attribution
dataset
numerical features
min, max
operators
candidate
tures by using operators to enumerate all the first-order transformations of base features, where each transformation uses one operator. The challenge of auto- mated feature generation often lies re- duction after expansion, i.e., how achieve both efficiency and effectiveness in eliminating redun- dant candidate features, especially when the number candidate features is huge. For reduction, propose a two-stage evaluation framework quickly reduce the number of candidate features. Finally, we include top-ranked candidate features in the base feature set. We employ a greedy ap- proach and repeat the above procedure for high-order feature generation.
categorical features
combine, groupby.
Figure 1: The overview of OpenFE.
3.2 FEATURE BOOSTING
One of the key challenges in automated feature generation is to accurately estimate the incremental performance of a new feature, i.e., how much performance improvement the new feature can offer when added to the base feature set. A standard evaluation procedure involves including the new feature in the base feature set, retraining the machine learning model, and observing the change in validation loss (Katz et al., 2016). However, the standard evaluation procedure is prohibitively expensive due to the substantial computing resources required for training a model from scratch convergence the whole dataset. Therefore, many existing methods rely statistical tests to determine a new feature effective (Christ et al., 2018; Kanter & Veeramachaneni, 2015; Shi al., 2020). However, significant features do not always translate into good predictors (Lo al., 2015). The effectiveness of a new feature may be encompassed by the base feature set. Motivated by this, we propose feature boosting method estimate the predictive power of new features in addition to the base feature set.
The inputs of feature boosting are dataset D, new features T', and base predictions á (see the def- initions below). Feature boosting outputs the reduction in loss A as an estimate of the incremental performance of new features. Feature boosting motivated by gradient boosting algorithms (Fried- man et al., 2000). Assume we have a dataset of samples D {(x,, U)), = 1,2, 1,2, n). Let F denote a set of models defined over the base features T. Assume we have a model f e that has been trained to convergence on D. We call the predictions 9i = f(x;) the base predictions, and define the loss L(f)) = Σ Iyy: f f(x,)) Let F" denote a set of models defined over new features T'. We want to find new model f E F' to boost the performance of and minimize L(f + f') Ei= I(y:. f(x,) + f(xx)). We can fix f and optimize f' by gradient descent (see more details in Appendix D.3). The reduction in loss A T(J) T( f") ) is an estimate of the
3

---

Under review as a conference paper at ICLR 2023
incremental performance of new features. We call this method "feature boosting," which uses new features to boost the performance of base features. Initialized by base predictions, feature boosting can estimate incremental performance by training new features opposed to the full feature set, resulting in faster convergence for efficiency. We provide a case study in Appendix C.6 to illustrate that feature boosting can efficiently estimate the incremental performance of new features.
Algorithm 1: OpenFE
Algorithm 2: Successive Pruning
input : DD: dataset
input DD: dataset, D: base predictions,
input : :: base features, O: operators
A(T): candidate feature set, 29: #data blocks
output: New feature set.
output: Pruned new feature set.
Initialize order f-
while order < predefined max order do
Divide D equally into 2 data blocks; Initialize Ao(T) = A(T):
Generate base predictions by T and D using cross-validation;
for i < 0 to do
Enumerate candidate features A(T) by O
for new feature T E T do
and T;
Use a subset D; with 2' data blocks to calculate the values of T: Calculate the score of
# two-stage evaluation
AT(T) = SuccessivePruning D,
A" (T) = FeatureAAtributtion""""" (A(T),,,
L A = Feature (D., 0);
T T + top_k(4 k<A' (T));
(T = delete_same (A, (T)) A (T) top (alf(AA((T)
order = order + I;
return T
return A AT+
3.3 A TWO-STAGE EVALUATION FRAMEWORK
Although feature boosting provides an efficient way estimate the incremental performance of candidate features (defined in Section 3.1), it is still computationally infeasible to calculate and evaluate the huge number of candidate features. Inspired by crucial fact that effective features usually sparse in the vast pool of candidate features, we propose a two-stage evaluation framework based on feature boosting efficiently eliminate redundant features and retrieve useful ones.
Before introducing the two-stage evaluation framework, we present the overall framework of OpenFE Algorithm OpenFE starts with generating the base predictions by training model on the base features T for feature boosting. Then we create a pool of candidate features by applying operators on the base features. Next, we propose a two-stage evaluation framework deal with the large number of candidate features. The first stage employs successive featurewise pruning algo- rithm. By using a bandit-based algorithm (Jamieson & Talwalkar, 2016) to dynamically allocate computing resources promising features, quickly eliminate redundant features by examining the effectiveness of each feature alone using feature boosting. In the second stage, we further take interaction effects into account and propose feature importance attribution algorithm to accredit the remaining features in reducing the loss function.
3.3. STAGE I: SUCCESSIVE FEATUREWISE PRUNING
In OpenFE, first use successive featurewise pruning method to quickly reduce the number of candidate features (see Algorithm 2). The method is motivated by the successive halving algorithm in multi-armed bandit problems (Even-Dar et al., 2006: Zhou et al., 2014). Successive halving dynamically allocates computing resources to promising arms. In our settings. first we split the dataset into 29 data blocks, where each data block has both training set and validation set, with a total of 2 samples. Then consider each candidate feature an arm, and a pull of the arm is to use feature boosting to evaluate the effectiveness of the single feature on the data blocks. The reward of pulling arm is the reduction in loss in feature boosting, which is calculated on the validation set. We allocate more data blocks (resources) to more promising candidate features. After successive featurewise pruning, only candidate features with positive reduction in loss A are returned.
Besides successive halving, we use a simple trick in our algorithm to eliminate redundant candidate features. It is common in the candidate feature set for two features to have exactly the same values.
4

---

Under review as a conference paper at ICLR 2023
For example, ma T2) and maz(r1, 73) are exactly the same if the minimum value of feature 71 is larger than the maximum value of та and T3. Features with the same values have the same score (i.e., reduction in loss A). Therefore, the delete same algorithm first ranks all the new features by their scores, and then compares the scores of adjacent features to remove redundant ones.
3.3.2 STAGE II: FEATURE IMPORTANCE ATTRIBUTION
There are two problems unaddressed in stage I: 1) There are still many redundant features in the candidate feature set after pruning. 2) Stage only evaluates the effectiveness of each feature alone. Therefore, in stage II, we further con- sider the interaction effects evaluate the ef- fectiveness of the remaining candidate features. We select the top-ranked candidate features and remove redundant ones to improve generaliza- tion performance. Assume the remaining can- didate features after pruning is A'))). We use candidate features A') and base features T together as the inputs to feature boosting. The reduction in loss A is an estimate of the incremental performance of the remaining candidate features A'T which considers the interaction effects between TT and A'(T). We attribute the reduction in loss A to each candidate feature as their importance. Popular methods for feature importance attri- bution include mean decrease in impurity (MDI) (Breiman, 2001), permutation feature importance (PFI) (Breiman, 2001), and SHAP (Lundberg et al., 2018). MDI is a popular method for feature attribution in tree ensembles, which sums up the total reduction of loss in all splits for given fea- ture. PFI measures the increase in the loss function after we randomly permute the feature's 's values. SHAP is game-the theoretic method for feature attribution.
Algorithm 3: Feature Attribution
input D: dataset, ÿ: base predictions, T: base features,
A'(T): candidate feature set output: Sorted A' (T).
= FeatureBoosting (D, T+ A'))), Attribute the reduction in loss A each feature in A'(T) as their importance; Sort (T) according importance; return sorted A (T)
3.4
IMPLEMENTATION
In OpenFE, we use gradient boosting decision trees (GBDT) (Friedman, 2001) to model tabular data for two reasons: 1) GBDT is usually the best performing model on tabular data where features are individually meaningful (Gorishniy et al., 2021; Borisov et al., 2021). 2) GBDT can automatically handle missing values and categorical features, which is convenient for automation (Ke et al., 2017). We use the popular LightGBM implementation (Ke et al., 2017). We use MDI for feature impor- tance attribution. We compare MDI, PFI, and SHAP in Appendix C.4. We show that MDI fast to compute and provides comparable performance to PFI and SHAP. Even though the feature genera- tion method relies on GBDT, the generated features can also enhance the learning performance of variety of DNNs (see Section 5.4).
4 THEORETICAL ADVANTAGE OF FEATURE GENERATION
In this section, we study the advantage of feature generation from a theoretical perspective. We present simple yet representative setting in which the test loss of empirical risk minimization augmented with feature generation converges to zero provably as the number of training samples increases, while the test loss for any learning model without feature generation is at least a pos- itive constant. In particular, we present transductive learning setting, which captures important characteristics of a class of datasets one may encounter frequently in data science applications, e.g., the IEEE-CIS Fraud Detection dataset (we also conduct experiments on this dataset in Section 5.5). Due to space limit, a formal and detailed description of the model can be found in Appendix A. We briefly introduce the high-level idea here.
Many tabular datasets contain both categorical and numerical attributes (i.e., features). A categorical feature partitions the dataset into groups (each associated with distinct category). For a data point (X,Y), Y). the target Y is correlated with not only the feature X, but also certain statistics of the group containing (X,Y), Y). Datasets with such characteristics are abundant in data science applications. As a concrete example, one may think each training data as a transaction, and one categorical feature is user id (each user may have many transactions in this table. These transactions form group). The
5

---

Under review as a conference paper at ICLR 2023
target Y we want to predict about the transaction (e.g.. probability of fraudulence) may depend on not only the features of this particular transaction, but also some statistics of this user (e.g., average size of his/her transactions). Hence, one see that operations such as GroupByTheMMean ThenMean (group by the user id) can provide statistical information about the user by aggregating the information from all data points associated this user.
We present a theoretical data model, which a two-phase data generation model, to capture the above characteristics. Under fairly standard learning theoretic assumptions (i.e., bounded Rademacher complexity), we prove that empirical risk minimization augmented with feature gen- eration (such the GroupByThen size and group size increase.
operation) can achieve vanishing test loss as the sample
Theorem 1. (informal) Assume the data set is generated according to the two-phase process de- scribed in Appendix A. Denole the number of groups in the training and test by kn and k2 respectively, and the number of data points in each group by h. There is a feature generationfuncciion function H, such that the test loss of the empirical risk minimizer f can be bounded by
Low., (H. S O [Rad,, (F) + vln(45 1)/k1 + Vdlln(Ad(ka + -226 1)hh)
with probability at least 1 - 5. In particular, assuming the Rademacher complexity Rada, (F) - as h1 + 3o. the test loss approaches to when k1, h oc.
On the other hand, if we do not use any feature generation, we prove that any predictor f (no matter how complicated f' is) incurs a non-vanishing constant test loss.
Theorem 2. (informal) In case that we do not use any feature generation, there exists a problem instance such that, no matter how large k1, K2, and are, for any predictor f X + V, the test loss D (f') 2 3 84
5 EXPERIMENTS
5.1 DATASETS AND EVALUATION METRICS
We use a diverse set of seven public datasets with two regression datasets, three binary classification datasets, and two multi-class clas- sification datasets. Each dataset has exactly one sain-validation--estt split, and all methods use the same split. The datasets we collect in- clude: California Housing (CA, real estate data, (Pace & Barry, 1997)),
Table 1: Properties of datasets used in our experiments. No- tation: "RMSE" ~ root-mean-qquare error, "AUC" ~ area- under-curve, "Acc." accuracy.
CA
MI
D3
NO
VF
JA
CD
#ubjects dnom.
20640
1200192
101756
34465
98528
83733
581012
100
54
features
34
features
mecric
RMSE
RMSE,
AUC
AUC
AUC
Acc.
Acc.
Pelasses
2
2
2
Microsoft (MI, search queries, (Qin & Liu, 2013)), Diabetes (DI, hospital records, (Strack et al., 2014)), Nomao (NO, data deduplication, (Candillier & Lemaire, 2012)), Vehicle (VE, vehicle clas- sification, (Siebert, 1987)), Jannis (JA, anonymized dataset, (Guyon et al., 2019)), Covertype (CO, forest characteristics, (Blackard Dean, 1999)). We summarize the dataset properties in Table 1. The features in each dataset are classified into numerical, categorical, and ordinal features. An ordinal feature can be treated as both categorical or numerical when generating transformed features.
5.2 BASELINE METHODS FOR COMPARISONS
The baseline methods for comparisons include: Base (the base feature set without feature gen- eration), FCTree (Fan et al., 2010). SAFE (Shi et al., 2020), AutoFeat (Horn et al., 2019), Au- toCross (Luo et al., 2019). We provide another baseline that uses the last hidden layer of DCN-V2 as new features. DCV-V2 (Wang et al., 2021) is a DNN architecture developed for modeling tabular data and claims to be able to capture feature interactions automatically. We denote this baseline NN. Most existing automated feature engineering methods do not open-source their codes. We reproduce FCTree, SAFE, and AutoCross according to the descriptions in these paper. We vali- date our reproduction by comparing the reproduced results with those in the corresponding papers

---

Under review as a conference paper at ICLR 2023
(see Appendix C.2). Some other learning-bssee methods, such as LFE (Nargesian et al., 2017), ExploreKit (Katz et al., 2016) (meta learning) and NFS (Chen et al., 2019), TransGraph (Khurana et al., 2018) (reinforcement learning), lack critical details for reproduction. For example, the choice of training datasets is crucial for generalization in learning-bass search did not describe which datasets were utilized for training (Nargesian et al., 2017; Khurana et al., 2018), We run OpenFE on the test datasets they use and compare our results with the numbers reported in their papers in Appendix C.I.
methods, however previous re-
Table 2: Comparisons between OpenFE and baseline methods. We report the mean and standard deviation of the evaluation metric on the test set over 10 random seeds. The top result for each dataset is marked in bold. SAFE and AutoCross can only run on binary-classffication datasets. See Table 1 for the evaluation metric of different tasks.
Method
CA J
MI 4
DI
NO
VE
JA f
CO
LightGBM
Base
a.
0.7043..3.0
0.7313.1.2-3
0. 0.9958.pa
0.9250 0-4
0.7212.1.0ca
0.968Gs1.1e8
NN
1.791:1...
0.7196. 4
0.7171:1.06-
0. 0.9921.5.0c
0.9211 66-4
0.7195 8.00
0.9659.200c4
FCTree
0,4321-240 2.76-a
0.71100
0.7313
0. 0.9956 2.0c-
0.9255
0.7190
0.9710
SAFE
0.7295 Desk
0.9058g3.00
0.9250a1
AutoFFea
0.4440s2.5e-
0.7438
0.7318st.tesa
0.9068-2.0e-4
0.9250 0ei
0.7205.7.0e4
0.9682s1:163
AutoCross
0.7319st.1es3
0. 0.9032.3.0e-4
0.9207a3.0e-
OpenFR
0.4206 1.63
0.7382ga.om Be-4
0.8878 Sw-
0.9967
0.9281 0--4
0.7286 1
0.9737
FT Franssosrrer
Base
0.4590
0.7461 -4.0c-4
0.7281
0. 0.99333 5.0cc4
0.9272 0e04
0.7328 228k
0.9696
NN
9.,0e4
0.76083-2.0-44
0.7112
0. 11.9221-2 -4.0e-4
0.92466.20ez 0e-4
0.7163-1.6e4
0.96663 9.0e-4
FCTree
0.4570.2.tea
0.7461.44.0--
0.7282 5.
0. 0.90
0.92778x4.0e-4 0e-4
0.7298
0.9713
SAFE
0.7281
1.11
0.9033
0.9276 0e33
AutoFeeat
0.4574g3.0-3
0.7463g35.0.
0.7281 4:2 dieck
0.9031 1.00-
0.9271 4d 0e-4
0.7337g1.263
0.9704g7.0-
AutoCross
0.72922 De-t
0.9005 000
0.9205 De-1
OpenFE
0. 0.4530_2765
0.7411:7.0e-
0.8870.1.2sm
0.9950
0.9289.3.om
0.7379
0.9708
5.3 COMPARISON RESULTS
In this section, we compare OpenFE with other baseline methods. We use two standard learning algorithms to evaluate the effectiveness of the new features generated by different methods. For GBDT, we choose the LightGBM implementation (Ke et al., 2017). For neural networks, we choose FT-Transformer (Gorishniy et al., 2021). Hyperparameter tuning follows a standard benchmark study (Gorishniy et al., 2021), and we tune the hyperparameters using the base feature set. For each dataset, we generate first-order features and include the same number of new features for different methods for a fair comparison. We discuss generating high-order features in Appendix E.2. We present the comparison results in Table 2. We can observe that OpenFE has clear advantages over baseline methods and achieves SOTA in most cases. The features generated by OpenFE greatly enhance the performance of both LightGBM and FT-Transformer. When baseline methods fail to generate effective new features, they include uninformative features in the base feature set, which brings additional noise and does harm the generalization of the learning model. We present the running time of different methods in Appendix C.3.
5.4 FEATURE GENERATION FOR NEURAL NETWORKS
In this section, we show that new features generated by OpenFE can greatly improve the performance of a variety of neural networks designed specifically for tabular data. The models include: Au- toInt (Song et al., 2019), DCN-V2 (Wang et al., 2021), NODE (Popov et al., 2019), TabNet (Arık & Pfister, 2021), FT-Transformer (Gorishniy et al., 2021). We follow the same implementations and hyperparameter tuning of these networks in (Gorishniy et al., 2021). We present the results in Table 3. The features generated by OpenFE greatly enhance the performance of different models in most cases. For Autolnt and DCN-V2 which claim to be able to learn feature interactions, feature generation can further improve the performance. Even though OpenFE relies on GBDT to measure the performance of new features, the generated features are also effective for variety of DNNs.
7

---

Under review as a conference paper at ICLR 2023
Table 3: The effect of OpenFE on variety of DNNs. We report the mean and standard deviations of the evaluation metric on the test set over 10 random seeds. The top results are shown in bold.
Model
Features
CA
MT
NO 1
VF
JA
CO
Base
0.1741
0.7301 15.0c.4
0.9927
0.9270
0.7207
0.93/12
Autolnt
OpenFE:
0.461825.te3
0..04
0.9943
0.9287a7.0e-4
0.7243-2.7m-
0.9415:4.1ea
Rase
0.4832a9.1e4
0.7486g7.00
0.992624.0e4
0.9236g1.5ea
0.7158a1.6ma
0.965321.3r4
DCN-V2
OpenFF:
0.4722xc.03
0..03
0.9944
0.9260g4..e-4
0.7163=1.6-3
0.9633:7.0e41
Base
0.609624.9e-
0.750%a1.1ea
0.9932a3.0e44
0.9236s1.0e-a
0.723724.3-3
0.9594-433-4
TabNet
OpenFF::
0.501
0.7464g6.0e-
0.9048a2.0-4
0.9258a7.0-41
0.7237a436-
0.9502,1.4m3
Base
0.4639:1.66a
0.7:5428.0e.4
0.9937.4.0e.4
0.926628.0e44
0.7271s1.6e.
0.9689
NODE
OpenFR
0.4560a1.5e-
0.739552
0.9953_5.0--
0.9289
0.7312a1.1e-
0.9632:9.0-41
Base
0..500.8.se.3
0.7.611.4.0c44
0.9833
0.9272.6.0.44
0.7328.2.8..
0.9696:8.0e44
FT-Transformer
Opent-t:
0.4530-2..ee
0.7411-7.0e-
0.9950
0.7379
0.9708.
Table 4: Results of OpenFE and Expert (feature generation by experts) in two Kaggle competitions. Notation: pub, ~ public, pri. ~ private. The final standing is determined according to the scores in the private leaderboard.
IEEE-CIS Fraud Detection
BNP Paribas Cardif Claims Management
feettree
onder
pub. rank
pri rank
pub.
rank
pri rank
Base
0.9 s 7
2408/6351
0,9182 8.2e-4
2286/6351
0,4382 1.7e-
38/2920
0,4359 2.0c
31/2920
first
0.9602 6e-4
52/6351
0.9327
70/6351
0.4334 Tesd
14/2920
0,4308
12/2920
Expert
high
0.9597 te-d
54/6351
0.9320
76/6351
0.4822g
13/2920
0.4301 2:22e-
12/2920
first
0.9617 5.5ee4
38/6351
0.9380 4.0e.4
44/6351
0.43 1.4319-1.444
22/2920
0,4324 2.0e4
16/2920
OpenFI:
0.961723.6e4
38/6351
0.9363
42/6351
0.4324a2.0e-
14/2920
0.4302
12/2920
5.5
CASE STUDIES: COMPARING OPENFE WITH MACHINE LEARNING EXPERTS
We present two Kaggle competitions to show that the features generated by OpenFE can result in competitive performance improvement against those provided by the competition's 's top winners.
The first Kaggle competition is IEEE-CIS Fraud Detec- tion?, where the goal predict whether an online trans- action is fraudulent. This competition is one of the largest and most competitive tabular data competitions on Kag- gle, with 6,351 data science teams participating. The competition's 's first place team made public the features they generated after the competition ended?, which we refer to as Expert. This competition heavily relics on feature generation, where baseline model of XGBoost without feature generation ranks at 2286 among 6351 teams the private leaderboard. However, the baseline model with features generated by the team ranks at 76/6351. The same baseline model with features generated by OpenFE ranks at 42/6351, which outperforms the features generated by the first-place team. We present the results in Table 4. We investigate the influence of the number of new first-order features in Figure 2. The advantage of OpenFE is that it can automatically explore and generate more useful feature transformations than Expert. We explain the useful feature transformations generated by OpenFE in Appendix E.
6.936
Openit
Expert
private
6.932
0.930
0.920
1.00
200
300
500
number of new features
Figure 2: Comparison between Expert
and OpenFE.
The second competition is BNP Paribas Cardif Claims Management', where the goal is to predict whether an insurance claim should be approved. We present the details in Appendix C.5.
hottps::/wwir kaggle com/ 'https: http://wwww...aaoooo..../caae/aa ctm/
'https: h/tp::/wwwwkkg.a wttt:/e....coooomm/eeeeee
8

---

Under review as a conference paper at ICLR 2023
Table 5: Results for ablation study, The model is LightGBM.
CA 4.
MI
DI 1
NO 7
VET
JA 1
C0 1
OpenFE
0.112611.6e3
0. 1..082
0.8878.2.5e-
0.9967.2.0c44
0.9281
0.725611.4e.
0.9737a5.8e44
OF 0.4299
0.7441 14.0es
0.8856
0.9959
0.9263-2.0e-4
0.7251
0.9701
OSS
0.428242.5w3
0.743945.0ed 0.04
0.8846:22263
0.995243.0e4
0.9267
0.7316x1.1es
0.9649+7.0w4
OpenFE-MI
0.4282222363
0.7439=3.0e4
0.995921.0m- 0.88672444m3
0.9261=3.0e-1
0.729321.2m
0.971325.0-1
5.6 ABLATION STUDY
We conduct an ablation study to justify the design choices of OpenFE. We name different variants of OpenFE as follows:
OpenFE FB. OpenFE without feature boosting. We train GBDT from scratch without using base predictions as the initial prediction. For example, in regression tasks the initial predic- tion is the average of the targets in the training set.
OpenFE SS. OpenFE without successive featurewise pruning. We subsample the data so that generating all the features can fit in the memory.
OpenFE-MI. We use mutual information between the feature and the target the scoring criterion successive featurewise pruning.
We present the results in Table 5. The results show that: 1) Feature boosting significantly improves the results. 2) Directly subsampling the data usually hurts the performance. 3) Mutual information does not provide desirable results in eliminating features.
RELATED WORK
Expand-annd-eduus arguably the most popular framework in automated feature generation. Most existing automated feature generation approaches employ statistical methods to identify and re- move redundant features. For example, Data Science Machine (DSM) (Kanter & Veeramachaneni, 2015) uses the f-value between the feature and the target, and One Button Machine (OneBM) (Lam et al., 2021) uses the Chi-square test to remove redundant features. DSM and OneBM focus on feature generation in relational databases, which is different from our setting in tabular data. FC- Tree (Fan et al., 2010) uses information gain to select new features during the construction of de- cision trees. SAFE (Shi al., 2020) uses information value to exclude uninformative features. AutoCross (Luo et al., 2019) and AutoFeat (Horn et al., 2019) uses the improvement on a linear regression model to determine whether new feature should be included. LFF (Nargesian al., 2017) and ExploreKit (Katz et al., 2016) design meta features based on statistical methods and use meta learning approach determine the effectiveness of new features. LFE and ExploreKit are confined to binary-classification tasks and specific set of operators. How to search for high-order feature transformations is challenging in automated feature generation due to the explosion in search space (Chen et al., 2019). Some learning based methods directly search for high-order feature trans- formations (Xie al., 2021). For example, Neural Feature Search (Chen al., 2019) utilizes a recurrent neural network controller to search for a series of transformations. Khurana et al. (2018) uses reinforcement learning to traverse a transformation graph for high-order features.
7 CONCLUSION
We propose OpenFE, a powerful automated feature generation that can effectively generate useful features to enhance the learning performance of tabular data. Extensive experiments show that OpenFE achieves SOTA on seven benchmark datasets and is competitive against human experts in feature generation. We facilitate future research by: 1) reproducing the main methods for automated feature generation and releasing the codes and datasets for fair comparisons. 2) providing empirical and theoretical evidence that feature generation is crucial component in modeling tabular data.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All the codes and datasets can be found in the supplementary materials, including the implementa- tion of OpenFE and other baseline methods. We also provide scripts (detailed experiment parame- ters) in the supplementary materials to reproduce the experiments. Readers can follow the "readme" to download the datasets and reproduce the results. Our theoretical contributions are clearly stated in Appendix A with definitions, assumptions, and complete proofs of all the theorems.
REFERENCES
Sercan O Ank and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, vol- ume 35, pp. (679-6687, 2021.
Jock A Blackard and Denis J Dean. Comparative accuracies of artificial neural networks and dis- criminant analysis predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 24(3)131-151, 1999.
Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint urXiv:2110.01800 2021.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Laurent Candillier and Vincent Lemaire. Design and analysis of the nomao challenge active learning in the real-world. In Proceedings of the ALRA: Active Learning in Real-world Applications, Workshop ECML-PKDDD. Citeseer, 2012.
Xiangning Chen, Qingwei Lin, Chuan Luo, Xudong Li, Hongyu Zhang, Yong Xu, Yingnong Dang, Kaixin Sui, Xu Zhang, Qiao, et Neural feature search: neural architecture for automated feature engineering. In 2019 IEEE International Conference Data Mining (ICDM), pp. 71-80. IEEE, 2019.
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (Isfresh-a python package). Neurocomputing, 307: 72-77, 2018.
Pedro Domingos. A few useful things to know about machine learning. Communications of the ACM, 55(10):78-88) 87, 2012.
Nick Brickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexan- der Smola. autogluon-ttbular: Robust and accurate automl for structured data. arXiv preprint 2055550003.06555. 2020.
Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
Wei Fan, Erheng Zhong, Jing Peng, Olivier Verscheure, Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. Generalized and heuristic-free feature construction for improved accuracy. In Pro- ceedings of the 2010 SIAM International Conference on Data Mining, pp. 6296669-66696 199.600.65 SIAM, 2010.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2): 337-407, 2000.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of
statistics, pp. 1189-1232, 2001.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Anem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Under review as a conference paper at ICLR 2023
Isabelle Guyon, Lisheng Sun- Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengy- ing Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, et al. Analysis of the automl challenge series. Automated Machine Learning, pp. 177, 2019.
Franziska Horn, Robert Pack, and Michael Rieger.
The autofeat python library for automated feature engineering and selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 111-120. Springer, 2019.
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pp. 240-248. PMLR, 2016.
James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analyt- ics (DSAA), pp. 1-10. IEEE, 2015.
Gilad Katz, Eui Chul Richard Shin, and Dawn Song, Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 979-984. IEEE, 2016.
Ambika Kaul, Saket Maheshwary, and Vikram Pudi. Autoleam- automated feature generation and selection. In 2017 IEEE International Conference on data mining (ICDM), pp. 217-226. IEEE, 2017.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Udayan Khurana, Horst Samulowitz, and Deepak Turaga. Feature engineering for predictive mod- eling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018.
Hoang Thanh Lam, Beat Buesser, Hong Min, Tran Ngoc Minh, Martin Wistuba, Udayan Khurana, Gregory Bramble, Theodoros Salonidis, Dakuo Wang, and Horst Samulowitz. Automated data science for relational data. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 2689-2692. IEEE, 2021.
Adeline Lo. Herman Chernoff, Tian Zheng, and Shaw-Hwa Lo. Why significant variables aren't 't automatically good predictors. Proceedings of the National Academy of Sciences, 112(45):13892- 13897, 2015.
Yifeng Lu. An end-to-end automl solution for tabular data kaggledays. Google Al Blog, 2019. URL https: //as:///a.../ /tt.ll/lgllbblg....ooooooo wtt.b//cc.cmm/201//0//0// an-end-to-end-auttom sot.auttooo--sstttoooooooooo html.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv: 1002v.880..00333 2018.
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Autocross: Automatic feature crossing for tabular data in real-world ap- plications. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1936-1945, 2019.
Mchryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana. Elias B Khalil, and Deepak Turaga Learning feature engineering for classification. Ijcai, pp. 2529-2535, 2017.
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (9):291-297, 1997.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning tabular data. arXiv preprint arXiv: 1rXiv:190..06312. 2019.

---

Under review as a conference paper at ICLR 2023
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances neural information processing systems, 31, 2018.
Tao Qin and Yan Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv:130..259) 2013.
Qitao Shi, Ya-Lin Zhang, Longfei Li, Xinxing Yang, Meng Li, and Jun Zhou. Safe: Scalable automatic feature engineering framework for industrial tasks. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 1656. IEEE, 2020.
J Paul Sicbert. Vehicle recognition using rule based methods. 1987.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1161-1170, 2019.
Beata Strack, Jonathan DeShazo, Chris Gennings, Juan Olmo, Sebastian Ventura, Krzysztof Cios, and John N Clore. Impact of hbalc measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed research international, 2014, 2014.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021, pp. 1785-1797, 2021.
Michael D Ward, Brian D Greenhill, and Kristin M Bakke. The perils of policy by p-value: Predict- ing civil conflicts. Journal of peace research, 47(4):363-31 375, 2010.
Ivo Welch and Amit Goyal. A comprehensive look at the empirical performance of equity premium prediction. The Review of Financial Studies, 1454):1455-1508 2008.
Yuexiang Xie, Zhen Wang, Yaliang Li, Bolin Ding, Nezihe Merve Gürel, Ce Zhang, Minlie Huang, Wei Lin, and Jingren Zhou. Fives: Feature interaction via edge search for large-scale tabular data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3795-3805, 2021.
Yuan Zhou, Xi Chen, and Jian Li. Optimal pac multiple arm identification with applications to crowdsourcing. In International Conference on Machine Learning, pp. 217-225. PMLR, 2014.
12

---

Under review as a conference paper at ICLR 2023
A THEORETICAL RESULTS
PROBLEM SETTING
In this section, study the advantage of feature generation from theoretical perspective. In particular, we present a simple yet representative setting in which empirical risk minimization aug- mented with feature generation can achieve smaller test loss than any learning model without feature generation.
Consider a regression problem with both numerical and categorical features in a transductive learn- ing setting. Denote Amm S I' as the numerical feature space. & S N the categorical feature space, and Y S [0 as the target domain. We assume X d-dimensional and convex. The training data Dunin consists of n training data {(X,, )=)..=...... where X; = ( : F.d)), T.o E x is the categorical feature value, (Izl..... Lid) E 2 are d numerical feature values. As concrete example, one may think each training data transaction, where Iio the user id and (Iai...., are some features about the user and the transaction, and Y is the target we want to predict about the transaction (e.g., lateness of payment, probability of fraudulence). The test data Diea consists of m data points {(X,, Yi) Jim++ fi m. but these Y; are the unknown targets that we try to predict.
The data model: The training set Drain and test set Prest are generated by two-phase process. Let (XXer) be the space all probability distributions defined over X and Q be a probability distribu- tion over (A Dunin is generated by repeating the following kı times: in the i-th iteration, we first take a sample P. A(A"t") from (P a distribution over )² Then, we sample a group of h training data points, each with its categorical feature value being c; E X and its d numerical fea- ture values being an i.i.d., sample from P. Hence, these h training data Xit+1,.... X(++1)A have the same categorical value C; Arm and they form the i-th group G; (e.g., the set of transactions of the i-th user). Dirain contains k groups G....., Gky For the i-th data point (X;, Y.). we use g(i) to denote the index the group that contains (X,, Y;). Hence, (X;, Y:) E Gg(t) Let F be the hy- pothesis class and the true hypothesis f X x Xeeat -> y/ such that Y; = = (* Fid: Z,) where 74 = Ex~Pets [X] (74 is d-dimensional vector) (e.g.. the target depends not only on the numerical feature values of this particular transaction, but also the mean the statistics the user).
The test dataset Dies is generated in the same way and it contains k2 groups GRy+l,.... Gk + We assume the categorical feature values cI.... Cky Cky+1 ... Cky are all distinct (e.g., a user id that appears in the test set does not appear in the training set).
Feature generation: Here we are interested in features generation using operations such as GroupByThenMent
In our setting, the groupby operation is based on the categorical feature X So, for a data point X,, we can generate of new features X; which can be computed from all data points in the same group Gg(a). Formally, let H. be the set of possible feature generation function and the new feature X, computed as follows:
X, H(Gg()) for some H H.
For feature generation function H and predictor f. the loss on the data point (X;, Y:) is L(H, f. Y.) ||, f(x, X111 f(X,,
|| f(x,, (Gx()))
Our goal is to find feature generation function H H and a predictor f F such that the test loss is minimized
Dr H,,
E [L.H VJ) m D
Σ
I| H(
If we do not use any feature generation, the loss of the predictor 8 is simply
DD. E.. - - 11
$ 'Note that the input dimension for the predictor f here is different since there is generated feature. But this is not issue for following argument.
13

---

Under review as a conference paper at ICLR 2023
In the following, show that under nature conditions, feature generation (as the number of training samples n and the minimum size of each group become larger). See Theorem 3. But if we only use the raw feature X; for predicting Y,, the expected test loss of any predictor is at least positive constant. See Theorrm 4.
achieve vanishing test loss by using
LEARNABILITY WITH FEATURE GENERATION
For particular feature generation function H, we Rad to denote the empirical Rademacher complexity of F over k random samples:
Radx(F) ((X,).)))
sup _E o.
where = (01,... ok) are independent Rademacher variables and X; is an ii.d. sample from P., which an iid, sample from Q, and Y; = fxX,, 2,) where Z; Exap, [X]. We assume that Rady, (F) + as k + do (for many hypothesis classes, Rade, scales as O(vI/ki)) (Mohri et al., 2018). We also assume any function ((: )) F is Lipschitz
There exists constant CF such that [f Z1) f(X, Z2)l < C Z2ll for any 21, 22, x E X' and f E F. We further assume there exists constant Xx such that supxex | $ Bx.
Theorem 3. There is a feature generation function H, such that the test loss of the empirical risk minimizer f can be bounded by
k2)6-1)/h with probability al least 1 - 5. In particular, the test loss approaches to O when ki, h + 0o.
L J) I 2Radu, (F) + 12 +ttp:///#+6xB2222
Proof. We fix the feature generation function to be GroupByThenMean. In concrete, we have X, = H (Ggg()) [G(v) ΣjeGue> X,. The algorithm then find f that minimizes the emprical risk:
arg min Dimi (H.
JEF
According to Hoeffding's inequality and union bound for each dimension, with probability least - 6/(2 + k2)), we have
[R 2:1l
Σ jEG
X,
[x]| < Bx 2d - +k2)6-1)/h,
(())
X
R(G)
for conerete g(i) By applying union bound all groups, this statement holds for all i with probability at least 1 5/2.
In this case, for any function f E F, we have
[f(X,, 2,) - Y2) - (((x;,81) (.)x)x)))) Y;)²
< (f((x,, 2,) - f(X,, X1)1 [(X,, 2i) + f(X,, - -Ys
$ 22xxCv 12)0-1)/h
where the last inequality holds since [(X,, 74) s(X,, Xi)1 S Crlla 2:ll and y [0. Define H be the function H" (Gg(t) = 7, -x [X]. We note this statement further implies
L (1I*, f) Lii.. (HI f) I /
+ 12)k--1//// and (1
(11, (11
/2d 1/dln(4ed) (n(4d(k) + k2)8-1)/h according to the definition of the loss function.
Note that each data point is not an i.i.d. sample (since two points in one group are correlated), we need to define the following group Rademacher complexity to bound the generalization error. We define group Rademacher complexity to be Rademacher complexity but defined over groups:
k sup
1 Σ L(H,
R by (F) = E Doni
Ki
T
JEG:
14

---

Under review as a conference paper at ICLR 2023
where (01, Tkk) are independent Rademacher variables. Note that if we view each group as one random sample, these groups are i.idd samples. Hence, we can apply the classical generalization bound via Rademacher complexity (Mohri et al., 2018), which asserts that with probability least 1 - 6/2 for any function f E F the following holds
DD.. f) L [H, f) s 2Rad (F) + 22
(3)
Moreover, one can see the group Rademacher complexity can be upper bounded by the ordinary Rademacher complexity for i.i.d. samples:
h
Rade ky (F) S
sup LEE j=1
9 T, Ya.s) = Radk, (F)
(4)
Dusis
kj
where Gig is the j-th element in Gi.
According to union bound, both equation 2 and equation satisfied for all f e F with probability at least 1 - 8. In this case, since f* is the ground truth, its empirical risks satisfies L (H* f²) In this case, we have
Dimm (H,f) L (H,
D (H, (T) L.D. (H² 18) L.D (H* + k2)5-1)/h
< 2BBxCr/2ddn((
(5)
where the first inequality dues to empirical risk minimization and the last inequality dues to equa- tion 2. As result, the population loss then be bounded by
L (H < L (H, LD.. f) + LD ","", L (D"
Dimi (H', f) Li.. (H, f) L (H,
< 2Bx C 2ndn(44(ki + k2) 1 + 2Rad (F) + ( 221n(4811)/k
+2Bx D2 k2) 1) 2Bx 2d k2)6-1)1
< 22Rad, (F) + v
Cd + 12)8-1)/h
where the second inequality holds due to equation 3 and equation while the last inequality follows from equation 4. This proves the desired statement.
Remarks:
We only consider the case where H. only contain one particular operation CroupByThenMers In fact, possible to extend the above setting to one with multiple fea- ture generation functions. Here we require that such feature generation functions be statistics of the corresponding group and can estimated using i.i.d. samples. In our two-phase generative process, each group contains exactly h data points. We can easily extend our theorem to the setting where the size of each group also random variable as long as takes value at least with high probability. The main idea is the same but the notations would become very tedious. Since our goal here is to illustrate the statistical advantage of feature generation, we choose to present a simplified yet representative setting.
WITHOUT FEATURE GENERATION
Theorem 4. In case that we do not use any feature generation, there exists problem instance such that, no matter how large kn (number of groups the training set), ko ((number of groups in the test set)), and h (the size of each group) are, for any fanction I X - y, the test loss is al least
3
LD (f') 64
Proof. Consider problem instance with X = y = [0. 1]. The data distribution is generated in
the following way: Each group contains h data points and is generated in the following way: The
15

---

Under review as a conference paper at ICLR 2023
distribution of i-th group Pε(e) is random between B and ( with equal probability where (p)) is the Bernoulli distribution such that Pr(B(p) = =
Pr(B(p) 0]. Thus,
= kcs)) [X] either or with equal probability for each group. X; is 0/1 random variable,
generated i.d. from either B if Z, and from either ( if Z; 1 We assume the target
of data point given by Y; f (X,, Z₁) Z,.
When X; = 1, the probability of Y is given by
Pr Y _ x, = Pr Z X, =
1
Pr|z, i: X, 1
1
Σ. Pr Pr|Z, = Pr 1| = 2] 4
Similarly, the probability of Y2 is Pr[Y, = - =1] = So for any predictor f : + y have
E - ((x)) (r(x.) 2)² 3 ( 3)²² 3
X,=I
64
The last inequality holds because the left hand side is quadratic function. With the same reasoning. we have Ex,;=olYY; ('(x.) 2
3 As result, the test loss of for any predictor I' is at least
3
E
IIY, f ((x))
DD
64
The above argument does not depend on how large k1, k2 and h are.
B DATA
Table 6: Datasets description
Name
Abbr
# Train
Validation
# Test
Num
# Cat
# Ord
Task type
California Housing
13209
3303
4128
7
Regression
Microsoft
723412
235259
241521
111
25
Regression
Diabetes
DI
65129
16283
20354
3
34
10
Binclass
Nomao
NO
22465
6000
6000
34
29
55
Binclass
Vehicle
VE
60000
18528
20000
100
0
Binclass
Jannis
53588
13398
16747
54
Multiclass
Covertype
CO
371847
92962
116203
9
0
15
Multiclass
We describe the details of the datasets in Table 6. All the datasets can be found in the supplementary materials.
C ADDITIONAL RESULTS
Table 7: Comparison between OpenFE and other baselines. The results of baseline methods are from the corresponding papers. Our results averaged by 10 different random seeds.
Dataset
Source
C)
Instances' Features
Random
TransGraph
LFE
NFS
OpenFE
Airfoil
UCIrvine
1503),5
0.753
0.801
0.796
0.808
German Credit
UCIrvine
1000)22
0.655
0.724
0.805
0.815
Higgs Boson Subset
UCIrvine
50000,28
0.699
0.729
0.731
0.741
Ionosphere
UCIrvine
351),34 1,34
0.934
0.941
0,932
0.969
0.986
SpamBase
UCIrvine
4601 (57
0.937
0.961
0.947
0.955
0.961
SpectF
UCIrvine
467),44
0.748
0.788
0.876
0.877
Sonar
UCIrvine
208\660
0.723
0.801
0.839
0.929
16

---

Under review as a conference paper at ICLR 2023
C.I COMPARISONS WITH OTHER BASELINES
We compare OpenFE with other baselines, including some learning-based methods that lack critical details for code reproduction. These mehtods include:
Random (Khurana et al., 2018). Randomly include features from candidate feature set mul-
tiple times and select new features with improvement according to CV scores.
TransGraph (Khurana et al., 2018). TransGraph uses reinforcement learning to traverse transformation graph for feature transformations.
LFE (Nargesian et al., 2017). LFE recommends feature transformations by meta learning approaches.
NFS (Chen et al., 2019). NFS uses recurrent neural network controller search for series of transformations.
Following previous studies (Chen et al., 2019), the metric for regression datasets (relative absolute enror) and the metric for classification datasets is Fl-score. We present the sults Table 7. OpenFE also surpasses other baseline methods in these datasets.
Table 8: Comparisons between the results of reproduced methods and results from the paper. The results averaged by 10 different random seeds.
Metric
FCTree (from paper)
FCTree (reproduced)
Transfusion
Accuracy
0.752
0.793 0.017
Nuclear(All)
AUC
0.629
0.625 + 0.009
(a) Comparison between the results of FCTree (from paper) and FCTree (repro-
duced).
Metric
SAFE (from paper)
SAFE (reproduced)
Magic
AUC
0.9288 0.9846
0.9370 + 0.0009 0.9837 + 0.0012
Spambase
AUC
(b) Comparison between the results of SAFE (from paper) and SAFE (re- produced).
Metric
AutoCross (from paper)
AutoCross (reproduced)
Bank
AUC
0.9455
0.9456 = 0.0008
Adult
AUC
0.9280
0.9251 = 0.0003
Credit
AUC
0.8567
0.8624 + 0.0002
(c) Comparison between the results of AutoCross (from paper) and AutoCross (reproduced).
C.2 REPRODUCTION
We reproduce FCTree, SAFE, and AutoCross according to the descriptions in their papers. In order to make sure that we reproduce a reasonable version of these methods, we compare the results of our reproduced methods with the ones in their paper. We present the results in Table 8. We most of the results of the reproduced methods closely match or even outperform the results from the papers.
that
C.3 RUNNING TIME
We present the running time of different methods in Table 9. The experimental environment is the same for all the methods. We can see that OpenFE is a fast method that terminates within reasonable amount of time even for large datasets.
17

---

Under review as a conference paper at ICLR 2023
Table The running time of different methods in minutes. IEEE: IEEE-CIS Fraud Detection, BNP: BNP Paribas Cardif Claims Management.
CA
MI
DI
NO
VE
JA
CO
IEEE
BNP
FCTree
2.3
1978
6.8
11
74
40
160
SAFE
0.9
1.3
10
AutoFeat
0.2
23
49
535
354
1484
AutoCross
169
148
146
OpenFE
0.1
31
4.7
3.3
3.5
20
92
0.9
Table 10: Comparisons between MDI, permutation, and SHAP in feature importance attribution.
MDI
permutation
SHAP
Rank
2.0720.93
1.7920.91
2.1420.69
Time
Os
25min
42s
C.4 COMPARING FEATURE IMPORTANCE ATTRIBUTION METHODS
In this section, we compare MDI, permutation, and SHAP in feature importance attribution in OpenFE. We use the results of OpenFE on seven benchmarking datasets rank these methods. The training model is LightGBM. We present the results in Table 10. We also present the running time of each method the Microsoft dataset, the benchmarking dataset with largest number of samples. Different feature attribution methods do not differ much on most of the datasets. MDI can be obtained for free after the training process of LightGBM, while permutation and SHAP may require longer running time, depending on the sizes of datasets.
C.5 KAGGLE COMPETITION
We present the details of the BNP Paribas Cardif Claims Management competition in this section. The goal of the competition is to predict whether personal insurance claim should be approved. The competition's 's 8th place team made public their generated features after the competition ended (the winners with higher rankings did not share their codes). We evaluate the performance of OpenFE in a similar way. A baseline model using Catboost (Prokhorenkova et al., 2018) without feature generation ranks at among 2920 teams. The baseline model with features generated by experts ranks 12/2920, while the baseline model with features generated by OpenFE ranks at 12/2920. We present the results in Table OpenFE generates 200 first-order features and 100 second-order features, while Expert generates 156 first-order features and 132 high-order features.
Table 11: The results of validating feature boosting. The number is the reduction in RMSE on the validation set. FB: feature boosting.
base feature
bf(1)
(2)
(13)
bf(4)
(()
by(())
bf(7)
6f(()
without FB
0.3256
0.0132
0.1008
0.0041
0.0011
0.0525
0.1441
0.1852
with FB
-0.0002
-0.0000
-0.0002
-0.0002
-0.0002
-0.0000
-0.0001
-0,0000
synthetic feature
s1)
(2)
(3)
sf
sf
f
sf(7)
s
with FB
0.0037
0.0029
0.0053
0.0052
0.0026
0.0028
0.0026
0.0022
18

---

Under review as a conference paper at ICLR 2023
C.6 VALIDATE FEATURE BOOSTING
We conduct experiments to validate that feature boosting can estimate the incremental performance of features. We show that:
effective in the presence of base features have zero negative reduction in loss A when evaluated using feature boosting. For example, base features are not effective in addition themselves, and they should have zero negative A.
Features that
Features that are effective in the presence of base features have positive reduction in loss A when evaluated using feature boosting. For example, synthetic feature generated using the information in the targets should have positive A.
We perform the experiment the CA dataset (D) with base features. We generate base predictions 9 by the 8 base features for feature boosting. The experiments include:
For each base feature, we train GBDT f' using the single base feature. We calculate A' L(0) - L(f') of each feature. We present the results in the second row of Table 11. We show through this experiment that each base feature can indeed explain the targets.
For each base feature bf, we calculate A FeatureBoosting bf. 9). We present the results in the fourth row of Table 11. We can see that all of the base features have zero or negative A.
We generate synthetic features using the information in the targets. The formula = 0.3 x y + 6, where follows the normal distribution with the mean and standard deviation the same as y. For each synthetic feature sf. we calculate A = FeatureBoosting ( 9). We present the results in the fifth row of Table 11. The synthetic features are effective in the presence of base features, and they have positive A in feature boosting.
D IMPLEMENTATION DETAILS
D.1 EXPERIMENTAL ENVIRONMENT
For the experiments, feature generation carried out on a workstation with Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz, 40 cores, 512G memory. Model tuning and model training are per- formed on one or more NVidia Tesla V100 16Gb.
Table 12: Unary operators.
Numerical
Categorical
Freq, Abs, Log,
Sqrt, Sigmoid,
Froq
Round, Residual
Table 13: Binary operators. (N refers Numerical, C refers to Categorical)
N N
N X
GreupByThennin, GroupByThenMMa GroupByThenMeen GroupBy CroupByThenMedieee Median, CroupByThenStee GroupByThene ThenRank
Combine,
Min, Max, +,-,x,
CombineThenFree
GroupByThen.Unniqe
D.2 OPERATORS AND FEATURE TRANSFORMATIONS
All the operators are classified into unary operators (Table 12) and binary operators (Table 13), where unary operators feature and binary operators act two features. Then the operators
19

---

Under review as a conference paper at ICLR 2023
further classified according the type of features they act For example, GroupByThenMean requires categorical feature and a numerical feature, while Max requires two numerical features. All the features in dataset classified into numerical features, categorical features, and ordi- nal features. The difference between ordinal features and categorical features that an ordinal feature has clear ordering of categories (such as "age"). Ordinal features are treated as both merical and categorical when generating features transformations. For example, we can calculate GroupByThenMeanneeee gender), which is the average age of each gender. We also calculate Thaneyeeeeeeeeennnnnnn age), which is the average income of people of different ages. For anonymized datasets where the meanings of features are unknown, features with string values are treated categorical features. Features with discrete values (the number of unique values is less than 100) are treated as ordinal features. Features with continuous values are treated as numerical features.
We enumerate all the first-order transformations form the candidate feature set. A first-order transformation uses one operator once to transform base features. For example, weight/ /height is first-order transformation of base features weight and height. BMI weight/height" is second- order transformation.
We list the details of all the operators below. stands for numerical feature and stands for categorical feature.
(f) The frequency of feature f-
Freq(e). The frequency of feature e.
Abs(f). Element-wise absolute value.
Log(f). Element-wise logarithm.
Sqri(S). Element-wise square root.
(f) Element-wisely apply function Ite-r-
Ro)nd(f). Element-wise rounding.
Residual(f). Element-wisell take decimal part.
Min(fi, f2). Element-wis, minimum.
Max(fi, (f1, f2). Element-wise maximum.
fi + f2. Element-wise addition.
fi f2. Element- subtraction.
fi x f2. Element-wise multiplication.
f1 f2. Element-wise division.
GroupByThenMiniin The minimum value of f in each category of feature c.
GroupByThenMax(f..cr The maximum value of f in each category of feature
GroupByThenMeent The average value of f in each category of feature e.
GroupByTheaMcianff...c The median of f in each category of feature GroupByThenstd((f.))) The standard deviation of f in each category of feature c.
GroupByTheen (henRank(J.c)) The ranking of f in each category of feature c. The rankings are normalized between 0 and
Combine(c1, 02). Comebine the categories in feature Ci and Co to be new categories. For example, for data point, "vocation" is "doctor" and "hobby" is "football", then the value for Combine(vocation hobby) of this data point is category of "doctor-football.
CombineeThenFreuu Same freq (Combine(cI, c2)).
o2). The number of unique values of Ci in each category of
feature 62.
20

---

Under review as a conference paper at ICLR 2023
D.3 FEATURE BOOSTING
We describe the main idea of feature boosting in Section 3.2. Here, we present the details of the implementation in this section. The first step in feature boosting to generate base predictions. In this paper, we use LightGBM to model tabular data and use 5-fold cross-validation to generate base predictions. For each fold, we first train LightGBM on the train set till early stopping on the validation set. Then we generate predictions on the validation set. The predictions on each fold are concatenated to yield the base predictions. The LightGBM trained using a set of default parameters for all the datasets. The default parameter has 10,000 estimators, 0.1 learning rate, 31 leaves, and 200 early stopping rounds. Other parameters follow the default settings in LightGBM.
The second step in feature boosting is to find new model f" E F' to boost the performance of base predictions and minimize LJ + (') = -i= (y4,, f((;,) + [(x))). In the implementation, 1.2....,11 as the initial predictions in LightGBM. The
the base predictions {f(x)), implementation of LightGBM can use new features T as the inputs to continue training from the base prediction. The model f' is the new LightGBM model after training. Feature boosting can also be extended to using neural networks. f is neural network parameterized by 8, we can calculate aL(s+(") and optimize 0 by back propaapppa.
Table 14: The parameters of OpenFE for each dataset. IEEE: IEEE-CIS Fraud Detection. BNP: BNP Paribas Cardif Claims Management.
CA
MI
DI
NO
VE
JA
CO
IEER
BNP
# data blocks
64
32
16
8
32
8
# new features
10
10
10
10
10
50
50
600
200
candidate features
275
78k
8k
81k
31k
9k
27k
456k
1753
D.4 PARAMETERS
We present the hyperparameters of OpenFE for each dataset in Table 14. For datasets whose sam- ples or the number of candidate features are not large, the number data blocks for successive featurewise pruning is set to be small. When the number of data blocks is one, we use all the data to calculate and evaluate new features in successive featurewise pruning, and eliminate new features with negative reduction in loss. For most datasets, the number of features is set to be 10. The effective new features are usually sparse in the vast pool of candidate new features. For two multi- classification datasets, the number of new features is set be 50. For fair comparison, all the baseline methods include the same number of new features, except for the NN baseline. For the NN baseline, the number of new features the number of hidden units in the last hidden layer, which determined by hyperparameter tuning.
In OpenFE, the default parameter of LightGBM in successive featurewise pruning has 1000 number of estimators, 0.1 learning rate, 16 leaves, and 3 early stopping rounds. The default parameter in feature importance attribution the same except for 50 early stopping rounds.
D.5 FEATURE IMPORTANCE ATTRIBUTION METHODS
MDI is the gain importance embedded LightGBM. For permutation feature importance, ran- domly shuffle the values of features on the validation set, and observe the change in validation loss. For SHAP feature importance, we average the SHAP values of each sample for each feature.
E DISCUSSION
E. DISCOVERED FEATURE TRANSFORMATIONS
In this section, present and explain the useful transformations discovered by OpenFE for the IEEE-CIS Fraud Detection competition and the DI (Diabetes) dataset. The goal of the IEEE com- petition is to predict whether an online transaction is fraudulent. The top-1 feature discovered by
21

---

Under review as a conference paper at ICLR 2023
0.10
104
135
SHAP values
SHAP values
8:37
3.00
SHAP
5.00
002
Combine ThenFreqluser month)
Sooup@yThenStdidee de user
15
Comb nn www.toonnhonooooooeetttoooo [id]
85 20
36
40
Figure 3: The SHAP values of discovered feature transformations. The x-axis is the feature values, and the y-axis is the SHAP values. Larger SHAP values indicate higher probability of fraudulent transactions.
OpenFE CombineThenFiee (ner id, month), which indicates how many transactions a user makes in month. We present the SHAP values of the feature in the trained XGBoost in Figure 3. The SHAP values tell us that when the user makes few transactions in a month, the transactions are more likely to be fraudulent. The top-2 feature is GroupBy CruupBBThhuSidddae ,ellta, user_id), which the standard deviation of the days between the current transaction and the first transaction. The SHAP values imply that when many transactions happen in short period (which corresponds to a small standard deviation), the transactions are more likely to be fraudulent. The top-4 feature is Combine'T ThenneeehheFeeeee (reeevienn,, (DeviceInfo, user id)r which indicates how frequently the user switches de- vices to make transactions. The actual meanings are masked for most of the features to protect the privacy of users. One of the advantages of OpenFE is the ability to generate useful features without knowing their actual meanings.
The DI (diabetes) dataset collects 10 years (1999-2008) of clinical care 130 US hospitals, where the goal is to predict the readmission (Strack et al., 2014). One can see from Table 2 that OpenFE outperforms other baseline methods and improves the learning performance by great margin. The top-1 feature discovered by OpenFE is freq(patient id), which is the number of times the patient has been admitted to the hospital and is highly predictive of whether the patient will be readmitted to the hospital. However, other methods may fail to find this new feature since the feature patient_ic id itself is not useful. For example, SAFE (Shi et al., 2020) only generates candidate features based on the base features that are used as splits in XGBoost. However, since patient_id _id itself is not useful, it will not be used for splits in XGBoost. This example also demonstrates that, reducing the number of candidate features by heuristic assumptions may risk missing useful candidate features.
E.2 HIGH-ORDER FEATURES
How to search for high-order feature transformations is challenging in automated feature generation due to the explosion in search space (Chen et al., 2019). Some previous methods argue that high- order features are useful by directly searching for high-order fcatures (Chen et al., 2019; Khurana al., 2018). However, the effectiveness of high-order feature transformations should be evaluated in light of all its low-order components. For example, a second-order feature transformation fa x fe fa is effective only if it has additional effectiveness to all their first-order components fi x f2. fi x f3. and f2 x f3. In addition, because high-order feature transformations are typically more difficult interpret than low-order ones, low-order feature transformations are favoured in industrial applications where interpretability is important. In word, searching for high-order features in hierarchical manner more appropriate than directly searching for high-order features in two aspects: 1) evaluating the effectiveness of high-order features in a more accurate way. 2) generating low-order features with better interpretability.
Whether it is necessary to generate high-order features is case-by-case. Because the search space of high-order features is usually incredibly huge (even if we limit the order), none of the existing methods can enumerate all the high-order features within reasonable computational resources. We can hardly claim that high-order features are not useful for a dataset. However, we do not find that generating high-order features is useful for all the benchmarking datasets in experiments. In the IEEE competition, generating high-order features does not seem to be useful for neither Expert nor OpenFE. In the BNP competition, generating high-order features provides small improvement in
22

---

Under review as a conference paper at ICLR 2023
the test score. We may conclude that first-order features are usually more important than high- order ones in feature generation.
F COMPLEXITY ANALYSIS
Complexity of Generating Base Predictions. Let n be the number samples, m be the number of base features of dataset, and be the number of folds. Complexity of generating base predictions times of GBDT training,
O dm:Tdmm)
where each GBDT contains I trees with maximum depth d.
Complexity STAGE I. Suppose we split the dataset into 24 data blocks, the number of candidate features is m There are 2 2m features remaining after successive featurewise halving. The complexity is Σ 2(r-9) m² C(2 1), where C(n, m) is the complexity of GBDT training with data shape m. If the GBDT contains trees with maximum depth of d1, we have the time complexity
(2="9Tidinn))
Complexity of STAGE II. The complexity stage is dominated by a single GBDT training. Suppose the GBDT has T2 trees with maximum depth of d2, then the time complexity of stage II
O (2-9Tzdnmm²)
Overall Complexity. In the implementation of OpenFE, the number of trees and the maximum depth of GBDT are predefined fixed constant. If we regard T and d as constant, the overall com- plexity OpenFR is
O 9gmm²)
II complexity analysis, we refer to the implementation of LightGBM. The dominant term in complexity is bui building histogram, i.e., O(rm) per depth per tree.
23


Reviews:
[{"review_id": "VJTxtFfahzs", "reviewer": "Reviewer_mToT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an evaluation method to eliminate redundant candidate features following the expand-and-reduce framework for automated feature generation. The proposed method achieves state-of-the-art performance on seven benchmarks and outperforms human exports for the first time. Moreover, a theoretical analysis is provided to prove the advantage of feature generation over the base feature, strengthening this task's importance and facilitating future research.", "summary_of_the_review": "The method proposed in this paper is not the most innovative, but the authors contribute to this field by theoretically proving the advantage of feature generation. Overall, the paper is marginally below the acceptance threshold of ICLR2023. \n\n======================================================================================================\nThanks for the responses. My main concerns have been answered and resolved by the authors, including the completeness of datasets, the overall clarity, the complexity and runtime of OpenFE, and a detailed explanation of the Kaggle competition. Also, according to the supplementary results on 20 datasets, OpenFE achieves a better overall performance than FETCH. Considering the authors’ response, the contributions of open-source implementations, and some issues that did exist in the first submission, I have decided to change my recommendation to 6: marginally above the acceptance threshold.\n", "strength_and_weaknesses": "The main strengths of this paper are as follows:\n1. The author has done sufficient experiments, to illustrate the effectiveness of the proposed method. Also, by reproducing the main methods of this task, this paper provides convenience for future work.\n2. This paper gives theoretical proof for the advantage of feature generation over the base feature set. \n3. The proposed method is well explained and the overall logic is smooth.\n\nThe main weaknesses of this paper are as follows:\n1. The method proposed in this paper is still restricted to the existing expand-and-reduce framework, so the innovation of the overall method is limited. One possible optimization direction is to eliminate part of the features when constructing the candidate set.\n2. The writing of this article is a little redundant. As a result, the important ablation study section has to be put in the appendix.\n3. In the experiments section, the definition of the experts is not clear enough. Especially when claiming to outperform human experts, the exact definition of the comparison object needs to be specified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "OhEP681JJL", "reviewer": "Reviewer_kN28", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to automated feature generation which is split into two subroutines. After new candidate features are generated, in the first subroutine the set of newly generated features is pruned via an adaptation of successive halving which is dubbed successive pruning. In the second stage a boosting approach is followed based on the effect of a feature to reduce the loss of a learner.\n", "summary_of_the_review": "To sum up, I have the overall impression that the paper is not yet ready for publication, especially not at such a prestigious venue as ICLR. While the proposed method is reasonable and the overall idea is novel, the paper falls short on the justifications for design decisions, scope of the experimental study, and discussion of limitations. Significant parts are outsourced to the appendix making the overall content hard to grasp. All in all, I recommend to reject the paper but I would like to encourage the authors to further pursue their work and revise the paper since the paper would make a great contribution also with the open source implementations of other approaches.\n\n\nEdit after rebuttal: My concerns vanished after the authors gave their rebuttal along with more experimental results. However, I am still missing some discussion of the limitations which might be hard to spot due to the outstanding performance.", "strength_and_weaknesses": "[+] Interesting and novel approach to automated feature engineering.\n[+] In the (quite limited) empirical study the proposed approach seems to perform very well.\n\n[-] While the proposed approach is indeed very intriguing and it seems to be a reasonable approach (also with respect to observed performance), the paper falls short on providing theoretical justifications for the made design choices. More specifically, the proposed method is only described as it is but not much is stated about why the two stage approach was followed nor was there some motivation on the concrete techniques employed.\n\n[-] The empirical study is quite limited. In total, 7 datasets were considered plus some Kaggle competitions. While such a small number of datasets might be enough to demonstrate the existence of advantages, it is not enough to state whether it really outperforms all the state-of-the-art methods.\n\n[-] Moreover, since OpenFE is a heuristic, it most likely will have some shortcomings. No such shortcomings are observed nor discussed. In particular, no limitations of the approach are discussed throughout the paper.\n\n[-] Another open question is on base of which datasets the method was developed. If it was with respect to the evaluated datasets there might be a positive bias in the data.\n\n[-] Furthermore, it was not stated how the datasets were picked and why other datasets were not considered.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nkfcjoyjzq", "reviewer": "Reviewer_YEEM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a feature generation and model boosting methodology, dubbed \"OpenFE\". \n\nOpenFE proceeds by \n1. Generating a set of possible features, by ways of atomic operations (elementary operations, as well as \"grouped\" operations that apply mean/min/max operations over subsets of the data corresponding to the same categorical feature).\n2. Pruning the bulk of redundant features based on (Jamieson & Talwalkar, 2016).\n3. Ranking remaining features by their relative importance (the top k acting as the final set of extracted features).\n\nThe authors verify OpenFE on seven datasets, as well as by comparing OpenFE to competitors on a past Kaggle benchmark. \n", "summary_of_the_review": "This is an interesting paper, but the overall clarity of the proposed method is insufficient in the paper's current state. Although empirical results show the value in OpenFE, it is difficult to put them in context. ", "strength_and_weaknesses": "# Strengths\n- The paper is well written, and addresses a crucial problem in real-world ML applications.\n- Experimental results across several datasets show that OpenFE outperforms competing feature generation baselines, such as DCN-v2 and TabNet.\n- The authors consider a variety of ablation experiments to verify the importance of the different OpenFE components.\n- The authors reimplement and open-source baselines considered in this paper.\n\n# Weaknesses\n- I believe that the main weakness of this work lies in its overall clarity. The authors provide a high-level description of OpenFE, but spend too little time discussing the specifics. For example, \n   + How important is it to increase the highest order of potential features in OpenFE? How does this trade off with the overall runtime?\n   + Similarly, the authors provide a theoretical analysis of when feature generation can provably improve model performance. However, the assumptions that are made about the training data - crucial to the analysis and overall scope of the result - are left entirely to the appendix. I would recommend providing a quick summary of the assumptions and why they are required for the proof in the main paper.\n- The authors do not provide an analysis of the overall runtime of OpenFE, despite the potential size of the explored feature space and resulting algorithms being major impediments to automated feature extractions. I recommend the authors include some formal analysis, including the complexity of OpenFE, as well as (ideally) some guarantees on the two feature refinement subprocesses. Would it be possible to state anything in terms of how well Algorithm 2 approximates its optimal output?\n- The authors compare their works on a Kaggle competition, but focus on comparing to methods that did not rank first in that competition. Although this may be reasonable (algorithmic constraints, etc.), doing so must be justified. What is the difference between the model that placed 42nd and the model that placed first? Do I understand correctly that they were proposed by the same team?\n\n# Questions for the authors\n- Could you confirm that in Table 3, the OpenFE results describe Gradient-Boosted Trees with augmented features learned with OpenFE (but without using expert features)?\n- Could sub-components of OpenFE also be combined with existing feature generation methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject