Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this research, for the task of tabular data learning, an algorithm that searches for features used and the base learner used jointly has been proposed. Within the research, it has been found that on a large number of different tasks, the method outperforms the existing baselines.
strength_and_weaknesses: Strengths:

1- This is a pioneering work that performs a search on features and learning models jointly (in the field of tabular data learning).

2- A solid number of experiments have been performed and the proposed algorithm outperforms various types of existing baseline models.

Weaknesses:

1- Some background information requires more detail (e.g. TGP, feature importance value). A more in-depth introduction to their design and their main characteristics will make the paper more throughout.

2- The influence of some design choices has not been analyzed empirically. For example, the design decision of using “self-competitive crossover” has not been analyzed.

clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality of the paper are sufficiently good.
summary_of_the_review: Although there are some slight limitations in the paper, but the idea of “searching features and learning model jointly” is effective and the experiments are solid.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper points out that neural-network-based features can be easy to over-fitting and proposes a an evolution-based feature engineering algorithm to imitate the manual feature construction process through trial and improvement. 
strength_and_weaknesses: Strength:

S1. The author illustrates that in the tabular data classification, the selected feature based on deep learning is prone to overfitting, and the proposed method based on the evolutionary algorithm can obtain more robust and better results.
S2. The proposed method demonstrates promising results compared with various baselines (e.g., ML-based, DL-based, ...).

Weaknesses:

W1. The technical novelty of this paper is limited. There is no novel technique has been proposed or discussed in this paper.
W2. In the real world, tabular data is usually very large-scale, and as the authors say in Section 5.1, they use part of the data due to limited computational resources. Therefore, an efficiency analysis (e.g., execution time, memory) of the proposed method is necessary, and a comparison with other baselines can also be added to comprehensively evaluate the applicability of each method.
W3.The insights for the proposed method are not clear. It would be better to provide a more explicit motivation and vital support for the proposed approach than to beat some approaches only.
clarity,_quality,_novelty_and_reproducibility: This paper cannot bring any new technical insight to the community.
summary_of_the_review: This paper focus on an interesting problem and gives some experimental analysis. However, The technical novelty of this paper is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the feature engineering problem for tabular data classification. It presents a new evolutionary algorithm for performing feature engineering. Each feature is generated by a feature tree, whose leaves are features and whose internal nodes are operators. The number of features is a hyperparameter. The evolutionary search process, finding those trees, is guided by the feature importance and training losses, computed by two base learners: logistic regression or decision tree. The best models at each stage are stored, and the final answer is their ensemble. Ablation study shows that considering both logistic regression and decision-tree is better than using just one of them.
The new method is applied to 119 PMLB datasets (with up to 10K samples each), and compared to two previous feature engineering methods, six classical ML methods and 4 deep-learning methods. The proposed method out performs all previous methods, with an advantage of about 0.8% average accuracy over the next best model, LightGBM. Previous work considered evolutionary algorithms for regression or image classification, but not for tabular data classification.

strength_and_weaknesses: The paper presents a new method for an important problem, and it is reasonably clear despite the space constraints. I think it would be interesting to see how sensitive are the results to the hyperparameter selection of the baselines compared. Specifically, what would be the results of the two top baselines (LightGBM and XGBoost) when simply using their default hyper-parameters? What would happen if the maximal number of estimators in the hyper-parameter search is allowed to be 5000 instead of 1000?
Also, the runtime measurements are not a main aspect of the results, but they are not fully clear (e.g., did each model run on a separate server, did all the servers have identical RAM, etc.).
clarity,_quality,_novelty_and_reproducibility: - The paper is quite clear, although it requires acquaintance with cited papers to understand what was done. The runtime measurements are not a main aspect of the results, but they are not fully clear (e.g., did each model run on a separate server, did all the servers have identical RAM, etc.).
- It does represent a novel method to use this type of feature engineering for tabular data classification.
- The provided code should enable reproducibility and the data is public. A question I have about the code is regarding the tests, which seem to relate to regression rather than classification. What is the reason for this? How was classification tested and how should it be run?
summary_of_the_review: This is an interesting paper that suggests a new method for an important problem. My main concern is whether the accuracy improvement could be due to non-optimal selection of hyper-parameters for the baselines. I suggest to try the default hyper-parameters for the two top baselines (XGBoost and LightGBM), and also try a hyperparameter search with more estimators (up to 5000). I also would like to have a clarification about the run-time measurements, as I mentioned above, and about my question above about the code. I will update my recommendation based on the author responses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduced an evolutionary algorithm to do feature engineering, while the neural network based features can be easy to overfitting. The proposed algorithm has been clearly introduced and the authors have compared the proposed algorithm with various baselines (e.g., ML-based, DL-based, etc). But its limited technical novelty could be a major weakness. 
justification_for_why_not_higher_score: The authors may have overclaimed the significance of the proposed study here. The authors tried to explore the case with heterogeneous features and a limited amount of data, but that is not the scenario for ordinary deep learning. The authors proposed an ad-hoc approach to select or design features for the classifiers, but they did not take the literature on feature selection, metric learning, or dimension reduction into account, which makes the current experimental design/comparison incomplete or unfair.
Decision: Reject