Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a feature generation and model boosting methodology, dubbed "OpenFE". 

OpenFE proceeds by 
1. Generating a set of possible features, by ways of atomic operations (elementary operations, as well as "grouped" operations that apply mean/min/max operations over subsets of the data corresponding to the same categorical feature).
2. Pruning the bulk of redundant features based on (Jamieson & Talwalkar, 2016).
3. Ranking remaining features by their relative importance (the top k acting as the final set of extracted features).

The authors verify OpenFE on seven datasets, as well as by comparing OpenFE to competitors on a past Kaggle benchmark. 

strength_and_weaknesses: # Strengths
- The paper is well written, and addresses a crucial problem in real-world ML applications.
- Experimental results across several datasets show that OpenFE outperforms competing feature generation baselines, such as DCN-v2 and TabNet.
- The authors consider a variety of ablation experiments to verify the importance of the different OpenFE components.
- The authors reimplement and open-source baselines considered in this paper.

# Weaknesses
- I believe that the main weakness of this work lies in its overall clarity. The authors provide a high-level description of OpenFE, but spend too little time discussing the specifics. For example, 
   + How important is it to increase the highest order of potential features in OpenFE? How does this trade off with the overall runtime?
   + Similarly, the authors provide a theoretical analysis of when feature generation can provably improve model performance. However, the assumptions that are made about the training data - crucial to the analysis and overall scope of the result - are left entirely to the appendix. I would recommend providing a quick summary of the assumptions and why they are required for the proof in the main paper.
- The authors do not provide an analysis of the overall runtime of OpenFE, despite the potential size of the explored feature space and resulting algorithms being major impediments to automated feature extractions. I recommend the authors include some formal analysis, including the complexity of OpenFE, as well as (ideally) some guarantees on the two feature refinement subprocesses. Would it be possible to state anything in terms of how well Algorithm 2 approximates its optimal output?
- The authors compare their works on a Kaggle competition, but focus on comparing to methods that did not rank first in that competition. Although this may be reasonable (algorithmic constraints, etc.), doing so must be justified. What is the difference between the model that placed 42nd and the model that placed first? Do I understand correctly that they were proposed by the same team?

# Questions for the authors
- Could you confirm that in Table 3, the OpenFE results describe Gradient-Boosted Trees with augmented features learned with OpenFE (but without using expert features)?
- Could sub-components of OpenFE also be combined with existing feature generation methods?
clarity,_quality,_novelty_and_reproducibility: # Clarity
Although the paper is well written, the proposed OpenFE method is not described in sufficient detail within the main paper. Similarly, the experimental results are difficult to put into context, as the baselines and the extent to which they resemble OpenFE are not sufficiently described (are there baselines that also generate new features? Which baselines simply consider pruning the feature space)?

# Quality
Experimental results across a variety of datasets showcase that OpenFE outperforms the considered baselines by a significant amount, while remaining comparatively computationally cheap (cf. Table 8). 

# Novelty
This works combines previous work into separate steps that make up OpenFE. Although this is not as of itself a bad thing, the novelty could potentially be improved by further theoretical analysis and worst-case bounds as discussed earlier.

# Reproducibility
The authors provide the source code for OpenFE.
summary_of_the_review: This is an interesting paper, but the overall clarity of the proposed method is insufficient in the paper's current state. Although empirical results show the value in OpenFE, it is difficult to put them in context.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an evaluation method to eliminate redundant candidate features following the expand-and-reduce framework for automated feature generation. The proposed method achieves state-of-the-art performance on seven benchmarks and outperforms human exports for the first time. Moreover, a theoretical analysis is provided to prove the advantage of feature generation over the base feature, strengthening this task's importance and facilitating future research.
strength_and_weaknesses: The main strengths of this paper are as follows:
1. The author has done sufficient experiments, to illustrate the effectiveness of the proposed method. Also, by reproducing the main methods of this task, this paper provides convenience for future work.
2. This paper gives theoretical proof for the advantage of feature generation over the base feature set. 
3. The proposed method is well explained and the overall logic is smooth.

The main weaknesses of this paper are as follows:
1. The method proposed in this paper is still restricted to the existing expand-and-reduce framework, so the innovation of the overall method is limited. One possible optimization direction is to eliminate part of the features when constructing the candidate set.
2. The writing of this article is a little redundant. As a result, the important ablation study section has to be put in the appendix.
3. In the experiments section, the definition of the experts is not clear enough. Especially when claiming to outperform human experts, the exact definition of the comparison object needs to be specified.

clarity,_quality,_novelty_and_reproducibility: The overall writing of this paper is clear but a little redundant, since the method is mentioned too many times before being officially introduced. Still, the framework has been well explained and supported by the experiments. Moreover, the mathematical derivation is complex but logical.
summary_of_the_review: The method proposed in this paper is not the most innovative, but the authors contribute to this field by theoretically proving the advantage of feature generation. Overall, the paper is marginally below the acceptance threshold of ICLR2023. 

======================================================================================================
Thanks for the responses. My main concerns have been answered and resolved by the authors, including the completeness of datasets, the overall clarity, the complexity and runtime of OpenFE, and a detailed explanation of the Kaggle competition. Also, according to the supplementary results on 20 datasets, OpenFE achieves a better overall performance than FETCH. Considering the authors’ response, the contributions of open-source implementations, and some issues that did exist in the first submission, I have decided to change my recommendation to 6: marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach to automated feature generation which is split into two subroutines. After new candidate features are generated, in the first subroutine the set of newly generated features is pruned via an adaptation of successive halving which is dubbed successive pruning. In the second stage a boosting approach is followed based on the effect of a feature to reduce the loss of a learner.

strength_and_weaknesses: [+] Interesting and novel approach to automated feature engineering.
[+] In the (quite limited) empirical study the proposed approach seems to perform very well.

[-] While the proposed approach is indeed very intriguing and it seems to be a reasonable approach (also with respect to observed performance), the paper falls short on providing theoretical justifications for the made design choices. More specifically, the proposed method is only described as it is but not much is stated about why the two stage approach was followed nor was there some motivation on the concrete techniques employed.

[-] The empirical study is quite limited. In total, 7 datasets were considered plus some Kaggle competitions. While such a small number of datasets might be enough to demonstrate the existence of advantages, it is not enough to state whether it really outperforms all the state-of-the-art methods.

[-] Moreover, since OpenFE is a heuristic, it most likely will have some shortcomings. No such shortcomings are observed nor discussed. In particular, no limitations of the approach are discussed throughout the paper.

[-] Another open question is on base of which datasets the method was developed. If it was with respect to the evaluated datasets there might be a positive bias in the data.

[-] Furthermore, it was not stated how the datasets were picked and why other datasets were not considered.

clarity,_quality,_novelty_and_reproducibility: - The approach itself combines and adapts already existing methods to the setting of feature engineering. To the best of my knowledge the exact use and application of these methods is novel.
- While the language of the paper is mostly clear and quite readable, Section 4 could benefit from some proofreading. "has benefit provably.", "Due to space limit", "one may think each training data as a transaction", "[...] (each user may have many transactions in this table. These transactions form a group).".
- The variables max_order in Algorithm 1 and q in Algorithm 2 are not defined.
- For permutation feature importance a wrong reference is given: (Breiman, 2001). In this paper nothing is stated about permutation feature importance.
- Bold claims are made about outperforming human experts. However, but the pub. rank is never 1 in the results which I interpret as that there exist expert approaches that perform better and human experts are not outperformed. It is not clearly stated how this conclusion is drawn.
- Several parts of the paper are outsourced to the appendix. If I counted correctly the appendix is referenced 19 times throughout the paper. For me as a reader, this makes the paper quite hard to follow and I wonder whether a 9-page conference paper is the right format for this work.

summary_of_the_review: To sum up, I have the overall impression that the paper is not yet ready for publication, especially not at such a prestigious venue as ICLR. While the proposed method is reasonable and the overall idea is novel, the paper falls short on the justifications for design decisions, scope of the experimental study, and discussion of limitations. Significant parts are outsourced to the appendix making the overall content hard to grasp. All in all, I recommend to reject the paper but I would like to encourage the authors to further pursue their work and revise the paper since the paper would make a great contribution also with the open source implementations of other approaches.


Edit after rebuttal: My concerns vanished after the authors gave their rebuttal along with more experimental results. However, I am still missing some discussion of the limitations which might be hard to spot due to the outstanding performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents OpenFE, which is an automated algorithm for feature generation for tabular data.  The algorithm proceeds via two stages.  First the algorithm does "successive featurewise pruning" using a bandit approach (successive halving) followed by sensible heuristics to estimate the effectiveness of individual features and prune less effective ones.  Then there is a feature importance attribution stage that uses a variety of heuristics along with the MDI algorithm.  The authors show strong results across a variety of experiments including comparing OpenFE to experts in a Kaggle competition and a variety of related algorithms.  

Initial reviews were just below borderline, with reviewers finding that method likely performs quite well in practice, but found it a bit heuristic without understanding justification for some choices.  They raised questions regarding the empirical evaluation, and particularly related to the choice of comparisons used (e.g. they didn't understand why certain Kaggle experts were compared to and others, e.g. the top ranked, were not).  A lot of questions related to clarity.  The paper describes a pipeline of algorithmic choices and it seems reviewers had trouble following what choices were made exactly and why at different stages.  In personal discussion, one reviewer remarked that from the text they wouldn't feel like they would be able to reimplement the pipeline based on the writing.  Another question relates to novelty, it's not entirely clear what the authors' contribution is and what is attributable to gradient boosted decision trees and MDI (perhaps this relates to the clarity point above).  It seems like these are the two main components of the two steps in the pipeline?

In the response phase, the authors presented additional results and in particular compared empirically to another paper in review at this conference with a higher average score.  One reviewer was compelled to raise their score to 8, based on this comparison ("It would be a pitty if the better approach would not be accepted. I increased my score to suggest acceptance accordingly.").  While such strong results should be applauded, such a comparison seems problematic.  Raw empirical performance is only one dimension of many that should be considered for acceptance.  In addition, we cannot base scores on those of another paper for which the review process is incomplete (e.g. there are comments on that paper questioning the empirical evaluation there).  Therefore, the reasoning for raising the score will be downweighted in the decision.  The reviewers other comments "the paper falls short on the justifications for design decisions, ..., and discussion of limitations. ... making the overall content hard to grasp... etc." seem to be shared by the other reviewers and should be addressed.

Therefore, the recommendation is to reject the paper.  It seems that improving the clarity of the work, and addressing other reviewer comments would go a long way to convincing reviewers for a subsequent submission.
justification_for_why_not_higher_score: The paper presents a pipeline of heuristics and existing methods for feature extraction.  While the presented empirical results seem strong, the paper lacks clarity and as such it's unclear what the novelty is, what is attributable to the existing methods used in the pipeline and reviewers seemed to not understand the method in its entirety.  As such, it doesn't seem ready for publication.
Decision: Reject