Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a first of its kind architecture framework for automated feature engineering called Fetch, a system based on brand new data driven Markov decision process. The authors identify critical gaps by stating the current methods for AutoFe are insufficient when it comes resembling human effort in handling datasets as the underlying Markov decision setup is different i.e. more based on trial and error, which leads to poor generalization across datasets and higher computational costs. The new method also has a key element of transferability, which is basically the ability to enable feature engineering on new datasets using prior policy networks trained on previous datasets, without the need for additional data exploration. The authors present evidence that Fetch is superior to the existing state of the art automated feature engineering methods such as Random, DFS, AutoFeat, NFS, DIFER as well as AutoML methods such as AutoSklearn and AutoGluon. The method is also tested for transferability by application on several datasets. 

The authors argue that the approach comes very close to mimicking human experts when it comes to handling new datasets when it comes to transferring experience. 
strength_and_weaknesses: Strengths: 
1. One of the key strengths of this paper is its ability to provide transferability between datasets which previous methods have not been able to provide. 
2. The method is highly flexible as it has been applied on different datasets as well different ML models. 
3. Overall there is sound discussion relevant work, gap analysis to find opportunity of development, strong mathematical rigour and experimental setup, evaluation and analysis. 


Weaknesses: 
1. The authors declare that to the best of their knowledge there aren't any autoFE/autoML workarounds for managing tabular data to accomplish transferability. However this could be supported by a stronger statement that can more concretely say if such methods exist or not by clearly stating a comprehensive survey of analysis did not yield any methods. 
2. This method is specifically geared towards tabular datasets. While tabular datasets prevail in several key applications, it would be interesting to know how this method applies/does not apply to any other form of datasets (eg. images, speech, unstructured datasets). At least there could be some discussion on consideration, if not a full scale evaluation.  

clarity,_quality,_novelty_and_reproducibility: Novelty: 
1. Paper proposes a new form of markov decision process to generate features. Previous methods take number of features and only update using sequences of previous actions, iteratively. Whereas Fetch provides Feature engineering actions and constructions actions, instead of features, iteratively based on the newly generated datasets (features). 
2. According to this paper, Previous methods for AutoFE have never explored transferability between datasets, which has been done by this method.


Clarity & Quality: The paper has high clarity with explicit background provided and methods explained with sufficient depth both in terms of mathematical equations, datasets and analysis of results. There is further in-depth explanation in the appendix section around usage of material and datasets which is crystal clear. 


Reproducibility: The method has been extensively tested on publicly available datasets which are linked. Algorithm and methods have been provided as well. The paper states that all experiments have been run with open source code provided.   

summary_of_the_review: I propose that we accept this paper on the basis of its merits around developing a new AutoFE method based on a novel markov decision process which not only is superior to the existing state of the art in terms of performance but also has a unique feature of transferability. Both strong points are well argued and evidenced in the paper throughout with sufficient gap analysis around opportunity areas, discussion of background and related work, strong mathematical rigor and sound experimental design setup, with thorough data analysis, discussion of evaluation and results. Although the work is limited to tabular data there is thoughtful coverage on several popular datasets including regression and classification examples and comparison to other AutoFE models and coverage across several ML models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Automated feature engineering is another needed ingredient in 'data science as a service.' Prior work focuses either on feature selection or feature engineering that is independent on the underlying data. The authors propose an RL algorithm that directly transforms data by applying a sequence of unary or binary operations. 

strength_and_weaknesses: Strengths: 
The idea of using data directly is novel. 
The use of data enables transfer learning. 

Weaknesses: 
Scalability is questionable. There is no deep discussion on computational times and comparisons with other algorithms. 
Explainability/interpretability is questionable. To the contrary, the resulting training data is not interpretable. 
clarity,_quality,_novelty_and_reproducibility: The paper is in general well written. There are a few loose points but nothing major. 
It is easy to read and comprehend. 
Reproducibility is questionable. There are not many details about how to reproduce the results. 
summary_of_the_review: The underlying idea of using the actual data in the RL setting is very novel. It is a substantial contribution. In my opinion it outweighs deficiencies in the work. 
Once the RL is established (the state space), everything else is standard (the network, algorithm). 

The authors don't mention feature selection (forward/backward selection, etc) which is perhaps injustice. Maybe feature selection should also be benchmarks (perhaps autoML methods include them). 

Minor remarks: 
Page 1: next to the last paragraph states that there is significant prior autoFE work. This statement contradicts some prior statements stating there is not a lot of work in this area. 
Hyperparameters being fixed is questionable since different subset of features definitely dictate possible different hyperparameters. I understand that this inclusion would be a significant complication and thus I don't take this against the authors. 
Page 3: 'specified feature x' excludes binary operations. This statement should be reworded. 
Page 4: weightheight^2 should be weightheight$^2$
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper firstly specifies the fact that AutoFE is underappreciated in the AutoML pipeline tailored to tabular data nowadays, and summarizes that previous AutoFE work faced two major drawbacks of unobserved data and lack of transferability. To tackle these problems, this paper proposes a novel RL-based AutoFE framework(called FETCH) by observing the behavior patterns of human experts, which is equipped with a data-driven MDP setup in its policy network. The key idea is to consider each table as a set of features and incorporate the table into the MDP state. Then FETCH learns how to map from data to feature engineering actions. Experiments show that FETCH outperforms other SOTA or popular feature engineering baselines and supports transferability by pre-training.
strength_and_weaknesses: Strengths:
1. The proposed framework is novel, simple and effective, which is easily to follow. The pre-training technique seems like a new schema in the tabular data mining field. 
2. The paper is well-written, and the related work is covered well.
3. The experiments are quite comprehensive. The authors also show that pre-training FETCH on more datasets results in additional performance gains, which further strengthens the claim of the model's transferability.

Weaknesses:
1. It is nice that the authors include 27 datasets as a benchmark and demonstrate that FETCH is much better than the other AutoFE baselines in most cases. However, I don't quite sure why the authors are comparing it to popular open-source AutoML packages (i.e. AutoSklearn and AutoGluon)? After all, these are two different domains. Is it to prove that AutoFE can power AutoML? Maybe I'm missing it, and please clarify.
2. Section 4.4 only covers 3 comparison algorithms, it would be better to add the enhancement effect on more algorithms such as LightGBM and CatBoost, which will be more convincing for the adaptability of FETCH on various algorithms.
3. FETCH seems to have been pre-trained on only 5 datasets, which seems somewhat simple. It would be preferable to have pre-training results on a larger number of datasets.
4. The theoretical analysis is a bit lacking. It is hoped that a theoretical analysis of effectiveness and experimental results can be provided as space permits. 

clarity,_quality,_novelty_and_reproducibility: This paper is overall well-written and clear. The code in the supplementary material seems to be well-organized and I believe it works as the paper claims, although I have not actually verified it.


summary_of_the_review: The idea of learning a data-driven policy network is well-motivated and novel, and the experiments are compelling.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors proposed a new automated feature engineering framework. The proposed framework can be transferred across different datasets, which is interesting. Empirical results are promising. One potential limitation is that the proposed framework may not be applied to different application domains beyond tabular datasets. Though the authors added some discussions about how to extend the framework to non-tabular data, they did not provide empirical evidence that the proposed framework can perform well on non-tabular data.

In summary, this is a good paper for ICLR-23.
justification_for_why_not_higher_score: No theoretical analysis is provided regarding the transferability of the proposed automated feature engineering framework.
justification_for_why_not_lower_score: The idea is interesting. The empirical results are promising, especially for the transfer learning setting.
Decision: Accept: notable-top-25%