Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors propose a differentiable AutoFE method, called DIFER, that is capable to produce high-order features. The general idea is to evolve a set of features, starting with the original features of the given dataset, so that in each iteration new features are added, either at random or based on the best performing features currently in the set. In the latter case, a so called "gradient-directed feature optimizer" is used, which consists of 1) an encoder that transforms a traversal string of the parse tree representation of a feature into a continuous representation (dense numerical representation of the string); 2) a predictor that estimates the performance of the continuous representation of the feature; 3) a decoder that transforms the continuous representation back into a traversal string of a parse tree. The main idea of this feature optimizer is to adapt the continuous representation of a feature, based on a gradient, until the predictor estimates a way better performance. Then the decoder is used to transform the continuous representation back into the parse tree of that feature. Note that the predictor is trained once in the beginning. This process is repeated until some maximum number of iterations is reached. The finally returned set of features is constructed based on the original features. Additionally, in each iteration the best performing feature from the previously grown set of features is selected for the final set, as long as there is improvement when adding the best performing feature.

The authors provide experiments of DIFER compared to 6 competitors, containing the effectiveness of DIFER, the efficiency of DIFER in terms of pipeline evaluations, the effectiveness of features having different (higher) order, as well as experiments covering different machine learning algorithms.
overall_review: Overall, the idea of the presented paper is very interesting. Unfortunately, the descriptions are often very short and for someone who did not see such a usage of an encoder decoder before, hard to understand. The images are missing important information, the ordering of the figures and tables is not reasonable. The evaluation is a good foundation, but the some experiment setups do not provide a fair comparison or do not cover all aspects or datasets. All in all, this paper seems to be not carefully finalized. 

The final decision will be dependent on the improvements on the methodology and evaluation (Sec. 3 and Sec. 4). If the concerns are tackled, I am willing to fully accept the paper. 
review_summary: Overall, the idea of the presented paper is very interesting. Unfortunately, the descriptions are often very short and for someone who did not see such a usage of an encoder decoder before, hard to understand. The images are missing important information, the ordering of the figures and tables is not reasonable. The evaluation is a good foundation, but the some experiment setups do not provide a fair comparison or do not cover all aspects or datasets. All in all, this paper seems to be not carefully finalized. 

The final decision will be dependent on the improvements on the methodology and evaluation (Sec. 3 and Sec. 4). If the concerns are tackled, I am willing to fully accept the paper. 


---
# Update after Rebuttal Phase

The authors addressed most of my concerns. Especially, changes have been made in Sec. 3 (Problem Formulation) and Sec. 4 (Experiments), which improved the correctness and clarity a lot. A section giving an overview of DIFER has been added, which helps to understand the overall approach, before going into details. The overview figure has been modified to correctly contain all key aspects of the approach. Some experiments have been added or modified to have a better evaluation of the approach and a better understanding of its results.  

Nevertheless, some descriptions or discussions are quite short and limited. Some experimental results are mentioned in discussions without having the results explicitly given in the paper. Further usage of the appendix would be suitable here.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors present a differentiable representation that improves evolutionary search for feature transformations.  The method combines a number of learning techniques into a single algorithm for automatic feature transformation learning.  Empirical comparison with other AutoFE methods suggest that their method (DIFER) yields more accurate feature transformations on a variety of benchmark datasets and is relatively efficient.
overall_review: The method proposed in this paper for AutoFE is interesting.  But it is also complex because it combines a number of learning algorithms in a complex system, and it is not clear what components of this system are critical for good performance, and which are not.  Overall it is difficult to understand exactly how and why the method works.  It would be good if the paper included concrete examples of the result of feature engineering and how different components of the system led it to devise those features.  The empirical results look reasonably strong, though there are some questions about the way computational performance is compared across competing methods.

The positive aspects of the paper are the use of a differentiable representation for AutoFE, and performing a fairly comprehensive empirical comparison.  the negative aspects of the paper are the clarity with which the method is explained and justified, and the lack of concrete examples that would further motivate the algorithm and the results of AutoFE.
review_summary: An interesting AutoFE method based on a differentiable representation that appears to outperform existing AutoFE methods.  Although it is not clear exactly how the components of this algorithm work together to generate those results, the results look good.  Perhaps if the paper did a better job explaining why and how the algorithm works, it would be easier to recommend it.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors propose an algorithm for automated differentiable feature engineering.
The algorithms utilize a basic evolutionary framework hybridized with and encoder-predictor-decoder neural network for feature optimization, which is trained via gradient descent, making the search space effectively differentiable.
overall_review: The paper is well written and structured with a well-maintained code repository allowing for easy reproducibility.
The authors propose an innovative, state-of-art-performance achieving method for automated feature engineering.
However, the evaluation regarding the efficiency of the model is lacking, and ethics statement is missing in any capacity.

review_summary: Overall, I really enjoyed reading the paper. It provides a novel and interesting approach to a relevant problem while providing state-of-the-art performance.
The remaining concerns I raised during this review can be addressed easily in the next phase of the review.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper introduces a novel approach to automated feature engineering. In order to tackle the computational complexity of finding the appropriate sequence of data transformations, they apply the method of mapping the discrete decision space to a continuous space. This allows to scale the search for which features to add. They show how their approach performs in various experimental evaluations.
overall_review: The approach employed here is based on the technique employed in Neural Architecture Optimize (NAO) as cited by the authors [12]. DIFER encodes the features into a continuous space. NAO encodes the neural network architecture into a continuous space. Both perform gradient descent in the direction to increase the predictors accuracy and the decoder is used for stability so that the encoder is embedding the features correctly. Then the loss function is also the same for both DIFER and NAO: reconstruction loss + predictors RMSE loss. There are some differences as the authors pointed out in some other rebuttal, and I think applying a technique in a different context is somewhat ok.

And credit to the authors, they did address quite a few issues over time in this paper (such as adding the number of added features in the table; trying different models). However, one other point that was made by various reviewers was the runtime complexity. It is fine to show the number of iterations, but they are not as revealing as the actual run time for the proposed method. Would it be possible to just add another column on runtime? I do not expect the numbers to be much worse than the ones obtained by other approaches such as RL based methods. Of course, an approach like LFE has a clear runtime advantage but also disadvantages.
Overall I think it would be really useful to expose the computational complexity of the method.

To me there is also still a bit of a lack of understanding of encoder/decoder and how all the pieces with together. Yes, I think I do understand it in general - maybe you need a concrete example that shows what exactly happens at each major component such as the embedding space and how to encode and decode? Maybe one issue is that there are so many parameters on how to create the parse tree, extend it etc, that it just becomes difficult to have a clear picture. And maybe it is just mostly performance driven anyway and that is just how things are selected in the end. 
Do you also know how accurate the predictions by the MLP actually are?

One other generic question I had is on why those rather simple transforms cannot be learned by an estimator? For instance, it is always claimed that Neural Networks learn their own feature representation - but they cannot learn a product and sum transform? Anyway, that is not really a question that is important for the acceptance of this paper. This brings of course the idea to mind to based on your work create an end to end NN that selects transforms but also makes the actual predictions at the end.

review_summary: The paper does adopt a method from NAS for AutoFE and is over all interesting.
I do still lack a bit of clarity on the approach including its computational complexity.
review_title: null

Meta-review: metareview: The reviewers have been a bit divided on this paper, and there was plenty of discussion. I do think that the paper could use some improvement, specifically about one concern risen on the explanation *why* the method works, which is also a concern to me. However, the topic is interesting, and reviewers expect some impact of this work.

In accordance with the reviewers, I consider the paper borderline. It is fair (and I would suggest) to accept it, but in its current shape it is clearly not a must-have for the conference.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept