Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new method to initialize neural networks using pertained tree based models for tabular related data. The new proposed framework includes two components: 
- [1] As proposed by the prior work 'Neural random forest', 2019, tree based models can be converted into a 3-layer MLP.
- [2] Design the non-linear function with proper selection of parameter to fit into gradient descent training framework.

Numerical result shows that MLP with tree based initialization method converges better than random initialization. Parameters distribution analysis also shows model trained with tree based initialization converges to a solution as compared to random initialization.
strength_and_weaknesses: Strength
[+]  Paper is well organized and well motivated. Literature is comprehensive and overall paper is well organized.

Weakness
[-] Novelty seems limited. The algorithm and contribution of the paper is mainly on developing a schema with neural network to handle tabular data and the major source of improvement is from tree based initialization and fine tuning process, which mostly seems to be obtained from Biau et al. (2019).

[-] Some details might need more clarification. 
Could the author provides an example or demo graph to illustrate more clearly on what the initialized the neural network look like. Figure 1 is obtained from the original paper to show how tree based model can be converted to neural nets in a intuitive way. Could some demo example listed here to more precisely show the initialization step? Same applies to section 2.3 on the details to convert the model to allow gradient descent training.

[-] Experimental results seem weak.
From the results in table 1, it seems MLP initialized with GBDT still lower perform than GBDT along. Is there any analysis on the source of changes, considering the neural networks is fine tuned on top of GBDT models? 

clarity,_quality,_novelty_and_reproducibility: Overall, quality and reproducibility are good. While clarity and novelty of the work can be improved. 
summary_of_the_review: The paper considers an interesting and important topics on building a neural network schema to handle tabular data. Initialization with tree based model with designed fine tuning process is reasonable sound. However, the novelty of the paper seems limited since majority of the work seems to be inherited from Biau et al. (2019). Experimental results seem weak as compared to the baseline models. Although there are some interesting findings on the converged models, while there is not theoretical analysis on those behavior.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to initialize MLP neural networks when working with tabular data. The method uses tree-based techniques to initialize the weights of the NN. The authors argue that using this initialization in the first layer of an MLP is sufficient to improve its performance (the initialization of the other layers is random).
strength_and_weaknesses: + The proposed approach is interesting and presents promising results. 
+ The idea of using decision trees to capture relevant features and their interactions and define a mapping to encode extracted relationships into a neural network is not new. Still, the authors presented a focused approach and interesting experiments to justify its use.

- The paper suggests the proposed approach operates in an implicit regularization during the NN training. However, I’m not convinced that it was entirely demonstrated in the experiments, results, and analysis.
- The authors compared the proposed NN initialization approach with random weights initialization. However, this is an active research area, and classical approaches should be considered for comparison:
   - Understanding the difficulty of training deep feedforward neural networks, Glorot and Bengio, 2010
   - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, Saxe et al., 2013
   - Random walk initialization for training very deep feedforward networks, Sussillo and Abbott, 2014
   - Data-dependent Initializations of Convolutional Neural Networks, Krähenbühl et al., 2015
   - All you need is a good init, Mishkin and Matas, 2015
   - Fixup Initialization: Residual Learning Without Normalization, Zhang et al., 2019
   - The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, Frankle and Carbin, 2019
- The authors do not consider the computational cost of training the tree-based methods to initialize the NN model. 
- What is the performance of the proposed approach on deeper models or using different activation functions? Please provide more comments on that.
- The authors did not discuss the limitation of the proposed method. Therefore, it will be meaningful to discuss the gap between the experiments in the current version of the paper and the real-world applications.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and logically structured. Unfortunately, the code is unavailable, making it difficult to reproduce the experiments.
The idea of using tree-based methods to initialize NN models is simple but seems effective considering the paper's results. However, the idea is not entirely new, and the authors should compare the proposed model with other approaches and with other initialization methods. Therefore, the comparison with random weight initialization seems not fair. 

summary_of_the_review: The proposed paper presents a promising approach with interesting results and analysis. The authors focus on comparing the proposed method with random initialization methods but should consider other initialization approaches to compare. The author also should discuss the proposed approach's limitations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors propose a method to 'translate' a Decision Tree (DT) into a Multi layer perceptron (MLP). They use this method as the basis for a Neural Net (NN) weights' initialization technique that allows them to considerably boost NN's performance on tabular (unstructured) data. Authors provide a thorough set of experimental tests where they compare the performances of their proposed approach to that of standard (random) NN init, as well as against the performances of different DT-based methods and another NN architecture aimed at  tabular data. The results obtained back the validity of their method.
strength_and_weaknesses: - The paper is very well written in structure, language and style, with only minor issues here and there.

- The proposed method is very interesting, although remarkably simple. Authors do a very good job explaining it, nonetheless.

- Experimental evidence provided is comprehensive, and provides good empirical support for the proposed approach.

- Results are highly encouraging to continue this line of research, where different ML algorithms combine and blend, in simple and very effective ways.

- Authors also do not fall short in providing discussion regarding all aspects of their works (inner workings and behavior of their methods, results, consequences and possibilities resulting from their  work).
clarity,_quality,_novelty_and_reproducibility: These are some suggestions that I'd like make to the authors in order to further improve their draft:

- In Related Works section, please make emphasis and/or describe briefly how your approach herein proposed for transforming a DT into a MLP differentiates from methods previously found in literature. In this section you mention that these _translations_ have been proposed as early as the 90s; since this is such a critical part of your approach (and also because is a fundamental good writing style) I think you should make clear as soon as possible how this new translating mechanism differentiates from other approaches previously proposed. Otherwise, novelty might be put into question.

- It would also be good if you discuss some of the motivations that led to your research. From the wording in the abstract and intro, I got the feeling that you consider neural nets shall "reign supreme" overall other ML methods, and if they lose against any other ML paradigm in certain areas of application, then we shall bring elements of such other family of methods to deep learning, until they are "the one ML method to rule them all"... that'd be a weird line of thought. Think of the following two questions => "Why cannot NN be as good, or even better, than tree methods in this or that kind of problems?" vs "How can we enhance the performance of NN by taking elements of other ML paradigms?" which one sounds more natural to you? Just a bit of rephrasing in abstract and intro would do the trick in my opinion.

- Did you consider discussing the implications of your research regarding the capabilities and limitations of gradient methods and the overall optimization complexity of neural nets? I think you might be sitting on something beyond just an initialization technique, but rather further evidence that gd is not as good as previously thought, which has been a recent line of research (e.g. lottery ticket hypothesis, etc.).
summary_of_the_review: In general, I think this paper is very good. My only doubt is how similar is this DT-MLP transformation technique to approaches previously proposed in the field. Other than that, I think the paper is well written and presented, experimental evidence and results check, and novelty and significance are good enough for ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP) towards tabular data. The method uses tree-based techniques to initialize the weights of the NN. The authors argue that using this initialization in the first layer of an MLP is sufficient to improve its performance. 

Reviewer c3Rb did not actively engage in the post-rebuttal discussion despite multiple nudges. The other two reviews remain positive. The AC also takes a close look at the paper & comments. The paper is very well written and easy to follow. The proposed method is simple and interesting. 

Two main concerns were raised by reviewers: 
(1) similarity to prior work  (Biau et al., 2019). The authors clarified that, unlike this prior art, they do not try to mimic exactly the tree behavior,  but instead remove the last layer of tree translation at initialization and add extra layers to perform prediction based on the tree-based feature extractor. 
(2) underwhelming performance. The authors added new baseline results with various architecture + initialization ways, and show they are all outperformed by the tree-initialized MLP on all but one data set

justification_for_why_not_higher_score: The innovations are moderate compared to prior arts; so are the performance improvements.
justification_for_why_not_lower_score: In general, I feel this paper is done solidly, though not particularly exciting. It might be of interest to the specific community
Decision: Accept: poster