Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper "Depth is More Powerful than Width in Deep Forest" studies the behavior of Deep Forest and shows that Deep Forests are consistent with a rate of $\mathcal O(1/M^2)$ where $M$ is the number of trees in the forest. To do so, the paper studies two-layered DF and its convergence in two different scenarios, one in which the number of data points is much larger than the number of leaves and  second when they are comparable equal. The resulting two Theorems establish the consistency of DF. Last, another Theorem establish the convergence speed of DF in terms of the number of trees which can be interpreted as "depth is more powerful than width". Experiments on synthetic data further supports the claim 
strengths_and_weaknesses: Overall I enjoyed reading the paper and I cannot criticize much about it. It is generally well-written and discusses an important topic. Deep Forest is arguably one of the more modern ensemble approaches (Ensembles of Neural Networks aside). While some theory has been developed for DF, it is not fully explored yet and hence the topic is of great interest. The paper heavily relies on results established for Random Forests, but extends them in a meaningful and non-trivial way. While the results are consistent with what one expect intuitively, I think it is important that they are tackled by rigors theory. Surprisingly, the results in the paper seem to be quite close to the practical application of DF. In summary:
- (+) Extends known results about Random Forest to Deep Forest thereby explaining some of the success of DF
- (+) Comparably easy to follow for such a theoretical paper. I was especially happy about the remarks and proof sketches, although I admit that I did not follow all proofs in the appendix. 
- (-) I think a discussion on DF with more than two layers would have been interesting, although it seems straight-forward to apply Theorem  2 and 3 recursively (see question below)
questions: - It seems that I can apply Theorem 2-4 recursively if I have a Deep Forest with more than layers. Is this correct, and if so, does this improve the convergence rate to $\mathcal O(1/M^3), \mathcal O(1/M^4), \dots$ with each layer? This seems like an unusual strong result. Is this correct?
limitations: I think the limitations are addressed adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, authors analyze the relationship between the depth and the number of trees of the deep forest. In recent years, deep forest has been widely studied due to its usability that combines the deep neural network with random forests. The authors validate their analysis through their experiments of the deep forests.
strengths_and_weaknesses: Strength
+) Motivation and background studies are sufficiently provided.
+) Appropriate assumptions and mathematical derivations for analyzing deep forests are included.

Weaknesses
-) Analyzing the depth and the number of trees of the deep forests is limited to claim the novelty.
-) There is a lack of experiments with popular tasks and datasets, such as in computer vision or NLP domains.

----- after rebuttal -----

In rebuttal phase, authors revise their papers according to the comments. I understand the importance of the deep forest and the author's finding that its depth is more important than its width. However, I'm not sure if this finding would apply to real problems. In the revised paper, authors introduce new results on several real-world datasets and compared it to a random forest. However, it is necessary to show how much the existing SOTA results in the housing dataset can be increased by the authors' finding. For example, if this finding is applied to previous deep forest based methods, there should be  performance improvement compared to the baseline deep forests.

Considering the author's response, I increase the score to borderline accept by understanding that the authors have responded to reviewer's comments sufficiently, and that their findings are important.
questions: Q1) Can you explain further why the analysis of the depth and the number of trees of the deep forest has a large impact? A more thorough explanation of why this analysis of deep forest is important should be added.

Q2) Does the experiment shown in this paper apply well to real vision or NLP applications? It would be nice to show the experimental results on more widely used task and dataset, but if that is not the case, authors need to provide evidence that the analysis in this paper can be applied to other applications.
limitations: There is no explanation for the limitations of the paper. Appropriate limitations should be included.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a theoretical analysis of deep random forests. Deep random forests stack layers of Breinman's random forest combined to completely random forest. The output of each layer in the regression setting is the average label of all sample falling in each leaf. Each layer takes as input a concatenation of the input features as well as previous features output by previous layers.

The authors focus on the two-layer deep forest and show multiple theoretical results aiming at explaining empirical performances.

Proposition 1 shows that stacking multiple layers make sense as the latter focus on the new features thus improving the regression function. Proposition 1 is proved in the infinite and finite regime (in the sample size).

Proposition 2 focuses on the variation of the true regression function f. The authors show that the variation of the regression function within a single cell is bounded in probability for any threshold and probability given that the sample size is large enough.

Theorem 3 recall the consistency of traditional random forests and states that the two layer deep forest are consistent as well. (given that the sample size per tree grows faster than the number of leaves which as stated by the authors is not what is usually done in practice).

Theorem 4 shows convergence speed to the infinite forest for the classical Breiman's forest and the two layer random forest. The former has a convergence rate of O(1/M) while the latter has O(1/M^2).

Concerning assumptions 1 and 2. Assumption 1 is a not too restrictive assumption probably used for simplicity in the proofs. Assumption 2 is less obvious to get at first sight and aims at showing that cells become less and less influenced by couple of samples labels and noise while the sample size tends to infinity.

Experiments on simulated data tend to confirm the theoretical claims and theorems of the paper.
strengths_and_weaknesses: - Strength
The paper is easy to read even for non-experts. Objects are well defined,
The paper proposes a theoretical justification for known empirical claims.
Deep forest is a recent contribution for which theoretical evidences lack and this paper contributes to the literature.

- Weaknesses
The assumptions, required for the proof, lack some analysis about their application in real life problems as well as motivation for their use.
questions: Would it be possible to give more elements about all the assumptions. Mainly:
- Do they apply in real life problem,
- What would we need to maintain the theoretical results while relaxing the assumptions ?

Would it be possible to evaluate experimentally the limit of these asumptions?

Equations 14 and 15 seem to have parenthesis problems.
limitations: Limitations of the work are clear from the core of the article but not particularly addressed in a specific area of the paper.
review_title: null

Meta-review: metareview: This paper establishes a set of theoretical results for characterizing the behavior of Deep Forest, an important model in deep learning. The analysis approach is novel and the results are of significant importance. The majority of the reviewers appreciate the authors’ contribution and all reviewers recommend acceptance. Thus I would strongly recommend accepting the paper. 
confidence: Certain
Decision: Accept