Depth is More Powerful than Width with Prediction Concatenation in Deep Forest
Authors: Shen-Huan Lyu
Yi-Xiao He
Zhi-Hua Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: ensemble learning
deep forest
consistency
convergence rate

Abstract: Random Forest (RF) is an ensemble learning algorithm proposed by \citet{breiman2001random} that constructs a large number of randomized decision trees individually and aggregates their predictions by naive averaging. \citet{zhou2019deep} further propose Deep Forest (DF) algorithm with multi-layer feature transformation, which significantly outperforms random forest in various application fields. The prediction concatenation (PreConc) operation is crucial for the multi-layer feature transformation in deep forest, though little has been known about its theoretical property. In this paper, we analyze the influence of Preconc on the consistency of deep forest. Especially when the individual tree is inconsistent (as in practice, the individual tree is often set to be fully grown, i.e., there is only one sample at each leaf node), we find that the convergence rate of two-layer DF \textit{w.r.t.} the number of trees $M$ can reach $\mathcal{O}(1/M^2)$ under some mild conditions, while the convergence rate of RF is $\mathcal{O}(1/M)$. Therefore, with the help of PreConc, DF with deeper layer will be more powerful than the shallower layer. Experiments confirm theoretical advantages.

Depth is More Powerful than Width with Prediction Concatenation in Deep Forest
Shen-Huan Lyu, Yi-Xiao He, Zhi-Hua Zhou National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China. (lvsh, heyx, zhouzh ) @ @landa... nju. waw..dd..c. cn
Abstract
Random Forest (RF) is an ensemble learning algorithm proposed by Breiman [1] that constructs large number of randomized decision trees individually and aggre- gates their predictions by naive averaging. Zhou and Feng [2] further propose Deep Forest (DF) algorithm with multi-layer feature transformation, which significantly outperforms random forest in various application fields. The prediction concate- nation (PreConc) operation is crucial for the multi-layer feature transformation in deep forest, though little has been known about its theoretical property. In this paper, analyze the influence of Preconc the consistency of deep forest. Especially when the individual tree is inconsistent (as practice, the individual tree is often set be fully grown, i.e., there is only one sample at each leaf node), we find that the convergence rate of two-layer DF w.e.t the number of trees M can reach (1/M²² under some mild conditions, while the convergence rate of RF is O(1/M), ). Therefore, with the help of PreConc, DF with deeper layer will be more powerful than the shallower layer. Experiments confirm theoretical advantages.
Introduction
Random forest (RF) [1] is a state-of-art classification and regression algorithm that constructs number of randomized individual decision trees during parallel training phase and predicts by naive averaging the results. Recently, Zhou and Feng [2] propose Deep Forest (DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-based representation learning. Benefiting from the multi-layer feature transformation in the cascade structure with prediction concatenation (PreConc), DF outperforms various tree-based algorithms [1, 3-5] in empirical study, and have been involved in real applications such as medicine [6], computer vision [7]. remote sensing [8], financial fraud detection [9], etc. Numerous variants have been extended to various tasks and meet with remarkable success in [10-13]. There are also variants [14, 15] aiming improving performance and reducing computational cost.
Empirical successes have attracted attention to the theoretical analysis of DF. Since its PreConc operation transforms features layer by layer, and each layer consists of complex non-parametric
random forest estimators, analyzing the impact of the new features extracted from these estimators on generalization performance is important. To start theoretical analysis on DF, Lyu al. [16] prove margin-based generalization bound for additive new features in DF. Then, Pang et al. [14] prove an upper bound on the sample efficiency and inspire an efficient improvement for reducing computational cost of DF. In the direction of consistency, Arnould et al. [17] provide tight lower and upper bounds on the excess risk of a shallow centered random tree network, which leverages the new features improve the performance of single centered random tree. Assuming that the original and new features are separated and independently used in two stages, Lyu et al. [18] prove that the new features based prediction easy to cause overfitting risk, and propose to use the interaction of decision rules to alleviate it.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, previous theoretical studies often choose to ignore the impact of the new features on generating forests. For example, recent works [16, 18, 14] assume that the forest estimator in each layer is a black-box model and Arnould et al. [17] assume that there is only single centered random tree in each layer. Such strong assumptions are in favor of theoretical analysis, but widen the gap between the theoretical results and practical deep forest architecture. To properly analyze DF, we must notice that, among the complex architecture in DF, both prediction concatenation (PreConc) [16] and the classification and regression trees (CART)- -pliit criterion [19] play critical roles. Therefore, in order to open the black box DE, we need to study the subtle combination of different techniques, Le., analyze the properties of the new feature and study how CART generates and utilizes it.
Contributions. We compare the advantages of depth (cascade structure with PreConc in deep forest) over width (bagging of trees in random forest) in the scope of LL3-consistency, based on the assumptions of the additive regression functions and uniform distribution over input space. The main contributions can be summarized as follows:
We are the first to establish consistency analysis of the prediction concatenation (PreConc) operation which is crucial for multi-layer feature transformation in deep forest, though based on a simplified version.
We prove the universal consistency of deep forest when the total number of leaves of individual trees is chosen properly.
In the practical setting, when the individual trees are fully grown, we prove that the convergence rate of two-layer deep forest reach ( w.r.f. the increasing number of trees M, while that of random forest is (L/M). This result reflects that deep forest with deeper layer will be more powerful than shallower layer.
Organization. The rest of this work is organized as follows: Section 3 shows the setting and notations related to tree-based estimators in this work. Section 4 recalls the original deep forest architecture and simplifies it into two-layer deep forest. Section 5 contains the properties of cascade structure with prediction concatenation. Section 6 proves the main result that depth is more powerful than width in deep forest architecture in the scope of consistency. Section 7 is devoted to the empirical studies of deep forest by verifying the theoretical results above. Section 8 concludes with future work. More experimental results and detailed proofs for theorems and propositions given in the supplementary material due to page limitation.
2 Related work
Despite the widespread use and remarkable success of random forest in real world applications, the theoretical properties of it are still not fully understood [20, 21]. Breiman [1] offers an upper bound on the generalization errоr of random forest in terms of correlation and accuracy of the individual trees. Lin and Jeon [22] establish a connection between random forest and a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye [23]. Meinshausen [24] studies the consistency of the quantile random forest for regression. In recent years, various theoretical works [20, 25-28] have been performed, analyzing the consistency of various simplified forests, and moving ever closer to practice. Denil al. [21] narrows the gap between theory and practice of random forests for regression. Scornet et al. [29] prove the first L Llj-connistency of Breiman's 's original random forest with CART-split criterion based on the assumptions of the additive regression functions and uniform distribution over input space. Scornet [30] prove that infinite forest consistency implies finite forest consistency. Gao and Zhou [31] then present the convergence rate of purely randomized trees and simplified variant of Breiman's original CART trees. Gao et al. [32] further expand it to multi-class setting. In addition, another research route is the theory analysis of feature importance [33-35]. Recently, Li et al. [36] derive non-asympiotic lower and upper bounds on the expected bias of MDI importance for random forests. Sutera et al. [37] establish a connection between MDI importance of pure random forest and Shapley values.
However, while deep forests further improve generalization performance, there no theory to prove the advantages brought by depth. Therefore, studying the influence of depth is the theoretical cornerstone for distinguishing deep forests from random forests. For example, in the well-known deep neural networks (DNNs), there are a lot of theoretical works to study the effect of depth and width

---

on its representation ability and generalization performance, which show the theoretical advantages of deep neural networks over shallow neural networks [38-42]. These works all contribute the understanding of deep learning and provide insight for designing algorithms.
3 Setting and notations
We first describe the setting and notations related to tree-based estimators in this work. For the sake of conciseness, we consider the regression setting.
Setting. We consider regression framework, where the training set S,, {( 11),,....... (ma, U/ ())
consists of [0, 1]d x R-valued independent random variables distributed as the prototype sample (xx y) with E oc. This underlying distribution, characterized by the marginal distribution Xx on [0, 1 and by the conditional distribution Dyjæ, can be written
= f()) +
(1)
where ((x) Elylæ] is the conditional expectation of y given æ. and a noise satisfy- ing Ele] = 0 and Varls] < o². The task considered in this paper is to output a randomized estimator fa, (-, S, 1] + I where θ is a random variable that accounts for the ran- domization procedure and independent of the training set Sn. To simplify notation, we denote hn (2x, Q) = (x, O S,.). The quality of randomized estimator h, is measured by its L2 risk
R(hm) E (1nn(zz θ) f(x))) -
(2)
where the expectation taken with respect (x, θ), conditionally on Sn,. As the training data size increases, we get sequence of estimators {/1, he...., ha.... J. A sequence of estimators {hn } said be consistent if R(h,,) + as + Co.
Trees and forests. A random forest estimator 1M. (x, Q) outputs the average prediction over M individual randomized trees (ar, 9,), Vj E [M]. Here, [M] = (1, 2....... of all individual randomized trees, where θ 1.....
. M) denotes the indexes ..... O are distributed identically and independently and denoted by generic random variable θ. The random variable & can be used to sample the training set and select the candidate dimensions and positions for splitting. Specifically, a recursive partition of [o, 1]d is built by performing successive axis-aligned splits according to θ
12 Yi 1x; E (z))
(xr θ)
N₂, (C,,
where CH,,, is the cell of the tree partition containing x and N₂ (CH.aax)) is the number of training samples falling into Cn,. (z) with convention that the estimation equals zero if the cell (T,). is empty. These trees are combined to form a finite forest estimation:
M
(0, Q) ha, OI...., OMr)
1
Σ
hn O.)
M
By the law of large numbers, for any fixed æ. conditionally on D,, the finite forest estimation converges to the infinite forest estimation:
hoa.n (x) = lim hM.(#, θ) =
(5)
ad +00
When the number of samples tends co, denote by hoo (xx, +) the randomized tree with infinite samples and M θ) the random forest with M trees and infinite samples.
Deep forest
We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deep forest algorithm in Section 4.2.
3

---

New Features
predictions
RF
Input Features
Fina Prediction
RF
CRF
CRF
CRF
CRF
Layer
Layer 2
Layer
concatenate
Figure 1: Deep forest architecture (the scheme is taken from Zhou and Feng [2]).
4.1 Original deep forest
Deep forest [2] tree-based deep model made up of non-differentiable forest modules without back-propagation. Each layer of DF is composed of two Breiman's 's random forest (RF) and two Completely-Randeeee Random Forest (CRF). The RF is composed of CARTs and CRF is composed of pure random trees. In the regression setting, each forest of each layer outputs prediction for any query point æ. corresponding to the average response the leaf node (cell) containing æ. At given layer, the predictions of all forests of this layer concatenated together with raw features. This prediction concatenation process is called PreConc, which is repeated for each layer until the best layer and construct a deep forest. For an overview of the architecture of deep forest, we refer readers the work [43, 2].
4.2 Simplified two-layer deep forest
In order narrow the gap between theoretical and empirical studies of deep forest, retain CART- split criterion and PreConc when simplifying the model. We define a simplified two-layer deep forest, whose each layer is composed of one Breiman's random forest. We denote by 1 (x, θ S,) the first-layer forest estimator and h M.n ([x, h], O, Sx) the second-layer forest estimator, where [z) hi] is the concatenation of the raw features and the new feature h. Then can define the two-layer deep forest as follows:
he S.S 11) hh (1) MM.. (z, S,
(6)
The complete algorithm is shown in Algorithm 1. This algorithm has three parameters: May {l....,d d} is the number of pre-selected directions for splitting, an E {l...., n. is the number of samples in each tree, t, {1....,a,, a is the number of leaves in each tree. In the default procedure, the parameters are set as follows: may is set to d. an is set to n/k; where A is the k-fold cross- validation in Zhou and Feng [2]'s deep forest. f, On means controls the subsampling rate an n = 1/k, which is proved by Scornet et al. [29] to be the key component for imposing tree diversity.
fully grown CART. Notice that
Given the above parameters, the most basic part of Algorithm Let C' be cell and N, (C) be the number of data points falling in C. A split in C' is pair (j. z), where is a dimension in .,....d) and z is the position of the split along the j-th dimension in cell C. Let Sc be the set of all possible splits in C. The CART-split criterion [19] takes the form
training process each CART.
Ln(j,2)= = (C(C).
Σ K0;EC
(y - (.,()))
(7)
Σ
(ys pn (CL)) (,,(RR) inneCR
Σ - (Cn
(,(CL)
in,ECL
where C2 = (a E C: (1)
<
2), Cn = fa E C: 2i1) 2 2), and (C,(C) = (.) Σivc
denotes the average response in any cell C (resp. pn (CL), pn (CR)), with the convention 0/0 0.

---

Algorithm 1: A simplified variant of Zhou's is original deep forest
Require: training set S, {(x) V1), ...,, (In: Yn)] the number of trees 2M, May (l,,..,,d} On E .I, nh, in (l,.... ., ap. and the query point [0. 11
Ensure: Two-layer deep forest h2 h M.n h M.n (.) and the prediction of æ.
1: for layer é in 2) do
2: if ( = then
3:
S, {( M. M.n [11) vi...... (( h M (xn)l.8)))
(1)
Prediction concatenation.
4:
end if
5:
for tree j E {1, 21,2,... ...,, . do
6:
Select an data points, without replacement, uniformly in S..
7:
Set Ilg = (0. } the partition associated with the root the tree.
8:
For all i v anz set II; = 0.
9:
Set Raddes = and level = O.
10:
while modes < fn do
11:
level then
12:
level level
13:
else
Let C be the first element in Hesel-- if C' contains exactly one point then
15:
11
1eeeel \. {C}.
17:
[u + [eve+++ U U(C}.
else
19:
Select the best split (Jn 2()) in C by optimizing the CART-split criterion along the D See details in Eq. (7).
dimension D in {l..... .,d].
20:
Split cell C along D according to the best split (i% 2,), Call Cz and Cr.
Thevel hevel \ {C).
22:
IT k f- [evel+1 U (CL) U (CR).
23:
Aodds f- Roodes + 1.
24:
end if
25:
end
26:
end while
Compute the predicted value (1) (x. Sn) at the query point æ equaling the average of 27:
the Yt' falling the cell of æ in partition H[vell U II
28:
end for
Compute the random forest estimation 11 (ar, O S,) at the query point according to Eq. (4).
29:
30: end for
31: Compute the two-layer deep forest estimation h2 (xx O, (x, the query point according
to Eq. (6).
At each cell C, the best split (ji 2 is selected by maximizing L, (j. 2) over Mury and Sc, that is.
(Jh 2ini) E arg max Ln(i.z) 2).
(8)
(e" JEM
Properties of prediction concatenation
In this section we show that the properties of the simplified deep forest enable us to analyze the influence of the concatenated new feature in deep forest and the local variation related to the empirical CART-split criterion.
We consider additive regression model satisfying the following assumption:
Assumption 1. The response y follows
y
Σ (z(
tE,
(9)
==1
5

---

is uniformly distributed over [0, 1]", e is an independent centered Gaussian noise with finite variance x² > 0 and each component f is continuous.
where
Stone [44], Hastie and Tibshirani [45] popularize these models, which decompose the regression function sum of univariate functions. Especially, Scornet et al. [29] prove the consistency of Breiman's 's original random forest under this assumption. On this basis, we study the impact of the new feature generated in the deep forest.
To start with, we analyze the priority of the new features generated by the previous layer in the selection of splitting features under the CART-split criterion, in both infinite sample regime and finite sample regime respectively.
Proposition (Priority of the new feature), Assume the data set follows Assumption I and the first-layer forest is consistent. The following results hold for any CART in the second-layed forest.
(2) 1. In the infinite sample regime (n = oo), we consider single second-layer CART 1ox (., (,,O). All splits in this CART are performed along the new feature only.
2. In the finite sample regime (n < oo), we consider a single second-layer CART 2 (., θ). Fix k E N' and p.P > O. Then, with probability 2 21- p. for all large enough, we have, the error of the first-layer forest is bound by &. As a consequence, the first k splits (J..((:)) S 9 S k) in this CART are performed along the new feature only.
Proof sketch. (P1.1). In the infinite sample regime, the random forest estimation of the first layer has 7ero error. Therefore, the new feature of the second layer is the target function 1 (x)) fxx). Obviously, the new feature is the most informative dimension. Therefore, when splitting in any cell, CART algorithm will select the new feature as the splitting dimension. (P1.2). In the finite sample regime, the random forest estimation of the first layer is not precise. Therefore, there is an error related to n between the new feature and the target function. Firstly, we prove that the distance between the theoretical (n oc) and empirical (n < oo) first splits of the CART algorithm is bounded by of with probability 2 p. when n is large enough. Connecting with the result of theoretical split above, the proof is completed.
Remark Proposition shows that the trees in the second layer primarily choose the new feature to split and the degree of this priority depends on the error of the first-layer forest estimator: This also reveals that the advantages of deep forest depend on the performance of the first layer. If the forest of the first layer does not return estimation with noise reduction, the performance of deep forest cannot be further improved through PreConc operation. This is consistent with the empirical results in previous work [2, 17].
In order to control the risk of deep forest, we need study the local variation property of the empirical CART-split criterion. For any cell C, the variation of regression function f((x) within C is defined as
A(f.C=
= sup lf(()) f(x').
(10)
2.,'EC
Proposition 2 (Variation of f in the empirical cell). Assume that Assumption 1 holds and the first-layer forest is consistent. The following results hold for any CART in the second-layer forest hi2)) (., θ). After splitting along the new feature, the CART will estimate the residual of the first-layer forest estimation. Then for all P.E & > O, there exists N E No such that, for all n > N.
Pr (J θ)) < 2 1-
(11)
Proof sketch. Firstly, we prove the variation of f(x) within the cell obtained by the theoretical CART-split criterion converges to zero. Secondly, we prove that the distance between the theoretical and empirical first k splits of the CART convergence zero. Finally, prove that the variation of f(z) within the empirical cell is close to the theoretical cell.
Proposition 2 shows that the variation of the regression function f(x) within a cell of random tree CT.,: is small provided n is large enough, thereby foreing the approximation error of DF to asymptotically approach zero.

---

Depth is more powerful than width
Our first result considers the in < On regime, where the number samples in each leaf node tends to 2o as f, - oo and 0₂ + co. We prove that controlling the depth of the trees through the number of leaves in sufficient to achieve consistency of deep forest.
Theorem 3 (Universal consistency). Let M 2 1. Consider two-layer deep forest h2as, given by Eq. (6) and Breiman's 's random forest hM,n given by Eq. (4) for the random CARTs satisfying an + oc, tn + oo and tn (log / - 0. Then under the setting described in Section and assume the data set follows Assumption 1,
1. [29, Theorem 11 the Breiman's 's random forest 12M is consistent for any M 2 1,
2. the two-layer deep forest f is consistent for any M 2 1.
Proof sketch. (T3.1). The universal consistency of Breiman's 's random forest is proved by Scornet et al. [29]. (T3.2). Firstly, already know that the variation of the target regression function fx) within a cell of randomized CART in the second-layer forest is small when n is large enough via Proposition 2. Similar as Scoree et al. [29], utilize Proposition 2 to control the approximation error of the two-layer deep forest. And the parameter (, allows us to control the size of the leaves of CART, which allows to have enough samples in each leaf node to smooth the impact of noise, so as to control the estimation error. Connecting the approximation and estimation error, the consistency of a CART of second-layer deep forest is proved. The universal consistency can be proved via [25, Proposition IJ. which guarantees that the error of forest estimator is no more than twice that of individual randomized CART.
Remark 2. Notice that under this setting, random forest and even deep forest have obvious advantages over single CART in theory. When we use forests in practice, we do not choose to control the depth of the trees. Empirical studies in Section 7.2 show that the forest with f, an always outperforms the forest with tn < a, Actually, the fully grown trees t, = an is the setting close to practical forest algorithm.
In order to deal with the ln = an regime, we need to introduce an assumption first proposed by Scornet al. [29]. We denote by Z, 1 ((x; (z)) the indicator that x; falls into the same cell as x in the random tree designed with Dn and the random parameter O Z = 1(x, E Cir (x)) another independent indicator. Then, we define the correlation between these two indicators conditionally on y/: yj or not, respectively
Φi,( vU) = ElZ,, Z}|x,, I1...., In. Vss Uy] and Φi, = E|Z
x.l .u.
(12)
Assumption 2. Let Zi, = (Zi, Zz). Then one of the following two conditions holds:
1. One has
lim (log an) )d 2 (logn)²B 2 max (1.,())))))) - M6s1 =0
(13)
igtj
2. There exists constant C > 0 and sequence (7x) )) - 0 such that, almost surely,
Cor [(v - ((ax:)), = (f1, 62) (2; 2., ,3,88)]l
max 0,11
/1// = (f2, E2) (2, x3,35]
< S'n
(14)
and
Cor (1ys -f((x))) 1(, = [1):)]| SC. -
max =0.1
(15)
(A2.1.) means that the influence of two labels yss Vj on the probability of connection of two couples of random points converge to zero as n -> 30. (A2.2.) means that the correlation between the noise and the probability of connection of two couples of random points vanishes quickly enough, as n + co. However, this assumption too strong for the Breiman's original random forest [1]. Scornet et al. [29] emphasize that they cannot know whether or not Assumption 2 is satisfied in random forest. In this paper, we recall this assumption and state that this assumption mild for the second-layer forest estimation in deep forest algorithm.
7

---

Since deep forest concatenates the prediction with the raw features as the imput for the next layer. the label information is encoded into the new feature of second layer. In this way, the influence of two labels on the connection probability of this pair of samples tends to zero, so (A2.1.) is mild for the second- -aaeer forest estimation. From another point of view, Proposition 1, the priority of new feature also shows that the partition can be independent of labels, and the information of new feature is enough to obtain appropriate partition results, so (A2.2.) is mild too.
Theorem 4 (Depth is more powerful than width). Let M Z 1. Consider two-layer deep forest h2M a given by Eq. (6) and Breiman's 's random forest hM.. given by Eq. for the random CART's satisfying an + 00, tn - oC, tu = an and on log n/n -+ Then under the setting described in Section 3 and assume the data set follows Assumption I and 2, the following results hold
1. 129, Theorem 2] [30, Theorem 3] The Breiman's 's random forest boon is consistent, and for all M, n E N,
0 S R( - R(hoo,a) < 811f12 + +10 + llogn)
(16)
M
2. The two-layer deep forest hoon is consistent, and for all M,n E N. If Cn, Q)) is small enough, then
0 S R(h2M,)) - R(hoo,n) S 6411f11 + 61 + 4logn)
(17)
M
Proof sketch. (T4.1). The consistency of the infinite Breiman's 's random forest is proved by Scornet et al. [29]. And the convergence rate of the finite random forest with the number of trees M is proved by Scornet [30]. (T4.2). Similar to Scornet et al. [29], we recall Proposition to control the approximation error of the two-layer deep forest. Then the estimation error is controlled by forcing the subsampling rate o,/n n to be / logn) Different from the bagging-style mechanism in random forest, the residual-style mechanism shown in Proposition 2 makes the second- forest in DF can reuse the first-layer estimation and focus on the residual learning.
Remark 3. In the Ln = on setting, Scornet et al. [29] show that the sub-sampling rate an the key component in random forest. Because the small rate ensures that query point is connected with enough different data points through different trees, the convergence rate of RF is (L/M)) w.z.t the number of trees M. Theorem 4 proves that, if the first layer forest can encode the regression function fx) into the new feature with noise reduction, the cascade structure with PreConc in DF can further accelerate the convergence. Because the second layer forest estimates the residual of the first layer, the trees in each layer of forest are more different. As a result, the convergence rate of deep forest will be improved to (1)/M²). This result reflects that deep forest with deeper layer will be more powerful than shallower layer.
7 Simulation experiments
7.1 Priority of the new feature
This experiment aims to verify the priority of the new feature in choosing which feature to split as suggested in Proposition We focus on a second layer decision tree built upon the first layer random forest. Since regression forest has only one output dimension, there is only one new feature for the second layer tree. More specifically, we count the maximum consecutive levels from root node that use the new feature to split, which we call effective depth for short.
The synthetic data set is generated as y = f(x) + E, where
1 f(x)
Σ Ij.
(18)
5 15153
is uniformly distributed over 10. c c~ N (0, where = 0.02. We vary the number of training samples and the number of trees in the first layer forest, and report the average effective depth of 5 runs in Figure 2(a). It is easy to observe that matter what the first layer's setting is, the effective depth of new feature
least 2.4. That to say, at the beginning of the growing of the second layer tree, CART will always choose the new feature to split, And we can see that with
8

---

Number of trees in the 1st layer
5.0
Effective depth of new feature
20
2:8
4
4.4
4.5
50
2.4
4
4.6
4.0
3.5
100
3.0
500
2.6
4.6
2.5
2.0
200
500
2000 10000 50000
0.03
0.04
0.05
0.06
Number of training samples
First layer
(a) Hcatmap of the effective depth new feature under (b) Effective depth of feature against the predic- different settings of the first layer forest.
tive error of the first layer.
Figure 2: Illustrations of the effective depth of the new feature (the consecutive levels from root node that split on the new feature only). The larger the effective depth, the higher priority the new feature takes in being chosen as the split feature under the CART-split criterion.
0.014
0.045
0.012
RMSE
RMSE
0.035
0.010
0.030
0.008
RF
0.025
0.006
DF
0.020
10
102
10
102
M
M
(a) Consistent trees (fn. < on))
(b) Inconsistent trees (fn. a.).
Figure 3: Root mean square with the increasing of trees.
the increase trees in the first layer and training samples, the CART's preference of new feature extend to deeper layers. Figure 2(b) further plots the effective depth against the predictive error of the first layer measured by root mean square error (RMSE), showing that the better the prediction performance of the first layer, the more favored the feature is for splitting.
7.2 Convergence rate w.r.f. the number of trees
To study the convergence rate of predictive error with respect the number of trees in the whole model, we fix the number of training samples to be 50,000. We compare 2-layer deep forest (DF) with M trees each layer to random forest (RF) with 2M trees, and we adopt 3-fold split training both DF and RF to ensure fairness. The performance is measured by RMSE with respect to the noise-free version of data generating function. Since the training data are with noise, fully grown tree is inconsistent. In Figure 3(a), set the minimum leaf size of the trees to be vnt, i.e., 233 in the case when training data size is 50,000. In Figure 3(b) the trees are fully grown with only one sample in each leaf. We plot the average RMSE of 5 runs against the increasing number of trees, with the colored band indicating the standard deviation.
Theorem 3 reveals that when the component trees are consistent, random forest and deep forest both consistent. However, the consistency analysis result cannot guarantee the finite sample performance in practice. Comparing Figure 3(a) to Figure 3(b), we can see that even though the training as large as 50,000, the performances of RF and DF using consistent trees are still much worse than using inconsistent trees. This observation confirms the efficacy of the default experimental setting that uses fully grown trees RF and DF. Figure 3(b) shows that DF enjoys faster improvement in RMSE with the increasing of M. More specifically, DF with M = 20 outperforms RF with M 500. This experimental result matches our theoretical result Theorem that DF has faster convergence rate w.e.t. the number of trees M.

---

8 Conclusion
In this paper, we prove that a two-layer deep forest has a faster convergence rate w.z. the number of component trees M than random forest. This work provides the first theoretical analysis of the prediction concatenation (PreCone) operation which crucial for feature transformation in deep forests, although based on very simplified structure where the concatenation of multiple random forests' predictions and completely-random forests' predictions each layer of deep forest has not been taken into account.
On the one hand, this paper focuses on the asymptotic consistency of deep forests, so the result is strictly true only when the number of samples tends to infinity. As for the generalization analysis of deep forests with finite samples, we leave used in this paper have certain limitations. Experiments on simulation and real-world data sets show that our theoretical results are valid in many objective function classes other than Assumption How to further relax the conditions in Assumption 1 will be an interesting problem. As for Assumption 2, still not strictly verified. However, quantifying the trade-off between label information and partition randomness will be very important topic in future work.
future work. On the other hand, the two assumptions
Acknowledgements
This research was supported by the NSFC (61921006) and the Collaborative Innovation Center of Novel Software Technology and Industrialization. The authors would like thank the anonymous reviewers for constructive suggestions, as well as Jin-Hui Wu, Qin-Cheng Zheng and Peng Tan for helpful discussions.
References
[1] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
[2] Zhi-Hua Zhou and Ji Feng. Deep forest. National Science Review, 6(1):74-86, 2019.
[3] Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statisticx, 29(5): 189-1232, 2001.
[4] Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine Learning, 63(1):3-22 2006.
[5] Tianqi Chen and Carlos Guestrin. XGboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 785-794, 2016.
[6] Ran Su, Xinyi Liu, Leyi Wei, and Quan Zou. Deep-resp-forest: A deep forest model to predict anti-cancer drug response. Methods, 166:91-102, 2019.
[7] Meng Zhou, Xianhua Zeng, and Aozhu Chen. Deep forest hashing for image retrieval. Pattern Recognition, 95:1 14-127, 2019.
[8] Yaakoub Boualleg, Mohamed Farah, and Imed Riadh Farah. Remote sensing scene classification using convolutional features and deep forest classifier. IEEE Geoscience and Remote Sensing Letters,
2019.
[9] Ya-Lin Zhang, Jun Zhou, Wenhao Zheng, Ji Feng, Longfei Li, Ziqi Liu, Ming Li, Zhiqiang Zhang, Chaochao Chen, Xiaolong Li, Yuan (Alan) Qi, and Zhi-Hua Zhou. Distributed deep forest and its application to automatic detection of cash-out fraud. ACM Transactions on Intelligent Systems and Technology, 10(5):55:1-555 2019.
[10] Lev V. Utkin and Mikhail A. Ryabinin. A siamese deep forest. Knowledge-Becn Systems, 139: 13-22. 2018.
[11] Lev V. Utkin and Mikhail Ryabinin. Discriminative metric learning with deep forest. International Journal Artificial Intelligence Tools, 19C:1195000711-9500771 2019.
10

---

[12] Liang Yang. Xi-Zhu Wu, Yuan Jiang, and Zhi-Hua Zhou. Multi-label learning with deep forest. In Proceedings of the 24th European Conference on Artificial Intelligence, volume 325, pages 1634-1641, 2020.
[13] Qian-Wei Wang, Liang Yang, and Yu-Feng Li. Learning from weak-label data: A deep forest expedition. In Pruceedings of the 34th AAAI Conference on Artificial Intelligence, volume 34, pages 6251-6258, 2020.
[14] Ming Pang, Kai-Ming Ting, Peng Zhao, and Zhi-Hua Zhou. Improving deep forest by confidence screening. IEEE Transactions on Knowledge and Data Engineering, 34(9):4298-431 2022.
[15] Yi-He Chen, Shen-Huan Lyu, and Yuan Jiang. Improving deep forest by exploiting high-order interactions. In IEEE International Conference Data Mining, pages 0030-1035, 2021.
[16] Shen-Huam Lyu, Liang Yang. and Zhi-Hua Zhou. A refined margin distribution analysis for forest representation learning. In Advances in Neural Information Processing Systems 32, pages 5531-5541, 2019.
[17] Ludovic Arnould, Claire Boyer, and Erwan Scornet. Analyzing the tree-layer structure of deep forests. In Proccedingy of the 37th International Conference on Machine Learning, pages 342-350, 2021.
[18] Shen-Huan Lyu, Yi-He Chen, and Zhi-Hua Zhou A region-based analysis for the feature concatenation in deep forests. Chinese Journal of Electronics, 2022.
[19] Leo Breiman, Jerome Friedman, Charles Stone, and Richard A Olshen. Classification and Regression Trees. CRC press, 1984.
[20] Gérard Biau. Analysis of random forests model. The Journal of Machine Learning Research, 13(1):1063-100) 2012.
[21] Misha Denil, David Matheson, and Nando De Freitas. Narrowing the gap: Random forests theory and in practice. Proccedings of the 30th International Conference on Machine Learning, pages 665-673, 2014.
[22] Yi Lin and Yongho Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association, 101(474):578-590 2006.
[23] Gérard Biau and Luc Devroye. On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. Journal of Multivariate Analysis, 101(10):2499-23 24182218,, 2010.
[24] Nicolai Meinshausen. Quantile regression forests. Journal of Machine Learning Research, 983-999, 2006.
[25] Gérard Biau, Lue Devroye, and Gäbor Lugosi. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research, 9(9), 2008.
[26] Hemant Ishwaran and Udaya B Kogalur. Consistency of random survival forests. Statistics & Probability Letters, 90(13-14):1056-0666 2010.
[27] Robin Genuer. Variance reduction in purely random forests. Journal of Nonparametric Statistics, 24(3):543-562, 2012.
[28] Ruoqing Zhu, Donglin Zeng, and Michael R Kosorok. Reinforcement learning trees. Journal of the American Statistical Association. 110511))1770-1788. 2015.
[29] Erwan Scornet, Gérard Biau, and Dean-Philippe Vert. Consistency of random forests. The Annals of Statistics, 13(4):1716-1744 2015.
[30] Erwan Scornet. On the asymptotics of random forests. Journal of Multivariate Analysis, 146: 72-83, 12-83, 2016.
[31] Wei Gao and Zui-Hua Zhou. Towards convergence rate analysis of random forests for classifi- cation. In Advances in Neural Information Processing Systems 33, pages 9300-9311, 2020.
11

---

[32] Wei Gao, Fan Xu, and Zhi-Hua Zhou. Towards convergence rate analysis of random forests for classification, Artificial Intelligence, 313:103788, 2022. doi: 00tp://000.rr/11011116///// 103788.
[33] Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformation,
8. 2007.
[34] Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. Understanding variable importances in forests of randomized trees. In Advances in Neural Information Processing Systems 26, pages 431 2013.
[35] Ludovic Amould, Claire Boyer, and Erwan Scornet. Is interpolation benign for random forests? arXiv preprint 2.8.022202.0366888 2022.
[36] Xiao Li, Yu Wang, Sumanta Basu, Karl Kumbier, and Bin Yu. debiased MDI feature importance measure for random forests. In Advances in Neural Information Processing Systems 32, pages 8047-8057, 2019.
[37] Antonio Sutera, Gilles Louppe, Vân Anh Huynh-Thu, Louis Wehenkel, and Pierre Geurts, From global to local MDI variable importances for random forests and when they are shapley values. In Advances in Neural Information Processing Systems 34, pages 3533-3543, 2021.
[38] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Proceedings of the 29th Conference on Learning Theory, pages 907-940, 2016.
[39] Matus Telgarsky. Benefits of depth in neural networks. In Proceedings of the 29th Conference on Learning Theory, pages 1517-1539, 2016.
[40] Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1979-29877 2017.
[41] Eran Malach and Shai Shalev-Shwartz. deeper better only when shallow is good? In Advances in Neural Information Processing Systems 32, pages 6426-6435, 2019.
[42] Amit Daniely and Eran Malach. Learning parities with neural networks. In Advances in Neural Information Processing Systems 33, pages 20356-20365, 2020.
[43] Zui-Hua Zhou and Feng. Deep forest: Towards an alternative to deep neural networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3553-3559,005 2017.
[44] Charles Stone. Additive regression and other nonparametric models. The Annals of Statistics, 13(2):689-705, 1985.
[45] Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-386, 1987.
[46] László Györfi, Michael Kohler, Adam Krzyzak, Harro Walk, et A Distribution- Free Theory of Nonparametric Regression, volume 1. Springer, 2002.
[47] Pengfei Ma, Youxi Wu, Yan Li, Lei Guo, He Jiang, Xingquan Zhu, and Xindong Wu. HW- Forest: Deep forest with hashing screening and window screening. ACM Transuctions on Knowledge Discovery from Data, 2022. doi: 10. 14533532299..
[48] Misha Denil, David Matheson, and Nando Freitas. Consistency of online random forests. In Proccedings of the 31st International Conference on Machine Learning, pages 1256-1264, 2013.
[49] Shai Shalev-Shwartz and Shai DeviD. Understanding Machine Learning: From Theory to Algorithms. Cambridge university press, 2014.
[50] Lue Devroye, László Györfi, and Gábor Lugosi. Probabilistie Theory of Patler Recognition. Springer, 1996.
12

---

[51] Luc Devroye. A note on the height of binary search trees. Journal of the ACM, 33(3):489-449 498, 1986.
[52] Sumanta Basu, Karl Kumbier, James B. Brown, and Bin Yu. Iterative random forests to discover predictive and stable high-order interactions. Proceedings of the National Academy of Sciences, 115(8):1943-1444 1948, 2018.
[53] Zhi-Hua Zhou. Why over perrpaaramttrizzttion of deep neural networks does not overfit? Science China Information Sciences, 64(1):1-3, 2021.
[54] Gérard Biau and Erwan Scornet, random forest guided tour. Test, 25(2):197-227, 2016.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 7.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] This is theoretical work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' rou're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [No]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [No]
14


Reviews:
[{"review_id": "kxpq4caSTG5", "reviewer": "Reviewer_8wUS", "summary": "The paper \"Depth is More Powerful than Width in Deep Forest\" studies the behavior of Deep Forest and shows that Deep Forests are consistent with a rate of $\\mathcal O(1/M^2)$ where $M$ is the number of trees in the forest. To do so, the paper studies two-layered DF and its convergence in two different scenarios, one in which the number of data points is much larger than the number of leaves and  second when they are comparable equal. The resulting two Theorems establish the consistency of DF. Last, another Theorem establish the convergence speed of DF in terms of the number of trees which can be interpreted as \"depth is more powerful than width\". Experiments on synthetic data further supports the claim ", "questions": "- It seems that I can apply Theorem 2-4 recursively if I have a Deep Forest with more than layers. Is this correct, and if so, does this improve the convergence rate to $\\mathcal O(1/M^3), \\mathcal O(1/M^4), \\dots$ with each layer? This seems like an unusual strong result. Is this correct?", "limitations": "I think the limitations are addressed adequately. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I enjoyed reading the paper and I cannot criticize much about it. It is generally well-written and discusses an important topic. Deep Forest is arguably one of the more modern ensemble approaches (Ensembles of Neural Networks aside). While some theory has been developed for DF, it is not fully explored yet and hence the topic is of great interest. The paper heavily relies on results established for Random Forests, but extends them in a meaningful and non-trivial way. While the results are consistent with what one expect intuitively, I think it is important that they are tackled by rigors theory. Surprisingly, the results in the paper seem to be quite close to the practical application of DF. In summary:\n- (+) Extends known results about Random Forest to Deep Forest thereby explaining some of the success of DF\n- (+) Comparably easy to follow for such a theoretical paper. I was especially happy about the remarks and proof sketches, although I admit that I did not follow all proofs in the appendix. \n- (-) I think a discussion on DF with more than two layers would have been interesting, although it seems straight-forward to apply Theorem  2 and 3 recursively (see question below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i9so4MXKHfc", "reviewer": "Reviewer_QMfU", "summary": "In this paper, authors analyze the relationship between the depth and the number of trees of the deep forest. In recent years, deep forest has been widely studied due to its usability that combines the deep neural network with random forests. The authors validate their analysis through their experiments of the deep forests.", "questions": "Q1) Can you explain further why the analysis of the depth and the number of trees of the deep forest has a large impact? A more thorough explanation of why this analysis of deep forest is important should be added.\n\nQ2) Does the experiment shown in this paper apply well to real vision or NLP applications? It would be nice to show the experimental results on more widely used task and dataset, but if that is not the case, authors need to provide evidence that the analysis in this paper can be applied to other applications.", "limitations": "There is no explanation for the limitations of the paper. Appropriate limitations should be included.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+) Motivation and background studies are sufficiently provided.\n+) Appropriate assumptions and mathematical derivations for analyzing deep forests are included.\n\nWeaknesses\n-) Analyzing the depth and the number of trees of the deep forests is limited to claim the novelty.\n-) There is a lack of experiments with popular tasks and datasets, such as in computer vision or NLP domains.\n\n----- after rebuttal -----\n\nIn rebuttal phase, authors revise their papers according to the comments. I understand the importance of the deep forest and the author's finding that its depth is more important than its width. However, I'm not sure if this finding would apply to real problems. In the revised paper, authors introduce new results on several real-world datasets and compared it to a random forest. However, it is necessary to show how much the existing SOTA results in the housing dataset can be increased by the authors' finding. For example, if this finding is applied to previous deep forest based methods, there should be  performance improvement compared to the baseline deep forests.\n\nConsidering the author's response, I increase the score to borderline accept by understanding that the authors have responded to reviewer's comments sufficiently, and that their findings are important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JJ-UUSToo6E", "reviewer": "Reviewer_iuCL", "summary": "The authors propose a theoretical analysis of deep random forests. Deep random forests stack layers of Breinman's random forest combined to completely random forest. The output of each layer in the regression setting is the average label of all sample falling in each leaf. Each layer takes as input a concatenation of the input features as well as previous features output by previous layers.\n\nThe authors focus on the two-layer deep forest and show multiple theoretical results aiming at explaining empirical performances.\n\nProposition 1 shows that stacking multiple layers make sense as the latter focus on the new features thus improving the regression function. Proposition 1 is proved in the infinite and finite regime (in the sample size).\n\nProposition 2 focuses on the variation of the true regression function f. The authors show that the variation of the regression function within a single cell is bounded in probability for any threshold and probability given that the sample size is large enough.\n\nTheorem 3 recall the consistency of traditional random forests and states that the two layer deep forest are consistent as well. (given that the sample size per tree grows faster than the number of leaves which as stated by the authors is not what is usually done in practice).\n\nTheorem 4 shows convergence speed to the infinite forest for the classical Breiman's forest and the two layer random forest. The former has a convergence rate of O(1/M) while the latter has O(1/M^2).\n\nConcerning assumptions 1 and 2. Assumption 1 is a not too restrictive assumption probably used for simplicity in the proofs. Assumption 2 is less obvious to get at first sight and aims at showing that cells become less and less influenced by couple of samples labels and noise while the sample size tends to infinity.\n\nExperiments on simulated data tend to confirm the theoretical claims and theorems of the paper.", "questions": "Would it be possible to give more elements about all the assumptions. Mainly:\n- Do they apply in real life problem,\n- What would we need to maintain the theoretical results while relaxing the assumptions ?\n\nWould it be possible to evaluate experimentally the limit of these asumptions?\n\nEquations 14 and 15 seem to have parenthesis problems.", "limitations": "Limitations of the work are clear from the core of the article but not particularly addressed in a specific area of the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strength\nThe paper is easy to read even for non-experts. Objects are well defined,\nThe paper proposes a theoretical justification for known empirical claims.\nDeep forest is a recent contribution for which theoretical evidences lack and this paper contributes to the literature.\n\n- Weaknesses\nThe assumptions, required for the proof, lack some analysis about their application in real life problems as well as motivation for their use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept