Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an extension of a neural network analytic tool
called Neural Tangent Kernels to its use for decision trees and
forests.  The focus is a previously proposed notion of soft trees.
A few analytical results are presented about the properties of the
extended notion, called Tree Neural Tangent Kernel, such as the
existence of some limiting deterministic form, how alternative
probabilistic tree ensembles would have their kernels converging to
this deterministic form, that this limiting kernel is positive
definite, and that the kernel remains stable during training,
equivalence of the kernels for one-level tree ensembles to those of two-layer perceptrons, etc.
Some support for the analytical claims is provided by simulation experiments.

main_review: The work attempts to generalize the recently proposed Neural Tangent
Kernel method to trees and forests, and attempts to use that to
analyze certain properties and training behavior of a special type of
decision trees (the so-called soft trees) and their infinite ensembles.
This could be of interest to followers of the Neural Tangent
Kernel method and its use in analyzing neural-network-like tree
structures.

For the rest of the audience, the paper has not made an attempt to
clearly lay out all the basics and the logic.  It relies heavily on
external references for many critical concepts and results.

For a start, the notion of a "soft tree" involves some mixing of the
following properties of trees:
(1) whether the splitting function is fixed in one shot (e.g. by
information theoretic choices) or adjustable and trainable (e.g. by
gradient descent) during learning, 
(2) during or after training, whether the split is deterministically
applied to every input, and
(3) whether the splitting function parameters are frozen by training or
continue to be samples from some probabilistic distribution that has
parameters estimated during training.
These concepts are never clarified in the paper, so when additional
notions like ensembles and infinitely large ensembles are brought in,
the confusion just grows.

One may wonder how the chosen definition of soft trees is related to
those of probabilistic decision trees [1] and neural trees [2].

[1] Michael I. Jordan,
 A Statistical Approach to Decision Tree Modeling,
 Machine Learning Proceedings 1994, Morgan Kaufmann, 1994, 363-370,
 https://www.sciencedirect.com/science/article/pii/B9781558603356500519

[2] J A Sirat & J-P Nadal,
 Neural trees: A New Tool for Classification,
 Network: Computation in Neural Systems, 1:4, 1990, 423-438,
 https://www.tandfonline.com/doi/abs/10.1088/0954-898X_1_4_003

Given that the work builds on a key concept in prior art, for the
paper to be self-contained, the notion of "Neural Tangent Kernel"
should first be explained (why it is called a Kernel, why Tangent, why
Neural, what is its significance, what new insight it has contributed,
...).   A basic understanding of this is important for the proposed
extension to have a sound footing. It should also be stated upfront
what is expected of the proposed extension.   What insight contributed
by the NTK do you expect to reproduce for tree ensembles?  What new
insight do you want it to bring on?

Many other improvements are needed in the presentation of the logic,
especially on the sudden appearance of certain claims and arguments
around or against them.  For example, when it is declared that trees
of depth larger than 1 have some distinctive feature from MLPs of any
number of layers, what assumption is being argued against?  Did
someone conjecture that they are the same, and why was there such a conjecture?

The paper is not readable due to its lack of clarity in the basic
definitions, a severe lack of connections in the logic, heavy
reliance on external references, and a lack of explanation of the
significance of the results. 

summary_of_the_review: The paper may have some new results but is better fitting for a forum for theoretical analysis and
preferably a journal where more space is available to fully elaborate on
the detailed definitions, assumptions, claims and their implications.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed the Tree Neural Tangent Kernel (TNTK) for tree ensembles. The proposed idea extends the NTK concept to tree ensemble models and enables ensembles of infinite soft trees. This paper provides theoretical studies to analyze the properties of the proposed TNTK. They also provide comprehensive experimental results to show the effectiveness of the proposed method. 
main_review: This paper proposed a new method called TNTK to ensemble infinite soft trees. It provides non-trivial theoretical studies of the TNTK. Their numerical experiments support their theoretical results. This paper is well-written and clearly organized. 

Compared with MLP-induced NTK and RBF, one of the advantages of TNTK will be the low computational complexity. It would be better to provide a detailed complexity analysis (e.g., memory cost, training time complexity, and inference time complexity) of the proposed TNTK. 

summary_of_the_review: Please refer to my main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors derive a neural tangent kernel for soft trees, and prove several properties of the kernel. These include the stability of the kernel in the large ensemble limit, applicability to oblivious tree ensembles, degeneracy with large tree depth, and comparison to NTK. They then evaluate the utility of the kernel on 90 UCI classification data sets using a kernel regression classifier. 
main_review: The authors define the TNTK and prove a number of properties which one would like to know about the kernel. 
Such as positive definiteness, and change during training.
The kernel shares some of the properties of the NTK, indicating that the TNTK can be used to understand training behavior. This is not surprising, considering that soft trees can be viewed as a neural network. This work offers some novelty in computing the kernel in the limit of infinite size of the ensemble, whereas naive application of NTK to large ensembles would be computationally intractible. 

The depth degeneracy property is a result with practical utility. The TNTK framework offers a clear understanding of this phenomenon and can help select model hyperparameters.

It is often unclear how to initialize soft-tree ensembles. Many works use classical greedy algorithms to initialize, while others fix the structure of the tree and randomly initialize the weights. It would be nice if the authors can remark whether TNTK can shed some light on initialization strategies.
summary_of_the_review: This paper represents the development of an important tool for understanding tree ensembles.
review_title: null

Meta-review: The paper investigates the neural tangent kernel NTK of infinitely wide ensembles of soft trees having a particular 
soft decision functions in their internal nodes. A closed form of the NTK is presented as well as a result bounding
the changes of the NTK during training. Implications for practical training procedures are briefly discussed and 
some experiments are also reported. 

The review and discussion phase were difficult with two rather uninformative but positive reviews and a negative 
but detailed review. The latter had, however, some serious flaws. For these reasons I carefully read the paper on my own,
too. In turned out that the criticized flaws in the presentation mentioned by the negative reviewer are baseless as long as 
one already knows what an NTK is. Given the title of the paper and the history of NTKs, I personally think that 
such knowledge can and should be assumed. 
Overall, I find the paper be actually very well written. The main issue I see is that the results are not overwhelmingly 
surprising. Nonetheless, this is a solid contribution, which deserves to be published.
Decision: Accept (Poster)