Title: A faster training algorithm for regression trees with linear leaves, and an analysis of its complexity
Venue: NeurIPS 2025
Accepted: yes
Keywords: regression trees

tree alternating optimization

Sherman-Morrison-Woodbury formula
Abstract: We consider the Tree Alternating Optimization (TAO) algorithm to train regression trees with linear predictors in the leaves. Unlike the traditional, greedy recursive partitioning algorithms such as CART, TAO guarantees a monotonic decrease of the objective function and results in smaller trees of much better accuracy. We modify the TAO algorithm so that it produces exactly the same result but is much faster, particularly for high input dimensionality or deep trees. The idea is based on the fact that, at each iteration of TAO, each leaf receives only a subset of the training instances. Thus, the optimization of the leaf model can be done exactly but faster by using the Sherman-Morrison-Woodbury formula. This has the unexpected advantage that, once a tree exceeds a critical depth, then making it deeper makes it faster to train, even though the tree is larger and has more parameters. Indeed, this can make learning a nonlinear model (the tree) asymptotically faster than a regular linear regression model. We analyze the corresponding computational complexity and verify the speedups experimentally in various datasets. The argument can be applied to other types of trees, whenever the optimization of a node can be computed in superlinear time of the number of instances.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on learning oblique regression trees (with linear leaves). They propose a tweak to the TAO algorithm that speeds up training for deep trees. This tweak is exact and makes use of the fact that fitting a linear model at a leaf uses few samples, often making the rank of the data matrix constrained by the number of samples rather than the number of features. This yields a speedup, which the authors provide a complexity analysis for and confirm numerically.\n\nSTRENGTHS AND WEAKNESSES:\n- S1: The authors propose an exact improvement to the TAO algorithm that can often provide a speed-up\n- S2: The algorithm is intuitive, simple, and straightforward to implement\n- S3: The experiments confirm the speed-ups, especially for deep trees\n- W1: The impact of the speed-up is unclear, as it tends to kick in for large depths or limited-sample regimes where decision trees may tend to overfit. Speed & performance comparisons to other algorithms (including greedy methods like CART) would be helpful to show that there are realistic settings where this speed up applies & yields strong performance.\n- W2: The introduction of the method and analysis of its complexity (Sec 3 & 4) are perhaps overly detailed for introducing the usage of the SMW formula, a fairly well-known trick\n- W3: For a paper like this that focuses on an efficiency improvement, it would be nice to see code provided\n\nQUESTIONS:\nCan the authors provide more in-depth exploration of when the speed-up applies and how it relates to a model's predictive performance / overfitting?\n\nCan the authors discuss in more detail how the method could be extended to other settings, e.g. learning sparse or axis-aligned splits? classification trees?\n\nLIMITATIONS:\nThe Limitations are not explicitly discussed in the Conclusion section, might be nice to discuss them more carefully there, esp how the speed-up relates to tree depths.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a faster training algorithm for oblique regression trees with linear predictors in the leaves, based on the Tree Alternating Optimization (TAO) framework. The authors modify the leaf-fitting step by applying the Sherman–Morrison–Woodbury (SMW) formula to invert an $N \\times N$ matrix instead of a $D \\times D$ one, where $N$ is the number of instances reaching a leaf and $D$ is the feature dimension. This results in an exact but significantly faster solution.\n\nThe key insight is that for deep trees exceeding a critical depth $\\Delta^* = \\log_2(N/D)$, each leaf receives fewer than $D$ samples, making SMW-based inversion more efficient. Surprisingly, making the tree deeper beyond $\\Delta^*$ can reduce total training time, even though the model has more parameters.\n\nThe authors analyze the complexity and demonstrate substantial speedups (up to $25\\times$) on high-dimensional ($D=2500$) regression tasks, with identical predictive performance to the original TAO algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The contributions of the paper are clearly stated (an exact faster leaf step, complexity analysis, experiments). The paper builds on the known TAO algorithm but identifies a nontrivial new insight that yields a significant practical gain.\n2. Faster training of linear-model trees is useful for regression tasks, especially since TAO-trained trees can sometimes outperform greedy methods such as CART and M5 in accuracy. The proposed idea also applies to other tree-learning methods, as the authors note.\n3. The use of the SMW identity is mathematically sound and yields exact equivalence to the original leaf solution. The derivations and complexity arguments are detailed and appear correct.\n4. The paper provides a careful time-complexity analysis, covering both “shallow” and “deep” regimes, and clearly explains why a deep tree can be faster to train than a linear model.\n5. The experiments on real and synthetic data demonstrate large speedups without loss of accuracy. \n\n\n**Weaknesses:**\n1. The main idea (use SMW for faster inversion) follows naturally from the observation that deeper trees have smaller leaf sample sizes. While the approach is effective, it builds on known mathematical tools and does not introduce a fundamentally new learning paradigm, but rather an optimization applied to TAO.\n2. The paper lacks an explicit discussion on limitations. The proposed method inherits all the limitations of TAO. Namely, TAO optimizes a tree of given structure (depth and topology). The structure is initialized often as a full binary tree, and TAO only optimizes the parameters, not the structure. This leads to hyperparameter sensitivity, and poor initialization or overly deep/shallow structures can result in significant overfitting or degraded performance. Additionally, while TAO forests can be smaller overall, individual trees can be quite heavy (because both the splitting functions and the leaf models contribute substantial parameter counts). This is one reason why TAO training can be significantly slower than greedy methods such as CART or M5.\n3. All experimental results focus only on TAO (adaptive) versus TAO (naive). No comparisons are shown against other linear-tree methods (e.g., CART, M5, or more recent PILOT [1]). The authors cite the TAO paper [2] to state that greedy trees have worse accuracy, but TAO is sometimes significantly slower. Since this paper accelerates TAO, it would be highly relevant to compare TAO (adaptive) to algorithms like XGBoost and Random Forests, as was done in the TAO paper [2]. \n4. The paper primarily discusses balanced trees. It would be helpful to discuss how the method performs on highly imbalanced splits.\n\n**Reproducibility.** The authors do not mention whether they plan to release code or a library. No code archive is attached to this submission.\n\n*[1] J. Raymaekers, P.J. Rousseeuw, T. Verdonck, and R. Yao. Fast linear model trees by PILOT. Machine Learning 113:6561–6610, 2024.*   \n*[2] A. Zharmagambetov, M. Carreira-Perpinan. Smaller, more accurate regression forests using tree alternating optimization. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:11398–11408, 2020.*\n\nQUESTIONS:\n**1. Scope of Experiments:** Have you considered comparing your approach to other linear-tree algorithms, such as XGBoost or Random Forests? Even though those methods follow a different (greedy) strategy, it would be informative to see how TAO (with your speedup) compares in terms of speed and accuracy on benchmark tasks.  \n**2. Hyperparameter Tuning:** TAO requires setting the tree depth and regularization strength. Since deeper trees become “cheaper to train” in your approach, do you recommend always searching for depths beyond the critical threshold? Is there a risk of overfitting when training very deep trees? How do these hyperparameters influence performance in practice?  \n**3. Imbalanced Leaves:** The paper mainly analyzes balanced trees. How does the performance behave when the tree is highly unbalanced?\n\nLIMITATIONS:\nThe paper does not explicitly discuss limitations. Some of the method’s limitations are mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an enhancement to the TAO (tree alternating optimization) algorithm for training regression trees with linear leaves. Specifically, they propose to optimize the leaves’ linear models using the Sherman-Morrison-Woodbury formula, taking advantage of the fact that each leaf only receives a potentially small subset of the full training set. On sufficiently deep trees, this has the potential to lead to significant speed ups, an observation that is verified experimentally across seven datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- A simple improvement to speed-up the computation of linear regression models in the leaves of TAO regression trees\n- The paper provides a clear analysis of the complexity of the proposed improvement\n- Experiments show clear speed-ups over standard TAO\n\n----\n\nWeaknesses:\nWhile the paper proposes a nice improvement that can speed up the computation of regression trees with high dimensionality, I found the paper to have limited novelty and limited significance.\n- Limited technical novelty: TAO is well known algorithm Section 3 is a description of previous work primarily. The main change is the application of the SMW formula to solve the leaf problem faster in the “deep regime”.\n- The experimental results show clear speed-up in performance, however significant gains are saved for very high-dimensional datasets with sufficiently deep trees such that the number of instances reaching a given leaf is smaller than the dimensionality. Gains in ensemble settings are more modest relatively.\n- Deeper trees lead to reduced RMSE on training set, but do not necessarily generalize better as they may overfit. The experiments in the paper are not focused on generalization and investigation of whether the observed gains (that are focused on deep trees) are useful in practice.\n- The work focuses on TAO regression trees exclusively. While the paper note that the main argument can be used to speed up greedy recursive partitioning algorithms such as CART, there is no experimental results demonstrating this. Further, the paper notes that this may extend to other types of trees but there is no additional information, examples, or experiments provided to substantiate this.\n- Overall, while the contribution in the paper seems correct and potentially useful in some settings, it seems to me very incremental with relatively limited significance.\n\n- minor:\n* “Gram matrix” is used in Section 5 but is not previously defined.\n* it would nice to have add y-axis labels to Fig 1 (especially as it seems the label for right-most figures is not the same as the rest?).\n\nQUESTIONS:\nNo specific questions, but please see weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper closely examines the Tree Alternating Optimization(TAO) algorithm for regression trees with linear leaves, which was introduced in an ICML 2020 paper [1]. TAO supposedly provides trees that are \"much smaller and much more accurate\" than those produced by other older popular algorithms like CART. This is believable and I did not verify the results from the ICML 2020 paper. TAO iterations consist of two types of steps, those optimizing the internal decision nodes of the tree, and those optimizing the leaf nodes. The leaf node optimization step is essentially the same as solving a ridge regression problem for the data points sent to that particular leaf. In the original paper introducing TAO, they perform a \"naive\" ridge regression whereas in this submission, they show that they can substantially speed up this step (at least for higher dimensionality of the data and/or higher tree depth) using the Sherman-Morrison-Woodbury formula. Basically, the trees produced by the algorithm provided in this paper are identical to those produced by TAO but they can compute them faster. The submission shows improved computational complexity by some basic analysis and furthermore they also experimentally show that their algorithm is faster than TAO.\n\nReferences:\n1) \"Smaller, more accurate regression forests using tree alternating optimization\" by A. Zharmagambetov and M. Á. Carreira-Perpiñán, ICML 2020\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The submission is technically sound with claims that are well-supported by both theoretical results and experimental validation. The methods used are appropriate and this is a complete piece of work.\n\nClarity: The submission is overall clearly written with minor typos which I list in the *Questions* section of the review.\n\nSignificance: The submission is certainly relevant to NeurIPS as work on regression trees has been appearing at NeurIPS and other related conferences for several decades. Yes, others are likely to build on this work and it is a clear improvement over the prior work on TAO. \n\nOriginality: I think this is the major weak point of this submission. My main qualm is whether the particular contribution of this submission warrants a separate NeurIPS (which is supposed to be one of the top venues) publication. Ideally, if the authors of the ICML 2020 paper had noticed this particular improvement in the work they had submitted then, it would have merited maybe only a paragraph discussion in that original paper with the rest of it pushed to the Appendix. Also, I believe the particular idea of using the Sherman-Morrison-Woodbury formula to speed up regression has cropped up in many other prior papers. But I agree that it is not something which the TAO paper used so this is novel when restricted to the TAO algorithm.\n\nQUESTIONS:\nMinor typos/clarifications:\n\nLines 54-58: I believe you want to mention $N'$ here instead of $N$ where $N'$ is the effective sample size $N2^{-\\Delta}$, which you mention at Line 240. Since you are using $N$ for the size of the dataset in the immediately preceding lines in this paragraph from Lines 47-53, it gets a bit confusing here.\n\nLine 56: but -> by\n\nLine 113: peocedure -> procedure\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}