Title: Empowering Decision Trees via Shape Function Branching
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

trustworthy AI

interpretable machine learning

decision tree
Abstract: Decision trees are prized for their interpretability and strong performance on tabular data. Yet, their reliance on simple axis‑aligned linear splits often forces deep, complex structures to capture non‑linear feature effects, undermining human comprehension of the constructed tree. To address this limitation, we propose a novel generalization of a decision tree, the Shape Generalized Tree (SGT), in which each internal node applies a learnable axis‑aligned shape function to a single feature, enabling rich, non‑linear partitioning in one split. As users can easily visualize each node's shape functions, SGTs are inherently interpretable and provide intuitive, visual explanations of the model's decision mechanisms. To learn SGTs from data, we propose ShapeCART, an efficient induction algorithm for SGTs. We further extend the SGT framework to bivariate shape functions (S$^2$GT) and multi‑way trees (SGT$_K$),  and present Shape$^2$CART and ShapeCART$_K$, extensions to ShapeCART for learning S$^2$GTs and SGT$_K$s, respectively. Experiments on various datasets show that SGTs achieve superior performance with reduced model size compared to traditional axis-aligned linear trees.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the Shape Generalized Tree, which learns a GAM with 1 or 2 features at each tree node (using a decision tree) rather than an axis-aligned split or a linear split. They learn these GAM splits in a recursive CART-style agorithm, and additionally allow for multi-way splits rather than binary splits. Experiments on 12 tabular datasets show that the resulting models can outperform traditional baselines for small model sizes while preserving a form of interpretability that allows for manual inspection of decision paths.\n\nSTRENGTHS AND WEAKNESSES:\n- S1: The authors tackle a well-motivated problem\n- S2: The authors propose an intuitive method and explore its natural variants\n- S3: The paper's writing is clear and the authors choices' for variations of the method seem reasonable\n- W1: The authors are missing comparisons to relevant baselines for compact interpretable models such as GAMs (e.g. the [explainable boosting boosting machine](https://github.com/interpretml/interpret)), optimal trees (e.g. [GOSDT](https://arxiv.org/abs/2006.08690)), regularized trees (e.g. [HSTree](https://arxiv.org/abs/2202.00858)), and tree sums with limited feature interactions per tree.\n- W2: The experimental results should be more thorough. The 12 selected datasets are somewhat arbitrary and could include a much wider set of a more standardized benchmark (e.g. [PMLB](https://github.com/EpistasisLab/pmlb)). For the depths that the authors evalute, all methods seem to underfit the data (atleast the performance continues to increase through depth 6, the highest depth evaluated). This suggests that in a realistic setting, users may require larger depths, especially after cross-validation. Perhaps the authors may want to evaluate on commonly used high-stakes datasets which are known to admit interpretable classification models (e.g. the classification datasets in [this work](https://arxiv.org/pdf/2201.11931))\n- W3: It is also unclear whether models of different depths are comparable when shape functions include more parameters and are more difficult to interpret.\n\nQUESTIONS:\nIt might be interesting to see more explorations of univariate shape function beyond using CART.\n\nLIMITATIONS:\nBesides the weaknesses listed above, the paper should include a more thorough discussion of the interpretability limitations introduced by the SGT. The nonlinearity introduced by the tree structure may make the SGT more difficult to interpret than a GAM, even one that has many more features (esp. if it makes the individual component functions for different features less understandable). Moreover, interpreting individual shape functions adds complexity beyond a typical binary axis-aligned split.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces SGTs, a novel type of decision trees that supports non-linear branching functions at each internal node. In doing so, these models are able to represent complex decision functions more concisely than traditional linear splitting decision trees. Additionally, algorithms for fitting SGTs (ShapeCART) , a generalization of SGTs that supports multi-way splitting (ShapeCART_k), and a generalization that supports bivariate splitting functions are introduced. These trees are shown to improve accuracy relative to baseline greedy decision tree methods across several datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This work is well motivated, and identifies a nuanced but clear problem in existing decision trees (inefficiency in representing some kinds of splitting functions).\n- The exposition is very clear, and the figures are helpful. In particular, the visualization technique used for SGTs is very clear.\n- Fitting SGTs is a somewhat difficult multipart optimization problem. This work presents reasonable simplifications to make this a feasible problem to solve; in particular, the approach of first learning a binning function, then learning bin assignments to learn each splitting function is quite elegant.\n- The evaluation is thorough and well described, considering multiple datasets of varying scale.\n\nWeaknesses\n- My primary concerns with this work have to do with the presentation of the experiments. If these can be addressed, I think this paper is a solid accept.\n  - The Z-score presentation of results is quite strange, and hard to interpret. Moreover, I see no reason to think that the aggregated test accuracies are normally distributed or support a CLT — they would be drawn from some kind of mixture distribution. As such, Z-scores are not an applicable tool. Additionally, it does not make sense to aggregate Z-scores over depth bounds. It may be more appropriate to present accuracy numbers from a reasonable subset of datasets in the main paper for comparison. I believe tables 1 and 2 could be merged if accuracy is reported directly, allowing space for 3-4 datasets to be included.\n  - In the presentation of results in the appendix, grouping tables by method makes comparisons quite difficult. I would recommend grouping them by dataset, making head to head comparisons easier.\n  - While greedy methods are the most direct competitors, globally optimal trees could be run for the shallower depth bounds, and should be included as competing methods. It does not necessarily take away from ShapeCART if they happen to find superior performance, since they likely scale less well.\n\nQUESTIONS:\nI have no major questions, but one minor copyediting note:\n- Should w_d be unbolded in Equation 6? My understanding is that it would be a scalar.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

comments: SUMMARY:
This paper proposes to extend the top-down induction tree learning paradigm from only considering linear axis-aligned splits to both (1) non-linear axis-aligned splits on a single feature by using shape functions, (2) non-linear splits on a pair of two features, and (3) multi-way splits (although in practice limited to three-way splits). In comparison with linear axis-aligned splits, the new splits can compactly capture more patterns, whereas in comparison to oblique splits, these shape functions remain more interpretable.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed method is simple and explained well
- The paper examines a variety of novel split types (shape functions, shape functions on pairs of features, multi-way splits)
- The formalism is useful and appropriate
- The related work contains a reflection on the different types of interpretability and the applicability to trees

Weaknesses:
- The experiment hyperparameter tuning needs more details. There are several hyperparameters that are tuned, but how? (1) L, Number of bins per shape function, (2) min. samples per bin, (3) gamma, regularization hyperparameter, (4) lambda, penalization of multi-way branching 

Small comments
- “For bivariate shape functions, we instead train a binary bivariate linear tree using BiCART”. More details is needed. Also, what is a linear tree?
- Eq. 11 needs a comma between the two constraints for readability
- In Def. 5, k is not defined (only later). Also k>=2 in the text, but in Eq. 5, k >= 1

QUESTIONS:
1. Categorical variables are transformed using one-hot encoding. This means the only possible shape function is the standard binary split, right? Are other encodings of categorical variables possible to exploit the value of shape functions?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ​Shape Generalized Trees (SGTs)​, a novel class of interpretable decision trees that replace traditional axis-aligned linear splits with ​learnable shape functions​ applied to individual features in each node. These non-linear functions capture complex feature-target relationships within single nodes, significantly reducing tree depth and size while maintaining model transparency. The authors develop ​ShapeCART, an efficient induction algorithm that constructs univariate SGTs using an interpretable binning-and-assignment approach, alongside extensions for bivariate splits (Shape²CART) and multi-way branching (ShapeCARTₖ). Experiments demonstrate that SGT variants consistently outperform conventional trees (CART, SERDT) and bivariate methods (BiCART) across tabular datasets, achieving ​higher accuracy with more compact architectures. Crucially, SGTs retain explainability through ​visualizable node-level decisions, as illustrated in the eye-movements dataset example where non-linear splits replace multiple linear thresholds. This work bridges the expressivity of generalized additive models with the structural clarity of decision trees.\n\nSTRENGTHS AND WEAKNESSES:\n**Related Work and Theory**: The paper establishes ​empirical superiority​ of SGTs over CART but omits ​theoretical guarantees​ comparable to recent CART analyses (Klusowski 2020, 2024; Zheng 2023). While ShapeCART's greedy optimization shares CART's structure, its ​non-linear shape functions​ fundamentally alter splitting mechanics. Future work should address: (1) ​Approximation bounds​ for function classes captured by piecewise-constant shape functions; (2) ​Consistency rates​ under the binning-and-assignment paradigm; (3) ​Sparsity guarantees​ given the coordinate descent assignment. Connecting to Klusowski's minimax optimality for additive functions or Zheng's consistency analysis would ground SGTs' advantages in modern learning theory.\n* Klusowski, Jason M., and Peter M. Tian. \"Large scale prediction with decision trees.\" Journal of the American Statistical Association 119.545 (2024): 525-537.\n* Klusowski, Jason. \"Sparse learning with CART.\" Advances in Neural Information Processing Systems 33 (2020): 11612-11622.\n* Zheng,  et al. \"On the consistency rate of decision tree learning algorithms.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n**More Experiments**: \n1) Hyperparameter sensitivity analysis​ lacks visualizations of key parameters' impact (e.g., bin count L, pairwise regularization γ) across tasks—critical given their effect on model compactness; ​\n2) Visualization limitations—while Figure 2 illustrates a univariate SGT node, the paper doesn't showcase ​bivariate/multi-way shape functions​ (promised in Sec. 4) or ​comparative architecture diagrams​ against linear trees (as implied in Fig. 1c); \n3) Baseline contemporaneity gaps—recent advances like NODE-GAM and interpretable oblique trees are absent despite relevance to SGT's \"interpretable non-linearity\" claims. While depth-controlled comparisons are rigorous (Tables 1-2), adding ​parameter sensitivity curves, ​bivariate shape visualizations, and ​modern baseline comparisons​ would better substantiate the novel structure-performance trade-offs central to SGTs' contribution.\n\nQUESTIONS:\nSee weaknessses\n\nLIMITATIONS:\nNan", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}