Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting
Authors: Hongyi Li
Han Lin
Jun Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Optimization
Regression trees
Newton method
Convergence
Primary area: optimization

Abstract: Oblique decision trees combine the transparency of trees with the power of multivariate decision boundaries—but learning high-quality oblique splits is NP-hard, and practical methods still rely on slow search or theory-free heuristics.
We present the Hinge Regression Tree (HRT), which reframes each split as a non-linear least-squares problem over two linear predictors whose max/min envelope induces ReLU-like expressive power.
The resulting alternating fitting procedure is exactly equivalent to a damped Newton (Gauss–Newton) method within fixed partitions.
We analyze this node-level optimization and, for a backtracking line-search variant, prove that the local objective decreases monotonically and converges; in practice, both fixed and adaptive damping yield fast, stable convergence and can be combined with optional ridge regularization.
We further prove that HRT’s model class is a universal approximator with an explicit $O(\delta^2)$ approximation rate, and show on synthetic and real-world benchmarks that it matches or outperforms single-tree baselines with more compact structures.

Published as a conference paper at ICLR 2026
HINGE REGRESSION TREE: A NEWTON METHOD FOR OBLIQUE REGRESSION TREE SPLITTING
Hongyi Li, Han Lin & Jun Xu*
School of Intelligence Science and Engineering Harbin Institute of Technology, Shenzhen Shenzhen, 518055, China
(23b904 000990045,,00 515, 24s15308 @ēstu. www. edt..ed.ccm
xu sujunggy@hitt.ede edw.
Code: httpa://github.. ctt. com.ooooy.....oo.iioooi.oo.. wttp:/ooogg----ssllinnn-e- Regression oi Tree
ABSTRACT
Oblique decision trees combine the transparency of trees with the power of multi- variate decision boundaries- learning high-quality oblique splits is NP-hard, and practical methods still rely on slow search theory-free heuristics. We present the Hinge Regression Tree (HRT), which reframes each split as a non- linear least-squares problem induces ReI.U-like expressive power. The resulting alternating fitting procedure is exactly equivalent to damped Newton (Gauss-Newton) method within fixed partitions. We analyze this node-level optimization and, for backtracking line- search variant, prove that the local objective decreases monotonically and con- verges; in practice, both fixed and adaptive damping yield fast, stable convergence and can be combined with optional ridge regularization. We further prove that HRT's model class is a universal approximator with explicit ( approxi- mation rate, and show on synthetic and real-world benchmarks that it matches or outperforms single-tree baselines with more compact structures.
linear predictors whose max/min envelope
INTRODUCTION
Decision trees are among the most influential models in supervised learning due their inter- pretability and ability to capture nonlinear relationships. The classical CART framework (Breiman al.) 1984) introduced axis-aligned recursive partitioning, which remains cornerstone modern tree-based methods. However, such axis-aligned trees often require deep structures to approximate even simple relationships high-dimensional or correlated settings, limiting efficiency and gener- alization (Hastic et al. 2009).
To address these issues, oblique regression trees extend splitting criteria from axis alisned thresholds to hyperplanes defined by linear combinations of features. This formulation yields more compact structures and improved predictive performance, particularly when features are correlated or rotated (Murthy al. 1994). Nonetheless, finding the optimal oblique hyperplane is NP-hard (Laurent) & Rivest 1976 Hancock et al.) 1996), and practical algorithms have relied on greedy heuristics, evolutionary methods, convex surrogates (Loh) 2014). Recent work explores aptimization-based formulations (Panda et al. 2024 Karthikeyan et al. 2022), but efficient and theoretically sound solutions remain limited.
We introduce novel oblique regression tree algorithm, termed the Hinge Regression Tree (HRT), that fundamentally redefines the node splitting problem. Specifically, we propose learn each node split nonlinear least squares optimization problem involving two distinct linear models. The basis function employs a hinge formulation to model this nonlinear optimization, intrinsically endowing HRT with ReLU-like nonlinear expressive power. To improve robustness under mul- ticollinearity, ridge regularization (Hoerl & Kennard 1970) optionally incorporated within the
Also with the Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments. "orresponning author: Jun Xu.

---

Published as a conference paper at ICLR 2026
fitting step. The resulting iterative optimization can be interpreted damped Newton (Gauss- Newton) method within fixed partitions. Furthermore, we show that, for a backtracking line-search variant, the node-level objective decreases monotonically and converges, and that, when the partition stabilizes, the iterates converge to the corresponding OLS minimizer, while in practice both fixed and adaptive damping exhibit fast and stable convergence. Separately, we establish that the induced piecewise linear model class enjoys strong universal approximation guarantees.
Our contributions can be summarized as follows: (1) We introduce novel HRT algorithm that refor- mulates node splitting as a nonlinear least squares optimization over two linear functions. The result- ing HRT model, by hierarchically composing its max/min envelope- baseed splits, intrinsically gains ReLU-like nonlinear expressive power and supports optional ridge regularization. (2) We charac- terize the node-level alternating optimization as a damped Newton/Gauss-NNew -Nwton method and, for a backtracking line-search variant, prove that the node-level objective decreases monotonically and converges, thereby providing a theoretical foundation for its efficiency and stable behaviour in prac- tice. (3) We establish that the resulting piecewise linear models constitute a universal approximator with an explicit (62) approximation rate, validating HRT's powerful function approximation ca- pabilities. (4) Through extensive experiments synthetic and real-world datasets, we demonstrate that HRT achieves competitive performance compared to single-tree baselines while maintaining a more compact structure. By integrating regression modeling objectives with optimization theory, our work advances oblique regression trees as both a practical and theoretically principled tool for nonlinear function approximation.
RELATED WORK
Oblique regression frees. The quest for optimal oblique splits dates back decades, driven by their ability to form more compact and expressive decision boundaries compared to axis-aligned trees Murthy et al.
1994 Loh, 2014). Early approaches largely relied on greedy heuristics, such as OC1 (Murthy et al.
like GUIDE 1994 which employed randomized search, or statistical tests for feature selection, However, the NP-hard nature of finding optimal oblique hyperplanes (Lau-) rent & Rivest 1976) has motivated shift towards more sophisticated optimization-ban -based method- ologies. These include alternating optimization strategies, which iteratively refine split parameters and data assignments (c.g.. TAO (Carreira- Perriiáá & Tavallali 2018 ). and gradicnt-bartt -basnd opti- mization techniques. Recent differentiable oblique trees, such as DGT (Karthikeyan et al.) 2022), train trees end-to-end using gradient descent with approximations like over-parameteriiatio and trrough estimators. Similarly, DTSemNeet (Panda et al. 2024) formulates oblique decision trees as neural networks, enabling optimization with standard gradient descent. While these methods have significantly advanced the field, they often rely on heuristics, approximations, or specific neu- ral network architectures. Our work distinguishes itself by reframing the core splitting problem as a direct, nonlinear least squares optimization with a clear equivalence to damped Newton method, providing theoretically grounded and efficient alternative.
Piecewise linear models and hinge functions. Regression trees are essentially a piecewise lincar approximation that recursively partitions the input space and fits simple model (typically constant or linear) within each region. The universal approximation property of piecewise linear functions is cornerstone of approximation theory, providing a theoretical foundation for the expressive power of such models (Cybenko) 1989 Hornik 1991). More recent developments confirm explicit conver- gence rates for oblique trees (Cattaneo et al. (2024), while deep networks provide complementary insights into piecewise linear expressivity (Hu) (202)). Hinge functions are indispensable piecewise linear primitives in machine learning models, valued for their moo-shottsssss and geometric inter- pretability (Ergen & Pilancil (02T). As pioneering work, hinging hyperplanes (Breiman 1993) construct global additive expansions of hinge units. Their influence extends broadly, from inspiring the hinge loss used in SVMs (Cortes & Vapnik, 1995) to motivating the ReLU activation func- tions that dominate modern neural networks (Nair & Hinton 2010; Glorot et al., 2011). Owing to their piecewise linear structure, hinge functions are particularly effective for defining efficient and interpretable decision boundaries. However, the reliance on additive combinations in hinging hyperplanes obscure the interpretability of the resulting models. In contrast, our method di- rectly leverages the characteristics of hinging hyperplanes to formulate node splits, while improving interpretability through explicit expressions of the corresponding subregions
2

---

Published as a conference paper at ICLR 2026
3 TREE CONSTRUCTION AND OBLIQUE SPLIT OPTIMIZATION
This section describes our procedure for constructing the HRT, an oblique decision tree. We first specify the construction objective, then detail how oblique splits are optimized each internal node, followed by the recursive routine used to construct the full tree, Our key insight is to formulate each split as nonlinear least-squares problem over two linear models. This formulation allows the tree to directly learn a locally adaptive piecewise-linee linear structure, from which the splitting hyperplane arises naturally.
3.1 CONSTRUCTION OBJECTIVE OF THE HRT
Consider dataset S { M1) 1- where B2 R and Uy R. suppose I Rd+1 is the augmented feature vector, i.e., 2 = [1,,.... Pd, 1]T Assume there are T nodes Dr, indexed in breadth-first order by f E T (1.....7) T). The nodes can be classified into two types: internal nodes, which execute branching tests and are denoted by E T1, and leaf nodes, which are utilized for prediction and are denoted by E Tr. Starting from the root node, the tree grows through the internal nodes to reach the leaf nodes. For each node De, E T. there is an associated parameter vector 0, Rd+1 describing the corresponding linear relationship. Let (x()) be the leaf index reached by input æ, and let its linear predictor be l, (x) = x (z))) The tree' 's output function is thus ((x) =
The global approximation accuracy refers to the training set is then described
J(D 0)
> (y ((x)))
where (xx)) (z.) (2,), meaning that the input x, belongs to unique leaf node (1,)),)) with (xx)) E TT. The indices of the internal nodes and leaf nodes, i.e. Tr and Tr, are dynamically changing during the construction the HRT.
3.2
OPTIMIZATION OF OBLIQUE SPLITS
3.2.1
OBJECTIVE FUNCTION
At any internal node Dr, / E TT, our goal is to find two sets of parameters, 0z 0,2 E RR+1 and these parameters define two linear functions, l,, (a:) a 0r, and 6,, (ar n 0r, that best fit the data. Specifically, we aim to minimize the following nonlinear least squares objective function:
V(0)
Σ ( ())
2
2, ED:
with respect to 0 =00T, 1T E (d= which is the total parameter vector, and h(æ,, 0) is defined as:
h( 0) = max ( 0r, B..) or Mæx, 0) = min (0 (2)00,, 0r, x. 0r.)
The basis function h(æ,, 0) is a hinge function. This function intrinsically defines a decision bound- ary in the data space, given by the hyperplane ã (0., 0.,) = O, where l,, (x) = l, (x)
(x) De-
pending which side of this hyperplane data point æ, lies (i.e., whether . 0,, $ 0., or
vice versa), the model selects either l,, (ær,) l, (ær)) for prediction. This hinge-enabled piecewise linear structure flexibly adapts to both convex and concave local data structures.
From model perspective, minimizing V(O) corresponds optimizing structure with one root and two leaves. The root uses the hyperplane (0,, 0.g) = O as its decision boundary to partition the data, i.e., the internal node Dr is partitioned into two leaf nodes Dr, and DD Each leaf corresponds to linear function E,, (z) or L,, (x) used for prediction in their respective data subsets.
3.2.2 OPTIMIZATION AS NEWTON METHOD
We address the nonlinear least squares optimization problem by employing an iterative procedure, which we formulate Newton method to derive efficient solution.
3

---

Published as a conference paper at ICLR 2026
Directly minimizing Equation is difficult due to the non-differentiable function. Our iterative algorithm operates by defining two dynamic partitions of the data based on the current parameters 0:
(i = {a, E Dr 05 2 0ta) S2(0) = ID, \ ())))))))))))))))()))
For a fixed partition S2), the objective function is differentiable with respect Oz, and Oz within each respective domain. Under this condition, the Newton update relies on the gra- dient VV and the Hcssian matrix V2V. Generally, Newton update is given by ((K+1) (()) - - In our case, due to the locally linear nature of h(,; 0) within each fixed partition, its second derivative is zero, which means the Newton approximation to the Hessian is exact. Combining this exact Hessian with the gradient structure (as derived in detail in Appendix A). this update simplifies to:
((k+1) = + (0 - (((b))
where (kk is the current parameter, and ( represents the optimal Ordinary Least Squares (OLS) solution independently computed for its respective data subset based on the current partitions (O(()) and (2(())) The term (0ors ((*)) corresponds to the Newton direction VV for the current fixed partition. f is the step size. After ((++1) is computed, the data points are then reassigned new partitions S (0(k+1)) and (2((((++)) for the next iteration, based on these updated parameters. This alternating procedure of parameter fitting and partition re-assignment forms the core of our optimization. Our iterative procedure employs step size j (0, 1]. When = the new parameters are directly set to the optimal OLS solution for the current partitions, i... ((K+1) olk) which represents a unit-step Newton update. For a detailed derivation of the general case and its equivalence to OLS when μ 1, please refer to Appendix [A] We support two step-size strategies in practice, both sharing the same Newton direction but differing in how (x()) is chosen: (i) fixed damping factor μ E (0, 1]. which yields a simple and efficient implementation. and (ii) an automatic backtracking line search (denoted as "auto" in our experiments), which selects (A()) adaptively to ensure sufficient decrease of the node-level objective.
3.3 RECURSIVE TREE CONSTRUCTION
The core of the HRT construction is recursive splitting process. Starting from the root node, each internal node undergoes the node-level optimization procedure described in Section 3.2.2 This procedure iteratively finds the optimal oblique hyperplane, thus creating linear decision boundary.
Data points are subsequently assigned to the left or right child node depending on whether a aTO 0,, 2 a" Ozg. The split recursively applied to the resulting child nodes until a stopping condition is satisfied, such reaching the maximum tree depth, falling below minimum number of samples, or achieving a Root Mean Squared Error (RMSE) below a predefined threshold. Through this recursive mechanism, the tree naturally composes hierarchy of hinge operations. This hierarchical structure endows the model with ReLU-like expressivity.
To enhance model robustness, all OLS fitting steps optionally incorporate ridge regression (L2 regularization). Furthermore, to ensure continuous tree growth, even if the node-level iterative op- timization fails to converge within specified number of iterations, simple fallback mechanism is employed to determine the split. Detailed information on these mechanisms, including the mathe- matical formulation of ridge regression and the specific implementation of the fallback strategy, can be found in Appendix C The detailed pseudocode for the optimal node-splitting mechanism and the overall buildinig procedure, along with their computational complexity analysis, can be found in Appendix D and respectively.
3.4 ILLUSTRATION OF THE RELU-LIKE EXPRESSIVE POWER
Each internal node D, carries two linear scores &, (z) = T T and = routes data points by the hinge test sx(x) L,, (x) L,, (x) 0 (left child) or 0 (right child). A depth-k tree induces a polyhedral partition of the input space by intersecting halfspaces along
paths.
According to Section 3.1 for each input ær, the tree's output function is ((x) (e) (2r), which
4

---

Published as a conference paper at ICLR 2026
is piecewise linear function defined most 2 regions. Define the ReLU function au(u) max(u, u, 0}. For any scalars a, b, we have max(a, b) = a + a(b-a) a) and min b) = a a -a-bb b). This implies that binary hinge operation at node is essentially single-unit ReLU gate acting on linear arguments. Composing these gates along the tree structure yields a computational circuit formed by linear maps and ReLU operations, which justifies its ReI.U-like expressive power.
THEORETICAL ANALYSIS
4.1 CONVERGENCE ANALYSIS
Our algorithm uses alternating optimization strategy. We update parameters by (k) OlS - (())
g (k++1) -b(4) - (()))(())
p(k)
e
where, as shown in Section 3.2, ( is a Bision/Gauss-Newtto direction under the current parti- tion.
In practice we instantiate this generic update rule two ways. The first uses a fixed damping factor ( μ (0, 1]: the value of ş (c.g.. 0.1, 0.5, 1) treated as a hyperparameter and selected on validation split. The second, denoted as auto in our experiments, employs simple monotone backtracking line search: starting from μ = 1 we geometrically decrease μ until a valid split with strictly smaller node-level RMSE is found, terminate if no such step exists.
For our theoretical analysis we focus on this line-search variant. In Appendix we show that, under mild assumptions, the resulting damped Newton method yields monotonically decreasing sequence of node-level objectives that converges, and that when the partition stabilizes the iterates converge to the corresponding OLS minimizer. Empirically, we observe that both the fixed-step and auto line-search schemes exhibit fast and stable convergence. Practical safeguards include optional ridge regularization the OLS fits to improve conditioning, minimum-sample and error-based stopping at nodes, and simple fallback (median split) if the node-level optimization fails to make progress within Taax iterations. We terminate when the relative change in V or falls below a tolerance, or when the partition stabilizes.
4.2 UNIVERSAL APPROXIMATION THEORY
This section establishes the theoretical foundation for HRT's expressive power. We prove that its piecewise linear model class a universal approximator for continuous functions and derive an explicit approximation rate that quantifies how its precision scales with domain partitioning.
Theorem 1.
Let F be the class of piecewise linear functions represented by finite oblique regression trees with linear models at the leaves. Let g K + R be a twice continuously differentiable compact set K. C. Rd. Furthermore, assume that for any constructed partition of KC into regions {Ra} } with diameter I 6, each region R, contains N; training data points (0):g(0,)
function (g E (²(K))
Let X; be the N; x (d 1) design matrix whose j-th row is a.} We further assume that X; satisfies Amin (XP X2) cN, for some constant > O. Also, sup |ãlla Dr for some bounded constant Dx- Then, there exists a constant C such that for any 8 > O, we can construct function f E F by partitioning the domain K. into regions {R₂} with max, diam (R,) I 5, satisfying the uniform error bound:
sup (f(x) 99(x) < 28
#EK
This approximation rate directly implies the universal approximation property, i.e., for any > O. there exists a function E F such that sup=ex f((x) - -=(=) <
For the detailed proof, please refer to Appendix
Intuition. Our HRTs serve as universal approximators by recursively partitioning the input space into convex regions and fitting local linear models in each. As partitions refine (region diameter 5 decreases), the piecewise linear predictor uniformly approximates g(z), achieving an O(82) ap- proximation rate under standard smoothness assumptions. We assume eigenvalue lower bound
5

---

Published as a conference paper at ICLR 2026
for each leaf Amia (X) cN, which keeps local OLS well posed so that its estimation error matches the (8))) approximation term. In practice. strong collinearity or small leaf sizes may vi- olate this condition; we therefore optionally employ ridge regularization, replacing the local closed For any a > O, this raises the smallest eigenvallu the penalized subspace and stabilizes the normal equations.
form with (X? X; +
EXPERIMENTS
To validate our algorithm's effectiveness and properties, we conducted experiments addressing three key questions: (1) Does the core splitting algorithm converge efficiently as predicted? (2) Can the resulting tree model effectively approximate complex continuous functions, verifying our universal approximation claim? (3) Does our method perform competitively on real-world regression tasks?
5.1 CONVERGENCE ANALYSIS OF THE SPLITTING ALGORITHM
This experiment investigates the convergence dynamics of our splitting algorithm, focus- ing on the critical role of the step size μ.. We aim to empirically demonstrate the trade-off between convergence speed and stability by testing the algorithm on two distinct synthetic datasets: one challenging and unstable, and another bell-behaved yet non-trivial. This dual approach validates our theoretical claim that while the algorithm equivalent
Objective:
damped Newton method within fixed partitions, a damped step size (μx < 1) is essential for robustness, whereas unit step can achieve rapid convergence in stable scenarios.
Setup: We generate two datasets, each of N = 1000 samples with Gaussian noise (e NOO, 1)).
For the unstable case, data are generated from the sinc function, y 5xæ 0.025e, on E [-1.5, 1.5). Its multiple local extrema and sharp oscillations create challenging optimization landscape. For the stable case, data are generated from a smooth, nonlinear function with distinct inflection point, y = i 0.8z + 0.025e, on [-3, 3]. This stedssigioooo function is an ideal well-behaved target, as it requires nonlinear fit but lacks the sharp features that cause instability. On both datasets, we test four step sizes: μ E {1.0, 0.5,0,1,...0.0000000 0.05).
sin(Smrr
Results and Analysis: The results in Figure T (unstable case) and Figure 2 (stable case), show the role of the step size. As shown in Figure [ the results on the sinc function highlight the necessity of damping for stability. The unit Newton step (μx 1.0) is unstable. Its aggressive updates cause partition collapse within the first few iterations, making the algorithm revert to a poor global linear fit. The large damped step (μ = 0.5), while avoiding outright collapse, becomes trapped in limit cycle; the model's 's parameters and the corresponding data partition oscillate between a small set of states without ever converging to a fixed point. Small steps (μ- 0.1, 0.05) prove robust, converging to a high-quality piecewise linear model that accurately captures the function's complex structure. This clearly demonstrates that for challenging problems, effective damping is not merely heuristic but a prerequisite for success.
Results on the twisted sigmoid function (see Figure for visualization and detailed analysis), illustrate the efficiency benefits of the Newton updates. Here, all step sizes converge to the same solution around the function's inflection point. Crucially, the unit Newton step (μ 1.0) exhibits the fastest convergence, reaching the optimal solution in just a few iterations. As the step size decreases, the number of iterations required for convergence predictably increases, showcasing the classic speed-stability trade-off.
Taken together, these experiments demonstrate our algorithm's behavior. functions as damped Newton method. On complex, unstable problems, small step size (μ < 1) is essential to ensure robust convergence by preventing partition collapse. On well-behaved problems, a unit step size (μ = 1) leverages the full power of the Newton update, achieving extremely rapid convergence. This confirms the algorithm's 's theoretical foundation and provides a clear practical guideline: the step size crucial hyperparameter balance convergence speed against stability, depending the nature of the problem. Additional ablation studies on the fallback mechanism are reported in A Appendix G
I serves

---

Published as a conference paper at ICLR 2026
Data
Initial Split
Start Point
Stepm1.0 1.0, single line
Stepm0,5
Stepm0.1
Step=0.05
37.5
35.0
0.0
Loss
32.5
0.5
30.0
27.5
-1.0
25.0
20
40
60
11,5
-1.0
-0.5
0.0
0.5
1.0
1.5
Iteration
x
Figure 1: Node-level convergence analysis on the unstable sinc function. Left: Objective value per iteration for single internal node (fixed initialization and data subset). The unit step (u 1.0, blue) and a large step (μ = 0.5, orange) do not decrease the objective monotonically in this example, and the latter gets trapped in a limit cycle. Smaller damping (μ4 0.1, green; l 0.05, red) yields much more regular local Newton dynamics this node. Right: Final fitted models for this controlled experiment. With large steps the node effectively collapses to a poor single linear fit, whereas sufficiently damped updates recover meaningful piecewise linear approximation. Note that this figure illustrates local node-level behaviour; full-tree performance with fallback is analysed in Appendix G
Data
Initial Split
Start Point
Step-1.0
Stop-0.5
Stap-0.1
Stop-0.05
100
2
80
LOSs
60
10
20
30
40
-2
-1
3
Iteration
X
function. Left: For this node, all step sizes lead to monotone decrease of the objective. The unit Newton step (μ = 1.0, blue) reaches the local minimum in the fewest iterations. Right: All step sizes arrive essentially the same high-quality piecewise linear fit around the function's 's inflection point. This illustrates that, stable problems, even aggressive Newton steps can behave well at the node level. Global training behaviour across all nodes and datasets is reported Tables 5
Figure 2: Node-level convergence analysis on the well-behaved
5.2 FUNCTION APPROXIMATION ON SYNTIIETIC DATA
This experiment validates the piecewise linear approximation of our proposed oblique regression tree and connect it with the theoretical Theorem demonstrating its superiority in fitting complex functions.
Objective:
Setup: We evaluate our method on both 2D and 3D regression tasks. For 2D tasks, we use sin(Srar)
5 on E [-1.5, 1.5] and the twisted.sigmoid d function y I+e 3- 0.8z r [-3.3]. For 3D tasks, evaluate method's performance on four oscillatory surface functions with complex characteristics (detailed equations available in Appendix H), with inputs æ1, 22 both ranging from [-3, 3]. All synthetic data generated by adding independent and identically distributed zero-mean Gaussian noise to the true function values. Specifically, the noise standard deviation is 0.025 for 2D tasks and 0.05 for 3D tasks. A detailed rationale for the selection of these synthetic functions and noise levels is provided in Appendix H For each task, we generate a sufficient number of data points (1000 total for 2D tasks, 10000 total for 3D tasks) and split them into 70% for training and 30% for testing.
two classic test functions: the sinc function =

---

Published as a conference paper at ICLR 2026
We compare our piecewise linear regression tree with two representative baseline methods, CART and XGBoost. These baseline models are implemented using the s scikit-learn library. Our proposed piecewise linear regression tree is trained using Algorithm + Hyperparameters for all models are optimized via five-fold cross-validation and grid search on the training set, with final performance reported the testing set.. For detailed hyperparameter configurations, please refer to A
Table I:
Performance comparison on 2D and 3D synthetic functions. All values are reported as mean + standard deviation over repetitions for 2D functions and five repetitions for 3D functions. For RMSE and MAE, lower is better (J). For R higher is better (1). Best results are bolded. Significant improvements over the best baseline are marked with Y (p- < 0.05).
Gin2 MAB())
tvisted sicmold
Model
RMSH(4)
R2th)
Model
RMSH(U)
MAB(J)
RK
CART
0.3355
0,0012
0,0254 +: 0.0009
0,9831 + 0.0049
CART
0,0312 +: 0.0011
0,0249 = 0.0099
0.9976 + 0.0002
0,0289
0,0228 +: 0.0099
0.9868
8.0011
0.02%
0.005
0.0226
0.007
0,9080
0.0001
Ours
0.0099 0,0280
0.0222 1: 0.0008
0.9876
1: 0.0028
Ours
0.0258 1 0,00047
0.0205
0.0003
0.998
1: 0.00012
fn (2x1 22)
(a x2)
CART
0.8522
0,0148
0.6530 1: 0.0127
0.9945
0.0001
CART
0.2721
0.0028
0.1972
:: 0.0024
0.300 :: 0.0011
XGB
0.3018
0.0083
0.2317 + 0.0060
0,9993
+ 0,0000
XGB8
0,0859
0.0615
0.0681
+ 0.0013
015920
+ 0.0003
Ours
0,07967 0.1646
0.1080 0.0322
0.9998
0.00031
0,0757
0.00201
0.0587
0,00161
0.9946
0.0003+
[a #22)
[4(11_
0,0587
0.9832
0.0005
CART
0.0014
0.0575
0.0005
0.9945
0.0002
XGB
0.537
0.0003
0,0428-2
0.0001
0.9914
0.0002
XGB
0.0556 0.0005
0.048
0.0003
0.9973
0.0001
Ours
0.0528 = 0.00847
0.0420 + 0.00033
0.9917 : (L00021
Ours
0,0555 + 0.0007
0.0436 = 0.0004
(.9973 = 0.0001
2D and 3D Experimental Results The results, summarized in Table I demonstrate our method's ability to effectively approximate complex functions across both 2D and 3D tasks. For detailed visual comparison of 2D approximation performance and visualizations for f1, f2, fas and fa. please refer to Appendix H The superior performance of our method across various synthetic functions arises from two fundamental design principles. First, the combination of piecewise linear modeling and Newton iterative optimization allows for precise and efficient local approximation. In contrast to traditional piecewise constant tree models, we explicitly fit linear planar functions within each partition and employ Newton updates to quickly converge to high-quality solutions. Second, flexible oblique splits significantly enhance the adaptability of feature-space partitioning. By overcoming the limitations of axis- axis-parallel splits, they yield decision boundaries that better align with the intrinsic geometry of the data, enabling more efficient and accurate region segmentation.
5.3 PERFORMANCE ON WORLD REGRESSION DATASETS
Objective: To evaluate the practical performance of our proposed model standard benchmark regression datasets and compare it against other well-established methods.
Setup: To assess the practical performance of the proposed method, we evaluate it on a diverse suite of publicly available regression datasets, including all seven benchmarks from ZZhrmagaabeeoy al. (202T) and several large-scale industrial datasets such as YearPred. These datasets cover a wide spectrum of feature dimensionalities (Ny) and sample sizes (N.)), from low-dimensional, small-sample tasks to high-dimensional, large-scale scenarios (see Table 2).
We compare against the following strong baselines: D'SemNett (Panda et al. 2024). DGT
(Karthikeyan al., 2022). TAO (oblique and axis-aligned) (Carreira-Perpinán & Tavallafi 2018 Zharmagambetto et al. 2021), CART (Breiman et al. 1984, M5 model trees ( (Wang & Witten, 1997). the linear trees implemented in the 1 iinear-tres library (Cerliani 2022). and XGBoost (Chen & Guestrin 2016) (an ensemble method).
For the seven datasets from Zharmagambetoov et al. 2021 1 we directly cite performance from that work. For the remaining datasets without published results, we reproduce results. The only excep- tion is TAO, which is not publicly available; its reported numbers are directly taken from the original paper. For reproduced experiments, hyperparameters are tuned via five-fold cross- vols-valio on the training set, with final performance reported on the testing set. Specifically. for datasets with pre- defined train/test splits, we adhere to these original partitions. For datasets without such predefined splits, we first randomly divide the entire dataset into a 0.5:0.5 train/test ratio. For detailed hyperpa- rameter configurations, please refer to Appendix [
8

---

Published as a conference paper at ICLR 2026
Table 2: Average RMSE results (lower is better) on regression tasks, std over five runs. The first column lists the dataset name, followed by the number of features Ny and the number of samples Ng. The first seven datasets are from Zharmagambetov et al. (20TT) The reported results for TAO (including TAO-A: axis-aligned and TAO-O: oblique) and CART are taken from Zharmagambetoo jet al. (2021); DGT results from Karthikeyan et al.] (2022); DTSemNet results from Panda et al. (2024); XGBoost results from Zharmagambetov & Carreira-Perpinan (2020). For datasets with reported results in the original papers, we directly cite their results. For datasets without reported results, reproduce the experiments under the same settings. The only exception is TAO, which is not publicly available. Note that for the Ailerons, D-Elevators, and D-Ailerons datasets, the reported RMSE values are scaled by x10- x10-3, and 10 respectively. A dash (-) indicates runtime exceeding 10 hours. Significant improvements over the best baseline are marked with t (p < 0.05).
Dutasct y. Na)
DTSemmNet
IXIT
TAO-A
TAD-O
CART
XGIB
M1
Lascar frce
HRT (ours)
Abolone (8. 43)
2.14 + 0.03
2.15 0:00
2.30
0.58
2.18 2 0.05
2:34
: 0.59
2.20
1.0.0
2.18 = 0.04
2.22
+ 0.05
2.11 =:: 0.06
CPU Sk)
2.65
1: 0.18
2,91 1 0:15
3.26
006:1:00
0.51
2.71 1 0.94
3.28
4: 0.44
2.57
1 0.00
4.16 3: 2.72
3,05
1 0.06
2.56
0005
Allerons (60, 14k)
1:66 0.011
1.72 1.000
2.55
1.76
0.02
2:85
1 0.57
1.72 0.00
1.68 0.00
1,75 1 0.00
1.64 0.60+
CTSlice
0:017
0.00
0.28
2.80
YoutPred (90,55
8.99
0.01
9,05 0:01
9.11
0.05
9.79
0.54
9.43 0.03
9:15
Concrese (8, 1k)
7.56
0.33
7.43 0.224
7.20.13.17
7.17
100111000 0.43
7.22
3.13
6.75
1. 0.21
6,63
0.12
6.93 0.74
6.92 0.24
Airfoil (5. 2k)
3.83
1: 0.16
3,72 1.0.0
2.73:10.62
3.13 -1 0.38
2.75
1: 0.62
2.77 0,10
3:21 1: 0.06
2.81 0:1.1
2,43
0.10
Fried 413)
1.51
0.00
2.27 1: 0.09
NIA
NIA
1.06
2: 0.01
1.09
1.59 0.02
1,10 1:
1,09
0
Elevators (6, 10K)
1.46
0.00
1.46 1: 0.01
NIA
NIA
1.50 1: 0.02
1.46
0.02
1.43
0.02
1.45 1: 0.02
1.43
0.01
D Aileeoos (5, 7k0
1.76
0.00
1.75 1 0.02
NIA
NIA
1,83 1 0.00
1.72 1 0.05
1,6
0.83
1.72
1 0.02
1.68 0.02
Kinemorics (8, 8k)
0.168
0.000
0.135
1: 0.003
NIA
0,300
4 0.033
0.125
0.001
0,175
0.004
0.141
1: 0.000
0.102:1 0.6031
(127, 20)
0,201
0.000
4:
0,160
0.142
0,004
0.145
0.004
0.009
0,140 0.004
Blog (250, 60k)
32.30
28.35
NIX
26.78
23.02 0.00
22.81
Results and Analysis: As shown in Table our method achieves the best or highly competitive RMSE across broad set of datasets. Among all single-tree models, our method demonstrates superior performance on the vast majority of datasets. In most datasets, the proposed approach yields notable reductions in RMSE. Even for large-scale datasets like YearPred, our performance remains on par with or occasionally superior to the best existing results (including ensemble meth- ods), demonstrating the scalability of our approach. For hyperparameter configurations, please refer to Appendix [
In addition to RMSE, Table 3 assesses the structural complexity of the learned models in terms of average depth (A) and average number of leaves (L) over five runs. Our method typically pro- duces significantly shallower trees with fewer leaves compared to other single-tree baselines. For instance, on Concrete, our model achieves competitive RMSE with a depth of only 3 and 5.8 leaves, demonstrating superior performance compared to CART, which requires depth of 11.2 and 113 leaves to attain higher error. This indicates that the proposed method attains a favorable trade-off between predictive performance and transparency. Regarding training time, as shown in Table 4 our method also shows efficient training performance on multiple datasets. While all experiments in this section focus on regression, preliminary extension of HRT to binary classification re- ported in Appendix K where the same framework achieves competitive AUC and Fl scores with substantially shallower trees.
Table 3: Average depths (A) and average number of leaves (L) over five repetitions for regression datasets. Only single-tree models are included in this comparison; XGBoost is excluded, its depth
and leaf statistics are not comparable to those of single decision trees. A dash (-) indicates that the
corresponding result was not reported in the original source. Datasets and results for TAO (including
TAO-A: axis- aliinnd and TAO-O: oblique) and CART are from Zharragambetoo et al. (2021) 1). DGT
results are from Karthikeyan et (2022). and DTSemNet results are from Panda et 2024
Dataset (Ny, Na)
DTSemNet
DGT
TAO-A
TAO-O
CART
Ouns
L.
Abalone
12.8
12.8
CPUae (21, 8K)
5.0
6.0
9.0
57.2
5.0
527
9.0
57.2
2,0
4.0
Allerons (40, 14K)
5.0
6.0
7.0
15.0
5.0
602
7.0
15.0
2,0
4,0
CTSlice (284. 51k)
5.0
10.0
36.0
700.0
7.0
748
36.0
7000
7.0
93.2
Yeadiee (M0, 515k)
135.0
1579
135.0
Concrete (8, 1k)
5,0
6.0
11.2
113.0
9.0
192.
11.2
113.0
3.0
5.8
Airfoil (5, 2k)
5.0
6.0
15.0
479.8
8.0
147.1
15.0
479.8
5,0
25.0

---

Published as a conference paper at ICLR 2026
Table 4: Training time (in seconds, lower better) on four regression datasets. The first column lists the dataset name, followed by the number of features Nj and the number of samples N,. The numbers parentheses represent the average number of iterations for HRT (ours).
Dutnset yN N₂)
DTSemNett
DGT
CAKT
XGB
M5
Lincar tree
HKT (ours)
D Ailerons (5, 7k)
9.557
00.850
0.008
2.757
0.344
3.001
0.183 (6.7)
Fiid (10, 41k)
69:979
595 243
0.171
9.722
8.322
25.745
2,807 (45.7)
Kinematies
14006
1.447
7,960
(15.8)
(127 2k)
$1.548
0.057
53,801
0.177
0.178
CONCLUSION
We presented the Hinge Regression Tree (HRT), a new oblique regression tree method that formu- lates each split as nonlinear least-squares fit over two linear models connected by hinge. This yields hierarchy of max/min envelope splits that provides ReLUUllike piecewise linear expressive- ness while preserving the interpretability of shallow decision trees.
We showed that the alternating fitting step at each node corresponds to a damped Newton/Gause Newton procedure over fixed partitions. With backtracking line search, we proved monotonic decrease of the node objective and convergence to the OLS solution once the partition stabilizes. In practice, both fixed and adaptive damping schemes converge quickly and stably, and can incorporate ridge regularization for robustness. From a functional viewpoint, we also established a universal approximation property for the resulting seccewise-lineat model class, with explicit O rate that links approximation error to partition granularity.
Experiments synthetic and real datasets demonstrate that HRT matches exceeds strong single- tree baselines while producing significantly shallower trees with fewer leaves, achieving a favorable balance between accuracy, compactness, and interpretability.
Future directions include extending the method to classification and structured outputs, develop- ing more adaptive step-size and regularization strategies for large-scale ill-conditioned settings, and integrating HRT-style node optimization into ensemble methods such as boosting and random forests.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of this work. The detailed mathematical derivations for our novel node-splitting algorithm's is optimization strategy and its equivalence to a damped Newton method are thoroughly presented in Appendix A Complete pseudocode for both the optimal node-splitting mechanism (Algorithm) 2 and the overall tree-building procedure (Algo- rithm 4 is provided in Appendix [D] Additionally, Appendix Couullnnes the optional ridge regression regularization and the fallback strategy for non-convergent nodes. The theoretical analysis, includ- ing convergence properties (further supported by empirical analysis in Section 55 and the proofs for the universal approximation property, are detailed in Appendix [ and Appendix [F respectively. For the experimental setup, comprehensive descriptions of synthetic data generation and definitions (Appendix H.. real-world datasets (Table 13), and hyperparameter configurations for both baseline methods and our approach (Tables 10 and 12] are fully documented in Appendix
ACKNOWLEDGMENTS
This work was supported in part by the Science and Technology Innovation Committee of Shen- zhen Municipality under Grant 129000200111000022000,,, in part by the Shenzhen Science and Technology Program under Grant SYSPG2024121117360000 and in part by Guangdong Provin- cial Key Laboratory Intelligent Morphing Mechanisms and Adaptive Robotics under Grant 0023B111201000
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Leo Breiman. Hinging hyperplanes for regression, classification, and function approximation. IEEE Transactions on Information Theory, 39(3):999-1013, 1993.
Leo Breiman, Jerome Friedman, Richard A Olshen, and Charles J Stone. Classification and Regression Trees. Wadsworth International Group, 1984.
Miguel A Carreira-Perpinán and Pooya Tavallali. Alternating optimization of decision trees, with application to learning sparse oblique trees. In Advances Neural Information Processing Sys- tems, 2018.
Matias D Cattaneo, Rajita Chandak, and Jason M Klusowski. Convergence rates of oblique regres- sion trees for flexible function libraries. The Annals of Statistics, 52(2):466 2024.
Marco Cerliani. Linear-tree: A python library for model trees with linear models at the leaves. 2022.
Tianqi Chen and Carlos Guestrin. Xgboost: scalable boosting system. In Proceedings of the 22nd ACM Sigkdd International Conference Knowledge Discovery and Data Mining, PD- 785-794, 2016.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Con- Irol, Signals and Systems, 2(4):303-314, 1989.
Tolga Ergen and Mert Pilanci. Convex geometry and duality of over-parameterized neural networks. Journal of Machine Learning Research, 22(212):1 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Thomas Hancock, Tao Jiang, Ming Li, and John Tromp. Lower bounds on learning decision lists and trees. Information and Computation, 126(2):114-12)) 1996.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, al. The Elements Statistical Learning. Springer, 2009.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 67, 1970.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, (2):251-257, 1991.
Xia Hu. Understanding deep neural networks from the perspective of piecewise linear property- PhD thesis, Simon Fraser University, 2021.
Ajaykrishna Karthikeyan, Naman Jain, Nagarajan Natarajan, and Prateck Jain. Learning accurate decision trees with bandit feedback via quantized gradient descent. Transactions Machine Learning Research, 2022.
Hyafil Laurent and Ronald L Rivest. Constructing optimal binary decision trees is NP-complete. Information Processing Letters. 5(1):15-17, 1976.
Wei- Wei-Yin Loh. Improving the precision of classification trees. The Annals of Applied Statistics, pp- 1710-1737, 2009.
Wei- Loh. Fifty years of classification and regression trees. International Statistical Review, 82 (3):329-348, 2014.
Sreerama K Murthy, Simon Kasif, and Steven Salzberg. system for induction of oblique decision trees, Journal of Artificial Intelligence Research, 2:1-32, 1994.

---

Published as a conference paper at ICLR 2026
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML), pp. 807-814, 2010.
Subrat Prasad Panda, Blaise Genest, Arvind Easwaran, and Ponnuthurai Nagaratnam Suganthan. Vanilla gradient descent for oblique decision trees. arXiv preprint arXiv:2408, 09135, 2024.
Yong Wang and Ian H Witten. Inducing model trees for continuous classes. In Proceedings of the ninth European conference on machine learning, volume 9, pp. 128-137, 1997.
Arman Zharmagambetov and Miguel Carreira-Perpinan. Smaller, more accurate regression forests using tree alternating optimization. In International Conference Machine Learning, pp. 11398- 11408. PMLR, 2020.
Arman Zharmagambetov, Suryabhan Singh Hada, Magzhan Gabidolla, and Miguel A Carreira- Perpinán. Non-greedy algorithms for decision tree optimization: An experimental comparison. In 2021 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
Detailed Derivations for Equivalence of Iterative Optimization to Newton Method
14
B
Node-Level Convergence of the Line-Search Newton Update
15
Details on Regularization and Fallback Strategies
18
Parameter Initialization Strategy and Detailed Algorithms
19
D.I Parameter Initialization Strategy
19
D.2 Detailed Algorithms
19
Time Complexity Analysis
23
Algorithm I: Find Optimal Split
23
E,2
Algorithm 2: Build Oblique Regression Tree
23
Proof of Universal Approximation and Approximation Rate
23
Ablation Study Step Size: Stability, Efficiency, and Fallback
26
Synthetic Data Description
28
Hyperparameter
Dataset Details
33
K Extending HRT to Binary Classification
34
he Use of Large Language Models (LI.Ss)
34
13

---

Published as a conference paper at ICLR 2026
A DETAILED DERIVATIONS FOR EQUIVALENCE OF ITERATIVE OPTIMIZATION TO NEWTON METHOD
This appendix provides detailed mathematical derivation demonstrating the equivalence of our iterative optimization procedure Newton method.
Recall from Section 3 that at any internal node of the tree D2, for all æ- E Dr, we aim to minimize the nonlinear least squares objective function:
V(0)
Σ haa,,
2 2, ED,
where 0 10%, OT 2( is the total parameter vector. The function h(a,, 0) is defined as 0.,). For clarity, we will provide the detailed derivation for the max form; the derivation for the min form follows analogously with adjusted partition definitions. As discussed in Section 3 directly minimizing V(0) challenging due to the non-differentiabiliyy of the hinge functions at the switching points.
h(x, 0) max (z,0,, 0,, -T or h 0) min (xT 0,, 2.
Our approach can be interpreted as an iterative procedure equivalent the Newton method within fixed partitions. We first define dynamic partitions of the data based on the current parameters 0, as introduced in Section
(1 = {2, = B. 15,00 2 20.021 0
(2) DD (
For fixed partition S2), the objective function is differentiable with respect to O,, and Or, within their respective domains. Under this condition, we can compute the gradient VV and the Hessian matrix V
Gradient Calculation. For almost all (where no data point lies exactly on the boundary 2 Or, E. 0z2),, the model function h(æ,, 0) is differentiable at each data point. Specifically, for h(,,, 0) max 0l).)
a or, > 0 0r,
0
Vh(m;, 0)
if if 0,, $
Thus, the gradient of the objective function is:
N
VV(0) Σ μ - M 0)) 0))
(us h(ar, 0))V), Σ j=1
1=1
Substituting Vh(n,, 0) and summing over the respective partitions:
V -- > (es 25 (v: x7 0.,) (1) ã (y a 0O.))
Hessian Matrix Calculation. For generic nonlinear least squares problem, the Hessian matrix is given by:
((Vhha VV(())= = ())Ma,, - -v; - Mar, 0) ()),"((a
However, in our specific case, because h(z,, 0) is locally linear within each fixed partition (i.e.. h, 0) is either n. 0., 0,,). its second derivative 7 0) is zero. This causes the second term of the Hessian to vanish, meaning the expression (Vh)) often used the Gauss- Newton approximation, becomes the exact Hessian.
Therefore, the exact Hessian matrix is:
V2V(0)- = Σ (es 2,07
where 0 is (d + 1) (d 1) zero matrix.
14

---

Published as a conference paper at ICLR 2026
Newton Update and OLS Equivalence. ((k+1) ((k) μ (VV Let ((h) = for Ot, and 0r, separately.
The Newton update at iteration is generally given by [0, (kk)TTT. We examine the updates
For Or,, the update is:
( 1 1
Σ
Σ (E),
3z(0; (
(as (E
1 +
Σ
Σ 83U1
Σ
a 2 0
(as (ES
(ES
(m..8s JES
!+ +
Σ
(m):Ns)ES!!))
(15)5)))
Let 0 OLS,
(E (E }هور)
Σ (e 2,ys)
be the OLS solution for the data
subset (()) Then the Newton update be written as:
e (++1) 0!
m(el!) OLS,1 - -
similar derivation holds for 02:
0 [++1)_ (k) 1 11 (K)
01S..2 )
where 0 the OLS solution for data subset SAA).
In our algorithm, the step size p tunable parameter. When unit step size employed (i.e., 1), the update formulas simplify to:
(++1) / 0
(++1)
(k)
02
OLS.2
This demonstrates that when a unit step size (f1 = 1) is employed, the alternating procedure of model fitting (performing OLS the current partitions) and partitioning (reassigning data points based on the new model parameters), as presented in Algorithm is precisely equivalent to executing a unit- step Newton method to minimize the original nonlinear objective function V(0). This choice of is common for this type of problem due to its direct connection to the optimal OLS solutions within fixed partitions.
B NODE-LEVEL CONVERGENCE OF THE LINE-SEARCH NEWTON UPDATE
This appendix provides a node-level convergence guarantee for the line-search Newton update de- scribed in Section 3.2.2] The analysis is local to single internal node and does not rely global tree construction. Throughout, we adopt the notation introduced in Section 3.2.1
Setup. At an internal node D,, the local objective is
V(0)
Σ (us - h(zz, 0))
= 0 E 2(d++))
(4)
2, E.
where
h 0) max vaxx(#}00 0,, #7 0r,)
ã [m] 1|
15

---

Published as a conference paper at ICLR 2026
Given 0 define the partition
(0) = (a, E Dr : 0, 2 0² 0z2}: ( = D, \ S (0).
(5)
(6)
On fixed partition (S,, S2), the hinge becomes inactive and
V(0)= -11:11 X X,0 12 + =llb2 X20 112
(7)
with gradient and Hessian
X (x yn)
X H(0)
0
(V =
(8)
(X202 y2)
0
X3 X2
where
X; [
ESES, E ER (()()(()
y: = [v0
E TR
i = 1,2
that is, X1 (resp. X2) is the matrix whose rows are the augmented feature vectors aj for all æj E Si (resp. 2. S2), and yı (resp- y2) is the vector of the corresponding responses. Here [S,] denotes the number of elements in S..
The OLS solution for this partition is
(xxxx)- yı (X X2) -22
ools
As shown in Section 3.2.2
p oors 0 -
(9)
the exact Newton/Gauss-Neet Newton direction on this partition. The following assumption requisite for the node-level convergence.
Assumption (Regularity). For every iterate (kk)
(i) (Nondegenerate hinge) For all j. ai (/ + az o/ Thus no sample lies exactly on the separating hyperplane.
(ii) (Uniform strong convexity each fixed partition) Each block matrix X? X, encountered along the iterates satisfies
ml 3 X LI,
i= 1.2,
for global constants 0 < m < < xc.
This condition is standard and matches the leaf-level eigenvalue assumption used our approxima- tion theory.
The following theorem demonstrates that the line-search Newton update can reach local opti- mum of (2).
Theorem 2 (Node-level convergence of the line-search Newton update). Under the node-level setup and Assumption] [ consider the Newton iteration at node D)
g ( = ( + (k) (k))
where (k)) = (k)) is the Newton/Gauss-Ner -ewsstt direction given by Equation 9). and ((()) is chosen by backtracking line search with parameter (0,1) and trial steps {Pos3 J120- Then the following hold:
(a) (Per-iterate descent and line-search termination) For any iterate ((k) with ((k) + OoLs, the Newton direction p(k) is strict descent direction and the backtracking line search produces step size 4 > 0 such that
V(O(k+1)) (()(4) + (k) (k9) ((O(t)).
(10)
16

---

Published as a conference paper at ICLR 2026
(b) (Monotone decrease and convergence of the objective) The sequence (V((())) is strictly decreasing whenever p * O and converges a finite limit Vo--
(c) (Convergence under fixed partition) If there exists K such that the partition is constant for all k 2 K, then {0(k}} converges to the unique OLS minimizer for that fixed partition.
(S,
Proof. We proceed in several steps.
Step 1: Newton direction a strict descent direction. For any iterate 0 (we temporarily drop the superscript k), the Newton direction satisfies
p(0) (() (VV(()).
so
VV OV p()) =V( (ii implies (0) 3 LI and (())) 0, and hence 11(0)-1 = H. Therefore,
Assumption
N 7 (V)
which yields
VV(0)) - -, IV
If 0 * ools, then (V(O) * 0, so the right-hand side is strictly negative. Thus p(0) is a strict descent direction.
Step Local decrease along p(0).
Fix and its Newton direction p p(0). Write
T
P1
p(0)
0
P2
conformably with the block structure in Equation(
Consider the univariate function
11 2 0. We show that there exists j > 0 such that V(0 pp) V For each define
(M) = v(0 -up),
() for all 0 < A.S..
a,(0) a 0,,
( 0 0,g
Assumption I i) states that (0) # (0) (0) for all j. ie.. the active branch in the max is unique at O.
Without loss of generality, suppose a, (0) > by (0); the other case analogous.
Along the ray 0 + we have
( + su)) (y 0 μu) (0) b, + - -xf
which is an affine function of and is nonzero at μ1 O. Therefore, for each j there exists J. > 0 such that the sign of a, ( - pp) 6, + μp) does not change for all 0 S μ S Py. In particular, the active branch in h( -) remains the same for all small μs.
Let J == min, [ > Then for all S μ I á the index of the active branch h(z,,)) fixed for every so V + /P)) coincides with the smooth quadratic representation along this segment. Hence (44) is differentiable on [0. [i] with
(0) (V(0)"p.
By Step 1, x'(0) = (V < O. By continuity of v/ at O, there exists < Ā S á such that (()) (0) = (()) for all 0 < p S f.
i.e.,
V + up) < (0) for all 0 < < p.
17

---

Published as a conference paper at ICLR 2026
Step 3: Backtracking line search terminates with decreasing step. Let the backtracking line search start from some initial step Ho > and generate the sequence μ4 = nu for t = 0, 1,
with 3 E (0, 1). Since 24 + 0, there exists t* such that pe S j.. For this t* we have 0 < 14.- S and hence, by Step 2,
V(O p4- < V(0).
The backtracking procedure selects the first such t`, so it terminates with a step size y( pe. satisfying (10). This proves part (a).
Step 4: Monotone decrease and convergence of V. Applying part (a) at each iteration k yields
((0(++1)) = ((0 + n (k)) (0)) < ((O())
whenever p # O. Since ( 2 0 for all 0, the sequence { [V (0())) is strictly decreasing and
bounded below, and thus converges to some finite limit Vo 2 0. This establishes part (b).
Step 5: Convergence under fixed partition. Finally, suppose there exists K such that the partition (8, (0(8),, (, is constant for all k 2 K. Then for all k > K, the objective V coincides with the smooth, strongly convex quadratic version 7 with Hessian
Xfxx
H
ml 3 H()) LI,
XT
by Assumption Ti). In this regime, the Newton direction simplifies to p(0) = fols - 0,
and the update becomes
(R+1) - Oors = (1 - (0) -010))
Under the standard backtracking line-search rule on strongly convex quadratic, the accepted step sizes (k) are uniformly bounded away from zero and from above, so the factor [ is strictly less than and uniformly bounded away from 1. Hence ((k) - OOLS as k - co. Equivalently, the iterates converge to the unique OLS minimizer for the fixed partition, which proves part (c) and completes the proof.
C DETAILS ON REGULARIZATION AND FALLBACK STRATEGIES
Incorporating Ridge Regression (1.2 Regularization). To enhance model robustness and effec- tively handle situations like multicollinearity or high-dimensional data, we introduce optional ridge regression (L2 regularization) in all OLS fitting steps within this algorithm. ridge regression modi- fies the OLS objective by adding a penalty on the L2-norm of the model coefficients, i.c., minimiz- ing:
1 No
min 2
Σ U
0)
0)
1=0
where a 0 is the regularization strength parameter. Notably, the bias term 0. (corresponding to the constant 1 in the augmented feature vector) is typically not regularized. Given a design matrix X (already augmented with column of ones for the bias) and target vector y, the ridge regression solution with regularization parameter 2 0 is given by:
- (XT aTo)
aTo)
where To is an identity matrix with its last diagonal element (corresponding to the bias term) set to zero, ensuring the bias is not regularized. This regularization mechanism effectively stabilizes parameter estimation, especially when the training data matrix X X might be singuai,,,,a,, which is particularly important in nigh-dimensional feature spaces or with limited sample sizes. The regu- larization strength is tuned hyperparameter during model training.
18

---

Published as a conference paper at ICLR 2026
Fallback Strategy for Non-convergent Nodes. If the iterative optimization does not converge within Tmax iterations, we fall back to a simple median split: a feature dimension & is chosen at random, and the data split at the median my of this dimension. This guarantees progress in tree growth and often decomposes a hard global problem into smaller subproblems that converge quickly under our Newton updates.
D PARAMETER INITIALIZATION STRATEGY AND DETAILED ALGORITHMS
This section provides the detailed pseudocode for the optimal node-splitting and tree-buidding pro- cedures discussed in Section
D.1 PARAMETER INITIALIZATION STRATEGY
The first step of Algorithms and 2 requires the initialization of the parameter vectors 01 and 02- A reasonable and stable initialization is important for the convergence of the subsequent alternating optimization. We employ a simple data-driven heuristic combined with small fallback procedure.
The initialization procedure is as follows:
1. Heuristic initial partition. At the current node, identify the feature dimension with the largest range in the local data. We then use the median value of this feature as a pivot to split the samples into two subsets. This aims create initial partition along the direction of greatest variation, which typically yields more balanced and meaningful starting point than a fully random split.
2. Partition-based initial (ridge) regression. Given these two subsets, we independently solve (ridge-regularized) least-squares problem on each subset to obtain the initial pa- rameter vectors 0 and 02. In our implementation we require that each subset contains least two samples; otherwise we treat the split as too small for reliable local regression.
3. Fallback for degenerate or very small partitions.
edge cases, such as when the node contains very few samples, the features are nearly constant, one of the subsets after the median split is too small, we fall back to a simple global initialization. Specifically, we first compute a global ridge regression solution Ggiobal using all samples at the node (or a constant predictor based on the mean response if the design matrix is degenerate). We then construct two initial parameter vectors and 9₂ by adding small independent random perturbations to fgloball which ensures non-trivial starting point even under poor initial splits.
4. Ensuring parameter diversity. Finally, we check whether the resulting and 02 are nearly identical. If so, we add another small perturbation (and, in a rare corner case, a tiny offset to the first coefficient) break the symmetry, so that the subsequent alternating optimization can proceed effectively.
This initialization strategy provides a data-dependent yet lightweight starting point for the node- wise optimization, and makes the overall splitting procedure robust to degenerate partitions and small-sample regimes.
D.2 DETAILED ALGORIHTM
19

---

Published as a conference paper at ICLR 2026
Algorithm Find Optimal Split (Min Variant)
Require: Dataset = {(r), yi)) (;))==1- Max iterations maxx Step size μ4, Tolerance Ensure: Optimal parameters Or 02
1: Initialize 0 o and partitions St, S2 using the strategy in Sec. D.1
2: for ( 1, ,,.., max do
3:
Model Fitring Step
4: Compute ((k)
OLS,1 and 0 (#) (with optional ridge regularization)
5:
g (k) + 1 (0 (8)
O1S,1
0 (R)
> Newton update with step size
6:
0g
(k) + u O1S,2
(k)
0 (k)
Newton update with step size
7:
11 Partitioning Step (min: assign to the smaller branch)
8:
S1.new +- 0, S2.new
9:
for all (æj, }3) S do
10:
if a 0, S 02 then
11:
Si.e < St U ((1j,,;)}}
else
13:
S2 S2 new U {a))( }ز}}
14:
end
15:
end for
16:
if 1 (+)) 1(4) +10@, (10g(+1)
0 || < then
> Check parameter changes Convergence
break
18:
end
S₁ S S2 Snnw
20:
end for
return 0,, 02
Algorithm 2 Find Optimal Split (Max Variant)
Require: Dataset S { j Mi)) ).V 1=1- Max iterations Tmans Step size
Ensure: Optimal parameters 0,,
Initialize of0) .0 and partitions Si, S2 using the strategy in Sec. D.I
2: for = 1,... Tmaz do
3:
Model Fitring Step
4: Compute (kk) and OLs (with optional ridge regularization)
5:
((++1)
(k) + 1 (k)
OLs,
0 (k)
> Newton update with step size
6:
0
((++1) + (k)) + p ((
0
> Newton update with step size
7:
11 Partitioning Step (max: assign to the larger branch)
8:
Si new +- 0, S2
9:
for all (0,, Uy) E do
10:
x. 0 2 0 then
11:
Sl.new f Si,new U ( w;)}
else
13:
S2,now + Seneew U ( }ز}}
end if
15:
end for
if (1
0(0011 +|| (++1) 0b5h11
then
Check parameter changes & Convergence
17:
break
18:
end
19:
S SI.new
Sy 5 S2.
20:
end for
21:
return 01,00
20

---

Published as a conference paper at ICLR 2026
Algorithm 3 Find Optimal Split (Min-or-Max by RMSE)
Require: Dataset = {(x), [ū)] Taxx, Step size μs, Tolerance
Ensure: Best parameters (07, 02), model type E (min, max)
1:
(0max 0 <- run Algorithm 2 (max variant) on S with (T M4 E)
2:
(0 0z run Algorithm (min variant) on S with (T H1, E)
3: Compute RMSEmax +
Ej= (us max(2; 0T 23 02x0
4: Compute RMSE +
N 2j=1 (vs 0 0n xz
1
enin)))) 2
5: if RMSSE &RMSEmax then
6: return 0pin Opain
7: else
8:
return 0pax 0
9:
end if
21

---

Published as a conference paper at ICLR 2026
Algorithm 4 Build Oblique Regression Tree
Require: Dataset S, current depth d, hyperparameters (max depth Dmaz, min samples Nmins RMSE threshold TRMSE, step size μi)
Ensure: Root node of (sub)tree
1: If Check stopping criteria for creating leaf
2: Fit a single linear model for the current node: Dleaf arg ming E(nwess x" 20 Solve
via OLS (with optional ridge regularization)
3: Calculate RMSE: RMSE = / [s] C(r.)ess n Oleaf)
4: if' Z D Mma or [S] < min or RMSE < TRMSE then
Check depth, samples, or RMSE
5: Create leaf node.
6: Store Bleaf in the node and return the node.
7:
end if
8:
9: Proceed with splitting the node
10: 07 02 -FindOptimalsple
p)
11: SL + (xx,)) E x= 1 2 x 0;
12: Sn S
13:
|SR| Nmi then 15: Create a leaf node.
if |r <min
Ineffective split, create leaf
Store the pre-computed flcaf from line 3 and return the node.
17: end if
19: Create an internal node, store split rule (0), 0%)
N.left_child <- Bui IdTreeeSS + 1, Dmnax Nmin TRMSE, 11)
21: N riihh <- Buil BuildTree d + 1, Daaxy Nmin, TRMSE, 11)
22: return the internal node
22

---

Published as a conference paper at ICLR 2026
E
TIME COMPLEXITY ANALYSIS
E.1 ALGORITHM 1: FIND OPTIMAL SPLIT
Overview: Algorithm is iterative algorithm designed to find the optimal splitting parameters 0, and 02. It alternates between model fitting and partition updating, running for most Tmax iterations.
Per Iteration:
Model Fitting: Solves OLS for two partitions S₁ and Sy. These OLS solutions can incorporate ridge regression. The complexity of each OLS problem is O(N₂ d²), where N₂ is the number of samples in the partition and d is the feature dimension. Since N = Ni + N2, the total complexity is O(N .
Partition Updating: For each the samples, computes two linear functions x 0, and x², 02 (each (dd) and reassigns partitions, yielding complexity of O(N d).
-Total Per Iteration: O(N d²) + O(N d) = O(N d²).
Total Iterations: At most maxx user-defined maximum.
Total Time Complexity: O N .dd.)
Note: In practice, convergence may before Timx, but the worst-case complexity sumes the full number of iterations.
E.2 ALGORITHM 2: BUILD OBLIQUE REGRESSION TREE
Overview: Algorithm 2 recursively constructs oblique regression tree with maximum depth Dmx and a minimum of Nmin samples per leaf. Each internal node invokes Algorithm 1, while leaf nodes fit final linear model.
Internal Nodes:
Splitting: Each internal node calls Algorithm 1, with complexity O - dd),
where Ngode is the number of samples at the node.
-Wrsst-Cse Assumption: The tree is balanced, with each split dividing the data ap- proximately in half.
Leevel Analysis: At level k, there are 2² nodes, each with approximately N/2 sam- ples. The total complexity per level is:
2 O (N/2*) d)) O(Tmax N -df).
- Total Levels: With Dmax levels, the complexity is O(Dmax Tmax N d2).
Leaf Nodes:
Fitting: Each leaf fits OLS model, which also incorporate ridge regression,
with complexity (NNaat dd), where Niat is the number of samples in the leaf.
Total Leaves: Up to O22Pum) leaves, but the total sample count across all leaves is N. Thus, the total complexity is O(N d.).
Total Time Complexity: maDmax I mmax N -dd) O(N dd) O(D Thax df)..
Note: In practical settings, Tima and D are often fixed constants, simplifying the com- plexity to O(N -)). In theory, Duns scales with N, the complexity increases accordingly.
F PROOF OF UNIVERSAL APPROXIMATION AND APPROXIMATION RATE
Theorem 1. Let be the class of piecewise linear functions represented by finite oblique regression frees with linear models at the leaves. Let 9 K. + R be twice continuously differentiable function (g E (²)()) on a compact set C Rd. Furthermore, assume that for any constructed partition of IC into regions (Ra) with diameter < 8, each region R2 contains N; training data points ()xg(( Let X; be the N₂ (d (d+1) 1) design matrix whose j-th row is az. We further assume
that X, satisfies Amin (X. X2) 2 for some constant e > 0. Also, supmex |x S Dx. for some bounded constant Drc Then, there exists a constant C such that for any 6 0. we can construct
23

---

Published as a conference paper at ICLR 2026
function f E F by partitioning the domain K. into regions {R²) with max, diam(R,) 1 6, satisfying the uniform error bound:
sup (f(1)( I C82
#EK'
This approximation rate directly implies the universal approximation property, i..,, for any 0,
there exists a function such that supœek: [(xx) - g(x)) <e.
Proof. This proof relies constructing first-order Taylor approximation within each partition and then quantifying the impact of OLS estimation. The condition that g is twice continuously differentiable (g (KC) the compact set K. is crucial throughout this proof.
We adopt the following notation for clarity and consistency: æ (or Zÿ) denotes a d-dimensional original feature vector. X (or x)) denotes the d + 1-dimensional augmented feature vector [x", 1 (or [x _]" respectively), which includes a bias term.
1. Domain Partitioning. As is standard, the compact set K. (representing the domain of original feature vectors) can be covered by a finite collection of disjoint convex polytopes { such that U R₂ = KC and the diameter of each polytope diam (R,) < 5. Since oblique decision trees perform recursive partitioning with hyperplanes, such partition can always be realized by a tree structure where the leaf nodes correspond to these regions.
Local Taylor Linear Approximation. This step introduces theoretical linear approximation. For each region R,, we select center point c, E R, and define fTaylor. the first-order Taylor expansion of around c;:
fTaylor.
g(c,) + Vg(c,)) g(c,)
c,) for all a E R.
This fraylor, (x) is linear function of It serves an analytical benchmark for how well g(x) can be approximated locally by a linear function.
3. Bounding the Taylor Approximation Error (Approximation Error). By Taylor's theorem with the Lagrange remainder, for any æ R2, there exists a point & on the line segment between æ and c, such that:
1 g(x) = g(c)) + Vglc:) - c;) 2/ -0-,;;)" -x- v.)' - c;)
where VEg(E) is the Hessian matrix of g at &. The Taylor approximation error is therefore the magnitude of the remainder term:
1 (x -., c.) c.)
ffaytor, g(x)و
2
Since E (1C), its Hessian bounded on the compact set K. Let M = supxek: [[]](2)] be the supremum of the spectral norm of the Hessian. We can then bound the error:
1 g(x)] S M|| c:l12
[Taylon;
2
As both and c, are in R2, the distance between them is bounded by the diameter, || - -cil2 S 6. This yields:
ITayloor (2x) -((=)) M8
Letting CA = M/2, this term bounded by CA52.
4. Quantification of OLS Estimation Error and its Impact on Convergence Rate. The preceding analysis assumes we can construct ftaylor.s exactly. In practice, however, we only have access to values of finite of data points. To construct function f E F (as described in the theorem), we use actual data to fit local linear models. Within each region R;, we employ OLS to estimate a N; linear model from a set of N; training data points {(x, 3;,,,1)) Uy)} j=1= where y, = g(xx)). Let (x) denote this OL Sesttmaated linear function, which takes the augmented input
24

---

Published as a conference paper at ICLR 2026
To quantify the impact of OLS estimation, decompose the total error for an original input into two parts:
(g(x) (x)] S 1g(x) - ftaylor, (x)) + IF (x) (x)|
The first term is the Taylor approximation error, which we have bounded by CA82. We now need to bound the second term, representing the difference between the OLS-estimated function and theoretical Taylor function. Letting fTaylor, (x) w;"x x, where w is the corresponding theoretical coefficient vector (i.e., w [Vg(c,)' ((,)) (g(c.)" c..l). The OLS-estimate linear function is (x) = w X, where w; is its coefficient vector.
We know that g(m;) = /Taykor, (mz) + R(xy), where [R(x,)] & CA82 is the Taylor remainder.
Utilizing the linear model representation, this can be written as yj = g( w, &3 + R(æ,) for each data point (æ M)}) in R4.
In matrix-vector notation, for all N training data points in R, we have:
y; X,w, + R;
where y; = -Vl..... JN.]"" is the vector of observed values, X, is the N, x (d+ 1) design matrix with rows X5. and R; -R., R(x ) is the vector of Taylor remainders.
The OLS estimator w; minimizes the empirical loss - - w(xx)² The solution to this minimization is given by the normal equations:
Because X? X; invertible (guaranteed by the data assumption in the theorem for sufficiently diverse data), we can solve for W;:
W = (x7x,
Substituting the expression for y; from above into the OLS solution:
w, = xXx (X, + RR)
= X, + (X7x))
Since = I (the identity matrix), the equation simplifies to:
W; w + (X? -x) R,
Rearranging this, we obtain the desired expression for the difference:
w ( -x)
Under the data assumption stated in the theorem, we can bound the uniform error of the OLS esti-
mate:
sup [Taylor - /x)ss;(x)I- = sup æc Rg
1x (w: w()l
=ER;
sup
||x la || w 1/2
=ER
Dx x ll2
< (x x,) -1x X || R
From the theorem's 's assumption, Amin (XF x,) cNV; for some constant c > This implies that the smallest singular value of X,, miin (X,) = / (X. 2 VeN, The spectral norm of the pseudoinverse X = (XF - is given by ||X; 1|2 (X,) Therefore, we have:
xIxxXx)) VeN.
Substituting this into the bound:
1 sup ITaylon - [x)s, S Dx VeN, E,
||Ra|/2
25

---

Published as a conference paper at ICLR 2026
Since |Rill2 I sN, [R(x))] S VN,CA6², obtain:
DxCAB Vc
VN.CABO (x/xylor,((x)) - [x)5, & DE VeN, mER;
sup
This term is of order O(5²), with a constant factor Co = DxCA/ve This indicates that as long as the data density and diversity are sufficient (ensuring N; large enough and is well above zero), OLS can converge to the theoretically optimal local linear approximation, and the additional error it introduces does not degrade the (b²²) convergence rate.
Combining Total Error: Combining both emror components, for each region R, and any original feature vector æ E R2 we have:
(g(x) - (x)s, & CAd² Coo (CA+ C
Let CA + Co = C1(1+ DK/ We define the constructed piecewise linear function f E F such that for R₁, fx) (x).. Thus, this bound holds for the entire domain K::
sup (9(x) f(x) s C82
=eK
This demonstrates that even with OLS for local linear fitting, provided sufficient data N; in each region to stabilize the OLS estimates, the O(6²) convergence rate is maintained. The universal approximation property also follows.
Corollary 1 (Curse of Dimensionalityy. To achieve partition with a maximum region diameter of 8, the depth of a well-balanced tree, Duax. scales with the dimension d. To halve a region's diameter, one must ideally cut it along all d orthogonal directions, which requires approximately d splits, or levels in the tree. This leads to the approximate relationship:
8(Dmax) diam 2 Damx/d
This implies that achieving a fixed error level (i.e., a fixed 8) requires the tree depth Dmax to grow linearly with dimension d (Duax d dloo(1/))), manifestation of the curse of dimensionality inherent to space- spaueppprtitioning methods.
G ABLATION STUDY ON STEP SIZE: STABILITY, EFFICIENCY, AND FALLBACK
The step size j (damping factor) is key hyperparameter in our node-splitting algorithm. It controls how aggressively each Newton update moves toward the local OL.S solution and therefore directly affects stability, efficiency, and final performance. To examine these effects empirically, perform an ablation over j on four datasets with different characteristics: the oscillatory sinc function, the smoother twist seteeddsiooo datasets. For the two real-world datasets, we additionally evaluate an automatic line-search step-size strategy (denoted by auto). Results are reported in Tables 5
function, and the real-world Kinemation ics and D -o-iierons
Damping and the fallback mechanism. On the synthetic datasets, smaller step sizes lead to very few fallbacks, while more aggressive steps can trigger the fallback mechanism more often. For sinc, the fallback rate is essentially negligible at μ 0.01 (0.39%), modest at μ = 0.05 (4.74%), and rises to 32.86 %, 60.94%, %, and 29.55% at μ 0.10. 0.50, and 1.00, respectively. A similar but milder trend appears on twi sted_si gmo i the fallback rate increases from 0.67 % at lr 0.01 to 34.67% % at μ 0.50, before dropping to 19.33% at μ. 1.00. On the real-world datasets, fallback rates are moderate and relatively stable across μ for Kinematics (between 6.87 % and 10.82%. with 9.51 % for auto), and identically auto. Overall, these results indicate that damping does help keep the fallback mechanism rare on more challenging synthetic objectives, while some benign regression problems remain stable for relatively aggressive step sizes.
D-Aileron
for all settings, including
26

---

Published as a conference paper at ICLR 2026
2. Step size, iterations, and training time. Larger step sizes do not automatically translate into faster convergence wall-clock time. On the Kinematics datasets, small or moderately damped step sizes typically lead to stable and fairly regular Newton dynamics, the node-wise optimisation terminates after only a few iterations on most splits. In contrast, aggressive step sizes can induce oscillatory irregular behaviour ically, and the parameters and partitions can keep moving without settling down quickly. Such os- cillatory runs require many more iterations before the stopping criterion is met, which substantially increases both the average iteration count and the total training time. As result, the dependence of fit time on j is non-monn m-nootone: large steps can be very fast when they converge cleanly, but they may be markedly less efficient once the cost of these unstable cases is taken into account.
nodes: the objective value may fail to decrease monoton-
3. Effect on predictive accuracy. The impact of on RMSE is dataset dependent but generally modest, with few notable cases. On sinc, the smallest step size μ 0.01 achieves the low- est error (0.02796), while = 0.10 is clearly suboptimal (0.05310); larger steps (p = 0.50 and 1.00) recover competitive errors but the cost of higher fallback rates and iteration counts. On twisted _udd__ssi iem.ddsssimioiommmmmmmm all choices of yield similar performance, with slight advantage around 00258-0.0283) On Kinemat ics, cs, performance improves as j
I = 0.50 and 1.00 (RMSE increases up to 0.50 (RMSE decreasing from 0.13055 at 0.01 to 0.10266 lr 0.50), and the aut o strategy yields the lowest error (0.10185) while using far fewer iterations than = 0.50. On D-Ailerone all step-size choices, including auto, achieve essentially identical RMSE (about 1.68 x 10 10-+), indicating that accuracy is largely insensitive to the damping factor on this dataset.
Overall takeaway. Taken together, these experiments suggest that the damping factor μ mainly controls trade-off between stability, efficiency, and the need for fallbacks, with only moderate in- fluence on accuracy in most settings. Small or moderate fixed step sizes tend to keep fallbacks rare and iterations moderate, especially synthetic problems such sinc and twisteds gmo Extremely aggressive fixed steps can lead to increased fallback usage and substantially more it- erations, particularly more structured high-dimensional data such K nematics. The automatic line-search step size (auto), which corresponds to the theoretically analysed variant in Appendix B consistently provides robust choice on the real-world datasets: it attains strong ac- curacy, moderate iteration counts, and stable fallback behaviour without requiring manual tuning of p.
important to note that Figures [nd focus on the dynamics of single internal node under different step sizes, with fixed initialization. In contrast, Tables & report averages over all nodes in full tree, including the effect of the fallback mechanism and stochastic variation across runs. As a result, local node-level stability and global tree-level efficiency are related but not identical quantities, and their trends need not coincide pointwise for every value of μ.
Table 5: Ablation study the sine dataset. Metrics averaged over 10 runs.
Step size (1)
RMSE
Leaves
Avg. Iters
Fit Time (s)
Avg. Fallbacks
Avg. Spliis
Fallback Rate (%)
0.01
0.02796
26.7
2.56
0.166
0.1
25.7
0.39%
0.05
0.02885
24.2
4.46
0.105
1.1
23.2
4.74%
0.10
0,05310
22.3
5.97
0.107
7.0
32.86%
0.50
0.03069
26.6
4.30
0.112
15.6
25.6
60.94%
1.00
0.02926
23.0
6.65
0.141
6.5
22.0
29.55%
Table Ablation study on the twisted.sigmol dataset. Metries are averaged over 10 runs.
Step size (1)
RMSE +
Leaves
Iters
Fit Time
Avg. Fallbacks
Avg. Splits
Fallback Rate (%)
0.01
0.026533
16.0
3.17
0.045
0.1
15.0
0.67%
0.05
0.03011
16.0
4.44
0.083
0.6
15.0
4.00%
0.10
0.028311
15.6
5.28
0.063
3.0
14.6
20.55%
0.50
0.02578
16.0
6.13
0.107
5.2
15.0
34.67%
1.00
0.02577
16.0
7.54
0.121
2.9
15.0
19.33%
27

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study on the Kinemat ics dataset. Metrics are averaged over five runs.
Step size (p)
RMSE
Leaves
Avg. Iters
Fit Time (8)
Avg. Fallbacks
Avg. Splits
Fallback Rate (%)
0.01
0.13055
57.0
10.89
0.49
4.6
56.0
8.21%
0.05
0.11367
59.2
19.53
0.62
4.0
58.2
6.87%
0.10
0.10631
57.2
26.65
0.72
6.0
56.2
10.68%
0.50
0.10266
56.6
59.77
1,53
5.8
57.2
10.14%
1.00
0,10702
48.6
83.43
1.88
5,8
53.6
10.82%
auto
0.10185
48.6
15.76
5.4
9.51%
Table 8:
Ablation study on the D-Ailerons dataset. Metrics are averaged over five runs. The
reported RMSE values scaled by x10-1
Step size (1)
RMSE
Leaves
Avg. Iters
Fit Time (8)
Avg. Fallbacks
Avg. Splits
Fallback Rate (%)
0.01
1.68
8
1.57
0.105
0
0%%
0.05
1.68
8
32.43
0.178
0
0%
0.10
1.68
30.09
0.179
0
0%
0.50
57.69
0.345
0%
1.00
92.60
0%
aulo
6.71
0.183
0%
H SYNTHETIC DATA DESCRIPTION
This appendix provides the detailed mathematical definitions of the 3D synthetic functions used in Section [.2] the rationale for the selection of these synthetic functions and noise levels, and additional visualizations of our method's 2D and 3D function approximation capabilities (Figures 3
and 4
For all 3D functions, the inputs , range from [-3,3].
3D Oscillatory Surface Functions:
f1 = (f.5 2x x3 + 3sin (4x1) cos (2x²) + 0.1 -
f2(31, = sin(3r1) + cos (2xz) + 0.5 sin(5x11 cos(4.22)
f2(311)22) 0.5+7
-i
(sin(r)e_ where V + 22z 10-6
f1(01,12) exp
(1)-1))+(02111)
exp
(1i+1)*+ (os++1))))
+ 0.5
Rationale for Selection of Synthetic Functions: These functions exhibit diverse nonlinear charac- teristics, including high-frequeet oscillations (sinc, 3D oscillatory functions), sharp changes, and inflection points (twis iiisteed s imo Successfully approximating such wide range of nonlin- earities, as demonstrated by our empirical results, provides strong empirical support for the universal approximation capability of our piecewise linear HRT model, which theoretically proven Theo- rem [ The s sinc function, with its numerous local extrema and oscillations, presents a particularly challenging landscape for optimization algorithms. It helps to expose potential instabilities (e.g.. non-monotonic convergence, limit cycles) in the splitting mechanism, thereby highlighting the ne- cessity and effectiveness of damped Newton updates for robust convergence, as analyzed in Section 5.1 Conversely, the twi sted s gmo i d function, while non-trivial, is smoother and less prone to pathological behaviors, allowing us demonstrate the efficiency of our Newton updates a more stable environment.
Rationale for Noise Levels: Introducing independent and identically distributed zero-mean Gaus- sian noise (with standard deviations of 0.025 for 2D tasks and 0.05 for 3D tasks) simulates this inher- ent uncertainty. This ensures that our model's performance is evaluated not just on ideal, noiseless functions, but data that more closely mimic practical applications. The selected noise levels are not so high that the primary challenge for the models remains the accurate approximation of the un- derlying functional relationship, rather than solely distinguishing signal from overwhelming noise.
28

---

Published as a conference paper at ICLR 2026
This allows to clearly observe and compare the models' capacities to fit the true, underlying non- linear structures. noise were excessively high, it would obscure the approximation performance, making it difficult to assess the inherent expressive power of the models.
Training
2
Tast Data method
Training
CART
-0.5
Test Data
21
XGBoost
CART
-1.0
0.05
0.00
0.00
-0.05
-0.05
1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
-3
-2
-1
(a)
+ 0.025c
(b) y
0.8w 0.025
Sma
Figure 3: Top: Approximation performance of various methods on sinc functions. Bottom: Residuals r = Spred Vtree representing the difference between predicted and true values for each method.
ne and twi stsedss gmo d
29

---

Published as a conference paper at ICLR 2026
Traaning Data
Tramnnng
Test Data
Test Data
Our Mechod
Our Meshod
40
2
y
20
-1
-2
3
2
2
-2
$
33
-2
0
0
(a) Our Method approximating (((11 + 0.05c.
(b) Our Method approximating y 0.05c.
f2(11, x2)
+
Traiinng Data
Traming
Tost Data
Tust Data
Our Method
Our Method
2
y
0.5
-10
- 82
7
82
1
(c) Our Method approximating fa(x1, + (d) Our Method approximating y fa(21, x22) + 0.05e.
0.05e.
Figure 4: 3D function approximation. We visualize the learned piecewise linear surface of our method along with training/testing points. Contours the floor help read depth. For clarity, only fitted surface of Our Method shown; detailed quantitative comparisons with baselines are provided in Table
30

---

Published as a conference paper at ICLR 2026
I HYPERPARAMETER
This appendix details the hyperparameter search grids used for tuning various models in 2D and 3D function approximation experiments. For each model, a grid search approach was employed to find the optimal hyperparameters that minimize the validation error. The specific ranges for each hyperparameter are listed in Table 9 Hyperparameters are tuned via five-fold cross-validation on the training set, with final performance reported on the testing set.
Table 9: Hyperparameter Search Grids for 3D Function Approximation Models
Model Hyperparameter
Search Grid
max_ pax_depth
(1, 8, 10, 12] [0.01, 0.03, 0.05]
ri ridge.alpin
HRT
step-size
(0.01, 0.5, 'uuto') (0.001,
threshold
max depth n samples split
[3,5, 113
CART
[2,5, 10)
min samples Leaf
(1,2,4)
n. nsess mators learning rate max max_depti depth
(50, 100,
(0.01, 0.2.
XGB
(2,3,5)
subsample
[0.7,
For HRT, we tuned the threshold parameter, which controls the splitting criterion, and max depth, which limits the maximum depth of the constructed tree structure. We also var- ied the ridge al pha, which adjusts the regularization strength ridge regression to mitigate overfitting, and the step_si ze, which determines the granularity of updates during optimization and affects convergence speed and stability. For CART, we explored variations max-depth, the minimum number of samples required to split an internal node (min.samples.spla) and the minimum number of samples required to be at leaf node moin.samples.lest For XGBoost, optimized the number of boosting rounds (n.estimators) the step size shrinkage (Learning .rate,, the maximum depth of tree (max (max_deptha) depth), and the subsam- ple ratio of the training instance (subsample). The objective function used for XGBoost was reg: squaredeueeeeeeeeee requrredrrrr' with rmse' as the evaluation metric.
Tables and 11 summarize the chosen hyperparameters via 5-fold cross- vross-validation, covering both our proposed method and all reproduced baseline methods. The tuned hyperparameters for DTSem- Net and DGT are listed Table 12
Table 10: Optimal Hyperparameters Across Various Functions for Different Models
Hyperparameters
CART
("mexulepch's
neingeaee leaf'; "mingrengless
sine
("leaminggratte 0.1, "moxueethh' 3, fruest instoraf 200, "xubxample's 0.7)
HRT
("max.depth" 6, "iiogeallpant 0.001, "atep_size" 0.01, "ehresheidr 0.03)
CART
('max dapcht 9, "min samples leaf': 2, 'min samples splis'; 5)
twisted sigmoid
XGB
('cearning.rate'' 0.1, "max max.doopth' 3, "nicstinsioes"' 100, "suacampie"; 0.7}
('maxxdecch' 4, "eldgeallpaass 0.001, rstopiaize":
"threshold': 0.01)
CART
("mexudepehht 11, metnsaanleessleff 1, "mingpempleeeppp 2) ("eearning.ratw/
(1(2))022)
XGB
"maxudepthl'
200, 'subsample's 0.7]
HRT
("max_depth's : 12, "ehreshole"s 0.01, 'ridge.alpha": 9, 'step.size": 1}
('max daptht "nin samples learr: 1, "min samples splitr: 2]
fa(m),0)))
XGB
('haariingratte 0.2, "maxdspthh" ; 5, "ness inslore's oes" 200, "subxample's 0.7]
HKT
('maxxlephh' 12, "Chreaboldr; 0.01, 'ridge.alpha" D, "stepaaiz" 1)
CART
("rexulepphht 11, "nax.dopta': 5, 'nucstinstors"' 200, "sussample': 0.73
4, "einnsomplesupplitr 2}
32(
XGB
('learnine_rato''
("max.dephh" 8, "threshold"; 0.05, "ridge.aipha":
'step.size": 1)
CART
('nax dapen' 'nin samples leaf': 1, 'min samples splitrs 2)
(4()))4g)
XGB
("earringg 0.1, "naxdepthh" 5, "mast inaloraas 200, "subsample's 0.7]
HKT
('maxidapch'' 12, "Chreahold", 0.03, 'sidge.alphe's 5, "step=ize" 1}
31

---

Published as a conference paper at ICLR 2026
Table 11: Optimal Hyperparameters Across Various Datasets for Different Models
Datavet
Model
Best Hopergarameters
M5
(rset: 10.03 maxudepphl" ('max deptl/
Abalane
Linzar tiee HRT
"nins 40)
'sidge aipha., b, istep_aizef
rchresteld": 1}
M5 HRT
('W: (.0) "maxaepth'' "ai.sorplies.eaat 19)
"max,dephh' 2, "ssegeualphe' 3, "etepusizer 0,5, "thresnold's 0.5]
('M): 40.0 }
Alleross
Lincar tiee
"manddppll/ :
"ains 19)
HKT
'maxxdepth''
"sidge,alpha": aitep 1, "chresheld"; Se-05)
M5 Lineur tree HRT
('M:: 20.0)
CTSlice
"maxdopthh' "min.corpiessesstt 19) maxxdepth' "sioge.alphe"' 'ssepuaizefs "threshold"s 0.2)
200.0]
YenPPred
miee
Sanudupilies, mingamples2ee6-200
HKT
"naxxdepth'' 4 "sidgo.alpha" D, "step:atze"; Nautor, "chreshold"; ; 0.5)
XGB
'lasrninauaate" 0.01, 'naxudapta'; 3, "nassinattzz"" 200, "sutsampie": 0.73
M5
"w: 6.0)
Concrete
Linear
maxxdepthh' "ninsserpllesseee:: 40)
"axxdopth' "gidze.alphe": 9.1, "stap_aiza"; 0.5, "threshoid": 6)
XGB
"lesening,sall' 0.01, "maxulephh:; 'maeoriaaoos' 200, "ubbaapple' 0.2) 6.0)
Ainiol
Lineat more
mnax (nax depthtt : 6, fnin samples leaff: leaf:
50
HRT
''ax,ilppli/ teidgwalphh'' 0.01, 'slepaaia's "auter, "Cheeahold"; 1.5)
CART
11, "min_aomlles.eeat: 6, "minasenglessap tts 10)
XGB
learsing.ratt'' 0.1, "mox.deoth": 'nsestimatore'
"subsampie": 0.7)
tw: 4.0)
Lincar tre
"nax.dophh' 5, 'minssamtiasdeaa 13)
HRT
"maxxdophh' "siape.aiph"" 3.1, "stop.size": 0.1, "threshoid"; 0)
CART
(manddeptii: 5, minssesples 2, iningsm 10)
XGB
leerning rate': 0.01, 'nax.depta'; 5, 'nessinntorra's 200, 'subsarple'' 0.2) 20.03
DVElevators
Diee
"aaxduplll" "minsaaplees11a2" 20)
HRT
faax /'ax depth" 5, "-idge alphe' 10, 'step size!s 1, "threshold"s 0}
CART
"nax.dopth': 5, "min.carolco.caa": 4, "ninassmpper.sppit: 10)
XGB
learning.sate" 0.1, "mex.depth": 3, "nLestimators": 200, 'subsampie's 0,3)
Dileerons
"W:: 20.0) "maxacpth'' "nsx.depth'' "ssepe.alphh'' 10, "step_cize"= "auto', "tareshoid'; 0}
"min.caroies.caaff 20)
HRT
"nax depth' Fleeening.aate' n.I, "nexueephh"; 5, "npesiinatrr''' 200, 'ucbsangle's 0.7)
'win sarples leat!; fnin camplee splitf; 18)
XGB
Kiinematics
M5
(or: 4.0)
depth/ "maxilpphh'
semples leaff; 40)
Piidgealphhe' 1, 'stepaaixe's fautor, "threshold': 0)
("sxxopphh" "min.comples.eeett 'min_eampter.pppitrr
XGB
"lasraingsaata' 0.01, "naxudapta": 5, "nasstnatozs'' 200, "schsampie"; 0.7)
C&C
M5
tw: 10.0)
"max.depth"' 'min.samoies.loa"" 13)
"naxdopth"' "sidze_alphe": 300, "stap_siza" 0.01, "chreshold': 0.0)
("nax
sarplas
samples spliers
Blog
MS
XGB
(urr 20.0)
meateningratte' 0.05, deax't 3, fnest instors' 460)
HKT
depth' 'zidge alpha': 100,
0.01, "chreshold': 5)
Table 12: Hyperparameters for DGT and DeSemNet
Parameters
D'SeeNett
DGT
Height
5
6
Learning Rate
0.005
0.01
Batch Size
32
128
Num Epochs
80
200
Optimizer
Adam
RMSprop
32

---

Published as a conference paper at ICLR 2026
J DATASET DETAILS
This dataset is designed to predict the age of abalone, given eight features including categorical and numerical measurements describing physical characteristics.
Abalone:
CPUact: This dataset is designed to predict the portion of time that CPUs run user mode, given numerical features describing system performance measurements.
Ailerons: This dataset designed to predict the aileron control command, given numerical fea- tures describing the status of the aircraft.
CTSlice: This dataset designed to predict the relative location of CT slices on the axial axis of the human body, given numerical features describing bone and air distribution patterns extracted from CT images.
This dataset is designed predict the release year of song, given numerical features describing audio characteristics.
YearPred:
Conerete: This dataset is designed predict the compressive strength of concrete, given eight quantitative mixture components together with the age of the concrete as input features.
Airfoil: This dataset is designed predict the scaled sound pressure level of airfoils, given five numerical features describing aerodynamic and geometric properties.
Fried: This dataset is designed predict continuous target variable, given ten numerical features generated independently and uniformly.
D-Elevators: This dataset is designed predict the variation of elevator control signals for an F16 aircraft, given six numerical features describing the aircraft state.
D-Ailerons: This dataset designed to predict the variation of aileron control signals for an F16 aircraft, given numerical features describing the aircraft state.
Kinematies: This dataset is designed predict the forward kinematics of 8-link robot arm, given numerical features describing joint configurations. The variant used (8nm) is highly nonlinear and moderately noisy.
C&C (Communities & Crime): This dataset combines census, law-enforcement, and crime statistics predict community crime rates. high-dimensional and heterogeneous, serving as standard benchmark for acco-demmogappicc prediction.
Blog: This dataset predicts the number comments on blog posts using text-derived features. It features a skewed target distribution and nonlinear relationships, making it suitable for modeling online content influence.
Table 13:
Dataset Details
Dataset
features
Total sample number
Source
Abalone
4177
UCI!
CPUact
21
8192
Delve
Ailerons
40
13750
LIACC
CTSlice
384
53500
UCI
YearPred
90
515345
UCI
Conerete
1030
UCI!
Airfoil
1503
UCP
Fried
10
40768
LIACC
D-Elevators
9517
LIACC
D-Ailerons
7129
LIACC
Kinematics
8192
LIACC
C&C
127
1994
UCI
Blog
280
60021
UCI
33

---

Published as a conference paper at ICLR 2026
K EXTENDING HRT TO BINARY CLASSIFICATION
Although HRT was originally developed for regression tasks, it naturally extends to binary classi- fication. We treat the binary labels {0, 1/ directly as continuous regression targets, allowing the model to learn locally optimal linear decision functions each leaf. At inference, we interpret the tree output 5 as a real-valued score, clip it to [0. 1] to obtain a class-probability estimate, and predict the positive class if (ax) 2 0.5 and the negative class otherwise.
To assess the classification capability of HRT, we evaluated it on five public binary classification datasets (Breast-w, Credit-g, Banknote, Spambase, and Diabetes), comparing its performance with that of CART and XGBoost. Across these datasets, HRT achieves AUC, Accuracy, and Fl scores that are broadly competitive with-and in several cases exceed-thos of XGBoost, while requiring substantially fewer leaf nodes. This reduction in model complexity contributes to improved inter- pretability without sacrificing predictive accuracy. The detailed results (mean + standard deviation over 5 random repetitions) are reported in Table 14
To ensure a fair and consistent comparison, we conducted independent hyperparameter tuning for each model on every dataset. Although XGBoost constitutes a strong ensemble baseline due to its use of multiple boosted trees, HRT employs only a single-tree structure. Despite this difference in model complexity, the empirical results indicate that HRT can reach competitive levels of predictive performance while maintaining a simpler and more transparent model form. The best hyperparam- eter configurations selected for all models are summarized in Table 15
Table 14: Performance comparison on five real-world datasets. Values for AUC, ACC, and F1 are reported mean + standard deviation, while values for Leaves and Fit Time reported mean only. For AUC, ACC, and FI, higher is better (1). For Leaves and Fit Time, lower is better (L). The best result in each column for each dataset is bolded. Significant improvements over the best baseline are marked with (p < 0.05).
Dataset y, N.)
Model
AUC(1))
ACC(1)
Fi(t)
Leaves(()
Fit Time (s)())
CART
0.960 + 0.006
0.931 0.013
0.899 + 0.022
Breast-w (9.699)
XGBeost
0.986 + 0.006
0.949 + 0.006
0.927 + 0.010
NIA
1.733
Our Method
0.993 0.0031
0.956 0.007
0.935 1 0.010
2.0
0.044
CART
0.707 + 0.019
0.710 +: 0.017
0,803 + 0.013
22.2
0.023
Credit-g (20,1000)
XGBoost
0.782
0.003
Our Method
0.791 + 0.009
0,760 + 0.004
0.840 + 0.005
1.0
0.040
CART
0.983 + 0.008
0.983 0.007
0 + 0.008
19.8
0.005
Banknote (4.1372)
XGBeost
0.999 + 0.000
0.987 + 0.007
0.986 + 0.008
N/A
1.114
Method
0.000
CART
0.924 + 0.010
0.900
0.009
0.868 + 0.011
22.2
0.030
Spambase (57,4601)
XCBeost
0.981 + 0.003
0.940
0.008
0.923 + 0.010
NIA
1.928
Our Method
0.966 1 0.006
0.922
0.006
0.900 1 0.007
8.0
0.901
CAKT
0.757 + 0.020
0.731 + 0.010
0.549 + 0.089
7.4
0.005
Diabetes (8,768)
XGBoost
0.030
Our Method
0.817 + 0.012
0.762 0.017
0.629 + 0.025
2.0
0.031
L THE USE OF LARGE LANGUAGE MODELS (LLMs)
Following the ICLR 2026 guidelines regarding the use of Large Language Models (LLMs), we hereby disclose that LLMs were utilized in the preparation of this paper:
Assisting in drafting and revising portions of the text, such as rephrasing existing sentences, checking grammar, and improving linguistic flow.
Text Drafting and Refinement:
Content Clarification and Summarization: Helping to clarify complex concepts or sum- marize lengthy explanations.
https://www.des
34

---

Published as a conference paper at ICLR 2026
Table 15: Hyperparameters for Classification Datasets
Dataset
Model
Best Hyperporamatters
CART
I'msx.cepth' : 5, imta.oomples.leott 4, "min.sompies.aplitt 2}
Breast-w
XGBooa
{ learningrrate" 0,35, "rasulepp: 5, "nuestinatarss 200, "subsemp 0.7)
Our Method
"'axxsopta': 1, "ridge.iipta' 'stepunize' "throshold";
CART
('nax.daphh"; 5, foin.aanleosteaf' 2, "minssumpees.ppii"" 10]
Credit
XGBooot
('caarnigg
0,32, dopsshf "stinntot"""" 200, 'subsampie"; 0.7)
"maxxBephhh' 1, 'ridge.alpha" 10, 'stepaize": 3.01, "Chreaholdd: 0.5)
CART
I'axxcepthh' 7. i
"minssmpieessppiees 2}
XCBBorat
(' ('larmnigrratte' 0.1, 'nas,depth'; 3. Inpestinatons 260, 'subaample's 1.0)
Our Method
I'max.captn': 1, "ridge.clpt":: 6, "stap.nize"; 0.01, "shreshold"; 0}
CART
("max.daphh'; 5, "oinseaplessdeatt' 4, 'min.omnpos.apiit' 10)
Spumbase
XGBBoost
('laarninq 0,35, " 5, "nessinators" 200, 'subsompie": 0.7)
Method
('maxdepthh'' fridgealgha""
'stepaize": "threahold';
CART
I'mxxceephh's 3, iniaeenpples.lett 2, 'min.somglesaplitt" 2}
Diahetos
XGBoodt
("laamiingraate"
"nexulepshhs inaestinattrr'' 'subxample's 0.7)
Our Metbod
('mex.copth' : 1, 'ridge.sipta': 1, "stepunize's 1, "shreshold': 0)
Formatting and Structural Suggestions: Providing advice on LaTeX formatting and as- sisting in structuring certain sections.
All content generated by the LLM underwent rigorous review, editing, and validation by the human authors. The human authors bear full responsibility for the entirety of the paper's content. This disclosure is made to ensure transparency and comply with ICLR's 's policy.
35


Reviews:
[{"review_id": "MvTateWfW1", "reviewer": "Reviewer_GmnV", "summary": "This paper proposes a novel splitting method for oblique regression trees. It uses two linear predictors and the splitting depends on which predictor is larger or smaller. This splitting problem is solved via an alternating fitting procedure which is equivalent to a damped Newton method within fixed partitions. This paper proves that the oblique regression tree with such a splitting mechanism is a universal approximator.", "questions": "On average, how many iterations are needed to split a node?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written with clear content organization, method description, and mathematical notation.\n2. The splitting mechanism of oblique regression tree is novel, i.e. the combination of two linear predictors and hinge function.\n3. This paper proves that the proposed method is a piece-wise linear model class which is a universal approximator. Thus its expressive power is underpinned by theoretical foundation as well as experimental results.", "weaknesses": "1. Some implementation details are missing, such as how to initialize two linear predictors.\n2. The formal global convergence proof is not provided. I understand this is challenging. But I think the authors can try some weaker conclusions. For example, under what conditions is the loss function monotonically decreasing?\n3. It is better to add experiments for binary classification tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "26QAmhL3Wg", "reviewer": "Reviewer_GFxH", "summary": "The paper proposes to learn an oblique decision tree with linear leaf predictors using the traditional greedy recursive partitioning approach but with the variation on the splitting procedure. The splitting procedure uses the hyperplanes at both leaves to define the splitting hyperplane: $\\mathbf{\\theta_1} - \\mathbf{\\theta_2}$. The learning algorithm uses the current split to define Newton updates, but it is not clear how it relates to finding the global optimum. Experiments are performed on synthetic and real datasets showing improved accuracy of the proposed method.", "questions": "No questions.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "Oblique decision trees are an important model class which has not been as widely studied. This paper proposes one approach in learning this relatively unexplored model class.", "weaknesses": "* Objective function is defined only for a splitting criterion. It is not clear what is the global objective being optimized.\n\n* During splitting, the internal decision node hyperplane is defined by its two linear leaf weights. Ideally, these sets of parameters (decision split hyperplane and its two children linear weights) must be independent. It is not clear why this way of coupling is used.\n\n* It is not clear whether the proposed algorithm optimizes eq. 1. No theoretical guarantees of convergence or optimality is shown.\n\n* The comparison experiments on real world data are not apples-to-apples. The baselines of CART and TAO, for example, use constant leaves, while the proposed approach uses linear leaves. Piecewise constant models are in general not suitable for regression problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YsRgpm30e6", "reviewer": "Reviewer_e2Hz", "summary": "The paper proposes the Hinge Regression Tree (HRT), an algorithm for building oblique regression trees by jointly learning two linear predictors at each split. Each node models its output as a hinge function, leading to a piecewise linear regression surface. Training alternates between (i) fitting the two linear models with ordinary least squares (optionally ridge-regularized) and (ii) reassigning data to the branch that yields the higher prediction. The authors describe this alternating fitting as a damped Newton method within fixed partitions and show that, with small damping factors, it converges stably in practice. They also restate a known universal-approximation result for piecewise linear models and present experiments on synthetic and real regression datasets, comparing HRT to CART, TAO, DGT, DTSemNet, and XGBoost.", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The node training procedure (alternate OLS fits for two linear predictors under a hinge) is simple, transparent, and easy to re-implement. The paper provides explicit pseudocode and a complete build procedure, which supports reproducibility.\n2. Compared to the baseline methods, HRT enjoys competitive performance.\n3. The paper is mostly clear and easy to understand.", "weaknesses": "1. The contribution is overstated. Within fixed partitions, the update equals a Gauss–Newton/OLS step (standard), which is not difficult to prove. However, the hard part in the decision tree method is partition switching. Unfortunately, there is no guarantee of monotone decrease or convergence of the alternating fit-reassign procedure. I am not convinced that this part should be left to future work, as this guarantee is far more interesting and important than the updates in each subspace to show the validity of the proposed method.\n\n2. The discussion of the damping effect is insufficient. For example, the results improvement for $\\mu=0.01$ (damping) and $\\mu=1.00$ (no damping) is only marginal in Tables 4 and 5. If using $\\mu=1.00$ (OLS) and the standard fallback algorithm can provide competitive results, then the advantage of the damping term, which is a contribution of the paper, is unclear, and the reformulation into Gauss-Newton update seems to be unnecessary.\n\n3. More benchmarks could be given to further show the effectiveness of the methods and examine the alternating fit-reassign procedure. In particular, high-dimensional datasets such as Communities & Crime (UCI), BlogFeedback (UCI), and baselines such as LightGBM can be included.\n\n4. Some of the improvements of HRT over other methods are marginal (within 1%). Statistical significance tests should be given to potentially make many differences statistically distinguishable.\n\n5. Although the authors discussed the complexity, there is no comparison of the computational time, especially how much time is sacrificed for using damped optimization. This makes the efficiency of the proposed method in practice hard to assess. \n\n6. The technical contribution of the theory is limited. The approximation of piecewise linear functions to any $C^2$ target in many similar cases is known, e.g., Breiman (1993), and ReLU approximation Barron (1993). The sample complexity of Oblique trees is also established in Cattaneo et al. (2024). Therefore, the approximation result is not surprising.\n\nBarron (1993): Universal approximation bounds for superpositions of a sigmoidal function, IEEE Transactions on Information Theory", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvFqkkEGJ4", "reviewer": "Reviewer_AC5G", "summary": "The paper introduces hinge regression tree (HRT), a new method for learning oblique regression tree splits. The key idea is to formulate each split as a nonlinear least-squares problem over two linear predictors, yielding an alternating fitting procedure that is mathematically equivalent to a damped Newton method within fixed partitions. HRT demonstrates competitive performance in benchmarking experiments compared to standard baselines. Overall, the proposed method appears promising. My comments are as follows:\n\n- Section 3: Consider adding a short subsection before Section 3.1 that briefly recaps oblique decision tree methods. The current text is related but does not explicitly state that this approach is an instance of oblique decision tree regression.\n\n- Line 154: I believe this represents a linear decision boundary, rather than a hinge-based one. \n\n- Line 191: The optimization behavior accounting for partition changes is not analyzed, leaving a disconnect between the theoretical guarantee and the practical performance. The empirical results show that the algorithm converges on real data, which is reassuring; however, I would still view the algorithm as heuristic, and suggest avoiding terms such as “rigorously” (line 49) or “solid theoretical foundation” (line 50).\n\n- The computational efficiency of the proposed algorithm has not been comprehensively evaluated.", "questions": "See summary above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "See summary above.", "weaknesses": "See summary above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)