Title: Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting
Primary area: optimization
Keywords: Optimization, Regression trees, Newton method, Convergence

Abstract: Oblique decision trees combine the transparency of trees with the power of multivariate decision boundaries—but learning high‑quality oblique splits is NP‑hard, and practical methods still rely on slow search or theory‑free heuristics.
We present the Hinge Regression Tree (HRT), which reframes each split as a non‑linear least‑squares problem over two linear predictors whose max/min envelope induces ReLU‑like expressive power.
The resulting alternating fitting procedure is exactly equivalent to a damped Newton method within fixed partitions, which in practice demonstrates fast, stable convergence and allows an optional ridge regularization.
We further prove that HRT’s model class is a universal approximator with an $O(\delta^2)$ rate, and validate it on synthetic and real‑world benchmarks.
HRT achieves competitive performance compared to single-tree baselines and maintains a more compact structure—making it a transparent and theoretically grounded regressor.