Jacobian Aligned Random Forests
Authors: Sarwesh Rauniyar
Venue: ICLR 2026 Poster (2026)
Keywords: Random forests; Decision trees; Axis-aligned splits; Oblique decision boundaries; Feature interactions; Supervised preconditioning; Gradient-based feature transforms
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Axis-aligned decision trees are fast and stable but struggle on datasets with rotated
or interaction-dependent decision boundaries, where informative splits require linear combinations of features rather than single-feature thresholds. Oblique forests
address this with per-node hyperplane splits, but at added computational cost.
We propose a simple alternative: JARF, Jacobian-Aligned Random Forests. Concretely, we fit a random forest to estimate class probabilities or regression outputs,
compute finite-difference gradients with respect to each feature, form an expected
Jacobian outer product/expected gradient outer product, and use it as a single
global linear preconditioner for all inputs. This preserves the simplicity of axisaligned trees while applying a single global rotation to capture oblique boundaries
and feature interactions that would otherwise require many axis-aligned splits to
approximate. On tabular benchmarks, our preconditioned forest matches or surpasses oblique baselines while training faster. Our results suggest that supervised
preconditioning can deliver the accuracy of oblique forests while keeping the simplicity of axis-aligned trees.

Published as a conference paper at ICLR 2026
JACOBIAN ALIGNED RANDOM FORESTS
Sarwesh Rauniyar
Department of Applied Mathematics and Statistics Johns Hopkins University
srauniyl y aal umni jh. edu
ABSTRACT
Axis-aligned decision trees are fast and stable but struggle on datasets with rotated or interaction-dependett decision boundaries, where informative splits require lin- ear combinations of features rather than single-feature thresholds. Oblique forests address this with per-node hyperplane splits, but at added computational cost. We propose a simple alternative: JARF, Ancobian-Aligned Random Forests. Con- cretely, we fit random forest to estimate class probabilities regression outputs, compute finite-difference gradients with respect to each feature, form an expected Jacobian outer product/expected gradient outer product, and use it global linear preconditioner for all inputs. This preserves the simplicity of axis- aligned trees while applying single global rotation to capture oblique boundaries and feature interactions that would otherwise require many axis-aligned splits to approximate. On tabular benchmarks, our preconditioned forest matches sur- passes oblique baselines while training faster. Our results suggest that supervised preconditioning can deliver the accuracy of oblique forests while keeping the sim- plicity of axis-aligen axis-aligned trees.
single
INTRODUCTION
On tabular data, tree-based ensemble methods widely used and often outperform deep networks on structured datasets (Breiman 2001] Grinsztajn et al. [2022). Methods like Random Forests and gradient boosting are popular for their strong performance with minimal tuning, robustness irrele- vant features, and inherent handling of mixed data types. However, these models are fundamentally built on axis-aligned decision trees, where each split considers only single feature. This design makes training fast, but it fails when the boundary depends such cases, an axis-aligned tree must simulate an oblique split through a series of orthogonal cuts, resulting in deeper trees and fragmented decision regions. This inefficiency can hurt accuracy and sample efficiency, especially on tasks with strong feature interactions.
rotated axis or a mix of features. In
Researchers have long recognized this limitation and explored oblique decision trees that split lin- ear combinations of features rather than single features. Oblique Random Forest variants have shown improved accuracy over standard forests by capturing feature interactions each node (Menze al., 2011 Katuwal et al. 2020). Unfortunately, these benefits come with significant drawbacks. Learn- ing the optimal linear combination at each node more complex optimization problem, often requiring iterative techniques or convex solvers that augment training cost (Murthy et al., [994] Menze al. 2011 Katuwal al. 2020). Oblique splits also tend introduce many more param- eters and can be prone to overfitting without careful regularization. As a result, oblique forests are often slower and less practical use than standard axis-eligned ones.
In this paper, we propose new approach to achieve this goal: a global, supervised feature transfor- mation that preconditions decision forests. We term our method JARE, short for Jacobian Aligned Random Forest. JARF learns mapping of the input features by leveraging information from the model's predicted class probabilities. In particular, we estimate the expected Jacobian outer product (EJOP) of the class probability function, which is covariance matrix that measures how sensi- tive the predicted class probabilities are to changes in each input direction (Trivedi et al., 2014). By rotating and scaling the original feature space along these directions, JARF creates new feature space where the most label- prediccive variations are axis-aligned. A standard Random Forest trained on this transformed space can then simulate oblique decision boundaries with simple axis-aligned

---

Published as a conference paper at ICLR 2026
splits. Crucially, this transformation is one-pass and model-agnostic: it requires only lightweight computations and does not alter the inner workings of the forest. The result is a middle ground between axis-aligned and fully oblique trees: retain the training speed, simplicity, and robust- ness of conventional Random Forests, while significantly boosting their ability to handle rotated or interacting features.
We demonstrate that applying JARF closes much of the accuracy gap between axis-aligned and oblique forests. In experiments, JARF achieves better accuracy than significantly more complex oblique-tree ensembles with substantially lower computational overhead, and also outperforms lighter, data-agnostic oblique variants (e.g., random- rotilmmrotttiooo/ooeetttonn forests) on most datasets. Through extensive experiments on diverse datasets, we show that JARF consistently improves the performance of baseline forests and gradient boosting models. These results highlight the effective- ness and generality of using probability gradients to inform feature space geometry in supervised learning.
2
RELATED WORK
2.1 SUPERVISED PROJECTION FOR DIMENSION REDUCTION.
Early work in statisties introduced supervised linear projections reduce dimensionality while preserving predictive information. Sliced Inverse Regression (SIR; Li 1991) and Sliced Average Variance Estimation (SAVE; Cook 2000) seek a low-dimensional subspace of features that most influences the response. These approaches identify directions in feature space that capture variation of Y given X, and they foreshadow modern gradient-based dimension reduction. Conceptually, they motivate using label information to precondition the inputs before fitting model, which is a perspective we adopt. For classification, including multiclass, SIR and SAVE apply directly by slicing class labels (Li) 1991 Cook 2000). Closely related, Fisher's linear discriminant analysis and its multiclass extension (Rao) learn at most one fewer projection than the number of classes, because only that many independent directions are needed separate the classes (Fisher 1936
Rao 1948).
2.2 GRADIENT- BASED GLOBAL SENSITIVITY (EJOP).
More recent methods leverage derivatives of predictive function with respect to inputs to find informative projections. In regression, the expected gradient outer product (EGOP) is
Ex {x{Vf(X)Vf(X)) and recovers effective dimension-reduction subspace (Trivedi et al. 2014
For multiclass settings, the expected Jacobian outer product (EJOP) is ((x))())((( returns class probabilities; the leading eigenvectors emphasize directions along which predictions change the most (Trivedi & Wang 2020). Researchers have applied these gradient- based summaries
where
to tasks like metric learning and sensitivity analysis (Perronnin et al.) 2010: Sobol' & Kucherenko)
2009). Our approach, JARF, follows this paradigm by computing a global, label-informed linear transform from EJOP/EGOP before training a forest.
2.3 OBLIQUE DECISION FORESTS.
Decision trees that split on linear combinations of features were shown early on to yield compact, accurate models when boundaries are tilted relative to the axes (Breiman, 200T). OC/ performs hill-climbing each node optimize hyperplane split, trading extra per-node computation for improved fit (Murthy et al. 1994). Rotation Forest applies unsupervised PCA-based rotations to random feature subsets independently per tree, so subsequent axis-aligned splits behave like oblique splits in the original space (Rodriguez et al. (006). Canonical Correlation Forests (CCF) compute supervised projections at each node via canonical correlation with the outputs, aligning splits with local predictive structure (Rainforth & Wood 2015). Another line samples random linear combi- nations for candidate splits; Breiman noted this idea in early forest variants (Breiman 2001). and Sparse Projection Oblique Random Forests (SPORF) constrain projections to be very sparse, im- proving interaction capture while mitigating overfitting (Tomita et al.) (020). While effective, these methods either increase per-node optimization (OC1, CCF) or rely on unsupervisee/random pro-
2

---

Published as a conference paper at ICLR 2026
jections (Rotation Forest, SPORF), that do not always align with predictive directions. This often means more trees or extra constraints are needed.
2.4 COMPARISON AND POSITIONING OF JARF.
Unlike node-wise oblique methods, JARF provides a one-pass, global, and supervised precondition- ing that leaves the tree learner unchanged. By constructing single EGOP/EIOP- transform shared across all trees, JARF supplies a coherent feature representation informed by all training la- bels, with negligible overhead during tree construction. This global projection amplifies directions along which p(y x) varies and attenuates irrelevant ones so that standard axis-aligned splits can approximate oblique boundaries. this way, JARF competes directly with oblique forests, aiming to deliver comparable accuracy with substantially lower complexity and simpler deployment.
METHODS
3.1 PROBLEM SETUP AND NOTATION
We consider multiclass classification with inputs x E Rd and labels y (1,..., C}. Let f: R A C--1 denote probabilistic classifier whose e-th component fz(x) estimates p(y = c 2). Standard Random Forests (eFiman 2001) build axis-aligned decision trees on -i..... F.,] which can require deep trees when informative directions are linear combinations of features. Our goal is
to learn a single, global, supervised linear map H Rdxd such that training an ordinary RF on the transformed data X H captures those predictive combinations with shallow, axis-aligned splits.
3.2 PROBABILIT-GRADEES PRECONDITIONING
The central object in JARF is EJOP-style matrix that summarizes how class probabilities change with small perturbations of z. Let X Fd denote random input drawn from the data eetta-enerating distribution Px: unless stated otherwise, expectations B[.] are taken with respect X Pxx Let [z(x) E RdxC be the Jacobian whose columns are gradients (x). The expected Jacobian outer product (EJOP)
Ho [x(()(()),(00 = E
(I)
matrix whose leading eigenvectors span the directions along which p(y ()) varies most (Trivedi et al.] 20144: Trivedi & Wang 2020). In practice, we replace Ex by an empirical average over the (subsampled) training inputs to estimate Ho, and use this estimate define global linear preconditioner H. For regression tasks with scalar output E I,, Equation I reduces to the expected gradient outer product (EGOP):
Ho = Fx (VXX)VV((X)
where f RR IR is the regression function. The same preconditioning procedure applies: estimate Hy via finite differences and use it to transform the feature space before training the forest.
Connection supervised dimension reduction. Equation [ is the gradient/Jacobian analogue of supervised projection methods such as SIR and SAVE Li 199| Cook 2000): instead of relying on first/second moments of X Y, JARF aggregates sensitivity of p(y x) to æ, producing label- informed geometry.
3.3 ESTIMATING Ho VIA FINITE DIFFERENCES
The estimator below is the EJOP estimator proposed by Trivedi Wang 2020)) Our only change the surrogate used to approximate p(y | æ): we use a random-forest classifier whereas Trivedi & Wang (2020) used a kemel (regression) estimator. We construct an empirical estimate of Ho. denoted Ho. in three steps:

---

Published as a conference paper at ICLR 2026
1. Probabilistic model. Fit random forest f on the training data Duain {(2,, Uc) Ji
equivalently, on the design matrix X = [2] ... ................................ " E I and label vector =
(V1....,,lh Un) E (1....,C)) C) This surrogate is used only to query class probabilities p(c z), not as the final predictor.
2. Per-feature probability gradients. For subsample {I,, ys ) estimate directional derivatives along each coordinate using a centered finite difference with step e > 0:
93 (1(2;)) c) 2 Se(xr + Bee) fe(xr; $ee)
where ej is the j-th basis vector. Stack gradients Gi(c) 1y ..r Ia(a e))
3. EJOP estimate. We use the following estimator:
m H.
! Σ C(()))((())) m
3.4 PRECONDITIONING MAP
We the EJOP estimate linear preconditioner. Define
ń Ho + +ha
(x
where the small diagonal term improves numerical conditioning. To keep feature scales comparable, we normalize
A tr(H)/d
(3)
We then map inputs:
((x) = RJ R
(4)
and train the forest on the transformed design matrix X M. This preserves dimensionality and emphasizes directions along which class probabilities vary.
3.5 TRAINING THE FOREST ON PRECONDITIONED FEATURES
After computing A once, we train a Random Forest { ((x)), de Pi::
RF H y).
At inference, we transform test point via ((x)= x. H and evaluate h h()())))
3.6 PRACTICAL CONSIDERATIONS
Surrogate model for EJOP estimation. Since the true Bayes-optimal class probabilities f(x) = p(y] x) are unknown, we require surrogate model f to estimate the EJOP matrix. This surrogate used solely to query class probabilities $(c .z) for gradient estimation. While any probabilistic classifier (logistic regression, kernel methods, neural networks) could serve this purpose, we choose random forests for three reasons: (1) they provide stable probability estimates due to ensemble aver- aging, (2) they are computationally efficient compared to alternatives like kernel regression, and (3) using the same model family for both EJOP estimation and final prediction maintains consistency.
Finite differences and non-differentiaailiyy, Our method computes directional sensitivities via rather
finite differences [p(x + {e,) p(x Se,)l/E rather than analytical derivatives, making it com- patible with non-smooth models like random forests whose predictions piecewise constant. The variance of these finite-difference estimates remains low despite the discontinuous nature of indi- vidual trees because ensemble averaging smooths the aggregate predictions. The adaptive step size ěÿ = MAD(X.;) ,) (x.x))//... /0.6745 and quantile-based clipping ensure that probe points typically cross informative split thresholds while remaining within the empirical data range, yielding meaningful gradient estimates even for free-based models.
4

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
We evaluate JARF against oblique forests on diverse datasets and check whether it preserves the simplicity and efficiency of Random Forests.
4.1 DATA AND PREPROCESSING
Real-data suite. We evaluate on a suite of tabular prediction tasks. Our primary classification benchmark consists of ten widely used OpenML/UCI datasets: adult, Dank-marketing, coverrype, phoneme, electricity, satimage, spambase, magic, letter, and vehicle. These span numeric and mixed-type features and range of sample sizes. To probe more challenging regimes, we addi- tionally include five five real-valued regression tasks from OpenML, where we apply the EGOP variant of our precondi- tioning.
tabular classification datasets with > 100 features and
For all tasks we use a 5x2 pross-validation protocol (five random 50/50 train/test splits, each eval- uated twice with roles swapped). For classification tasks the splits are stratified. All methods share identical folds. All preprocessing is only on the training portion of each fold and applied to the corresponding test split to avoid leakage. The JARF transform H is likewise learned only from the training fold and then applied to transform the corresponding test fold. For the simple global projection baselines (PCA+RF and LDA+RF) fit the PCA or LDA map the training fold and reuse the same projection to embed the associated test fold before training a standard random forest on the projected features.
Simulated suite. To evaluate JARF under controlled conditions that are known to disadvantage axis-aligned splits, we create a synthetic problem. This setting contains a single linear decision boundary that is not aligned with the coordinate axes. We draw (0, Ia) with dl E {10, 50, 100) and fix a rotation angle E (15", 30%, 45%, 60%). We define a unit normal in the (e1, e2 -
= cos 1 + 02
and assign labels by noisy halfspace
H x
~~~0000"") 0.2,
which avoids perfectly separable cases. This matters because an axis-aligned tree must approximate the tilted boundary with many splits; oblique split (or global preconditioner) solves it with far fewer nodes.
4.2 METHODS COMPARED
We call a tree/forest axis-aligned if each split tests a single coordinate æy S r; it is oblique if splits test a linear combination w T x S with w E TRd. In our comparison, RF and XGBoost use axis-aligned splits: RotF, CCE, and SPORF employ obligue hyperplanes. Our method learns a single global linear map H using EJOP/EGOP and then trains an axis-aligned forest on X H; in the original coordinates the induced splits are shared oblique hyperplanes He, S (same H for all trees/nodes). Below we outline each method, its split type, and where supervision or extra complexity appears.
RF (axis-aligned). Random Forests (RF; (1Fiman 2001) use CART nodes with axis axis-aligned tests T S bagging, and feature subsampling. We use 200 trees, Gini impurity, and standard defaults. This is the fastest and most robust baseline; all trees remain strictly axis-aligned.
Rotation Forest (oblique via global per-tree rotation). Rotation Forest (RotF; Rodríguez et al., 2006) builds each tree after applying block-diagonal PCA rotation / learned from disjoint subsets of features (here K=6 subsets). The tree then makes axis- axis-aligned splits in the rotated space X R, which correspond oblique hyperplanes w TT x < r in the original coordinates. Rotations unsu- pervised (label-age agnossicc) and are recomputed independently per tree (global pre-tree transform, not per node).
5

---

Published as a conference paper at ICLR 2026
Canonical Correlation Forests (oblique per node). Canonical Correlation Forests (CCF; Rain- forth & Wood 2015) compute a supervised canonical correlation analysis (CCA) projection at each node using the node's data and the current labels; the split is then taken along one of the projected coordinates. Thus, CCF induces oblique hyperplanes that adapt to the local class structure. Because projection is learned at every node, training cost is higher than RF/RotF.
SPORF (sparse oblique per node). SPORF (Tomita al. 2020) samples small set of sparse random directions w at each node, evaluates impurity reductions, and chooses the best direc- tion/threshold. This yields oblique but interpretable splits with controllable complexity through sparsity. We use 200 trees and the authors' recommended sparsity/number of candidate directions.
XGBoost aliisealigned boosting). XGBoost (Chen & Guestrin 2016) fits an additive ensemble of shallow CART trees with axis axis-aligned splits Ij via gradient boosting. We include small shared grid over depth, learning rate, and L2 penalty. It is a strong tabular baseline and its nodes are axis aligned.
PCA+RF (global unsupervised projection). As simple "one- "oot=shoo" projection baseline fit single PCA (principal component analysis) transform WPCA E Rdxd on the training features each fold (ignoring the labels) and rotate all inputs to X WPCx. We then train standard axis aligned RF on these rotated features using the same hyperparameters as the RF baseline. Splits are axis aligned in PCA space but correspond to fixed set of oblique directions in the original coordinates.
LDA+RF (global supervised projection). Analogously, construct global supervised projec- tion using linear discriminant analysis (LDA). For each training fold we fit an LDA map WLDA using the class labels, embed the data into the resulting LDA space, and train standard axis-aligned RF on these transformed features with the same hyperparameters as RF. Here label information used once, form single global projection shared by all trees; in the original coordinates the splits again correspond to oblique hyperplanes.
JARF (global transform, axis-aligned trees). Our method learns a single supervised linear trans- form A on the training fold by estimating the EJOP/EGOP matrix from finite-difference probability gradients (we choose per-feature steps E a (XAD(X.))/O.... /0.6745 with differences when x, E lies within the empirical range of feature j. otherwise a one-sided differ- ence). We set H Ho (adding small 7/2 for conditioning) and then train standard RF (200 trees) on the transformed features XH. Splits are axis axis-aligned in the transformed space, which correspond to shared oblique hyperplanes x: He, S r in the original coordinates. This preserves RF's simplicity and training profile while injecting label-awart geometry common to all trees.
0.1; we use centered
4.3 METRICS AND STATISTICAL TESTING
Our primary metric is Cohen's (chance-corrected accuracy) both the synthetic and real datasets we use and R for the regression tasks we test. For each dataset and algorithm A we report the effect size (AA) x (RF) *(AA ()); negative values indicate outperforms RF and positive values indicate RF is better (visualized with beeswarm plots across datasets). Next, we test whether our global transform aligns with oblique split directions using principal angle analysis between subspaces. Finally, we measure training time for each method we compare and perform ablation studies.
5 RESULTS
We present results controlled simulations (to isolate phenomena that favor oblique splits) and on the real-data suite from Sec. 4
5.1 SIMULATED STUDIES
We evaluate a canonical setting where axis-aligned trees are known to be inefficient and oblique methods help: a rotated hyperplane classifier where the boundary forms angle 0

---

Published as a conference paper at ICLR 2026
{15", 30,, 45", 60%} with the coordinate RF, RotF, CCF, SPORF, JARF, XGB, and the PCA+RF and LDA+RE projection baselines. As grows, RF and XGB degrade the fastest, while PCA+RF and LDA+RF give only modest improve- ments over RF and remain well below the oblique forests. JARF consistently achieves the highest moderate and large rotation angles. These results show that EJOP-based preconditioning finds directions that line with the oblique boundary, letting the forest build efficient trees even when the decision surface is far from axis-aligned. For small rotations all methods are fairly close and RF remains competitive, suggesting that JARF's advantages manifest primarily when axis- axis-alignment assumptions are substantially violated.
Figure 1 reports Cohen's function of 0 for
Simulated rotated boundary
0.925
0.900
Cohen's K
0.875
RotF
0.850
CCF
SPORF
JARF
0.825
XGB
PCA+RF
0.800
LDA+RF
20
40
60
Rotation angle (degrees)
Figure 1: Cohen's versus rotation angle 9 for RF, RotF, CCF, SPORE, JARF, XGB, and the PCA+RF and LDA+RF baselines on the simulated rotated hyperplane problem. JARF attains the highest at moderate and large rotations, while PCA+RF and LDA+RF offer only modest gains over RF and all axis aligned methods (RF, XGB, PCA+RF, LDA+RF) degrade more quickly than the oblique forests increases.
5.2 REAL-WORLD BENCIMARKS
Tables and report par-dataset test performance on the extended real-data suite (Sec. 4) which includes the I0 core OpenML/UCI classification tasks, five additional nigher-dimensional tabular classification datasets with d > 100, and five regression tasks. Across the 15 classification datasets, JARF attains the best result on 12 tasks and is never worse than RF by more than one standard error. On average, JARF achieves the highest Cohen's with a mean of ()810 compared to 0.704 for RF, 0.715 for RotF, 0.715 for CCF, 0.723 for SPORF, 0.709 for XGB, 0.692 for PCA+RF, and 0.697 for LDA+RF. The largest gains appear datasets with complex high-dimensional decision boundaries, such as electricity, magic, letter, and the d>1100 benchmarks (higgs, madelon, bioresponse, jannis, mnist-784), where JARF typically improves over RF by roughly 0.08-0.1 0.11 in k. On the five regression tasks (Table 2). JARF also attains the best 12 on every dataset, with a mean of 0.836 compared 0.776 for RF and lower values for all other baselines, indicating that the benefits of EJOP-based preconditioning extend beyond classification.
Figure summarizes effect sizes relative to RF via A(A) x <(RF) - x(A). The beeswarm plot shows that JARF consistently improves over RF (the vast majority of points lie below zero), whereas other oblique methods and the simple global projection baselines (PCA+RE, LDA+RF) cluster much closer to zero and sometimes degrade performance. This pattern supports the view that the EJOP- based preconditioning is doing more than generic global PCA/LDA step.
5.3 EFFICIENCY AND COMPUTE
We measure training time on the same CPU. For JARF, the total cost has three parts: (i) fitting the surrogate RF used to estimate the conditional class probabilities vi(x;), (ii) computing the EJOP
7

---

Published as a conference paper at ICLR 2026
Table 1: Real-data classification performance (Cohen's K, mean + s.e. over CV splits).
Dataset
RF
RotF
OCF
SPORF
XGBB
PCA+RF
LDAIRF
JARF
adult
0.605 + 0.3062
0.630 + 0.0067 0.627 0.0070 0.629 : 0.0688 0,618 : 0.0059 0.595 = 0,000 0.600 0,0051
0.720
0.0063
bank marketntti
0.0081
0.601
0.602 0.0755
0.700
0.0084
coventyp
0.612 + 0.0041
0.616 +: 0.0043 0.631 +: 0.0040 0.633 :: 0.0022 0.622 # 0.0055 0.602 = 0.0041
0.607
0.0044
0.790
0.0047
phoneme
0.659 + 0.3098 0.652
0.00%6 0.649 0.0094 0.662 + 0.097 0.657 : 0.0101 0.649 0.006
0.654
0,0097
0
0.0999
electricity
0.0051
0.703
0.689
0.780
0
satimage
0.731 +: 0.0050 0.840 +: 0.0053 0.737 1: 0.0051 0.741 #: 0.0054 0.743 :: 0.0049 0.721 = 0.0050
0.726
0.0051
0.830
0.0048
spambise
0.751 + 0:3095 0.770 0.0097
0.766 0.0K98 0.774 + 0,0891 0,764 : 0.0093 0,741 0.0064
0.746
0,0085
0
0.0990
magic
0.297 0.0672.
0.808 0.0076 0.890 + 0.0080 0.794 +
0,792
0.0079
letier
0.795 + 0.0108 0.796 + 0.0111 0.803 : 0.0109 0.812 = 0.0116 0.799 + 0.0113 0.765 = 0.0109
0.790 2: 0.0110 0.860
0.0112
vehicle
0,900 4: 0.0137 0.880 0,0134
0.877 0.0131 0.879 : 0.0135 0.570 :: 0.0160 0.872 0.0136
0.577 0.0133
0.890
0.0135
higgs
0.690 4: 0.9045
0.708
0.0048 0.712 + 0.0029 0,700 +:
0.790
0.0050
midelion
0.640 + 0.3080 0.655 + 0,0081 0.660
0.0031 0.662 : 0.0082 0.648
+ 0.0081 0.630- = 0.007
0.635 = 0.008)
0,770
0.00S5
bioresponse
0.675 4: 0.0065 0.688
0,0067
0.690
0.0068
0.692 + 0.0069
0.682 + 0.0066 0.665
0.0061
0.670
0,0065
0.800
0.0070
junnis
4: 0.0050 0.722
0.725
0.9052 0.728 0.0053
0.0051 0.700
0.705
0.0049
0.830
0.0054
mnist-784
0,720 + 0.0040 0.732 + 0.0042
0.735 + 0.0043 0.737 + 0,0044 0.725
+ 0.0041 0.710 = 0.0040
0.715
0.0044
0.850
0.0045
Mean + s.c.
0.701 + 0.0100 0.715 0.0102
0.715 0.0103 0.723 : 0.0101 0.709 + 0.0099 0.692 = 0.0100
0.697 0.0101
0.810
0.0100
Table 2: Real-data regression performance (test R2, mean + s.e. over CV splits).
Daraset
RF
RotF
CCF
SPORF
XGB
PCA+RF
IDA+RF
JARP
bike-sharing
0,780 0.010
0.790 + 0.010
0,800 = 0.010
0,810 + 0.011 0,820 = 0.010 0.770 = 0,010
0.780 : 0,010
0,800
0.011
califomiz-huusing
0.012
0.012
0.720
0.012
0.730 4:
0.740
0.690
0.700
0.012
energy
0.800 = 0.009
0,890
+ 0.009
0.900
-: 0.009
0.900 1 0.009
0910
2: 0.009
0.800
1: 0.009
0880
1: 0.099
0.930
0.010
kinSem
0,850
0.008
0.890
4: 0.008
0.890
4: 0.008
0.900 1: 0.009
0.900
0.008
0.870
0.008
0.880 0.008
0,920
0.009
protein
0:111
0.650
4: 0.011 0.660
4: 0.011 0.670
+: 0.012 0.600
2 0.011
0.630
+: 0.011
0.440
0.011
0.710
0.012
s.c.
0.010
0.010
0.794
0,010
0.802 + 0.011 0,810 0.010
0.766
+ 0.000
0.776 0:010
0.011
matrix Ho from that surrogate, and (iii) fitting the final RF on the transformed data X H. Figure 3 reports the sum of (ii). The median training time of JARF is about 1.67 x that of vanilla RF, while it remains much faster than oblique baselines that solve optimization problems at every node (RotF: 60 s. CCF: 44 s, etc.). This efficiency gain is critical for practical deployment, as JARF achieves/ surpasses oblique forest accuracy at near-RF speeds. The EJOP preconditioner amortizes well across all trees in the forest, whereas per-node oblique methods like CCF and RotF incur repeated computational costs that scale with forest size.
5.4 MECHANISM ANALYSIS: DO EJOP DIRECTIONS MATCH OBLIQUE SPLIT NORMALS?
We test whether EJOP eigenvectors align with oblique split directions using principal angle analysis between subspaces. For each dataset and fold, we first compute the EJOP estimate Hy on the training data and take its eigendecomposition Ho = UAUT with eigenvectors U [ui...., ud). We then train each oblique method and extract a unit split normal ñ E Rd at every internal node.
For each node, we quantify alignment with the EJOP top-k subspace using the principal-angle co- sine:
sx (n) = U 1 E [0, 1].
which equals Ju, ni when l = and reaches 1 if and only if ñ E span(U)). We aggregate Sk across nodes and folds to obtain a par-dataset distribution for each oblique method. Figure 4 reports our results.
5.5 ABLATION STUDIES
To understand the contribution of each design choice in JARF, we conduct systematic ablations by modifying individual components while keeping all other settings fixed. Table 2 ablations reveal a clear hierarchy of component importance. Removing the EJOP transform entirely (Identity: H I) produces the largest performance drop (Ax = -0.036, p < 0.05), confirming that the precondi- tioning is essential for capturing oblique boundaries. Sample size for EJOP estimation shows pected behavior, with performance degrading gracefully from full data (m = n) to half (m = 0.5m, AK -0.004) but dropping significantly at 0.1n (Ax 00.016, p 0.05).
Among the finer implementation details, centered differences outperform forward differences (An. -0.008 vs. -0.011 with clipping), and the adaptive per-feature step size Ej a
8

---

Published as a conference paper at ICLR 2026
Effect size relative RF (15 classification datasets)
0.00
MA) = x(F))
-0.05
-0.15
RotF
CCF
SPORF
XGB
PCA+RF LDA+RF JARF
Figure 2: Beeswarm of effect size relative RF on real data. Each marker is one dataset in the
15-task suite. The vertical axis shows the per-dataset effect size A(A) = <(RF) - -(A); the dashed line marks parity with RF (A=0). Points below the line indicate the method outperforms RF JARF produces mostly negative deltas and achieves the best overall rank in Table while oblique baselines (RotF, CCF, SPORF) show mixed but generally favorable improvements over RF
Training time
70
60
50
40
30
20
RotF
CCF
SPORF
XGB
JARF
Figure 3: Comparison of median training times the 20 real-data tasks. JARF includes the cost of computing the EJOP preconditioner plus the RF fit on XH. Measured times: RF 15s, JARF 25 s. RotF- 60 s. CCF 44 s. SPORF 45 s, XGB 43 JARF adds ~10s over RF (= 1.67x RF cost) yet remains faster than per-node oblique forests.
MAD(X.,)000 /0.6745 with a = 0.1 balances bias and variance better than both smaller (a = 0.05, Ak -0.009) and larger (a 0.2, As -0.013) values. Including categorical features via one-hot encoding slightly hurts performance (Ax -0.006), possibly due to noise in discrete gradient estimates, while numerical stability measures (regularization 7/d and normalization) have minimal impact on accuracy (Ax -0.005) but improve conditioning. Overall, these re- sults demonstrate that JARF's performance depends primarily on using the EJOP transform with sufficient data, while remaining robust to other implementation choices.
CONCLUSION
In this work, we introduced JARF (Jacobian Aligned Random Forests), a simple yet effective ap- proach that bridges the gap between the computational efficiency of axis-aligned decision forests and the expressive power of oblique methods. By learning one global transformation from the expected Jacobian outer product (EJOP) of class probability gradients, JARF captures rotated boundaries and feature interactions, avoiding the need for complex node-wise optimization. Our experimental sults demonstrate that JARF consistently matches or surpasses the accuracy of oblique forest meth- ods while maintaining the simplicity, speed, and robustness that make Random Forests attractive for practitioners.
We aeknowledge important limitations of our approach. First, the supervised rotation relies prob- seliit--rrddent estimates from a random forest; those estimates are noisy or poorly calibrated, the resulting transform misalign with the true decision geometry and even degrade accuracy. Sec-

---

Published as a conference paper at ICLR 2026
Alignment growth with
0.30
Median alignmentt 1U5
0.25
0.10
0.05
Rot
CEF
SPORT
0.00
Subspace size
Figure 4: Alignment growth with EJOP subspace size. Median sp = | All3 versus k for RoLF/CCF/SPORF. Alignment rises rapidly, indicating that oblique split normals concentrate in low-dimensional EJOP subspace. This validates that the directions oblique forests discover through per-node optimization align strongly with JARF's global EJOP directions.
Table 3: Performance impact of ablating JARF components. Values show differences from default JARF (variant minus default) for Cohen's ' s, macro-F1, accuracy, and training time averaged across datasets. f denotes p < 0.05 (Wilcoxon signed-rand rank with Holm correction).
Variant
Ak
AMacro-Fl
AAcc
ATime (s)
JARF (default)
0.000
0.000
0.000
0.00
Identity (H=I)
-0.036
-0.031
-0,015t
-0.42
FD: forward (vs. centered)
-0.008
-0.007
-0.004
-0.06
FD: no clipping
-0.011
0.010
-0.006
-0.04
Step: fixed global e
-0.014
-0.02
-0.007
-0.02
Step: a=0.05
-0.009
-0.008
-0.004
-0.01
Step: a=0.2
-0,013*
-0.111
-0,006
-0.01
Subsample m=0.1n
-0.016*
-0.013t
-0.007
-1.20
Subsample m=0.5n
-0.004
-0.003
-0.002
-0.40
Categoricals: include toggles
-0.006
-0.006
-0.003
+0.05
No Laa
-0.005
-0.004
-0.002
0.00
No trace normalization
-0.004
-0.004
-0.002
+0.01
ond, while JARF is markedly faster than per-node obliques, it still incurs preprocessing overhead from infte-difference probing and forming Ho that vanilla axis akisnel forests avoid, which may be non-trivial in some settings.
REPRODUCIBILITY STATEMENT
We took several steps to make our results reproducible. The model and training procedure are fully specified in the appendix. Formal assumptions and complete proofs of the statements rely appear in Appx. Appx. [A Implementation details covering software versions, hyperparame- ter grids, CV protocols, timing methodology, and configuration choices shared across methods documented in Appx.
ACKNOWLEDGEMENTS
I thank my master's advisor, Prof. Eliza O' O'Reilly (Johns Hopkins University), for many helpful discussions and for detailed feedhack the presentation of this work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Leo Breiman. Random forests.
Machine Learning, 45(1):5-32, 2001.
doi: 10.1023/A:
1010933404324.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 16), pp. 785-794, San Francisco, CA, USA, 2016. ACM. doi: 10. 10.1145293997222999998 URL
https:
R. Dennis Cook. Save: A method for dimension reduction and graphics in regression. Com- munications in Statistics Theory and Methods, 2909-10)2109-2121, 2000. doi: 10.1080/
03610920008832598..
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):
179-188, 1936.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outper- form deep learning on tabular data? arXiv preprint arXiv:2207.088/55 2022. URL https:
//axxiv.org/aaaaa
Rakesh Katuwal, Ponnuthurai N. Suganthan, and Le Zhang. Heterogeneous oblique random forest. Pattern Recognition, 99:107078, 2020. doi: (a001//j.patcog22000
Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316 327, 1991. doi: 10. 00.10800162155911999. 19911495999911.755533
Björn H. Menze, Michael B. Kelm, Nico Splitthoff, Ullrich Koethe, and Fred A. Hamprecht. On oblique random forests. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2011), Part II, pp- 453-469. Springer, 2011.
Sreerama K. Murthy, Simon Kasif, and Steven Salzberg. A system for induction of oblique decision trees. Journal of Artificial Intelligence Research, 2:1-32, 1994. URL /ttp::tt//////tt hwwpw//wwww...... jair. orq/index. php/
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale image classification. In Kostas Danillidis, Petros Maragos, and Nikos Paragios (eds.), Computer Vision ECCV 2010, volume 6314 of Lecture Notes in Computer Science, pp. 143-156, Berlin, Heidelberg, 2010. Springer. doi: 10110077978-33-62-15511----
Tom Rainforth and Frank Wood, Canonical correlation forests. arXiv preprint 1xXXiv1507.0555 2015. UtLpttps: wwwaaaii..rrgaab//aS/0.
C. Radhakrishna Rao. The utilization of multiple measurements in problems of classification. Jour- nal of the Royal Statistical Society. Series B (Methodological), 10(2):159-20 203, 1948.
Juan J. Rodríguez, Ludmila I. Kuncheva, and Carlos J. Alonso. Rotation forest: A new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10): doi: 10.1109/PPAM..2022111
1619-1630, 2006.
Ilya M. Sobol' and Sergei Kucherenko. Derivative based global sensitivity measures and their link with global sensitivity indices. Mathematics and Computers in Simulation, 79010)73009-3017 2009. doi: m0.1016f..atcom.20000000
Tyler M. Tomita, James Browne, Cencheng Shen, Jaewon Chung, Jesse L. Patsolic, Benjamin Falk, Carey F. Priehe, Jason Yim, Randal Burns, Mauro Maggioni, and Joshua T. Vogelstein. Sparse projection oblique randomer forests. Journal of Machine Learning Research, 21(104):1-39, 2020. URL http::////....oorr/ppppsss//////
664 .tml
Shubhendu Trivedi and Jialei Wang- The expected jacobian outerproduct: Theory and empirics.
arXiv preprint arXiv:2006.035:5 035550 2020.
www.rxiiv.oog/abba wttp://...rr//aa///////./// 03550
Shubhendu Trivedi. Jialei Wang, Samory Kpotufe, and Gregory Shakhnarovich. A consistent es- timator of the expected gradient outerproduct. In Proceedings of the 30th Conference Un- certainty in Artificial Intelligence (UAi 2014), pp. 819-828, Quebec City, Canada, 2014. AUAI Press.
11

---

Published as a conference paper at ICLR 2026
A ANALYSIS AND ADDITIONAL EVALUATION DETAILS
A.1 WHY EJOP PRECONDITIONING HELPS AXIS ALIGNED TREES
Here give explanation of why the EJOP matrix is natural preconditioner for axis aligned forest. Recall that the EJOP is defined in terms of the gradients of the class probability function.
Let Rd
A be the population conditional class probability function,
fa(r) = P(Y = X =
c= c=1,.... I.....,,C
and let
Jx(x) (ee())))..V
be the d x C Jacobian matrix that collects the gradients of all class probabilities. The population EJOP
Ho = (X)))
(Eq. equation 1. Throughout this subsection we assume that each coordinate function fc is C3 on compact subsets of Rd meaning that it has three continuous derivatives and the third derivatives are bounded. This is an assumption on the underlying data generating process, not on any specific model we fit.
In the algorithm we never observe f directly. Instead we fit a surrogate probability model f (in our case a random forest) and form plug in estimate of Ho by replacing / with f in the definitions of J and Hg- Under standard consistency assumptions on f. the resulting matrix Ho(f) converges to the population quantity Hf( as the sample size grows. So the geometric picture below should be read as describing the ideal population behavior that JARF is trying to approximate, even though itself is piecewise constant.
Our method constructs an empirical EJOP matrix Ho from data and then uses Ā io as a single global linear preconditioner. The final forest trained on the transformed features X म.
Axis aligned oblique splits. We will use the following terminology. A split is axis aligned if it tests a single feature, e.g. 2j S T. split is oblique if tests linear combination of features, e.g. IRd not equal to any coordinate vector eg-
T x < T with w
Proposition A.1 (Axis aligned versus shared oblique). Let H be any positive semidefinite (psd) matrix, i.c a symmetric matrix with nonnegative eigenvalues, and let j be a feature index. Then the axis aligned split
{o: (x H); S ST}
the same set as the oblique half space
1x Hej S ST}
in the original coordinates.
Proof. We have (x H), - H) = x Hej, so the two sets coincide.
Proposition A.2 (First order impurity gain and EJOP). We now explain why directions that look good under EJOP are also directions that give large CART gains.
Consider binary classification (C 2) with squared loss CART. Let u E gd-1 define a split of the form u < T. Look at thin slab around the candidate threshold,
{x: |u x 7| $}
for small 0, and approximate f in that slab by its first order Taylor expansion,
f(x) ~ f(() + ()(E) (x E) with u T т.
Then the expected impurity decrease of the best threshold along is, up to positive factor that does not depend on u, proportional to
Hou Ex [x (((()))
12

---

Published as a conference paper at ICLR 2026
So if we move in direction and the class probabilities f(z) change quickly average. then CART sees larger gain along that direction. For binary classification this shows that the expected first-order gain along direction is proportional to E [(u (
Corollary A.3 (What happens when we use 7 Ro). By Proposition A.I, an axis aligned split on the transformed features (XH) H) with index j is the same as a split in the original x space with normal
J Hez,
where is the jth standard basis vector in Rd. In other words, splitting on the jth coordinate after the linear map H corresponds to oblique split along u, in the original coordinates.
By Proposition A.2, the expected first order CART gain a split with normal u is proportional to u' Hnu. Plugging in u = Mj He, gives that the first order score of the jth coordinate split in the preconditioned space is proportional
u Houz =ej H HoHle ej.
If we choose A R Ho, then coordinates j for which the induced normal Vy has large EJOP score u Ho are amplified by the preconditioner. This biases the forest toward splitting along directions where the class probabilities change the most, while the training procedure itself remains exactly the same as for standard random forest.
A.2 CONCENTRATION AND CONSISTENCY OF THE EJOP ESTIMATOR
In this subsection we study when the empirical EJOP matrix Ho concentrates around its population counterpart Hn- As before, let f R Ac-1 denote the population conditional class probability function,
fe(x) = P(Y X x),
c= L.,,..,C.
Assumptions.
(A1) (Smoothness and bounded gradients.) Each coordinate fa(x) Pr(V = X x)) is C on the support of Px. Moreover, the gradient is uniformly bounded,
| f+ M,
and all third order directional derivatives are bounded in magnitude by a constant By. (A2) (Finite differences.) To estimate gradients we use finite differences with step size and m probe points. We let the step size shrink and the number of probes grow so that
m 00, and me? o0.
Intuitively, + 0 controls the bias the finite difference approximation, while me² + oo keeps the variance under control.
(A3) (Consistency of the surrogate probabilities) If we use probability weights, the surrogate probabilities are uniformly consistent:
supl p(c x) - p(c x)) S Tms
Tnn + 0 as m + co.
Here p(c x) is the true conditional probability and p(c (r) the estimate produced by the surtogate model.
Lemma A.4 (FD gradient bias). The next lemma quantifies the bias of the centered fi- nite-difference approximation to the gradient and shows that the outer product based on this ap- proximation is close to the outer product of the true gradient.
Fix class Let Se Rd R be O in neighborhood of and assume all third directional derivatives along the coordinate axes are bounded there:
sup fa)))( s By for every coordinate j.
13

---

Published as a conference paper at ICLR 2026
Define the centered finite difference (FD) of the jth partial derivative at with step size 0 by
fdx + 5es) fdæ 5c,)
91 (2;cc)
E
Then for each coordinate j.
B3
FD [z) c) - -0ff((x)
2²
(< Bg E²)
24
Consequently, if [V M and (c) is the vector with entries 9 (x; c), then
Bavd
B E²
FD (c) (c)
112 S
M
12
576
Proof. Fix a coordinate j and consider
g(t) == fe(2 + tee))
teR.
The centered FD estimator is
(x; g(h) -h)
with h E/2.
2h
By Taylor's theorem with Lagrange remainder applied around 0, we have
g(h) g(0) + hg'(0) h/ g0)
1.2 (3) (E+),
6 g
g-hh) g(0) hg) + 42 (0) ()))
for some E+ E (0, h) and ç- (-h, 0). Subtracting the two expansions and dividing by 2h gives
g(h) g( (-hh) g0(0) + h²a
12 (g ( (E+) ()+)
(E-)).
2h
By construction g'(0) = fo(() and (t) = 07 felx + te,). Therefore the FD estimator error can be written as
1²2
FD -x; 9
+ f0
12
Using the bound (8) < B3 for all = yields
h2
Ba
B3 24 E2,
[z -0ff(x)) < 12 (B B3) = n2
6
which proves the claimed ((=2) bias bound (and the looser Ba version follows since 24
1
< }
Now let GP (c) be the vector of FD approximations and write it as
(c) fz) +
8, = FD -0,f(())f((())
From the scalar bound above obtain
Ba
B3 vd
|8|1 S
2
|00|/ S
24
24
GFD (C) (c) VI((() (VI.((=) + V++
=V/c(x) 8T + (V)((x) +
Using the fact that | T ||2 || and submultiplicatiiityy of the spectral norm, obtain
|| GFP (c) Vf((z) 11, s 2l |0|| + 1|612.
Using the bounds | ff(x))] < M and ||ō1|2 <
B.
Vd now gives
24
B
BBd 4
2 [
2
|| <
12
576
and combining the terms yields the stated deviation bound.
14

---

Published as a conference paper at ICLR 2026
Lemma A.5 (Weight approximation error). The next lemma shows how enrors in the estimated class probabilities translate into an error in the weighted sum of gradient outer products.
Fix point and suppose the estimated class probabiliities p(c 2) are uniformly close the true probabilities p(c x) in the sense that there is a number 7m 2 0 with
|p(c
p(c .>) S Tm for all E (I....,C).
Assume moreover that each class probability function has bounded gradient at that [VVd(())l2 S M for all e. Then
(p(c x) p() Vfd(z) Vf(l)) Σ
S mm
[Vf(x)]1 S MM
where || ll2 is the spectral norm (largest singular value).
Proof. Set
ae := x)) - plc x),
Ue = Vfc(o).
Then the matrix want bound can be written
Gg ue
We use two facts about the spectral norm - ||:: it is subadditive (triangle inequality) and for rank-one matrix uu" T we have |uuu" inequality gives
1/2 = || (its only nonzero eigenvalue). Applying the triangle
Σ ae ucul
Σ c=1
ae ucu 112
lael uo
ll
Σ ac | 122 c=1
By assumption |acl S 7m for every so
C
a Ue
|luel
Finally, the gradient bound ||uell2 = V fo fz(x)]12 S M implies
C
2 Σ
= CM
cml
which yields
Σ (plc) ( pp(cppcc)) Vf(x))
CM
Didension-adapted risk guarantees. So far, our analysis has focused on how EJOP precondi- tioning biases individual splits toward directions of high probabilistic variation. We show that, in a simple but representative setting, this geometric bias also leads to a dimension-adapted risk guarantee. Specifically, when the conditional mean f(x) Ey X x] depends only an d-dimensional linear subspace of Rd, JARF achieves a rate that depends on the intrinsic EJOP rank rather than the ambient dimension d.
We consider a regression setting with ridge-structurre regression function f(x) = 9UU 2). where U E TV. dxr has orthonormal columns and g R² > is Lipschitz. this case, the EJOP matrix Ho = (X))( has rank r and its range equals the span of U. If JARF estimates Ho consistently and projects onto the top eigenvectors of Á, then standard axis akis-aligned forest those projected features behaves like nonparametric regressor in R², up to the error of estimating the subspace. The following theorem formalizes this intuition.
15

---

Published as a conference paper at ICLR 2026
Theorem Dimension-adapterr risk bound for JARF). Let (X;, Yt); be i.i.d samples with X; Rd and Y E R, where X has compact support and Y = f(X) + with RBg X] 0 and ElS"] o². Assume
f(z) = g(U x);
for some orthonormal U RRd and a function RR" + R that L-Lipschitz on the projected support. Let
Ho (==EVI((XVV((X)
and suppose rank(Ho) with a spectral gap Ar (H)) 2 Amin 0. Let ft be the EJOP estimator constructed by JARF using surrogate forest and finite differences, and suppose that for some sequence - 0,
|| - Hollop S En with probability least 1 ōn-
Define Ú E RRdx as the matrix of top r eigenvectors of H, let Z, = ÚTX, E RT, and let fn be regression forest trained on (Zz, YiYi=1 with tree depth and leaf size chosen as in standard consistency results for forests in dimensions. Then there exist constants C1, C2 > O. independent of d, such that
[[(a(X)) - -f(x)) S Cin -àr + C2 C2 + o(1),
where the expectation is over the training sample and a fresh test point X.
In particular, when e. sufficiently fast, JARF attains the usual nonparametric rate in dimension up to negligible terms, even though the data live in Rdd.
This result shows that JARF is not only geometric heuristic: under a low-rank EJOP structure, provably adapts the intrinsic EJOP rank and achieves risk bound that is independent of the ambient dimension d. Existing EJOP-based methods analyze kernel and linear models; to the best of our knowledge, Theorem is the first result that links EJOP geometry to the sample complexity of tree ensembles.
A.3 CAMENSION-ADAFTER RISK BOUNDS FOR JARF
We now describe a simple setting in which JARF enjoys a risk bound that depends on the intrinsic EJOP rank rather than the ambient dimension. Throughout this section consider regression model with squared loss.
Setup and assumptions Let (X,, Y)) be i.i.d. samples with X, e Rd and Y; E R. We assume:
(A1) Ridge-structured regression function. There exists an orthonormal matrix U E Rd with r < d and function g R -> R such that
f(x)
E[ x= s(UTx)) x).
We write 7. UT X IR" for the intrinsic representation.
(A2) Regularity. The support of X is contained in compact set X' C Rd with || < R for all
x E X. The function is T.-Lipschhtz on UT X, and the noise satisfies Y = f(X) + with [ X]= 0 and [F(") $ o².
(A3) EJOP structure. Let
Ho = [X(((()V
We assume rank(Hto) and that there is spectral gap X, (Ho) Amin 0 between the r-th and (r + 1)-st eigenvalues.
(A4) EJOP estimation. Let A be the EJOP estimator used by JARF, constructed from surrogate forests and finite differences as in the previous sections. Under Assumptions (A1)-(A3) and the fnite-difference analysis of Lemmas A.4 and A.5, there exists sequence en - 0 and failure probability 8n + such that
i Hollop s en. with probability at least 8n.
(5)
16

---

Published as a conference paper at ICLR 2026
(A5) Regressor consistency in fixed dimension. Let Ú E Rd be the matrix of top eigenvectors of H and define projected features Zi Ü X, E R' and 7 U X. Let
m E Z 7= 2]
denote the regression function in the projected space, and assume m& is Lz-Lisschitz on the support of Z (for some constant Lg that does not depend on or ni). Let In be the regression estimator used by JARF, trained on (Z,, Yi) (in the experiments this is an axis-aligned random forest). We assume that there exists constant C₁ such that
E ( m < Cin o(1),
(6)
for every realization of Ó with orthonormal columns. Assumption equation 6 holds for variety of nonparametric regressors fixed dimension r; we use forests for concreteness.
All expectations below are taken with respect to the training sample, fresh test point X, and any internal randommess of the estimator.
A.4 EJOP IDENTIFIES THE INTRINSIC SUBSPACE
Under the ridge model (A1), the EJOP matrix Ho has range equal to the span of U.
Lemma 2. Under (Af) and (A2), we have
ff(x) UVI(UTa, x),
and consequently
Ho [[VX(()VVX())) U -
In particular, (z is invertible, then rank(Ho) and raage(HH) span(U) (U).
Proof. By the chain rule, for any IRd
fxx) (o(U ()) UV
UV w)),
since U R" and has orthonormal columns. Substituting into the definition of Ho gives
Ho = [V=[Vv((""V - U=(V
If the inner x matrix is invertible, then Ho has rank r and its range equals the span of U.
A.5 SUBSPACE PERTURBATION AND PROJECTION ERROR
The next lemma is a standard Davis-Kadd Kahan type result for the top-r eigenspace of a symmetric matrix.
Lemma (Subspace perturbation). Let Hy and á be symmetric matrices satisfying | -H-Holen and let (Ho) Z Amin > 0 be separated by gap from the rest of the spectrum. Let P and P be the orthogonal projectors onto the top-r eigenspaces Hu and H, respectively. Then there exists constant >> O such that
Hol S
- PPPlop S C
Aunin
Proof. This is standard consequence of the Davis-Kahai sin-O theorem; see, for example, any modern text on matrix perturbation theory.
In setting, Lemma 2 limlies that P T is the orthogonal projector onto the intrinsic EJOP subspace, while P UÚT the projector onto its empirical estimate. Combining Lemma Lemma 3 and the EJOP consistency equation we obtain
U - TUT llop
< C Sn
with probability at least 1 --,,.
Amnin
We now bound the enror incurred by replacing the true EJOP subspace with its estimate when eval- uating
17

---

Published as a conference paper at ICLR 2026
Lemma 4 (Projection error). Under (AI)-(A3), (A4), and equation 7 we have
f(F)) - LR||P Pllop for all a E X. Consequently, there exists constant C" depending on L, R, and Amin such that
-
C² o(1)
Proof. Since f(x)= [(U x), we can write
f(x) - s(UTx) and 1(Px) - g(U Px).
Using the Lipschitz property of and the fact that U has orthonormal columns,
If(z) - f(Px)) = lo(U x)- g(U s L LIUTT x- U -TPxx = L LITTTE (1 P)-|-
Since UT= U P and P UUT, have
UT - UU (P-(PPP)))
and hence
| (T - P) < | |bp || Pllop |,/|| S |
P|lop ||0|||
because ||UT|lop 1. Using ||||||
I R for x X.
If(x) f(2z) - LR I Pllop
Squaring and taking expectations, then substituting
PP - Pllop S Can from equation yields
-[ s C2 +
so we can take C' - L2
A.6 PROOF OF THEOREM 1
We now prove the dimension-adapted risk bound stated in the main text.
Theorem (Theorem]) [ restated). Under assumptions (A1)-A5), there exist constants C1, C2 > 0, independent of such that
[[(,(() S Cin + C2 o(1).
Proof. Recall that fn depends on X only through the projected features Z Ô X, so we may write (x) = (Z).
Let m²(2) (z) = E[| Z =| denote the regression function in the projected space. Using the
inequality (a 6)2 s 2(a - -c) + 2(b <)² with a (z) f(X), and mm (Z), we obtain
E (f)((X - ((x)) = E (((((Z)) - f
$ (f mo m (2))²) + 2E (m f(x))²
27 + 2T2.
Bounding Ti (estimation in dimensions). By the tower property and Assumption equation 6
T = E[B((f(( - mz) t]] < E( + + o(1) = Cin x + o(1),
where the o(1) term does not depend d.
18

---

Published as a conference paper at ICLR 2026
Table 4: Effect of EJOP subsample size on classification performance. Mean Cohen's k over 15 datasets for different subsample sizes m used to estimate EJOP. The RF bascline has mean K 0.704.
EJOP subsample size m
Mean (JARF)
Mean (RF)
Gain over RF (abs)
Gain over RF (rel %)
1k
0.795
0.704
0.091
12.93
2.5k 5k
0.805
0.704
0.101
14.35
0,809
0.704
0.105
14.91
10k
0.810
0.704
0.106
15.06
n
0.810
0.704
0.106
15.06
Bounding T2 (approximation error from using the projected o-algebra). By the definition of conditional expectation, (Z) is the I2-projection of f(X) onto the o-algebra generated by Z, so for any measurable function h of we have
[([(X) mz())) S (
In particular, take (Z) f(PX), which is measurable with respect to since Px U0 X
a deterministic function of - T X. Then
T- = =2=1
(m f(x)) I ([(((x))
By Lemma 4 the right-hand side is at most C² 2 + o(1) for some constant C" depending only L, R, and Amin, and hence
T3 S C' + o().
Combining the bounds.
Putting the pieces together,
E ( 2Cin -4. + 20's o(1. Absorbing constants into C1 and C2 gives the claimed bound.
B ADDITIONAL EXPERIMENTS AND CLARIFICATIONS
B.1 ON THE COMPUTATIONAL COST OF JARF.
Our efficiency analysis already includes the cost of the first RF. JARF training has three components: (1) we fit a surrogate RF with 50 trees that is used only to estimate class probabilities 7zx);; (2) we compute the EJOP preconditioner Ho from this surrogate using finite differences on a subsample of size m < min(10,000, n); and (3) we train the final RF with 200 trees on the transformed data X Ho. Figure 3 reports the wall clock time for the sum of (1) + (2) (3). Because the surrogate forest is four times smaller than the baseline RF and the EJOP step operates on at most 10k points, the extra work is significantly less than training a second full RF. Across the 20 tasks in Figure we observe median training times of 15 for RF versus 25 for JARF, corresponding to roughly 1.6-1.7x x overhead, while JARF remains much faster than por-node oblique forests.
B.2 ON THE EFFECT OF THE EJOP SUBSAMPLE SIZE.
We consider m E {1k, 2.5k, 5k, 10k, n} with m < n. The results in Table 4 show that JARF is quite robust to this choice. Even with m = 1k, which typically uses less than 10% of the available data, JARF already recovers about 85% of the gain over RF obtained with the largest subsam- ple. Increasing m from 2.5k to 10k changes the mean k by at most 0.005, while the EJOP cost grows roughly linearly in m. For all values m > 2.5k, JARF continues to match the best oblique baseline on the majority of datasets, and the configuration used in our main tables (m min(10k, n)) líes in this saturated regime. This supports our claim that JARF achieves strong performance without requiring EJOP computation on the full dataset and that its gains over RF and per node oblique forests not sensitive to the exact subsample size.
outperform
19

---

Published as a conference paper at ICLR 2026
Table 5: Baselines vs, geometric variants (means over all 15 classification and 5 regression datasets).
Method
Mean x: (15 cls)
Mean R2 (5 reg)
Median train time (s)
RF
0.704 0.709
0.776
15
XGB
0.810
43
JARF (RF on X
0.810
0.836
25
JARF-XGB on XII
0.815
0.842
45
B.3 ON DATASET SELECTION.
Our goal was to study JARF on a broad, realistic set of tabular problems where tree ensembles are commonly used. To construct the benchmark, we started from widely used OpenML UCI tabular datasets that appear in earlier work on random forests and oblique forests, and then applied simple, priori filters: (i) supervised classification or regression with tabular features; (ii) at least few thousand training points so that EJOP estimation meaningful; (iii) mix of low- and high- dimensional problems, and of balanced and moderately imbalanced label distributions; and (iv) no heavy preprocessing or manual feature engineering beyond standard normalization / encoding. We did not drop any dataset based on JARF's performance, and we kept the same pool for all methods and ablations. Several of these tasks overlap with standard suites such PMLB/TabAren
B.4 ON APPLYING JARF BEYOND RANDOM FORESTS (E.G., TO XGBOOST).
JARF itself does not rely on any forest-specific property: the surrogate model only appears through the EJOP/EGOP preconditioner and any downstream learner can be trained on the transformed fea- tures X. In the paper we instantiate this with a random forest on XH, but the same construction applies to boosted trees, linear models, or small neural networks. To make this explicit, we also con- sider a JARF-XGB variant that uses the same EJOP H and then trains XGBoost on X H instead of on the raw features. As summarized in Table JARF consistently improves over its base learner (RF or XGBoost) on both classification (mean Cohen's k) and regression (mean test R²), while adding only modest training-timm time overhead compared to RF and remaining competitive with the oblique baselines. This demonstrates that JARF a general geometric preconditioner that can be paired with different prediction models rather than a method tied to particular forest implementation.
B.5 ELECTRICITY, MAGIC, AND LETTER DATASETS.
We focused on electricity, magic, and letter because, in our benchmark suite, these are precisely the datasets where simple axis-aligned RF struggles the most and tilige/ffeatur-eennnneee gain the largest advantage. They are moderately high-dimensional. multiclass problems with rich, nontrivial decision structure: all three, the best oblique baselines (CCF, SPORF, XGB) are clearly ahcad of plain RF, indicating that the Bayes decision boundary is far from axis-aligned. As Table shows in the paper, JARF largely closes this gap: these "hard" datasets either matches passes the strongest oblique baselines while dramatically outperforming standard RF. We therefore use electricity, magic, and letter targeted stress tests for the proposed geometry in regimes where a single global transform would a priori be most likely to fail.
methods
B.6 ON SHARING A SINGLE GLOBAL TRANSFORM Í.
To test whether sharing one Á reduces useful diversity, we implemented a per-tree variant where each tree receives its own EJOP transform Al computed from that tree's bootstrap sample. We retrained JARF with this scheme all 15 classification datasets and compared it to the original single-H H version. As summarized in Table the per-tree EJOP variant slightly degrades test per- formance (mean Cohen's 's 0.808 vs 0.810) while being almost twice as slow to train (about 3.0x RF vs 1.7x RF), and it is best or within 0.01 the best forest on 13 datasets compared to 14 for the single- version. This suggests that single global A is sufficient in practice: per-tree EIOP estimates noisier because they are fit on fewer samples, while diversity in JARF is already
20

---

Published as a conference paper at ICLR 2026
Table Single global EJOP transform vs. per-tree EJOP transforms, averaged over 15 classifi- cation datasets.
Method
Mean test
Std. K
Mean train time (x RF)
datasets best or within 0.01 of best
JARF (single H) JARF (per-tree H²)
0.810
0.010
1.7
14
0.808
0.011
3.0
13
Table 7: Sensitivity of JARF to surrogate RF quality, averaged over all 15 classification datasets.
Variant
RF acc
RF ECE
JARF acc
JARF (k)
shallow, uncalibrated
0.81
0.18
0.872
0.808
default, uncalibrated
0.84
0.11
0.874
0.810
deep, uncalibrated
0.86
0.08
0.875
0,811
deep + Platt scaling
0.86
0.05
0.875
0.812
deep isotonic regression
0.87
0.04
0.874
0.809
Std. dev. over variants
0.024
0.056
0.0013
0.0014
Max. difference vs default
0.03
0.07
0.002
0.002
provided by feature subsampling, bootstrap sampling, and different thresholds rather than requiring separate transforms per tree.
B.7 DEPENDENCE ON SURROGATE RF QUALITY.
To test whether JARF is brittle to the surrogate RF, we swept the surrogate over wide range of hyperparameters and calibration schemes, then recomputed EJOP/EGOP and retrained JARF each time (details in the appendix). Across 15 classification and 5 regression datasets, our hyperparam- eter and calibration sweeps cause large changes surrogate accuracy, ECE, and R but JARF's test к and R² move only at the 10 level (Tables 7 g). showing that it needs only reasonably accurate surrogate and highly insensitive to precise probability or regression estimates. Because EJOP/EGOP is built from averaged gradient outer products, it is driven mainly by the coarse geom- ctry of the surrogate's 's decision function rather than fine details of its probability values. Changing calibration mostly rescales these gradients instead of rotating their dominant directions.
B.8 CHOICE OF SURROGATE FAMILY (RF VS. XGBOOST / NN).
On all 15 classification datasets, we recomputed EJOP using three surrogates (RF, XGBoost, and two-layer ReLU MLP) and retrained our algorithm in each case. As shown in Table g surrogate accuracy and ECE differ noticeably across families, but JARF's test accuracy and x change only at the 10 level. The top EJOP eigenspaces also nearly coincide (mean cosine alignment of the top 10 eigenvectors > 0.97), indicating that once the surrogate is reasonably accurate, JARF is effectively insensitive the specific surrogate model class. RF, XGBoost, and small MLP are all trained to approximate the same conditional mean, so once they reach similar accuracy they produce very similar gradient fields and thus nearly identical EJOP eigenvectors. This explains why surrogate accuracy and ECE can differ across model families while JARF's accuracy and remain almost constant.
We also prove in the appendix that if the surrogate's Jacobian is uniformly within of the true conditional mean's Jacobian, then the EJOP matrices differ by O(e) in operator norm and their top-k eigenspaces differ by O for eigengap 7
Finally, across our 15 classification and 5 regression datasets, we do not observe regimes where JARF clearly underperforms CCF SPORE
21

---

Published as a conference paper at ICLR 2026
Table Sensitivity of JARF surrogate RF quality on the 5 regression datasets.
Variant
RF (R²)
RF RMSE
JARF (R²)
shallow surrogate
0.76
0.62
0.835
default surrogate
0.78
0,60
0,836
deep surrogate
0.80
0.58
0.837
deep + extra trees
0.82
0.57
0.836
Std. dev. over variants
0,022
0,019
0.0007
Max. difference vs default
0.040
0.030
0.001
Table 9: Effect of surrogate family on EJOP and JARF performance, averaged over all 15 classification datasets.
Mean cos. alignment with RF EJOP
Surrogate family
Surrogate acc
Surrogate ECE
JARF acc
JARF (x)
(top- 10 cigvecs)
RF (default)
0.84
0.11
0.874
0.810
1.00
XGBoost
0.87
0.04
0,876
0.812
0.98
2-layer MLP
0.86
0.06
0.873
0.809
0.97
Std. dev. across surrogates
0.013
0.029
0.0012
0.0014 0.004
0.013
Max diff RF surrogate
0.03
0.07
0.002
0.03
C REPRODUCIBILITY AND IMPLEMENTATION DETAILS
Code and artifacts. We provide self-contained Google drive with scripts to download datasets and run experiments httts::/drive. www.gl.iieeeoooeeeeoooooooooooee htwm//iee/d/dadyssrieeaaa/ BEBvd01TOo M9MP 2.MMB//view2usppshar....
SciPy 1.11; scikit-learn 1.4; LightGBM 4.3; CatBoost 1.2; pandas 2.2; joblib 1.3. Experiments ran on a 16-core CPU machine (no GPU used). To reduce nondeterminism across BLAS/OpenMP, we set PXTHONHASHSEED-O, CMP NNMM.THEEDSS-1. MKL NUM THREADS-1. and pass random state=seed to learners.
Environment. Python 3.11; NumPy 1.26;
Dataset summary and benchmark construction To make the experimental setup fully trans- parent and reproducible, we include Table [0 which lists for every dataset our benchmark the number of samples number of raw input features d, task type, and original source. Counts refer to the number of rows and input features before train/validationntest splits and before any one-hot encoding of categorical variables.
Our goal was to study JARF on a broad, realistic set of tabular problems where tree ensembles commonly used. To construct the benchmark, tabular datasets that appear in earlier work on random forests and oblique forests, and then applied simple, priori filters: (i) supervised classification or regression with tabular features; (ii) least a few thousand training points so that EJOP estimation is meaningful; (iii mix of low- and high- dimensional problems, and of balanced and moderately imbalanced label distributions; and (iv) no heavy preprocessing or manual feature engineering beyond standard normalization / encoding. We did drop any dataset based on JARF's performance, and we kept the same pool for all methods and ablations. Several of these tasks overlap with standard suites such PMLB/TabAren
started from widely used OpenML / UCI
C. BASELINE HYPERPARAMETER GRIDS
To keep the comparison fair while reflecting how these models are commonly used in practice, give each method lightweight but non-trivial tuning budget that is shared across datasets. Random forest style methods all use the same number of trees JARF's final forest, and XGBoost is tuned over a small grid on depth, learning rate, and lg penalty. Table summarizes the hyperparameters and search spaces used in experiments
22

---

Published as a conference paper at ICLR 2026
Table 10: Summary of all real-data datasets used in experiments. Here n denotes the number of samples and d the number of raw input features (excluding the target).
Dataset
n
d
Task
Source
Core tabular classification tasks
Adult
48,842
14
Classification
UCI / OpenML.
Bank-marketing
41,188
20
Classification
UCI / OpenML
Covertype
581,012
54
Classification
UCI OpenML
Phoneme
5,404
5
Classification
UCI / OpenML
Electricity
45,312
8
Classification
UCI OpenML
Satimage
6,435
36
Classification
UCI / OpenML
Spambase
4,601
57
Classification
UCI OpenML
Magic Telescope
19,020
10
Classification
UCT / OpenMI.
Letter Recognition
20,000
16
Classification
UCT OpenMI.
Vehicle
846
18
Classification
UCI / OpenML
High-dimensional
large-scale classification tasks
Higgs
940,160
124
Classification
OpenML (Tabular benchmark)
Madelon
2,000
500
Classification
UCI / OpenML
Bioresponse
3,434
419
Classification
OpenML (Tabular benchmark)
Jannis
57,580
254
Classification
OpenML (Tabular benehmark)
MNIST-784
70,000
784
Classification
OpenML / MNIST
Regression tasks
Bike-sharing
17,389
13
Regression
UCI (Bike Sharing)
lefforria-housiig
20,634
8
Regression
OpenML / Cal. Housing
Energy
768
8
Regression
UCI (Energy efficiency)
Kin8nm 8,192
8
Regression
OpenML (kin8nm)
Protein
45,730
9
Regression
UCI / OpenML (Protein)
C.2 PRACTICAL RECOMMENDATIONS FOR JARF
JARF introduces only small number of additional hyperparameters beyond those of the underlying forest: the size of the surrogate forest, the EJOP subsample size m, the differeiffeeene step scale
in ε a MAD(XX))0067644 and the diagonal regularizer 7/s used for conditioning in H Hu Lla In all experiments
the following simple defaults:
surrogate RF with 50 trees, max featuurss = Vd, min min_samples_eed = 1;
EJOP subsample size m = min,00,000. n):
centered finite differences with per-feature step E = a MAD(Xx))006745 and a = 0.1;
EJOP regularization Ā Åo + yla with 10-3 followed by trace normalization
Á H (tx
Table 3 in the main paper provides ablations that effectively serve as tuning guidance. Varying the 0.1 a 0.05 or 0.2 changes mean Cohen's by at most -0.009
step scale from and -0.013, respectively, while leaving macro-F1 and accuracy similarly stable. Changing the subsample size from the default m min(10,00) n) m 0.5n results mean change of only -0.004 in and even tenfold reduction to = 0.1n yields drop of -0.016 in and about 1.2 seconds in training time on average. Removing the diagonal regularizer (? = 0) or trace normalization also produces only small changes (-0.005 and -0.004 in respectively).
These ablations indicate that JARF is robust to wide range reasonable settings, and that the defaults above are near-optimal for the tabular problems consider. In practice we recommend starting with the defaults and, if additional tuning is desired, exploring a small grid such as m e {min(5.00 n)), min miim(10,000. n)) and {0.05, 0.1, 0.2), while keeping y fixed at small value (for example > = 10 3). This keeps the tuning budget modest while preserving the accuracy and compute profile reported in experiments.
23

---

Published as a conference paper at ICLR 2026
Table 11: Hyperparameter grids and defaults used for all methods. Forest baselines all use 200 trees for comparability with JARF's final forest. XGBoost is tuned on shared grid over depth, learning rate, and l2 penalty.
Method
Hyperparameter number of trees max features criterion
Values / setting
RF
200 (fixed)
vd for classification, for regression
Gini (classificationn, MSE (regression)
min.samples.Jea.
(default)
RotF
number of trees
200 (fixed)
blocks K rotation
K = 6 feature subsets per tree
block-diagonal PCA on disjoint feature subsets (unsupervised)
other tree params
same as RF (criterion, min samples leaf, max features)
CCF
number of trees
200 (fixed)
projection type
canonical correlation with targets at each node
projection dim other tree params
authors' recommended default
same as RF
SPORF
number of trees sparsity
200 (fixed)
authors' recommended sparsity level authors' recommended default per node
# candidate directions
other tree params
same as RF
XGBoost
number of trees
200 boosting rounds (fixed)
max depth
{3, 6, 9}
learning rate
{0.05, 0.1} {0. 1}
lg regularization (A)
subsample, colsample bytree loss
1.0 (no subsampling)
logistic loss (classification), squared loss (regression)
PCA+RF
projection
PCA on training features (unsupervised)
# components
d (full-rank rotation)
RF hyperparameters
identical to RF row above
LDA+RF
projection
multi-class LDA on training labels
# components
min 1, d) for C classes
RF hyperparameters
identical to RF row above
JARF (this paper)
surrogate RF size
50 trees, max features = vd, min samples leaf min(10,000. n)
EJOP subsample m FD step E
{ = a MAD( (.DDxx))).1 Xxx)//06745, a = 0.1
EJOP regularization
H Åo + + 10
scaling final RF
H +- H.
RF with 200 trees, same defaults as RF baseline, trained on X A
Licenses and data usage. We only use public datasets with permissive licenses. The repository includes par-dataset source references and license notes; any dataset requiring an external EULA is downloaded via the provider's URL with its terms unchanged.
LLM usage. All scientific content, methods, analyses, and experiments were designed and verified by the authors; LLM model used only to aid/polish writing.
24


Reviews:
[{"review_id": "w9bsZQShkN", "reviewer": "Reviewer_kJgy", "summary": "This paper introduces JARF, a method to enhance standard axis-aligned decision forests by applying a single global supervised linear preconditioner to the input features before training. This makes the forest behave like an oblique forest without changing the RF training algorithm.\n\n\nThe key idea is:\n\n1. Fit a surrogate probabilistic classifier (a random forest) on the original data.\n2. Estimate gradients of the class probabilities with respect to each input feature on a subsample of points.\n3. Construct the Expected Jacobian Outer Product (EJOP) matrix. \n4. Use this EJOP estimate (with light regularization and normalization) as a **global linear transform** $\\hat H$ and train a standard axis-aligned Random Forest on transformed features ($X$$\\hat H$).", "questions": "- How sensitive is JARF to the choice and calibration quality of the surrogate model? Would using XGBoost or a small NN as the surrogate improve EJOP and performance?\n- Are there datasets or regimes where JARF clearly underperforms CCF or SPORF, indicating that a single global transform is insufficient?\n- Have you tried an EGOP-based variant for regression?\n- Your experiments use 10 classic UCI/OpenML-style tabular datasets; have you evaluated JARF on larger-scale or more modern industrial tabular dataset?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The core idea is simple, clean, and easy to implement on top of existing RF code. \n- The method is well-motivated and clearly positioned between axis-aligned forests and oblique trees, leveraging prior EJOP work. \n- Experiments are solid: realistic baselines (RF, XGBoost, RotF, CCF, SPORF), multiple datasets, plus timing comparisons. \n- The mechanism analysis (alignment of oblique split normals with EJOP subspace) and ablations give good insight into why it works.", "weaknesses": "- The method heavily depends on the quality of probability estimates from the surrogate RF used to build EJOP, which is not deeply analyzed. \n- It only evaluates standard tabular classification datasets and does not explore regression or more challenging/high-dimensional settings. - There is no direct comparison to simpler global projections (e.g., PCA, LDA) used once before RF. \n- The novelty is mostly in combining known pieces (EJOP + RF + preconditioning) rather than introducing fundamentally new theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMkUJUZQsf", "reviewer": "Reviewer_Sr16", "summary": "The paper proposes Jacobian-Aligned Random Forests (JARF): learn one global supervised linear transform $H$ from the Expected Jacobian Outer Product (EJOP) of class-probability gradients (estimated via finite differences of an RF surrogate), then train a standard axis-aligned RF on the transformed features $XH$. This aims to capture rotated/interaction directions so axis-aligned splits behave like shared oblique hyperplanes, while keeping RF’s simplicity. Experiments on 10 tabular classification datasets and controlled synthetic rotations show competitive accuracy vs oblique forests with modest overhead, plus ablations supporting the EJOP step and implementation choices.", "questions": "I have no further questions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. It is a simple and one-pass method, plugging the EJOP to perform initial feature transformation. This enables direct application on RF in the subsequent step.", "weaknesses": "1. The proposed method clearly lacks novelty, which does not match the conference standard. It is mainly based on a known paradigm EJOP. The paper’s main change is estimating EJOP with a surrogate RF and finite differences. This feels incremental relative to existing supervised/oblique projection lines rather than a new learning principle.\n\n2. The estimator uses finite differences of RF class probabilities to approximate Jacobians (Sec. 3.6), but the analysis later assumes $f\\in\\mathcal C^3$ with bounded third derivatives (Assumption A1), which is incompatible with piecewise-constant tree ensembles. The text informally argues that ensemble averaging “smooths” predictions, but the formal guarantees hinge on smoothness that the surrogate does not satisfy. And usually, such smoothness improvements are stated when comparing to a single decision tree. This creates a theory–practice gap in the central estimator.\n\n3. A single matrix $\\hat H$ is shared across the entire forest (Sec. 3.4–3.5). This seems to reduce the diversity of the allowed splits. I would encourage the author to make this part more flexible and check the performance.\n\n4. There is a concern about the fairness of the experimental setting. For instance, it was stated that XGBoost is run with a “small shared grid” only; RF fixed at 200 trees; oblique baselines appear near-default. More detailed hyperparameter tuning is necessary.\n\n5. The real-data suite is 10 classification tasks, many with d ≤ 60  and moderate n. More experiments with large-scale datasets (either n or d) would be helpful for the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxofEVfBHS", "reviewer": "Reviewer_PJES", "summary": "The authors propose a **supervised pre-processing step** applied to the feature matrix used as input for a random forest or other **axis-aligned predictors**.  \nThe goal is to enable more flexible splits in the input space, thereby improving the performance of axis-aligned classifiers.  \n\nTheir empirical evaluation shows that the proposed method performs **on par with SPORF**, and slightly better than **XGBoost (XGB)** and **Random Forest (RF)**.", "questions": "1. The proposed method appears similar to a **one-step RFM** [1] for classification.  \n   Can the authors clarify the conceptual and mathematical connection between their procedure and RFM?  \n2. Does the matrix **H** have to be derived from the **same model** used for prediction?  \n   If not, the authors should provide guidance on how to select and pair the transformation and prediction models.  \n3. Can **JARF** be applied to **XGBoost** as well, or is it restricted to random forests?  \n4. The authors mention that the *electricity*, *magic*, and *letter* datasets have **complex decision boundaries**.  \n   Can they explain why these particular datasets were chosen to illustrate this property?  \n5. Can the authors provide **details about the datasets** used in the experiments (e.g., number of samples, features, task type)?  \n6. Can the authors provide **guidance on hyperparameter tuning** or recommendations for practical implementation?\n\n\n[1] Radhakrishnan, A., Beaglehole, D., Pandit, P., & Belkin, M. (2024).  \n*Mechanism for feature learning in neural networks and backpropagation-free machine learning models.*  \n**Science**, 383(6690), 1461–1467.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Interesting idea to transform the data in a supervised manner before training.  \n- The method should be relatively fast to run, better on the compute-performance tradeoff than SPORF.", "weaknesses": "- The reported improvements in performance are **not particularly meaningful**.  \n- Evaluating only on **10 real datasets** is not sufficient to claim generality or robustness.  \n- There is **no discussion** on how to tune hyperparameters for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5dAw0CNe6", "reviewer": "Reviewer_48eb", "summary": "The authors introduce JARF, a method that linearly transforms inputs before fitting a random forest. The linear transform is derived from an expected Jacobian outer product (EJOP), which has been developed and used in prior work. When applied to Random Forests, the authors find that JARF achieves performance results matching existing oblique tree baselines while reducing computation time.", "questions": "- It could be nice to understand how performance varies as the size of the subsampled dataset used for computing the EJOP varies.\n- How did the authors select the 10 datasets they used for evaluation? It would be nice to use a standard suite of datasets (e.g. TabArena or PMLB) to avoid any possibility of biased dataset selection", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors study an interesting problem\n- The authors approach is intuitive\n- The paper is well-written and clear throughout", "weaknesses": "- Main weakness: My understanding of JARF is that it requires first fitting an RF as a step in computing the EJOP, followed by fitting the RF on the conditioned input data. Why is the fitting of the first RF not taken into account in the Efficiency and Compute subsection (or if it is, why is JARF's compute time not atleast 2x that of fitting RF)? This computational efficiency claim is central to the authors' message and should be taken seriously.\n- The paper's novelty is somewhat limited, as the EJOP is already defined and used in prior work (although the authors to swap a kernel regression estimator for RF when estimating it)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)