Oblique Bayesian Additive Regression Trees
Authors: Paul-Hieu V. Nguyen
Ryan Yee
Sameer Deshpande
Venue: Accepted by TMLR ()

Abstract: Current implementations of Bayesian Additive Regression Trees (BART) are based on axis-aligned decision rules that recursively partition the feature space using a single feature at a time. Several authors have demonstrated that oblique trees, whose decision rules are based on linear combinations of features, can sometimes yield better predictions than axis-aligned trees and exhibit excellent theoretical properties. We develop an oblique version of BART that leverages a data-adaptive decision rule prior that recursively partitions the feature space along random hyperplanes. Using several synthetic and real-world benchmark datasets, we systematically compared our oblique BART implementation to axis-aligned BART and other tree ensemble methods, finding that oblique BART was competitive with --- and sometimes much better than --- those methods.

Published in Transactions on Machine Learning Research (04/2025)
Oblique Bayesian Additive Regression Trees
Paul-Hieu V. Nguyen
wwngguyen5@wiisccoomm eduu
Deportment of Statistics
University of Wisconsin-Madison
Ryan Yee
ryee&@wisc. edu
Department Statistics
University of Wisconsin Madison
Sameer K. Deshpande
samcer.deskponddeensss...
Department of Statistics
Universily of Wisconsin- Madison
Wisconsin-Maaiion
Reviewed OpenReview: https: openreviee. net/ forum? id= 140mj4 (.00mmttH... tHBm
Abstract
Current implementations of Bayesian Additive Regression Trees (BART) are based on axis- aligned decision rules that recursively partition the feature space using single feature at time. Several authors have demonstrated that oblique trees, whose decision rules are based on linear combinations of features, sometimes yield better predictions than axis- aligned trees and exhibit excellent theoretical properties. We develop an oblique version of BART that leverages a data-adaptive decision rule prior that recursively partitions the feature space along random hyperplanes. Using several synthetic and real-world benchmark datasets, we systematically compared our oblique BART implementation to axis-aligned BART and other tree ensemble methods, finding that oblique BART was competitive with and sometimes much better than those methods.
1 Introduction
1.1 Motivation
Trse- methods like CART (Breiman al., 1984), random forests (RF; Breiman, 2001), and gradient boosted trees (GBT; Friedman, 2001) are extremely popular and effective machine learning algorithms. They are simple to train and typically deliver accurate predictions (Hastie & Friedman, 2005, $9.2).
Compared to these models, the Bayesian Additive Regression Trees (BART; Chipman et al., 2010) model is used much less frequently in the wider machine learning community. Like RF and GBT, BART approximates unknown functions using regression tree ensembles. But unlike RF and GBT, BART based on a fully generative probabilistic model, which facilitates natural uncertainty quantification (via the posterior) and allows to be embedded within more complex statistical models. For instance, BART has been extended to models for survival (Sparapani et al., 2016; Linero et al., 2022) and semi-continuous (Linero et al., 2020) outcomes; conditional density regression (Orlandi et al., 2021; Li et al., 2023b); non-homogeneous point process data (Lamprinakon et al., 2023); regression with betcroskedastic errors (Pratola et al., 2020); and integrating predictions from multiple models (Yannotly et al., 2024a;b). BART and its extensions typically return accurate predictions and well-calibrated uncertainty intervals without requiring users to specify the functional form of the unknown function and without hyperparameter tuning. Its casc-of-use and generally excellent performance makes BART an attractive "off-the-shelf" modeling tool, especially for estimating heterogeneous causal effects (Hill, 2011; Dorie et al., 2019; Hahn et al., 2020).

---

Published in Transactions on Machine Learning Research (04/2025)
Virtually every implementation of RF, GBT, and BART utilizes axis-aligned regression trees, which partition the underlying predictor space into rectangular boxes (Figure la) and correspond to piecewise constant step functions. Although step functions (i.e., regression trees) are universal function approximators, several authors have proposed modifications to make regression trees even more expressive and flexible. One broad class of modifications replaces the constant output each leaf node with simple paramelric models (e.g., Quinlan, 1992; Landwehr al., 2005; Chan & Loh, 2004; Künzel al., 2022) or nonparametric models (e.g., Gramacy & Lee, 2008; Starling al., 2020; Maia et al., 2024). Another broad class involves the use of "oblique" decision rules, which allow trees to partition the predictor space along arbitrary hyperplanes (see, e.g., Figure 1b). As we detail in Section 2.1, several authors have found that oblique trees and ensembles thereof often outperform their axis-aligned counterparts in terms of prediction accuracy. For instance, Bertsimas Dunn (2017) reported oblique decision trees can outperform CART by 7% while Breiman (2001) noted that an oblique version of RF can improve performance by 3%. Beyond these empirical results, Cattanco et al. (2024) recently demonstratod that oblique decision trees and their ensembles can sometimes obtain the same convergence rates neural networks. To the best of our knowledge, however, there has been no systematic exploration of oblique trees in the Bayesian setting.
Xi 0.4
X1 X2 0.
P2
P3
X2 <
X1 0.4
X1 2Xg
Pi
81
(a)
(b)
Figure 1: Example of step functions defined over [-1, 1]2 and their corresponding axis-aligned (a) and oblique (b) regression tree representations.
1.2 Our contributions
In this work, we introduce obliqueBART and systematically compare its predictive ability to the original BART model and several other tree-based methods. As preview, consider the functions in Figures 2a and 2d, whose discontinuities are not closely aligned with the coordinate axes. Although axis-aligned BART captures the general shape of the decision boundaries, requires very deep trees to do so and inappropriately smoothes over the functions' sharp discontinuities (Figures 2b and 2e). ObliqueBART, in sharp contrast, recovers the boundaries much more preciscly and with shallower trees (Figures 2c and 2f). We will return to these examples in Section 4.1.
The rest of the paper is organized as follows. In Section 2, we review the literature on oblique trees and briefly review the basic axis-aligned BART model. In Section 3, we introduce our prior on oblique decision rules and discuss our implementation of obRT We compare the performance of obliqueBART to axis-aligned BART, several popular troc-based machine learning models, and other oblique ensembles across range of synthetic and benchmark datasets in Section 4. We discuss potential extensions and further refinements in Section 5.
2 Background
Before reviewing existing oblique tree models (Soction 2.1) and BART (Section 2.2). we introduce some notation and briefly review RF and GBT. For simplicity, we focus on the standard nonparametric regression problem with continuous predictors: given observations (æ1, y1 ) ...., (B Ua) with n; E [-1,1] from the
2

---

Published in Transactions on Machine Learning Research (04/2025)
60
-05
0.6
X
(a)
(b)
(c)
-10
0.5
0.0
0.6
1.0
-10
0.5
0.0
0.5
1.0
-1.
0.5
0.0
0.5
1.0
X
X
(d)
(e)
Figure 2: True function (a,d), axis-aligned BART estimate (b.c), and obliqueBART estimate (c.f).
model N (f(a), wish to recover the unknown function f and residual variance o². A decision tree over 1-1, 1|P is pair (T, D) containing (i) a binary tree T that contains several terminal or leaf nodes and several non-terminal or decision nodes and (ii) collection docision rules D, one for cach decision node in T. Axis-aligned decision rules take the form {X, < c} while oblique decision rules take the form TT < c} where Φ E R2. Notice that axis-aligned rules form a subset of all oblique rules.
trace path from the root unique leaf follows. For axis-aligned trees (resp. oblique trees), starting from the root, whenever the path reaches node with decision rule (X, < c) (resp. (Øb x <c)), the path proceeds to the left if x; < (resp. & a < c) and to the right otherwise. We let f(x; T, D) denote the leaf reached by w's decision-following path. By associating a scalar μe to each leaf in T, the regression tree (T, M. represents a piecewise constant step function over [|. We use M denote the set of all ue's 's associated with tree T. Given a regression tree (T),.M), we denote the evaluation function that returns the scalar associated the leaf reached by æ T,; PE(=:T,D)
Given decision tree (T, D), for each point E [|-,,1|,
In the nonparametric regression problem, CART estimates single regression tree such that f(x) g(x; M)) using iterative approach. After initializing T as the root node, CART visits every current leaf node, finds an optimal decision rule for that leaf, and attaches two children to the leaf, turning it inlo a decision node, This process continues until each leaf contains a single observation. Typically CART trees are pruned back according optimal decision rule using a variance reduction criteria. See Hastie & Friedman (2005, $9.2) for an overview of the CART algorithm.
vross-validation criterion. At each step, CART exhanstively scarches for the
3

---

Published in Transactions on Machine Learning Research (04/2025)
RF, GBT, and BART instead learn collection (or ensemble) of M regression trees & = {(Tms Dm Dm:MMm) such that f(x) Im=1 g(2; Tm D Pm) These methods differ significantly in how they learn E.
RF estimates each tree in & using independent bootstrap sub-samples of the training data. Each tree in an RF ensemble is trained using an iterative procedure similar to CART. However, instead of exhaustively scarching over features, at each itcration, RF identifies the optimal rule over a random subsct of features. When the number of training observations n is very large, each tree-growing iteration of RF can be slow, as it involves searching over all possible cutpoints for the randomly selected features. The Extremely Randomized Trees (ERT) procodure of Geurts et al. (2006) overcomes this limitation by randomizing both the splitting variable and cutpoint. That is, in each iteration, ERT draws a small collection of splitting variable and cutpoint pairs and then finds the optimal rule among this small collection. ERT also differs from RF in that each tree is trained using the whole sample, rather than independent bootstrap re-samples. Typically, the trees in an RF or ERT ensemble are un-pruned.
GBT, on the other hand, trains the trees sequentially, with each tree trained to predict the residuals based on all preceding trees. Specifically, for each m, GBT trains (Tm, Mm) to predict the residu- als yı 2m g(v;; Tm,, Dm;: Unlike with CART, RF, and ERT, when running GBT, practitioners conventionally fix the tree to have shallow depth. Although BART shares some algorithmic similarities with ERT and GBT, there are crucial differences, which we will exploit when developing obliqueBART.
2.1 Inducing oblique trees
Conceptually, extending CART to use oblique rules seems straightforward: one need only find an optimal direction ø and cutpoint e that most decreasos the variance of the observod responses that reach a given node. Unfortunately, the resulting optimization problem extely challenging; in the case of classification, the problem is NP-complete (Heath et al., 1993, Theorem 2.1). Consequently, researchers rely on heuristics to build oblique trees. These heuristics fall into three broad classes. The first class of heuristics are model-based: at each decision node, one selects ø after fitting an intermediate statistical model and then finds an optimal cutpoint for the feature Φ x. For instance, Menze et al. (2011) set & to be the parameter estimated by fitting a ridge regression at each decision node. Zhang & Suganthan (2014) and Rainforth & Wood (2015) instead performed linear discriminant, principal component, and canonical correlation analyses at each decision node to determine Φ. Compared to the axis- aligned random forests, ensembles of these model-based oblique trees respectively improved prediction accuracy by 20%, 2.9%, and 28.7%.
The second class of heuristics involve randomization: instead of searching over all possible directions s, one draws D random directions $1. ΦD: computes D new features o oZ.; and then identifies the optimal axis-aligned rulc among these new features. Notable examples of randomization--asod approaches are Breiman (2001), Blaser & Fryzlewicz (2016), Tomita et al. (2020), and Li et al. (2023a), which differ primarily in the distribution used to draw the candidate directions Φ1, .1,...,,1 ΦD. Breiman (2001), for instance, restricted the dx's to have & non-zero elements drawn uniformly from 11. The resulting random combination forests algorithm had decreased out-of-sample errors by 2-3% compared to the axis-aligned random forests. By allowing a variable number of non-zero entries in øa but restricting those entries to +1, Tomita et al. (2020) obtained similar performance gains.
Methods based on the first two classes of heuristics estimate or propose new oblique rules in each iteration of the tree growing procedure. An alternative strategy involves pre-computing a large number of feature rotations or projections and then training standard axis-aligned model using the newly-constructed features. Rodriguez al. (2006), for instance, combine a PCA pre-processing step with the standard random forest algorithm. Blaser & Fryzlewicz (2016) instead generate large number of randomly rotated features before building a random forest ensemble. Their random rotation random forest method outperformed axis-aligned methods in about two-thirds of their experiments.
Before proceeding, it is worth mentioning Bertsimas et al. (2021) does not utilize model- or randomization- basod hcuristics to grow oblique trees. They instcad find optimal oblique decision rules using mixed-integer programming. Their procedure, however, restricts the depth and structure of the tree T a priorí, generally not scalable beyond depth three, and led to small increases in the predictive R²a
4

---

Published in Transactions on Machine Learning Research (04/2025)
2.2 Review of BART
Like RF, ERT, and GBT, BART expresses the unknown regression function f(z) with an ensemble of trees & = {(Tm: Pm, Mm)). Unlike these methods, which learn only a single ensemble, BART computes entire posterior distribution over tree ensembles. Since the posterior is analytically intractable, BART uses Markov chain Monte Carlo (MCMC) to simulate posterior samples.
2.2.1 The BART prior
Key to BART's empirical success is its regularizing prior over the regression tree ensemble. BART models cach tree (Tm, Dm. a priori independent and identically distributed. We can describe the regression tree prior compositionally by explaining how to sample from it. First, draw the tree structure T by simulating branching process. Starting from the root node, which is initially treated as a terminal node at depth 0, whenever a terminal node at depth d is created, the process attaches two child nodes to it with probability a(1 + d) Then, at each decision node, decision rules of the form (X, < c} are drawn conditionally on the rules at the nodes' ancestors. Drawing a decision rule involves (i) uniformly selecting the splitting axis J: (ii) computing the interval of valid values of X, at the current tree node; and (iii) drawing the cutpoint uniformly from this interval. The set of valid Xj values at any node is determined by the rules at the node's ancestors in T. For instance in Figure 1a, if we were to draw decision rule at node 5, 0.4) is the set of valid X1 values and [0.2, 1] is the set of valid X₂ values. Finally, conditionally on T, the leaf outputs in M are drawn independently from a N (0,7"/M) distribution.
Chipman et al. (2010) completed their prior by specifying o² ~ Inv. Gamma (M/2, v2/2). They further recommended default values for cach prior hyperparamcter. They suggested sctting a = 0.95 and ,3 2, so that the branching process prior concentrates considerable prior probability on trees of depth less than 5. For any æ, the marginal prior for ((x) is N (0, Chipman et al. (2010) recommended setting + that this marginal prior places 95% probability over the range of the observed data. They similarly recommended setting y= = 3 and À so that there was 90% prior probability on the event that was less than the standard ioviation of the observed outcome. Finally, they recommended setting M = 200. Although some of these choices are somewhat ad hoc and data-dependent, they have proven tremendously effective across a range of datasets.
2.2.2 Sampling from the BART posterior
To simulate draws from the tree ensemble posterior, Chipman el al. (2010) introduced a Metropoliswwithin- Gibbs sampler. In each iteration, the sampler sweeps over the entire ensemble and sequentially updates each regression tree (Tm,
conditionally fixing the remaining M - trees. Each regression tree update consists of two sleps. First, a new decision tree (T, D) drawn from its conditional distribution given the data, a, and all other regression trees in the ensemble. Then, new leaf outputs M are drawn conditionally given the new decision tree, the data, o, and the remaining regression trees. In the standard nonparametric regression setting, the leaf outputs are conditionally independent given the tree structure, which allows to draw M with standard normal-normal conjugate updates. Most implementaations of BART update the decision tree (T,D) by first drawing a new decision tree from a proposal distribution and accepting that proposal with a Metropolis-Hastingg step. The simplest proposal distribution involves randomly growing or pruning (T, D) with equal probability.
In each MCMC itcration, BART updates cach regression tree conditionally fixing the other troes in the ensemble E. Consequently, each individual regression tree in a BART ensemble does not attempt to estimate the true regression function well. Instead, like GBT, each tree is trained to fit partial residual based on the fits of other tres. However, unlike GBT, which trains trees sequentially, each tree in the BART ensemble is dependent on every other tree a posteriori. This is in sharp contrast to RF and ERT, which independently train each tree to predict the outcome using a bootstrap sample (RF) or full training data (ERT).
BART is, nevertheless, similar to ERT, insofar as both methods grow trees using random decision rules. Recall that at each iteration, ERT grows a tree by selecting the best decision rule among small collection of random proposals. BART, on the other hand, uses Metropolis- inst step to accept or reject single
5

---

Published in Transactions on Machine Learning Research (04/2025)
random proposal. As a result, BART can sometimes grow a tree with rule that does not significantly improve overall fit to data or remove a split that does. Thus, unlike CART, RF, ERT, and GBT, we cannot regard the decision rules appearing in a BART ensemble as optimal in any sense.
3 BART with oblique decision rules
Suppose that we observe n pairs (211 11,,,)),.... Bn) of p-dimensional vectors of predictors æ and scalar outcomes y. For regression problems, we model y ~ N (f(x)) and for binary classification problems, we model Py = 1) (x)))) where & is the cumulative distribution function for a standard normal distribution. Further suppose that have Pcont contimuuus predictors and Peat = p Poout categorical predictors. Without loss of generality, we will arrange continuous predictors first; assume that all continuous predictors lie in interval |-1,1] 1] and that the j-th categorical predictor lies in a discrete set Kj. That is, we assume that all prodictor vectors æ = (2 T T lic in the product space [-1, 1| x K, X .... Keess
ObliqueBART expresses the unknown regression function f(x)) with an ensemble of M regression trees D1, & = [ Mi).,...... (TM, PM,M whose decision rules take the form
Feont < {oTTeont or (Xpcom E CH where j (1,.... ......Praa } and C " Kj. Formally, this involves specifying a prior and computing a posterior over E, from which we approximately sample using MCMC. Like in the original BART model, we model the individual trees (Tm,
a priori i..dd Similarly, we simulate posterior samples using a Mctropolis- within-Gibbs sampler that updates each regression tree one-at-a-time while fixing the others. The main differences between obliqueBART and the original, axis-aligned BART are the decision rule prior and the conditional regression tree updates.
3.1 The obliqueBART prior
For obliqueBATT, we adopt exactly the same priors for T and M as Chipman et al. (2010). That is, we specify the same branching process prior for T and independent normal priors for outputs in M with the same default hyperparameters described in Section 2.2. We specify the obliqueBART decision rule prior implicitly, by describing how draw a rule at a non-terminal node a tree T. To this end, suppose we are internal node nx in T and that we have drawn rules at all of nx's ancestors. With probability Peat/p, we draw categorical decision rule of the form (Xrea-j E C} where C C K, and j E (l,...,Peat] and with probability Peont/p, we draw continuous decision rule of the form {Ø Teomm < c).
Drawing categorical decision rule involves (i) selecting the splitting variable index j: (ii) computing the set A of valid values of the j-th categorical predictor X Xpomnt5 +J; and (iii) forming a subset C C A by randomly assigning each element of A to with probability 1/2. The set A is determined by the decision rules of nx's ancestors in the tree. Although Chipman et al. (1998) initially used such categorical decision rules in their Bayesian CART procedure, most implementations of BART do not use this prior and instead one-hot encode categorical features. A notable exception is Deshpande (2024), who demonstrated these categorical decision rules often produced more accurate predictions than one-hot encoding.
Drawing a continuous decision rule involves (i) drawing a random vector Φ; (ii) computing the interval of valid values of Φ Dcont at nx; and (iii) drawing the cutpoint e uniformly from that interval. Both Breiman (2001) and Tomita et al. (2020) recommend the use of sparse ø's when inducing oblique decision trees. While Breiman (2001) fixed the number of non-zero elements of Φ, Tomita et al. (2020) demonstrated that allowing the number of non-zero entries vary across decision rules improved prediction. Motivated by their findings, we specify a hicrarchical spiko-and-slab prior for é, which encourages sparsity and also allows the number of non-zero eutries of & to vary adaptively with the data.
Formally, we introduce a parameter 0 E [0, 1] that controls the overall sparsity of Φ. Conditionally on 0, we draw p binary indicators ?:....: P L.d.d Bernoulli (8). Then, for each j = 1,.... .oout,, we draw Φ; ~ N (0, 1) if 7/ = 1 and Φ; = 0 otherwise. Finally, we re-scale & to have unit norm. By specifying a further prior on 6, we allow obliqueBART to learn an appropriate level of sparsity of o from the data. For simplicity, we specify conjugate Beta (ag,bo) prior with fixed og, bo 0.
6

---

Published in Transactions on Machine Learning Research (04/2025)
Once we draw 6, draw uniformly from the range of valid values of Φ Dcout available at nx. This range is determined by the continuous decision rules used at the ancestors of nx in the tree and can be computed by solving two linear programs maximizing and minimizing o Zcont over the polytope corresponding to nx.
ObliqueBART depends on several hyperparmeters: the number of trees M, the hyperparameters ag. by > 0 for 8, and v and X for the prior on o² (for regression). Following Chipman et al. (2010), we recommend setting M 200, V 3, and tuning so that there 90% prior probability on the event that o² is less than the observed variance of Y. For the spike-and-slab prior, recommend fixing ag = M and by that the prior mean of 0 is 2/ We have found that obliqueBART's performance is not especially sensitive to these choices; see Appendix C.
3.2 Posterior computation
For simplicity, describe the posterior sampling algorithm used for regression. For binary classification, we use essentially the same algorithm but with an additional data augmentation step, which was first described in Albert & Chib (1993).
To simulate draws from the obliqueBART posterior, we deploy a vetropolis-within. witbin-GGibbs sampler that is almost identical the original BART sampler described in Section 2.2. In each Gibbs sampler iteration, we update each of &, and o² conditionally the other two. Simple conjugate updates are available for 0 and o². Like Chipman et al. (2010), update the trees in È one-at-a-time in two steps: marginal decision tree update followed by conditional update for the leaf outputs. We further utilize grow and prune proposals to update each decision tree. The only difference between our obliqueBART sampler and the original BART sampler lies in the generation of grow proposals.
3.2.1 Regression tree updates
To describe the update of the m-th regression tree (Tm, Dm, - let E- denole the collection of the remaining M - regression trees in the ensemble. For every node nx in T, let I(nx) denote the set of indices the obsorvations æ; whose decision-following paths visit the node nx. Further let 7ax = [[(x)] count the number of observations which visit nx. The full conditional posterior density of the m-th regression tree factorizes over the leaves of? T:
P(T, MTBDDM.BEE E0 ,0,0) & D, N0MMM,E00,,000 D)M E,, D. ,02,0)
}}
p(T x II
1
(r: ne) + r 22
(1)
2
leaves
EEZ(E)
where r; the i-th partial residual basod on the trees in E- given by r; = yi E Em g( Trn, M Mm)).
By integrating out the pe's from Equation (1), we compute
O 2P: }
K E- ,2,0 x D(0) x II
P
exp
(2)
leaves
where, for any node nx in T, Pax naxo + - and Oax = o =0 EiEI(nx) Ps..
We can further compute
{ Pelue - Po10.)²
P(MIT, D.E (MMT..DE x P
P exp
(3)
2
leaves
We see immediately from Equation (3) that, given the decision tree structure, the leaf outputs are con- ditionally independent and normally distributed. Specifically, in leaf L, we have br N (P.-0,, P) conditionally on T. T,y, g(), and o². This means that when n, large, ut's conditional posterior sharply concentrated near the average of the partial residuals in the leaf. This is in marked contrast to CART, RF. and ERT where the leaf outputs exactly the average of the outcomes in the leaf.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Efficient decision tree updates. It remains to describe how to sample a new decision tree from the distribution in Equation (2). Although its density is available in closed form, this distribution does not readily admit an exact sampling procedure. Instead, assuming that currently (Tm; Dm) = (T, we use a single Mctropolis-Hai Matrooolis-Haattnnss (MH) step in which we first propose a random perturbation (T", , Dr) of the (T,D) and then set (Tm; Pm) = (T*, ,D) with probability
aTT D= min { ('(T.) x "T,, 00000) (
(4)
where q(-l-) is a to-be-specified proposal kernel.
In obliqueBART, we use a simple proposal mechanism that randomly grows or prunes the tree, each with probability 1/2. To generate grow proposal, we (i) attach two child nodes to leaf node selected uniformly at random; (ii) draw a new decision rule to associate with the selected node; and (iii) leave the rest of the tree and the other decision rules unchanged. To generate a prune proposal, we (i) delete two leaves with a common parent and their incident edges in T; (ii) delete the decision rule associated with the parent of the deleted leaves; and (iii) leave the rest. of the tree and the other decision rules unchanged. See Figure 3 for a cartoon illustration.
Prume
Current tree
Grow
O Zennt < d)
4 Zeeet <
+ dFrontt < c)
dy Wreent < 63
dy Miout
Og Mecut < Og
og Feout
of Ront <
Figure 3: Cartoon illustration of grow and prune move with oblique, continuous decision rules
The local nature of grow and prune proposals they change at most two leaves - and the fact that the conditional posterior density of (T, D) factorizes over leaves (Equation (2)) yields considerable cancelation in the acceptance probability in Equation (4); see Appendix B.
During a grow move, we must draw a decision rule to associate to the newly created decision node from some proposal distribution; in Figure 3, this rule is highlighted in dark green. In principle, we can use any distribution over decision rules for the proposal. For simplicity, we choose to propose decision rules grow moves from the prior described in Section 3.1. That is, we first randomly decide whether draw a categorical continuous decision rule. If we draw categorical rule, form a random subset of the available levels of that variable. Tf we decide to draw continuous decision rule, then, conditionally on 0, we draw the entries of á independently from the spike-and-slab prior.
select the splitting variable and
Note that there positive probability that the proposed ó contains all zeros. When this occurs, we set c = 1 so that the decision rule (OTa x < 1) sends all observations to the left. child and none to the right. child the proposed docision tree (T*, (T*, Such proposals provide exactly the same
the data as the original tree but receive less prior support due to their increased complexity. Consequently, these proposals tend to be rejected in the MH step. When contains least one non-zero element, solve two linear programs to compute the maximum and minimum values of Φ &cont over the lincar polytope corresponding to the node nx. In our implementation, we solve these programs using the numerical optimization library ALGLIB 4.01.0 (Bochkanov, 2023).
At first glance, it may seem counter-intuitive to propose rules completely at random and independently of the data. One might anticipate that completely random rules would lead to very low MH acceptance probabilities. One might further expect that proposing new rules in data-dependent fashion, for instance
8

---

Published in Transactions on Machine Learning Research (04/2025)
using the model-based heuristics of Menze et al. (2011), Zhang & Suganthan (2014), or Rainforth & Wood (2015), could lead to larger acceptance probabilities and more efficient MCMC exploration. In fact, drawing overly-informed proposals can result in slower MCMC exploration than drawing proposals from the prior.
To develop some intuition for this phenomenon, we note that the MH acceptance probability can be decom- posed into the product of two terms (Equation (B1)). The first balances tree complexity against data fit while the other is the ratio between the prior and proposal probabilities of the new rule. Although informed pro- posals might increase the first term by improving the data fit, they generally make the second term extremely small, deflating the acceptance probability. The resulting Markov chain tends to explore the posterior very slowly. See Deshpande (2024, $2.2 and Appendix B) for more discussion about this phenomenon.
To summarize, in each Gibbs sampler iteration, while keeping o² and 0 fixed, we first sweep over the trees in E, updating each one conditionally on all the others. Each regression tree update involves (i) sampling a new decision tree from Equation (2) with a MH step and grow/prunc mechanism and then (ii), conditionally on the new decision tree, drawing leaf l's outpul from a (P, P) distribution. When generating grow proposals, draw new decision rules from the prior described in Section 3.1.
3.2.2 Updating 0 and の²
Once we update each tree in E, we draw o² and 0 from their full conditional posterior distributions, which are both conjugate. We compute
2² y. E, Inv. Gamma
V v+n vt
1 12
Σ
Σ
M
2) g Tm Den Mn))
1/1
2 2
2
y. & 0² Beta (ag + no. bo + zz)
where no (resp. 2Φ) counts the number of non-zero (resp. zero) entries in the é's appearing in E. An R package implementing obliqueBART is available at htpp::///ittii com com/paulhnguuen/ obliqueBART.
4 Experiments
We compared the performance of obliqueBART to other tree models using several synthetic and benchmark datasots. In Section 4.1, we use the synthetic data from Figure 2 to compare obliqueARRT to axis-aligned BART with the original features and several randomly rotated versions of the features. Then, in Sec- tion 4.2, we compare obliqueBART
other axis-aligned and oblique tree methods, as well as axis-aligned tree ensemble methods fit with randomly pro-rotated fcatures benchmark datascts for both regression and classification problems. Generally speaking, obliqueBART run with its default hyperparameter values performed very well on regression tasks and was competitive with the other tuned methods on classification tasks. We further found that randomly pre-rotating features before fitting axis-aligned RF, ERT, or GBT models tended to yield worse results than simply fitting obliqueBART.
We performed all of our experiments on a shared high-throughput computing cluster. For obliqueBART and BART, we compute posterior means of f(x (for regression) and IP(y = 1|x) (for classification) based on 1000 samples obtained by simulating single Markov chain for 2000 iterations and discarding the first 1000 "burn-in." We fit BART, RF, ERT, and XGBoost models using implementations available in the R packages BART (Sparapani et al., 2021), randomForest (Liaw & Wiener, 2002), ranger (Wright & Ziegler, 2017), and xgboost (Chen et al., 2024). We also compared BAl'' performance to rotated versions of RF, ERT, XGB, and BART with R = 200 random rotations. Additionally, we fit oblique tree ensembles from the ODRF (Zhan et al., 2022) and aorsf (Jacger et al., 2022) packages on both the classification and regression settings. We also compared obliqueBART to oblique tree-based classifiers implemented in the RPEnsemble (Cannings & Samworth, 2017) and rotationForest (Ballings & Van den Poel, 2017) packages.
9

---

Published in Transactions on Machine Learning Research (04/2025)
4.1 Synthetic data experiments
We compared obliqueBART to axis-aligned BART with data from the two simple functions in Figure 2. Since obliqueBART partitions the fcature space along randomly selected directions, we additionally compared it with a hybrid procedure that first computes R random rotations of the feature space and then trains an axis-aligned BART model using the rotated features. This random rotation BART model is analogous to the random rotation ensemble methods studied in Blaser & Fryzlewicz (2016), but it does not screen tune the random rotations.
For these experiments, we sampled n covariate vectors 2; Uniform((0,1) 1]2) and generated ~ N (f(æ;;;; 0, A), 1),, 1), where the function f(æ; A) takes on two values =A and depends on a parameter We considered two such functions, the rotated axes function (Figure 2a) and the simusoid function (Fig- ure 2d).
Given any 0 E [0, 1/4], let u(x) be the point obtained by rotating 0 radians counter- clockwise around the origin. The value of rotated axes function is determined by the quadrant in which u lies. Specifically, for rotation angle E {0,
Rotated axes.
and jump A E {0.5, 1, 4}, we set f(æ; 0, =
A x (2 x 1 (N1 > 0) 1)..
Sinusoid. The value of f(x; 0, A) depends on which side of a particular sinusoid the point a lies. Specifically, for a given amplitude 0 (0,0.1,...,1I) and jump A {0.5, 1,2,4), we f(x; 0, A)
Á (2 (x2 ssn (10zz)) 1).
For each combination of n E (100, 1000, 10000), function f, jump A, and parameter 0, we generated 20 datasets of size n. Figure 4 shows how axis-aligned BART and random rotation BART's out-of-sample RMSE (averaged over 20 replications) relative to obliqueBART changes as the decision boundaries becomes less and less aliseeligned (i.e., as increases) with A 4.
Rotated axes partition
Sinusoidal partition
3.0
2,5
2.5
AA
RMSE ratio (test)
R4
-4-
124
2.0
RS0
RMSE
+ R50
R1G0
R100
19
R20D
R200
x/16
x/8
3ml 16
4/4
0.0
0.2
0.4
0.8
0.8
(a)
(b)
Figure 4: Performance of axis aligned BART (AA) and axis-aligned BART with random rotations relative to obliqueBART terms of out-of-sample predictive error in the (a) rotated axes partition and (b) sinusoidal partition, as pictured in Figure 2. This figure depicts the experiments run with n = 1000 and A = 4. Shaded regions depict 95% confidence intervals for the ratio of each model's out-of-sample RMSE and obliqueBART's out-of-sample RMSE are shaded (i.e., the saa-le error relative to obliqueBART).
As we might expect, obliqueBART substantially outperformed axis-aligned BART on the rotated axes prob- lem. Interestingly, we see that obliqueBART even out- out-performs. axis-aligned BART when 0 = 0 and the true decision boundary is axis-aligned. When 0, the obliqueBART ensemble used, on average, axis-aligned rules 70.2% of the time compared to 52.9% of the time when = 7/4. This suggests that obliqueBART, through the hierarchical spike-and-slab prior for o can adaptively adjust the number of axis-aligned
10

---

Published in Transactions on Machine Learning Research (04/2025)
oblique rules. ObliqueBART further outperformed axis-aligned BART the sinusoid problem, even though the decision boundary was highly non-linear (see Figure 4b).
Although obliqueBART also outperformed random rotation BART with a small number of random rotations, the gap between the methods diminished as the number of rotations R increased. For these data, random rotation BART run with at least 50 rotations performed as well as and, at times, slightly outperformed obliqueBART. As we discuss in Section 4.2, matching obliqueBART's performance with a random rotation ensemble is much harder when the mumber of contimuous predictors Peont is large.
4.2 Results benchmark datasets
The regression datasets contain between 96 and 53,940 observations and betwoen 4 and predictors and the classification dalasets contain between 61 and 4601 observations and between 4 and 72 predictors. In our experiments, contimuous variables were scaled to [-1,1] and observations with missing values were dropped before we fit any models to the data. We obtained most of these datasets from UCI Machine Learning Repository (uttps://archive ics ius.uci........c.ccc.m.......iii edu); the Journal of Applied Econometrics data archive (http://qed) econ. queensu. cateaauu.aaaaaaeee ca/jae/); and from several R packages. See Table A1 for the dimensions of and links to these datasets. For each competing method, we tuned hyperparameters using 5-fold cross- validation on each training datasct. Table A2 shows the grids of values considered for cach method's hyperparameters. These grids mirror those used in Chipman et al. (2010, Table 2).
We created 20 random 75%-25% training-testing splits of each dataset. For regression tasks, computed each methods' standardized out-of-sample mean square error (SMSE), which is defined the ratio be- -1 tween the the mean square errors of model, Niest Eee (Yuest, 82)² and the mean of the training data -1 Ees (Yrest,s - Y ). For classification problems, we formed predictions by truncating the outputted class probability at 50% and computed out-of-sample accuracy. We performed one-sided paired t-tests to determine whether obliqueBARTT error were significantly less than the competing methods' errors.
4.2.1 Comparison to axis aligned methods
Figure 5 compares obliqueBART's SMSE and accuracy to those of the axis-aligned methods across every fold and datasct. We defer datasct-by-datase tabulations of these crror metrics to Tables A3 and A4 in Appendix A.
Regression
Classification
XGB
XGB
ERT
ERT
RF
RF
BART
BART
obliqueBART
obliqusBART
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
SMSE
Accuracy
(a)
(b)
Figure 5: ObliqueBART's SMSE (a) and accuracy (b) across all splits and datasets, compared to XGB, ERT, RF, and BART. Models with lower SMSE's and higher accuracies are preferred.
Although no single method performed the best across all regression datasets, obliqueBART had the smallest overall mean SMSE (0.297). For comparison, the mean SMSEs for the tuned axis-aligned methods were 0.316 (BART), 0.314 (RF), 0.309 (ERT), and 0.299 (XGB). The median SMSEs were respectively 0.210 (obliqueBART), 0.261 (BART), 0.264 (RF), 0.260 (ERT), and 0.219 (XGB). ObliqueBART's SMSE was also statistically significantly lower (at the 5% level) than BART's nine datasets, RF's on eight datasets,
11

---

Published in Transactions on Machine Learning Research (04/2025)
ERT's six datasets, and XGB's five datasets. It is notable that the "off-the-shelf" performance of obliqueBART on regression tasks was somewhat better than the performance of its tuned competitors.
ObliqueBART had the lowest classification accuracy (0.846) averaged across all folds and datasets while ERT had the highest (0.867). ERT had the largest accuracy on eleven datasets while obliqueBART was the best-performing method for four datasets. Generally speaking, however, obliqueBART was still competitive with the other methods for classification. For example, the difference in accuracy between obliqueBARII and ERT, the best method, less than 2% in 13 of the 22 datasets. Similarly, the difference in accuracy between obliqueBART and BART was less than 1% in 13 of the datasets.
Because BART and obliqueBART generate many samples of the regression tree ensemble, we would expect them to be slower than RF, ERT, and XGB, which only generale a single ensemble. While BART and obliqueBART typically take longer to compute compared to non-Bayesian tree ensembles, their ability to quantify uncertainty often make the increased computation time worthwhile. Further, because obliqueBART involves solving two linear programs when proposing new decision rules, we would expect it to be slower than BART. This was generally the case in our experiments: for most datasets, obliqueBART was about twice slow as BART and slower than RF, ERT, and XGB. Surprisingly, obliqueBARTT was much faster on the diamonds dataset than BART.
4.2.2 Comparison with oblique ensembles
Next, we compared obliqueARRT's performance on the benchmark datasets with available oblique tree ensemble implementations in R. For each competitor method, we tuned hyperparameters based on 5-fold cross validation within each training set.
Regression
Classification
ODRF
RPEnsemble
rotatonForest
sorsf
ODRF
aoraf
obliqueBART
obliqueBART
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
SMSE
Accuracy
(a)
(b)
Figure 6: ObliqueBART's SMSE (a) and accuracy (b) across all splits and datasets, compared to ODRF, aorsf, RPEnsemble, and rotationForest. Models with lower SMSE's and higher accuracies are preferred.
ObliqueBART still performs quite well in the regression setting compared to other oblique ensembles. ObliqueBART has the lowest mean and modian SMSE across all datascts and splits 0.297 and 0.210, respectively; ODRF has mean and median SMSE at 0.339 and 0.318, and aorsf has mean and median SMSE 0.343 and 0.309. For classification, obliqueBART is average compared to the other oblique ensembles, and specific method stands out from the rest, generally.
One important distinction in these results is the reliability of each implementation. ObliqueBART runs for all 40 benchmark datasets, for each training and testing split. our experiments, we found that some other methods often failed, either because they hit memory limits or encountered values a categorical level in testing that were not seen in training. Tables A5 and A6 contain the mean SMSE and accuracy for each individual dataset.
4.2.3 Comparison with random rotation ensembles
Next, we compared BbliqueBART's performance randomly rotated versions of BART, RF, ERT, and XGB with R E {1, 4, 16, 50, 100, 200) random rotations. We report dataset-by-dataset SMSFs and accuracies in
12

---

Published in Transactions on Machine Learning Research (04/2025)
Tables A7 and A8. We additionally determined how many random rotations were needed for each of these methods to match obliqueBARTTs performance (Tables A9 and A10). Figure 7 shows the SMSFs and accuracies of obliqueBART and the random rotation ensembles with R = 200 for all datasets and folds.
Regression
Classification
rotXGB
roXGBB
IlER
noL_RRT
JOURF
NOIRF
rotBART
relBART
abliqueBART
obliqueARRT
0.0
1.0
1.5
2.0
3.0
0.2
0.4
0.8
1.0
SMSE
Accuracy
(a)
(b)
Figure 7: ObliqueBART's SMSE and accuracy (resp. left and right) across all splits and datasets, compared to XGB, ERT, RF, and BART with 200 random rotations. Models with lower SMSE's and higher accuracies are preferred.
Somewhat surprisingly, the randomly rotated random forests (rotRF) performed markedly worse than the other methods. For regression, obliqueBART and the rotated version of BART (rotBART) had the smallest average SMSEs (0.297 and 0.296, resp.). ObliqueBART was also competitive with the rotERT and rotXGB for classification; its average accuracy was 0.846 while the best performing method, rotERT, had an average accuracy of 0.859. We additionally observed that obliqueBART was, average, 20 times faster than rotRF and twice as fast as rotBART run with R = 200 random rotations.
Interestingly, there was substantial variation in the minimum number of rotations needed for rotBART, rotRF, rotERT, and rot to match obliqueBART's performance. For instance, rotBART was unable to match obliqueBARTTss performance on 19 datasets even with 200 rotations and required only one rotation for datasets; four rotations for two datasels; 16 rotations for one dataset; and rotations for one dataset. rotXGB, on the other hand, was unable to match obliqueBART's performance on 24 datasets using 200 rotations and required one rotation for nine datasets; four rotations for four datasets; 16 rotations for dataset; 50 rotations for one dalaset; and 100 rotations for one dataset. These results suggest that running OART is often more effective and faster than tuning the number of random pre-rotations used to train random rotation ensemble.
5 Discussion
We introduced obliqueBART, which extends the expressivity of the BART model by building regression trees that partition the predictor space based random hyperplanes. Unlike oblique versions of CART or RF, obliqueBARTT does not search for optimal decision rules and instead grows trees by randomly accepting (via a Metropolis-Hastte outperform BART across the benchmark datasets we considered, its performance generally not signifi- cantly worse than BART's and can sometimes be substantially better. On this view, we would not advocate for wholesale replacement axis-aligned BART in favor of our obli implementation. It is possible, for instance, that alternative priors for ò may yield somewhat larger improvements.
step) completely random decision rules. Although obliqueBART does not uniformly
While we have focused primarily on the tabular data setting, we anticipate that our obliqueBART implemen- lation can be fruitfully exlended to accommodale structured input like images. Li al. (2023a) demonstrated that oblique tree ensembles can close the performance gap between non-neural network methods and convo- lutional deep networks for image classification. Their manifold oblique random forests (MORF) procodure recursively partitions images based on the average pixel value within random rectangular sub-regions of an image. Developing BART analog of MORF would involve modifying the decision rule prior to ensure that
13

---

Published in Transactions on Machine Learning Research (04/2025)
the non-zero elements of o correspond to connected sub-region of an image. We leave this and similar extensions for other types of structured inputs (e.g., tensors, functional data) to future work.
References
James H Albert and Siddhartha Chib. Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88(422):669 679, 1993.
Michel Ballings and Dirk Van den Poel. rotationForest: Pit and Deploy Rotation Forest Models, 2017. R package version 0.1.3.
Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106(7):1039-1082, 2017. 2017.
Dimitris Bertsimas, Jack Dunn, and Yuchen Wang. Near-optimal nonlinear regression trees. Operations Research Letters, 49(2):201-20) 2021.
Rico Blaser and Piotr Fryzlewicz. Random rotation ensembles. Journal of Machine Learning Research, 17 (1):126-14) 151, 2016.
Sergey Bochkanov. ALGLIB. www.alglib.net, 2023.
Leo Breiman. Random forests. Machine Learning, 45:5 32, 2001.
Leo Breiman, Jerome H Friedman, Richard A Olshen, and Charles J Stone. Classification and Regression Trees. Routledge, 1984.
Timothy Cannings and Richard Samworth. Randon-projection ensemble classification. Journal of the Royal Statistical Society Series B, 79(4):959 1035,105555000 2017.
Matias D. Cattaneo, Rajita Chandak, and Jason M. Klusowski. Convergence rates of oblique regression trees for flexible function librarics. Annals of Statistics, 52(2):466-490, 2024.
Kin- Yee Chan and Wei- Yin Loh. Lotus: An algorithm for building accurate and comprehensible logistic regression trees. Journal of Computational and Graphical Statistics, 13(4):826-88) 852, 2004.
Tianqi Chen, Tong Hc, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, Mu Li, Junyuan Xie, Min Lin, Yifeng Geng, Yutian Li, and Jiaming Yuan. rgboost: Extreme Grudient Boosting, 2024. R package version 1.7.8.1.
Hugh A Chipman, Edward George, and Robert E. McCulloch. Bayesian CART model search. Journal of the American Statistical Association, 93((43):9335 948, 1998.
Hugh A Chipman, Edward George, and Robert E. McCulloch. BART: Bayesian additive regression trees. Annals of Applied Statistics, 4(1):266-298, 2010.
Sameer K. Deshpande. flexBART: Flexible Bayesian regression trees with categorical predictors. Journal of Compulational and Graphicul Stalistics, 2024.
Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. Automated versus do-il-yourself methods for causal inference: Lessons learned from data analysis competition. Statistical Science, 34(1): 43 68, 2019.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of Stotistics, 29 (5):1 10:1189--122, 2001.
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine Learning, 63: 3-42, 2006.
Robert B Gramacy and Herbert KH Lec. Bayesian treed Gaussian process models with an application computer modeling. Journal of the American Statistical Association, 103(483): 2008.
14

---

Published in Transactions on Machine Learning Research (04/2025)
P. Richard Hahn, Jared S. Murray, and Carlos M. Carvalho. Bayesian regression models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesion Analysis, 15(3):965-10) 10566, 2020-
Trevor Hastie and Robert Tibshirani Jerome Friedman. The Flements of Stalistionl Learving: Dala Mining, Inference, and Prediction. Springer, 2005.
David G. Heath, Simon Kasif, and Steven L. Salzberg. Induction of oblique decision trees. In International Joint Conference on Artificial Intelligence. 1993.
Jennifer L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphicul Stalislies, 20(1):217 240, 2011.
Byron C. Jaeger, Sawyer Welden, Kristin Lenoir, and Nicholas M. Pajewski. aorsf: An R. package for supervised learning using the oblique random survival forest. Journal of Open Source Sofbware, 7(77): 4705, 2022.
Sören R Künzel, Theo F. Saarinen, Edward W. Liu, and Jasject S. Sekhon. Linear aggregation in treo-based estimators. Journal of Computational and Grophical Statistics, 31(3):917 934, 2022.
Stamatina Lamprinakou, Mauricio Barahona, Seth Flaxman, Sarah Filippi, Axel Gandy, and Emma McCoy. BART-based inference for Poisson processes. Compulational Statisties Dala Analysis, 180: 107658, 2023.
Niels Landwehr, Mark Hall, and Eibe Frank. Logistic model trees. Machine Learning, 59:161-201 205, 2005.
Adam Li, Ronan Perry, Chester Huynh, Tyler M. Tomita, Ronak Mehta, Jesus Arroyo, Jesse Patsolic, Ben Falk, Sridevi Sarma, and Joshua Vogelstein. Manifold oblique random forests: Towards closing the gap convolutional deep networks. SIAM Journal on Mathematics of Data Science, 5(1):77-96, 2023a.
Yinpu Li, Autonio R. Linero, and Jared Murray. Adaptive conditional distribution estimation with Bayesian decision tree ensembles. Journal of the American Statistical Association, 118(543): :2129 -212,, 2023b.
Andy Liaw and Matthew Wiener. Classification and regression by randomForest. R News, 2(3):18 22, 2002.
Antonio R. Linero, Debajyoti Sinha, and Stuard R. Lipsitz. Semiparametric mixed-scale models using shared Bayesian forests. Biometrics, 76(1):131-144) 2020.
Antonio R. Linero, Piyali Basak, Yinpu Li, and Debajyoti Sinha. Bayesian survival tree ensembles with submodal shrinkage. Bayesian Analysis, 17(3):997 -99--1020,, 2022.
Mateus Maia, Keefe Murphy, and Andrew C. Parnell. GP-BART: A novel Bayesian additive regression trees
approach using gaussian processes. Computational Statistics Data Analysis, 190:107858, 2024.
Bjoern H Menze, B Michael Kelm, Daniel N Splitthoff, Ullrich Koethe, and Fred A Hamprocht. On oblique random forests. In Machine Learning and Knowledge Discovery in Dalabases, 2011.
Vittorio Orlandi, Jared Murray, Antonio R. Linero, and Alexander Volfovsky. Density regression with Bayesian additive regression trees. arXiv:2112.12250, 2021.
Matthew T. Pratola, Hugh A Chipman, Edward George, and Robert. F. McCulloch. Heteroscedastic BART using multiplicative regression trees. Journal of Computational and Graphical Statistics, 2952):405-417, 2020.
John R Quinlan. Learning with continuous classes. In 5th Australion Joint Conference on Artificial Intelli- gence, 1992.
Tom Rainforth and Frank Wood. Canonical correlation forests. arXiv:1507.05444 2015.
J.J. Rodríguez, L.I. Kuncheva, and C.J. Alonso. Rotation forest: new classifier ensemble method. JEEE Transactions on Patlern Analysis and Machine Intelligence, 28(10):1619-16)) 1630, 2006.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Rodney Sparapani, Brent R Logan, Robert E. McCulloch, and Purushottam W. Laud. Nonparametric survival analysis using Bayesian Additive Regression Trees. Stalistics in Medicine, 35(1))2744127733 2016.
Rodney Sparapani, Charles Spanbauer, and Robert McCulloch. Nonparametric machine learning and efficient computation with Bayesian additive regression trees: The BART R package. Journal of Statistical Software, 97(1):1 66, 2021.
Jennifer Starling, Jared S Murray, Carlos M Carvalho, Radek K Bukowski, and James G Scott. BART with targeted smoothing: An analysis of patient-specific stillbirth risk. Anmals of Applied Stalistics, 14 (1):28-50, March 2020.
Tyler M. Tomita, James Browne, Cencheng Shen, Jaewon Chung, Jesse L. Patsolic, Benjamin Falk, Carey E. Pricbe, Jason Yim, Randal Burns, Mauro Maggioni, and Joshua T. Vogelstein. Sparse projection oblique randomer forests. Journal of Machine Learning Research, 21044): 39, 2020.
Marvin N. Wright and Andreas Ziegler. ranger: A last implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, (7(1):1-17, 2017. doi: (1118667/jss.077....
John C. Yannolty, Thomas J. Santner, Richard J. Furnstahl, and Matthew T. Pratola. Model mixing using Bayesian additive regression trees. Technometrics, 66(2):196-25)) 2076 2024a.
John C. Yannotty, Thomas J. Santner, Bo Li, and Matthew T. Pratola. Combining climate models using Bayesian regrossion trees and random paths. arXiv 2407.13169, 2024b.
Haoran Zhan, Yu Liu, and Yingcun Xia. ODRF: Consistency of the oblique decision tree and its random forest. arXiv:2211 2265.121111,1,, 2022.
Le Zhang and Ponnuthurai Nagaratnam Suganthan. Random forests with ensemble of feature spaces. Pattern Recognition, 37(10)33429 3437, 2014.
16

---

Published in Transactions on Machine Learning Research (04/2025)
A Additional benchmarking details and results
Table A1 lists the dimensions and sources of each benchmark dataset. Most datasets are from the UCI data repository, though some are from individual R packages (boldod), the Journal of Applied Econometrics data archive, and the CMU Statlib data repository.
Table Al: Dimensions and hyperlinked sources of benchmark datasets
Regression Data
Classification Data
Data (Source)
n
Poant
Data
p
Pront
abalone (UCI)
4177
Danknote(UC))
1372
4
ais (DAAG)
202
12
10
blood transfusion (UCI)
748
4
ammenity (JAE)
3044
20
breast cancer diag (UCI)
30
attend (JSE)
838
9
6
breast cancer (UCI)
683
9
(ISL))
16
breast cancer prog. (UCI)
194
32
basketball (SMIS)
96
4
3
climate crashes (UCT)
540
18
18
boston (MASS)
506
3
connectionist sonar (UCT)
208
60
budget (JAE)
1729
10
10
credit approval (UCI)
653
15
cane (OzDASL)
3775
31
25
echocardiogram (UCI)
579
10
cpu (UCT)
209
6
fertility (UCI)
100
9
diabetea (lars)
442
10
9
german credit (UCI)
1000
20
diamonds (ggplot2)
53940
heptatitis (UCI)
19
edu (JAE)
2338
5
ILPD (UCT)
579
10
labor (Ecdat)
5320
ionosphere (UCI)
351
34
mpg (UCI)
392
8
7
ozone1 (UCI)
1848
72
72
rice (JAb)
1026
14
ozone8 (UC1)
1847
72
servo (UCT)
167
4
2
parkinsons (UCI)
195
22
22
strikes (Statlib)
625
planning relax (UCI)
182
12
qsar bio. (UCI)
1055
41
38
seismic bumps (UCI)
2584
18
spambase (UCI)
4601
57
57
spectf heart (UCI)
267
44
44
Table A2 shows the values considered for each competing method's operational parameters. We used func- tions included in the RPEnsemble package to choose the best values for its hyperparameters.
Table A2: Operational parameters for competing methods
Method
Parameter
Values considered
Random Forest
# of trees
10, 50, 100, 200
% of variables sampled to grow node
10, 33, 100 ypt 50
Gradient Boosting
# of trees
50, 100, 200
Shrinkage
001, .05, .1, .25
Max depth for each tree
1, 3,
Extremely Randomized Trees
# trees
10, 50, 100, 200
% of variables sampled to grow node
10, 33, 100 50
vP
# of random splits to consider
Rotation Forest.
# of trees
10, 50, 100, 200
% of variables sampled to grow node
10, 33, 100
ODRF AORSF
# of trees
10, 50. 100, 200
# trees
10, 50, 100, 200
% of variables sampled to grow node
10, 33, 100 50
# of cut-points assessed
1, 2,
17

---

Published in Transactions on Machine Learning Research (04/2025)
Tables A3 and A4, respectively, show the out-of-sample standardized mean square and accuracy of oblique- BART, RF, ERT, BART, and XGB for each benchmark dataset, averaged over 20 training-lesting splits. We marked entries in these tables with asterisks whenever obliqueBART achieved statistically significantly lower SMSE or higher accuracy than the competing method. Throughout, we assessed significance using a paired I-test. and a 5% threshold. We report analogous results from comparison obliqueBART to rotated versions of RF, ERT, BART, and XGB run with 200 random rotations in Tables A7 and A8.
Table A3: Standardized mean square errors on regression benchmark datasets, averaged across 20 training- testing splits, for obliqueBART and axis-aligned methods. Best performing method is bolded and errors that are statistically significantly larger than obliqueBARTSS
marked with asterisk.
data
obliqueBART
BART
ERT
RF
XGB
abalone
0.44
*0.449
#0.45
*0.449
*0.454
ais
0.122
*0.133
0.108
0.112
0.131
amenity
0.283
0.278
0.279
0.282
0.274
attend
0.246
*0.302
*0.412
*0.428
0.246
baseball
0.39
*0.399
*0.439
*0.419
*0.461
basketball
0.762
0.669
0.656
0.708
0.753
boston
0.778
0.6
0.587
0.62
0.639
budget
0.003
0.003
0.002
0.004
0.003
0.183
*0.193
0.18
0.182
0.173
0.139
*0.272
0.149
*0.178
0.153
diabetes
0.495
*0.5
*0.526
*0.546
*0.537
diamonds
0.02
0.019
0.018
0.018
0.02
edu
0.02
0.02
0.02
*0.02
0.02
labor
0.221
*0,693
*0.604
*0.535
*0.349
mpg
0.135
0.125
0.127
0.135
*0.144
rice
0.015
*0.027
*0.019
*0.027
0.012
servo
0.167
0.149
0.169
0.152
0.073
strikes
0.926
0.863
0.816
0.836
0.94
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table A4: Accuracies on classification benchmark datasets, averaged across 20 training-testinng splits for obliqueBART and axis-aligned methods. Best performing method bolded and errors that are statistically significantly larger than obliqueBART's are marked with an asterisk.
data
obliqueBART
BART
ERT
RF
XGB
banknote
0.997
0.997
0.998
*0.991
0.995
blood transfusion
0.769
0.795
0.79
0.767
0.783
breast cancer diag.
0.922
0.966
0.965
0.958
0.965
breast cancer
0.975
*0.973
0.976
0.972
*0.655
breast cancer prog.
0.758
0.76
*0.744
*0.745
0.751
climate crashes
0.917
0.944
0.944
0.918
0.948
connectionist
0.802
0.824
0.861
0.808
0.831
credit approval
0.866
0.87
0.87
0.871
0.865
echocardiogram
0.714
0.712
0.719
*0.699
*0.944
fertility
0.86
0.86
0.852
*0.836
0.852
hepatitis
0.865
0.87
0.86
0.85
ILPD
0.714
0.71
0.719
+0.699
*0.694
ionosphere
0.902
0.926
0.941
0.92
0.923
ozone
0.97
0.97
0.969
0.97
0.969
ozone
0.933
0.938
0.941
0.938
0.939
parkinsons
0.865
0.861
0.913
0.89
0.892
planning relax
0.717
*0,708
*0.695
*0.677
0.71
qaar biodegradable
0.823
0.845
0.866
0.86
0.863
seismic bumps
0.935
0.935
0.935
*0.931
*0.934
spambase
0.771
0.932
0.956
0.948
0.952
spectf heart
0.799
0.813
0.795
0.806
0.787
statlog german cred.
0.751
0.76
0.759
0.757
0.749
Table A5: SMSE for obliqueBART and other oblique ensemble methods on regression benchmark datasets, averaged across 20 training-eesting splits. The best performing method is bolded; model and data combina- tions that were not succesfully across all 20 train/test splits marked with parentheses.
data
obliqueBART
ODRF
aorsf
abalone
0.44
0.414
0.419
ais
0.122
0.121
0.114
amenity
0.283
0.319
0.324
attend
0.246
0.446
0.491
baseball
0.39
0.427
0.477
basketball
0.762
0.636
0.659
boston
0.778
0.585
0.67
budget
0.003
0.01
0.013
0.183
0.186
0.259
0.139
0.282
0.28
diabetes
0.495
0.494
0.493
diamonds
0.02
NA
0.018
edu
0.02
0.019
0.021
labor
0.221
0.674
0.787
mpg
0.135
0.132
0.135
rice
0.015
0.03
0.038
servo
0.167
0.192
0.218
strikes
0.926
0.795
(0.801)
19

---

Published in Transactions on Machine Learning Research (04/2025)
Table A6: Accuracies for obliqueBART and other oblique ensemble methods on classification benchmark datascts, averaged across 20 maining-testing splits. The best performing method bolded; model and data combinations that were not run succesfully across all 20 train test splits are marked with parentheses.
data
obliqueBART
ODRF
RPEnsemble
aorsf
rotationForest
banknote
0.997
0.994
0.985
0.998
0.993
blood transfusion
0.769
0.797
0.772
0.793
0.789
breast cancer diag.
0.922
0.975
0.943
0.977
0.954
breast cancer
0.975
0.973
0.972
0.976
0.975
breast cancer prog.
0.758
0.727
0.756
0.756
(0.748)
climate crashes
0.917
0.956
0.917
0.92
0.934
connectionist.
0.802
0.8
0.757
0.788
(0.783)
credit approval
0.866
0.872
0.871
0.867
(0.871)
echocardiogran
0.714
0.703
0.714
0.707
0.706
fertility
0.86
0.842
0.85
0.86
NA
hepatitis
0.835
0.845
0.84
(0.885)
ILPD
0.714
0.703
0.714
0.707
0.706
ionosphere
0.902
NA
0.895
NA
(0.946)
ozone 1
0.97
0.968
0.97
0.97
0.97
ozone 8
0.933
0.94
0.933
0.939
0.942
parkinsons
0.865
0.878
0.855
0.87
(0.881)
planning relax
0.717
0.645
0.716
0.711
0.696
qsar biodegradable
0.823
0.866
0.709
0.864
0.858
aeismic bumps spanbase
0.935
NA
0.935
NA
0.934
0.771
0.951
(0.724)
0.947
(0.925)
spectf heart
0.799
0.802
0.799
0.808
(0.806)
statlog german cred.
0.751
0.76
0.746
(0.759)
0.763
Table A7: Standardized mean square errors on regression benchmark datasets, averaged across 20 training- testing splits, for obliqueBART and rotated versions of axis-aligned methods with 200 random rotations. Best performing method is bolded and errors that are statistically significantly larger than BAl' marked with asterisk. NA indicates that the method could not be run with 200 random rotations.
data
obliqueBART
rotBART
rotERT
rotRF
rotXGB
abalone
0.44
0.448
0.45*
1.08*
0.47*
ais
0.122
0.129
0.151
1.33*
0.167*
amenity
0.283
0.281
0.349*
1.07*
0.357*
attend
0.246
0.297*
0.304
1.06*
0.305
baseball
0.39
0.406*
0.446*
1.07*
0.544*
basketball
0.762
0.691
0.68
1.06*
0.767*
boston
0.778
0.577
0,833
22.6*
0.853
budget
0.003
0.003
0.010
L.200*
0.019*
cane
0.183
0.191
0.315
1.580*
0.291*
cpu
0.139
0.313*
0.201 *
1.480*
0.249*
diabetes
0.495
0.499
0.520
0.967*
0.606*
diamonds
0.020
0.020
NA
NA
NA
edu
0.020
0.020
0.022*
1.150*
0.024
labor
0.221
NA
NA
NA
NA
mpg
0.135
0.138
1.0g*
0.16*
rice
0.015
0.026*
0.204*
1.670*
0.234*
servo
0.167
0.196*
0.25*
2.24*
0.225*
strikes
0.926
0.805
0.915
36.2*
1.31
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table A8: Accuracies on classification datasets, averaged across 20 training-testing splits, for obliqueBARII and rotated versions of axis-aligned methods with 200 random rotations. Best performing method bolded and accuracies that are statistically significantly worse than obliqueBARTTs are marked with an asterisk.
data
obliqueBART
rotBART
rotERT
rotRF
rotXGB
banknote
0.997
0.999
*0.451
0.998
blood transfusion
0.769
0.793
*0.734
*0.748
0.789
breast cancer diag.
0.922
0.973
0.977
*0.635
0.97
breast cancer
0.975
0.975
0.973
*0.647
0.972
breast cancer prog-
0.758
0.759
0.751
*0.303
*0.733
climate crashes
0.917
0.932
0.926
*0.56
0.952
connectionist sonar
0.802
0.801
0.849
*0.474
0.824
credit approval
0.866
*0.859
0.87
*0.453
0.871
echocardiogram
0.714
0.708
0.712
*0.343
*0.695
fertility
0.86
0.86
*0.848
*0.726
*0.84
hepatitis
0.84
0.858
0.858
*0.725
0.842
ILPD
0.714
*0.7
0.711
*0.336
0.708
ionosphere
0.902
*0.881
0.93
*0.361
0.892
ozone 1
0.97
*0.97
*0.968
0..0314
0.969
ozone 8
0.933
0.941
0.939
0.0695
0.939
parkinsons
0.865
0.859
0.911
*0.265
0.894
planning relax
0.717
*0.711
*0.653
*0.283
*0.63
qaar bio.
0.823
0.847
0.851
*0.568
0.857
seismic bumps
0.935
*0.934
*0.919
*0.0648
*0.929
spambase
0.771
0.936
0.951
*0.413
0.945
spectf heart
0.799
0.817
0.813
0.799
0.791
Table A9: Number of random rotations of the data for difference between obliqueBART's and rotated method's MSE to be statistically insignificant. Model and data combinations with f_" had MSE's that were larger than obliqueBART, even with 200 random data rotations.
data
rotBART
rotERT rotRF
rotXGB
abalone ais
amenity
attend
baseball
basketball
boston
budget
cane
50
cpu
diabetes
1
diamonds
edu
labor
mpg
16
rice
servo
100
strikes
21

---

Published in Transactions on Machine Learning Research (04/2025)
Table A10: Number of random rotations of the data for difference between obliqueBART's and rotated method's accuracies to be statistically insignificant. Model and data combinations with s_" had smaller accuracies than obliqeeBRTT, even with 200 random data rotations.
data
rotBART
rotERT
rotRF
rotXGB
banknote
blood transfusion
breast cancer diag.
breast cancer
1
breast cancer prog.
climate crashes
4
connectionist sonar
credit approval
echocardiogram
fertility
16
hepatitis
16
4
ILPD
1
ionosphere
50
ozone 1
ozone
parkinsons
planning relax
16
qsar bio.
seismic bumps
spambase
spectf heart
4
Metropolis-Hastings acceptance probabilities
Grow move. Suppose are updating the m-th decision tree (Tm, Dm) = (T,D) Further suppose that we formed the proposal (T*,D*) by growing (T, D) from an existing leaf nx at depth d(nx) and then drawing a new rule rule to associate with nx in (T*, Let q(rule]T, D) denote the proposal probability of drawing rule at nx. Additionally, let n.eaf(.) and nnog(.) count the number leaf nodes and decision nodes with no grandchildren in a tree. The acceptance probability of grow move decomposes the product of three terms.
(T,,D+T' a(1 d(nx)) a22 + d(nx)) -812
nleaf (T)
aal + d(nx))
nnog (T*)
x -1 x
PaxPPxx
X exp
6 nxl
+
e nxr
+
(B1)
Pax
2P
2P Pxr
2P
x p(rule| TT,D")
((rule|T,1)
The terms in the first two lines of Equation (B1) respectively compare the complexity (i.e., depth) and overall fit to the partial residual R of (T* D*) and (T, D). More specifically, the term in second line tends to be larger than one whenever splitting the tree at nx along rule yiclds better fit to the current partial residual than not splitting the tree at nx. The term in the final line is the ratio between the prior and proposal probability of drawing the rule rule. When we draw rule from the prior, the MH acceptance probability depends only on terms that compare the fit and complexity of the two trees. However, if the proposal distribution is much more sharply concentrated around rule than the prior, this term will artificially deflate the acceptance probability.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Prune move. Suppose instead that we form (T" D by removing leafs nxl and nxr and turning their common parent nx into leaf. Let rule denote the rule associated with nx in (T, D). The acceptance probability of a prune move is
- a(1 + d(nx))
nnog(
a(,,D-
a(1 + d((x)) 8 a(2 d(nx))
-8
nleaf
(T*)
x
Pax
7
exp
6 mx1
nxr
(B2)
Pax Paze
2PPx
2P
2P
(rule) T (D)
p(rule|T,D
C Hyperparameter Sensitivity Analysis
As noted in Section 3, obliqueBART depends on several hyperparameters. In this section, we perform a sensitivity analysis using the benchmark datasets from the experiments Section Specifically, we investigate the sensitivity to the hyperparameters for the sparcity parameter 0, which regulates the number of non-zero entries in Φ. We also assess the scnsitivity to the number of trees, M, on the performance of obliqueBART.
Sensitivity to
We ran obliqueBART with multiple combinations for the hyperparameters on the Beta (as, ba) prior on y/ 1).
Specifically, we ran obliqueBART with as E {1, M), with M 200, and by such that K[0] E { p pt p + Holding by constant, increasing allows obliqueBART to be less sensitive to the data. Recall that the conjugate update for 0 is
0 E, o² Beta (ag + ns, be 26)
So, when ag small, the effect of ns (resp. zg); the number of non-zero (resp. zero) entrics, is more substantial, compared to when ag is large. The other hyperparameter, be is chosen to control the sparsity of Φ and we test the performance of obliqueBART when the prior mean for dependent and independent of p. Figure CS depicts boxplots for out of sample SMSE across all splits and datasets for the difference combinations of (ag, bp).
Regression
Classification
M
21
1:30
1/2
0.0
0.5
10
20
000
0.2
0.50
0.5
1.00
SMSE (out-of-sample)
Accuracy (out-ct-sample)
(a)
(b)
Figure C8: ObliqueBART's SMSE (a) and accuracy (b) across all splits and datasets, compared at different combinations of (ag,bo) Models with lower SMSE's and higher accuracies are preferred.
ObliqueBART seems insensitive to the choices of and bg- For the vast majority of datasets, predictive accuracy is very similar. We note that, generally, setting by so that E[(] is dependent on p tended to produce
better results than ][0] = 1/2. We move forward with ag = M and by set so that [[ = 2/p for the rest of the experiments and recommend this the dcfault sctting for obliqueBART.
Sensitivity to M
23

---

Published in Transactions on Machine Learning Research (04/2025)
Fixing ag to M and by that B[Ø] 2/p, obliqueBART with M {10, 25, 100, 200). Figure C9 shows the resulting boxplots for out of sample SMSE across all splits and benchmark datasets. We find that M = 200 generally produced the best predictive accuracy, although the accuracies were quite similar, but slightly worse, for 100.
Regression
Classification
100
100-
25-
0.00
0.25
0.50
0.76
1.00
SMSE (out- of sample)
Accuracy (out-ol- sal-ol-sample)
Figure C9: ObliqueBART's SMSE (a) and accuracy (b) across all splits and datasets, compared at different levels of M. Models with lower SMSE's and higher accuracies preferred.
24


Reviews:
[{"review_id": "6wtZAYNIvE", "reviewer": "Reviewer_wbnm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a conservative extension of BART. For example, the priors on T\nand M as Chipman et al. This seems a sensible approach: BART is a\nsuccessful approach so it makes sense to 'just' add in the capability\nto have oblique decision rules and see what happens.\n\nThe paper is clearly written and not a hard read. There are no\ntheoretical results, just a description and justification for the\ndesign of oblique BART and empirical results. A useful finding from\nthe empirical work is an indication that the hierarchical\nspike-and-slab prior allows oblique BART to adjust the ratio between\naxis-aligned and oblique rules.\n\nThere are no big surprises in the empirical results. Naturally when\ntrue decision boundaries are oblique a model which allows oblique\ndecision boundaries has an advantage. It was somewhat interesting that\nrandomly rotated versions of BART, RF, ERT, and XGB did fairly\nbadly. On the benchmark datasets oblique-BART's performance \"is generally not\nsignificantly worse than BART's and can sometimes be substantially\nbetter.\" It's good to see the authors not overselling their algorithm.\n\nI think a future version of this paper might well be suitable for\npublication in TMLR: I do not have a problem with the \"incrementality\"\nof the contribution. Most papers make such a contribution (or none at\nall). Some might view the empirical results as mainly negative (for\nexample oblique BART had the worst average classification accuracy of\nall methods), but the point is that we have useful results, whether we\nchoose to interpret them as negative or otherwise.\n\nSMALL POINTS\n\nin (1) etc \"leafs\" -> \"leaves\"\n\n\"Such proposals ... receives\" ->\n\"Such proposals ... receive\"\n\nFigure 4a shows how axis-aligned BART and random rotation.. ->\nFigure 4 shows how axis-aligned BART and random rotation..\n\np11\nit's average accuracy ->\nits average accuracy\n\nMy guess (I should not have to guess) is that the \"average\" SMSE of\n0.296 mentioned on p11 is mean SMSE and the vertical line on the plots\nin Fig 5 indicates the median (which would be a standard for a\nboxplot). The vertical line for XGB is to the left of that for ERT\neven though its \"average\" SMSE (0.342) is higher than that of ERT\n(0.332). It would be good to report median values in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors develop an oblique-split version of the BART\nalgorithm. Since this is a Bayesian approach, this requires choosing a\nprior on the oblique decision rules. The method is presented (after\nsuitable background is given) and empirical results presented.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "What I find missing here is an empirical comparison where there is\n*no* random rotation of the data. My assumption is that oblique-BART\nmight do worse than regular BART (and perhaps also other comparators),\nbut I don't know for sure. Please include that.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "92XmhiE7ET", "reviewer": "Reviewer_oFXc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* The proposed model and algorithm demonstrate novelty, as there has not been a BART variant with oblique splits before.\n* The method of oblique partitioning and prior distributions for the parameters used in partitioning are clearly defined and explained for both continuous and discrete predictor variables.\n* The sampling method from the posterior distribution is technically correct.\n* The model shows expected superior performance on synthetic datasets where oblique BART is intuitively well-suited.\n* Guidelines for hyperparameter tuning are provided.\n\n## Weaknesses\n\n* There are numerous deficiencies in the description.\n* There is room for improvement in the clarity of the derivation of equations (1), (2), and (3).\n* The performance improvement on benchmark datasets is not substantial. However, since significance is not heavily emphasized in TMLR, this is not a major concern.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research proposes a variant of Bayesian Additive Regression Trees (BART) that partitions the feature space using non-axis-aligned hyperplanes, termed as oblique BART. The authors also propose a sampling method from the posterior distribution of oblique BART. The model can handle both continuous and categorical predictor variables and is applicable to both regression and classification problems. On synthetic datasets where oblique BART is intuitively expected to perform well, it demonstrates superior performance compared to conventional methods as expected. In terms of predictive performance on benchmark datasets, oblique BART shows slightly better results than existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n\n* Add more detailed explanations for the derivation of equations (1), (2), and (3)\n  * The derivation of equations (1), (2), and (3) appears to have some logical gaps. Please include the complete transformations based on conditional independence derived from model assumptions without omissions. For instance, I believe equation (1) uses conditional independence of $(\\mathcal{T}, \\mathcal{D})$ and $(\\mathcal{E}^-, \\sigma^2)$ given $\\theta$ after transformations such as:\n$$p(\\mathcal{T}, \\mathcal{D}, \\mathcal{M} | \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) \\propto p( \\boldsymbol{y} | \\mathcal{T}, \\mathcal{D}, \\mathcal{M}, \\mathcal{E}^-, \\sigma^2, \\theta) p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\mathcal{E}^-, \\sigma^2, \\theta) p(\\mathcal{T}, \\mathcal{D} | \\mathcal{E}^-, \\sigma^2, \\theta).$$\nPlease include such complete transformations for equations (2) and (3) as well.\n* Address the following descriptive deficiencies:\n  * Title: Capitalize the first letter of each word appropriately.\n  * Page 2, 3rd line from bottom: $\\mathcal{M}$ is not explicitly defined. Is it a set of $\\mu_l$?\n  * Page 5, line 7: $\\mathrm{Inv. Gamma}(3/2, 3\\lambda/2)$ should include $\\nu$.\n  * Sometimes, the word \"obliqueBART\" is not properly capitalized:\n    * Page 5, 2nd line from bottom\n    * Page 6, 10th line from bottom\n    * Page 11, line 12\n  * Page 6, 17th line from bottom: $\\gamma_p$ should be $\\gamma_{ p_\\mathrm{cont} }$\n  * Before equation (1), either explicitly state that you are considering the case where $(\\mathcal{T}_m, \\mathcal{D}_m, \\mathcal{M}_m) = (\\mathcal{T}, \\mathcal{D}, \\mathcal{M})$, or add subscript $m$ to $\\mathcal{T}$, $\\mathcal{D}$, $\\mathcal{M}$ in equations (1), (2), and (3).\n  * Page 9, 10th line from bottom: Figure 2a should be Figure 2d.\n  * Equation (B1): $\\mathcal{D}^\\star$ is missing in the numerator and $\\mathcal{D}$ is missing in the denominator of the last term.\n  * Equation (B2): There is an unnecessary bracket $]$ at the end of the numerator in the first term.\n  * Equation (B2): The superscripts $\\star$ on $\\mathcal{T}^\\star$ and $\\mathcal{D}^\\star$ in the denominator of the last term are unnecessary.\n\n## Personal Suggestion\n\nThe following modifications are not mandatory for acceptance. Please consider them if you find them agreeable.\n\n### **Regarding the derivation order of equations (1), (2), and (3)**\n\nIn my understanding, equations (2) and (3) are derived by comparing the following two equations, where the first equation should correspond to equation (1) in the manuscript:\n$$\\begin{align*}\n  p(\\mathcal{T}, \\mathcal{D}, \\mathcal{M} | \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) &\\propto p( \\boldsymbol{y} | \\mathcal{T}, \\mathcal{D}, \\mathcal{M}, \\mathcal{E}^-, \\sigma^2, \\theta) p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\mathcal{E}^-, \\sigma^2, \\theta) p(\\mathcal{T}, \\mathcal{D} | \\mathcal{E}^-, \\sigma^2, \\theta) \\\\\\\\\n  p(\\mathcal{T}, \\mathcal{D}, \\mathcal{M} | \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) &= p( \\mathcal{T}, \\mathcal{D} | \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta)\n\\end{align*}$$\nWhile this approach is valid, it might be more straightforward to first derive equation (3) directly using basic properties of probability and the definition of the model:\n$$\\begin{align*}\n&p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) \\\\\\\\\n&\\propto p(\\boldsymbol{y} | \\mathcal{T}, \\mathcal{D}, \\mathcal{M}, \\mathcal{E}^-, \\sigma^2, \\theta)p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\mathcal{E}^-, \\sigma^2, \\theta) \\\\\\\\\n&= (2\\pi)^{-\\frac{n}{2}} \\sigma^{-n} \\exp \\left( -\\frac{1}{2 \\sigma^2} \\sum_{i=1}^n r_i^2 \\right) \\prod_{l: \\mathrm{leaf}} \\frac{1}{\\sqrt{2\\pi \\tau^2}} \\exp \\left( -\\frac{1}{2} P_l (\\mu_l - P_l^{-1} \\Theta_l )^2 + \\frac{ \\Theta_l^2 }{ 2P_l } \\right)  \\qquad ( * ) \\\\\\\\\n&\\propto \\prod_{l: \\mathrm{leaf}} \\exp \\left( -\\frac{1}{2} P_l (\\mu_l - P_l^{-1} \\Theta_l )^2 \\right)\n\\end{align*}$$\nThen, by substituting $( * )$ and using the fact that $\\int \\exp \\left( -\\frac{1}{2} P_l (\\mu_l - P_l^{-1} \\Theta_l )^2 \\right) \\mathrm{d} \\mu_l = \\sqrt{\\frac{2 \\pi}{P_l}}$, equation (2) can be derived:\n$$\\begin{align*}\n&p( \\mathcal{T}, \\mathcal{D} | \\boldsymbol{y}, \\mathcal{E}^-, \\sigma^2, \\theta) \\\\\\\\\n&\\propto p(\\mathcal{T}, \\mathcal{D} | \\mathcal{E}^-, \\sigma^2, \\theta) \\int p(\\boldsymbol{y} | \\mathcal{T}, \\mathcal{D}, \\mathcal{M}, \\mathcal{E}^-, \\sigma^2, \\theta)p(\\mathcal{M} | \\mathcal{T}, \\mathcal{D}, \\mathcal{E}^-, \\sigma^2, \\theta) \\mathrm{d} \\mathcal{M} \\\\\\\\\n&= p(\\mathcal{T}, \\mathcal{D} | \\theta) \\int (2\\pi)^{-\\frac{n}{2}} \\sigma^{-n} \\exp \\left( -\\frac{1}{2 \\sigma^2} \\sum_{i=1}^n r_i^2 \\right) \\prod_{l: \\mathrm{leaf}} \\frac{1}{\\sqrt{2\\pi \\tau^2}} \\exp \\left( -\\frac{1}{2} P_l (\\mu_l - P_l^{-1} \\Theta_l )^2 + \\frac{ \\Theta_l^2 }{ 2P_l } \\right) \\mathrm{d}\\mathcal{M} \\\\\\\\\n&\\propto p(\\mathcal{T}, \\mathcal{D} | \\theta) \\prod_{l: \\mathrm{leaf}} \\tau^{-1} P_l^{-\\frac{1}{2}} \\exp \\left( \\frac{ \\Theta_l^2 }{ 2P_l } \\right)\n\\end{align*}$$\nNote that the second equality uses the conditional independence of $(\\mathcal{T}, \\mathcal{D})$ and $(\\mathcal{E}^-, \\sigma^2)$ given $\\theta$.\n\n### **Regarding the distinction between conventional and proposed methods**\n\nWhile the current manuscript clearly states that the originality of the proposed method lies solely in the generalization of grow proposals, to make the distinction between conventional and proposed methods even clearer, you might consider moving the following sections into Section 2.2.2:\n\n* Section 3.2.1 (excluding the Efficient decision tree update part)\n* Section 3.2.2", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uIKMOba830", "reviewer": "Reviewer_47xj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Generalizing BART to use oblique splitting may seem like a straightforward extension, yet it has not been explored before. This paper may have the potential to contribute an important piece to the machine learning community. Additionally, the paper provides intuitive visualizations using figures and a thorough introduction to both BART and oblique splitting, making it accessible even to readers unfamiliar with these concepts. The approach described in the paper for extending BART with oblique splitting is straightforward and does not introduce significant technical novelty. However, since TMLR’s evaluation criteria do not primarily emphasize technical contribution and significance, this is not a major concern. \n\nOn the other hand, I see several issues with the presentation and experimental demonstration. In particular, the discussion on aspects unique to BART is rather limited. The experiments compare a model using oblique splitting with one using axis-aligned splitting, which is a major limitation. If this were the main focus, traditional oblique decision trees could be used instead. Therefore, it is crucial to clearly present the advantages of using oblique trees specifically within BART. I will elaborate on this point in detail in the Requested Changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for constructing a BART model that performs oblique splitting and evaluates its effectiveness through several experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- The experiments primarily compare the proposed method with axis-aligned models, but this does not seem to differ significantly from the claims made in prior work on oblique trees. For example, nowhere in the paper is a direct comparison made between typical oblique trees and obliqueBART. Furthermore, the discussion on BART-specific characteristics, such as uncertainty estimation, is lacking. As a result, the current presentation does not provide a strong motivation for using oblique BART over a standard oblique tree.\n\n- Due to sampling, proposed obliqueBART is likely to have a higher computational cost compared to simple oblique trees. This aspect should be evaluated. Including such a comparison is essential to justify why obliqueBART is preferable to standard oblique trees.\n\n- Compared to simple axis-aligned decision trees, oblique splitting is more sensitive to feature scaling and handling of missing values, as these factors directly impact performance. However, the paper does not mention how these preprocessing steps were handled.\n\n- Although the Appendix discusses the hyperparameters of oblique BART, the benchmark methods do not tune hyperparameters. This makes the performance comparison unfair.\n\n- Minor points:\n\n  - The introduction and other sections repeatedly emphasize performance improvements using numerical percentages, but this is not a common practice and should be avoided.\n\n  - The quality of figures presenting experimental results is insufficient. For example, Figure 4 lacks error evaluation, and the x-axis of the left-side plot in Figure 4 appears to use decimal notation with an upper limit of $\\pi/4$, which is unintuitive and difficult to interpret.\n\n  - In Section 2, there is a statement suggesting that GBT requires trees to have a fixed shallow depth. However, this is not a general constraint. There are several instances where conventions and specifications seem to be conflated.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a Bayesian Additive Regression Trees (BART) method that incorporates oblique splitting. While the reviewers acknowledged that the techniques used are a combination of existing approaches, they agreed that this work presents the first instance of BART considering oblique splitting, thus considered novel.
However, the reviewers also noted that the significance of this research is somewhat limited. Through discussion, the authors addressed concerns by improving the paper's readability and notation, and by updating the experiments to provide further evidence of the proposed method's effectiveness. Based on these revisions, the paper is now considered suitable for acceptance by TMLR.

The following are the final revisions requested by the reviewers for the paper's acceptance:

* Section 4: "Generally speaking, Generally speaking"
* "smallest classification accuracy" -> "lowest classification accuracy"
* Figure 4 should be more self-contained. This means a longer caption which repeats the information about what information Fig 4 is showing. This information is currently in various places in the main text. Also is the information in Fig 4 for some particular choice of n, or are the RMSEs for different values of n combined?
* In Table A.2, please ensure consistency in the capitalization of terms.
* Reproducible code has not yet been released.
Decision: Accept (poster)