Title: Generative Forests
Venue: NeurIPS 2024
Accepted: yes
Keywords: tabular data

generative models

boosting

trees
Abstract: We focus on generative AI for a type of data that still represent one of the most prevalent form of data: tabular data. We introduce a new powerful class of forest-based models fit for such tasks and a simple training algorithm with strong convergence guarantees in a boosting model that parallels that of the original weak / strong supervised learning setting. This algorithm can be implemented by a few tweaks to the most popular induction scheme for decision tree induction (*i.e. supervised learning*) with two classes. Experiments on the quality of generated data display substantial improvements compared to the state of the art. The losses our algorithm minimize and the structure of our models make them practical for related tasks that require fast estimation of a density given a generative model and an observation (even partially specified): such tasks include missing data imputation and density estimation. Additional experiments on these tasks reveal that our models can be notably good contenders to diverse state of the art methods, relying on models as diverse as (or mixing elements of) trees, neural nets, kernels or graphical models.

Generative Forests

Abstract

We focus on generative AI for a type of data that still represent one of the most prevalent form of data: tabular data. Our paper introduces two key contributions: a new powerful class of forest-based models fit for such tasks and a simple training algorithm with strong convergence guarantees in a boosting model that parallels that of the original weak / strong supervised learning setting. This algorithm can be implemented by a few tweaks to the most popular induction scheme for decision tree induction ( i.e. supervised learning ) with two classes. Experiments on the quality of generated data display substantial improvements compared to the state of the art. The losses our algorithm minimize and the structure of our models make them practical for related tasks that require fast estimation of a density given a generative model and an observation (even partially specified): such tasks include missing data imputation and density estimation. Additional experiments on these tasks reveal that our models can be notably good contenders to diverse state of the art methods, relying on models as diverse as (or mixing elements of) trees, neural nets, kernels or graphical models.

1 Introduction

There is a substantial resurgence of interest in the ML community around tabular data, not just because it is still one of the most prominent kind of data available [6]: it is recurrently a place of sometimes heated debates on what are the best model architectures to solve related problems. For example, even for well-posed problems with decades of theoretical formalization like supervised learning [39], after more than a decade of deep learning disruption [22], there is still much ink spilled in the debate decision forests vs neural nets [28]. Generative AI * makes no exception. Where the consensus has long been established on the best categories of architectures for data like image and text (neural nets), tabular data still flourishes with a variety of model architectures building up - or mixing - elements from knowledge representation, logics, kernel methods, graph theory, and of course neural nets [4, 9, 12, 31, 34, 41, 43, 44] (see Section 2). Because of the remarkable nature of tabular data where a single variable can bring considerable information about a target to model, each of these classes can be a relevant choice at least in some cases (such is is the conclusion of [28] in the context of supervised learning). A key differentiator between model classes is then training and the formal guarantees it can provide [31, 44].

In this paper , we introduce new generative models based on sets of trees that we denote as generative forests (GF), along with a training algorithm which has two remarkable features: it is extremely simple and brings strong convergence guarantees in a weak / strong learning model that parallels that of the original boosting model of Valiant's PAC learning [18]. These guarantees improve upon the best state of the art guarantees [44, 31]. Our training algorithm, GF.BOOST, supports training from data with missing values and is simple enough to be implementable by a few tweaks on the popular induction scheme for decision tree induction with two classes, which is supported by a huge number

domain

1 tree, 50 splits

GF, 50 stumps

of repositories / ML software implementing algorithms like CART or C4.5 [1, 35, 3, 36, 45]. From the model standpoint, generative forests bring a sizeable combinatorial advantage over its closest competitors, generative trees [31] (see Table 1 for an example) and adversarial random forests [44]. Experiments on a variety of simulated or readily available domains display that our models can substantially improve upon state of the art, with models of ours as simple as a set of stumps potentially competing with other approaches building much more complex models.

The models we build have an additional benefit: it is computationally easy to compute the full density given an observation, even partially specified ; hence, our generative models can also be used for side tasks like missing data imputation or density estimation. Additional experiments clearly display that our approach can be a good contender to the state of the art. To save space and preserve readability, all proofs and additional experiments and results are given in an Appendix.

2 Related work

It would not do justice to the large amount of work in the field of "Generative AI" for tabular data to just sample a few of them, so we devote a part of the Appendix to an extensive review of the state of the art. Let us just mention that, unlike for unstructured data like images, there is a huge variety of model types, based on trees [7, 31, 44], neural networks [12, 14, 20, 47], probabilistic circuits [5, 43, 38], kernel methods [4, 34], graphical models [41] (among others: note that some are in fact hybrid models). The closest approaches to ours are [44] and [31], because the models include trees with a stochastic activation of edges to pick leaves, and a leaf-dependent data generation process. While [31] learn a single tree, [44] use a way to generate data from a set of trees - called an adversarial random forest - which is simple: sample a tree, and then sample an observation from the tree. The model is thus simple but not economical: each observation is generated by a single tree only, each of them thus having to represent a good sampler in its own. In our case, generating one observation makes use of all trees (Figure 1, see also Section 4). We also note that the theory part of [44] is limited because it resorts to statistical consistency (infinite sample) and Lipschitz continuity of the target density, with second derivative continuous, square integrable and monotonic. Similarly, [31] resort to a wide set of proper losses, but with a symmetry constraint impeding in a generative context. As we shall see, our theoretical framework does not suffer from such impediments.

3 Basic definitions

Perhaps surprisingly at first glance, this Section introduces generative and supervised loss functions. Indeed, our algorithm, which trains a data generator and whose overall convergence shall be shown on a generative loss, turns out to locally optimize a supervised loss. For the interested reader, the key link, which is of independent interest since it links in our context losses for supervised and generative learning, is established in Lemma A (Appendix).

@ k P N ¡ 0 , let r k s . GLYPH<16> t 1 , 2 , ..., k u . Our notations follow [37]. Let B . GLYPH<16> p π, A , B q denote a binary task , where A , B (and any other measure defined hereafter) are probability measures with the same support, also called domain , X , and π P r 0 , 1 s is a prior . M . GLYPH<16> π GLYPH<4> A GLYPH<0> p 1 GLYPH<1> π q GLYPH<4> B is the corresponding mixture measure. For the sake of simplicity, we assume X bounded hereafter, and note that tricks can be used to remove this assumption [31, Remark 3.3]. In tabular data, each of the dim p X q features

ARFs [44]

GF (this paper)

can be of various types , including categorical, numerical, etc., and associated to a natural measure (counting, Lebesgue, etc.) so we naturally associate X to the product measure, which can thus be of mixed type. We also write X . GLYPH<16> GLYPH<2> d i GLYPH<16> 1 X i , where X i is the set of values that can take on variable i . Several essential measures will be used in this paper, including U , the uniform measure, G , the measure associated to a generator that we learn, R , the empirical measure corresponding to a training sample of observations. Like in [31], we do not investigate generalisation properties.

Loss functions There is a natural problem associated to binary task B , that of estimating the probability that an arbitrary observation x P X was sampled from A - call such positive - or B - call these negative -. To learn a supervised model X GLYPH<209> r 0 , 1 s for such a class probability estimation (CPE) problem, one usually has access to a set of examples where each is a couple (observation, class), the class being in set Y . GLYPH<16> tGLYPH<1> 1 , 1 u (= t negative, positive u ). Examples are drawn i.i.d. according to B . Learning a model is done by minimizing a loss function: when it comes to CPE, any such CPE loss [2] is some glyph[lscript] : Y GLYPH<2> r 0 , 1 s GLYPH<209> R whose expression can be split according to partial losses glyph[lscript] 1 , glyph[lscript] GLYPH<1> 1 , glyph[lscript] p y, u q . GLYPH<16> J y GLYPH<16> 1 K GLYPH<4> glyph[lscript] 1 p u q GLYPH<0> J y GLYPH<16> GLYPH<1> 1 K GLYPH<4> glyph[lscript] GLYPH<1> 1 p u q . Its (pointwise) Bayes risk function is the best achievable loss when labels are drawn with a particular positive base-rate,

where B . GLYPH<16> Bernoulli random variable for class 1. A fundamental property for a CPE loss is properness , encouraging to guess ground truth: glyph[lscript] is proper iff L p p q GLYPH<16> E Y GLYPH<18> B p p q glyph[lscript] p Y , p q , @ p P r 0 , 1 s , and strictly proper if L p p q GLYPH<160> E Y GLYPH<18> B p p q glyph[lscript] p Y , u q , @ u GLYPH<24> p . Strict properness implies strict concavity of Bayes risk. For example, the square loss has glyph[lscript] SQ 1 p u q . GLYPH<16> p 1 GLYPH<1> u q 2 , glyph[lscript] SQ GLYPH<1> 1 p u q . GLYPH<16> u 2 , and, being strictly proper, Bayes risk L SQ p u q . GLYPH<16> u p 1 GLYPH<1> u q . Popular strictly proper ML include the log and Matusita's losses. All these losses are symmetric since glyph[lscript] SQ 1 p u q GLYPH<16> glyph[lscript] SQ GLYPH<1> 1 p 1 GLYPH<1> u q , @ u P p 0 , 1 q and differentiable because both partial losses are differentiable.

In addition to CPE losses, we introduce a set of losses relevant to generative approaches, that are popular in density ratio estimation [29, 40]. For any differentiable and convex F : R GLYPH<209> R , the Bregman divergence with generator F is D F p z } z 1 q . GLYPH<16> F p z qGLYPH<1> F p z 1 qGLYPH<1> p z GLYPH<1> z 1 q F 1 p z 1 q . Given function g : R GLYPH<209> R , the generalized perspective transform of F given g is ˇ F p z q . GLYPH<16> g p z q GLYPH<4> F p z { g p z qq , g being implicit in notation ˇ F [26, 27, 32]. The Likelihood ratio risk of B with respect to A for loss glyph[lscript] is

with g p z q . GLYPH<16> z GLYPH<0> p 1 GLYPH<1> π q{ π in the generalized perspective transform. The prior multiplication is for technical convenience. D glyph[lscript] is non-negative; strict properness is necessary for a key property of D glyph[lscript] : D glyph[lscript] GLYPH<16> 0 iff A GLYPH<16> B almost everywhere [31].

4 Generative forests: models and data generation

Architecture We first introduce the basic building block of our models, trees .

Definition 4.1. A tree Υ is a binary directed tree whose internal nodes are labeled with an observation variable and arcs are consistently labeled with subsets of their tail node's variable domain.

Consistency is an important generative notion; informally, it postulates that the arcs' labels define a partition of the measure's support. To make this notion formal, we proceed need a key definition.

Personalstatus&sex

Number existing credits

{A93,A94,A95}

{A91,A92}

{4,5,6}

{0,1,2,3,7,8}

Numberexistingcredits

Job

{0,1,2,3,4}

{5,6,7,8}

{A172,A173}

{A171, A174}

Personalstatus&sex

{A93,A95}

{A94}

R=UCIGermanCredit

.

For any node ν P N p Υ q (the whole set of nodes of Υ , including leaves), we denote X ν GLYPH<132> X the support of the node. The root has X ν GLYPH<16> X . To get X ν for any other node, we initialize it to X and then descend the tree from the root, progressively updating X ν by intersecting an arc's observation variable's domain in X ν with the sub-domain labelling the arc until we reach ν . Then, a labeling of arcs in a tree is consistent iff it complies with one constraint (C) :

(C) for each internal node ν and its left and right children ν f , ν t (respectively), X ν GLYPH<16> X ν f Y X ν t and the measure of X ν f X X ν t with respect to G is zero.

For example, the first split at the root of a tree is such that the union of the domains at the two arcs equals the domain of the feature labeling the split (see Figure 2). We define our generative models.

Definition 4.2. A generative forest ( GF ), G , is a set of trees, t Υ t u T t GLYPH<16> 1 , associated to measure R (implicit in notation).

Figure 2 shows an example of GF. Following the consistency requirement, any single tree defines a recursive partition of X according to the splits induced by the inner nodes. Such is also the case for a set of trees, where intersections of the supports of tuples of leaves (1 for each tree) define the subsets:

( Λ p Υ q GLYPH<132> N p Υ q is the set of leaves of Υ ). Notably, we can construct the elements of P p G q using the same algorithm that would compute it for 1 tree. First, considering a first tree Υ 1 , we compute the support of a leaf, say X λ 1 , using the algorithm described for the consistency property above. Then, we start again with a second tree Υ 2 but replacing the initial X by X λ 1 , yielding X λ 1 X X λ 2 . Then we repeat with a third tree Υ 3 replacing X by X λ 1 X X λ 2 , and so on until the last tree is processed. This yields one element of P p G q .

Generating one observation Generating one observation relies on a stochastic version of the procedure just described. It ends up in an element of P p G q of positive measure, from which we sample uniformly one observation, and then repeat the process for another observation. To describe the process at length, we make use of two key routines, INIT and STARUPDATE, see Algorithms 1 and 2. INIT initializes "special" nodes in each tree, that are called star nodes , to the root of each tree (notation for a variable v relative to a tree Υ is Υ .v ). Stochastic activation, performed in STARUPDATE, progressively makes star nodes descend in trees. When all star nodes have reached a leaf in their respective tree, an observation is sampled from the intersection of the leaves' domains (which is an element of P p G q ). A Boolean flag, done takes value true when the star node is in the leaf set of the tree, indicating no more STARUPDATE calls for the tree ( . is Iverson's bracket, [21]).

J K STARUPDATE is called with a tree of the GF for which done is false, a subset C of the whole domain and measure R . The first call of this procedure is done with C GLYPH<16> X . When all trees are marked done , C has been "reduced" to some C s P P p G q , where the index reminds that this is the last C we obtain, from which we s ample an observation, uniformly at random in C s. Step 1 in STARUPDATE is

After INIT

STARUPDATE on

ν

GLYPH<14>

GLYPH<16>

Q

STARUPDATE on

ν

GLYPH<14>

GLYPH<16>

R

STARUPDATE on

ν

GLYPH<14>

GLYPH<16>

N

STARUPDATE on

ν

GLYPH<14>

GLYPH<16>

N

STARUPDATE on

ν

GLYPH<14>

GLYPH<16>

Q

X

fundamental: it relies on tossing an unfair coin (a Bernoulli event noted B p p q ), where the head probability p is just the mass of R in X Υ .ν GLYPH<14> t X C relative to C . Hence, if X Υ .ν GLYPH<14> t X C GLYPH<16> C , p GLYPH<16> 1 . There is a simple but important invariant (proof omitted).

Lemma 4.3. In STARUPDATE , it always holds that the input C satisfies C GLYPH<132> X Υ .ν GLYPH<14> .

We have made no comment about the sequence of tree choices over which STARUPDATE is called. Let us call admissible such a sequence that ends up with some C s GLYPH<132> X . T being the number of trees (see INIT), for any sequence v P r T s D p C s q , where D p C s q is the sum of the depths of all the star leaves whose support intersection is C s , we say that v is admissible for C s if there exits a sequence of branchings in Step 1 of STARUPDATE, whose corresponding sequence of trees follows the indexes in v , such that at the end of the sequence all trees are marked done and the last C GLYPH<16> C s . Crucially, the probability to end up in C s using STARUPDATE, given any of its admissible sequences, is the same and equals its mass with respect to R .

The Lemma is simple to prove but fundamental in our context as the way one computes the sequence - and thus the way one picks the trees - does not bias generation: the sequence of tree choices could thus be iterative, randomized, concurrent ( e.g. if trees were distributed), etc., this would not change generation's properties from the standpoint of Lemma 4.4. We defer to Appendix (Section II) three examples of sequence choice. Figure 3 illustrates a sequence and the resulting C s .

Missing data imputation and density estimation with GFs A generative forest is not just a generator: it models a density and the exact value of this density at any observation is easily available. Figure 4 (top row) shows how to compute it. Note that if carried out in parallel, the complexity to get this density is cheap, of order O p max t depth p Υ t qq . If one wants to prevent predicting zero density, one can stop the descent if the next step creates a support with zero empirical measure. A GF thus also models an easily tractable density, but this fact alone is not enough to make it a good density estimator. To get there, one has to factor the loss minimized during training. In our case, as we shall see in the following Section, we train a GF by minimizing an information-theoretic loss directly formulated on this density (2). So, using a GF also for density estimation can be a reasonable additional benefit of training GFs.

The process described above that finds leaves reached by an observation can be extended to missing values in the observation using standard procedures for classification using decision trees. We obtain a simple procedure to carry out missing data imputation instead of density estimation: once a tuple of leaves is reached, one uniformly samples the missing features in the corresponding support. This is fast, but at the expense of a bit more computations, we can have a much better procedure, as explained in Figure 4 (bottom row). In a first step, we compute the density of each support subset corresponding to all (not just 1 as for density estimation) tuples of leaves reached in each tree. This provides us with the full density over the missing values given (i) a partial assignment of the tabular domain's variables and (ii) the GF. We then keep the elements corresponding to the maximal density value and finally simultaneously sample all missing features uniformly in the corresponding domain. Overall, the whole procedure is O p d GLYPH<4> p GLYPH<176> t depth p Υ t qq 2 q .

density estimation

missing data imputation

R

observation

X

Vol()

ai

2

d3

d2

d=0

y

～u([a,b])

a

+partial observation(y"unknown)

partialobservation("yunknown)*

Algorithm 3 GF.BOOST( R , J , T )

5 Learning generative forests using supervised boosting

From the GAN framework to a fully supervised training of generative models It can appear quite unusual to train generative models using a supervised learning framework, so before introducing our algorithm, we provide details on its filiation in the generative world, starting from the popular GAN training framework [12]. In this framework, one trains a generative model against a discriminator model which has no purpose other than to parameterize the generative loss optimized. As shown in [33], there is a generally inevitable slack between the generator and the discriminator losses with neural networks, which translates into uncertainty for training. [31] show that the slack disappears for calibrated models, a property satisfied by their generative trees (and also by our generative forests). Moreover, [31] also show that the GAN training can be simplified and made much more efficient for generative trees by having the discriminator (a decision tree) copy the tree structure of the generator, a setting defined as copycat . Hence, one gets reduced uncertainty and more efficient training. Training still involves two models but it becomes arguably very close to the celebrated boosting framework for top-down induction of decision trees [19], up to the crucial detail that the generator turns out to implements boosting's leveraged "hard" distribution. This training gives guarantees on the likelihood ratio risk we define in (2). Our paper closes the gap with boosting: copycat training can be equivalently simplified to training a single generative model in a supervised (2 classes) framework. The two classes involved are the observed data and the uniform distribution. Using the uniform distribution is allowed by the assumption that the domain is closed, which is reasonable for tabular data but can also be alleviated by reparameterization [31, Remark 3.3]. The (supervised) loss involved for training reduces to the popular concave "entropic"-style losses used in CART, C4.5 and in popular packages like scikit-learn . And of course, minimizing such losses still provides guarantee on the generative loss (2). While it is out of scope to show how copycat training does ultimately simplify, we provide all components of the "end product": algorithm, loss optimized and the link between the minimization of the supervised and generative losses via boosting.

The algorithm To learn a GF, we just have to learn its set of trees. Our training algorithm, GF.BOOST (Algorithm 3), performs a greedy top-down induction. In Step 1, we initialize the set of T trees to T roots. Steps 2.1 and 2.2 choose a tree ( Υ GLYPH<6> ) and a leaf to split ( λ GLYPH<6> ) in the tree. In our implementation, we pick the leaf among all trees which is the heaviest with respect to R . Hence, we merge Steps 2.1 and 2.2. Picking the heaviest leaf is standard to grant boosting in decision tree induction [19]; in our case, there is a second benefit: we tend to learn size-balanced models. For example, during the first J GLYPH<16> T iterations, each of the T roots gets one split because each root has a larger mass (1) than any leaf in a tree with depth ¡ 0 . Step 2.4 splits Υ GLYPH<6> by replacing λ GLYPH<6> by a stump whose corresponding splitting predicate, p , is returned in Step 2.3 using a weak splitter oracle called splitPred . "weak" refers to boosting's weak/strong learning setting [18] and means that we shall only require lightweight assumptions about this oracle; in decision tree induction, this oracle is the key to boosting from such weak assumptions [19]. This will also be the case for our generative models. We now investigate Step 2.3 and splitPred .

The weak splitter oracle splitPred In decision tree induction, a splitting predicate is chosen to reduce an expected Bayes risk (1) ( e.g. that of the log loss [36], square loss [1], etc.). In our case, splitPred does about the same with a catch in the binary task it adresses , which is † B GEN . GLYPH<16> p π, R , U q (our mixture measure is thus M . GLYPH<16> π GLYPH<4> R GLYPH<0> p 1 GLYPH<1> π q GLYPH<4> U ). The corresponding expected Bayes risk that splitPred seeks to minimize is just:

The concavity of L implies L p T q / L p π q . Notation P p . q overloads that in (3) by depending explicitly on the set of trees of G instead of G itself. Regarding the minimization of (4), there are three main differences with classical decision tree induction. The first is computational: in the latter case, (4) is optimized over a single tree: there is thus a single element in P p T q which is affected by the split, X λ GLYPH<6> . In our case, multiple elements in P p T q can be affected by one split, so the optimisation is more computationally demanding but a simple trick allows to keep it largely tractable: we do not need to care about keeping in P p T q support elements with zero empirical measure since they will generate no data. Keeping only elements with strictly positive empirical measure guarantees a size | P p T q| never bigger than the number of observations defining R . The second difference plays to our advantage: compared to classical decision tree induction, a single split generally buys a substantially bigger slack in L p T q in our case. To see this, remark that for the candidate leaf λ GLYPH<6> ,

(because L is concave and GLYPH<176> C P P p T q , C GLYPH<132> X λ GLYPH<6> p R r C s GLYPH<16> p R r X λ GLYPH<6> s ). The top-left term is the contribution of λ GLYPH<6> to L p T q , the bottom-right one its contribution to L pt Υ GLYPH<6> uq (the decision tree case), so the slack gives a proxy to our potential advantage after splitting. The third difference with decision tree induction is the possibility to converge much faster to a good solution in our case. Scrutinizing the two terms, we indeed get that in the decision tree case, the split only gets two new terms. In our case however, there can be up to 2 GLYPH<4> Card pt C P P p T q : C GLYPH<132> X λ GLYPH<6> uq new elements in P p T q .

Boosting Two questions remain: can we quantify the slack in decrease and of course what quality guarantee does it bring for the generative model G whose set of trees is learned by GF.BOOST ? We answer both questions in a single Theorem, which relies on a weak learning assumption that parallels the classical weak learning assumption of boosting:

Definition 5.1. (WLA p γ , κ q ) There exists γ ¡ 0 , κ ¡ 0 such that at each iteration of GF.BOOST , the couple p λ GLYPH<6> , p q chosen in Steps 2.2, 2.3 of GF.BOOST satisfies the following properties: (a) λ GLYPH<6> is not skinny: p M r X λ GLYPH<6> s ¥ 1 { Card p Λ p Υ GLYPH<6> qq , (b) truth values of p moderately correlates with B GEN at λ GLYPH<6> : GLYPH<7> GLYPH<7> p R r p | X λ GLYPH<6> s GLYPH<1> p U r p | X λ GLYPH<6> s GLYPH<7> GLYPH<7> ¥ γ , and finally (c) there is a minimal proportion of real data at λ GLYPH<6> : πp R r X λ GLYPH<6> s{ p M r X λ GLYPH<6> s ¥ κ .

The convergence proof of [19] reveals that both (a) and (b) are jointly made at any split, where our (b) is equivalent to their weak hypothesis assumption where their γ parameter is twice ours. (c)

T

=

J

=20(

GLYPH<6>

)

T

=

J

=100(

GLYPH<6>

)

T

=200,

J

=500

T

=200,

J

=1500

Ground truth

GF

"

"

GF

9

5

3

.

postulates that the leaf split has empirical measure at least a fraction of its uniform measure - thus, of its relative volume. (c) is important to avoid splitting leaves that would essentially be useless for our tasks: a leaf for which (c) is invalid would indeed locally model comparatively tiny density values. Theorem 5.2. Suppose the loss glyph[lscript] is strictly proper and differentiable . Let G 0 (= U ) denote the initial GF with T roots in its trees (Step 1, GF.BOOST ) and G J the final GF , assuming wlog that the number of boosting iterations J is a multiple of T . Under WLA p γ , κ q , we get the following on likelihood ratio risks: D glyph[lscript] p R , G J q / D glyph[lscript] p R , G 0 q GLYPH<1> κ γ 2 κ 2 8 GLYPH<4> T log GLYPH<0> 1 GLYPH<0> J T GLYPH<8> , for some constant κ ¡ 0

It comes from [25, Remark 1] that we can choose κ GLYPH<16> inf t glyph[lscript] 1 GLYPH<1> 1 GLYPH<1> glyph[lscript] 1 1 u , which is ¡ 0 if glyph[lscript] is strictly proper and differentiable. Strict properness is also essential for the loss to guarantee that minimization gets to a good generator (Section 3).

6 Experiments

Our code is provided and commented in Appendix, Section V.2. The main setting of our experiments is realistic data generation ('LIFELIKE'), but we have also tested our method for missing data imputation ('IMPUTE') and density estimation ('DENSITY'): for this reason, we have selected a broad panel of state of the art approaches to test against, relying on models as diverse as (or mixing elements of) trees, neural nets, kernels or graphical models, with MICE [42], adversarial random forests (ARFs [44]), CT-GANs [47], Forest Flows [17], Vine copulas auto-encoders (VCAE, [41]) and Kernel density estimation (KDE, [4, 34]). All algorithms not using neural networks were ran on a low-end CPU laptop - this was purposely done for our technique. Neural network-based algorithms are run on a bigger machine (technical specs in Appendix, Section V.2). We carried out experiments on a total of 21 datasets, from UCI [10], Kaggle, OpenML, the Stanford Open Policing Project, or simulated. All are presented in Appendix, Section V.1. We summarize results that are presented in extenso in Appendix. Before starting, we complete the 2D heatmap of Table 1 by another one showing our models can also learn deterministic dependences in real-world domains (Table 2). The Appendix also provides an example experiment on interpreting our models for a sensitive domain (in Section V.V.3.1).

Generation capabilities of our models: LIFELIKE The objective of the experiment is to evaluate whether a generative model is able to create "realistic" data. The evaluation pipeline is simple: we create for each domain a 5-fold stratified experiment. Evaluation is done via four metrics: a regularized optimal transport ("Sinkhorn", [8]), coverage and density [30] and the F1 score [17]. All metrics are obtained after generating a sample of the same size as the test fold. Sinkhorn evaluates an optimal transport distance between generated and real and F1 estimates the error of a classifier (a 5-nearest neighbor) to distinguish generated vs real (smaller is better for both). Coverage and density are refinements of precision and recall for generative models (the higher, the better). Due to size constraint, we provide results on one set of parameters for "medium-sized" generative forests with T GLYPH<16> 500 trees, J GLYPH<16> 2 000 total splits (Table 5), thus learning very small trees with an average 4 splits per tree. In the Appendix, we provide additional results on even smaller models ( T GLYPH<16> 200 , J GLYPH<16> 500 )

UCI

abalone

OpenML

analcatdata_supreme

perr

(cat. variables)

rmse

(num. variables)

perr

rmse

vs GF,

J

GLYPH<16>

20

0.75

14

err (categorical variables)

0.55

0.7

generative tree

marginal

0.5

marginal

120

110

marginal

err

0.65

0.55

0.6

0.5

arginal

generativforests

8

6

4

generative tree

generative forests

imputation

0.45

0.35

0.25

0.3

0.4

nice.rf (4000 trees)

generative tree

100

0000700504030

mice.rf (4000 trees)

generative tree

0.45

mice.n(4500 trees)

2

mice.rf（4500 tree)

0.2

generative forests

generative forests

0.4

0

0.15

20

2

4

6

8

#trees (T)

10

1214161820

2

4

6

8

#trees (T)

10

12

14

16

18

20

2

4

6

8

#trees (T)

10

12

1416

18

20

2

4

6

8

#trees (T)

10

12

14161820

and additional results on one metric (Sinkhorn) against contenders with various parameterizations on more domains. In Table 5, contenders are parameterized as follows: ARFs learn sets of 200 trees. Since each tree has to be separately a good generative model, we end up with big models in general. CT-GANs are trained for 1 000 epochs. Forest Flows and VCAEs are run with otherwise default parameters. Table 5 contains just a subset of all our domains, furthermore not the same for each contender. This benchmark was crafted to compare against Forest Flows, our best LIFELIKE contender but a method requiring to re-encode categorical variables. Since our method and ARFs process data natively, we selected only domains with numerical or binary variables (8 domains). On some of these domains, CT-GANs or VCAE crashed on some folds so we do not provide the results for the related domains. Globally, Generative Forests are very competitive against each contender, significantly beating all VCAEs on all metrics and CT-GANs on coverage and density. Tree-based contender methods appear to perform substantially better than neural networks, with FFs performing better than ARFs. Ultimately, our models compare favorably or very favorably against both FFs and ARFs, while on average being much smaller - for example, each FF model can contain up to 6 750 total splits. As our experiments demonstrate (Appendix, Table A6), learning much smaller generative forests with 500 total splits can still buy an advantage in most cases, a notable counterexample being density for FFs. From the memory standpoint, our code (Java) is not optimized yet managed to crunch domains of up to m GLYPH<2> d GLYPH<19> 1.5 M while always using less than 1.5Gb memory to train our biggest models. Finally, it clearly emerges from our experiments that there is a domain-dependent "ideal" size ( T, J ) for the best generative forests. "Guessing" the right size is a prospect for future work. Missing data imputation with our models: IMPUTE We compared our method against a powerful contender, MICE [42], which relies on round-robin prediction of missing values using supervised models. We optimized MICE by choosing as supervised models trees (CART) and random forests (RFs, we increased the number of trees to 100 for better results). We create a 5-fold experiment; in each fold, we remove a fixed % of observed values (data missing completely at random, MCAR). We then use the data with missing values as input to MICE or us to learn models which are then used to predict the missing values. We compute a per-feature discrepancy, the average error probability ( perr , for categorical variables), and the root mean square error ( rmse , numerical variables). We also compute one of the simplest baselines, which consists in imputing each variable from its empirical marginal . Results summary Table 4 puts the spotlight on two domains. The Appendix provides many more plots. From all results, we conclude that generative forests can compete or beat MICE.RF while using hundred times less trees (eventually using just stumps, when J GLYPH<16> T GLYPH<16> 20 ). From all our results, we also conclude that there is a risk of overfitting if J and / or T are too large. This could motivate further work on pruning generative models. For GF, an unexpected pattern is that the pattern "small number of small trees works well" can work on real-world domains as well (see also Appendix), thereby demonstrating that the nice result displayed in Table 1 generalises to bigger domains. Density estimation: DENSITY We compared GF vs kernel density estimation (KDE) [4, 24, 34]. The experimental setting is the same as for LIFELIKE: in each of the 5-fold stratified cross validation fold, we use the training sample to learn a model (with KDE or GF.BOOST) which is then used to predict the observation's density in the the test sample. The higher the density, the better the model. The

Domain

Sinkhorn

GLYPH<211>

Coverage

GLYPH<210>

Density

GLYPH<210>

F1 measure

GLYPH<211>

us (GF)

ARF

pval

us (GF)

ARF

pval

us (GF)

ARF

pval

us (GF)

ARF

pval

ring

GLYPH<14>

0.285

GLYPH<8>

0.008

0.286

GLYPH<8>

0.007

0.62

GLYPH<14>

0.968

GLYPH<8>

0.010

0.960

GLYPH<8>

0.008

0.05

GLYPH<14>

1.031

GLYPH<8>

0.049

0.976

GLYPH<8>

0.030

0.11

GLYPH<14>

0.070

GLYPH<8>

0.030

0.086

GLYPH<8>

0.047

0.54

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.355

GLYPH<8>

0.005

0.33

0.945

GLYPH<8>

0.015

GLYPH<14>

0.962

GLYPH<8>

0.021

0.35

GLYPH<14>

0.993

GLYPH<8>

0.045

0.954

GLYPH<8>

0.011

0.05

0.514

GLYPH<8>

0.529

GLYPH<14>

0.507

GLYPH<8>

0.032

0.92

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.394

GLYPH<8>

0.002

0.02

GLYPH<14>

0.975

GLYPH<8>

0.009

0.908

GLYPH<8>

0.010

glyph[trianglesolid]

GLYPH<14>

0.995

GLYPH<8>

0.077

0.630

GLYPH<8>

0.050

glyph[trianglesolid]

GLYPH<14>

0.017

GLYPH<8>

0.015

0.043

GLYPH<8>

0.012

0.04

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.288

GLYPH<8>

0.002

0.36

GLYPH<14>

0.961

GLYPH<8>

0.012

0.953

GLYPH<8>

0.004

0.29

GLYPH<14>

0.979

GLYPH<8>

0.023

0.940

GLYPH<8>

0.027

glyph[trianglesolid]

GLYPH<14>

0.014

GLYPH<8>

0.013

0.029

GLYPH<8>

0.010

glyph[trianglesolid]

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.099

GLYPH<8>

0.031

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.929

GLYPH<8>

0.010

0.14

GLYPH<14>

0.961

GLYPH<8>

0.026

0.801

GLYPH<8>

0.028

glyph[trianglesolid]

GLYPH<14>

0.459

GLYPH<8>

0.031

0.531

GLYPH<8>

0.028

glyph[trianglesolid]

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.150

GLYPH<8>

0.021

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.946

GLYPH<8>

0.012

0.10

GLYPH<14>

0.989

GLYPH<8>

0.017

0.941

GLYPH<8>

0.013

glyph[trianglesolid]

GLYPH<14>

0.492

GLYPH<8>

0.037

0.500

GLYPH<8>

0.027

0.04

comp

GLYPH<14>

0.532

GLYPH<8>

0.008

0.535

GLYPH<8>

0.033

0.86

0.556

GLYPH<8>

0.021

GLYPH<14>

0.560

GLYPH<8>

0.037

0.75

0.430

GLYPH<8>

0.008

GLYPH<14>

0.440

GLYPH<8>

0.011

0.11

GLYPH<14>

0.496

GLYPH<8>

0.021

0.520

GLYPH<8>

0.020

0.26

arti

GLYPH<14>

0.821

GLYPH<8>

0.004

0.849

GLYPH<8>

0.010

glyph[trianglesolid]

GLYPH<14>

0.947

GLYPH<8>

0.003

0.892

GLYPH<8>

0.014

glyph[trianglesolid]

GLYPH<14>

0.967

GLYPH<8>

0.013

0.747

GLYPH<8>

0.013

glyph[trianglesolid]

GLYPH<14>

0.429

GLYPH<8>

0.056

0.512

GLYPH<8>

0.037

glyph[trianglesolid]

us (GF)

CT

pval

us (GF)

CT

pval

us (GF)

CT

pval

us (GF)

CT

pval

ring

GLYPH<14>

0.285

GLYPH<8>

0.008

0.351

GLYPH<8>

0.042

0.01

GLYPH<14>

0.968

GLYPH<8>

0.010

0.798

GLYPH<8>

0.050

glyph[trianglesolid]

GLYPH<14>

1.031

GLYPH<8>

0.049

0.757

GLYPH<8>

0.025

glyph[trianglesolid]

GLYPH<14>

0.070

GLYPH<8>

0.030

0.100

GLYPH<8>

0.073

0.40

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.435

GLYPH<8>

0.075

0.07

GLYPH<14>

0.945

GLYPH<8>

0.015

0.734

GLYPH<8>

0.041

glyph[trianglesolid]

GLYPH<14>

0.993

GLYPH<8>

0.045

0.401

GLYPH<8>

0.033

glyph[trianglesolid]

GLYPH<14>

0.514

GLYPH<8>

0.529

0.746

GLYPH<8>

0.033

glyph[trianglesolid]

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.408

GLYPH<8>

0.019

0.01

GLYPH<14>

0.975

GLYPH<8>

0.009

0.828

GLYPH<8>

0.053

glyph[trianglesolid]

GLYPH<14>

0.995

GLYPH<8>

0.077

0.649

GLYPH<8>

0.058

glyph[trianglesolid]

GLYPH<14>

0.017

GLYPH<8>

0.015

0.034

GLYPH<8>

0.011

0.21

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.327

GLYPH<8>

0.024

0.03

GLYPH<14>

0.961

GLYPH<8>

0.012

0.659

GLYPH<8>

0.049

glyph[trianglesolid]

GLYPH<14>

0.979

GLYPH<8>

0.023

0.582

GLYPH<8>

0.035

glyph[trianglesolid]

GLYPH<14>

0.014

GLYPH<8>

0.013

0.079

GLYPH<8>

0.053

0.06

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.384

GLYPH<8>

0.047

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.808

GLYPH<8>

0.016

glyph[trianglesolid]

GLYPH<14>

0.961

GLYPH<8>

0.026

0.589

GLYPH<8>

0.129

glyph[trianglesolid]

GLYPH<14>

0.459

GLYPH<8>

0.031

0.654

GLYPH<8>

0.030

glyph[trianglesolid]

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.158

GLYPH<8>

0.009

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.894

GLYPH<8>

0.026

glyph[trianglesolid]

GLYPH<14>

0.989

GLYPH<8>

0.017

0.953

GLYPH<8>

0.035

0.05

GLYPH<14>

0.492

GLYPH<8>

0.037

0.581

GLYPH<8>

0.043

glyph[trianglesolid]

us (GF)

FF

pval

us (GF)

FF

pval

us (GF)

FF

pval

us (GF)

FF

pval

ring

0.285

GLYPH<8>

0.008

GLYPH<14>

0.276

GLYPH<8>

0.001

0.07

GLYPH<14>

0.968

GLYPH<8>

0.010

0.957

GLYPH<8>

0.020

0.09

1.031

GLYPH<8>

0.049

GLYPH<14>

1.045

GLYPH<8>

0.020

0.28

0.070

GLYPH<8>

0.030

GLYPH<14>

0.051

GLYPH<8>

0.031

0.07

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.354

GLYPH<8>

0.003

0.27

0.945

GLYPH<8>

0.020

GLYPH<14>

0.956

GLYPH<8>

0.015

0.25

GLYPH<14>

0.993

GLYPH<8>

0.045

0.989

GLYPH<8>

0.027

0.79

GLYPH<14>

0.514

GLYPH<8>

0.529

0.530

GLYPH<8>

0.028

0.46

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.393

GLYPH<8>

0.001

0.04

GLYPH<14>

0.975

GLYPH<8>

0.009

0.954

GLYPH<8>

0.012

0.01

0.995

GLYPH<8>

0.077

GLYPH<14>

1.013

GLYPH<8>

0.050

0.21

GLYPH<14>

0.017

GLYPH<8>

0.015

0.045

GLYPH<8>

0.007

glyph[trianglesolid]

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.287

GLYPH<8>

0.001

0.43

GLYPH<14>

0.961

GLYPH<8>

0.012

0.927

GLYPH<8>

0.011

0.02

0.979

GLYPH<8>

0.023

GLYPH<14>

1.000

GLYPH<8>

0.008

0.05

GLYPH<14>

0.014

GLYPH<8>

0.013

0.028

GLYPH<8>

0.008

0.09

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.030

GLYPH<8>

0.029

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.954

GLYPH<8>

0.021

0.71

0.961

GLYPH<8>

0.026

GLYPH<14>

1.001

GLYPH<8>

0.034

0.04

0.459

GLYPH<8>

0.031

GLYPH<14>

0.458

GLYPH<8>

0.052

0.97

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.097

GLYPH<8>

0.007

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.945

GLYPH<8>

0.007

glyph[trianglesolid]

GLYPH<14>

0.989

GLYPH<8>

0.017

0.970

GLYPH<8>

0.023

0.13

GLYPH<14>

0.492

GLYPH<8>

0.037

0.498

GLYPH<8>

0.041

0.78

comp

GLYPH<14>

0.532

GLYPH<8>

0.007

0.891

GLYPH<8>

0.007

glyph[trianglesolid]

GLYPH<14>

0.556

GLYPH<8>

0.021

0.548

GLYPH<8>

0.027

0.32

GLYPH<14>

0.430

GLYPH<8>

0.008

0.404

GLYPH<8>

0.013

glyph[trianglesolid]

GLYPH<14>

0.496

GLYPH<8>

0.021

0.526

GLYPH<8>

0.018

0.04

arti

GLYPH<14>

0.821

GLYPH<8>

0.004

0.834

GLYPH<8>

0.016

0.12

GLYPH<14>

0.947

GLYPH<8>

0.003

0.879

GLYPH<8>

0.008

glyph[trianglesolid]

GLYPH<14>

0.967

GLYPH<8>

0.013

0.774

GLYPH<8>

0.016

glyph[trianglesolid]

GLYPH<14>

0.429

GLYPH<8>

0.056

0.530

GLYPH<8>

0.032

glyph[trianglesolid]

us (GF)

pval

us (GF)

pval

us (GF)

pval

us (GF)

pval

ring

GLYPH<14>

0.285

GLYPH<8>

0.008

0.392

GLYPH<8>

0.005

glyph[trianglesolid]

GLYPH<14>

0.968

GLYPH<8>

0.010

0.364

GLYPH<8>

0.037

glyph[trianglesolid]

GLYPH<14>

1.031

GLYPH<8>

0.049

0.132

GLYPH<8>

0.013

glyph[trianglesolid]

GLYPH<14>

0.070

GLYPH<8>

0.030

0.291

GLYPH<8>

0.035

glyph[trianglesolid]

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.415

GLYPH<8>

0.012

glyph[trianglesolid]

GLYPH<14>

0.945

GLYPH<8>

0.015

0.620

GLYPH<8>

0.020

glyph[trianglesolid]

GLYPH<14>

0.993

GLYPH<8>

0.045

0.265

GLYPH<8>

0.015

glyph[trianglesolid]

GLYPH<14>

0.514

GLYPH<8>

0.529

0.805

GLYPH<8>

0.012

glyph[trianglesolid]

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.400

GLYPH<8>

0.003

glyph[trianglesolid]

GLYPH<14>

0.975

GLYPH<8>

0.009

0.165

GLYPH<8>

0.037

glyph[trianglesolid]

GLYPH<14>

0.995

GLYPH<8>

0.077

0.042

GLYPH<8>

0.012

glyph[trianglesolid]

GLYPH<14>

0.017

GLYPH<8>

0.015

0.065

GLYPH<8>

0.002

glyph[trianglesolid]

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.414

GLYPH<8>

0.003

glyph[trianglesolid]

GLYPH<14>

0.961

GLYPH<8>

0.012

0.317

GLYPH<8>

0.033

glyph[trianglesolid]

GLYPH<14>

0.979

GLYPH<8>

0.023

0.156

GLYPH<8>

0.018

glyph[trianglesolid]

GLYPH<14>

0.014

GLYPH<8>

0.013

0.224

GLYPH<8>

0.019

glyph[trianglesolid]

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.105

GLYPH<8>

0.042

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.913

GLYPH<8>

0.013

0.04

GLYPH<14>

0.961

GLYPH<8>

0.026

0.821

GLYPH<8>

0.041

glyph[trianglesolid]

GLYPH<14>

0.459

GLYPH<8>

0.031

0.537

GLYPH<8>

0.007

glyph[trianglesolid]

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.181

GLYPH<8>

0.019

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.938

GLYPH<8>

0.008

glyph[trianglesolid]

GLYPH<14>

0.989

GLYPH<8>

0.017

0.907

GLYPH<8>

0.026

glyph[trianglesolid]

GLYPH<14>

0.492

GLYPH<8>

0.037

0.527

GLYPH<8>

0.028

glyph[trianglesolid]

us (GF)

VC-D

pval

us (GF)

VC-D

pval

us (GF)

VC-D

pval

us (GF)

VC-D

pval

ring

GLYPH<14>

0.285

GLYPH<8>

0.008

0.390

GLYPH<8>

0.011

glyph[trianglesolid]

GLYPH<14>

0.968

GLYPH<8>

0.010

0.331

GLYPH<8>

0.067

glyph[trianglesolid]

GLYPH<14>

1.031

GLYPH<8>

0.049

0.122

GLYPH<8>

0.028

glyph[trianglesolid]

GLYPH<14>

0.070

GLYPH<8>

0.030

0.319

GLYPH<8>

0.037

glyph[trianglesolid]

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.411

GLYPH<8>

0.004

glyph[trianglesolid]

GLYPH<14>

0.945

GLYPH<8>

0.015

0.649

GLYPH<8>

0.055

glyph[trianglesolid]

GLYPH<14>

0.993

GLYPH<8>

0.045

0.269

GLYPH<8>

0.026

glyph[trianglesolid]

GLYPH<14>

0.514

GLYPH<8>

0.529

0.813

GLYPH<8>

0.018

glyph[trianglesolid]

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.398

GLYPH<8>

0.002

glyph[trianglesolid]

GLYPH<14>

0.975

GLYPH<8>

0.009

0.162

GLYPH<8>

0.034

glyph[trianglesolid]

GLYPH<14>

0.995

GLYPH<8>

0.077

0.043

GLYPH<8>

0.009

glyph[trianglesolid]

GLYPH<14>

0.017

GLYPH<8>

0.015

0.064

GLYPH<8>

0.000

glyph[trianglesolid]

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.414

GLYPH<8>

0.003

glyph[trianglesolid]

GLYPH<14>

0.961

GLYPH<8>

0.012

0.312

GLYPH<8>

0.040

glyph[trianglesolid]

GLYPH<14>

0.979

GLYPH<8>

0.023

0.149

GLYPH<8>

0.018

glyph[trianglesolid]

GLYPH<14>

0.014

GLYPH<8>

0.013

0.225

GLYPH<8>

0.017

glyph[trianglesolid]

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.383

GLYPH<8>

0.037

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.868

GLYPH<8>

0.042

0.02

GLYPH<14>

0.961

GLYPH<8>

0.026

0.738

GLYPH<8>

0.030

glyph[trianglesolid]

GLYPH<14>

0.459

GLYPH<8>

0.031

0.587

GLYPH<8>

0.019

glyph[trianglesolid]

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.484

GLYPH<8>

0.056

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.876

GLYPH<8>

0.027

glyph[trianglesolid]

GLYPH<14>

0.989

GLYPH<8>

0.017

0.891

GLYPH<8>

0.010

glyph[trianglesolid]

GLYPH<14>

0.492

GLYPH<8>

0.037

0.608

GLYPH<8>

0.037

glyph[trianglesolid]

us (GF)

VC-R

pval

us (GF)

VC-R

pval

us (GF)

VC-R

pval

us (GF)

VC-R

pval

ring

GLYPH<14>

0.285

GLYPH<8>

0.008

0.388

GLYPH<8>

0.004

glyph[trianglesolid]

GLYPH<14>

0.968

GLYPH<8>

0.010

0.331

GLYPH<8>

0.065

glyph[trianglesolid]

GLYPH<14>

1.031

GLYPH<8>

0.049

0.124

GLYPH<8>

0.022

glyph[trianglesolid]

GLYPH<14>

0.070

GLYPH<8>

0.030

0.322

GLYPH<8>

0.023

glyph[trianglesolid]

circ

GLYPH<14>

0.351

GLYPH<8>

0.005

0.402

GLYPH<8>

0.003

glyph[trianglesolid]

GLYPH<14>

0.945

GLYPH<8>

0.015

0.664

GLYPH<8>

0.017

glyph[trianglesolid]

GLYPH<14>

0.993

GLYPH<8>

0.045

0.274

GLYPH<8>

0.006

glyph[trianglesolid]

GLYPH<14>

0.514

GLYPH<8>

0.529

0.806

GLYPH<8>

0.022

glyph[trianglesolid]

grid

GLYPH<14>

0.390

GLYPH<8>

0.002

0.399

GLYPH<8>

0.003

glyph[trianglesolid]

GLYPH<14>

0.975

GLYPH<8>

0.009

0.153

GLYPH<8>

0.032

glyph[trianglesolid]

GLYPH<14>

0.995

GLYPH<8>

0.077

0.038

GLYPH<8>

0.006

glyph[trianglesolid]

GLYPH<14>

0.017

GLYPH<8>

0.015

0.065

GLYPH<8>

0.001

glyph[trianglesolid]

rand

GLYPH<14>

0.286

GLYPH<8>

0.003

0.417

GLYPH<8>

0.009

glyph[trianglesolid]

GLYPH<14>

0.961

GLYPH<8>

0.012

0.315

GLYPH<8>

0.023

glyph[trianglesolid]

GLYPH<14>

0.979

GLYPH<8>

0.023

0.145

GLYPH<8>

0.021

glyph[trianglesolid]

GLYPH<14>

0.014

GLYPH<8>

0.013

0.229

GLYPH<8>

0.022

glyph[trianglesolid]

wred

GLYPH<14>

0.980

GLYPH<8>

0.032

1.365

GLYPH<8>

0.098

glyph[trianglesolid]

GLYPH<14>

0.962

GLYPH<8>

0.028

0.839

GLYPH<8>

0.027

glyph[trianglesolid]

GLYPH<14>

0.961

GLYPH<8>

0.026

0.716

GLYPH<8>

0.072

glyph[trianglesolid]

GLYPH<14>

0.459

GLYPH<8>

0.031

0.596

GLYPH<8>

0.047

glyph[trianglesolid]

wwhi

GLYPH<14>

1.064

GLYPH<8>

0.003

1.353

GLYPH<8>

0.038

glyph[trianglesolid]

GLYPH<14>

0.963

GLYPH<8>

0.002

0.747

GLYPH<8>

0.014

glyph[trianglesolid]

GLYPH<14>

0.989

GLYPH<8>

0.017

0.942

GLYPH<8>

0.029

0.02

GLYPH<14>

0.492

GLYPH<8>

0.037

0.729

GLYPH<8>

0.024

glyph[trianglesolid]

GF models we consider are the same as in LIFELIKE ( T GLYPH<16> 500 , J GLYPH<16> 2 000). Results summary Table (3, right) summarizes the results (plots: Appendix, Section V.V.3.7). The leftmost column ("GF " KDE") counts domains for which there exists an iteration j for GF after which we tend to be statistically better (and never statistically worse) than KDE. The rightmost column ("KDE " GF") counts domains for which KDE is always statistically better than GF. The last type of plots appears to be those for which there is no such statistical difference (central column). The results support the conclusion that GF can also be good models to carry out density estimation.

7 Conclusion

In this paper, we have introduced new generative models based on ensemble of trees and a top-down boosting-compliant training algorithm which casts training in a supervised 2-classes framework. Those generative forests allow not just efficient data generation: they also allow efficient missing data imputation and density estimation. Experiments have been carried out on all three problems against a variety of contenders, demonstrating the ability of small generative forests to perform well against potentially much more complex contenders. A responsible use of such models necessarily includes restriction to tabular data (Section 1). Experiments clearly show that there is a domain-dependent "best fit" size for our models; estimating it can be done via cross-validation but an important question, left for future work, is how to directly "guess" the right model size. A pruning algorithm with good generalization seems like a very promising direction.

Acknowledgments

The authors thank the reviewers for engaging discussions and many suggestions that helped to improve the paper's content.

References

L. Breiman, J. H. Freidman, R. A. Olshen, and C. J. Stone. Classification and regression trees . Wadsworth, 1984.

A. Buja, W. Stuetzle, and Y. Shen. Loss functions for binary class probability estimation ans classification: structure and applications, 2005. Technical Report, University of Pennsylvania.

T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In 22 nd KDD , pages 785-794, 2016.

Y.-C. Chen. A tutorial on kernel density estimation and recent advances. Biostatistics & Epidemiology , 1:161-187, 2017.

Y. Choi, A. Vergari, and G. Van den Broeck. Probabilistic circuits: a unifying framework for tractable probabilistic models, 2020. http://starai.cs.ucla.edu/papers/ProbCirc20. pdf .

M. Chui, J. Manyika, M. Miremadi, N. Henke, R. Chung P. Nel, and S. Malhotra. Notes from the AI frontier . McKinsey Global Institute, 2018.

A.-H.-C. Correia, R. Peharz, and C.-P. de Campos. Joints in random forests. In NeurIPS'20 , 2020.

M. Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In NIPS*26 , pages 2292-2300, 2013.

A. Darwiche. A logical approach to factoring belief networks. In Dieter Fensel, Fausto Giunchiglia, Deborah L. McGuinness, and Mary-Anne Williams, editors, KR'02 , pages 409420. Morgan Kaufmann, 2002.

D. Dua and C. Graff. UCI machine learning repository, 2021.

V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A.-C. Courville. Adversarially learned inference. In ICLR'17 . OpenReview.net, 2017.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS*27 , pages 2672-2680, 2014.

L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on tabular data? In NeurIPS'22 Datasets and Benchmarks , 2022.

A. Grover and S. Ermon. Boosted generative models. In AAAI'18 , pages 3077-3084. AAAI Press, 2018.

G.-E. Hinton. The forward-forward algorithm: Some preliminary investigations. CoRR , abs/2212.13345, 2022.

R.C. Holte. Very simple classification rules perform well on most commonly used datasets. MLJ , 11:63-91, 1993.

A. Jolicoeur-Martineau, K. Fatras, and T. Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. In AISTATS'24 , volume 238, pages 1288-1296. PMLR, 2024.

M.J. Kearns. Thoughts on hypothesis boosting, 1988. ML class project.

M.J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proc. of the 28 th ACM STOC , pages 459-468, 1996.

D.-P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR'14 , 2014.

D.-E. Knuth. Two notes on notation. The American Mathematical Monthly , 99(5):403-422, 1992.

A. Krizhevsky, I. Sutskever, and G.-E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS*25 , pages 1106-1114, 2012.

S. Lang, M. Mundt, F. Ventola, R. Peharz, and K. Kersting. Elevating perceptual sample quality in pcs through differentiable sampling. In NeurIPS 2021 Workshop on Pre-Registration in Machine Learning, 13 December 2021, Virtual , volume 181 of Proceedings of Machine Learning Research , pages 1-25. PMLR, 2021.

Q. Li and J.-S. Racine. Nonparametric estimation of distributions with categorical and continuous data. Journal of Multivariate Analysis , 86:266-292, 2003.

Y. Mansour, R. Nock, and R.-C. Williamson. Random classification noise does not defeat all convex potential boosters irrespective of model choice. In ICML'23 , 2023.

P. Maréchal. On a functional operation generating convex functions, part 1: duality. J. of Optimization Theory and Applications , 126:175-189, 2005.

P. Maréchal. On a functional operation generating convex functions, part 2: algebraic properties. J. of Optimization Theory and Applications , 126:375-366, 2005.

D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When do neural nets outperform boosted trees on tabular data? In NeurIPS'23 Datasets and Benchmarks , 2023.

A. Menon and C.-S. Ong. Linking losses for density ratio and class-probability estimation. In 33 rd ICML , pages 304-313, 2016.

M.-F. Naeem, S.-J. Oh, Y. Uh, Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. In ICML'20 , volume 119 of Proceedings of Machine Learning Research , pages 7176-7185. PMLR, 2020.

R. Nock and M. Guillame-Bert. Generative trees: Adversarial and copycat. In 39 th ICML , pages 16906-16951, 2022.

R. Nock, A.-K. Menon, and C.-S. Ong. A scaled Bregman theorem with applications. In NIPS*29 , pages 19-27, 2016.

S. Nowozin, B. Cseke, and R. Tomioka. f -GAN: training generative neural samplers using variational divergence minimization. In NIPS*29 , pages 271-279, 2016.

E. Parzen. On estimation of a probability density function and mode. The Annals of Mathematical Statistics , 33:1065-1076, 1962.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

J. R. Quinlan. C4.5 : programs for machine learning . Morgan Kaufmann, 1993.

M.-D. Reid and R.-C. Williamson. Information, divergence and risk for binary experiments. JMLR , 12:731-817, 2011.

R. Sánchez-Cauce, I. París, and F.-J. Díez. Sum-product networks: A survey. IEEE Trans.PAMI , 44(7):3821-3839, 2022.

L.-J. Savage. Elicitation of personal probabilities and expectations. J. of the Am. Stat. Assoc. , pages 783-801, 1971.

M. Sugiyama and M. Kawanabe. Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation . Adaptive computation and machine learning. MIT Press, 2012.

N. Tagasovska, D. Ackerer, and T. Vatter. Copulas as high-dimensional generative models: Vine copula autoencoders. In NeurIPS*32 , pages 6525-6537, 2019.

S. van Buuren and K. Groothuis-Oudshoorn. mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software , 45(3):1-67, 2011.

A. Vergari, Y. Choi, and R. Peharz. Probabilistic circuits: Representations, inference, learning and applications, 2022. NeurIPS'22 tutorials.

D.-S. Watson, K. Blesch, J. Kapar, and M.-N. Wright. Adversarial random forests for density and generative modeling. In AISTATS'23 , Proceedings of Machine Learning Research. PMLR, 2023.

I. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations . Morgan Kaufmann, 1999.

C. Xiao, P. Zhong, and C. Zheng. BourGAN: Generative networks with metric embeddings. In NeurIPS'18 , pages 2275-2286, 2018.

L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni. Modeling tabular data using conditional GAN. In NeurIPS*32 , pages 7333-7343, 2019.

Appendix

To differentiate with the numberings in the main file, the numbering of Theorems, etc. is letter-based (A, B, ...).

Table of contents

I Related work

The typical Machine Learning (ML) problem usually contains at least three parts: (i) a training algorithm minimizes (ii) a loss function to output an object whose key part is (iii) a model. The main problem we are interested in is data generation as generally captured by "Generative AI". The type of data we are interested in still represents one of the most prevalent form of data: tabular data [6]. When it comes to tabular data, a singular phenomenon of the data world can be observed: there is a fierce competition on part (iii) above, the models . When data has other forms, like images, the ML community has generally converged to a broad idea of what the best models look like at a high-level for many generic tasks: neural networks ‡ . Tabular data offers no such consensus yet, even on well-defined tasks like supervised learning [13]. In fact, even on such "simple tasks" the consensus is rather that there is no such consensus [28]. It is an understatement to state that in the broader context of all tasks of interest to us, a sense of a truly intense competition emerges, whose "gradient" clearly points towards simultaneously the most complex / expressive / tractable models, as shown in [43, Slides 27, 53]. One can thus end up finding models based on trees [7, 31, 44], neural networks [12, 14, 20, 47], probabilistic circuits [5, 43, 38], kernel methods [4, 34], graphical models [41] (among others: note that some are in fact hybrid models).

So the tabular data world is blessed with a variety of possible models to solve problems like the ones we are interested in, but - and this is another singular phenomenon of the tabular data world -, getting the best solutions is not necessarily a matter of competing on size or compute. In fact, it can be the opposite: striving for model simplicity or (non exclusive) lightweight tuning can substantially pay off [28]. In relative terms, this phenomenon is not new in the tabular data world: it has been known for more than two decades [16]. That it has endured all major revolutions in ML points to the fact that lasting solutions can be conveniently addressing all three parts (i-iii) above at once on models, algorithms and losses.

From this standpoint, the closest approaches to ours are [44] and [31], first and foremost because the models include trees with a stochastic activation of edges to pick leaves, and a leaf-dependent data generation process. While [31] learn a single tree, [44] use a way to generate data from a set of trees called an adversarial random forest - which is simple: sample a tree, and then sample an observation from the tree. Hence, the distribution is a convex combination of the trees' density. This is simple but it suffers several drawbacks: (i) each tree has to be accurate enough and thus big enough to model "tricky" data for tree-based models (tricky can be low-dimensional, see the 2D data of Table 1, main file); (ii) if leaves' samplers are simple, which is the case for [31] and our approach, it makes it tricky to learn sets of simple models, such as when trees are stumps (we do not have this issue, see Table 1). In our case, while our models include sets of trees, generating one observation makes use of leaves in all trees instead of just one as in [44] (Figure 1, main file). We note that the primary goal of that latter work is in fact not to generate data [44, Section 4].

Theoretical results that are relevant to data generation are in general scarce compared to the flurry of existing methods if we omit the independent convergence rates of the toolbox often used, such as for (stochastic) gradient descent. Specific convergence results are given in [44], but they are relevant to statistical consistency (infinite sample) and they also rely on assumptions that are not realistic for real world domains, such as Lipschitz continuity of the target density, with second derivative continuous, square integrable and monotonic. The assumption made on models, that splitting probabilities on trees is lowerbounded by a constant, is also impeding to model real world data.

In the generative trees of [31], leaf generation is the simplest possible: it is uniform. This requires big trees to model real-world or tricky data. On the algorithms side, [31] introduce two training algorithms in the generative adversarial networks (GAN) framework [12], thus involving the generator to train but also a discriminator, which is a decision tree in [31]. The GAN framework is very convenient to tackle (ii) above in the context of our tasks because it allows to tie using first principles the problem of learning a density or a sampler and that of training a model (a "discriminator") to distinguish fakes from real, model that parameterizes the loss optimized. An issue with neural networks is that this parameterization has an uncontrollable slack unless the discriminator is extremely complex [33]. The advantage of using trees as in [31] is that for such classifiers, the slack disappears because the models are calibrated, so there is a tight link between training the generator and the discriminator. [31] go

Algorithm 4 ITERATIVESUPPORTUPDATE( t Υ t u T t GLYPH<16> 1 )

Algorithm 5 CONCURRENTSUPPORTUPDATE

further, showing that one can replace the adversarial training by a copycat training, involving copying parts of the discriminator in the generator to speed-up training (also discussed in [15] for neural nets), with strong rates on training in the original boosting model. There is however a limitation in the convergence analysis of [31] as losses have to be symmetric, a property which is not desirable for data generation since it ties the misclassification costs of real and fakes with no argument to do so in general.

Our paper lifts the whole setting of [31] to models that can fit more complex densities using simpler models (Table 1), using sets of trees that can be much smaller than those of [44] (Figure 1); training such models is achieved by merging the two steps of copycat training into a single one where only the generator is trained, furthermore keeping strong rates on training via the original boosting model, all this while getting rid of the undesirable symmetry assumption of [31] for the loss at hand.

II Additional content

In this additional content, we provide the three ways to pick the trees in a Generative Forest to generate one observation (sequential, concurrent, randomized), and then give a proof that the optimal splitting threshold on a continuous variable when training a generative forest using GF.BOOST is always an observed value if there is one tree, but can be another value if there are more (thus highlighting some technical difficulties of one wants to stick to the optimal choice of splitting).

II.1 Sequentially choosing trees in a GF for the generation of observations

II.2 Concurrent generation of observations

We provide a concurrent generation using Algorithm 5, which differs from Algorithm 4 (main file). In concurrent generation, each tree runs concurrently algorithm UPDATESUPPORT (hence the use of the Java-style this handler), with an additional global variables (in addition to C s , initialized to X ): a Boolean semaphore accessible implementing a lock, whose value 1 means C is available for an update (and otherwise it is locked by a tree in Steps 1.2/1.3 in the set of trees of the GF). We assume that INIT has been run beforehand (eventually locally).

II.3 Randomized generation of observations

We provide a general randomized choice of the sequence of trees for generation, in Algorithm 6.

Algorithm 6 RANDOMIZEDSUPPORTUPDATE

III Supplementary material on proofs

III.1 Proof of Lemma 4.4

Given sequence v of dimension dim p v q , denote t C j u j Pr 1 GLYPH<0> dim p v qs the sequence of subsets of the domain appearing in the parameters of UPDATESUPPORT through sequence v , to which we add a last element, C s (and its first element is X ). If we let X j denote the support of the corresponding star node whose Bernoulli event is triggered at index j in sequence v (for example, X 1 GLYPH<16> X ), then we easily get

indicating the Bernoulli probability in Step 1.1 of STARUPDATE is always defined. We then compute

(7) holds because updating the support generation is Markovian in a GF, (8) holds because of Step 3.2 and (9) is a simplification through cancelling terms in products.

III.2 Proof of Theorem 5.2

Notations: we iteratively learn generators G 0 , G 1 , ..., G J where G 0 is just a root. We let P p G j q denote the partition induced by the set of trees of G j , recalling that each element is the (non-empty) intersection of the support of a set of leaves, one for each tree (for example, P p G 0 q GLYPH<16> t X u ). The criterion minimized to build G j GLYPH<0> 1 from G j is

The proof entails fundamental notions on loss functions, models and boosting. We start with loss functions. A key function we use is

which is concave [37, Appendix A.3].

Lemma A.

Proof. We make use of the following important fact about GFs:

We have from the proof of [31, Lemma 5.6],

We note that (F1) implies (13) is just a sum of slacks of Jensen's inequality. We observe

where P split J contains all couples p X t , X f q such that X t , X f P P J and X t Y X f P P J GLYPH<1> 1 . These unions were subsets of the partition of X induced by the set of trees of G J GLYPH<1> 1 and that were cut by the predicate p put at the leaf λ GLYPH<6> that created G J from G J GLYPH<1> 1 . To save space, denote X tf . GLYPH<16> X t Y X f .

We now work on (14). Using (F1) , we note ‡ X tf dG J GLYPH<1> 1 GLYPH<16> ‡ X tf dR since X tf P P J GLYPH<1> 1 . Similarly, ‡ X v dG J GLYPH<16> ‡ X v dR , @ v P t t , f u , so we make appear R in (14) and get:

where we let for short

We finally get

D glyph[lscript] p R , G J GLYPH<1> 1 q GLYPH<1> D glyph[lscript] p R , G J q

as claimed. The last identity comes from the fact that the contribution to L p . q is the same outside P J GLYPH<1> 1 for both G J GLYPH<1> 1 and G J .

Algorithm 7 MODABOOST p R , glyph[lscript], WL , J, T q

Input: measure R , SPD loss glyph[lscript] , weak learner WL, iteration number J ¥ 1 , number of trees T ; Output: PLM (partition-linear model) H J with T trees; .

Step 1 :

Γ

0

GLYPH<16> t

Υ

t

u

T

t

GLYPH<16>

1

-t

root

,

root

, ...,

root

u

;

P

p

Γ

0

q - t

X

u

;

Step 2 :

for

j

GLYPH<16>

1

,

2

, ..., J

Step 2.1 : pick t P r T s ;

Step 2.2 : pick λ P Λ p Υ t q ;

Step 2.3 : h j - WL p X λ q ;

// call to the weak learner for some h j of the type h j . GLYPH<16> 1 p p . q GLYPH<4> K j , // K j constant, p p . q a splitting predicate ( e.g. x i ¥ a )

Step 2.4 : for C P P λ p Γ j GLYPH<1> 1 q ,

Step 2.5 : // update P p Γ . q :

We now come to models and boosting. Suppose we have t trees t Υ 1 , Υ 2 , ..., Υ t u in G j . We want to split leaf λ P Υ t into two new leaves, λ f , λ t . Let P λ p G j q denote the subset of P p G j q containing only the subsets defined by intersection with the support of λ , X λ . The criterion we minimize can be reformulated as

Here, Υ is any tree in the set of trees, since Y λ P Λ p Υ q P λ p G j q covers the complete partition of X induced by t Υ 1 , Υ 2 , ..., Υ t u . If we had a single tree, the inner sum would disappear since we would have P λ p G j q GLYPH<16> t X λ u and so one iteration would split one of these subsets. In our case however, with a set of trees, we still split X λ but the impact on the reduction of L can be substantially better as we may simultaneously split as many subsets as there are in P λ p G j q . The reduction in L can be obtained by summing all reductions to which contribute each of the subsets.

To analyze it, we make use of a reduction to the MODABOOST algorithm of [25]. We present the algorithm in Algorithm 7. The original MODABOOST algorithm trains partition-linear models , i.e. models whose output is defined by a sum of reals over a set of partitions to which the observation to be classified belongs to. Training means then both learning the organisation of partitions and the reals which are just the output of a weak classifier given by a weak learner, times a leveraging constant computed by MODABOOST. As in the classical boosting literature, the original MODABOOST includes the computation and updates of weights .

In our case, the structure of partition learned is that of a set of trees, each weak classifier h is of the form

where K is a real and p is a Boolean predicate, usually doing an axis-parallel split of X on an observed feature, such as p p x q GLYPH<17> 1 x i ¥ a .

From now on, it will be interesting to dissociate our generator G j - which includes a model structure and an algorithm to generate data from this structure- from its set of trees i.e. its model structure which we denote Γ j . GLYPH<16> t Υ t u T t GLYPH<16> 1 . MODABOOST learns both Γ j but also predictions for each possible

C

= {Y1,Y2}

Card(Px(F)) = 3

outcome, predictions that we shall not use since the loss we are minimizing can be made to depend only on the model structure Γ j . Given the simplicity of the weak classifiers h and the particular nature of the partitions learned, the original MODABOOST of [25] can be simplified to our presentation in Algorithm 7. Among others, the simplification discards the weights from the presentation of the algorithm. What we show entangles two objectives on models and loss functions as we show that

MODABOOST learns the same structure Γ as our GF.BOOST , and yields a guaranteed decrease of L p G q , and it is achieved via the following Theorem.

Theorem B. MODABOOST greedily minimizes the following loss function:

Furthermore, suppose there exists γ ¡ 0 , κ ¡ 0 such that at each iteration j , the predicate p splits X λ of leaf λ P Λ p Υ q into X p λ and X p λ (same nomenclature as in (17) , (19) ) such that:

Then we have the following guaranteed slack between two successive models:

where κ is such that 0 GLYPH<160> κ / inf t glyph[lscript] 1 GLYPH<1> 1 GLYPH<1> glyph[lscript] 1 1 u .

Proof sketch The loss function comes directly from [25, Lemma 7]. At each iteration, MODABOOST makes a number of updates that guarantee, once Step 2.4 is completed (because

the elements in P λ p Γ j q are disjoint)

where the weights are given in [25]. Each expression in the summand of (26) is exactly the guarantee of [25, ineq. before (69)]; all such expressions are not important; what is more important is (26): all steps occurring in Step 2.4 are equivalent to a single step of MODABOOST carried out over the whole X λ . Overall, the number of "aggregated" steps match the counter j in MODABOOST. We then just have to reuse the proof of [25, Theorem B], which implies, in lieu of their [25, eq. (74)]

with L SQ p u q . GLYPH<16> u p 1 GLYPH<1> u q . Noting 2 L SQ p u q ¥ min t u, 1 GLYPH<1> u u , we then use (22) - (24), which yields the statement of the Theorem.

As a consequence, if we assume that the total number of boosting iterations J is a multiple of the number of trees T , it comes from [25, eq. (29)] that after J iterations, we have

Using Lemma A and the fact that the induction of the sets of trees in MODABOOST is done in the same way as the induction of the set of trees of our generator G in GF.BOOST, we get:

as claimed.

Remark C. One may wonder what is the relationship between the loss that we minimize and "conventional" losses used to train generative models. The advantage of our Theorem 5.2 is that by choosing different proper losses, one can get convergence guarantees for different "conventional" losses. Let us illustrate this with the KL divergence between measures A and B , noted KL p A } B q .

Lemma D. Suppose the prior satisfies π ¡ 0 and R absolutely continuous with respect to G . Then for the choice glyph[lscript] . GLYPH<16> glyph[lscript] LOG = log-loss, we get

Proof. We first recall that for any convex F , we have with our choice of g (assuming π ¡ 0 ),

The log-loss has glyph[lscript] LOG 1 p u q GLYPH<16> GLYPH<1> log u, glyph[lscript] LOG GLYPH<1> 1 p u q GLYPH<16> GLYPH<1> log p 1 GLYPH<1> u q . It is strictly proper and so it comes

We thus get, in our case

Personalstatus&sex

(0.1,{A93, A94,A95))

(0.9, {A91, A92))

Numberexistingcredits

(0.4,{0, 1,2,3,4))

(0.6,(5,6,7,8)

Personalstatus&sex

(0.7,{A93,A95))

(0.3, {A94))

and using (32) and the fact that for the log-loss pGLYPH<1> L q 1 p u q GLYPH<16> log p u {p 1 GLYPH<1> u qq ,

Finally, we get, assuming R absolutely continuous with respect to G ,

D glyph[lscript] LOG p R , G q

as claimed (end of the proof of Lemma D).

Because of the joint convexity of KL divergence, we always have

which yields (another proof) that our loss, D glyph[lscript] LOG p R , G q , is lowerbounded by 0. In general, if we let γ ¡ 0 such that KL p π dR GLYPH<0> p 1 GLYPH<1> π q dU } π dG GLYPH<0> p 1 GLYPH<1> π q dU q / p 1 GLYPH<1> γ q GLYPH<4> π KL p R } G q (The strict positivity of γ is also a weak assumption as long as R , G sufficiently differ from the uniform distribution), then

so any upperbound on D glyph[lscript] LOG p R , G q (such as obtained from Theorem 5.2) translates to an upperbound on the KL divergence. Note that the condition for absolute continuity in Lemma D is always met with the models we learn (both generative forests and ensembles of generative trees).

IV Simpler models: ensembles of generative trees

Storing a GF requires keeping information about the empirical measure R to compute the branching probability in Step 1 of STARUPDATE. This does not require to store the full training sample, but requires at least an index table recording the t leaves u GLYPH<2> t observations u association, for a storage cost in between Ω p m q and O p mT q where m is the size of the training sample. There is a simple way to get rid of this constraint and approximate the GF by a set of generative trees (GTs) of [31].

Models Simply put, a GT is a simplified equivalent representation of a generative forest with 1 tree only. In this case, the branching probabilities in Step 1 of STARUPDATE depend only on the tree's star node position. Thus, instead of recomputing them from scratch each time an observation needs to be generated, we compute them beforehand and use them to label the arcs in addition to the features' domains, as shown in Figure IV.1. This representation is equivalent to that of generative trees [31]. If we have several trees, we do this process independently for each tree and end up with an ensemble of generative trees (EOGT). The additional memory footprint for storing probabilities ( O p GLYPH<176> i | Λ p Υ i q|q ) is offset by the fact that we do not have anymore to store associations between leaves and observations for generative forests, for a potentially significant reduction is storing size. However, we cannot use anymore STARUPDATE as is for data generation since we cannot compute exactly the probabilities in Step 1. Two questions need to be addressed: is it possible to use an ensemble of generative trees to generate data with good approximation guarantees (and if so, how) and of course how do we train such models.

Data generation We propose a simple solution based on how well an EOGT can approximate a generative forest. It relies on a simple assumption about R and X . Taking as references the parameters in STARUPDATE, we now make two simplifications on notations, first replacing Υ .ν GLYPH<14> v by ν GLYPH<14> v (tree implicit), and then using notation C v . GLYPH<16> X ν GLYPH<14> v X C for any v P t t , f u , the reference to the tree / star node being implicit. The assumption is as follows.

Assumption A. There exists κ P p 0 , 8q such that at any Step 1 of STARUPDATE , for any v P t t , f u , we have

A simple condition to ensure the existence of κ is a requirement weaker than (c) in Definition 5.1: at all Steps 1 of STARUPDATE, p R r C t | C s P p 0 , 1 q , which also guarantees p R r C f | C s P p 0 , 1 q and thus postulates that the branching in Step 1 of STARUPDATE never reduces to one choice only. The next Lemma shows that it is indeed possible to combine the generation of an ensemble of generative trees with good approximation properties, and provides a simple algorithm to do so, which reduces to running STARUPDATE with a specific approximation to branching probabilities in Step 1. For any GF G and C P P p G q , depth G p C q is the sum of depths of the leaves in each tree whose support intersection is C . We need the definition of the expected depth of G .

Definition B. The expected depth of GF G is depth p G q . GLYPH<16> GLYPH<176> C P P p G q p G r C s GLYPH<4> depth G p C q .

depth p G q represents the expected complexity to sample an observation (see also Lemma C).

Lemma C. In Step 1 of STARUPDATE , suppose p R r C t X C | C s is replaced by

with p ν GLYPH<14> . GLYPH<16> p R GLYPH<16> X ν GLYPH<14> t | X ν GLYPH<14> GLYPH<24> the probability labeling arc p ν GLYPH<14> , ν t q in its generative tree. Under Assumption A, if we denote G the initial GF and ˆ G the EOGT . P using (40) , the following bound holds on the KL divergence between G and ˆ G :

Proof. Because C GLYPH<132> X Υ .ν GLYPH<14> at any call of STARUPDATE, we have

(43) is due to the fact that, since C GLYPH<132> X Υ .ν GLYPH<14> , p R r C v s GLYPH<16> p R GLYPH<16> C v X X Υ .ν GLYPH<14> v GLYPH<24> GLYPH<16> p R GLYPH<16> C v | X Υ .ν GLYPH<14> v GLYPH<24> p R GLYPH<16> X Υ .ν GLYPH<14> v GLYPH<24> , and then using (39). (44) is obtained by dividing numerator and denominator by p R GLYPH<16> X Υ .ν GLYPH<14> t GLYPH<24> GLYPH<0> p R GLYPH<16> X Υ .ν GLYPH<14> f GLYPH<24> GLYPH<16> p R r X Υ .ν GLYPH<14> s .

G and ˆ G satisfy P p G q GLYPH<16> P p ˆ G q so

Finally, p G r C s and p ˆ G r C s are just the product of the branching probabilities in any admissible sequence. If we use the same admissible sequence in both generators, we can write p G r C s GLYPH<16> -n p C q j GLYPH<16> 1 p j and p ˆ G r C s GLYPH<16> -n p C q j GLYPH<16> 1 ˆ p j , and (44) directly yields p j / exp p 2 κ q GLYPH<4> ˆ p j , @ j P r n p C qs , n p C q being a shorthand for depth G p C q GLYPH<16> depth ˆ G p C q (see main file). So, for any C P P p G q ,

and finally, replacing back n p C q by notation depth p C q ,

as claimed.

Note the additional leverage for training and generation that stems from Assumption A, not just in terms of space: computing p U GLYPH<16> C v | X ν GLYPH<14> v GLYPH<24> is O p d q and does not necessitate data so the computation of key conditional probabilities (40) drops from Ω p m q to O p d q for training and generation in an EOGT. Lemma C provides the change in STARUPDATE to generate data.

Training To train an EOGT, we cannot rely on the idea that we can just train a GF and then replace each of its trees by generative trees. To take a concrete example of how this can be a bad idea in the context of missing data imputation, we have observed empirically that a generative forest (GF) can have many trees whose node's observation variables are the same within the tree. Taken independently of the forest, such trees would only model marginals, but in a GF, sampling is dependent on the other trees and Lemma 4.4 guarantees the global accuracy of the forest. However , if we then replace the GF by an EOGT with the same trees and use them for missing data imputation, this results in imputation at the mode of the marginal(s) (exclusively), which is clearly suboptimal. To avoid this, we have to use a specific training for an EOGT, and to do this, it is enough to change a key part of training in splitPred , the computation of probabilities p R r . s used in (4). Suppose κ very small in Assumption A. To evaluate a split at some leaf, we observe (suppose we have a single tree)

(and the same holds for λ t ). Extending this to multiple trees and any C P P p T q , we get the computation of any p R r C f s and p R r C t s needed to measure the new (4) for a potential split. Crucially, it does not necessitate to split the empirical measure at C but just relies on computing p U r C v | C s , v P t f , t u : with the product measure and since we make axis-parallel splits, it can be done in O p 1 q , thus substantially reducing training time.

More with ensembles of generative trees we can follow the exact same algorithmic steps as for generative trees to perform missing data imputation and density estimation, but use branching probabilities in the trees to decide branching, instead of relying on the empirical measure at tuples of nodes, which we do not have access to anymore. For this, we rely on the approximation (40) in Lemma C. A key difference with a GF is that no tuple can have zero density because branching probabilities are in p 0 , 1 q in each generative tree.

V Appendix on experiments

V.1 Domains

ringGauss is the seminal 2D ring Gaussians appearing in numerous GAN papers [46]; those are eight (8) spherical Gaussians with equal covariance, sampling size and centers located on regularly spaced (2-2 angular distance) and at equal distance from the origin. gridGauss was generated as a decently hard task from [11]: it consists of 25 2D mixture spherical Gaussians with equal variance and sampled sizes, put on a regular grid. circGauss is a Gaussian mode surrounded by a circle, from [46]. randGauss is a substantially harder version of ringGauss with 16 mixture components, in which covariance, sampling sizes and distances on sightlines from the origin are all random, which creates very substantial discrepancies between modes.

V.2 Algorithms configuration and choice of parameters

GF.BOOST We have implemented GF.BOOST in Java, following Algorithm 3's blueprint. Our implementation of tree and leaf in Steps 2.1, 2.2 is simple: we pick the heaviest leaf among all trees (with respect to R ). The search for the best split is exhaustive unless the variable is categorical with more than a fixed number of distinct modalities (22 in our experiments), above that threshold, we pick the best split among a random subset. We follow [31]'s experimental setting: in particular, the input of our algorithm to train a generator is a .csv file containing the training data without any further information . Each feature's domain is learned from the training data only; while this could surely and trivially be replaced by a user-informed domain for improved results ( e.g. indicating a proportion's domain as r 0% , 100% s , informing the complete list of socio-professional categories, etc.) - and is in fact standard in some ML packages like weka 's ARFF files, we did not pick this option to alleviate all side information available to the GT learner. Our software automatically recognizes three types of variables: nominal, integer and floating point represented.

Comments on implementation For the interested reader, we give here some specific implementation details regarding choices mades for two classical bottlenecks on tree-based methods (this also applies to the supervised case of learning decision trees): handling features with large number of modalities and handling continuous features.

We first comment the case where the number of modalities of a feature is large. The details can be found in the code in File Algorithm.java (class Algorithm ), as follows:

method public GenerativeModelBasedOnEnsembleOfTrees learn_geot() implements both GF.Boost (for generative forests) and its extension to training ensembles of generative trees (Appendix) as a single algorithm.

method Node choose_leaf(String how_to_choose_leaf) is the method to choose a leaf (Step 2.2). Since it picks the heaviest leaf among all trees, it also implements Step 2.1 (we initialize all trees to their roots; having a tree = root for generation does not affect generation).

method public boolean split_generative_forest(Node leaf, HashSet <MeasuredSupportAtTupleOfNodes> tol) finds the split for a Generative Forest, given a node chosen for the split. Here is how it works:

It first computes the complete description of possible splits (not their quality yet), using method public static Vector<FeatureTest> ALL_FEATURE_TESTS(Feature f, Dataset ds) in Feature.java . The method is documented: for continuous features, the list of splits is just a list of evenly spaced splits.

then it calls public SplitDetails split_top_down_boosting_generative_forest_fast(Node leaf, boolean [] splittable_feature, FeatureTest [][] all_feature_tests, HashSet <MeasuredSupportAtTupleOfNodes> tol) , which returns the best split as follows: (i) it shuffles the potential list of splits and then (ii) picks the best one in the sublist containing the first Algorithm.MAXIMAL_NUMBER_OF_SPLIT_TESTS_TRIES_PER_BOOSTING_ITERATION elements (if the list is smaller, the search for the best is exhaustive); this class variable is currently = 1000.

Last, we comment how we handle continuous variables: the boosting theory tells us that finding a moderately good split (condition (b) Definition 5.1, main file) is enough to get boosting-compliant convergence (Theorem 5.2, main file), so we have settled for a trivial and efficient mechanism: cut-points are evenly spaced and the number is fixed beforehand. See method public static Vector<FeatureTest> ALL_FEATURE_TESTS(Feature f, Dataset ds) in Feature.java for the details. It turns out that this works very well and shows our theory was indeed concretely used in the design/implementation of the training algorithm.

MICE We have used the R MICE package V 3.13.0 with two choices of methods for the round robin (column-wise) prediction of missing values: CART [1] and random forests (RF) [42]. In that last case, we have replaced the default number of trees (10) by a larger number (100) to get better results. We use the default number of round-robin iterations (5). We observed that random forests got the best results so, in order not to multiply the experiments reported and perhaps blur the comparisons with our method, we report only MICE's results for random forests.

TENSORFLOW To learn the additional Random Forests involved in experiments GEN-DISCRIM, we used Tensorflow Decision Forests library § . We use 300 trees with max depth 16. Attribute sampling: sqrt(number attributes) for classification problems, number attributes / 3 for regression problems (Breiman rule of thumb); the min # examples per leaf is 5.

CT-GAN We used the Python implementation ¶ with default values [47].

juv_fel_count

{0}

{1…10}

race_African-American

true

false

two_year_recid

true

false

~56%

~44%

------------------------------------------------------------------------

(name

=

#0.0 |

depth

=

8

|

#nodes

=

17)

[#0:root]

internal

{5278}

(juv_fel_count

<= 0

?

#1

:

#2)

|-[#1]

internal {5087}

(age_cat_Greaterthan45 in

{0}

?

#3

:

#4)

|

|-[#3]

internal

{4004}

(age

<=

41 ?

#5

:

#6)

|

|

|-[#5] internal

{3760}

(race_African-American

in

{0}

?

#7 :

#8)

|

|

|

|-[#7]

leaf

{1321}

|

|

|

\-[#8]

internal

{2439}

(age <=

19

?

#9

:

#10)

|

|

|

|-[#9]

leaf

{13}

|

|

|

\-[#10]

internal

{2426} (age

<=

39

?

#11 :

#12)

|

|

|

|-[#11]

internal {2347}

(age

<=

34

? #13

:

#14)

|

|

|

|

|-[#13]

internal

{2024}

(two_year_recid in

{0}

?

#15

:

#16)

|

|

|

|

|

|-[#15] leaf

{886}

|

|

|

|

|

\-[#16] leaf

{1138}

|

|

|

|

\-[#14]

leaf {323}

|

|

|

\-[#12]

leaf

{79}

|

|

\-[#6] leaf

{244}

|

\-[#4]

leaf {1083}

\-[#2]

leaf

{191}

Adversarial Random Forests We used the R code of the generator FORGE made available from the paper [44] || , learning forests containing a variable number of trees in t 10 , 50 , 100 , 200 u . We noted that the code does not run when the dataset has missing values and we also got an error when trying to run the code on kc1 .

Vine Copulas AutoEncoders We used the Python code available from the paper [41] ** , which processes only fully numerical datasets. We got a few errors when trying to run the code on compas .

Forest Flows We used the R code available from the paper [17] †† . Hyperparameters used were default parameters ‡‡ .

Kernel Density Estimation We used the R code available in the package npudens with default values following the approach of [24]. We tried several kernels but ended up with sticking to the default choices, that seemed to provide overall some of the best results. Compared to us with GF, KDE's code is extremely compute-intensive as on domains below tictactoe in Table A1: it took orders of magnitude more than ours to get the density values on all folds, so we did not run it on the biggest domains. Notice that in our case, computation time includes not just training the generative model, but also, at each applicable iteration J in GF.BOOST, the computation of the same number of density values as for KDE.

Computers used We ran part of the experiments on a Mac Book Pro 16 Gb RAM w/ 2 GHz Quad-Core Intel Core i5 processor, and part on a desktop Intel(R) Xeon(R) 3.70GHz with 12 cores and 64 Gb RAM. CT-GANs and VCAEs were run on the desktop, the other algorithms on the laptop.

V.3 Supplementary results

Due to the sheer number of tables to follow, we shall group them according to topics

V.V.3.1 Interpreting our models: 'SCRUTINIZE'

Table A2 provides experimental results on sensitive real world domain compas . It shows that it is easy to flag potential bias / fairness issues in data by directly estimating conditional probabilities: considering the GF of this example and ignoring variable age for simplicity, conditionally to having 0 felony count and race being African-American, the probability of generating an observation with two_year_recid = true is . 562 (=1138/2024).

T

=

J

=20(

GLYPH<6>

)

T

=

J

=100(

GLYPH<6>

)

T

=100,

J

=200

T

=100,

J

=500

Ground truth

Ens. of gen. trees

Generative forests

V.V.3.2 More examples of Table 1 (MF)

Tables A3 completes Tables 2 and A2 (main file), displaying that even a stochastic dependence can be accurately modeled.

We provide in Table A4 the density learned on all our four 2D (for easy plotting) simulated domains and not just circgauss as in Table 1 (MF). We see that the observations made in Table 1 (MF) can be generalized to all domains. Even when the results are less different between 50 stumps in a generative forest and 1 generative tree with 50 splits for ringgauss and randgauss , the difference on gridgauss is stark, 1 generative tree merging many of the Gaussians.

GF:1treew/50splits

circgauss

ringgauss

gridgauss

randgauss

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

V.V.3.3 The generative forest of Table 1 (MF) developed further

One might wonder how a set of stumps gets to accurately fit the domain as the number of stumps increases. Table A5 provides an answer. From this experiment, we can see that 16 stumps are enough to get the center mode. The ring shape takes obviously more iterations to represent but still, is takes a mere few dozen stumps to clearly get an accurate shape, the last iterations just finessing the fitting.

V.V.3.4 Experiment LIFELIKE in extenso

The objective of the experiment is to evaluate whether a generative model is able to create "realistic" data. Tabular data is hard to evaluate visually, unlike e,g. images or text, so we have considered a simple evaluation pipeline: we create for each domain a 5-fold stratified experiment. After a generative model has been trained, we generate the same number of observations as in the test fold and compute the optimal transport (OT) distance between the generated sample and the fold's test sample. To fasten the computation of OT costs, we use Sinkhorn's algorithm [8] with an ε -entropic regularizer, for some ε GLYPH<16> 0 . 5 which we observed was the smallest in our experiments to run with all domains without leading to numerical instabilities. To balance the importance of categorical features (for which the cost is binary, depending on whether the guess is right or not) and numerical features, we normalize numerical features with the domain's mean and standard deviation prior to computing OT costs. We then compare our method to three possible contenders: Adversarial Random Forests (ARF) [44], CT-GANs [47], Forest Flows [17] and Vine copula autoencoders (VCAE) [41]. All these approaches rely on models that are very different from each other. Adversarial Random Forests (ARF) and Forest Flows (FFs) are tree-based generators. ARFs work as follows to generate one observation: one first sample uniformly at random a tree, then samples a leaf in the tree. The leaf is attached to a distribution which is then used to sample the observation. As we already pointed out in Section 2, there are two main differences with our models. From the standpoint of the model's

distribution, if one takes the (non-empty) support from a tuple of leaves, its probability is obtained from a weighted average of the tree's distributions in [44] while it is obtained from a product of theirs in our case. Hence, at similar tree sizes, the modelling capability of the set of trees tips in our favor, but it is counterbalanced by the fact that their approach equip leaves with potentially complex distributions (e.g. truncated Gaussians) whereas we stick to uniform distributions at the leaves. A consequence of the modeling of ARFs is that each tree has to separately code for a good generator: hence, it has to be big enough or the distributions used at the leaves have to be complex enough. In our case, as we already pointed out, a small number of trees (sometimes even stumps ) can be enough to get a fairly good generator, even on real-world domains. Forest Flows use tree-based models to estimate parameters of diffusion models. Their tree-based models are not necessarily big but they need a lot of them to carry out training and generation, which is a big difference with us. Our next contender, CT-GAN [47], fully relies on neural networks so it is de facto substantially different from ours. The last, VCAE [41] relies on a subtle combination of deep nets and graphical models learned in the latent space.

Table 5 (main file) with smaller generative forests Table A6 provides the equivalent of Table 5 (with one additional contender, which did not fit in the space allotted for the main file), but in which our generative forests are much smaller: T GLYPH<16> 200 trees, j GLYPH<16> 500 total splits (so each tree is on average barely bigger than a decision stump). One can check that our models are still competitive with respect to all contenders. The best contender, Forest Flows, beat our models on density, but it must be remembered that Forest Flows uses many tree-based models to carry out both training and generation.

Focus on Sinkhorn for different parameterizations of contenders We now investigate how changing the parameterization of various contenders affect their performance against our approach. Here, ARFs are trained with a number of trees T P t 10 , 50 , 100 , 200 u (ARFs include an algorithm to select the tree size so we do not have to select it). CT-GANs are trained with a number of epochs E P t 10 , 100 , 300 , 1000 u .

We compare those models with generative forests with T GLYPH<16> 500 trees and trained for a total of J GLYPH<16> 2000 iterations, for all domains considered. Compared to the trees learned by ARFs, the total number of splits we use can still be small compared to theirs, in particular when they learn T ¥ 100 trees. Each table against ARFs and CT-GANs has a different size and contains only a subset of the whole 21 domains: ARFs do not process domains with missing values and we also got issues running contenders on several domains: those are discussed in Appendix, Section V.2.

domain

us (GF)

Adversarial Random Forests (ARFs)

tag

T

=500,

J

=2 000

T

GLYPH<16>

10

p

-val

T

GLYPH<16>

50

p

-val

T

GLYPH<16>

100

p

-val

T

GLYPH<16>

200

p

-val

0.423

GLYPH<8>

0.033

0.530

GLYPH<8>

0.064

0.0039

0.517

GLYPH<8>

0.081

0.0866

0.466

GLYPH<8>

0.050

0.1452

0.502

GLYPH<8>

0.043

0.0044

ring

0.285

GLYPH<8>

0.008

0.289

GLYPH<8>

0.006

0.1918

0.286

GLYPH<8>

0.007

0.8542

0.288

GLYPH<8>

0.010

0.7205

0.286

GLYPH<8>

0.007

0.6213

circ

0.351

GLYPH<8>

0.005

0.354

GLYPH<8>

0.002

0.2942

0.356

GLYPH<8>

0.008

0.2063

0.350

GLYPH<8>

0.002

0.9050

0.355

GLYPH<8>

0.005

0.3304

grid

0.390

GLYPH<8>

0.002

0.394

GLYPH<8>

0.003

0.1376

0.391

GLYPH<8>

0.002

0.3210

0.392

GLYPH<8>

0.001

0.2118

0.394

GLYPH<8>

0.002

0.0252

for

1.108

GLYPH<8>

0.105

1.272

GLYPH<8>

0.281

0.2807

1.431

GLYPH<8>

0.337

0.1215

1.311

GLYPH<8>

0.255

0.0972

1.268

GLYPH<8>

0.209

0.0616

rand

0.286

GLYPH<8>

0.003

0.286

GLYPH<8>

0.003

0.9468

0.290

GLYPH<8>

0.005

0.2359

0.289

GLYPH<8>

0.005

0.3990

0.288

GLYPH<8>

0.002

0.3685

tic

0.575

GLYPH<8>

0.002

0.577

GLYPH<8>

0.001

0.2133

0.578

GLYPH<8>

0.001

0.1104

0.577

GLYPH<8>

0.001

0.2739

0.577

GLYPH<8>

0.001

0.2773

iono

1.167

GLYPH<8>

0.074

1.431

GLYPH<8>

0.043

0.0005

1.469

GLYPH<8>

0.079

0.0001

1.431

GLYPH<8>

0.058

0.0015

1.420

GLYPH<8>

0.056

0.0035

stm

0.975

GLYPH<8>

0.026

1.010

GLYPH<8>

0.015

0.0304

1.015

GLYPH<8>

0.018

0.0231

1.001

GLYPH<8>

0.010

0.0956

1.014

GLYPH<8>

0.021

0.0261

wred

0.980

GLYPH<8>

0.032

1.086

GLYPH<8>

0.036

0.0003

1.072

GLYPH<8>

0.055

0.0133

1.086

GLYPH<8>

0.030

0.0003

1.099

GLYPH<8>

0.031

0.0001

stp

0.976

GLYPH<8>

0.018

0.999

GLYPH<8>

0.011

0.0382

1.012

GLYPH<8>

0.010

0.0250

1.006

GLYPH<8>

0.011

0.0280

1.003

GLYPH<8>

0.003

0.0277

ana

0.368

GLYPH<8>

0.014

0.370

GLYPH<8>

0.012

0.7951

0.382

GLYPH<8>

0.015

0.1822

0.368

GLYPH<8>

0.006

0.9161

0.369

GLYPH<8>

0.007

0.8316

aba

0.490

GLYPH<8>

0.028

0.505

GLYPH<8>

0.046

0.5251

0.484

GLYPH<8>

0.024

0.7011

0.503

GLYPH<8>

0.035

0.6757

0.481

GLYPH<8>

0.031

0.4632

wwhi

1.064

GLYPH<8>

0.003

1.159

GLYPH<8>

0.011

ε

1.159

GLYPH<8>

0.006

ε

1.170

GLYPH<8>

0.014

ε

1.150

GLYPH<8>

0.021

0.0005

comp

0.532

GLYPH<8>

0.008

0.531

GLYPH<8>

0.012

0.8278

0.534

GLYPH<8>

0.009

0.8438

0.551

GLYPH<8>

0.023

0.1371

0.535

GLYPH<8>

0.033

0.8642

arti

0.821

GLYPH<8>

0.004

0.849

GLYPH<8>

0.004

0.0007

0.847

GLYPH<8>

0.010

0.0005

0.843

GLYPH<8>

0.009

0.0006

0.849

GLYPH<8>

0.010

0.0005

0.929

GLYPH<8>

0.002

0.929

GLYPH<8>

0.002

0.6044

0.930

GLYPH<8>

0.002

0.6536

0.929

GLYPH<8>

0.002

0.5937

0.930

GLYPH<8>

0.001

0.2447

elec

1.483

GLYPH<8>

0.033

1.577

GLYPH<8>

0.088

0.1052

1.543

GLYPH<8>

0.017

0.0081

1.552

GLYPH<8>

0.039

0.0522

1.581

GLYPH<8>

0.025

0.0095

wins / lose for us

GLYPH<209>

15 / 1

17 / 1

16 / 1

17 / 1

Results vs Adversarial Random Forests Table A7 digs into results obtained against adversarial random forests on the Sinkhorn metric. A first observation, not visible in the table, is that ARFs indeed tend to learn big models, typically with dozens of nodes in each tree. For the largest ARFs with 100 or 200 trees, this means in general a total of thousands of nodes in models. A consequence, also observed experimentally, is that there is sometimes little difference in performance in general between models with a different number of trees in ARFs as each tree is in fact an already good generator. In our case, this is obviously not the case. Noticeably, the performance of ARFs is not monotonic in the number of trees, so there could be a way to find the appropriate number of trees in ARFs to buy the slight (but sometimes significant) increase in performance in a domain-dependent way. In our case, there is obviously a dependency in the number of trees chosen as well. From the standpoint of the optimal transport metric, even when ARFs make use of distributions at their tree leaves that are much more powerful than ours and in fact fit to some of our simulated domains (Gaussians used in circgauss , randgauss and gridgauss ), we still manage to compete or beat ARFs on those simulated domains.

Globally, we manage to beat ARFs on almost all runs, and very significantly on many of them. Since training generative forests does not include a mechanism to select the size of models, we have completed this experiment by another one on which we learn much smaller generative forests. The corresponding Table is in Appendix, Section V.V.3.4. Because we clearly beat ARFs on student_performance_mat and student_performance_por in Table A7 but are beaten by ARFs for much smaller generative forests, we conclude that there could exist mechanisms to compute the "right size" of our models, or to prune big models to get models with the right size. Globally however, even with such smaller models, we still manage to beat ARFs on a large majority of cases, which is a good figure given the difference in model sizes. The ratio "results quality over model size" tips in our favor and demonstrates the potential in using all trees in a forest to generate an observation (us) vs using a single of its trees (ARFs), see Figure 1.

Results vs CT-GANs Table A8 summarizes our results against CT-GAN using various numbers of training epochs ( E ). In our case, our setting is the same as versus ARFs: we stick to T GLYPH<16> 500 trees trained for a total number of J GLYPH<16> 2000 iterations in the generative forest, for all domains. We see that generative forests consistently and significantly outperform CT-GAN on nearly all cases. Furthermore, while CT-GANs performances tend to improve with the number of epochs, we observe that on a majority of datasets, performances at 1 000 epochs are still far from those of generative forests and already induce training times that are far bigger than for generative forests: for example, more than 6 hours per fold for stm while it takes just a few seconds to train a generative forest. Just like we did for ARFs, we repeated the experiment vs CT-GAN using much smaller generative forests ( T GLYPH<16> 200 , J GLYPH<16> 500 ). The results are presented in Appendix, Section V.V.3.4. There is no change in the conclusion: even with such small models, we still beat CT-GANs, regardless of the number of training epochs, on all cases (and very significantly on almost all of them).

More results with small generative forests Tables A9 and A10 present the results of generative forests vs adversarial random forests and CT-GANs, when the size of our models is substantially smaller than in the main file ( T GLYPH<16> 200 , J GLYPH<16> 500 ). We still manage to compete or beat ARFs on simulated domains for which modelling the leaf distributions in ARFs should represent an advantage (Gaussians used in circgauss , randgauss and gridgauss ), even more using comparatively very small models compared to ARFs. We believe this could be due to two factors: (i) the fact that in our models all trees are used to generate each observation (which surely facilitates learning small and accurate models) and (ii) the fact that we do not generate data during training but base our splitting criterion on an exact computation of the reduction of Bayes risk in growing trees. If we compute aggregated statistics comparing, for each number of trees in ARFs, the number of times we win / lose versus ARFs, either considering only significant p -values or disregarding them, our generative forests always beat ARFs on a majority of domains. Globally, this means we win on 34 out of 52 cases. The results vs CT-GANs are of the same look and feel as in the main file with larger generative forests: generative forests beat them in all cases, and very significantly in most of them.

V.V.3.5 Comparison with "the optimal generator": GEN-DISCRIM

Rather than compare our method with another one, the question we ask here is "can we generate data that looks like domain's data" ? We replicate the experimental pipeline of [31]. In short, we shuffle a 3-partition (say R 1 , R 2 , R 3 ) of the training data in a 3! GLYPH<16> 6 -fold CV, then train a generator G 1

domain

us (GF)

CT-GANs

tag

T

=500,

J

=2 000

E

GLYPH<16>

10

p

-val

E

GLYPH<16>

100

p

-val

E

GLYPH<16>

300

p

-val

E

GLYPH<16>

1000

p

-val

ring

0.285

GLYPH<8>

0.008

0.457

GLYPH<8>

0.058

0.0032

0.546

GLYPH<8>

0.079

0.0018

0.405

GLYPH<8>

0.044

0.0049

0.351

GLYPH<8>

0.042

0.0183

circ

0.351

GLYPH<8>

0.005

0.848

GLYPH<8>

0.300

0.0213

0.480

GLYPH<8>

0.040

0.0019

0.443

GLYPH<8>

0.014

ε

0.435

GLYPH<8>

0.075

0.0711

grid

0.390

GLYPH<8>

0.002

0.749

GLYPH<8>

0.417

0.1268

0.459

GLYPH<8>

0.031

0.0085

0.426

GLYPH<8>

0.022

0.0271

0.408

GLYPH<8>

0.019

0.1000

for

1.108

GLYPH<8>

0.105

9.796

GLYPH<8>

5.454

0.0049

1.899

GLYPH<8>

0.289

0.0045

1.532

GLYPH<8>

0.178

0.0244

1.520

GLYPH<8>

0.311

0.0410

rand

0.286

GLYPH<8>

0.003

0.746

GLYPH<8>

0.236

0.0119

0.491

GLYPH<8>

0.063

0.0021

0.368

GLYPH<8>

0.015

0.0002

0.327

GLYPH<8>

0.024

0.0288

tic

0.575

GLYPH<8>

0.002

0.601

GLYPH<8>

0.003

0.0002

0.581

GLYPH<8>

0.001

0.0082

0.586

GLYPH<8>

0.006

0.0294

0.584

GLYPH<8>

0.003

0.0198

iono

1.167

GLYPH<8>

0.074

2.263

GLYPH<8>

0.035

ε

2.084

GLYPH<8>

0.052

ε

1.984

GLYPH<8>

0.136

ε

1.758

GLYPH<8>

0.137

0.0002

stm

0.980

GLYPH<8>

0.032

1.511

GLYPH<8>

0.074

ε

1.189

GLYPH<8>

0.045

0.0002

1.168

GLYPH<8>

0.054

0.0056

1.167

GLYPH<8>

0.041

0.0013

wred

0.980

GLYPH<8>

0.032

2.836

GLYPH<8>

0.703

0.0044

2.004

GLYPH<8>

0.090

ε

1.825

GLYPH<8>

0.127

0.0001

1.384

GLYPH<8>

0.047

ε

stp

0.976

GLYPH<8>

0.018

1.660

GLYPH<8>

0.161

0.0006

1.141

GLYPH<8>

0.018

0.0001

1.206

GLYPH<8>

0.046

0.0007

1.186

GLYPH<8>

0.052

0.0019

ana

0.368

GLYPH<8>

0.014

0.542

GLYPH<8>

0.135

0.0050

0.439

GLYPH<8>

0.056

0.0202

0.404

GLYPH<8>

0.012

0.0397

0.436

GLYPH<8>

0.036

0.0051

aba

0.490

GLYPH<8>

0.028

1.689

GLYPH<8>

0.053

ε

1.463

GLYPH<8>

0.094

ε

0.754

GLYPH<8>

0.040

0.0009

0.657

GLYPH<8>

0.026

0.0006

wwhi

1.064

GLYPH<8>

0.003

1.770

GLYPH<8>

0.160

0.0005

1.849

GLYPH<8>

0.064

ε

1.284

GLYPH<8>

0.029

ε

1.158

GLYPH<8>

0.009

ε

arti

GLYPH<6>

0.821

GLYPH<8>

0.004

1.388

GLYPH<8>

0.109

4

0.0020

1.209

GLYPH<8>

0.018

4

0.0005

1.026

GLYPH<8>

0.013

4

0.0001

0.959

GLYPH<8>

0.013

3

0.0038

wins / lose for us

GLYPH<209>

14 / 0

14 / 0

14 / 0

14 / 0

from R 1 (we use generative forests with GF.BOOST), then train a random forest (RF) to distinguish between G 1 and R 2 , and finally estimate its test accuracy on G 1 vs R 3 . The lower this accuracy, the less distinguishable are fakes from real and thus the better the generator. We consider 3 competing baselines to our models: (i) the "optimal" one, COPY, which consists in replacing G 1 by a set of real data, (ii) the "worst" one, UNIF(orm), which uniformly samples data, and (iii) GF.BOOST for T GLYPH<16> 1 , which learns a generative tree (GT). We note that this pipeline is radically different from the one used in [44]. In this pipeline, domains used are supervised (the data includes a variable to predict). A generator is trained from some training data, then generates an amount of data equal to the size of the training data. Then, two classifiers are trained, one from the original training data, one from the generated data, to predict for the variable to predict and their performances are compared on this basis (the higher the accuracy, the better). There is a risk in this pipeline that we have tried to mitigate with our sophisticated pipeline: if the generator consists just in copying its training data, it is guaranteed to perform well according to [44]'s metric. Obviously, this would not be a good generator however. Our pipeline would typically prevent this, R 2 being different from R 3 .

Results Tables A11 and A12 provide the results we got, including statistical p -values for the test of comparing our method to COPY. They display that it is possible to get GFs generating realistically looking data: for iris , the Student t -tests show that we can keep H 0 = "GFs perform identically to COPY" for J GLYPH<16> T GLYPH<16> 40 ( p ¡ . 2 ). For mat , the result is quite remarkable not because of the highest p -val, which is not negligible ( ¡ 0 . 001 ), but because to get it, it suffices to build a GF in which the total number of feature occurrences ( J GLYPH<16> 80 ) is barely three times as big as the domain's dimension ( d GLYPH<16> 33 ). The full tables display a clear incentive to grow further the GFs as domains increase in size. However, our results show, pretty much like the experiment against Adversarial Random Forests, that there could be substantial value of learning the right model size: iris and student_performance_por clearly show nontrivial peaks of p -values, for which we would get the best models compared to COPY.

domain

us (GF)

Adversarial Random Forests (ARFs)

tag

T

=200,

J

=500

T

GLYPH<16>

10

p

-val

T

GLYPH<16>

50

p

-val

T

GLYPH<16>

100

p

-val

T

GLYPH<16>

200

p

-val

0.529

GLYPH<8>

0.099

0.530

GLYPH<8>

0.064

0.9890

0.517

GLYPH<8>

0.081

0.8645

0.466

GLYPH<8>

0.050

0.1099

0.501

GLYPH<8>

0.043

0.6548

ring

0.283

GLYPH<8>

0.005

0.289

GLYPH<8>

0.006

0.0651

0.286

GLYPH<8>

0.007

0.4903

0.288

GLYPH<8>

0.010

0.4292

0.286

GLYPH<8>

0.007

0.7442

circ

0.356

GLYPH<8>

0.002

0.354

GLYPH<8>

0.002

0.4561

0.356

GLYPH<8>

0.008

0.7882

0.350

GLYPH<8>

0.002

0.0591

0.355

GLYPH<8>

0.005

0.7446

grid

0.392

GLYPH<8>

0.002

0.394

GLYPH<8>

0.003

0.4108

0.391

GLYPH<8>

0.002

0.3403

0.392

GLYPH<8>

0.001

0.8370

0.394

GLYPH<8>

0.002

0.3423

for

1.086

GLYPH<8>

0.124

1.272

GLYPH<8>

0.281

0.1420

1.431

GLYPH<8>

0.337

0.1142

1.311

GLYPH<8>

0.255

0.1215

1.268

GLYPH<8>

0.209

0.0370

rand

0.290

GLYPH<8>

0.007

0.286

GLYPH<8>

0.003

0.4488

0.290

GLYPH<8>

0.005

0.9553

0.289

GLYPH<8>

0.005

0.9004

0.288

GLYPH<8>

0.002

0.6542

tic

0.574

GLYPH<8>

0.001

0.577

GLYPH<8>

0.001

0.0394

0.578

GLYPH<8>

0.001

0.0325

0.577

GLYPH<8>

0.001

0.0657

0.577

GLYPH<8>

0.001

0.0914

iono

1.203

GLYPH<8>

0.068

1.431

GLYPH<8>

0.043

0.7307

1.469

GLYPH<8>

0.079

0.3402

1.431

GLYPH<8>

0.058

0.7046

1.420

GLYPH<8>

0.056

0.8737

stm

1.044

GLYPH<8>

0.013

1.010

GLYPH<8>

0.015

0.0080

1.015

GLYPH<8>

0.018

0.0094

1.001

GLYPH<8>

0.010

0.0037

1.014

GLYPH<8>

0.021

0.0329

wred

1.027

GLYPH<8>

0.037

1.086

GLYPH<8>

0.036

0.0327

1.072

GLYPH<8>

0.055

0.1432

1.086

GLYPH<8>

0.030

0.0095

1.099

GLYPH<8>

0.031

0.0044

stp

1.029

GLYPH<8>

0.027

0.999

GLYPH<8>

0.011

0.0826

1.012

GLYPH<8>

0.010

0.3421

1.006

GLYPH<8>

0.011

0.0542

1.003

GLYPH<8>

0.003

0.0764

ana

0.367

GLYPH<8>

0.018

0.370

GLYPH<8>

0.012

0.8229

0.382

GLYPH<8>

0.015

0.2318

0.368

GLYPH<8>

0.006

0.9264

0.369

GLYPH<8>

0.007

0.8722

aba

0.476

GLYPH<8>

0.017

0.505

GLYPH<8>

0.046

0.2264

0.484

GLYPH<8>

0.024

0.2162

0.503

GLYPH<8>

0.035

0.1847

0.481

GLYPH<8>

0.031

0.7468

wwhi

1.120

GLYPH<8>

0.013

1.159

GLYPH<8>

0.011

0.0013

1.159

GLYPH<8>

0.006

0.0014

1.170

GLYPH<8>

0.014

0.0026

1.150

GLYPH<8>

0.021

0.0327

comp

0.538

GLYPH<8>

0.015

0.531

GLYPH<8>

0.012

0.5639

0.534

GLYPH<8>

0.009

0.5706

0.551

GLYPH<8>

0.023

0.4057

0.535

GLYPH<8>

0.033

0.9017

arti

0.830

GLYPH<8>

0.007

0.849

GLYPH<8>

0.004

0.0078

0.847

GLYPH<8>

0.010

0.0012

0.843

GLYPH<8>

0.009

0.0003

0.849

GLYPH<8>

0.010

0.0010

0.928

GLYPH<8>

0.001

0.929

GLYPH<8>

0.002

0.1720

0.930

GLYPH<8>

0.002

0.0646

0.929

GLYPH<8>

0.002

0.2675

0.930

GLYPH<8>

0.001

0.0081

elec

1.503

GLYPH<8>

0.035

1.577

GLYPH<8>

0.088

0.1222

1.543

GLYPH<8>

0.017

0.0880

1.552

GLYPH<8>

0.039

0.1228

1.581

GLYPH<8>

0.025

0.0027

wins / lose for us

GLYPH<209>

13 / 5

11 / 5

12 / 5

12 / 6

V.V.3.6 Full comparisons with MICE on missing data imputation

The main file provides just two examples of domains for the comparison, abalone and analcatdata_supreme . We provide here more complete results on the domain used in the main file and results on more domains in the form of one table for each additional domain:

Table A13: experiments on analcatdata_supreme completing the results shown in Table 4 (MF);

Table A14: experiments on abalone completing the results shown in Table 4 (MF); (tables below are ordered in increasing domain size, see Table A1)

Table A15: experiments on iris ;

Table A16: experiments on ringgauss ;

Table A17: experiments on circgauss ;

Table A18: experiments on gridgauss ;

Table A19: experiments on randgauss ;

Table A20: experiments on student_performance_mat ;

Table A21: experiments on student_performance_por ;

Table A22: experiments on kc1 ;

Table A23: experiments on sigma_cabs ;

Table A24: experiments on compas ;

Table A25: experiments on open_policing_hartford ;

The large amount of pictures to process may make it difficult to understand at first glance how our approaches behave against MICE, so we summarize here the key points:

domain

us (GF)

CT-GANs

tag

T

=200,

J

=500

E

GLYPH<16>

10

p

-val

E

GLYPH<16>

100

p

-val

E

GLYPH<16>

300

p

-val

E

GLYPH<16>

1000

p

-val

ring

0.283

GLYPH<8>

0.005

0.457

GLYPH<8>

0.058

0.0029

0.546

GLYPH<8>

0.079

0.0020

0.405

GLYPH<8>

0.044

0.0048

0.351

GLYPH<8>

0.042

0.0299

circ

0.356

GLYPH<8>

0.002

0.848

GLYPH<8>

0.300

0.0217

0.480

GLYPH<8>

0.040

0.0024

0.443

GLYPH<8>

0.014

0.0001

0.435

GLYPH<8>

0.075

0.0816

grid

0.392

GLYPH<8>

0.002

0.749

GLYPH<8>

0.417

0.1279

0.459

GLYPH<8>

0.031

0.0096

0.426

GLYPH<8>

0.022

0.0233

0.408

GLYPH<8>

0.019

0.1239

for

1.086

GLYPH<8>

0.124

9.796

GLYPH<8>

5.454

0.0241

1.899

GLYPH<8>

0.289

0.0028

1.532

GLYPH<8>

0.178

0.0020

1.520

GLYPH<8>

0.311

0.0313

rand

0.290

GLYPH<8>

0.007

0.746

GLYPH<8>

0.236

0.0129

0.491

GLYPH<8>

0.063

0.0017

0.368

GLYPH<8>

0.015

0.0003

0.327

GLYPH<8>

0.024

0.0124

tic

0.574

GLYPH<8>

0.001

0.601

GLYPH<8>

0.003

ε

0.581

GLYPH<8>

0.001

0.0024

0.586

GLYPH<8>

0.006

0.0299

0.584

GLYPH<8>

0.003

0.0228

iono

1.203

GLYPH<8>

0.068

2.263

GLYPH<8>

0.035

ε

2.084

GLYPH<8>

0.052

ε

1.98

GLYPH<8>

0.136

0.0001

1.758

GLYPH<8>

0.137

0.0005

stm

1.044

GLYPH<8>

0.013

1.511

GLYPH<8>

0.074

0.0001

1.189

GLYPH<8>

0.045

0.0106

1.168

GLYPH<8>

0.054

0.0346

1.167

GLYPH<8>

0.041

0.0036

wred

1.027

GLYPH<8>

0.037

2.836

GLYPH<8>

0.703

0.0049

2.004

GLYPH<8>

0.090

ε

1.825

GLYPH<8>

0.127

ε

1.384

GLYPH<8>

0.047

0.0002

stp

1.029

GLYPH<8>

0.027

1.660

GLYPH<8>

0.161

0.0010

1.141

GLYPH<8>

0.018

0.0004

1.206

GLYPH<8>

0.046

0.0015

1.186

GLYPH<8>

0.052

0.0057

ana

0.367

GLYPH<8>

0.018

0.542

GLYPH<8>

0.135

0.0399

0.439

GLYPH<8>

0.056

0.0196

0.404

GLYPH<8>

0.012

0.0231

0.436

GLYPH<8>

0.036

0.0018

aba

0.476

GLYPH<8>

0.017

1.689

GLYPH<8>

0.053

ε

1.463

GLYPH<8>

0.094

ε

0.754

GLYPH<8>

0.040

ε

0.657

GLYPH<8>

0.026

ε

wwhi

1.120

GLYPH<8>

0.013

1.770

GLYPH<8>

0.160

0.0009

1.849

GLYPH<8>

0.064

ε

1.284

GLYPH<8>

0.029

0.0006

1.158

GLYPH<8>

0.009

0.0117

arti

GLYPH<6>

0.830

GLYPH<8>

0.007

1.388

GLYPH<8>

0.109

4

0.0021

1.209

GLYPH<8>

0.018

4

ε

1.026

GLYPH<8>

0.013

4

0.0002

0.959

GLYPH<8>

0.013

3

0.0051

wins / lose for us

GLYPH<209>

14 / 0

14 / 0

14 / 0

14 / 0

first and most importantly, there is not one single type of our models that perform better than the other. Both Generative Forests and Ensembles of Generative Trees can be useful for the task of missing data imputation. For example, analcatdata_supreme gives a clear advantage to Generative Forests: they even beat MICE with random forests having 4 000 trees (that is hundreds of times more than our models) on both categorical variables (perr) and numerical variables (rmse). However, on circgauss , student_performance_por and sigma_cabs , Ensembles of Generative Trees obtain the best results. On sigma_cabs , they can perform on par with MICE if the number of tree is limited (which represents 100+ times less trees than MICE's models);

as is already noted in the main file (MF) and visible on several plots (see e,g, sigma_cabs ), overfitting can happen with our models (both Generative Forests and Ensembles of Generative Trees), which supports the idea that a pruning mechanism or a mechanism to stop training would be a strong plus to training such models. Interestingly, in several cases where overfitting seems to happen, increasing the number of splits can reduce the phenomenon, at fixed number of trees: see for example iris ( 50 GLYPH<209> 100 iterations for EOGTs), gridgauss , randgauss and kc1 (GFs);

some domains show that our models seem to be better at estimating continuous (regression) rather than categorical (prediction) variables: see for example abalone , student_performance_mat , student_performance_por and our biggest domain, open_policing_hartford . Note from that last domain that a larger number of iterations or models with a larger number of trees may be required for the best results, in particular for Ensembles of Generative Trees;

small models may be enough to get excellent results on real world domains whose size would, at first glance, seem to require much bigger ones: on compas , we can compete (rmse) or beat (perr) MICE whose models contain 7 000 trees with just 20 stumps;

it is important to remember that our technique does not just offer the capability to do missing data imputation: our models can also generate data and compute the full density conditional to observed values, which is not the case of MICE's models, crafted with the sole purpose of solving the task of missing data imputation. Our objective was not to beat MICE, but rather show that Generative Forests and Ensembles of Generative Trees can also be useful for this task.

J =80

J=100

0.9

0.9

0.9

0.9

0.85

0.8

generative tree

0.85

uniorm

generativetree

0.85

untorm

generativetree

0.8

0.85

generativetree

0.75

generativeforests

j0.75

0.75

0.75

0.7

0.7

generative forests

0

generative forests

0.7

generative forests

0.65

0.65

0.65

0.65

S

0.6

copy

0.6

copy

copy

0.6

copy

0.55

0.55

0.55

0.55

5

10

152025

3035

40

5

10

152025

3035

40

5

10

152025

3035

40

5

10

152025

6303540

#trees (T)

#trees (T)

#trees (T)

#trees (T)

unitorm

unitorm

0.95

0.95

0.95

0.95

0.9

generative tree

0.9

generativetree

0.9

generative tree

0.9

generative tree

0.85

0.85

0.85

qenerativeforests

0.85

aenerative forests

0.8

0.8

0.8

0.8

0.75

0.75

0.75

0.75

0.7

copy

0.7

copy

i 0.7

copy

i0.7

copy

0.65

0.65

0.65

0.65

5

10

1520

25

30

35

40

5

10

1520

25

30

35

40

5

10

1520

25

30

35

40

5

10

1520

25

30

3540

#trees(T)

#trees (T)

#trees (T)

#trees (T)

uniform

uniform

unitorm

0.95

generative tree

0.95

0.95

0.95

0.9

0.9

generative tree

0.9

generative tree

0.9

generatvetree

0.85

0.85

0.85

anorativre.forests

0.85

rafive_forests

0.8

0.8

0.8

0.75

0.75

0.75

10.75

0.7

copy

0.7

copy

0.7

copy

0.7

copy

5

10

152025

303540

5

10

152025

303540

5

10

152025

303540

5

10

152025

303540

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.95

1

.9

0.95

generativerveforests

0.95

generative tree

0.95

unirorm

generativetree

0.85

0.9

0.85

0.85

generative forests

0.85

generative forests

0.8

0.8

.8

0.75

0.75

0.75

0.75

uo

0.7

0.7

0.7

1

0.65

0.65

0.65

aba

0.6

0.6

0.6

0.6

5

10

152025

63035

40

5

10

152025

3035

40

5

10

152025

3035

40

5

10

152025

303540

#trees(T)

#trees (T)

#trees(T)

#trees(T)

37

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.55

180

0.55

130

(categorical

0.45

0.5

marginal

160

140

0.45

0.5

marginal

120

100

110

marginal

0.35

0.25

0.4

0.3

mice.rf (4000 trees)

ens.

dener-

100

120

80

60

marginal

0.35

0.25

0.4

0.3

0.2

ice.rf (4000 trees)

06

30

80

mice.rf (4000 trees)

generative forests

20

0.2

2

8101214161820°

40

6

81012

14161820

0.15

generative forests

2

4

81012

14161820

20

2

81012

14161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.55

140

0.55

130

0.45

0.5

0.4

marginal

130

110

100

120

marginal

0.45

0.5

0.4

marginal

120

100

110

90

marginal

90

0.35

80

uo

0.35

0.25

0.3

0.2

ens. of generative-tre

2

4

8

101214161820

50

80

70

60

40

2

4

8

10

ens.of generative trees

12

14

16

18

20

0.15

0.25

0.3

0.2

generative forests

2

mice.rf (4000 trees)

4

6

8

10

12

14161820

60 50 4

70

30

2

mice.rf (4000 trees)

6

1012

generative foreets

14161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.55

(sn n) s n

130

0.55

(n n) es und

130

0.45

0.5

marginal

120

100

110

marginal

0.45

0.5

marginal

120

110

100

marginal

t0

90

80

ba

0.35

0.4

90

80

0.35

0.3

ens. of qenerative

70

50

60

mice.rf (4000 trees)

ens.of generative trees

0.25

0.3

mice.rf (4000 trees)

5040

70

60

mice.rf (4000 trees)

0.25

0.2

mice.rf (4000 trees)

2

4

8101214161820

40

30

2

6

81012

1416

0.15

0.2

2

generatve forests

4

6

81012

14161820

30

4

6

1012

gonorative-foroot

14161820

2

#trees (T)

#trees (T)

#trees (T)

J =40

J = 80

J =100

0.006

0.3

0.018

0.012

0.016

0.005

0.25

0.01

0.014

0.01

30.008

S

0.001

0.05

0.004

0.002

0.002

5

10152025303540

10

152025303540

0

5

10152025303540

0

5

10152025303540

i

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.0007

0.0013

0.005

0.0016

0.0006

0.0012

0.0011

0.0045

0.004

0.0014

0.0004

国

0.001

0.0003

0.0008

0.0002

0.0005

0.001

0.0001

000°0

0.0005

000°0

10152025303540

0000

10152025303540

10152025303540

0.0002

510152025303540

#trees (T)

#trees (T)

#trees (T)

0.000013

0.00010

0.0002

0.0005

0.000012

.0

0.00008

0.00009

0.00016

0.00018

0.00035

0.00045

0.0004

0.003

0.00006

.005

0.00015

0.000008

00000

0.00002

2x10-5

4x10-5

0.0001

5x10-5

0.000007

510152025303540

0.00001

5

10152025303540

0

5

10152025303540

10152025303540

#trees(T)

#trees(T)

#trees(T)

#trees (T)

3x10-6

0.00035

0.0006

2.5x10-6

0000

90000

p-val us vs copy

2×10-6

0.

abalone

0

1.5x10-6

0.00015

0.5

1x10-6

5x10-7

5x10-5

0.0001

10

15202530

#trees (T)

35

5

10152025303540

#trees (T)

0

5

10152025303540

#trees (T)

10152025303540

#trees (T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

T

perr

rmse

perr

rmse

0.75

(sn an) s on

16

0.75

(sn n) as uond

14

0.7

14

0.7

12

marginal

ions

(categorical

0.65

90

atarguege

marginal

ens.orgenerative trees

0.65

0.6

manginal

forests

10

8

iterat

0.55

0.5

6

4

0.55

0.5

6

4

generative forests

0.45

mice.rf (4500 trees)

2

mice.rf (4500 trees)

ndu

0.45

mice.r (4500 trees)

2

mice.rf(4500 tree

0.4

6

8101214161820

0

4

6

101214161820

0.4

2

4

6

8101214161820

0

4

6

101214161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

(

0.75

(n n  n

18 16 14 12 10 8 6

20

0.7

(n en) es und

14

0.7

0.65

marginal

12

marginal

0.65

0.6

crative forests

10

ion:

0.6

marginal

8

0.55

0.55

ens. of generative trees

6

0.45

0.5

mice.rf (4500 trees)

2

4

mice.rf (4500 trees)

0.45

0.5

mice.rf (4500 trees)

2

mice.rf (4500 trees)

qenerative foreet

0

0.4

2

4

6

8101214161820

0

4

6

8101214161820

0.4

2

4

6

8101214161820

4

6

8101214161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

r(categoricalvariables)

4

0.8

0.8

ions

0.6

0.4

Graina

2.5

2

(categorical

0.5

0.7

0.6

marginal

2 15

malrglr

0.2

0

mice.rf (2500 trees)

0.5

1.5

mict d (25ou trees)

ens.ofg

0.4

0.3

0.2

0.1

generative forests

0.5

mice.rf (2500 trees)

generative forests

-0.2

8101214161820

0

2

4

8101214161820

mice.rf (2500 trees)

2

4

6

8101214161820

。

2

8101214161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.9

3.5

0.9

4

0.8

3.5

uor

S

(categorical

0.5

0.4

amarginalene

euees

2.5

2

rical

(categori

0.7

0.5

0.6

marginal

marginal

0.1

0.3

0.2

mice.rf (2500 trees)

0.5

1.5

mice.rf (2500 trees)

.oi generative trees

n

0.4

0.3

0.1

0.2

erativeforeots

0.5

1.5

mice.rf (2500 trees)

nerative forests

50

-0.1

2

4

6

8

101214161820

0

2

4

6

8

1012

14161820

0

mice.rf (2500 trees)

2

8101214161820

0

6

81012

14161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

3.5

0.9

( n

4

0.8

0.7

0.5

yenerative trees

2.5

marginal

0.6

marginal

imputation

0.4

0.3

0.2

0.1

1.5

2

1

mice.rf (2500 trees)

ens. of generative trecs

0.5

0.4

0.3

0.2

106

er.eraive forcosts

1.5

mice.rf (2500 trees)

gencrative forests

mice.rf (2500 trees)

0.5

0.1

0.5

-0.1

4

6

8101214161820

#trees (T)

0

2

4

6

81012

#trees (T)

214161820

2468101214161820

mice.rf (2500 trees)

#trees (T)

6

81012

#trees (T)

14161820

Ens.ofGenerativeTrees(EoGT)

Generative Forests (GF)

J

rmse

rmse

2.6

2.5

3

generativetree

2.4

2.5

2.3

622

vetree

misnga100u trees)

20 iterations

2

1.5

generative forests

ens.of generative trees

1.8

2

1012

1416182

20

2

4

6

8

101214161820

#trees(T)

#trees(T)

2.7

2.6

generativetree

2.5

2 2 2 13

100 iterations

1.8

1.6

ens.of generative trees

generative forests

1.4

1.8

2

4

6

8

10

12

14

41618

20

2

46

8101214161820

#trees(T)

#trees(T)

486222222

2.6

generativetree

2.4

generative tree

200 iterations

2

generative forests

ens.of generative trees

1.8

1.6

1.6

2

4

6

8

10

12

14

16

18

20

2

4

6

8

101214161820

#trees(T)

#trees(T)

2.5

3

2.4

2 2 24

2.3

generativetree

2.2

MirgHan

400 iterations

S

1.8

cns.og

1.9

generative forests

1.6

1.8

2

4

6

10

12

14

161820

2

6

8101214161820

#trees(T)

#trees(T)

Generative Forests (GF)

J

rmse

rmse

22

22

20

2 2019 18 716 1 14 13 12 11

generativetree

18 16

generativetree

14

nice.rf (1000 trees)

arginal

20 iterations

12

mice.rf (1000 trees)

10

8

rginal

6

ens.ofgenerativetrees

generative forests

2

4

1012141618

20

4

6

8

101214161820

#trees (T)

#trees(T)

22

24

20 1816

22

generativetree

generativetree

20

18

ice.rf (1000 trees)

100 iterations

14

16

marginal

12

14

marginal

10

ens.of generative trees

12

generative forests

8

10

2

4

6

8

10

12

14

16

18

320

2

4

6

8

10

12

214161820

#trees(T)

#trees(T)

18

2 2 2019 1817 1615 1413 12

17

generativetree

16

15

nice.rf (1000 trees)

generativetree

200 iterations

14

13

ginal

12

ens. of generative trees

marginal

generative forests

11

2

4

6

8

10

12

14

16

18

20

2

4

6

8

10

12

14161820

#trees(T)

#trees(T)

2 2019 18 7 16 15 14 13 12

18

generativetree

17

generativetree

16

15

mice.rf (1000 trees)

400 iterations

14

ce.rf(1000 trees)

13

marginal

marginal

generativeforests

ens.ofgenerative trees

12

11

11

2

4

6

8101214161820

4

6

8

101214161820

#trees(T)

#trees(T)

Ens.ofGenerativeTrees(EoGT)

Generative Forests (GF)

J

rmse

rmse

22

18

20

17 16

18 1614

generativetree

151413 121

nice.rt (1000 trees)

marginal

harginal

20 iterations

12

generative forests

10

11

8

ens.of generative trees

6

2

4

6

8

1012

14161820

2

4

6

8

1012

14161820

#trees(T)

#trees(T)

201918 171615 1413 121109

2 2 2019 18 7 16 15 14 13 12

generativetree

generativetree

ice

A(1000trees)

nargiral

100 iterations

mice.rf（1000 trecc)

ens.ofgenerative trees

marginal

qenerative forests

2

4

6

8

10

12

14

16

1820

2

4

6

8

10

12

14161820

#trees(T)

#trees(T)

24

2 2 20 19 18 7

2 20 18 16 4

generativetree

generative tree

200 iterations

micenou0trees)

marginal

16 15 14 13

12

mice.rf(100u trees)

marginal

10

ens.of generative trees

8

2

4

6

8

10

12

14

16

18

20

6

8

10

12

14161820

#trees(T)

#trees(T)

2 2019 18 716 15 14 13 12

20

19

18

generativetree

qenerativetree

17

400 iterations

T

16

narginal

marginal

generative forests

ens.of generative trees

11

13

2

4

6

101214161820

24

6

8

101214161820

#trees(T)

#trees(T)

Ens. of Generative Trees (EoGT)

Generative Forests (GF)

J

rmse

rmse

imputation rmse(numerical variables)

2 2 1 1 14 12 10 8 6

22

20 18 16 14

aenerative tree

marainal

generativetree

12

20 iterations

10

ens.ofgenerative trees

98

generativeforests

4

4

2

4

6

8

10

12

14

161820

2

4

6

8

1012

14161820

#trees (T)

#trees(T)

20

20 18 16 4

18

marqinal

marqinal

16

141210

generativetree

12

100 iterations

10

ens.of generative trees

generativetree

6

generative forests

4

2

2

4

6

8

10

12

14

16

1820

4

6

8

10

12

14161820

#trees(T)

#trees (T)

20

20 18 16 14 12 8

18 164 12

marcinal

marcinal

generativetree

200 iterations

generativetree

10

8

ens.ofgenerative trees.

6

mice.rf(1000trees)

generative forests

2

2

4

6

8

101214

161820

2

4

6

8

1012

14161820

#trees(T)

#trees(T)

2 20 18 16 14 12 10 8 6

20 18 16 14 12

maroinal

generativetree

400 iterations

10

8

generativetree

6

con(1000trees)

mice.n(ioootrees)ens.orgenerativeuees

generative forests

4

2

2

4

6

1012

214161820

2

6

8

101214161820

#trees(T)

#trees(T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.6

140

0.6

140

0.55

marginal

120

marginal

0.55

marginal

120

marginal

0.5

ens.of generative trees

imputation rmse (numerical

100

0.5

100

suo

80

80

0.45

0.45

generative forests

60

60

0.35

0.4

mice.rf (16500 trees)

40

20

ens.ofgenerative trees

0.35

0.4

mice.rf (16500 trees)

40

20

generative-lorests

0.3

46

8101214161820

0

mice.rf(16500 trees)

4681012

14161820

0.3

2

4

8

101214161820

0

246

mice.rf(16500 trees)

8101214161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.6

140

0.6

140

0.55

marginal

120

marginal

0.55

marginal

120

marginal

0.5

100

0.5

generative forests

100

80

80

0.45

ens.of generative trees

60

0.45

60

0.4

40

0.4

40

0.35

mice.rf (16500 trees)

20

ens.of generative trees

0.35

mice.rf (16500 trees)

20

generative forests

00

0.3

8

101214161820

0

mice.rf(16500trees)

2

46

81012

14

4161820

0.3

2

1012

14161820

0

mice.rf(16500 trees)

2

46

81012

14161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.6

(numerical variables)

140

0.6

140

0.55

marginal

120

marginal

0.55

marginal

120

marginal

ions

0.5

100

80

60

0.5

yenerative forests

100

80

0.45

0.45

0.4

ens.of generative trees

20

40

ens. of generative trees

0.4

40

60

0.35

mice.rf (16500 trees)

0

mice.rf(16500 trees)

0.35

mice.rf (16500 trees)

20

generative forests

00

0.3

8101214161820

-20

4

8101214161820

0.3

4

8101214161820

mice.rf(16500 trees)

2

8101214161820

2

#trees (T)

#trees (T)

#trees (T)

() s0#

0.6

180

0.6

140

0.55

0.5

marginal

rmse (numerical

160

140

120

marginal

0.55

0.5

marginal

generativeTorests

120

100

marginal

100

80

0.45

ens.of generative trees

80

0.45

60

0.4

60

0.4

40

0.35

mice.rf(16500 trees)

nd

20

40

ens.of generative trees

0.35

mice.rf(16500 trees)

20

generative forests

0.3

8101214161820

mice,rf(16500 trees)

2

4

8101214161820

0.3

2

6

81012

14161820

0

mice.rf(16500 trees)

2

4

6

8101214161820

4

#trees (T)

#trees (T)

#trees (T)

#trees (T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.6

35

s

0.6

(sn en) as nd

35

0.55

marginal

30 25

marginal

0.55

marginal

marginal

0.5

ens.of generative trees

20

(categorical

0.5

20

0.45

15

ens.of generative trees

0.45

generative forests

15

0.4

10

n

0.4

10

generative forecte

0.35

mice.rf (16500 trees)

5

0.35

mice.rf (16500 trees)

5

2

0.3

2

468101214161820

mice,rf(16500 trees)

2

468101214161820

0.3

2

46

81012

14161820

0

mice,rf(16500trees)

2

468101214161820

#trees (T)

#trees(T)

#trees (T)

#trees (T)

0.6

(sn n) as n

40

0.6

(sn en) as und

35

0.55

marginal

35 30 25 20 15

0.55

marqinal

30

marginal

0.5

marginal

0.5

gernerative forests

25

20

0.45

0.45

15

0.4

ens. of generative trees

10

ens.of generative trees

0.4

10

0.35

mice.rf (16500 trees)

5

0.35

mice.rf (16500 trees)

5

0

0.3

246

8101214161820

mice,rf(16500.trees)

2468101214161820

0.3

24

8

1012

14161820

2468101214161820

mice,rf(16500trees)

#trees(T)

#trees (T)

#trees (T)

#trees (T)

0.6

15154053025201 10 5 8

0.6

35

0.55

marginal

0.55

marginal

30

marginal

25

0.5

0.5

generative forests

0.45

marginal

0.45

20

15

0.4

ens. of generative trees

0.4

10

0.35

mice.rf (16500 trees)

ens.of generative trees

0.35

mice.rf (16500 trees)

5

0.3

8101214161820

0

mice.rf (16500.trees)

24681012

14161820

0.3

2

46

8

101214161820

0

2468101214161820

mice,rf(16500trees)

2

#trees (T)

() s

(） s#

#trees (T)

0.6

(san anu) as oend

15 0 5 0 25 20 15

0.6

imputationmse(numerical variables)

35

0.55

marginal

0.55

marqinal

30

marginal

25 20

0.5

marginal

0.5

generativeforests

0.45

0.4

ens.of generative trees

10

ens.of generative trees

n

0.45

0.4

15

10

generative forests

0.35

mice.rf(16500 trees)

5

0.35

mice.rf(16500 trees)

5

400

0.3

2468101214161820

0

mice,rf (16500.trees)

468101214161820

0.3

8101214161820

0

mice,rf(16500trees)

2

468101214161820

#trees (T)

#trees (T)

#trees (T)

#trees (T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.55

0.6

1.8x109

2x109

0.55

0.6

2x109

0.45

0.35

0.25

0.5

0.3

0.4

A.of generative trees

arginal

1.6x109

1.4x109

1.2x109

8x108

4x108

1x109

6x108

enernuvetree

ens. of generative trees

0.45

0.25

0.35

0.5

0.3

0.4

rainal

IOTE

1.5x109

5x108

1x109

generativetree

0.15

0.2

mice r (11000 trees)

2×108

0

0.15

0.2

mice r (11000 trees)

0

0.1

-2×108

0.1

-5x108

5101520253035404550

5101520253035404550

5101520253035404550

5101520253035404550

#trees (T)

#trees (T)

#trees(T)

#trees (T)

(

0.6

(numerical variables)

1.8x109

0.6

saen enu s uen

1.8x109

0.55

1.6x109

0.55

1.6x109

0.5

rainal

1.4x109

0.5

maroinal

1.4x109

0.45

1.2x109

0.45

1.2x109

1ons

0.4

1x109

0.4

1x109

0.35

0.25

0.15

0.3

0.2

micerf (11000 trees)

ens.ofgenerative trees

8x108

6x108

2x108

4x108

0

generative tree

n

0.35

0.15

0.25

0.3

0.2

generative-foroel

mice rf (11000 trees)

8x108

6x108

4x108

2x108

0

neralivekeeforess

0

0.1

-2x108

0.1

-2×108

5101520253035404550

5101520253035404550

5101520253035404550

5101520253035404550

#trees(T)

#trees (T)

#trees (T)

#trees (T)

variables)

0.65

0.6

1.8x109

1.6x109

0.55

0.6

(s n  n

1.8x109

1.6x109

0.55

1.4x109

0.5

marainal

1.4x109

ions

0.45

0.5

m

1.2x109

1x109

0.45

0.4

1.2x109

1x109

0.35

0.25

0.15

0.3

0.2

0.4

ens.of generative trees

mice f (11000 trees)

8x108

4x108

6x108

2x108

0

generative tree

0.35

0.25

0.15

0.2

0.3

generativeforests

mice rf (11000 trees)

8x108

6x108

4x108

2x108

0

ceneralpetrtieefores

0

0.1

-2x108

0.1

-2x108

5101520253035404550

510 1520253035404550

5101520253035404550

5101520253035404550

2

#trees (T)

#trees (T)

(） s

#trees (T)

0.6

1.8x109

0.6

1.8x109

0.55

1.6×109

0.55

1.6x109

0.5

marginal

1.4x109

0.5

marainal

1.4x109

0.45

1.2x109

0.45

1.2x109

ions

0.4

1x109

0.4

1x109

iterati

i

0.35

0.25

0.3

ens.of generative trees

4x108

6x108

8x108

generativetree

(cat

0.35

0.25

0.3

6x108

8x108

4x108

500

0.15

0.2

0.1

mice rf (11000 trees)

imputation

-2x108

2×108

0

0.15

0.2

0.1

micerf(11000frees)

-2×108

2x108

0

generaiveee forests

5101520253035404550

510 15 202530 3540 4550

5101520253035404550

510 15 20 2530 3540 4550

#trees (T)

#trees (T)

#trees(T)

#trees (T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.75

marginal

0.75

0.7

0.7

0.65

enerativetree

0.65

generative tree

0.55

0.6

mice.rf (6500trees)

of generative trees

ens.of generative trees

0.55

0.6

enerativeforests

0.5

102030405060708090100

102030405060708090100

0.5

102030405060708090100

102030405060708090100

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.75

0.75

marginal

marginal

0.65

0.7

L0

generative tree

0.65

0.6

#

generativeforests

0.55

Hice9t

generative tree

imputation err (

0.6

0.5

ns.of generative trees

0.55

mice.rf (6500 trees)

0.45

0.5

10203040506070809010

102030405060

708090100

102030405060708090100

102030405060708090100

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.75

0.75

marginal

marginal

0.7

0.7

generativetree

0.65

0.65

generative forests

0.6

-generative tree

0.6

auveforests

0.55

enisedfCheralv

cno.ofgcnierative trees

0.55

mice.rf (6500 trees)

0

0.5

102030405060708090100

102030405060708090100

0.5

102030405060708090100

102030405060708090100

2

#trees(T)

#trees(T)

#rees (T)

#trees(T)

Ensembles of Generative Trees (EoGT)

Generative Forests (GF)

perr

rmse

perr

rmse

0.55

0.55

0.5

marqinal

0.45

0.5

0.45

Lns.of generative trees

0.4

0.4

0.35

0.3

err

0.35

0.3

50

generativetree

ens.of generative trees

0.25

0.15

0.2

ice.rf(7000trees)

50

generative tree

0.25

mice.rf (7000 trees)

ndui

0.1

erative forests

0.2

。

0.05

0

102030405060708090100

102030405060708090100

102030405060708090100

102030405060708090100

#trees (T)

#trees (T)

#trees (T)

0.55

0.55

0.5

marqinal

0.5

marainal

ions

0.45

0.4

uobe

0.45

0.4

0.35

ens.of generative trees

0.35

0.3

generative tree

0.3

50

n

0.25

mice.rf ru00 trees)

5

0.25

mice.rf (7000 trees)

90:rf (7000-Ir90

0.2

enerative forests

00

0.2

102030405060708090100

0

102030405060708090100

0.15

102030405060708090100

0

102030405060708090100

#trees (T)

#trees (T)

#rees(T)

#trees (T)

0.55

0.55

0.5

marqinal

0.5

marqinal

0.45

0.45

uobal

t0

0.4

0.35

0.35

0.25

0.3

eis. or generative rees

mice.rf (7000 trees)

50

geerativegeerativetees

0.25

0.3

50

generativetree

0.2

102030405060708090100

0

102030405060708090100

0o.r1 (7000-

0.2

102030405060708090100

0

102030405060708090100

eroboaf

5

#trees (T)

#trees (T)

#rees (T)

#trees (T)

EnsemblesofGenerativeTrees (EoGT)

GenerativeForests(GF)

perr

rmse

perr

rmse

0.7

marainal.

0.7

marqinal

0.6

ve-trees

0.6

0.5

220

0.5

220

suo

0.4

200

18

ens.of generative trecs

0.4

180

200

iterat

0.3

0.3

generative forests

0.2

0.2

gereraavecative.tree.

0.1

mice.r.(10.000.trees

mice.rf(10 000 trees)

0.1

ice.rf(10.000trees

mice.rf (10 000 trees)

20

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.7

0.7

narqinal

marginal

0.6

marginal

0.6

ens.of generative trees

0.5

0.5

suo

0.4

0.4

yenerativeforests

0.3

0.3

0.2

0.1

mice.rf (10.000.trees

50

0.2

0.1

ce.rf (10.000.trees)

80

ativetree

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

#trees (T)

#trees (T)

#trees (T)

#trees (T)

0.7

0.7

marqinal

marginal

marainal

marginal

0.6

0.6

lons

0.5

ens.of generative

180

0.5

nciauiveTorests

220

0.4

160

0.4

10

0.3

0.3

0.2

0.2

0.1

mice.rf (10.000.trees)

80

60

cns.Uranuum

0.1

mice.rf (10.000.trees)

80

mice.rf (10 000 trees)

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

5

#trees (T)

#trees (T)

#trees (T)

#trees(T)

0

KDE

-0.5

-1

expected log-dens

-1.5

-2

-2.5

only

sometreesnotstumps

stumps

-3

-3.5

-4

-4.5

0

500

1000

1500

2000

# splits GF

T

GLYPH<16>

500

, J

GLYPH<16>

2000

T

GLYPH<16>

15

, J

GLYPH<16>

10000

10*109

KDE

10*109

KDE

100*106

100*106

expected dens

1*106

dens

1*106

10*103

expected

10*103

100*100

100*100

1*100

1*100

10*10-3

10*10-3

0

500

1000

1500

2000

0

2000

4000

6000

8000

10000

#splits GF

#splits GF

V.V.3.7 Full comparisons with KDE on density estimation

For each domain we either use the expected density, or the expected negative log-density, as the metric to be maximized. We consider the expected density to cope with the eventuality of zero (pointwise) density predictions. Tables A28 and A29 present the results against KDE (details in Table A28). Table A26 singles out a result on the domain gridGauss , which displays an interesting pattern. The leaf chosen for a split in GF.BOOST is always the heaviest leaf; hence, as long as the iteration number j / T (the maximum number of trees, which is 500 in this experiment), GF.BOOST grows stumps. In the case of gridGauss , like in a few other domains, there is a clear plateau in performances for j / 500 , which does not hold anymore as j ¡ 500 . This, we believe, characterizes the fact that on these domains, which all share the property that their dimension is small, training stumps attains a limit in terms of performances. Table A27 shows that by changing basic parameters - here, decreasing the number of trees, increasing the total number of iterations -, one can obtain substantially better results.

ringGauss

circGauss

forestfires

randGauss

tictactoe

ionosphere

student_performance_mat

winered

1.5

-2.4

1*106

KDE

1

KDE

-2.6

KDE

-2.8

expected dens

10*103

expected log-dens

0.5

expected log-dens

E-

0

-3.2

100*100

-0.5

-3.4

-1

-3.6

1*100

-3.8

-1.5

-4

10*10-3

-2

-4.2

-2.5

-4.4

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

# splits GF

# splits GF

# splits GF

0.01

-1

1*10-3

KDE

KDE

KDE

0.0001

-1.5

1x10-6

1log-dens

-2

expected dens

1x10-8

expected dens

-2.5

1x10-10

expectedle

-3

100*10-6

1x10-12

-3.5

1x10-14

1x10-16

-4

1x10-18

-4.5

1x10-20

-5

10*10-6

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

#splits GF

# splits GF

# splits GF

20

1*1015

KDE

1*106

KDE

15

10

10*10-9

10*109

expected log-dens

5

100*10-12

expected dens

0

expected

1*10-12

100*103

-5

10*10-15

-10

1*100

-15

100*10-18

-20

1*10-18

10*10-6

-25

10*10-21

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

#splits GF

# splits GF

# splits GF

student_performance_por

analcatdata_supreme

winewhite

compas

artificial_characters

10*10-3

100*106

KDE

10*1039

KDE

100*10-12

KDE

100*1033

1*106

1*10-15

1*1030

dens

10*10-9

dens

10*1024

dens

10*10-21

expected

100*10-12

100*1018

expected

100*10-27

1*10-12

1*1015

1*10-30

10*10-15

10*109

10*10-36

100*10-18

100*103

100*10-42

1*10-18

1*100

10*10-21

1*10-45

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

# splits GF

#splits GF

# splits GF

10*109

KDE

100*1078

KDE

KDE

10*109

10*1069

1*1060

100*103

expecteddens

100*103

dens

100*1048

dens

expected

10*1039

expected

1*100

1*100

1*1030

100*1018

10*10-6

10*109

10*10-6

1*100

100*10-12

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

# splits GF

#splits GF

# splits GF

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims are supported by both the theoretical results presented in the paper and extensive experimental results, summarized in the main paper but otherwise presented in extenso in the Supplement.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Limitations are discussed, in particular in the experimental section vs the SOTA (see e.g. experiment Lifelike).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are detailed, formal results given and full proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: (i) full code provided, (ii) extensive README.txt, (iii) coding choices discussed in Appendix, (iv) all scripts provided for easy running of each experiments, (v) special visualization routines (e.g. heatmaps) also included in the code, (vi) example domains (public and simulated) provided for quick assessment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: (i) full code provided, (ii) extensive README.txt, (iii) coding choices discussed in Appendix, (iv) all scripts provided for easy running of each experiments, (v) special visualization routines (e.g. heatmaps) also included in the code, (vi) example domains (public and simulated) provided for quick assessment.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in main text and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: standard devs and inferential statistics test done (details in main file + appendix).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: Our code runs on any standard computer. Details of codes, computer and SOTA used (version, etc.) in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research of the paper follows the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No release of data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Appropriate credit / referencesse are provided. Licenses listed in Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets provided.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generative model for tabular data based on a forest of trees. It is evaluated on the quality of the generated samples (using an optimal transport based distance to a held out test set), on data imputation and density estimation.\n\nSTRENGTHS:\nGenerative modeling for tabular data is an important research topic, and exploiting the strengths of trees (ubiquitous in this modality) for this generative task is a promising direction.\n\nThe framework inherits the flexibility of boosting, and the theoretical results (under weak learning assumptions) provide guarantees that the underlying distribution can be approximated well enough with enough trees.\n\nWEAKNESSES:\nFurther empirical comparison could be conducted. For example, recent works have applied diffusion for tabular data (Kotelnikov et al.) as well as flow matching (Jolicoeur-Martineau et al.). It might be worth comparing to these more recent approaches.\n\nFurthermore, more metrics could be included for the LifeLike experiment to provide more nuanced results. See Jolicoeur-Martineau et al. for examples of such metrics.\n\nReferences:\n\nKotelnikov, A., Baranchuk, D., Rubachev, I., & Babenko, A. (2023, July). Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning (pp. 17564-17579). PMLR.\n\nJolicoeur-Martineau, A., Fatras, K., & Kachman, T. (2024, April). Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. In International Conference on Artificial Intelligence and Statistics (pp. 1288-1296). PMLR.\n\nQUESTIONS:\nWhat is the computational cost for training and inference compared to Generative Trees? (It would be interesting to provide wall-clock times)\n\nHow are the parameters chosen (number of trees, number of splits, ...)?\n\nLIMITATIONS:\nSee questions", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces generative forests (GF), a new class of generative models designed for tabular data and based on sets of trees (forests), and a training algorithm called GF.BOOST. The algorithm is designed to be simple, making it easy to implement with minor modifications to existing CART-style decision tree induction schemes. \n\nThe GF models and GF.BOOST algorithm are evaluated across various domains, demonstrating their capability in tasks such as data generation, missing data imputation, and density estimation. The results show that even a small number of trees within a generator can be effective, highlighting the potential of the proposed approach in handling tabular data.\n\nSTRENGTHS:\n- I believe the paper is a nice addition to the ML community. Prevailing methods in data imputations for DTs are based on heuristics and tricks, whereas this paper approaches this problem with sounding probabilistic point of view, which makes a lot of sense to me. This naturally leads to the ability of generating more data, which may have serious implications in synthetic data generation literature. \n- The method is relatively easy to implement and can be done by straightforward extension of CART-style tree induction methods.\n\nWEAKNESSES:\n- Experiments section may need some improvements:\n     - Most datasets in Table 2 (which seem to be the main results table) are standard ML benchmarks. It would be really nice to have real-world kaggle-style datasets with missing values and compare with strong baselines therein.\n     - Adding traditional forests is also necessary to compare against established methods (xgboost, lightgbm, etc.)\n\n- I find most of the math notations hard to follow and can be significantly simplified.\n\nQUESTIONS:\n- How computationally efficient to compute Eq. 4? From my understanding it is used for splitting a tree\n- Is there any mechanism to prune some trees in a forest (reduce T)? Traditional tree boosting builds trees sequentially, so there is opportunity for early stopping. I was wondering how critical here to have predefined T...\n- Is there any separate ablation how well this method works for categorical features?\n\nLIMITATIONS:\n- there seems to be some limitations in convergence analysis (e.g. symmetry of l). \n- otherwise, having separate (sub)section on limitations would be nice (e.g. runtime compared to traditional boosting methods).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generative model based on an ensemble of trees (forest) for tabular data. The proposed model enjoys the following peculiar properties:\n1. Compared to generative models based on a single tree, it offers improved **expressiveness** in terms of partitioning the input space (linear for generative trees vs. polynomial for generative forest in terms of decision nodes in a tree).\n2. The model offers **unbiased** density estimation for sufficient large capacity, thanks to the fact that each node in each tree leverage decisions in proportion to the empirical distribution.\n3. Learning the structure of the trees is performed by minimizing an objective function based on the likelihood ratio risk. \n4. Computationallly speaking, data generation and density estimation can be performed in **parallel** with respect to the different trees. Indeed, results can be aggregated a posteriori by taking the intersection of selected regions.\n5. Apart from **data generation** and **density estimation**, the model can be extended to deal missing data and perform **data imputation**.\n\nExperiments are conducted on a series of tabular datasets from the UCI repository. The proposed solution is shown to outperform/being on par with existing tabular data generation (the most relevant being adversarial random forests), data imputation (e.g. MICE) and density estimation baselines (e.g. kernel density estimation).\n\nSTRENGTHS:\n1. The problem of learning a good generative model for tabular data is relevant **Relevance**\n2. The solution based on a tree ensemble represent a significant advance over solutions based on a single decision tree  **Significance**.\n3. The proposed model offers the possibility to tackle three different tasks, including density estimation, data generation and data imputation. Moreover, experiments demonstrate the efficacy of the solution when compared with corresponding baselines for each task **Significance**.\n4. The theory of the paper seem correct, however I haven’t gone through the proofs **Correctness**\n5. Code is available at the time of submission. No additional check has been conducted to verify the reproducibility of the results **Reproducibility**\n\nWEAKNESSES:\n1. One of the major concerns is with the novelty of the proposed solution **Novelty**. What is the relation with the work in [1]? The work in [1] is also providing a model able to tackle the three above-mentioned tasks. Would it be possible to provide a comparison?\n2. The paper is sometimes overloaded by unnecessary formalism, rendering the text hard to follow **Clarity**. For instance,\n    1. Line 92-109, why is the whole theory about risk minimisation for binary classification introduced, when the goal is to perform density estimation? Would simple likelihood learning suffice?\n    2. Definition of tree. “…labeled with subsets of their tail node’s variable domain”. Do you mean “”labeled with subset of attribute value for the corresponding variable?\n    3. Line 160-163. Can you make an example or in case leave out the sentence? Am I correct to say that the permutation invariance property is due to the final intersection operation or am I missing something?\n    4. Algorithm 3 in Step 2.1 How is node selection performed, uniformly at random? Moreover, can you provide some more details about the resulting trees, are they balanced?\n    5. Can you elaborate on the motivation of proving theorem 5.2 and discuss the significance of its results or link with the results in the experiments?\n3. All experiments are conducted on datasets with less 34 variables **Soundness**. Would it be possible to provide some comparisons with [1] on larger dimensional datasets (see the reference for example) and/or discuss the computational requirements?\n\n**Reference**\n\n[1] Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees. AISTATS 2024\n\nQUESTIONS:\nOverall, I like the extension of generative trees to ensembles. I’m willing to raise my score if all weaknesses are addressed during the rebuttal phase.\n\nLIMITATIONS:\nThere are no additional suggestions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel boosting algorithm based on ensembles of generative trees, addressing tasks such as data binary classification, missing data imputation, and density estimation. The proposed algorithm optimizes specific loss functions and leverages the structure of the models to efficiently estimate densities given a generative model and partially specified observations. The authors demonstrate the practicality of their approach through comprehensive experiments, highlighting its effectiveness in binary classification compared to diverse state-of-the-art methods, including their competitors like Adversarial Random Forests (ARF), CT.GAN,  Vine copulas auto-encoders (VCAE) or Kernel Density Estimations(KDE). \n\nThe authors introduce a unique training methodology for these generative models, which is based on class probability estimation (CPE) in binary classification tasks. The loss functions are decomposed into partial losses for positive and negative classes, with the Bayes risk function defined as the optimal achievable loss given a certain positive base-rate. Properness and strict properness of the loss functions are critical properties, ensuring that the loss is minimized when the predicted probability matches the true class probability. The paper discusses various proper losses, such as square loss, logarithmic loss, and Matusita's loss, which are symmetric and differentiable. Furthermore, the paper explores loss functions in the context of density ratio estimation and introduces the likelihood ratio risk for assessing the performance of generative models. Using Bregman divergence and the generalized perspective transform, the authors propose a framework for evaluating the density ratios of different distributions. \n\nThis theoretical foundation supports the practical applications of their models, which show competitive performance over the dataset proposed. The authors carried out experiments on a total of 21 datasets from sources such as UCI, Kaggle, OpenML, and the Stanford Open Policing Project, or simulated datasets. These include datasets like iris, ringGauss, gridGauss, forestfires, tictactoe, ionosphere, student performance, winered, abalone, kc1, sigma-cabs, compas, artificial characters, jungle chess, open-policing-hartford, and electricity, with varying attributes and missing data conditions. The paper concludes that their models, trained with the novel generative tree algorithm, are strong contenders against existing state-of-the-art techniques.\n\nSTRENGTHS:\nRegarding to the originality, the paper introduces a novel approach to generative modeling by creating a generative forest based on Class Probability Estimation (CPE). This method represents a significant advancement over current state-of-the-art algorithms. By leveraging ensemble methods and decision tree induction for generative tasks, the paper presents a fresh perspective that has not been extensively explored in the literature. The combination of well-known techniques in a new context adds substantial value to the field and showcases the authors' innovative approach to generative modeling.\n\n\nThe quality of the submission is commendable. The paper is well-organized and meticulously indexed, with a comprehensive appendix that supports the main content. The methodology is robust, encompassing a detailed mathematical explanation, pseudocode for the proposed algorithm, and extensive experimental validation. The authors have tested their model on 21 diverse datasets from reputable sources such as UCI, Kaggle, OpenML, and the Stanford Open Policing Project. This thorough evaluation across various tasks, including data generation, missing data imputation, and density estimation, provides strong evidence of the model's performance and versatility. Additionally, the paper continues the line of research in generative models, building on previous work and demonstrating significant progress in this domain.\n\n\nThe paper is clearly written and well-organized, ensuring that readers can easily follow the authors' reasoning and methodology. The detailed descriptions of the mathematical foundation, the pseudocode, and the experimental setup are particularly noteworthy. These elements allow that an expert can reproduce the results. Furthermore, the inclusion of a comprehensive appendix adds to the clarity by providing additional details without cluttering the main text.\n\n\nThe results presented in the paper are of high value.  In that line, the authors have made a commendable effort by comparing their method with the closest approaches (ARF, copycat) cited in [42] and [29]. Furthermore, they have clearly outlined the distinctions and enhancements of their algorithm in comparison to these approaches.\n\nThe authors suggest that generative forest can effectively eliminate \"the slack\" when utilizing decision trees as discriminators. This implies that decision trees exhibit a more regulated and foreseeable behavior in contrast to neural networks. Decision trees are inherently more calibrated and provide more deterministic outcomes. As a result, the feedback they provide to the generator during training is more accurate and reliable. The proposed method seems to enhance binary classification performance compared to existing approaches and also addresses challenging tasks such as missing data imputation and density estimation. The model's ability to produce competitive results across a diverse array of datasets and tasks underscores its potential impact on both the research community and practical applications.\n\nWEAKNESSES:\nIn section 1, the authors delve into the use of generative AI in the ML community and highlight two significant features of their generative forest. They emphasize its simplicity and the strong convergence guarantees it offers in a weak/strong learning model, akin to Valiant’s PAC learning boosting model. However, it remains unclear whether this generative forest, based on Class Probability Estimation (CPE), can effectively handle multiclass scenarios.\n\nSection 3 introduces the supervised loss for the generative tree. Nevertheless, from a synthesized perspective, the drafting in line 75 lacks clarity. The authors should consider enhancing the clarity of these lines.\n\nMoving on to Section 4, in line 125, the authors refer to 'c', but it is not clearly connected to the context provided in the text.\n\nRegarding the supplementary material, Table A3 is challenging to read as the footnotes overlap with the image.\n\nQUESTIONS:\nIn Table 2, the final row displays the wins and losses for us. I am not closely following the numbers as you have already chosen the green one, which includes p-values indicating significance. Another query pertains to comparing the number of trees. It seems unfair to compare T=500 with competitors having fewer trees (10, 50, 100, 200). In simpler terms, I coudn’t comprehend why you mentioned that all these approaches rely on models that are vastly distinct from each other. I know that in the article the authors mentioned that ARFs are trained with a varying number of trees T ∈ {10, 50, 100, 200}. Additionally, ARFs incorporate an algorithm for tree size selection, eliminating the need for manual selection. On the other hand, CT-GANs undergo training with a different number of epochs E ∈ {10, 100, 300, 1000} but I don't follow why these models are then compared with generative forests comprising T = 500 trees and trained over a total of J = 2000 iterations. That approach was done for Table 2 and 4.\n\nLIMITATIONS:\nThe work presented by the authors is highly promising since they have introduced a novel methodology for constructing generative tree models based on class probability estimation (CPE) as a loss function in binary classification tasks. However, the paper would benefit from a dedicated section discussing the current limitations of this method. Specifically, it would be useful to understand whether this approach can be applied with the same efficacy to multiclass classification or, with some modifications, to regression tasks.\n\nAdditionally, it would be valuable to identify the types of tabular data that the proposed method might struggle with or the conditions under which its performance may be suboptimal. For instance, are there specific data types or distributions where this model does not perform as well? A thorough examination of these aspects could enhance the article.\n\nMoreover, while the authors have laid a strong foundation for binary classification problems in their work, constructive suggestions for improvement include providing an analysis or discussion on how the current method could be extended or adapted to handle multiclass classification tasks or regression problems, which would be helpful.\n\nIn that line, another interesting thing could be identifying the specific types of tabular data that might present challenges for the proposed method, which would be beneficial. Under what conditions does the model's performance decline, and what are the characteristics of datasets where this approach is less effective?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel generative model (Generative Forests - GF) for sampling and density estimation, leveraging an ensemble of trees. Their method partitions the input domain by considering all possible intersections of the supports of the leaves across the ensemble, resulting in a much finer partitioning than a single tree could achieve. For each subset of the input domain thus obtained, the model estimates a uniform density based on an empirical training measure. Efficient algorithms for sampling from such a model and evaluating the densities are derived.\n\nTo train the model, the authors propose an algorithm that iteratively splits a selected leaf node in one of the ensemble's trees. This splitting is guided by minimizing a strictly proper loss for a binary class probability estimation task.\n\nThe authors evaluate their method's performance against competing approaches across three main dimensions: the quality of synthetic data generation, missing data imputation, and density estimation.\n\nSTRENGTHS:\n- I find that the paper reads resonably clearly. The language and notation used throughout the paper is very precise although it occasionally appears overly complex.\n\n- The proposed method is interesting and I believe enjoys reasonable novelty.\n\n- The method and theoretical results all appear to be sound.\n\n- The proposed method's ability to estimate densities makes it more versatile than other generative methods commonly used for tabular synthetic data generation. Few methods can match its generality in handling various types of tabular data, except for other tree-based methods like Adversarial Random Forests (ARF). In this context, the positive results presented in comparison to ARF are promising.\n\n- The authors have made their implementation code available.\n\nWEAKNESSES:\nWhile the authors adequately explain their ensemble tree-based model and its sampling and density computation methods, the explanation of the training algorithm is less thorough. There is insufficient discussion on how the method scales computationally with the number of trees and splitting iterations (see my first question). Additionally, more discussion on the motivation behind the formulation of the CPE task would be welcome.\n\nAccording to the paper's Lemma 4.4, GF estimates a different density value at each non-empty intersection of the supports of each leaf, derived directly from the empirical training measure. This is similar to ARF or other works such as [1] or [2] when using uniform densities at the leaves with the exception that these methods would output an averaged density estimated independently from leaves of each tree in the ensemble.\nIt is unclear why the proposed approach should be better. From a generalization perspective, I would intuitively assume that averaging densities from multiple trees has a regularizing effect compared to using a single tree (see [2]).\nOn the other hand, GF's density estimation is based on a finer partitioning, whose cardinality grows with the product of the depths of all trees in the ensemble. While this could providing greater representational capacity, it would seem to make the method more susceptible to overfitting.\n\nThe theoretical analysis provided also focuses only on approximating this empirical training measure, without addressing overfitting and the approximation of the data-generating measure.\nFor a model such as described above, it seems obvious that the less coarse the partitioning becomes, the more faithfully an empirical distribution can be approximated. I am therefore unsure of the practical significance of these results and I think the paper would benefit more from a discussion of generalization ability.\n\nMy main concern with the paper, however, lies with the evaluation and comparison to other methods:\n- For the many analyses presented in the main paper and the appendices, different methods are compared against and for each method/scenario combination different datasets are used. This lack of consistency in comparisons and datasets in the presented results, leaves an impression that results are missing.\n- The choice of datasets focuses on relatively small numbers of data points and mixes both synthetic toy data and real datasets together. This is compounded by some of the main results being presented only as summary data (#wins/#ties/#losses) over this heterogeneous group of datasets.\n- There is no serious attempt to compare the proposed method with state-of-the-art approaches for synthetic data generation, particularly neural network-based ones:\n    - The comparison is limited to CT-GAN, which has been shown to underperform almost every other popular neural network method, including the TVAE method put forward as a baseline in the same paper.\n    - Additionally, the authors do not tune any hyperparameters for CT-GAN beyond the number of epochs. As a GAN, CT-GAN can have convergence issues, and without proper tuning, it may fail to converge. This lack of tuning undermines the fairness of the comparison, especially since it is unclear how much the authors optimized the hyperparameters of their own method.\n    - The authors also report unusually long training times on a very small dataset, presumably because no GPU is used to train these models and possibly training for too long to no avail. While I understand that the proposed method can be trained on a CPU alone, and maybe that is the point being made, I don't think this remark is made in good faith. In my experience, methods like CT-GAN, TVAE, and TabDDPM can achieve reasonable performance quickly (within minutes to tens of minutes) when trained on a consumer GPU (e.g., 2080) even on larger datasets than those used in the reported experiments.\n    - Results for CT-GAN on the two larger datasets are not reported, so the comparison primarily relies on very small datasets where neural networks might perform worse.\n\n  A comparison with TabDDPM [3], a strong neural network-based generative method, would be valuable, as it is currently a standard benchmark in papers evaluating synthetic data and, in my opinion, a consistent contender for a state-of-the-art method.\n\n- The evaluation of generated samples relies solely on an Optimal Transport distance metric. Although I am not well-versed in the practical computation of this metric, the authors' comment about selecting an entropic regularization parameter that avoids numerical instabilities across all domains is not particularly reassuring. It raises the question of why other metrics were not used in conjunction.\n  - Additionally, the setup described by the authors (GEN-DISCRIM), which uses Random Forests to distinguish real from synthetic data, appears both reasonable and well-founded. I am curious why this wasn't the primary evaluation method for comparing with other approaches. Instead, it is only used to compare GF against optimistic and pessimistic baselines.\n  - On this topic, the authors might also want ot look at e.g., [4] for utility metrics in machine learning tasks that are common in comparing synthetic data generators, although I wouldn't personally fault them if they choose to disregard such metrics.\n\n- Finally, given that the proposed model is able to estimate densities and that evaluating the log-likelihood of held-out data is the gold standard for objectively measuring the quality of such models I don't understand why this evaluation doesn't have a more prominent place in the paper:\n  - The authors only compare their method to KDE, which GF seems to underperform on several datasets, particularly those with a moderate number of dimensions. This is concerning, and I think it is important to investigate how the proposed method scales to higher-dimensional data compared to KDE. \n  - ARF is missing as a baseline in these evaluations which is strange since it is another model that is capable of estimating densities. Additionally, other methods, such as [2], report consistently outperforming KDE, making it worthwhile to include as an additional comparison in my view.\n  - Finally, the evaluation setup is not throughly explained in the main paper and only summary results against KDE are presented. It seems that the authors don't always use log-likelihood to compare these models due to their proposed model predicting 0 density for test samples (see question 3 below). This issue is only mentioned in the appendix but should, in my view, be disclosed in the main paper.\n\n\n[1] Alvaro H. C. Correia, Robert Peharz, Cassio de Campos. Joints in Random Forests, 2020 (https://arxiv.org/abs/2006.14937)\n\n[2] Hongwei Wen, Hanyuan Hang. Random Forest Density Estimation, 2022 (https://proceedings.mlr.press/v162/wen22c.html)\n\n[3] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, Artem Babenko. TabDDPM: Modelling Tabular Data with Diffusion Models, 2022 (https://arxiv.org/abs/2209.15421)\n\n[4] Lasse Hansen, Nabeel Seedat, Mihaela van der Schaar, Andrija Petrovic. Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark, 2023 (https://arxiv.org/abs/2310.16981)\n\nQUESTIONS:\n1. In line 218 the authors mention regarding finding the optimal split in $\\text{splitPred}$ that \"the optimisation is no more computationally demanding\" than for decision tree induction. I'm not sure if this is a typo but it seems to me that, as the trees in the ensemble grow deeper, finding the best split could quickly become intractable.\nAs an example, in the case where only single stumps are used, the potential cardinality of $\\mathcal{P}(\\mathcal{T})$ at iteration $j$ is $2^j$ which means that intersections with that many subsets have to be considered when finding the best split point. This seems like an untenable scaling of the computation with the iteration number but maybe I am misunderstanding something about how the best split is selected as the authors' explanation is not clear.\n\n2. Regarding density modelling the authors only compare with KDE. Since ARF also provides density estimates why not compare against this method as well? Additionally [2] could also be worthwhile comparing against.\n\n3. Could the authors clarify what they mean by \"expected densities\" that are used \"to cope with the eventuality of zero (pointwise) density predictions\" in appendix section V.V.3.7? If their proposed model assigns 0 probability to a subset of the domain that has non-zero mass under the data generating distribution I would say that this is a defect of the model and a sign of poor regularization. Changing from the commonly used log-likelihood metric to one that doesn't penalize this so harshly should be surfaced and discussed in the main text in my opinion. Furthermore, it seems that this choice is made on a dataset-by-dataset basis, I can only assume depending on whether the proposed model yields $-\\infty$ log likelihoods which seems to happen for most non-toy datasets. As it stands, these details are hidden in the appendix while the main paper presents a potentially misleading picture based only on #wins.\n\n4. Still regarding 0 density predictions, the authors propose a way to avoid this issue for GFs in line 173. It would be interesting if this was explored further and the resulting (more regularized) density model compared to other approaches for density estimation based on log-likelihood.\n\n[2] Hongwei Wen, Hanyuan Hang. Random Forest Density Estimation, 2022 (https://proceedings.mlr.press/v162/wen22c.html)\n\nLIMITATIONS:\nI find that the authors hardly discuss the limitations of their method. Most importantly there is little discussion of training times and how they scale with the number of iterations/data dimensions/number of samples. There is also no discussion of how performance and generalization is expected to scale with these factors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}