<figure><img src="image_1.png" /></figure>

HAL
open science

# Goal oriented random forest (GORF)

Kevin Elie-Dit-Cosaque, Bérénice-Alexia Jocteur, Véronique

Maume-Deschamps, Clémentine Prieur, Pierre Ribereau, Ri Wang

<figure><img src="image_2.png" /></figure>

## ► To cite this version:

Kevin Elie-Dit-Cosaque, Bérénice-Alexia Jocteur, Véronique Maume-Deschamps, Clémentine Prieur, Pierre Ribereau, et al. Goal oriented random forest (GORF). Séminaire Phimeca 2023, Nov 2023, Paris, France. hal-04294065

HAL Id: hal-04294065

https://hal.science/hal-04294065v1

Submitted on 16 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

# Goal oriented random forest (GORF).

Véronique Maume-Deschamps
joint work with

Kevin Elie-Dit-Cosaque (SCOR SE), Bérénice-Alexia Jocteur
(Natixis, ICJ), Clémentine Prieur (LJK - UGA), Pierre Ribereau
(ICJ - UCBL), Ri Wang (ICJ).

17 novembre 2023

<figure><img src="image_10.png" /></figure>

Lyon 1

<figure><img src="image_11.png" /></figure>

Institut
Camille
Jordan
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

# Why considering GORF?

Random Forests are mainly designed for **Regression** or **Classification** purposes. In these cases, the target is observed.
Other aims may be pursued, e.g.:

* conditional quantiles

* conditional average treatment effects (CATE)

* other conditional risk measure such as expectiles

* ...

⇒ **Distributional Random Forest<fnref n="1" />** vs **Goal Oriented Random Forest.**

<footnote n="1">¹ Domagoj Ćevid et al. (2022). In: *Journal of Machine Learning Research*
Qiming Du et al. (2021). In: *International Conference on Artificial
Intelligence and Statistics*</footnote>
<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

Recall CART

Classification And Regression Tree<fnref n="2" />. Input variables:

X = (X₁, ..., X₉), Output variable: Y. Tree: constant piecewise
predictor, obtained by binary recursive partitioning.

Separate the data from the cur-
rent node, by looking for the split
reducing the most the heterogene-
ity of Y at the two child nodes.

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<footnote n="2">² Leo Breiman (2001). In: *Machine learning*</footnote>
<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

Loss function

In the **regression** context, $Y = m(\mathbf{X}) + \varepsilon$, the goal is to estimate
$\mathbb{E}(Y|\mathbf{X}) = m(\mathbf{X})$. Consider a random sample
$\mathcal{D}_n = (\mathbf{X}^i, Y^i), i = 1, \dots, n$, the heterogeneity of $Y$ is measured by
the intra-groups variance, so that we shall maximise:

$$
\mathcal{L}_{\mathrm{C}}^{n}(j, z) = \frac{1}{\#\mathrm{C}} \sum_{i=1}^{n} (Y^{i} - \bar{Y}_{\mathrm{C}})^{2} \mathbf{1}_{\{\mathbf{X}^{i} \in \mathrm{C}\}} - \\
\left[ \frac{1}{\#\mathrm{C}} \sum_{i=1}^{n} (Y^{i} - \bar{Y}_{\mathrm{C}_{\mathrm{L}}})^{2} \mathbf{1}_{\{\mathrm{C}_{\mathrm{L}}\}} + \frac{1}{\#\mathrm{C}} \sum_{i=1}^{n} (Y^{i} - \bar{Y}_{\mathrm{C}_{\mathrm{R}}})^{2} \mathbf{1}_{\{\mathrm{C}_{\mathrm{R}}\}} \right],
$$

where $C$ is the current cell, ie an hyper-rectangle $\prod_{j=1}^{d} [a_j, b_j]$, $\#C$ is
the number of elements of $\mathcal{D}_n$ for which $\mathbf{X}^i$, $i=1, \dots, n$ belongs
to $C$; $C_L = C \cap \{x_j \le z\}$; $C_R = C \cap \{x_j > z\}$.
<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

# Random Forests

Aggregate several CART's to reduce the estimation variance ⇒ **Bootstrap aggregating**

* Training sample: $D_n = (\mathbf{X}^i, \mathbf{Y}^i), i = 1, \dots, n$

* $\Theta_\ell, \ell = 1, \dots, k$ are independent random variables, following $\Theta = (\Theta^1, \Theta^2)'$ law: $\Theta^1$ provides the bootstrap indices on $D_n$ and $\Theta^2$ gives which *mtry* variables are considered for the splits of each node. $\Theta_\ell$ is assumed to be independent of $D_n$.

* $A_n(\mathbf{x}; \Theta_\ell, D_n)$: the leaf that is obtained when dropping $\mathbf{x}$ down the tree.

* $N_n(\mathbf{x}, \Theta_\ell, D_n)$: the number of points which are in $A_n(\mathbf{x}; \Theta_\ell, D_n)$.

* $N_n^b(\mathbf{x}, \Theta_\ell, D_n)$: the number of points of the bootstrapped sample, which are in $A_n(\mathbf{x}; \Theta_\ell, D_n)$.
<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

# CART estimation

In the regression framework, $\mathbb{E}(Y|\mathbf{X}) = m(\mathbf{X})$ is estimated on the random forest by. Let $B_j(\Theta_l, \mathcal{D}_n)$ be the number of times that the observation $(\mathbf{X}^j, Y^j)$ has been drawn from the original dataset for the $l$-th tree construction. Consider the weights:

$$ \omega_{n,i}(\mathbf{x}, \Theta) = \frac{1}{k} \sum_{j=1}^{k} \frac{\mathbf{1}_{\mathbf{x}^i \in A_n(\mathbf{x}, \Theta_j, \mathcal{D}_n)}}{N_n(\mathbf{x}, \Theta_j, \mathcal{D}_n)}, $$

$$ \omega_{n,i}^{b}(\mathbf{x}, \Theta) = \frac{1}{k} \sum_{\ell=1}^{k} \frac{B_{i}(\Theta_{\ell}, \mathcal{D}_{n}) \mathbf{1}_{\mathbf{x}^{i} \in A_{n}(\mathbf{x}; \Theta_{\ell}, \mathcal{D}_{n})}}{N_{n}^{b}(\mathbf{x}; \Theta_{\ell}, \mathcal{D}_{n})}, $$

and the corresponding estimations of $m(\mathbf{x})$:

$$ \hat{m}_{n}^{b}(\mathbf{x}) = \sum_{i=1}^{n} \omega_{n,i}^{b}(\mathbf{x}) Y^{i}. $$
<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

# Focus on causality and quantile estimation

* **Conditional quantile estimation** : $\alpha \in ]0, 1[$, the conditional quantile $q_{\alpha}(Y|\mathbf{X})$ could be estimated by inverting the estimated conditional distribution function: and the corresponding estimations of $F(y|\mathbf{X} = \mathbf{x})$ (introduced in Meinshausen 2006 and a.s. consistency proved in Elie-Dit-Cosaque and Maume-Deschamps 2022):

$$ \hat{F}_n^b(y|\mathbf{X} = \mathbf{x}) = \sum_{i=1}^{n} \omega_{n,i}^b(\mathbf{x}) \mathbf{1}_{\{Y_i \le y\}} $$

* **Conditional Average Treatment Effect**
<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

# Focus on causality and quantile estimation

* Conditional quantile estimation

* Conditional Average Treatment Effect:

$CATE(\mathbf{X}) = E(Y(1) - Y(0)|\mathbf{X})$ where $Y(W)$ is a target variable (eg some biological quantity) in presence ($Y(1)$) or absence ($Y(0)$) of a treatment $W$. Some causal models write: $Y = \tau(\mathbf{X})W + \gamma(\mathbf{X})$, in this case, $CATE(\mathbf{X}) = \tau(\mathbf{X})$. Modifications on the CART construction has to be done in order to estimate $CATE$, since for each individual, we observe either $Y^i(1)$ or $Y^i(0)$.
<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

Other forests

* Generalized **Random Forests**<fnref n="3" /> consider $\theta$ the target (eg conditional quantiles or **CATE** and use the following loss function:

$$ \frac{\#C_L\#C_R}{\#C^2} [\hat{\theta}_{C_L} - \hat{\theta}_{C_R}]^2 $$

where $\hat{\theta}_C$ is an estimator of the target $\theta$ on the cell C.

* Distributional **Random Forests**<fnref n="4" /> use Maximal Mean Discrepancy as loss function or the Wasserstein distance <fnref n="5" /> for the construction of the split criterion.

* **Trees designed for Extreme Value Analysis**<fnref n="6" />.

<footnote n="3">³Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>

<footnote n="4">⁴Domagoj Čevid et al. (2022). In: *Journal of Machine Learning Research*</footnote>

<footnote n="5">⁵Qiming Du et al. (2021). In: *International Conference on Artificial Intelligence and Statistics*</footnote>

<footnote n="6">⁶Sébastien Farkas et al. (2021). In: *arXiv preprint arXiv:2112.10409*</footnote>
<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

# Goal oriented loss function

We propose loss functions specifically designed for

* CATE estimation,

* Conditional quantile estimation,

with a common **proof scheme** for a.s. consistency.
<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

# HTERF

Heterogeneous Treatment Effect Random Forest<fnref n="7" />

Consider the loss function:

$$ \mathcal{L}_C(j, z) = \frac{\#C_L \#C_R}{\#C^2} \left( \left( \bar{Y}_{C_{L1}} - \bar{Y}_{C_{L0}} \right) - \left( \bar{Y}_{C_{R1}} - \bar{Y}_{C_{R0}} \right) \right)^2, \quad (1) $$

where $C_{L1} = \{\mathbf{X}^i \in C_L, W^i = 1\}$, $C_{L0} = \{\mathbf{X}^i \in C_L, W^i = 0\}$, $C_{R1} = \{\mathbf{X}^i \in C_R, W^i = 1\}$, $C_{R0} = \{\mathbf{X}^i \in C_R, W^i = 0\}$,

<footnote n="7">⁷ Bérénice-Alexia Jocteur, Véronique Maume-Deschamps, and Pierre Ribereau (2023). In: https://hal.science/hal-04112079</footnote>
<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

# HTERF estimation

For the estimation of CATE we use:

$$ \widehat{\mathrm{CATE}}(\mathbf{x}) = \sum_{i:W^i=1} \omega_{n,i}(\mathbf{x}, \Theta) Y^i - \sum_{i:W^i=0} \omega'_{n,i}(\mathbf{x}, \Theta) Y^i, \quad (2) $$

where $\omega$ (resp. $\omega'$) are the weights associated to observations such as $W^i = 1$ (resp. $W^i = 0$).
<figure><img src="image_98.png" /></figure>

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

# Pin-ball loss

Consider the pin-ball function<fnref n="8" />: $\psi_{\alpha}(y, \theta) = (y - \theta)(\alpha - \mathbf{1}_{\{y \le \theta\}})$

Recall that the $\alpha$-quantile is given by:

$q^{\alpha}(Y) = \arg \min_{\theta} \mathbb{E}[\psi_{\alpha}(Y, \theta)]$.

Consider the loss function<fnref n="9" />:

$$ \mathcal{L}_C(j, z) = \sum_{i=1}^{n} \psi_{\tau}(Y^i, \hat{\theta}_C) \mathbf{1}_{\mathbf{X}^i \in C} - \left[ \sum_{i=1}^{n} \psi_{\tau}(Y^i, \hat{\theta}_{C_L}) \mathbf{1}_{\mathbf{X}^i \in C_L} + \sum_{i=1}^{n} \psi_{\tau}(Y^i, \hat{\theta}_{C_R}) \mathbf{1}_{\mathbf{X}^i \in C_R} \right], $$

where $\hat{\theta}_C$ is an estimator of the $\alpha$-quantile in C.

Estimate the conditional distribution function then the quantile as before.

<footnote n="8">⁸It is also named check function, quantile loss.</footnote>

<footnote n="9">⁹Harish S Bhat, Nitesh Kumar, and Garnet J Vaz (2015). In: 2015 IEEE International Conference on Big Data (Big Data). IEEE</footnote>
<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

# Consistency of random forests

Results by Scornet, Biau, Vert (2015) in a linear model context:

$$Y = m(X) + \varepsilon \quad \text{with} \quad \varepsilon \rightsquigarrow \mathcal{N}(0, \sigma^2) \quad \text{and} \quad m(X) = \sum_{j=1}^{d} m_j(X_j).$$

Under various assumptions **including tree size wrt $n$** and a forest correlation control, for $\mathbf{X} \rightsquigarrow \mathcal{U}[0, 1]^d$,

$$\mathbb{E}[(m_n(\mathbf{X}) - m(\mathbf{X}))^2] \to 0, \text{ with } m_n = \mathbb{E}_{\Theta}(\hat{m}_n).$$

* The bootstrap is not taken into account in $\Theta$

* No results for $m(\mathbf{x})$

* Results for fully grown trees and for limited grown trees.
<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

Consistency of GRF

Asymptotic normal laws obtained<fnref n="10" /> under

* regularity assumptions of the target function,

* constraints on the tree construction

<footnote n="10">¹⁰Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>
<figure><img src="image_122.png" /></figure>

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /></figure>

# Consistency of GRF

Asymptotic normal laws obtained<fnref n="10" /> under

* regularity assumptions of the target function, which write in the CATE estimation case:

$x \mapsto \mathbb{E}[Y(u)|\mathbf{X} = \mathbf{x}]$ and $x \mapsto \mathbb{E}[Y(u)^2|\mathbf{X} = \mathbf{x}]$ are

Lipschitz-continuous, $\mathrm{Var}[Y(u)|\mathbf{X} = \mathbf{x}] > 0$ and

$\mathbb{E}[|Y(u) - \mathbb{E}[Y(u)|\mathbf{X} = \mathbf{x}]|^{2+\delta}|\mathbf{X} = \mathbf{x}] \le M$ for some constants $\delta, M > 0$ uniformly over all $\mathbf{x} \in [0, 1]^d$.

* constraints on the tree construction

<footnote n="10">¹⁰Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>
<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>

<figure><img src="image_133.png" /></figure>

# Consistency of GRF

Asymptotic normal laws obtained<fnref n="10" /> under

* regularity assumptions of the target function,

* constraints on the tree construction:

* at every step of the tree building procedure, the probability that the next split is done along the $j$-th feature is bounded below by $\pi/d$ for some $0 < \pi \le 1$ for all $j = 1, \dots, d$ random split hypothesis.

* for some fixed $\gamma$, each split leaves at least a fraction $\gamma$ of the available training sample on each side of the split, $\gamma$-regularity hypothesis

* for some fixed $p$, the leaf containing $x$ has at least $p$ observations for each treatment group and the leaf containing $x$ has either less than $2p-1$ observations with $W^i = 0$ or $2p-1$ observations with $W^i = 1$.

<footnote n="10">¹⁰Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>
<figure><img src="image_134.png" /></figure>

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

Consistency of GRF

Asymptotic normal laws obtained<fnref n="10" /> under

* regularity assumptions of the target function,

* constraints on the tree construction

The **bootstrap is not taken into account**, the proof is done for **honest forests** (ie independence of the sample used for the tree construction and the estimation).

<footnote n="10">¹⁰Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>
<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

# General scheme for the a.s. consistency

## Conditions

Relations between $k$ (number of trees) and $N_n^b(x; \Theta, D_n)$ (number of bootstrap observations in a leaf node):

* $k = O(n^\alpha)$, with $\alpha > 0$.

* $\forall x$, $N_n^b(x; \Theta, D_n) = \Omega(\sqrt{n}(\ln(n))^\beta)$, with $\beta > \frac{5}{2}$, a.s.ª or an assumption on mean and variance of $N_n^b(x; \Theta, D_n)$

The variation of the target function $\theta$ is small on the trees' leaves: for any $x$,

$$ \sup_{z, z' \in A_n(x, \Theta_j)} |\theta(z) - \theta(z')| \xrightarrow{n \to \infty} 0^a. $$

$$ {}^a f(n) = \Omega(g(n)) \iff \exists k > 0, \exists n_0 > 0 \mid \forall n \ge n_0 \ |f(n)| \ge k \cdot |g(n)| $$
<figure><img src="image_146.png" /></figure>

<figure><img src="image_147.png" /></figure>

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

<figure><img src="image_151.png" /></figure>

# Consistency: result

## Theorem

Assume the conditions above are verified then

$$|\hat{\theta}_n(\mathbf{x}) - \theta(\mathbf{x})| \underset{n \to \infty}{\xrightarrow{a.s.}} 0$$

Proof for conditional distributions<fnref n="11" /> and CATE<fnref n="12" /> but the proof scheme applies more generally.

<footnote n="11">¹¹ Kevin Elie-Dit-Cosaque and Véronique Maume-Deschamps (2022). In: *Electronic Journal of Statistics*</footnote>

<footnote n="12">¹² Bérénice-Alexia Jocteur, Véronique Maume-Deschamps, and Pierre Ribereau (2023). In: https://hal.science/hal-04112079</footnote>
<figure><img src="image_152.png" /></figure>

<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

<figure><img src="image_157.png" /></figure>

# Remark on the variation hypothesis

In order to get the variation hypothesis we need the continuity of $\theta$ and either

* the random split and $\gamma$-regularity hypothesis or,

* the convergence of the empirical loss function to the theoretical one (with some uniformity on the cells C) and the fact that the theoretical loss function is 0 on a cell C of a **theoretical tree**<fnref n="13" /> implies that $\theta$ is zero on C. This last condition is true e.g. in the CATE estimation setting for a large class of functions (including sums, products, dense classes).

<footnote n="13">¹³introduced in Scornet, Biau, and Vert 2015 and deeply used in Elie-Dit-Cosaque and Maume-Deschamps 2022</footnote>
<figure><img src="image_158.png" /></figure>

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>

<figure><img src="image_162.png" /></figure>

<figure><img src="image_163.png" /></figure>

## Theroetical trees

A **theoretical tree** is grown following the same rules as an empirical tree, except that the theoretical equivalent of the empirical split criterion on a node C is used to choose the best split. E.g. for the CART-tree, the theoretical split criterion is:

$$ \mathcal{L}_C^*(j, z) = \mathrm{Var}(Y | \mathbf{X} \in C) - \mathrm{P}(\mathbf{X} \in C_L | \mathbf{X} \in C) \mathrm{Var}(Y | \mathbf{X} \in C_L) - \mathrm{P}(\mathbf{X} \in C_R | \mathbf{X} \in C) \mathrm{Var}(Y | \mathbf{X} \in C_R). $$

Hence, a theoretical tree is obtained thanks to the best consecutive cuts $(j^*, z^*)$, among $j \in \mathcal{M}_{\text{try}}$, $z \in \mathcal{C}^j$ optimizing the previous criterion $\mathcal{L}_C^*(\cdot, \cdot)$.
<figure><img src="image_164.png" /></figure>

<figure><img src="image_165.png" /></figure>

<figure><img src="image_166.png" /></figure>

<figure><img src="image_167.png" /></figure>

<figure><img src="image_168.png" /></figure>

<figure><img src="image_169.png" /></figure>

# Empirical vs theoretical trees

Consider the model $Y = m(X) + \varepsilon$ with $m(\cdot)$ in the ♠-class<fnref n="14" /> and $\varepsilon$ with ligth tails.

## Proposition

For any $h \in \mathbb{N}$ fixed, for any empirical tree with node sizes greater than $C\sqrt{n}(\ln n)^\beta$, $\beta > \frac{5}{2}$, consider a node at height $h$ in the theoretical tree (resp. empirical tree) and $T_h$ the set of theoretical trees of height $h$, then

$$ \text{let } A = \prod_{j=1}^{d} [a_j, b_j] \text{ and } A^n = \prod_{j=1}^{d} [a_j^n, b_j^n]. \text{ We have:} $$

$$ \inf_{T_h} \max_{j=1,...,d} \max (|a_j - a_j^n|, |b_j - b_j^n|) \to 0 \text{ a.s. as } n \to \infty. $$

<footnote n="14">¹⁴Functions in this class satisfy that if $L_c^*(j, z) = 0$ for all $j, z$ the $m$ is constant on $C$.</footnote>
<figure><img src="image_170.png" /></figure>

<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>

<figure><img src="image_174.png" /></figure>

<figure><img src="image_175.png" /></figure>

# Empirical vs theoretical trees

## Proposition

For any $h \in \mathbb{N}$ fixed, for any empirical tree with node sizes greater than $C\sqrt{n}(\ln n)^\beta$, $\beta > \frac{5}{2}$, consider a node at height $h$ in the theoretical tree (resp. empirical tree) and $T_h$ the set of theoretical trees of height $h$, then

$$ \text{let } A = \prod_{j=1}^{d} [a_j, b_j] \text{ and } A^n = \prod_{j=1}^{d} [a_j^n, b_j^n]. \text{ We have:} $$

$$ \inf_{T_h} \max_{j=1,...,d} \max (|a_j - a_j^n|, |b_j - b_j^n|) \to 0 \text{ a.s. as } n \to \infty. $$

Proved for CART trees<fnref n="14" /> and for causal forests<fnref n="15" />.

<footnote n="14">¹⁴ Kevin Elie-Dit-Cosaque and Véronique Maume-Deschamps (2022). In: *Electronic Journal of Statistics*</footnote>

<footnote n="15">¹⁵ Bérénice-Alexia Jocteur, Véronique Maume-Deschamps, and Pierre Ribereau (2023). In: https://hal.science/hal-04112079</footnote>
<figure><img src="image_176.png" /></figure>

<figure><img src="image_177.png" /></figure>

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

<figure><img src="image_180.png" /></figure>

<figure><img src="image_181.png" /></figure>

## The two samples method.

One of the main idea to prove the consistency of random forests is to use an auxiliary sample: let ($\mathbf{X}^{i\diamond}, Y^{i\diamond}, i = 1, \dots, n$) be a second sample, independent from ($\mathbf{X}^i, Y^i, i = 1, \dots, n$) and consider the weights

$$ \omega_{n,i}^{\diamond}(\mathbf{x}, \Theta) = \frac{1}{k} \sum_{j=1}^{k} \frac{\mathbf{1}_{\mathbf{x}^{i\diamond} \in A_n(\mathbf{x}, \Theta_j, D_n)}}{N_n^{\diamond}(\mathbf{x}, \Theta_j, D_n)} $$

and the corresponding estimator $\theta_n^{\diamond}$ of $\theta$. We prove:

* $|\hat{\theta}_n(\mathbf{x}) - \theta_n^{\diamond}(\mathbf{x})| \xrightarrow{n \to \infty} 0$, uses a Hoeffding like inequality + Vapnik-Chervonenkis classes$^{16}$ (proximity of $N^{\diamond}$ and $N^b$),

* $|\theta_n^{\diamond}(\mathbf{x}) - \theta(\mathbf{x})| \xrightarrow{n \to \infty} 0$, uses Vapnik-Chervonenkis classes again and the variation hypothesis.

<footnote n="16">16 V. N. Vapnik and A. Ya. Chervonenkis (1971). In: *Theory of Probability and its Applications*</footnote>
<figure><img src="image_182.png" /></figure>

<figure><img src="image_183.png" /></figure>

<figure><img src="image_184.png" /></figure>

<figure><img src="image_185.png" /></figure>

<figure><img src="image_186.png" /></figure>

<figure><img src="image_187.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
CATE estimation: a first example

We consider simulated data close to causal frameworks previously studied <fnref n="17" />. $X \sim U([0, 1]^p)$, $W \sim Bern(0.5)$ and $Y = \tau(X)W + \beta\gamma(X)$, $p = 10$, $\tau(x) = \sin(x_1)$ and $\gamma(x) = \cos(2x_2 + 3x_3)$. The scalar $\beta$ allows to consider the impact of the magnitude of $\tau$ relative to $\gamma$.

<table><thead><tr><td>β</td><td>GRF</td><td>HTERF</td></tr></thead><tbody><tr><td>5</td><td>0.276</td><td>0.117</td></tr><tr><td>1</td><td>0.122</td><td>0.012</td></tr><tr><td>0.2</td><td>0.079</td><td>0.004</td></tr></tbody></table>

**Table:** Mean squared errors of GRF and HTERF methods that estimate heterogeneous treatment effect, with 500 tree forests.

<footnote n="17">¹⁷ Susan Athey, Julie Tibshirani, Stefan Wager, et al. (2019). In: *The Annals of Statistics*</footnote>
<figure><img src="image_188.png" /></figure>

<figure><img src="image_189.png" /></figure>

<figure><img src="image_190.png" /></figure>

<figure><img src="image_191.png" /></figure>

<figure><img src="image_192.png" /></figure>

<figure><img src="image_193.png" /></figure>

CATE estimation: interpretability

<table><thead><tr><th rowspan="2">β</th><th colspan="4">GRF</th><th colspan="4">HTERF</th></tr><tr><th>dep.3</th><th>dep.5</th><th>dep.10</th><th>imp.</th><th>dep.3</th><th>dep.5</th><th>dep.10</th><th>imp.</th></tr></thead><tbody><tr><td>5</td><td>0.870</td><td>0.378</td><td>0.150</td><td>0.852</td><td>1</td><td>0.498</td><td>0.175</td><td>0.985</td></tr><tr><td>1</td><td>0.874</td><td>0.526</td><td>0.174</td><td>0.866</td><td>1</td><td>0.995</td><td>0.282</td><td>1</td></tr><tr><td>0.2</td><td>0.875</td><td>0.627</td><td>0.2</td><td>0.866</td><td>1</td><td>1</td><td>0.603</td><td>1</td></tr></tbody></table>

Table: Frequencies of splitting on $X_1$ at depths 3, 5 and 10 and importance of $X_1$.
CATE estimation: a non linear framework

Let $\mathbf{X} \sim U([0, 1]^p)$, $W \sim \text{Bern}(0.5)$ and $Y = \sin(X_1)(W + 2)^3 + \cos(X_2)$, where $p = 3$. Hence we have CATE that satisfies: $\tau(\mathbf{x}) = 19 \sin(X_1)$.

<table><thead><tr><td>Method</td><td>RMSE</td><td>importance</td></tr></thead><tbody><tr><td>GRF</td><td>0.321</td><td>0.777</td></tr><tr><td>HTERF</td><td>0.209</td><td>1</td></tr></tbody></table>

**Table:** Root mean squared errors of GRF and HTERF methods that estimate heterogeneous treatment effect, with 500 tree forests. We also consider the importance of $X_1$.
<figure><img src="image_194.png" /></figure>

<figure><img src="image_195.png" /></figure>

<figure><img src="image_196.png" /></figure>

<figure><img src="image_197.png" /></figure>

<figure><img src="image_198.png" /></figure>

<figure><img src="image_199.png" /></figure>

Conditional quantiles estimation: two examples<fnref n="18" />

**Example 1:**

$Y | \mathbf{X} \sim \mathcal{N}(0, 1 + \mathbf{1}(X_1 > -0.5)), p = 10, X_j, j = 1, 2, \dots, 10$ are independent draws from uniform distribution $\mathcal{U}(-1, 1)$, $\mathbf{x}_i = (x_i, 0, 0, \dots, 0)$, and $x_i$ is taken from regular grid over $[-1, 1]$.

**Example 2:** $Y = X_1 - X_2 + \varepsilon, X_j, j = 1, 2, \dots, 10$ are independent draws from $X_j \sim \text{Exp}(1), \varepsilon \sim \mathcal{N}(0, 1)$, $\mathbf{x}_i = (x_{i1}, x_{i2}, 1, 1, \dots, 1)$, and $(x_{i1}, x_{i2})$ are taken from regular grid over $[0, 5]^2$.

<footnote n="18">¹⁸ Véronique Maume-Deschamps, Clémentine Prieur, and Ri Wang.</footnote>
<figure><img src="image_200.png" /></figure>

<figure><img src="image_201.png" /></figure>

<figure><img src="image_202.png" /></figure>

<figure><img src="image_203.png" /></figure>

<figure><img src="image_204.png" /></figure>

<figure><img src="image_205.png" /></figure>

Conditional quantiles estimation: Example 1

<table><caption>Table: MSE<fnref n="19" /> for Example 1: $Y|X \sim N(0, 1 + 1(X_1 > -0.5))$</caption><thead><tr><th>α</th><th>n</th><th>GRF</th><th>Quantile06</th><th>DRF</th><th>Pin-ball</th></tr></thead><tbody><tr><td rowspan="3">0.1</td><td>500</td><td>0.197</td><td>0.219</td><td>0.107</td><td><strong>0.085</strong></td></tr><tr><td>2000</td><td>0.038</td><td>0.140</td><td><strong>0.027</strong></td><td>0.028</td></tr><tr><td>4000</td><td>0.025</td><td>0.110</td><td>0.019</td><td><strong>0.016</strong></td></tr><tr><td rowspan="3">0.5</td><td>500</td><td><strong>0.015</strong></td><td>0.016</td><td>0.017</td><td>0.035</td></tr><tr><td>2000</td><td><strong>0.007</strong></td><td><strong>0.007</strong></td><td>0.009</td><td>0.010</td></tr><tr><td>4000</td><td><strong>0.006</strong></td><td><strong>0.006</strong></td><td>0.010</td><td>0.010</td></tr><tr><td rowspan="3">0.9</td><td>500</td><td>0.205</td><td>0.221</td><td>0.112</td><td><strong>0.083</strong></td></tr><tr><td>2000</td><td>0.037</td><td>0.137</td><td>0.025</td><td><strong>0.022</strong></td></tr><tr><td>4000</td><td>0.026</td><td>0.119</td><td>0.017</td><td><strong>0.014</strong></td></tr></tbody></table>

<footnote n="19">¹⁹100 repetitions, number of trees $B = 200$, min.node.size = 20, mtry = p. Other parameters use the default setting.</footnote>
<figure><img src="image_206.png" /></figure>

<figure><img src="image_207.png" /></figure>

<figure><img src="image_208.png" /></figure>

<figure><img src="image_209.png" /></figure>

<figure><img src="image_210.png" /></figure>

<figure><img src="image_211.png" /></figure>

Conditional quantiles estimation: Example 2

<table><thead><tr><th>τ</th><th>n</th><th>GRF</th><th>Quantile06</th><th>DRF</th><th>Pin-ball</th></tr></thead><tbody><tr><td rowspan="3">0.1</td><td>500</td><td>5.700</td><td>2.679</td><td>3.074</td><td><strong>1.629</strong></td></tr><tr><td>2000</td><td>4.754</td><td>1.770</td><td>1.949</td><td><strong>0.763</strong></td></tr><tr><td>4000</td><td>3.959</td><td>1.133</td><td>1.373</td><td><strong>0.474</strong></td></tr><tr><td rowspan="3">0.5</td><td>500</td><td>1.848</td><td>1.166</td><td>1.713</td><td><strong>0.817</strong></td></tr><tr><td>2000</td><td>0.761</td><td>0.405</td><td>0.644</td><td><strong>0.383</strong></td></tr><tr><td>4000</td><td>0.411</td><td>0.243</td><td>0.341</td><td><strong>0.235</strong></td></tr><tr><td rowspan="3">0.9</td><td>500</td><td>5.869</td><td>2.833</td><td>3.022</td><td><strong>1.812</strong></td></tr><tr><td>2000</td><td>4.842</td><td>1.717</td><td>1.809</td><td><strong>0.847</strong></td></tr><tr><td>4000</td><td>4.093</td><td>1.201</td><td>1.332</td><td><strong>0.497</strong></td></tr></tbody></table>

Table: MSE<fnref n="20" /> for Example 2: $Y = X₁ - X₂ + ε$

<footnote n="20">²⁰100 repetitions, number of trees $B = 200$, min.node.size = 20, mtry = p. Other parameters use the default setting.</footnote>
<figure><img src="image_212.png" /></figure>

<figure><img src="image_213.png" /></figure>

<figure><img src="image_214.png" /></figure>

<figure><img src="image_215.png" /></figure>

<figure><img src="image_216.png" /></figure>

<figure><img src="image_217.png" /></figure>

# Plan

1. Introduction

2. Alternative loss functions

3. On the consistency of RF

4. Simulation studies

5. Conclusion
<figure><img src="image_218.png" /></figure>

<figure><img src="image_219.png" /></figure>

<figure><img src="image_220.png" /></figure>

<figure><img src="image_221.png" /></figure>

<figure><img src="image_222.png" /></figure>

<figure><img src="image_223.png" /></figure>

Conclusion.

* Use Goal Oriented Random Forest for specific purposes:

* better estimation (in general)

* better interpretability.

* Results developed for *CATE* and conditional quantile estimations but other quantities are reachable.

* General scheme for a.s. consistency results.

* Theoretical random forest = a powerful tool to better understand the random forests methods.
References I

<figure><img src="image_224.png" /></figure>

Athey, Susan, Julie Tibshirani, Stefan Wager, et al. (2019). "Generalized random forests". In: *The Annals of Statistics* 47.2, pp. 1148-1178.

<figure><img src="image_225.png" /></figure>

Bhat, Harish S, Nitesh Kumar, and Garnet J Vaz (2015). "Towards scalable quantile regression trees". In: *2015 IEEE International Conference on Big Data (Big Data)*. IEEE, pp. 53-60.

<figure><img src="image_226.png" /></figure>

Breiman, Leo (2001). "Random forests". In: *Machine learning* 45.1, pp. 5-32.

<figure><img src="image_227.png" /></figure>

Cévid, Domagoj et al. (2022). "Distributional random forests: Heterogeneity adjustment and multivariate distributional regression". In: *Journal of Machine Learning Research* 23.333, pp. 1-79.

<figure><img src="image_228.png" /></figure>

Du, Qiming et al. (2021). "Wasserstein random forests and applications in heterogeneous treatment effects". In: *International Conference on Artificial Intelligence and Statistics*, pp. 1729-1737.

<figure><img src="image_229.png" /></figure>

Elie-Dit-Cosaque, Kevin and Véronique Maume-Deschamps (2022). "Random forest estimation of conditional distribution functions and conditional quantiles". In: *Electronic Journal of Statistics* 16, 6553-6583.

<figure><img src="image_230.png" /></figure>

Farkas, Sébastien et al. (2021). "Generalized Pareto Regression Trees for extreme events analysis". In: *arXiv preprint arXiv:2112.10409*.

<figure><img src="image_231.png" /></figure>

Jocteur, Bérénice-Alexia, Véronique Maume-Deschamps, and Pierre Ribreau (2023). "Heterogeneous Treatment Effect based Random Forest: HTERF". In: https://hal.science/hal-04112079.

<figure><img src="image_232.png" /></figure>

Maume-Deschamps, Véronique, Clémentine Prieur, and Ri Wang. "Work in progress".

<figure><img src="image_233.png" /></figure>

Meinshausen, Nicolai (2006). "Quantile regression forests". In: *Journal of Machine Learning Research* 7.Jun, pp. 983-999.
<figure><img src="image_234.png" /></figure>

Scornet, Erwan, Gérard Biau, and Jean-Philippe Vert (2015). "Consistency of random forests". In: *The Annals of Statistics*.

<figure><img src="image_235.png" /></figure>

Vapnik, V. N. and A. Ya. Chervonenkis (1971). "On the Uniform Convergence of Relative Frequencies of Events to their Probabilities". In: *Theory of Probability and its Applications* 16.2, pp. 264–280.
<figure><img src="image_236.png" /></figure>

<figure><img src="image_237.png" /></figure>

<figure><img src="image_238.png" /></figure>

<figure><img src="image_239.png" /></figure>

<figure><img src="image_240.png" /></figure>

<figure><img src="image_241.png" /></figure>

Thanks for your attention.
Merci pour votre attention.