Scaling Laws of Distributed Random Forests
Authors: Katharina Flügel
Charlotte Debus
Markus Götz
Achim Streit
Marie Weiel
Venue: Accepted by TMLR ()

Abstract: Random forests are a widely used machine learning technique valued for their robust predictive performance and conceptual simplicity. They are applied in many critical applications and often combined with federated learning to collaboratively build machine learning models across multiple distributed sites. The independent decision trees make random forests inherently parallelizable and well-suited for distributed and federated settings. Despite this perfect fit, there is a lack of comprehensive scalability studies, and many existing methods show limited parallel efficiency or are tested only at smaller scales. To address this gap, we present a comprehensive analysis of the scaling capabilities of distributed random forests on up to 64 compute nodes. Using a tree-parallel approach, we demonstrate a strong scaling speedup of up to 31.98 and a weak scaling efficiency of over 0.96 without affecting predictive performance of the global model. Comparing the performance trade-offs of distributed and local inference strategies enables us to simulate various real-life scenarios in terms of distributed computing resources, data availability, and privacy considerations. We further explore how increasing model and data size improves prediction accuracy, scaling up to 51 200 trees and 7.5 million training samples. We find that while distributing the data across nodes leads to super-scalar speedup, it negates the predictive benefit of increased data. Finally, we study the impact of distributed and non-IID data and find that while global imbalance reduces performance, local distribution differences can help mitigate this effect.

Published in Transactions on Machine Learning Research (11/2025)
Scaling Laws of Distributed Random Forests
Katharina Flügel
katharina, Baeeeialaaaeeeeaeaaaaaaaa edu
Karlsruhe Institute of Technology (KIT), Scientific Computing Center (SCC) Helmholts Al
Charlotte Debus
charlotte. debiteeeebbbsbiii edu
Karlsruhe Institate of Technology (KIT), Scientific Computing Center (SCC)
Markus Götz
markus. Sarkkus.goetzükhi.c edu
Karlsruhe Institute of Technology (KIT), Scientific Computing Center (SCC)
Helmholtz
Achim Streit
achim.streit edu
Karlsruhe Instätude of Technology (KIT), Scientific Computing Center (SCC)
Marie Weiel
marie, weiel edu
Karlsruhe Institute of Technology (KIT), Scientific Computing Center (SCC)
Helmholtz Al
Reviewed on OpenReview: https: https:://ppenreviauu openreview. net/ porm? id= ICHoTlgnSy
Abstract
Random forests widely used machine learning technique valued for their robust predic- tive performance and conceptual simplicity. They are applied in many critical applications and often combined with federated learning to collaboratively build machine lcarning mod- els across multiple distributed sites. The independent decision trees make random forests inherently parallelizable and well-suited for distributed and federated settings. Despite this perfect fit, there a lack of comprehensive scalability studies, and many existing meth- ods show limited parallel efficiency are tested only at smaller scales. To address this gap; present comprehensive analysis of the scaling capabilities of distributed random forests on up to 64 compute nodes. Using a troe-parallel approach, we demonstrate strong scaling speedup of up to 31.98 and a weak scaling elliciency of over 0.96 without affecting predictive performance of the global model. Comparing the performance trade-offs of dis- tributed and local inference stratogics enables us to simulate various real-life sccnarios in terms of distributed computing resources, data availability, and privacy considerations. We further explore how increasing model and data size improves prediction accuracy, scaling up to 51 200 trees and 7.5 million training samples. We find that while distributing the data across nodes leads to super-scalar speedup, it negates the predictive benefit of increased data. Finally, we study the impact of distributed and non-IID) data and find that while global imbalance reduces performance, local distribution differences can help mitigate this effect.
1 Introduction
Interpretability and transparency are crucial for machine-learning applications in fields like healthcare and critical infrastructure, where decisions must. be both data-driven and explainable. In such high-stakes do- mains, black- box models like deep neural networks often face skepticism due to their lack of transparency. As result, practitioners often prefer rely on more interpretable alternatives like docision trees. Random forests (Breiman 2001) are ensembles of independent decision trees. By aggregating the predictions of indi- vidual trees, they improve generalization, enhance robustness to noise, and mitigate overfitting. Combining

---

Published in Transactions on Machine Learning Research (11/2025)
conceptual simplicity with competitive performance, especially on high-dimensional tabular data, they have
been employed in various fields like healthcare (Khalilia et al. 2011 Nguyen et al. 2013 Abdoh et al.
2018
Kaur et al.
2019; Alam et al. 2019 Wang et al.
2020], environmental sciences (Zhang & Yuan, 2015
Benali
ct al.]
2019 Tyralis et al. 2019 Asgari et al.,
2022 He ct al. 2022), remote sensing (Belgiu & Drägut 2016
material sciences Lii el al. 2022), intrusion dotection (Masaral et al. 2016 Resende & Drummond
2018
Liu et al.
2021), and finance Madaan et al.
2021 Uddin et al. 2022].
In light of the rising awareness around data security and privacy, federated learning (MMMahan et al. 2017 Yang et al. 2019) has emerged promising approach for applications involving personal data distributed across multiple sites. In federated learning, multiple independent clients collaboratively train a machine learning model with limited data sharing between participants. Critical applications, such as hospitals and financial institutions, often combine federated learning with random forests (Liu et al. 2020; Hauschild et al. 2022 As ensembles of independent trees, random forests are inherently well-suited for parallelization and distribution. This enables the training of larger forests on more extensive datascts. However, many existing implementattioss suffer from suboptimal parallel efficiency and often lack a thorough investigation of scaling capabilities. Furthermore, the data used in federated learning is typically assumed to be independent and identically distributed (IID) over all participating clients. In practice, this assumption rarely holds, as the local data distributions can vary significantly across the clients. Understanding how non-IID data affects the performance of distributed machine learning models is crucial for enhancing their practical application in federated scenarios.
This paper aims to address these gaps with a comprehensive study of the scalability and data heterogeneity in distributed random forests. To this end, we utilize hybrid parallel implementation of distributed random forests, combining the message- messaggepassing interface MPI with shared-memory parallelism to enable efficient communication and scalability in serrrbutdd-emoor environments. In comprehensive scalability study, we thoroughly investigate the scaling capabilities of distributed random forests with respect to the number of samples, troes, and machines, as well as their impact on prediction accuracy. We demonstrate the strong and weak scalability of this approach, utilizing model parallelism by distributing subforests and optionally data parallelism by distributing the training data. We explore when distributing trees and samples is beneficial and how it impacts the final model's prediction accuracy. By comparing different inference flavors in terms of efficiency and memory consumption, we simulate various real-life scenarios in terms of distributed compute power, data availability, and privacy restrictions. Finally, we study the impact of data imbalance on federated learning scenarios by treating each machine as an individual client with its local data, To explore the impact of non-IID data on the predictive performance of federated random forests, we evaluate the effect of class imbalance in both global and local datasets. This provides deeper insights into how random forests behave in federated learning settings and their robustness to heterogeneous data distributions.
Our key contributions are:
An experimental study of the scaling laws of random forests with increasing model and data size.
An HPC-adapted, hybrid parallel implementation of distributed random forests.
A parallel scalability study on up to 64 compute nodes, achieving up to 31.98 strong scaling speedup and over 0.96 weak scaling efficiency without affecting predictive performance of the global model.
An evaluation of the effects of data distribution and non-IID class distributions on distributed random forests.
2
Related Work
Parallel Random Forcsts Multiple shared-memory parallel implementations of random forests in R have been suggested (Genuer et al. 2017 Wright & Ziegler 2017 Azizah et al., 2019), which parallelize over the trees on up to 15 cores, but lack a thorough evaluation of parallel scalability. Random forests have also been parallelized using accolerators such as GPUs and FPGAs. For instance, |Van Essen et al. (2012) parallelize the inference over the sample domain, while Senagi & Jouandeau (2022) parallelize the training over the trees, achieving average speedups of up to 3.57 with dynamic GPU parallelism.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Distributed Random Forests with MapReduce Many approaches apply MapReduce (Dean & Ghe- mawal, 2008) to distribute random forests across multiple worker nodes. They differ in the exact frameworks used, the computational partitioning, and the evaluation scale, often utilizing only a small number of parallel nodes. Panda ct al. (2009) parallelize best-split selection along tree nodes using MapReduce and report strong scaling results for training a single tree on up to 200 workers. [Basilico et al. (2011) propose a MapReduce- basod parallclization over the trees of the forest, assigning each worker node its own data subset. They evaluate their approach on 65 parallel worker nodes with four cores each, reaching speedups of up to ten, but do not report detailed parallel scaling results. Han et al. (213)) implement another parallelization along the tree dimension. They evaluate their approach on up to ten workers in a weak-scaling experiment by scaling up the number of trees with the workers, but do not report timing or speedup results. Masarat et al. (0016) and Liu et al. (2021) apply a similar approach to intrusion detection. [Zhang & Yuan 2015) use MapReduce to parallelize along the sample dimension, first computing local splitting criteria before aggregating them. They evaluate their approach on up to cight cores across four dual-core CPUs but achieve no significant parallel speedup. This is also confirmed by Asgari et al. (2022), who report speedups of at most 1.1 across four compute nodes using the same approach. |Wakayama et al. (2015) train random forests with large datasets on MapReduce clusters, where each node holds a local training dataset. To address overfitting due to limited data access per node, they construct "shared forests" on the manager node. These shared forests are adapted to the local training data on each cluster node using transfer learning, and then returned to the manager for further processing. Chen et al.] (2017) proposed a random forest algorithm that combines data and task parallelism using a task-directed acyclic graph. The data is partitioned vertically along the feature axis. During training, trees are constructed in parallel, with multiple feature variables in each tree caleulated concurrently for node splitting. They evaluate their approach on up to 100 parallel workers with two cores each and report speedups of up to 87.3 at 100 worker nodes.
Other Approaches to Distribute Random Forests Most closely related to our work are approaches using general distributed computing in high-performance computing (HPC) environments. Mitchell et al.] Parallel Random Forest Classifier for R, distributing bootstrap
(201)) introduce nanager-worker-base samples and trees among available parallel processes. Communication is facilitated through a custom re- duction function, akin to MPI. They report parallel efficiency over 0.5 on up to 128 cores for strong scaling and 512 cores for weak scaling with increasing forest size. [Wang et al. (2018) introduce DistForest, a paral- lel random forest for supercomputers using MPI. Their manager-worker- based approach distributes dataset features across the parallel workers, utilizing multiple manager nodes to facilitate parallel tree training. Dis- tForest is evaluated on up to 128 nodes; however, the parallel efficiency remains below 0.5. Cliff et al. (2019) implement iterative random forests, which add an iterative boosting process standard random forests, for HPC applications using C++ and MPI. Their parallclization approach aligns with the shared-global-modde variant in our paper, where subforests are distributed across compute nodes and then aggregated into a shared model. Similar to Mitchell et al. (201I), they focus on the application to biological datasets where the features often outnumber the samples. They evaluate an iterative random forest for gene expression on up to 1600 threads across ten nodes. However, their parallel evaluation is limited to a single strong-scalin scalling setup with 1000 trees, and parallel efficiency drops below 0.5 after five nodes. VazqueezNovoa et al. (2023) use the task-based programming model COMPs to apply task parallelism along both the tree and data dimensions. They evaluate strong and weak scaling with increasing number of trees on up to 16 compute nodes, each with 48 cores, reporting strong scaling speedups of up to six; weak scaling efficiency at 16 nodes is approximately 0.65.
Federated Learning Distributed random forests are closely related to federated learning (McMahan al, 2017 Yang et al. ).19), where multiple participants collaboratively train model without sharing their data, often motivated by concerns over data privacy and security. Federated learning can be categorized into three types: horizontal federated lcarning, where all participants share the same features but different samples; vertical federated learning, where the participants hold different features of the same samples; and federated transfer learning, where there is no overlap between either features or samples. With the independence of individual trees, random forests lend themselves to foderated lcarning, and multiple approaches have been introduced for both horizontal and vertical federation. Most random-forest based approaches for horizontally partitioned data first train independent subforests on each participant's 's local data and then aggregate them
3

---

Published in Transactions on Machine Learning Research (11/2025)
into a global ensemble. Different aggregation approaches have been introduced, such weighting the trees by their performance (Tsou et al., 2018 Geneturk et al. 2022) and selecting only a subset of all trees (Markovic
et al. 2022 Hauschild et al. 2022 Most approaches for tree-based vertical federated learning focus on
growing decision trees collectively, node by node, to find the best split across all features. These approaches
can be adapted to random forests by applying the same approach to multiple trees. Liu et al. (2020) introduce Foderated Forest for vertical federated learning. In cach node, clients provide potential splits and corresponding scores, while a central server determines the best split based on these scores. [W et al. 2220)) have clients collectively determine the best split using multi-party computation and homomorphic encryption. FedTTree (I.i et al., 2023) adopts similar approach to Federated Forest (Liu et al. 2020) for training gradient-boosted decision trees in both vertical and horizontal federated learning settings. Clients send local histograms to a central server, which aggregales these histograms using summation (for horizontal federation) or concatenation (for vertical federation) and decides on the split in the current node.
Non-IID Data Many distributed random forest approaches assume the data to be independent and iden- tically distributed (TD) across all partitions. However, in practical federated applications, the local data subsets are often non-llI, which can pose an additional challenge (McMahan et al. 2017 Lu et al. 2024). There are multiple ways in which the local subsets can differ, ranging from differences in sheer size to dif- ferences in feature and class ristributions, both individually and jointly. Multiple categorization ions have been suggested Criado et al.) 2022 Ma et al. 2022 Milashouski et al. 2024). In this paper, we focus on a common type of non-III data: varying class (im)balances, where the share of different classes varies between the sub- sets. Multiple studies examine the effects of both generally imbalanced and non-IID data on the predictive quality of random forests. However, most approaches focus on the special case of binary classification and consider either an imbalance in the global dataset or between the local subsets, but not their interaction. For example, Hauschild et al. (2022) investigate how the number of data splits, imbalances in local data sizes, and global class imbalances affect the performance of random forests for binary classification in horizontal fedorated lcarning. |Atonio Eng Lim & Hee Park (2024) build a foderated random forest by growing cach tree collaboratively across clients. They evaluate the impact of varying class imbalance between the local subsets for multi-class classification while keeping the global class balance fixed. [hhen et al. (2022) study the effects of class imbalance for image classification with neural networks, suggesting an optimization- based meea-algorithm to improve performance in the minority classes through iterative commuication rounds. The evaluate their approach combining both global imbalance using up to three minority classes with local imbalance following the approach by Hsu et al. (0199)
Scaling Laws Neural scaling laws (Kaplan et al. 2020 study how the performance of machine learning models improves as a function of compute, data, and model size. Kaplan et al. (2020) demonstrate empirically that the performance of language models scales as a power law with the number of parameters, dataset
size, and amount of compute. Multiple follow-up studies (Sharma & Kaplan 2022 Michaud et al. 2023
Bahri et al. 2024) examining the theory behind it. While initially focused on transformer- language models, they have since been extended to other architectures and applications like image generation and classification (Henighan et al. 2020 Alabdulmohsin et al. 2022), reinforcement learning (Gao et al. 2023
and time-series forecasting (Shi et al. 2024). For random forests, Biau (2012) gives a theoretical analysis of how their performance scales with the number of features and shows that the convergence is independent of the number of noise features, and multiple studies (Oshiro et al.] 2012] Probst & Boulesteix, 2018)) analyze how the predictive performance scales with the number of trees. We extend this analysis with an empirical
evaluation of the scaling with both the number of trees and samples.
3 Background: Random Forests
A random forest [Breiman 2001 F = (Ti...., Ti} is an ensemble of t decision troes T. Random forests can be applied to both classífication and regression lasks, yielding robust, and accurate predictions for many applications. The fundamental building blocks of random forests are the decision trees T,, which recursively partition the feature space. Multiplc algorithms exist for constructing decision trees, which can be combined with random forests. These include CART (Breiman 1984), ID3 [Quinlan| 1986). AID20 (Morgan et al., 1963), and THAID (Morgan & Messenger 1973). To train a random forest on dataset. D consisting of
4

---

Published in Transactions on Machine Learning Research (11/2025)
(b) Scaling trees z and samples
(c) Scaling trees r and samples
(a) Scaling the number trees
Bq 1.17 10
Rj
1.17 10
Text Accuracy
100%
Hights betoer
Text Accuracy
100%
1 2=2-nin=2on A-m
Test Accuracy
100%
75%
75% -
75%
50%
50%
50%
0.75 x 10 =5
25%
25%
25%
110000 11 0.35 x 10", -
-1. 8
0%
0%
0%
500
10000
1500
2000
20
20
60
Scaling factor 2.
Scaling factor A
Scaling factor
Figure 1: What is the impact of the model size (number of trees t t= --) and training data (samples n) on the predictive performance of random forests? (a) When increasing the model size while keeping the data size n constant, we observe a consistent improvement from adding more trees that slowly saturates for both baselines l = 25 with n 0.75 x 10 and 4 = 7 with n 0.75 10". (b, e) The accuracy benefits further from simultaneously scaling the dataset to À ni samples compared to increasing only the model size and keeping the data static at n n1. More details on the experimental sctup are given in Section 5
n samples with m features, each tree is trained individually by growing from its root node. At each node, only a random subset of u features is considered for splitting ("feature bagging"), often set to u = vm. This helps reduce correlation among trees and improve gencralization. Among the selected features, the best split is chosen to divide the samples at a node into subsets while maximizing the purity of the subsets. Multiple optimization metrics exist to measure this impurity, such as the Gini index, used in this paper, and information gain. This splitting is repeated in the child nodes until a stopping criterion is reached, for example, & certain purity in the leaf nodes. The leaf node is assigned a prediction based on voting or averaging over the remaining training samples within its region. Bootstrapping is used to create multiple training datasets by randomly sampling a bootstrap D, with replacement from the original dataset D for each tree T,, typically [Di|= = n. This process is known as bagging (bootstrap aggregating) and introduces diversity among the trees in the forest. Overall, the complexity of growing a single tree up to depth O (log considering n samples and u features at each split, can be summarized trained independently, resulting in O (t n l) to train the entire forest. During inference, sample is processed by all decision trees. Each tree makes a prediction by passing the sample from the root to the corresponding leaf node. The random forest aggregates the individual predictions into the ensemble result.
O (u n nlog n). The t trees are
4 Distributed Random Forests
The predictive performance of random forests improves with both the size of the random forest and the amount of data it is trained on, as demonstrated in Figure However, with growing datasets and models, the computational and memory demands can surpass the capacity of a single machine. This highlights the need for distributing random forests across multiple machines. Yet, there is a distinct lack of comprehensive studies the scalability of distributed random forests. Since the individual decision troes of the random forest are independent, they are ideal target for parallelization. We therefore partition the t trees and distribute them equally among the processing units. During training, each node holds a local subforest Frecal of t/p trees and trains them using either shared global data or node-local data D. Training the subforests does not require any communication between compute nodes. Figure aa) illustrates this parallel training process. As most modern compute nodes follow a multi-core architecture, is often beneficial to further parallelize over the t/p local troes within cach nodc's subforests. We thus employ a hybrid parallclizatio scheme, using distributed computing between the p compute nodes and shared- memory parallelization within each node.
For global inference, a sample needs to be processed by all trees; however, during training, the global ensem- ble is distributed across compute nodes. We consider Iwo different approaches for inference on a distributed random forest. First, one can aggregate a global model by collecting the local subforests Flocal and con- structing the global forest Fglobal = =U (f). Afterwards, the global model can be used independently of other nodes to conduct inference for arbitrary, potentially local samples (see Figure bb). This can be ad- vantageous when local test data cannot be shared among nodes, such as in federated applications. However,
5

---

Published in Transactions on Machine Learning Research (11/2025)
Grapex Nodo 3
Infermncc
Inference Yoring
AlB-Guther
Independent Inference
All--oddac
Jil
(1)
Fi.m(1)
Jul
Fiwal?)
Floelf)
Phocle)
Figure Example of parallel random forest using compute nodes 1.2....P each node training three trees in their local subforests (i), for global forest Fglabal of nine trees. (a) Parallel Training: Node- local subforests are trained independently of other nodes, either shared global data or local slices D. (b) Inference with Global Model: Aggregate local subforests to global model after training, allowing local inference independent other nodes. (c) Inference with Global Voting: Share evaluation data Deval: perform distributed inference on local subforcsts, and aggrogate local results to global prodiction.
aggregating a global model requires a significant one-time communcatiion overhead for gathering all local subforests and does benefit further from distribution in the inference phase. Alternatively, the compute nodes continue collaboration during the inference phase via global voting, illustrated in Figure (c). Each compute node continues to hold only its local subforest Flocal. Test samples processed in parallel on all local subforests. To aggregate the results to the global prediction, each subforest computes a histogram of its votes. These histograms are summed using collective reduction, after which the global prediction can be determined from its maximum. With this approach, the predictions are identical to those obtained by an aggregated global model. AL the same time, continues to benefit from distributed computing the infer- ence phase, which both speeds up the inference and reduces the memory requirement on each node. This, however, requires the test data to be shared among all nodes, which may violate data privacy requirements in some applications. The pseudocodes for distributed training and inference given in Appendix A.1
5 Empirical Results
5.1
Datasets
We evaluate the parallel scalability of distributed random forests in terms of both strong and weak scaling and their application to federated learning. Details the implementation and computational environment are given in Appendix A.2 We use synthetic data generated with scikit-learn' 's make faake_classi- cntassiiiiiionn as this allows us to scale both the number of samples n and features m freely and adjust the class balance. We keep the fraction of informative and redundant features at 10 % each, with 0% repeated features, and all remaining features filled with noise. The number of classes is set to ten, with one cluster per class. Except for the experiments with imbalanced data Section 5.5 we use balanced classes. For most experiments, we use the largest dataset we could fit together with the local model onto standard compute node, resulting in a total of 10"h values. We use two datasets with different feature-to-sample ratios, resulting in two distinct difficulties. The 1M dataset consists of Rootll 1 x 10t samples and 1 x 10a features, while the 10M x 10? samples and m = 1 x 103 features. The 1M datasct is more challenging to
dataset consists of Notal solve, as it has an order of magnitude more features to learn from fewer samples. When comparing the two inference variants, we need to both the data and the global model into the memory of individual compute nodes. We thus use two additional, smaller datasets with only 1 10 overall values, as the 10 value datascts exceed our memory capacity. The 100K dataset has Neotal 1 x 105 samples and m 1 features, while the 1M-b dataset has Rootal 106 samples and 1 102 features. Additionally, we

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Test Accuracy 1M- Datasel
(b) Test Accuracy 10M- -atasst
(c) Speedup
64
Tout Accuracy
100%
.... Global Model
Local Medels
Hights betts
Tea Accuracy
32
Linear
75%
75%
Speodup
16
10M
Hight
Hightr
50%
50%
25%
25%
Global Medel
2-
Local Models
0%
0%
16
32
64
16 Number of Nodes
4
64
16
32
64
Number of Nodes
Number Nodes
Figure 3: Strong scaling experiments: two fixed size problems training t = 1600 trees on the 1M dataset and t == 448 trees on the 10M dataset while increasing the number of compute nodes p. Each node trains a subforest of size t/p. The global accuracy not affected by parallclization and remains constant. The speedup initially increases linearly but saturates as the work assigned to each node becomes too small for effective parallelization.
extend the strong and weak scaling experiments to the HIGGS dataset (Baldi et al. 2014 W Whiteson 2014).
a binary classification task distinguish between signal and background events in particle collision data. The dataset is nearly balanced, containing about 53 % positive (signal) samples. With Reotal 1.1 10 samples and m 28 features, it is similar in size to the 10M synthetic dataset. For all datasets, 75 % of the samples 0.75 Rrotal used training set, while the remaining 25 % are used as test set. During bootstrapping, cach tree draws a random set of n samples with replacement.
5.2 Scalability of Distributed Training
To evaluate the parallel scalability of distributed random forests, we use two common types of scaling experiments: strong and weak scaling, In strong scaling, the goal is to solve a fixed problem faster by increasing the computing resources. In our case, we train a random forest using a fixed number of trees on the same dataset while increasing the number of compute nodes. We run two series of experiments: training = 1600 trees on the 1M dataset and t = 448 trees on the 10M dataset on E (1, 4, 8, 16, 32, 64) compute nodes. Each node trains local forest of t/p trees. The number of trees was chosen as the maximum multiple of 64 we could train within 100 min on single node. For the serial baseline with p= we train scikit-learn
asssifiissiirrr on all t trees without MPI but with the shared-memory parallelization. Note that since the full-sized global forest does not fit onto a standard compute node, the serial runs were conducted on megmomemory nodes with twice the main memory but otherwise identical hardware. Figure BigureBjillstraae illustraaee the results in terms of test accuracy and speedup; detailed results of training time and speedup are given in Table D All accuracics are given as the mean over three independent runs with separate random seeds passed to the model. As expected, the accuracy of the global model remains unchanged when increasing p, as this does not affect the global number of trees t. This confirms that our parallelization docs not impact the random forest's 's predictive performance. Comparing the two datascts, we observe that the model performs significantly better on the 10M dataset than on the 1M dataset, even with considerably fewer trees in the forest. This is expected, as the ratio of samples to features is 100 times higher for 10M than for 1M, making it harder to learn. In contrast to the global model, the mean accuracy of the local models decreases with as their size t/p decreases. Finally, we evaluate the parallel scalability using the speedup Sp)) = /eeg/ Tpp), where Tseq is the training time of the sequential baseline and T(p) is the parallel training time with p compute nodes. Ideally, the speodup would scale lincarly with the number of compute nodes p. Our results show linear speedup initially, but this saturates as the number of nodes increases. This effect is more pronounced for the 10M dataset compared to the 1M dataset. This is expected as the work assigned to each node reduces with increasing p. At p 64, the subforests on each node contain only 25 (1M) and 7 (10M) trees, limiting the gain from further parallelization.
In weak scaling, we aim to solve a larger problem by increasing the computing resources while the problem size per node remains constant. In our case, we scale up the number of trees as we increase the number of compute nodes p. Specifically, we train a global forest with t = t1 p trees, where & the bascline number of trees, and each node trains local subforest with 4 trees. As for strong scaling, we train
7

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Test Accuracy 1M- Datasel
(b) Test Accuracy 10M- Dataset
(c) Efficiency
.....
Global Model
Tou Accuracy
15%
Local Models
Hightr ia bette
Teut Accuracy
Efficiency
Hight
90%
0.9
50%
Ideal
25%
80%
Medel Local Models
0.8
1004
0%
16
16 Number Nodes
64
64
Number of Nodes
Number Nodes
11 .p p in the random forest while increasing the number compute nodes p for two baseline problems, training t = 800 trees on the 1M dataset and 224 trees on the 10M datasct. Each node trains a subforest of size f1. The global accuracy increases with p as the number of trees in the global forest increases. The parallel efficiency remains over 0.96 for both datasets and all tested scales p.
Figure 4: Weak scaling experiments: increasing the global number trees =
Table 1: The global training time T(p), strong scaling speedup S(p), and weak scaling efficiency E(p) for I,... 64 nodes with the corresponding number of cores, trees and trees per node t/p.
Strong Scaling
Weak Scaling
Nodes p Cores
Trees t
t/p
T(p)
S(p) +
Trees t
t/p
T(p) +
E(p) t
76
1600
1600
5923.104s
1.000
800
800
2870.629
1.000
152
1600
800
2911.042s
2.035
1600
800
2930.780
0.979
304
1600
400
1528.165s
3.876
3200
800
2907.165s
0.987
1M
608
1600
200
784.943
7.546
6400
800
2941.895
0.976
16
1216
1600
100
4
12.969
12800
800
2915.327s
0.985
32
2432
1600
50
2
24.875
25600
800
2936.716s
0.977
64
4864
1600
25
185.237 s
31.976
51200
800
2930.885
0.979
76
448
448
8839.3934
1.000
224
224
2881.014s
1.000
152
448
224
2938.5228
1.987
448
224
2964.515s
0.972
4
304
448
112
1681.794g
3.472
896
224
2961.187
0.973
10M
8
608
448
56
874.932
6.674
1792
224
2987.3883
0.964
16
1216
448
28
662.736s
8.811
3584
224
2990.356
0.963
32
2432
448
14
617.535
9.456
7168
224
2992.089s
0.963
64
4864
448
7
606.976s
9.620
14336
224
2980.575
0.967
forest with p 21, 4, 16, 16, 64) nodes on the 1M and 10M datasets. We use i 800 for 1M and ti 224 for 10M. That is, the global forests trained by strong scaling correspond those trained at p in the weak scaling experiments. Figure 4 and Table give accuracy of the local forests remains constant while the global forest improves of compute nodes p and, in turn, the global model size = t p. The improvement in global accuracy is significantly higher for the 1M compared to the 10M dataset, as there is much more room for improvement from the t baseline. Since the problem size increases with p. we measure the parallel performance in terms of weak scaling efficiency F(p) /Tee//(()) instead of speedup. Efficiency typically ranges from zero to one, with higher values indicating better performance and one being the expected idcal. Our implementation demonstrates good scalability with elficiencies over 0.96 across all scales. In contrast strong scaling, we observe saturation
results. In contrast to strong scaling, the
increase the number
increase p and scale successfully up to 64 compute nodes (the maximum tested). Since we scale the problem size in tandem with the available resources, we avoid the issue of having too little work left on each node.
Figure gives the strong and weak scaling performance on the HIGGS dataset. We train a global forest of 640 trees for strong scaling and scale up from , trees in weak scaling, cach with up to nodes. The scaling behavior for both predictive and parallel performance matches our results on synthetic data. The accuracy is not affected by the parallelization and depends only the model size. In strong scaling,
8

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Strong Scaling Accuracy
(b) Weak Scaling Accuracy
(c) Strong Scaling Speedup
(d) Weak Scaling Efficiency
100%
... Ciiolal Model
Local Models
Hightr la Tone
100%
... Glotal Model
Local Models
Hligher is NOH
32-
Linzer HIGGS
1.00
Text Accuracy
90%
Test Accuracy
90%
Spoodup
16
Hligher
Emiciencc
0:75
Hiihht
80%
80%
0.50
70%-
70%.
0.25
Ideal
HIGGS
60%
60%
0.00
2
16 64
2
16 32 Number of Nodes
64
1
2
4
16 32
64
1
2
4
16
32
64
Number of Nodes
Number of Nodes
Number of Nodes
Figure 5: Strong scaling with 640 global trees and weak scaling with f1 10 trees per node on the HIGGS dataset. The results mirror the behavior on synthetic data: The accuracy is not affected by parallelization and depends only on model size. The strong scaling speedup initially increases linearly but saturates as the work per node decreases, while weak scaling efficiency remains high throughout all scales.
the accuracy of the global model remains constant while that of the local models decreases slightly as they become smaller. In weak scaling, the local models have constant size and accuracy while the accuracy of the global model improves with more nodes, and thus trees. As with the synthotic data, the strong scaling speedup increases linearly at first but saturates after about 16 nodes. Similarly, the weak scaling efficiency remains close to one at all tested scales.
5.3 Scalability of Distributed Inference
We compare the two approaches to inference introduced in Section in terms of their time and memory consumption. With global voting, each node i holds only the local subforest. (i) it trained. For inference on the global model, each test sample must be shared across all nodes, which perform independent inference on local subforests. The resulting local predictions are then aggregated via voting to obtain the global result. In contrast, building a shared global model would aggregate the local subforests Flocal once after training and combine them into a global model Fglobal = - (1) on each node. For inference, each node can use its own global model independent of the other nodes, meaning test data does not need to be shared among nodes. This approach is also motivated by federated learning, where clients may not want to send real test data to others due to privacy concerns. Sharing the model allows each individual node to act independently of the others during the inference phase. In contrast, global voting would be more suited to a single stakeholder distributing the random forest for computational performance, as in an HPC enviromment. Note that they do not affect the predictive performance of the global model.
To fit both the data and the global model into the memory of individual compute nodes, we use the smaller datasets 100K and 1M-b and I, = 76 local trees. We use a weak scaling selup, meaning the global model and thus the communication volume increase with p. The complexity of all-gathering a message of size m across p nodes is O(a log p + 8pm), where a is the message startup latency and 33 is the communication cost per word (Sanders et al. 2019 Chapter 13). To all-gather the global model, each node sends the t1 trees in its local subforest. The resulting message size m is about 461.97 MiB for 100K and 2.11 GiB for 1M-b. All-reducing the histograms for global voting has complexity (( log p + Bm). The size of the histograms m depends on the number of classes and test samples and is about 90 B per sample for both datasets.
Figure G illustrates our results; detailed values are provided in Table First, we consider the overhead of all-gathering the global model after training. Figure G(a) gives the time to train and aggregate the t --trre forest on p nodes, normalized with the single-node training time Tanq of p-tree forest. As expected, the time to train the local subforest Tirain remains constant, independent of the chosen inference variant. Compared to the sequential time, the training time thus decreases with increasing p. Aggregating the global model adds an overhead Teather to the training. This overhead grows with increasing p and thus forest size, while remaining constant with respect to the sequential training time. At = 64 nodes and t = 4864 trees, all- aalggathering the model takes between 3.7 (1M-b) and 5.1 (100K) times as long as the parallel training itself. Despite this overhead, both variants are significantly faster than sequential training. Figure (b) considers the inference time Trst the global test set, again normalized with the sequential time Tseq on
9

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Training Time
(b) Inference Time
(c) Memooy Consumption
TxIp) (Tee
1.0
All Gather Tanee
Training Fashs
Lover benes
Teee (m) Tou
1.0-
Menony Node (GBB)
150
Lower benee
Dutaset
[00K Giiebll Viring
Appreach
100
Glibal Model
0.5
0.5
IM b
Global Vating
50-
IM-b Global Model
0.0
2.0 0.0
2
4
8
16
32
64
2
4
8
16
32
64
8
16
32
64
Number of Nodes
Number of Nodes
Number of Nodes
Figure 6: Comparison of two inference variants: aggregating a shared global model on each node after training or performing distributed inference via global voting for two bascline problems, 100K and 1M-b with 4 = 76 trees per node in a weak-scaling setup. Building a shared global model comes at both a one-time overhead to aggregate the local models after training and a consistent overhead during the serial inference phase for both inference time and memory consumption. In contrast, global voting requires less time and memory per node but utilizes all p nodes during inference. Training and inference times are normalized with the corresponding single-node time seq on the same-size forest with 4 p trees.
the corresponding model size. With shared global model, a node needs to process the entire forest and is thus on par with sequential inference (Tuet (T)e RJ 1). This is a significant overhead compared to global voting, which distributes the inference across all p nodes, each processing only their local subforest and thus reducing the inference time with increasing p. Finally, Figure G(c) illustrates the memory consumption per node. With global voting, the memory consumption remains constant, while using a global model increases it with the size of the aggregated global model, growing more than 150 GiB. This memory consumption drastically limits the base problem size we could use in this experiment compared to using global voting, Overall, the aggregation of a shared global model comes at both the one-time cost of the all-gathering overhead and the recurring cost of holding and operating on the entire global model. In contrast, global voting reduces the overall wall-clock time and consumes less memory per node. However, it requires using all p nodes and the parallel efficiency can drop below 0.5, especially for small datasets and forests. One thus needs to carcfully consider the trade-off between wall-clock time and efficient use of computing resources for the given use case. In some cases, the inference approach can also be prescribed by the application. For example, in federated learning settings, it is often not possible to share local test data with other participants, requiring the entire global model to be held locally. In other cases, the global model may be too big to fit onto a single node, requiring distributed inference.
5.4 Training with Distributed Data
In Sections 5.2 and 5.3 we demonstrate the parallel scalability of distributed random forests using shared training data. However, distributed machines may not always have access to the same data. For example, federated learning scenarios, participants might not wish to share their training data, or data is gathered in different locations with limited communication capacities. We therefore investigate the impact of partitioning the training data across nodes on both the predictive and the computational performance. This approach is related to data parallel training employed for neural networks, where the data is partitioned across machines that collaboratively train a global model (Ben-Nun & Hoefler 2019). We repeat the strong scaling experiment from Section 5.2 but in addition to the trees, we also partition the training data and distribute it across nodes. Thus, each node trains t/p trees on n/p samples compared to the global trees and n samples, using the 1M dataset with 1600 global trees and the 10M dataset with t 448 global trees. Figure 7 and Table 5 present the predictive and computational performance for p E {1. 4, 8, 16, 32, 64) compute nodes. For comparison, the corresponding values from the strong scaling experiment (Figure 3) are included with low transparcncy. For both datasets, the accuracy of the trained forest decrcases significantly as the number of nodes increases and the local datasets reduce in size. Compared to strong scaling without data distribution, the accuracy of the local subforests decreases more strongly, and even the global accuracy decreases. With distributed datasets, the bagging of samples on each node is restricted to much smaller base datasot. This decrease in accuracy more pronounced for the 1M dataset. With n 0.75 10 samples, only n/p- 1.17 x 10a samples per node remain p 64 nodes or about 1.17 samples per feature. Due to
10

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Test Accuracy 1M- Datasel
(b) Test Accuracy 10M -ataset
(c) Speedup
Teut Accuracy
100%
Global Model
Local Models
No DD
Hightr betts
Teu Accuracy
1024
-... 1M
1OM
No DDD
75%
75%
Speedup
Linear
Hight
64-
Higher
50%
50%
25%
25%
Global Model
No
Local Models
No DD
0%
16
32
16 Number Nodes
4
64
16
64
Number of Nodes
Number Nodes
Figure 7: Results for training with distributed data, where the training data is partitioned and distributed over the nodes together with the trees. We solve two fixed-size problems, training t 1600 trees on the 1M datasct and f 448 trees on the 10M dataset while incrcasing the number of compute nodes p. Each node trains a subforest of size l/p on n/p samples. The corresponding strong scaling results from Figure 3 i.e., without data distribution (No DD), are superimposed with low transparency. With data partitioning, both local and global accuracy degrade with increasing p; this effect is stronger for 1M with fewer samples overall. However, we observe super-lincar speedup of the overall work decreases with more compute resources.
50%
Piea
Reas Hpstica
Global
40%-
Class Frequency
Local
30%
20%
Res = Rurik
Biu 2, Aprice =2
Global
10%
Local
0
2
4
6
8
0
3
5
8
2
4
6
7
9
Class
Class
Class
Figure 8: (a) The class frequencics for ten classes with μ = foo, 10, 5, 2, 0.5} and a peak at class five. (b) The class frequencies of the global dataset and the different local subsets on = 4 nodes for different combinations of data and partition imbalance. The global data balanced with Jddatn = oo in the top row and imbalanced in the bottom row with Mdata 2. The partition into local subsets is balanced in the left column (ppartition oo) and imbalanced in the right column (Apartition = 2).
its higher ratio of samples to features, the 10M datasel retains about 117.18 samples per feature per node at 64. While the predictive performance decreases, we observe a super-linear speedup in training time. This is expected as the total workload decreases from O (t u in moggn) n) to O (t u n/plog(n/p) as cach local forest is trained on only n/p instead of n samples.
5.5 Breaking the IID Assumption
So far, have assumed the training data to be independent and identically distributed (IID). However, in many federated settings, this assumption longer holds, for example, due to different data collection methods or class prevalence. In this section, we examine the behavior of distributed random forests when breaking the IID assumption. Specifically, we vary the class probabilities of both the global dataset and the local subsets independently and examine the impact on the global and local accuracy. We use the probability mass function of the Skellam distribution p(k; f1, P2) (11++)2) (μn /2) 4/2 (2///p2))) [Iwwin 1937 Skellam 1946) with μ = distribution which tends toward the normal distribution (O, 2u) for large (Irwin 1937). To obtain the class p)) at E (c C/2 c e Zc). Decreasing
to generate different class imbalances, as it is a parametrizal ble discrete
weights for the C classes Z = {0, 1,..., 1),,, we sample p( μ. μ thus increases class imbalance. We use the edge case μ oo to indicate balanced classes. Figure B(a) illustrates the resulting class frequencies for different, μ. In this experiment, we consider two different. kinds of
11

---

Published in Transactions on Machine Learning Research (11/2025)
class imbalances. First, consider the imbalance of the global dataset, that is, when generating the global dataset, we use the class frequencies prescribed by Mdaat- Second, we vary the imbalance of the data partition into local datasets to create non-llD data, i.e, the class frequencies of the node-local datasets are different across nodes and compared to the global dataset. For this, we use a class distribution with Apartition where each node i E Zp uses a cyclically shifted distribution, shifting the assignment of sampling position & to class e by (c l C/p + 0.5l) mod C. We then partition the global dataset class-wise, assigning each node a fraction of the samples labeled with that class according to their relative frequency compared to other nodes. Depending on the global class imbalance, this may not result in the exact local class frequencies prescribed by Ppartition, but rather the closest match to partition the global dataset according to these weights. Tt can also lead to an imbalance in the size of the local datasets across nodes, as requesting large share from the majority class results in more samples than a large share of a minority class. Figure B(b) illustrates the resulting global and local class frequencies for all four combinations of Jdata = {00, 2} and Ppartition = {co, 2).
We evaluate the prodictive performance of distributed random forests trained on distributed data with different. factors of imbalance. We train three dalaset sizes 100K with 224 trees, 1M with / t= 800, and 10M with 224 trees on p = 16 compute nodes, partitioning and distributing both trees and training data over the nodes. Using the process described above, we generate global datascts ranging from no class imbalance (Muata = xo) to a strong imbalance of up to Pdata = 0.5. These global datasets are then partitioned and distributed among the 16 nodes, using varying partition imbalances Ppartition: which affect the difference in data distribution across the 16 local subforests. Overall, we evaluate six values μ = {0o,, 10, 5, 2, 1, 0.5) for both Pdata and Mpartition, resulting in a total of 36 combinations per dataset size. To avoid the over- representation of majority classes, we use balanced accuracy (macro average). Figure g gives the balanced accuracy on three datasets 100K, 1M, and 10M, while varying both the data and partition imbalance from do to 0.5. Additional metrics are given in Figures 10 and [] The imbalance of the global dataset is shown on the x-axis, with the imbalance increasing as Udata decreases. As expected, the classification gets increasingly harder, and the balanced accuracy decreases with a stronger imbalance. This applics across all three datasets and for all partition imbalances. Interestingly, increasing the partition imbalance (by decreasing ('partiiion) can actually improve the predictive performance of the global model. For imbalanced global datasets with Udata S 10, the accuracy consistently improves as the local imbalance increases from ppartition = 3o to ppprtition = 1. For balanced global dataset Idata 0c, we observe almost no difference between /pprttiionn = xo to /lpartition = 1. Only Mpartiiioon = 0.5 performs significantly worse than Mpartition oc for some cases, especially for datasets with fewer samples and more balanced global class frequency. This suggests that there is a limit to how imbalanced the local subsets may be before performance degrades. Overall, our results suggest that some class imbalance in the local datasets has only a small impact on accuracy when the global dataset is balanced and may even improve performance on imbalanced global data. Appendix A.55 investigates this effect in more detail by comparing local and global confusion matrices for IID and non-IID partitions. An important limitation of this experiment is the restriction to synthetic data; however, this allows us to vary the dataset size and imbalance freely, focusing purely on the effects of non-IID data without side effects. Repeating these experiments with real-world data will be an important next step in evaluating the practical implications of these observations.
6 Conclusion
This paper studies the scalability of distributed random forests. By distributing independent subforests across compute nodes and utilizing shared-memory parallelization within each node, we implement a hybrid parallel approach to distributed random forests. We examine the parallel scalability of this approach in both strong and weak scaling experiments, scaling up to 64 compute nodes with 76 cores each, to total of 4864 cores. We achieve strong scaling speedups of up to 31.98, which level out as the work per node diminishes, and weak scaling efficiencies above 0.96 for all tested scales. The parallelization docs not impact the random forest's 's predictive performance and the accuracy of the global model with / trees remains unchanged when increasing the compute nodes p. In contrast, the mean accuracy of the local models decreases with p as their size t/p decrcasos. We compare two approaches to inference using either a distributed model with global voting or aggregating the global model on a single node, followed by sequential inference. The global voting approach benefits from low memory consumption per node and faster inference at the cost of using more
12

---

Published in Transactions on Machine Learning Research (11/2025)
[a) 100K -ataset
(b) M-Dataset
[c) 10M- Dutaset
100%
100%
1009-
Balanced Accuracy
Highor is better
Balnnced Accuracy
Balanced Accuracy
Higher letite
75%
75%
75%
Higher
500%
50%
50%
25%
25%
0.5
0%
0%
09
10
0.5
10
0.5
Plans
Pau
Paa
Figure 9: The balanced accuracy for different combinations of class imbalance in the global data (Adata) and the local subsets (Apartition) using three different problem sizes: 100K with t = 224 trees, 1M with = 800, and 10M with 224 trees on p = 16 compute nodes. The smaller lA, the stronger the imbalance. Increasing the imbalance of the global data consistently decreases predictive performance, while an increased partition imbalance can improve performance imbalanced global data.
compute nodes. In contrast, aggregating global model introduces a computational and memory overhead, yet. requires only a single node and sharing of inference data. We further investigate how model size, the number of training samples, their partitioning, and class distribution impact the predictive performance of large-scale random forests. As expected, increasing both the number of trees and the number of training samples improves results. However, partitioning the training data, for example, across distributed machines, has a significant impact on the resulting accuracy. In testing various combinations of class imbalance in the global data and the partition into local subsets, we find that increasing global imbalance complicates the classification of minority classes, Introducing partition imbalance can counteract this increasing diversity between subforests. This work improves the understanding of the performance characteristics of distributed random forests and their limitations at scale. While the training scales near perfectly, given enough work is available, inference involves tradeoff between memory consumption, compulation time, and data privacy. Our results on data distribution and heterogeneity provide key insights for applying distributed ensembles like random forests in practice and in the context of federated learning. In the future, we aim to extend our experiments to real-world datasets, non-classification tasks like regression, and a broader range of ensemble methods.
Acknowledgments
This work was funded by Helmholtz Association's 's Initiative and Networking Fund through the Helmholtz AT platform grant. The authors gratefully acknowledge the computing time provided on the high-peeformance computer HoreKa by the National Perforeeffom Computing Center at KIT (NHR@KIT). This center is jointly supported by the Federal Ministry of Education and Research and the Ministry of Science, Research and the Arts of Baden-Wünttemberg, as part of the National High-Performane Computing (NHR) joint funding program https///wwww.hh-eeeieie /edd/ee//uurprrrtersss HoreKa is partly funded by the German Research Foundation (DFG).
References
Sherif F. Abdoh, Mohamed Abo Rizka, and Fahima A. Maghraby. Cervical Cancer Diagnosis Using Random Forest Classifier With SMOTE and Feature Reduction Techniques. IEFE Access, 6:59475-599 594885, 2018. ISSN 2169-3536. /
Ibrahim M. Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting Neural Scaling Laws in Language and Vision. Advances in Neural Information Processing Systems, 55:22300-22312 December 2022.
Md. Zahangir Alam, M. Saifur Rahman, and M. Sohel Rahman. A Random Forest based prodictor for medical data classification using feature ranking. Informatics in Medicine Unlocked, 15:100180, Jamuary 2019. ISSN 2352-9148. 40010.00006/imm.200.100011
13

---

Published in Transactions on Machine Learning Research (11/2025)
Penjan Antonio Eng Lim and Cheong Hee Park. A collaborative ensemble construction method for fed- erated random forest. Expert Systems with Applications, 255:124742, December 2024. ISSN 0957-4174. doi 100_:000000/1.eva2222..2222.1
Marjan Asgari, Wanhong Yang, and Mahdi Farnaghi. Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data spark dis- tributed framework. Environmental Technology & Innovation, 27:102776, August 2022. ISSN 2352-1864.
doi 1att110110000 (0t.//..ti.2022.1.1... 102776
N Azizah, LS Riza, and Y Wihardi. Implementation of random forest algorithm with parallel computing R. In Journal of Physics: Conference Series, volume 1280, pp. 022028. 1OP Publishing, 2019.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neu- ral scaling laws. Proceedings of the National Academy of Sciences, 121(27)x2231188122,, July 2024. doi 10.1000..11000 http//aas..... 2311878121
P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5(1):4308, July 2014. ISSN 2041-1723. 1oi110.0038/ncome ncomms530
Justin D. Basilico, M. Arthur Munson, Tamara G. Kolda, Kevin R. Dixon, and W. Philip Kegelmeyer. COMET: A Recipe for Learning and Using Large Ensembles on Massive Data. In 2011 IEEE 11th International Conference on Data Mining, pp. 41 50, December 2011. 1t.11000111CD..... 2374-8486.
ISSN:
Mariana Belgiu and Lucian Drägut. Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrummetry and Remote Sensing, 114:24-31, April 2016. ISSN 0924- 2716. ht000///.spprspps2201....00
Tal Ben-Nun and Torsten Hocfler. Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis. ACM Compuling Surveys, 52(4))65:1 65:43, August 2019. ISSN 0360-0300.
L. Benali, G. Notton, A. Fouilloy, C. Voyant, and R. Dizene. Solar radiation forecasting using artificial neural network and random forest methods: Application to normal beam, horizontal diffuse and global compo- nents. Renewable Energy, 132:871-888 March 2019. ISSN 0960-1481. doi 10.10166...eennne2200 0.08.044 044
Gérard Biau. Analysis of a random forests model. Journal of Machine Learning Research, 13(1):1063 1095, April 2012. ISSN 1532-4435.
Lco Breiman. Classification and regression trees. The Wadsworth statistics/probaaiiit series. Wadsworth & Brooks/Cole, Pacific Grove, California, 1984.
Leo Breiman. Random Forests. doi 10.103/ A:1010933:043241
Machine Learning, 45(1):5 32, October 2001.
ISSN 1573-0565
Jianguo Chen, Kenli Li, Zhuo Tang, Kashif Bilal, Shui Yu, Chuliang Weng, and Kegin Li. A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment. IEEE Transactions on Parallel and Distributed Systems, 2844)9919-933 2017. T...0.1199/TPDS..O 2016.2603511
Ashley Cliff, Jonathon Romero, David Kainer, Angelica Walker, Anna Furches, and Daniel Jacobson. A High-Peeformance Computing Implementation of Iterative Random Forest for the Creation of Predictive Expression Networks. Genes, 10(12):996, December 2019. ISSN 2073-4425. doi 10.3390/ 209966
Publisher: Multidisciplinary Digital Publishing Institute.
Marcos F. Criado, Fernando E. Casado, Roberto Iglesias, Carlos V. Regueiro, and Senén Barro. Non-lII data and Continual Learning processes in Federated Learning: A long road ahead. Information Fusion 88:263-280 December 2022. ISSN 2535. doi 110.1016/..inffi 2tt.f/s..22.07.0022
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: simplified data processing on large clusters. Commun. ACM, 51 5111:107711 January 2008. ISSN 0001-0782. 110.114451337452.1515
14

---

Published in Transactions on Machine Learning Research (11/2025)
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In Pro- ceedingg of the 40th International Conference on Machine Learning, pp. 10835-10866. PMLR, July 2023. ISSN: 2640-3498.
Mert Gencturk, A. Anil Sinaci, and Nihan Kesim Cicekli. BOFRE: A Novel Boosting- Based Federated Random Forest Algorithm on Horizontally Partitioned Data. IEEE Access, 10:89835-89857 2022. ISSN 2169-3536. doi T00.T1000AACCSSS20022222222
Robin Genuer, Jean-Michel Poggi, Christine Tuleau-Malot, and Nathalie ViilanVialaniix. Random Forests for Big Data. Big Data Research, 9:28 46, September 2017. ISSN 2214-5796.
Jiawei Han, Yanheng Liu, and Xin Sun. scalable random forest algorithm based on MapReduce, In 2013 IEEE 4th International Conference on Software Engineering and Service Science, pp. 849-852. IEEE, 2013. doi 10.1 10.11009/CCSSSSS2013...... /C////CSSSSS.211.615555533
Anne- Chrr-hhisttnn Hauschild, Marta Lemanczyk, Julian Matschinske, Tobias Frisch, Olga Zolotareva, Andreas Holzinger, Jan Baumbach, and Dominik Heider. Federated Random Forests can improve local performance of predictive models for various healt hcare applications. Bioinformatics, 38(8):2278 2286, April 2022. ISSN 1367-4803. doi 10.1093/ bioinformatics/laeeeeie hatpaaatttcccttttt55.. /btac665
Song He, Jianhua Wu, Dan Wang, and Xiaodong He. Predictive modeling of groundwater nitrate pollution and evaluating its main impact factors using random forest. Chemosphere, 290:133385 March 2022. ISSN
0045-6535. doi 20.1000/f.cheensspeeee222111..1.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramosh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modcling, November 2020. arXiv:2010.1470 [cs].
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the Effects of Non-Identical Data Distri- bution for Federated Visual Classification, Seplember 2019. arXiv:1909.0665 06633 [cs].
Joseph Oscar Irwin. The frequency distribution the difference between two independent variates following the same poisson distribution. Journal of the Royal Statistical Society Series A: Statistics in Society, 100 (3):415-416 416, 1937.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jelfrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020.
arXiv:2001.0880 [cs, stat].
Pavleen Kaur, Ravinder Kumar, and Munish Kumar. A healthcare monitoring system using random forest and internet of things (IoT). Multimedia Tools and Applications, 18(14):19905-19916 July 2019. ISSN 1573-7721. 00010000777//111110099-7777
Mohammed Khalilia, Sounak Chakraborty, and Mihail Popescu. Predicting disease risks from highly im- balanced data using random forest. BMC Medical Informatics and Decision Making, 11(1):51, July 2011.
ISSN 1472-6947.
Hong Li, Jiajian Lin, Xiaobao Lei, and Tianxia Wei. Compressive strength prodiction of basalt fiber reinforced concrete via random forest. algorithm. Materials Today Communications,, 30:103117, March 2022. ISSN 2352-4928. doi /tt0://..ncoom.20211..000
Qinbin Li, Wu Zhaomin, Yanzheng Cai, Ching Man Yung, Tianyuan Fu, Bingsheng He, et al. FedTree: A Federated Learning System for Trees. Proceedings of Machine Learning and Systems, 5:89 103, 2023.
Chao Liu, Zhaojun Gu, and Jialiang Wang. A Hybrid Intrusion Detoction System Based on Scalable K-Means+ Random Forest and Deep Learning. IEEE Access, 9:75729-75740, 2021. ISSN 2169-3536. 0tw.0.1000/ACCCSS22202.....
15

---

Published in Transactions on Machine Learning Research (11/2025)
Yang Liu, Yingting Liu, Zhijie Liu, Yuxuan Liang, Chuishi Meng, Junbo Zhang, and Yu Zheng. Federated forest. IEEE Transactions on Big Dala, 8(3):843 11843-854, 2020.
Zili Lu, Heng Pan, Yueyue Dai, Xueming Si, and Yan Zhang. Federated Learning With Non-IID June 2024. ISSN 2327-4662.
Data: A Survey. JEEE Internet of Things Journal,
Xiaodong Ma, Jia Zhu, Zhihao Lin, Shanxuan Chen, and Yangjie Qin. A state-of-the-art survey on solving non-IID data in Federated Learning. Future Generation Computer Systems, 135:244-25$ 258, October 2022. ISSN 0167-739X.
www.1011//...uuur.e.. watlburee2222.00 2022.05.003
Mchul Madaan, Aniket Kumar, Chirag Keshri, Rachna Jain, and Precti Nagrath. Loan default prodiction using decision trees and random forest: A comparative study. JOP Conference Series: Materials Science and Engineering, (102(4):, January 2021. ISSN 1757-899X. 1oi 00000.0088/1777-000 17571177777711111 /00T77/-899X/0222/2/00000 Publisher: IOP Publishing.
Tijana Markovic, Miguel Leon, David Buffoni, and Sasikumar Punnekkat. Random Forest Based on Feder- ated Learning for Intrusion Detection. In llias Maglogiannis, Lazaros lliadis, John Macintyre, and Paulo Cortez (ods.), Artificial Intelligence Applications and Innovations, pp. 182-144, Cham, 2022. Springer International Publishing. ISBN 778-3-031-083333 doi 1001007/9988---031-08883
11
Saman Masarat, Saced Sharifian, and Hassan Taheri. Modified parallel random forest for intrusion detection systems. The Journal of Supercompuling, 72(6):2235 2258, June 2016. ISSN 1573-0484.
10.107 000007/11120 1111227-01161177--
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas, Communication--Fifeeeeet Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pp. 1273 1282. PMLR, April 2017.
ISSN: 2640-3498.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The Quantization Model of Neural Scaling, Advances in Neural Information Processing Systems, 36:28699 28722, December 2023.
U. Milasheuski, L. Barbieri, B. Camajori Tedeschini, M. Nicoli, and S. Savazzi. On the Impact of Data Het- erogeneity in Federated Learning Environments with Application to Healthcare Networks. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp. 1017-1023, June 2024. doi 10.1109///0 N0.1119/CA15956 200400699.0224.00555
Lawrence Mitchell, Terence M Sloan, Muriel Mewissen, Peter Ghazal, Thorsten Forster, Michal Pi- otrowski, and Arthur S Trew. A Parallel Random Forest Classifier for R. In Proceedings of the Sec- ond International Workshop on Emerging Computational Methods for the Life Sciences, pp. 1-6, 2011. doi 011445/1996023.1199
James N Morgan and Robert C' Messenger. THAID, a sequential analysis program for the analysis of nominal scale dependent variables, 1973.
James N. Morgan, and John A. Sonquist. Problems in the Analysis of Survey Data, and a Pro- posal. Journal of the American Statistical Association, 58(302))4155443) June 1963. ISSN 0162-1459. doi 10i110.0000001600 010000006114491199 10500855
Cuong Nguyen, Yong Wang, and Ha Nam Nguyen. Random forest classifier combined with feature selection for breast cancer diagnosis and prognostic. 2013, May 2013.
2013..e..111....07 Publisher:
Scientific Rescarch Publishing.
Thais Mayumi Oshiro, Pedro Santoro Perez, and José Augusto Baranauskas. How Many Trees in a Random Forcst? In Petra Perner (od.), Machine Learning and Data Mining in Pattern Recognition, pp. 154-168, Berlin, Heidelberg, 2012. Springer. ISBN 978-3-642-31537-4. doi 1o1[0.0007//76--3222-333377 13.
Biswanath Panda, Joshua S. Herbach, Sugato Basu, and Roberto J. Bayardo. PLANET: massively parallel learning of tree ensembles with MapReduce. Proc. VLDB Endow., 2(2):1426-144 1437, August 2009. ISSN 2150-8097. 1o.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Philipp Probst and Anne-Laure Boulesteix. To Tune Not to Tune the Number of Trees in Random Forest. Journal of Machine Learning Research, 18(181):1-18, 2018. ISSN 1533-7928.
J. Ross Quinlan. Induction of Decision Trees. Machine Learning, 1:81-106, 1986.
Paulo Angelo Alves Resende and André Costa Drummond. A Survey of Random Forest Based Methods for Intrusion Detection Systems. ACM Comput. Surc., 51(():48:1-448) 48:36, May 2018. ISSN 0360-0300.
1o.104.445/317855
Peter Sanders, Kurt Mehlhorn, Martin Dietzfelbinger, and Roman Dementiev. Sequential and Parallel Algorithms and Data Structures: The Basic Toolbor. edition, 2019. ISBN 178-3-030-25209-1
Kennody Senagi and Nicolas Jouandcau. Parallel construction Random Forest on GPU. The Journal of Supereonpuling 78(8):10480 10500, May 2022. ISSN 1573-0484. 000100000/111222-00-0002222
Utkarsh Sharma and Jared Kaplan. Scaling Laws from the Data Manifold Dimension. Journal of Machine Learning Research, 23(9):1-34, 2022. ISSN 1533-7928.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to federated learning with class imbalance. In International Conference on Learning Representations, 2022.
Jingzhe Shi, Qinwei Ma, Huan Ma, and Lei Li. Scaling Law for Time Series Forecasting, Advances in Neural Information Processing Systems, 37:83314-83344 December 2024.
John G Skellam. The frequency distribution of the difference between poisson variates belonging to different populations. Journal of the Royal Statistical Society Series A: Statistics in Society, 109(3): 296-296, 1946.
Yu-Lin Tsou, Hong-Min Chu, Cong Li, and Shao-Wer Yang. Robust Distributed Anomaly Detection Using Optimal Weighted One-Class Random Forests. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 1272 1277, November 2018. doi /CC...201..0.00171 2001:00000000///0MMM00000..00000
ISSN: 2374-8486.
Hristos Tyralis, Georgia Papacharalampous, and Andreas Langousis. A Brief Review of Random Forests for Water Scientists and Practitioners and Their Recent History in Water Resources. Water, 11(5):910, May 2019. ISSN 2073-4441. doi (00.3390/w1105000 Number: Publisher: Multidisciplinary Digital Publishing Institute.
Mohammad S. Uddin, Guotai Chi, Mazin A. M. Al Janabi, and Tabassum Habib. Leveraging random forest in micro-enterprises credit risk modelling for accuracy and interpretability. International Journal of Finance & Economics, 27(3):3713-377 3729, 2022. ISSN 1099-1158. 1a010..002/iiff.222
Brian Van Essen, Chris Macaraeg, Maya Gokhale, and Ryan Prenger. Accelerating Random Forest Classifier: Multi-Core, GP-GPU, or FPGA? In 2012 IEEE 20th International Symposium on Field- Programmable Custom Computing Machines, pp. 232-239, April 2012. doi
Fernando Vázquez-Novoa, Javier Conejero, Cristian Tatu, and Rosa M. Badia. Scalable Random Forest with Data-Parallel Computing. In José Cano, Marios D. Dikaiakos, George A. Papadopoulos, Miquel Pericàs, and Rizos Sakellariou (eds.), Euro- 2023: Parallel Processing, pp. 397-410, Cham, 2023. Springer Nature Switzerland. ISBN 078-3-031-39698-4
__2
Ryoji Wakayama, Ryuei Murata, Akisato Kimura, Takayoshi Yamashita, Yuji Yamauchi, and Hironobu Fujiyoshi. Distributed forests for MapReduce- basod machine Icarning. In 2015 3rd IAPR Asian Conference on Pallern Recogration (ACPR), pp. 276 280. IEEE, 2015.
Chenxu Wang, Tingting Cai, Guang Suo, Yulong Lu, and Enqiang Zhou. Fost.ooest:: A Parallel Random Forest Training Framework Based on Supercomputer. In 2018 IEEE 20th International Conference on High Performance Computing and Communications: IEFE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC//SartCityy/ /DSS), pp. 196 204204. June 2018. /000
17

---

Published in Transactions on Machine Learning Research (11/2025)
Sutong Wang, Yuyan Wang, Dujuan Wang, Yunqiang Yin, Yanzhang Wang, and Yaochu Jin An improved random forest-based rule extraction method for breast. cancer diagnosis. Applied Soft Compuling, 86: 105941, January 2020. ISSN 1568-4946. 101000.0000//1a00222901111111111
Daniel Whiteson. HIGGS, 2014. URL http:://wwarraee............e.ee
Marvin N. Wright and Andreas Ziegler. ranger: A Fast. Implementation Random Forests for High Di- mensional Data in C++ and R. Journal of Statistical Software, 77:1 17, March 2017. ISSN 1548-7660. doi htw.116677/ss.v07....
Yuncheng Wu, Shaofeng Cai, Xiaokui Xiao, Gang Chen, and Beng Chin Ooi. Privacy preserving vertical federated learning for tree-based models. Proc. VLDB Endow. 13(12):2090 2103, July 2020. ISSN 2150- 8097. doi
Qiang Yang, Yang Liu, Tianjaan Chen, and Yongxin Tong. Federated Machine Learning: Concept and Applications. ACM Trans. Intell. Syst. Technol, 10(2):12:1112:11 January 2019. ISSN 2157-6904.
doi110.115/3329695
Chuanting Zhang and Dongfeng Yuan. Fast Fine- Grained Air Quality Index Level Prediction Using Random Forest Algorithm Cluster Computing of Spark. In 2015 IEEE 12th Inll Conf on Ubiquilous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC- ScalCom), pp. 9299999999999 934,,,,, August 2015. doi 10.100//ICCAATC--aaCCo...mm Caw.aaCooooooDDCCoooo..00.......
18

---

Published in Transactions on Machine Learning Research (11/2025)
A Appendix
A.1 Distributed Random Forests Pseudocode
AggorthhmSsunnn summarizes the tree-parallel training of distributed random forests. A describe the two variants for inference, either aggregating a global model or using global voting for distributed inference.
Algorithm 1: Training of local subforest on each compute node.
Input: Local training data Diocal, number of global trees (global, number of compute nodes p
1 Hoocll f- 'global/
>> Global forest distributed over nodes
Fiocal randomfoorstt
Create independent local subforest
3 foreach E Flocal do in parallel
Shared-menory parallelization within each node
grow Diocal)
Algorithm 2: Aggregation of the global model and local inference.
Input: Local inference data D, local forest Fiocal Fglohal + all_aaaaaa aalll_gthrr((_ (Focat)
b Collect global model, only once after training Local inference on global model
return Fglobal. predict(D)
Algorithm 3: Inference with the distributed model via global voting.
Input: Shared inference data D, local forest Flocal
1 Mocal + Flocal-
(isttitt__stttra(()) lgtobal +- all aallaaleeeee lelu_reduceeee (Munca)
b Collect local predictions as histogram
Sum local histograms
return argmax(( (hglobal)
Global prediction is the most-voted class
A.2
Experimental Setup
A.2.1 Implementation
We implement the distributed random forest in Python using scikit-learn and MPI. Each node-local sub- forest a scikit-learn annddooForestcasssiiiie and trained via the fit method, passing in local training data and labels train each individual decision tree in the ensemble. For shared-memory parallelization, we enable scikit-learn's parallelization in the local subforests via the n_jobs parameter. All-gathering the global model and aggregating the local histograms for global voting is implemented in mpi4py. We train random forests of differrnt sizes, varying the number of trees but keeping to the default hyperparameters defined by scikit-learn. Our code is open-source and publicly available at ggidab.com/Hccmmott--All Hemmhollzllll Energy /spccccccaall-osoousss
A.2.2 Computational Environment
All experiments were conducted up to 64 compute nodes, each of which has two Intel Xcon Platinum 8368 processors for total of 76 cores, kB L1 and 1 MB L2 cache per core, and 57 MB L3 cache per processor. Most experiments used standard compute nodes with 256 GB main memory. The exception is the serial baseline for the strong scaling experiments, which used high-memory nodes with 512 GB main memory but otherwise identical hardware to fit the model and data. All nodes are connected with InfiniBand 4X HDR 200 Gbit/s interconnect. All experiments used OpenMPI v4.1.6, Python v3.11.2, mpi4 v4.0.1, numpy v2.2.2, scikit-learn v1.6.1, and scipy v1.15.1.
19

---

Published in Transactions on Machine Learning Research (11/2025)
A.3 Detailed Results Scaling Laws of Random Forests
Tables and give the results illustrated in Figure in labular format.
Table 2: The predictive performance in test accuracy when scaling the total number of trees & X for two baselines 1M with & 25 and 10M with 8 =
1M
10M
A
11
Trees t
Accuracy
t1
Trees
Accuracy
7.5 X 10$
25
25
0.136
7.5 X 106
0.490
7.5 x 105
25
50
0.152
7.5 x 106
0.632
7.5 x 10$
25
100
0.176
7.5 106
28
0.767
7.5 x 105
25
200
0.213
7.5 X 106
56
0.862
16
7.5 X 10s
25
400
0.271
7.5 X 106
7
112
0.915
32
7.5 x 10
25
800
0.357
7.5 X 106
224
0.941
32
7.5 X 10s
25
800
0.358
7.5 x 106
224
0.941
64
7.5 x 10s
25
1600
0.476
7.5 x 106
448
0.953
64
7.5 X 10$
25
1600
0.476
7.5 106
448
0.953
128
7.5 x 105
25
3200
0.613
7.5 106
896
0.958
256
7.5 x 105
25
6400
0.736
7.5 106
1792
0.961
512
7.5 x 10$
25
12 800
0.820
7.5 X 106
7
3584
0.962
1024
7.5 x 105
25
25 600
0.867
7.5 x 106
7
7168
0.963
2048
7.5 x 10$
25
51 200
0.892
7.5 x 106
14336
0.963
Table 3: The predictive performance in test accuracy when scaling the total number of trees X A- with without data scaling. With data scaling, the number of training samples grows with À n X 71 Without data scaling, the training set does not grow with X, remaining al a constant T4-
With Data Scaling
Without Data Scaling
11
t
A
Trees t
Accuracy
n
Accuracy
1.2 x 104
25
25
1.2 X 104
0.111
1.2 x 104
0.111
1.2 x 104
25
50
2.3 x 104
0.116
1.2 104
0.113
1.2 X 10a
25
100
4.7 104
0.129
1.2 104
0.117
1.2 10a
25
8
200
9.4 x 104
0.165
1.2 10a
0.127
1.2 10a
25
16
400
1.9 X 10s
0.214
1.2 104
0.145
1.2 x 10a
25
32
800
3.8 x 105
0.315
1.2 x 104
0.160
1.2 x 1
25
64
1600
7.5 x 10a
0.474
1.2 x 104
0.188
1.2 105
7
7
1.2 10S
0.276
1.2 10a
0.276
1.2 105
7
2
14
2.3 x 105
0.415
1.2 x 103
0.357
1.2 105
7
4
28
4.7 x 105
0.588
1.2 X 105
0.471
1.2 x 105
7
8
56
9.4 x 105
0.767
1.2 x 105
0,600
1.2 105
7
16
112
1.9 x 106
0.882
1.2 x 105
0.721
1.2 x 105
7
32
224
3.8 x 106
0.933
1.2 x 10
0.810
1.2 x 105
7
448
7.5 x 106
0.953
1.2 10
0.862
20

---

Published in Transactions on Machine Learning Research (11/2025)
A.4 Detailed Results Parallel Scalability
Tables and give the results presented in Section 5 in tabular format.
Table 4: Comparison of the two inference variants either aggregating the global model using distributed inference via global voting in weak scaling setup, giving the parallel training time Twain time to aggregate the global model (p), and time to perform inference on the entire test set (p). Note that the global model performs sequential inference on a single node, while the inference with global voting is parallelized across p nodes. We further include the memory consumption (Mem) per nodc. When aggregating a global model, the memory consumption per node scales with the global model size i, while with global voting, it scales only with the local model size t/p.
Global Model
Global Voting
p
t
t/p
T (p)
T Tatther (p) +
Trest
Mem +
T T (p)
Tust(p) +
Mem
2
152
76
5.776s
0.550 s
0.179s
3.3 GiB
5.677
0.110s
2.5 GiB
4
304
76
5.588
3.673
0.307
GiB
5.584
0.115s
2.6
100K
608
76
5.677s
6.052
0.558
GiB
5.632
0.123s
2.6
1216
76
s
7.756
1.059
9.7
5.739
0.112s
GiB
2432
76
5.944
1
2.047s
GiB
6.022
0.123
G.B
64
4864
76
6.412s
23.226
4.015
31B5.iB
6.514
0.122
2.B
152
76
30.119s
4
0.908
GiB
30.396 s
0.523
6.B
304
76
3
6.135
1.672s
GiB
30.100
0.527
8.1
1M-b
608
76
30.051
13.004
3.206
GiB
30.002
0.545
8.6
16
1216
76
30.327s
23.451
6.313s
GiB
30.280
0.545
6.7 GiB
32
2432
76
30.519s
42.331
12.504s
82.6 GiB
30.632
0.546
7.B
64
4864
76
31.089
115.327
25.889
159.8 GiB
31.016s
0.546
GiB
Table The global training time T(p), speedup S(p), and accuracy for p 1,..., the training data, i.c., each node trains t/p trees on n/p local training samples.
64 nodes when distributing
Nodes p
Trees t
t/p
n/p
T(p) +
S(p) +
Accuracy f
1
1600
1600
7.5 X 105
5923.104s
1.000
0.476
2
1600
800
3.8 x 105
1361.426s
4.351
0.420
4
1600
400
1.9 X 10a
331.704s
17.857
0.365
1M
8
1600
200
9.4 x 10*
79.640s
74.373
0.315
16
1600
100
4.7 x 104
22.1138
267.854
0.269
32
1600
50
2.3 104
5.848
1012.908
0.232
64
1600
25
1.2 10a
2.534
2337.587
0.196
1
448
448
7.5 x 106
8067.080
1.000
0.953
2
448
224
3.8 x 106
1331.368
6.059
0.942
448
112
1.9 x 106
350.751s
22.999
0.929
IONI
8
448
56
9.4 105
86.204s
93.581
0.913
16
448
28
4.7 105
28.089$
287.201
0.892
32
448
14
2.3 x 105
11.861s
680.150
0.868
64
448
7
1.2 x 105
5.584
1444.747
0.838
21

---

Published in Transactions on Machine Learning Research (11/2025)
A.5 Additional Details the Effects of Class Imbalance
Figures 10 and 11 give the predictive performance with differing class imbalance using two additional metrics. The micro average of the accuracy weighs each sample with the same importance, thus placing a greater importance on majority classes. This effect can be observed in Figure [0] where the accuracy increases at strong imbalance as the majority class makes up an increasing share of all samples, especially in subfigure (b). Figure 11 gives the Cohen's kappa score
Po Pe
K
1- Pe
comparing the observed agreement Po between prediction and true label to the expected agrecment Pe. basod on the underlying class imbalance.
(a) 100K-Dataset
(b) IM-Dataset
(c) 10M-Dataset
100%
100%
100%
Upattee
Highes ta beroe
75%
75%
75%
Accuracy
Accurccy
Accuracy
Highe
Righe
50%
50%
50%
25%
25%
0.5
0%
0%
0%
0.5
10
Paa
Pau
Pan
Figure 10:
The (micro-average) accuracy for different combinations class imbalance in the global data (Maata) and the local subsets (Hpartition).
CUUK-Dataset
1M
10M-Dataset
100%
Higherr bener +
100%
100%
Cohen's Kappa x
Cohen's 's Kappa K
Coloc''s Kappa K
75%
75%
75%-
10
50%
50%
50%
Cohee's
25%
25%
25%
0%
0%
10
0.5
10
5
0.5
0.5
Paats
Figure 11: The Cohen's Kappa k. for different combinations of class imbalance in the global data (Jidata) and the local subsets (Ppartition).
To further investigate the beneficial effect of partition imbalance, we provide more detailed examination of the prodictive performance imbalanced data. Figures and 13 13 global and local confusion matrices for Adata = 2, comparing balanced partition with Ppartition oo and an imbalanced partition with ppartition 1. Figure 12 presents the confusion matrix of the global model and exemplary local model (for node eight, using a balanced data partition). While the majority classes are retrieved mostly correctly, the global model struggles to detect the minority classes. As the data partition is balanced, all subforests are trained using the same class imbalance (that of the global data (daaa). As a result, all local subforests fail to detect the rare classes. Figure 13 gives the corresponding global and local confusion matrices, but using an imbalanced data partition with Ppartition 1. The class distribution used train the local subforests thus varies significantly from the global distribution. While this can reduce the predictive performance of individual subforests the global test set, it strengthens the performance of the collective ensemble. In contrast to the balanced data partition, a non-lll) data partition can increase the diversity between the local subforests, thereby improving the performance of the global ensemble. Table D gives the predictive performance of the global and local models with Pdata = 2 and Apactition = oo compared to ppartition 1.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Global Model
Local Model
Class Distribution Test
Prediction
Prediction
2
5
8
9
0%
25%
21%
21%
40%
189g
1758
2
10%5
30%
True Label
1%
1%
4
0%
0%
0%
20%
0%
7
1%
16
10%
8
10%
9
175
17%
0%
Class Distribution
0%
0%
Prediction
25%
25%
Class Distribution Training 25%
0%
25%
Figure 12: The confusion matrix the global model (left) and the local model on node eight (right) using imbalanced data with (daaaa = and balanced data partition with Mpartition = co. We also provide the class distribution of the global test set, the global and local training sets, and the predictions of the global and local models. The class distribution of the local model identical the global distribution, and both fail at detecting the minority classes to 7.
Table The predictive performance with Pdata 2 and IID (A4prtittion = oc) or non-IID (Apartition = 1) data for both the global model and as average over all local models. While the local models trained on non-IID data perform worse individually, their ensemble outperforms the training IID data thanks to increased diversity between the subforests.
Mpartition
P'pattiiinn
Metric
Global Model
Local Models
Global Model
Local Models
Accuracy Balanced Accuracy Cohen's Kappa
0.761
0.745
0.826
0.432
0.473
0.462
0.560
0.331
0.711
0.691
0.791
0.346
23

---

Published in Transactions on Machine Learning Research (11/2025)
Global Model
Local Model
Class Distribution Test
Prediction
Prediction
8
9
0%
25%
0%
40%
18%
30%
True Label
2%
4
0%
216
5
0%
20%
0%
7
3%
10%
8
11%
9
17%
0%
0%
Class Distribution Prediction
0%
0%
25%
25%
Class Distribution Training 25%
0%
25%
Figure 13: The confusion matrix of the global model (left) and the local model on node eight (right) using imbalanced data with Adata = 2 and an imbalanced data partition with Apartition 1. We also provide the class distribution of the global test set, the global and local training sets, and the predictions of the global and local models. The class distribution of the local model differs significantly from the global distribution. The increased diversity between the local models slightly improves the detection of minority classes compared to the balanced partition in Figure 12
24


Reviews:
[{"review_id": "mbldGop9vn", "reviewer": "Reviewer_kMux", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions are empirical. \n\n1. Experimental study of scaling laws of random forests with varying model size and data size. \n2.  An implementation of distributed random forests for HPC. This is validated and shows a speed-up. \n3. An evaluation of the effects of data distribution and class imbalance.", "claims_and_evidence_yn": "No", "claims_explanation": "I believe that to claim scaling laws, a significnatly more extensive experimental setup should be considered. In looking at the figures, the reader cannot see a trend or a power law. \nAlso, the authors do not present any sort of approximation or fitting to the power laws that predict how random forrest will work.", "audience_yn": "No", "audience_explanation": "There are no findings or conclusions. This paper seems more like a technical report than a paper. The authors implemented a distributed RF implementation, and then run it on a couple of setups. I fail to see the utility of this for the ML community. \n\nIn all, no rate is given, and no law presented.\nIt is an empirical study, but, not even a fitting of the law w.r.t to the number of nodes is given.", "requested_changes": "1. Remove paragraph 3.2. I believe it cuts the flow of the paper. The content of this can be moved to the caption of Figure 1. \n2. Before Section 5, the authors explain: \"For shared-memory parallelization [...] \" That whole parragraph should be in the appendix.", "broader_impact_concerns": "No comment on this regard.", "additional_comments": "I believe there are citations missing like:\n\n1. \"An Agnostic Approach to Federated Learning with Class Imbalance\" Shen et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPTKCGMBJB", "reviewer": "Reviewer_b4LG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\nRandom forests are a popular class of machine learning algorithms that are particularly amenable to parallelization. Training random forests in parallel can result in large efficiency gains, and may be required in settings such as federated learning with sensitive datasets. The paper generates synthetic data to examine the performance of random forests (a) as the number of parallel compute resources increases and (b) as both the data size and compute resources increase. The paper considers cases in which the distributed nodes share a global dataset, and where each node has a local subset of the data. Finally, the authors examine the case where the local datasets at each node have class imbalance.\n\n### Strengths and weaknesses\n\n*Imbalanced data*. I think the analysis of imbalanced data settings is particularly interesting for practical uses of distributed random forests. For example when hospitals with different datasets use federated random forests, disparities in accuracy for under-resourced locations or between classes would be a major concern. One concern is that in the imbalanced setting, the authors mention that their technique for sampling from $\\mu_{partition}$ results in different dataset sizes between nodes, and the nodes with higher weights on the majority class tend to have larger sizes. I'm concerned that in real cases, the split between nodes would be constant, and that this correlation between local dataset size and class prevalence could interact with the results.\n\n*Related work.* The related work is thorough and detailed. However, it is often unclear how the prior findings listed such as Eng and Park (2024), Mitchell et al. (2011), Vazquez-Novoa et al. (2023), etc, compare to the contributions of this paper: it's difficult to tell what is the contribution of the work at hand. An additional specific concern is that Mitchell et al.'s results appear to show that distributed random forests do not scale well for real data, which questions the generalizability of these synthetic experiments.\n\n\n*Synthetic data*. This paper uses exclusively synthetic data. This is a strength for the imbalanced data case, because the authors can control the parameter $\\mu$ of the data to directly assess the effect of class imbalance. However, in the remaining experiments the use of synthetic data does not seem to be well-justified, because without having some parameter of the distribution being controlled, it's not clear to me why the authors could not use real data for this. \n\nSince the paper only considers a single distribution (independent normal distributions with variance 1) for the scaling results, it's difficult to have intuition for how these results would extend to more realistic distributions. If the variables interact in more complex ways -- as in real data -- the local dataset distributed settings would likely scale differently (likely more poorly). \n\nThe plots consider runs for only 3 random seeds and do not report standard error bars, so it's difficult to tell how significant the differences are between the curves (especially in the imbalanced setting where the curves are all very close together)\n\nFurthermore, the test accuracy on the 1M dataset is no better than random in the best case, indicating that this may not be a realistic approximation for a setting in which random forests would be used. \n\n\n*Implementation*. It would be helpful if the authors could clarify the sense in which the implementation is a contribution. Is the code released as a usable package that would easily work on other compute clusters? Furthermore, is the parallelization implemented here challenging to implement?", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, on the whole the claims of the paper are backed by evidence. However, the paper does not report standard deviations / error bars for the plots shown. Moreover, the generalizability of the paper is limited as the random forests are trained on synthetic data from a single distribution.", "audience_yn": "Yes", "audience_explanation": "Yes, researchers training distributed random forests in federated learning settings would be interested in this work.", "requested_changes": "Critical adjustments:\n1. Please include error bars in your plots\n2. Please either consider additional synthetic data distributions, or supplement the synthetic results with real data, or else provide a strong justification for why the results for this distribution would extend to real datasets\n3. Please provide justification for why it is reasonable to have different-sized local datasets in the imbalanced class setting, and why this does not contribute to the surprising result that imbalanced local data can be helpful\n4. Please clarify the relationship to related work\n5. Please explain why it is reasonable to consider settings where the best-case test accuracy is as good as random. \n\nMinor adjustments:\n6. It would be helpful to clarify the sense in which the implementation is a novel contribution", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w2gjfYn991", "reviewer": "Reviewer_FoSJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a thorough review, both in literature and via experiments, of parallel scaling for learning random forests on synthetic data.  There were not new algorithms presented, and the resulting experiments seem fairly standard.  Yet, this measured and fairly comprehensive report provides interesting results.  \n\n\n### Strengths:\n + The paper provides a really broad review of different attempts (many recent) in the literature of parallelizing the learning of random forests.  This part alone makes the paper useful for TMLR readers.  \n\n + The experimental evaluation considers several ways of scaling, with different data sizes in data set size and dimension, and synthetic data class balance.  \n\n + I found the results interesting (although perhaps for different reasons than the authors).  \n\n + The writing was well done, very thorough and efficient as it explained many experiments.  \n\n### Weaknesses:\n - The writing and prose overly focuses on the scaling concerns, and not enough on the effect on accuracy.  If accuracy of an approach drops by even 10%, it is going to be very hard to convince someone to use it in a meaningful prediction situation-- even if they have to wait 10 or 100 times a long.  Strong parallel scaling effects that leads to a noticeable drop in accuracy (of say 5%) compared to a global model should be strongly discounted.  \n  (the paper is right that there may be privacy or distribution shift reasons that could explain this away -- but I'd say these are often secondary to the iid test accuracy.)  \n\n - I would have liked to have seen discussion of XGBoost, and related methods that build trees dynamically -- and seemingly sequentially.  Random Forests have the most consistent benefits in this model.  However, because of the boosted nature, it appears much hard to parallelize.  \n\n - The fourth line of the paper claims that random forests are \"interpretable.\"   This is not a common view.  Maybe decision trees are (which are the base unit of random forests), but random forests then blend them together in a complicated way, so that the effect of individual components is lost in the mix.  They are an important learning model nonetheless, so I would just remove this justification.  \n\n - The experiments are entirely from synthetic data.  Sometimes the method of generating synthetic data has quirks that is not reflected in real data.  It would be useful to run a few simple experiments to see if some of the main results are mirrored on some similarly scale real data sets -- in particular in how the accuracy is affected.  \n\n\n\n### Overall:\nI liked the paper and found it interesting and useful to read.  It provides a nice review, and measured evaluation.  It leaves open some big issues that could improve the paper and its impact (see **Weaknesses**) but is already a self-contained paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper does not make strong claims, it presents a review.  I think the review is thorough and balanced.", "audience_yn": "Yes", "audience_explanation": "I found reading the results interesting.  Generic random forests can scale fairly well via parallelism, but splitting data or a lack of strong global optimization can lead to drastic drops in accuracy.  This was demonstrated believably in the experimental results.", "requested_changes": "The first and third weaknesses should definitely be addressed in the writing.  \n\nThe second weakness (on XGBoost, and relatives) is something that could be shown as a baseline, and I would find it useful.  If it does not show much improvement over the synthetic data, it may indicate that the data inputs used are not very interesting.  \nI expect scaling XGBoost would require more advanced techniques than this paper explores to understand how it could be scaled effectively.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)