RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting
Authors: Hsiang Hsu
Ivan Brugere
Shubham Sharma
Freddy Lecue
Chun-Fu Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Rashomon effects; predictive multiplicity; gradient boosting; tabular data
Primary area: safety_in_machine_learning
TL;DR: Theoretical and empirical analysis on the Rashomon effect and predictive multiplicity of gradient boosting, with a new methodology to audit and reduce predictive multiplicity.

Abstract: The Rashomon effect is a mixed blessing in responsible machine learning. It enhances the prospects of finding models that perform well in accuracy while adhering to ethical standards, such as fairness or interpretability. Conversely, it poses a risk to the credibility of machine decisions through predictive multiplicity. While recent studies have explored the Rashomon effect across various machine learning algorithms, its impact on gradient boosting---an algorithm widely applied to tabular datasets---remains unclear. This paper addresses this gap by systematically analyzing the Rashomon effect and predictive multiplicity in gradient boosting algorithms. We provide rigorous theoretical derivations to examine the Rashomon effect in the context of gradient boosting and offer an information-theoretic characterization of the Rashomon set. Additionally, we introduce a novel inference technique called RashomonGB to efficiently inspect the Rashomon effect in practice. On more than 20 datasets, our empirical results show that RashomonGB outperforms existing baselines in terms of improving the estimation of predictive multiplicity metrics and model selection with group fairness constraints. Lastly, we propose a framework to mitigate predictive multiplicity in gradient boosting and empirically demonstrate its effectiveness.

RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting
Hsiang Hsu Ivan Brugere', Shubham Sharma², Freddy Lecue', and Chun-Fu Chen *JPMorganChase Global Technology Applied Research
2 MJPMorganChase Al Research (hsiang. hau, ivan brugere, shubham. x2. sharmam @ pmchase. com (freddy. lecue, richard. cf. chen @jpmchase.. com
Abstract
The Rashomon effect is a mixed blessing in responsible machine learning. It enhances the prospects of finding models that perform well in accuracy while ad- hering to ethical standards, such as fairness or interpretability. Conversely, it poses a risk to the credibility of machine decisions through predictive multiplicity. While recent studies have explored the Rashomon effect across various machine learning algorithms, its impact on gradient boosting an algorithm widely applied to tabular datasets-remaing unclear. This paper addresses this gap by systematically analyz- ing the Rashomon effect and predictive multiplicity in gradient boosting algorithms. We provide rigorous theoretical derivations to examine the Rashomon effect in the context of gradient boosting and offer the Rashomon set. Additionally, we introduce novel inference technique called RashomonGB to efficiently inspect the Rashomon effect in practice. On more than 20 datasets, our empirical results show that RashomonGB outperforms existing baselines in terms of improving the estimation of predictive multiplicity metrics and model selection with group fairness constraints. Lastly, we propose a framework to mitigate predictive multiplicity in gradient boosting and empirically demonstrate its effectiveness.
cetemmationnthhhoeee characterization
Introduction
Large-scale, complex data and the pursuit of superior performance in machine learning (ML) models have led to increased complexity in both the models themselves and the training algorithms [44]. As a result. is more likely to find a plethora of distinct models, such as those found local minima, that exhibit statistically indistinguishable performance (e.g., test accuracy) [23]. This phenomenon, known the Rashomon effect [12], has urged researchers reconsider its impact on ML models when deployed in real-world scenarios [22. 10, 30, 26].
The impacts of the Rashomon effect reveals two sides of the coin in responsible ML. On one hand, it benefits the current trend of developing algorithms that prioritize responsible ML principles beyond merely optimizing for accuracy. These principles often include interpretability [64], causality [37], group fairness [20], counterfactual explanations [39], and feature interactions [51]. The abundance of models with competing performance allows compliance with these principles without significant compromises in performance. For instance, algorithmic fairness often faces trade-off with accuracy; the Rashomon effect allows fairness intervention algorithm to identify more optimal balance among models with statistically similar performance [20]. On the other hand, the Rashomon effect presents a risk the credibility machine decisions known as predictive multiplicity [56]. where competing models, generated by simply varying randomness' in the training processes, yield
Such randomness includes difforent seeds, weight initialization, splits of mini-batches, etc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

conflicting predictions for some individual samples. If left unaddressed, conflicting predictions can lead to discrimination and unfaimesss hidden under the guise of algorithmic randomness. This can adversely affect certain individuals without revealing significant statistical differences from non- discriminatory models [22]. The negative societal impacts of predietive multiplicity and inconsistent decisions have been recently studied under various frameworks' such as prediction uncertainty [35, 2], predictive churn [59, 75J, and predictive multiplicity [56, 67, 9, 42].
To understand the Rashomon effect and address predictive multiplicity, recent studies have focused on characterizing competing models and efficiently searching for them across different ML models. For instance, predictive multiplicity caused by linear classifiers can be computed via mixed integer programming [56, 74]. Competing models from sparse decision trees are exactly characterized and searched by sue-tree pruning [76]. In special cases such as ridge regression and generalized additive models, the forms of the set of competing models are analytically derived [67, 16). More recently, test-time dropout has been utilized to search for competing models for neural networks [43].
In this paper, we focus on another ML algorithm, gradient boosting [65], which is widely applied to tabular datasets [36]. Gradient boosting differs fundamentally from other ML algorithms in its sequential approach: rather than training a model single entity, gradient boosting breaks down the training process into a sequence of sub-learning problems. This sequential training pipeline not only facilitates the analysis of the Rashomon effect but also offers new methodologies for model selection and reducing predictive multiplicity. To the best of our knowledge, our work is the first to explore the Rashomon effect for gradient boosting. The main contributions of this work include:
1. We formalize and investigate the Rashomon effect induced by gradient boosting, employing statistical learning and information theory to analyze its behavior (Section 3). Specifically, we leverage coormatiin-thhoretie measures to characterize the impact of dataset properties on the Rashomon effect.
2. We introduce RashomonGB, an efficient method that explores exponential search space versus baseline methods which search linearly. (Section 3.1).
3. We implement RashomonGB on several real-world (large-scale) tabular and image datasets, and empirically demonstrate the competing models obtained with RashomonGB can greatly improve the estimation of predictive multiplicity (Section 4.1), and model selection with additional responsible ML principles such as group faimeess (Section 4.2).
4. We propose two methods to mitigate predictive multiplicity for gradient boosting and experimentally validate the methods on 18 tabular datasets (Section 4.3).
Onitted proofs, additional explanations and discussions, details on experiment setups and training, and additional experiments are included in the Appendix. Code to reproduce our experiments can be accessed at https: : / //gtthub.. com/ htt///jooganhhaee wwwaaaaaaaeaaaaooonn...o.ooo RRashooon gradient- -oostttoo..on.....o
Background and related work
Consider dataset S = (s:) drawn i.i.d. from Ps. where each s; is a pair (x;, It) consisting of feature vector x, = [x11. xia]" E X C Ied and a target y: E R. Let X and Y be the random variables for the feature x; and target y: respectively, and S = x Y. We denote by
hypothesis space of functions that map from X' to y. The loss function used to evaluate model performance is denoted by E : H. x S + R+ and Lps (h) & Eps [(h,, S)] the population risk. As usual. the population risk is approximated by the empirical risk (s(h) - E s(h, We denote the empirical risk minimizer as h² argminge Ls(h) E H. We denote V Vze(z) the gradient of v(x) w.r.t. x, and [: the indicator function. For two random variables X and Y. the mutual information between them is defined as I(X; Y)- = (Kx, P| Py) [21], where (PIIQ) Ep [log P/Q] is the Kullback-Leeibe Leibler (KI. divergence [47]. Finally, we let [k] (1.... R...
The Rashomon sets and exploring models therein. The studies on the Rashomon effect typically start with searching for models in the Rashomon set [67], the set of all models in the hypothesis space
Both prediction uncertainty and predictive multiplicity consider the arbitrariness of machine outputs, with predictive multiplicity specifically addressing models with competing performance. Predictive churn, on the other hand, focuses on the instability of decisions before and after updating models with new data.

---

H. whose population risks are comparable to that of a given model' h² E H. i.e.
R,H, .c) é (h E H: Lp (h) S LP, (h²) +
(I)
where c 2 Rashomon parameter that determines the size of the Rashomon set. However, when the hypothesis space H. is large (e.g., neural network architectures, tree ensembles, etc.), exhaustively identifying all models within the Rashomon set becomes computationally infeasible. Therefore, it is customary to approximate the full Rashomon set by a subset with m models called an RR"(H, empirical Rashomon set, R. (H, S. h²,e) - [hI,,... hm H: h2 E RRH, S, s,,",,,,,, ,)); Vi E [m]]. In practice, the m models in the empirical Rashomon set are mainly obtained by re-training (See Appendix B.2 for more discussions). The re- re-training strategy re-trains models with different random initializations and rejects those that disobey the loss deviation constraint in Eq. (1) until m models are collected [67, 48]. However, re-training models repeatedly is time-consuming with large datasets or complex architectures. To improve efficiency, a recent strategy involves collecting distinct models with competing performance during the inference phase. For example, Hsu et al. [43] proposed an inference-tive time dropout strategy for convolutional neural networks (CNNs). Despite these advances, exploring Rashomon sets for ensemble learning methods like boosting remains unexplored.
Predictive multiplicity. Predictive multiplicity undermines the credibility of decisions made by ML algorithms, making its measurement an active research area. Predictive multiplicity metrics can be categorized based on whether they are defined on output decisions (i.e., thresholded predictions/scones after argmax) or on output scores in the probability simplex. For example, ambiguity and discrepancy, measure the proportion of samples with conflicting decisions from models within the Rashomon set [56]. Disagreement, in a similar flavor, assess the probability of conflicting decisions per sample [48]. In contrast, bosee metrics estimate various aspects such as score variance/std. [52, 17, 9], the viable range of scores (termed viable prediction range (VPR) by Watson-Daniels et al. [74])), or score spread in the probability simplex (referred to as Rashomon Capacity (RC) by Hsu and Calmon [42]). Due to space constraints, we defer the mathematical formulations of these predictive multiplicity metrics and discussions estimating these metries with the empirical Rashomon sets to Appendix B.2. These predictive multiplicity metrics are often estimated using the empirical Rashomon set R' (H,, S, h", ,e). As m increases, the empirical Rashomon set better approximates the Rashomon set, leading to more precise estimations of predictive multiplicity metrics.
Mitigating predictive multiplicity. Mitigating predictive multiplicity ensures that decisions made by ML algorithms are consistent. The main strategy for this is to combine decisions from competing models. Roth et al. [63] reconcile conflicting decisions from two different models in the Rashomon set to improve the disagreement in predictions. Combining decisions from multiple models falls under the umbrella of model averaging in ensemble learning [9, 42, 52]. Model averaging is a special ensemble learning that collects multiple base models, often referred to weak learners, and combines them in parallel. As averaging model outputs reduces the variance, it is a natural choice for diminishing predictive multiplicity and has been reported in several studies. For instance, Black et al. [9] proposed a selective averaging that leverages certifiably-robust predictions to mitigate the problem of inconsistency with a probabilistic guarantee. Hsu and Calmon [42, Section A.4.5] observe that random forest classifiers exhibit a lower Rashomon Capacity compared to decision tree classifiers. Furthermore, Long et al. [52] demonstrate that the probability of significant deviated predictions in model averaging diminishes exponentially with the number of models in the average.
Ensemble learning is not limited parallel combinations. Another popular branch involves com- bining models sequentially, known as boosting algorithms. Despite the widespread use of boosting algorithms and their superior performance over neural networks tabular datasets [36], boosting algorithms have been mostly overlooked in the literature on the Rashomon effect and predictive mul- tiplicity. This paper aims address this gap, outlined in the next section. The works most closely related to ours involve prediction uncertainty estimation in gradient boosting, such as NGBoost (27, 55), PGBM [70], and IBUG [13], which consider probabilistic predictions from regression trees in Bayesian settings. However, these studies do not frame the concept of arbitrariness in predictions within the context of the Rashomon effect and may overestimate arbitrariness with models that do not have similar performance. To the best of our knowledge. this work is the first that investigates the impact of the Rashomon effect and predictive multiplicity on gradient boosting.
It is common to choose the model as an empirical risk minimizer.
3

---

3 Analyzing the Rashomon effect in gradient boosting
We begin with brief introduction to gradient boosting and discuss how its sequential combination approach can inspire new method for finding competing models in the Rashomon set. We then provide a high-probability bound on the Rashomon set using information theory, making the first connection between the size of the Rashomon set and data quality as measured by mutual information. The proofs of the propositions in this section are included in Appendix A.
Boosting algorithms select a sequence of weak learners", ho. ... ,h E H. such that the additive expansion fx) = Ei= (x) E F with a > 0 minimizes the empirical risk Ls(fr) [34, 57, 66, 19]. Different choices of H. and the method for selecting h2 E H have led various boosting algorithms (see Appendix B.1). Here, our focus lies on gradient boosting [66, 34], which starts with a constant model hu(x)= = argmin Ex=1 (hho, st), and iteratively extends the model fi (x;) with
fx(x) - Ix- + argmin| MI
LLs(si1) hell2 - fx-1(x)= + argmin NE
[1bx(xx) - -rkl
(2)
where Tri 1 is the pseudo-residual of sample from weaker learner ht- Indeed, for
35--1
regression problem with a mean squared error (MSE) loss function, hig > y:/n and Tri 2(y (x:)) part of the y/ that cannot be explained by the current model fx=((x;), (x:), For a binary classification problem with cross-entropy (CE) loss, hi log Σx= [ 1 Σ [yi Oj. and can be understood as regression on the log- li-liieiihood.. We set a = in this section for the sake of theoretical analysis. For the convergence and consistency analysis of gradient boosting, see, e.g.. Zhang and Yu [79] and Telgarsky [71].
3.1 Building Rashomon sets for gradient boosting
The iterative training procedure of gradient boosting in Eq. (2) allows us to convert the original learning problem minjer Lf( into sequence of learning problems ming,e Ls, (ht), for e [1. 21, .T) where S = {(;,, r=) In other words, the weak learner h2 aims to fit the pseudo- residuals in each iteration, and inducing a residual Rashomon set (ef. Figure 1),
R₂ St, & {he E Hi LP, (he) LPs, (h²) +e).
where hū is any given model such as the empirical risk minimizer in each iteration. Eq. (3) suggests an alternative to building the entire Rashomon set R(F,S, S, Te) by iteratively building the residual Rashomon sets for each iteration, i.c.
(4) Eq. (4) comes from the fact that, for classification tasks, gradient boosting actually performs a regression on the log- likelihoddood using the MSE loss. The pseudo-residual of the MSE loss exhibits linear relationship between the prediction and the output of each iteration, allows us to aggregate the losses across iterations. Equation (4) will be clarified further in Proposition 2.
R1 (H, S1, hj,c) x ..... x S,, x... x Ry RT, 17. R(F,S,
In practice, we perform mn re-tranning in each iteration, and obtain the empirical residual Rashomon set R.T (H, S,, hi, c), the model (x) can be expressed as fr(x) = ha(x) + E h (x).. th, E RT (H, Se, e).. Since in each of the T iterations, there are m candidate models in R,"" (H, St, hif e), there are a total of m" possibilities for fx(x) E R,F,S, / Te) (see Ap- pendix B.3 for visualization). We term the process of building the empirical residual Rashomon sets for the entire Rashomon set R(F, S. Te) as Rashomon gradient boosting or RashomonGE in short. RashomonGB can be straightforwardly implemented by training m models (i.c., weak Icarners) at each iteration that meet the loss constraints defined by the Rashomon set. This is fundamentally different from simply performing m re-training of gradient boosting to solve the original learning problem min fer Ls(f f), since the m models at each iteration obtained by RashomonGB share the same residual from the previous iteration. During the inference phase RashomonGB carries a sig- nificant benefit over simply performing m re- ne-training of gradient boosting botthnnggt-hhhhhoooo re- ve-training of gradient boosting and RashomonGB requires m x T training when considering all the iterations.
"The weak learners in gradient boosting are usually decision tree regressors. However, the choice of the weak learners can be generalized to more complex families of functions [49] such as neural networks with [25] or without regularizations [6, GrowNet], and Gaussian processes [69].

---

However, RashomonGB, using a different way to output the predictions at the inference phase, yields exponentially many more models with the same training cost. So far, we have informally introduced RashomonCB. In the rest of this section, we provide rigorous analysis to show the effectiveness of RashomonGB by characterizing the Rashomon sets of gradient boosting (or more generally, the Rashomon sets of the iterative training procedure). The analysis collectively ensure that RashomonGB is not only practical but also robust, enhancing its applicability in various real-world scenarios.
Gradient Booeting
Reshoeen6B
hj
12 &t,
ht"
Re S.
3.2 Information- characterization of the Rashomon set
Characterizing the entire Rashomon set, given a learning problem and dataset, has posed as significant computational challenge (refer to Sec- tion 2). Specifically, for gradient boosting, delineating the Rashomon set R(F S, frr e) entails identifying all models fr E FF such that LPs (fr) < LP (fT) + e for given model f7. However, Eq. (4) presents a novel approach to approximating the Rashomon set by decomposing it into residual Rashomon sets concerning the weak learners -tis method offers valuable insights into analyzing the Rashomon effect in gradient boosting.
hf
G000 Σ $500 (T.
Figure 1: Gradient boost- ing and RashomonGB with T iterations.
Existing literature explores how various hypothesis spaces influence the Rashomon set but frequently neglects the impact of datasets, despite the Rashomon set being inherently tied to the dataset itself (cf. Eq. (1)). In this context, we derive novel bound on the size of the Rashomon set by expanding the scope of the Rashomon set beyond mere consideration of hypothesis spaces, incorporating the influence of datasets through the lens of statistical learning and information theory. Particularly in the analysis of ihforeition-theoretiic generalization error bounds [77], learning algorithm A(S) conceptualized as a random variable H that generates models within the hypothesis space H. Moreover, the loss function E(h, S) is also treated as a random variable, and is further assumed to be suub-Gaussian" [73], property that effectively generalizes the boundedness assumption of the loss function in information-theorrtiia analysis. Sub-Gaussianity is practical property since for a bounded loss function / E {a. b). which can be readily satisfied by clipping the loss", l(h, S) is guaranteed to be (b a)/2-sube 22-sub-Gaussian With these properties in place, the mutual information [S; H) between H and the random variable of dataset S emerges as a pivotal metric for assessing generalizability, By establishing a connection between generalization error bounds and the definition of a Rashomon set in Eq. (1), and leveraging the properties of a aub-Gaussian loss, we derive the following high- ceob-prbbability bound for Rashomon set.
Proposition 1. For a dataset S. given an empirical risk minimizer h" = argmingen (s) and a o-Sub- Guub-GGussiin loss E, with probability least 1 p. we have
802
H) P
4
2 In
hER R
H,S, h2,
In
+
(5)
P
In contrast to existing analyses of the Rashomon set primarily focused on optimization perspectives, Proposition I offers a characterization of the size of the Rashomon set in terms of the controllable probability of the Rashomon parameter 6. This sheds light on understanding the Rashomon set from statistical learning standpoint. Furthermore, the mutual information [(S; H) in Eq. (5) quantifies the uncertainty of the learning algorithm with respect to the dataset S. In essence, [(S); H) serves as metric for the multiplicity of models, thereby contributing to predictive multiplicity. By chain rules, we can decompose the mutual information into two components, each delincating a distinct source responsible for inducing multiplicity, i.e.,
T(S: H) T(Y, X;
T(X: H
Model Uncertainty
Quality Data
When 1(X; H) is small, the model derived from the learning algorithm becomes nearly deterministic, indicating minimal uncertainty regarding model selection. Similarly, (XX) can be expressed
'A random variable U is sub-Gaussian if log
AAU AAU-BU>]
< X for all À e R.
'The MSE loss is also qualified as sub- Gaussian loss function.
5

---

as = = - where 9 denotes the (conditional) Shannon's 'sso entropy [21]. The first component, g(Y evaluates the data quality; if the channel from X to Y is devoid of noise, the conditional entropy will be low. Conversely, if the channel is noisy, the conditional entropy will be lower-bounded by the entropy of the noise source.
The mutual information / (S; H) indeed contains intricate details about the dataset, allowing us to incorporate the dataset's influence into depicting a Rashomon set. This is precisely why we opt for information-thcoretii tools over Rademacher complexity [68]. While Rademacher complexity offers detta-deeendent" generalization bounds, it lacks the ability to provide substantial insights into the dataset itself. For completeness, we include a theoretical analysis of the size of Rashomon sets based on Rademacher complexity, akin to the high probability bound in Proposition 1, in Appendix C.
Now we move back the characterizing the Rashomon set for RashomonGB. Recall that for each iteration t (7) in gradient boosting, we may define a residual Rashomon set S.. By plugging S into S in Proposition 1, and using the decomposition of the entire Rashomon set RFF,S, fr. f Te) in Eq. (4), we are now equipped re-evaluate the entire Rashomon set of gradient boosting.
Proposition 2. Let hy be the empirical risk minimizer for each boosting iteration St, and f7 Ei hi, then for a a a-sub--Gautd Gaussian loss l, with probability at least p,, we have
T
802
(2I H) + In 1 )
fr ER
F. F.S.
FL
7
n
P
Proposition 2 suggests that the Rashomon set grows with the boosting iterations- it is due to the increased complexity of the overall model ft. Unless the number of samples n goes to infinity, the Rashomon set has non-zero size.
3.3 The Rashomon effect in each iteration of gradient boosting
The information-theorede analysis presented in the previous section underscores the importance of the mutual information (IX) in determining the size of the Rashomon set and hence the severity of predictive multiplicity. However, existing literature lacks an in-depth analysis of the information contained within the pseudo-residuals for gradient boosting algorithms. Let Re represents the random variable associated with the residuals {ru: J As ( grows, fitting Rt with X becomes progressively challenging, akin to the gradient vanishing problem observed in deep learning [31, 40), thereby resulting in larger conditional mutual information with data X. We formalize this observation in the following proposition.
Proposition 3. For both the MSE and CE losses, the mutual information between and the pseudo- residuals conditioned on the features X is non-decreasing with respect to the boosting iteration, iee, let 0 S t S t2 I T, then [H;; Rr, [X I [H;; Re, (X).
The essence of Proposition lies in the fact that the information contained in Re, be understood as a combination of the information in R2, and the additional information provided by the models fitted to the residuals between iterations t and l2. Proposition hint at a counter-intuitive observation: the size of the residual Rashomon set could potentially increase with more boosting iterations, i.e., for 0 < 6 S l2 S T, we have Et_ I Etz and hence RR (H. Sr, hi <t,) C R. Stg, (t2) This implies that conducting additional boosting iterations? may not only exacerbate over-fitting but also result in larger Rashomon set and heightened predictive multiplicity. Note that we do not start with the assumption that e increases with each iteration; rather, this conclusion emerges from Proposition 3. With constant p~as defined in Proposition 1- additional iterations result in increased conditional mutual information, which in turn necessitates a larger e.
Proposition I, and Eq. (6) collectively
Figure provides a simulation with a 20-dimensional Gaussian synthetic datasets with 100 samples, trained with gradient boosting for 10 iterations, where each iteration contains m 100 models. Here, we show the conditional entropy g R& X) instead of the mutual information T (R, X; H) in
AA similar phenomenon where additional information impacts the performance of boosting algorithms, has also been noted in previous studies such as Friedman et al. [33] and Long and Servedio [53]. Esttmated with local linear regression [18], which has consistently smaller MSE for small datasets.

---

Eq. (6) as the estimation of the mutual information is in general hard task [60]. It is clear that the conditional entropy- and consequently the mutual information inneeases as the boosting procedure iterates, leading to larger Rashomon set. This occurs because the Rashomon effect accumulates across the sequential learning problems addressed each iteration, highlighting the cumulative impact on diversity within the model space. The Ablation study in Appendix E.4 further clarifies the selection of e through its iterations. Figure E.9 demonstrates that fixing e while re- re-training with different random seeds results in a decreasing percentage p of models in the Rashomon set. This implies that to maintain a consistent p. the chosen Rashomon parameter e must increase.
As a remark, consider training m models per iteration, and construct the overall Rashomon sets using models obtained from the Ti-th and T2-th iterations with Ti < Tz With the same threshold e for the Rashomon set, let f1 and P2 represent the probabilities for iterations Ti and T2, respectively, then by Proposition 3, we have
Loss
0.8
Accuracy
Loss
Acc.
Top 10% VPR
1.0
f1 - P2- the number of models from the Ti-th iteration that are included in the Rashomon set with threshold e will be mi x (1 P1). Similarly, for the Tz-th iteration, the count will be m= x (1 p2). It is important to note that although p decreases, this reduction is linear with respect to the number of iterations (as suggested by the term - Tp in Proposition 2). However, the number of models generated by RashomonGB grows exponentially with the number of iterations. Thus, the total number models in the Rashomon set will still asymptotically increase with the number of iterations.
0.5
giR:120
0.0
1
3
9.
Boosting Iterationn
Figure 2: Gradient boosting for binary classification. The conditional entropy of the residuals and the predictive mul- tiplicity (measured by VPR) increase along with the boosting iteration, which matches Proposition 1.
Applications of Rashomon gradient boosting
We present three use cases demonstrating how RashomonGB can be deployed in practice to explore models more effectively in the Rashomon sets: (i) improving the estimation of predictive multiplicity metrics, (ii) fair model selection, and (iii) mitigating predictive multiplicity. Given that there are no existing algorithms specifically designed to explore the Rashomon set for gradient boosting, apart from re-training with different seeds [67, 48], we use the re-training strategy as our primary baseline for comparison through this section. It's noteworthy that both the re-training strategy and RashomonGB share the same training complexity. However, during the inference phase, RashomonGB exponentially expands the empirical Rashomon set (i.e., with a large m following Section 3.1). In our experiments, we adopt the settings in Friedman [34] with decision tree regressors as weak learners for tabular datasets and CNNs for images. See Appendix D.I for detailed descriptions and pre-processing of the datasets, Appendix D.2 for detailed training setups, and Appendix E. for additional experiments including ablation studies.
4.1 Improving the estimation of predictive multiplicity metrics
We estimate the predictive multiplicity metrics (cf. Appendix B.2) on three tabular datasets with binary classes. Two of these datasets are from the financial domain (ACS Income [24] and Credit Card (78)), while the other is from the medical domain (Contraception(5)). The datasets are particularly chosen as predictive multiplicity in these domains could have profound implications for fairness and justice. Note that the ACS Income dataset is an extension of the widely-used UCI Adult dataset [5] with many more samples (= million vs. <50k in the UCI Adult), allowing us to compare methods with higher precision. Beyond binary classification and decision-tree weak learners, use CIFAR- 10 [46] as a multi-class case study with CNNs as weak learner, i.c., a setting similar to Badirli et al. [6. GrowNet].
Figure 3 summarizes the estimation of 4 predictive multiplicity metries using the empirical Rashomon sets obtained from re- re-trarttt trrininn and RashomonGB. For a more detailed explanation on how to interpret Figure 3, see Appendix E. We conduct (and 50 for the CIFAR- 10 dataset) re-training of gradient boosting with different random seeds; each gradient boosting has T = 10 (and T = 6 for CIFAR-10) 10) iterations and m = 10 (and m = 50 for CIFAR-10) models in each iteration. We randomly select 2 out of m models in each boosting iteration and perform RashomonGB to obtain
7

---

Re- Re-training
RachomenGB
0.00
ACS
0.50
CE
X
CF lore
0.50
0.45
78
0.80
0,020.00
0.25
0.50
0.75
1.0
12
0.0
0.5
1.0
0.0
0.1
0.2
Contracaption contracaption
0.03
Love
CE
0.55
0.65
0.0
1.0
0.0
CIPAR 10
2.50
0.78
0.80 Accuracy
0.82
0.00 0.02
0.64 0.06
1.0
1.1
1.20.0
0.5
1.0
0.0
0.1
0.2
VPR
Reshomon Capacity
Disagreement
Discrepancy
Figure 3: Re-Ree--riie Re-training vs. RashomonGB in exploring the Rashomon set for predictive multiplicity metrics estimation. In the leftmost column, cach marker represents a model. The rightmost 4 figures in row share the same y-axis for the loss difference (values shown at the right), i.c. LPg (h") ] e in Eq. (1). Higher predictive multiplicity values mean a better estimate. RashomonGB, with more models the Rashomon set, offers more accurate multiplicity estimates under the same loss deviation constraints.
2 1024 (and 972 models for CIFAR-10) models The leftmost column shows the CE loss vs. accuracy for the models obtained by the two methods. It is clear that with the same training cost, RashomonGB offers many more models in the Rashomon set that spread wider in the loss-accuracy plane. The rest of the four columns show the estimates of predictive multiplicity metrics given different loss deviation constraints. Since VPR, Rashomon capacity, and disagreement are defined per sample, we plot the mean and std. of the top 10% samples instead
As observed Figure 3, RashomonGB consistently outperforms re- te-training in the ACS Income and Credit Card datasets for all predictive multiplicity metrics. The RashomonGB has more advantages especially when re-training is incapable of exploring enough models under the loss deviation constraint. For example, in the Contraception dataset, when the loss constraint is under 0.585, there only one model obtained from re- re-training, and therefore the corresponding values of predictive multiplicity metrics remain O. As the loss constraint increases, the Rashomon set from re-training has more models and its estimates converge with that of RashomonGB. The explanations above indicate that the exploration of diverse models under the same loss constraint largely affects the estimation of predictive multiplicity metrics. For the CIFAR- 10 dataset, both VPR and RC values are small, while seciision-baeedd metrics, such disagreement and discrepancy, are large. This implies that despite small score variations, a significant number of samples have scores close to the decision boundary. Consequently, a slight perturbation in scores from a different model in the Rashomon set could lead to a different class after applying argmax.
We include additional results on three other UCI datasets in Appendix E.2 and comparison of the computational time to obtained one model from re- ve-training and from RashomonGB in Appendix E.3. The ablation studies different types of weak learners H (e.g.. linear regression), depths of decision tree weak learners, the number of boosting iteration T and the number of model in each iteration m are also include in Appendix E.5 to E.8. To elucidate the distinctions between predictive multiplicity and prediction uncertainty estimation in gradient boosting (cf. Section 2), we have compared our re- re-training strategy, RashomonGB against NGBoost [27, 55], PGBM [70], and IBUG [13] with the UCI Contraception dataset in Appendix E.9.
For 3 models per iteration, RashomonGB produces 310 xe 59k models, exceeding our storage limit. Prrddictivee multiplicity occurs only small portion of samples. The choice of 10% is data-dependent. For the ACSIncome dataset, the inference time per model is 0.4 seconds for re- training, compared to just 0.02 seconds for RashomonGB. This indicates that, with the same training cost (54.67 seconds). RashomonGB is 20 times more efficient in generating models from the Rashomon set than the traininni strategy.
8

---

4.2 Selecting models with group fairness constraints
We conduct re-training and RashomonGB on two standard datasets in the algorithmic fairness community, the UCI Adult and COMPAS recidivism datasets [3], with the goal of selecting model that exhibits better fairuacy trade-offs. We follow the same training procedure and gradient boosting architecture in Section 4.1. We assess the bias across different groups (c.g.. female vs. male) by two group-fairness fairness m mean equalized odds (MEO) [38] and statistical parity (SP) [29] (cf. Appendix B.4 for details). In both datasets, the group attributes are the binary "race" label. Figure 4 illustrates that with RashomonGB, a practitioner has a greater chance to select model that better satisfies group-fairness constraints without significant drop in accuracy. For the UCI Adult dataset, despite that re- re-training achieves the highest accuracy (ss 84.5%), it violates the MEO with 0.01. On the other hand, RashomonGB provides model that perfectly complies with fairness constraint (MEO ~ 0) with a drop in accuracy less than 1%. Similar observations apply to the COM- PAS dataset regarding SP. Moreover, we include four additional faimess intervention baselines, EqOdds [38], Rejection [45], Reduction [1], and FaiRS [20]. See Appendix B.4 for a brief in- troduction of the fairess intervention baselines. The most relevant baseline to RashomonGB FaiRS, which modifies the Reduction ap- proach to address fairness intervention problems across the models in the Rashomon set. FaiRS is originally implemented for logistic regression, and we adapt it to gradient boosting.
UCI
COMPAS Recidivism
0.84
0.66
Accuracy
Accure
0.84
0.02
0.62
0.80
0.50
0.00
0.07
0.01
0.08
0.00
0.2
0.4
0.6
Keen Equalliee Odds
Statistical Parily
Re-tmining
Redicction
Rejection
RanhomenGi
PARS
Fopasss
Figure 4: Fairness Fairness-accuracy off for re- treaninnn vs. RashomonGB on test set. Each marker represents a model. A hetter trade-off means smaller group-fairess metric (MEO or SP) and higher accuracy, i.c., the top left area. For both datasets, RashomonGB provides more better models to select that complies with the fairness constraints whilst having the highest accuracy. For UCI Adult, the CE loss of the models from RashomonGB is 0.38 0.02 and 0.64 0.02 for COMPAS.
For both UCI Adult and COMPAS datasets, RashomonGB encompasses nearly all models (excluding explicit fairness cervvention-iniccale
EqOdds) from the fairness intervention that RashomonGB gives a "rich" Rashomon set for model selection with additional faimess consider- ations. The advantage RashomonGB becomes even more pronounced when dealing with larger datasets. In such scenarios, re-training and re-training-baser fairness intervention algorithms, such as Reduction (and FaiRS) and Rejection, may incur significantly higher training costs. Model selection with RashomonGB is not limited the group fairness purposes; this experiment serves as an initial demonstration of the diversity of RashomonGB use cases. In Section 4.3 we demonstrate explicit feedback in the model selection process.
4.3 Mitigating predictive multiplicity
The RashomonGB framework, which involves training m distinct models in each iteration, offers the added benefit of reducing predictive multiplicity in gradient boosting. The m models in the empirical Rashomon set for RD (H. St. h2 <) for the tth iteration can either be selected (based on the least losses) or aggregated (similarly to model averaging in Section 2). Building on these concepts, we propose two approaches to reduce predictive multiplicity in gradient boosting: (i) model selection with reweighted loss (MS), and (ii) intermediate ensembles during boosting iterations (IE).
Let lg = {(hj,ss) 8;) be the loss evaluated at sample s; for model hy E RTH Sr, hū,e), e), and let 7, - Σリ fi,j be the mean loss. The MS method considers the reweighted loss for each model in
the empirical Rashomon set using L,, A ((,ss//)))))))) and selects the top l models with the smallest b where & S m. MS simplifies to re- re-training at À O. The model with the smallest ly is used to compute the residuals for the next iteration, and we return the top i models at the last boosting iteration. The intuition of this reweighting is that the loss contribution of sample s; is rewarded at an exponential scaling factor 2 0 when the model hú produces lower loss than the average for s; over all models in R (H, St, ee). In contrast, the IE method constructs U ensembles he E [U] in each iteration, where each ensemble consists of E randomly selected
12 MEO and SP respectively quantifies the discrepancy in the sum of True Positive Rate and False Positive Rate, and in the probability of the model outputting class L.

---

0.02
0.00
plleptc-seezuur
Wine
0.01
crditcard
dropour 7
talvon credit
creditcard
heart-clevelard
sccuracy
0.00
approcol
heart-dleveladd
A accuracy
0.00
sushrosm dew.nddoodddddrgg
Mer bands
mnmmmo
syfinder bands
gomaa--credt
-0.01
mushroomm secondary
0.01
german-cruff
iinequual
ontracc
-0.02
-0.02
-0.0
-0003
0.00
0.05
0.30
0.15
0.20
0.25
0.30
0.00
0.05
0.30
0.35
0.20
0.25
0.30
A O-disagreement
A 0-disagreement
Figure 5: reetrranning vs. MS with À (left) and IE with E 20 (right) to mitigate predictive multiplicity on 18 UCI datasets. Each point is averaged over 20 random train-test splits (std. omitted for clarity). Dashed lines are the mean of each axis. Higher values are better for both axes.
models from R (H, S hz.c). The model hy constructed by an additive weighted sum of the outputs of the E models, i.e., hx,))) = E Ee wehe(x:), where the weights We = (1/En.)ww and v is the harmonic mean of all losses {en, Te=1-
For evaluation, we extend the sample-wise metric disagreement ((s;) in Kulynych et al. [48] (cf. Ap- pendix B.2) to consider the disagreement across all samples in the whole dataset. We call this new metric p-dixagreement1' defined as d(S. p) A Σn [u(s;) < p]. In Figure 5 we report the reduction of predictive multiplicity (A0-disagreement) and improvement of accuracy (Aaccuracy) vs. a rr-trra te-training baseline, on 18 UCI tabular datasets [5]. In our experiments, U = m = 100, k 25, and E 20. For fair comparison, MS and IE share the same training procedure as re ve-training in all iterations we select the top k = 25 from m = 100 models.
We observe that IE outperforms MS in disagreement reduction while both methods yield similar accuracy to re-rrerrrreeee to-triininn.. However, IE has the cost of increasing the overall model complexity by factor of E 20, which may be undesirable for interpretability or auditing. It is noteworthy that a small Aaccuracy could lead to a great reduction in disagreement. For example, in "epileptic- seizure", re-training has O-disagreement of 0.208 and IE reduces it to 0.041 with only slight improvement of 0.014 + 0.002 in accuracy. For ablation studies on the hyperparameters E, and and more explanations, see Appendix 1.
5 Discussion
Here we reflect on the limitations and highlight interesting avenues for future work.
Limitations. While re-training with different random seeds offers a "global" exploration of models within the Rashomon set, RashomonGB conducts a "local" exploration. Therefore, the effectiveness of RashomonGB depends on selecting diverse models in each iteration; similar models reduce its efficiency in exploring Rashomon sets. However, RashomonGB demonstrates greater efficiency than the re- re-training strategy, highlighting trade-off between the efficiency of exploring the Rashomon set and the effectiveness of capturing model diversity. Finding the soub-)optimal strategy for exploring the Rashomon set remains an active area of research. Our theoretical analysis, developed for gradient boosting, needs an extension to other algorithms like adaptive boosting to validate its applicability. Additionally, the large number of models generated by RashomonGB poses storage challenges, requiring new data structures for efficient management.
Future directions. First, our analysis in Scction 3 links the size of the Rashomon set to dataset quality, providing foundation for studying how dataset propertics impact the Rashomon effect. Sec- ond, combining models in each RashomonGB iteration is similar to model stitching [50], suggesting potential insights if adapted for neural networks. Third, varying the number of models selected per iteration could enhance RashomonGB's s flexibility and effectiveness.
d(SS 0) counts the fraction of samples that have disagreement among the models in the Rashomon
10

---

Disclaimer. This paper was prepared for informational purposes by the Global Technology Applied Research center and Artificial Intelligence Research group of JPMorgan Chase & Co. This paper is not product of the Research Department of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, accounting effects thereof. This document is not intended as investment research or investment advice, or as a recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product service, to be used in any way for evaluating the merits participating in any transaction.
References
[1] A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, and H. Wallach. A reductions approach to fair classification. In Proceedings of the International Conference on Machine Learning, pages 60-69. PMLR, 2018.
[2] A. N. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[3] J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias. In Ethics of Data and Analytics, pages 254-264. Auerbach Publications, 2022.
[4] S. Arimoto. An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Transactions on Information Theory, 18(1):14-20) 1972.
[5] A. Asuncion and D. Newman. UCI machine learning repository, 2007.
[6] S. Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi, Gradient boosting neural networks: GrowNet. arXiv preprint arXiv:2002.07977 2020.
[7] R. K. E. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, Mehta, A. Mojsilovic, Nagar, N. Ramamurthy. Richards, D. Saha, Sattigeri, M. Singh, K. R. Varshney, and Y. Zhang. AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, Oct. 2018. URL httpp://aaae prg/abs/1810 01943.
[8] Bird, M. Dudík, R. Edgar, Horn, R. Lutz, V. Milan, M. Sameki, H. Wallach, and K. Walker. Fairlearn: A toolkit for assessing and improving fairness in AI. Microsoft, Technical Report MSR-TR TR-2020-32 2020.
[9] E. Black, K. Leino, and M. Fredrikson. Selective ensembles for consistent predictions. In Proceedings of International Conference on Learning Representations, 2022.
[10] E. Black, M. Raghavan, and S. Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the Conference Fairness, Accountability, and Transparency. ACM, 2022.
[11] R. Blahut. Computation of channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460-473, 1972.
[12] L. Breiman. Statistical modeling: The two cultures. Statistical science, 16(3):199-23) 2001. 2001.
[13] J. Brophy and D. Lowd. Instance-based uncertainty estimation for gradient-boossted regression trees. In Advances in Neural Information Processing Systems, 2022.
[14] F. Calmon, D. Wei. B. Vinzamuri, K. Natesan Ramamurthy, and K. R. Varshney. Optimized pre-processing for discrimination prevention. In Proceedings of Advances in Neural Information Processing Systems, 2017.
[15] T. Chen, T. He, M. Benesty, Khotilovich, Tang, H. Cho, Chen, R. Mitchell, I. Cano,
T. Zhou, et al. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1-4, 2015.
11

---

[16] Z. Chen, C. Zhong, M. Seltzer, and C. Rudin. Understanding and exploring the whole set of good sparse generalized additive models. In Proceedings of Advances in Neural Information Processing Systems, 2023.
[17] A. F. Cooper, K. Lee, M. Z. Choksi, Barocas, C. Sa, Grimmelmann, Kleinberg, S. Sen, and B. Zhang. Arbitrariness and social prediction: The confounding role of variance in fair classification. Proceedings of the AAAI Conference on Artificial Intelligence, pages 22004-22012, 2024.
[18] C. Correa. A new estimator entropy. Communications MtatisticsThheor and Methods, 24(10):249922449 1995.
[19] C. Cortes, M. Mohri, and U. Syed. Deep boosting. In Proceedings of the International conference machine learning, pages 1179-1187. PMLR, 2014.
[20] A. Coston, A. Rambachan, and A. Chouldechova. Characterizing fairness over the set of good models under selective labels. In Proceedings of the International Conference Machine Learning. PMLR, 2021.
[21] T. M. Cover. Elements of information theory. John Wiley & Sons, 1999.
[22] K. Creel and D. Hellman. The algorithmic leviathan: Arbitrariness, fairness, and opportu- nity in algorithmic decision making systems. In Proceedings of the Conference Fairness, Accountability, and Transparency. ACM, 2021.
[23] A. D'Amuur, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23(():10237-10297 2022.
[24] F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. In Proceedings of Advances in Neural Information Processing Systems, volume 34, pages 6478-6490, 2021.
[25] J. Dong, Y. Chen, B. Yao, X. Zhang, and N. Zeng. A neural network boosting regression model based on xgboost. Applied Soft Computing, 125:109067, 2022.
[26] J. Donnelly, S. Katta, C. Rudin, and E. Browne. The rashomon importance distribution: Getting rid of unstable, single model-based variable importance. In Proceedings of Advances in Neural Information Processing Systems, volume 36, 2024.
[27] T. Duan, A. Anand, D. Y. Ding, K. K. Thai, S. Basu, A. Ng, and A. Schuler. Ngboost: Natural gradient boosting for probabilistic prediction. In Proceedings of the International Conference on Machine Learning, pages 2690-2700. PMLR, 2020.
[28] C. L. DuBois. edu.
UCI network data repository, 2008. URL http://networkdata .ucs.uci.
[29] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasuuraamanian. Certifying and removing disparate impact. proceedings of the =SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259-268. ACM, 2015.
[30] A. Fisher, C. Rudin, and Dominici. All models are wrong, but many are useful: Learning a variable's 's importance by studying an entire class of prediction models simultaneously. The 2019.
Journal of Machine Learning Research, 20(177):
[31] J. Frery, A. Habrard, M. Sebban, O. Caelen, and I.. He-Guelton. Efficient top rank optimization with gradient boosting for supervised anomaly detection. Proceedings of Machine Learning and Knowledge Discovery in Databases, pages 20-35. Springer, 2017.
[32] Y. Freund and R. F. Schapire. decision-theoreti generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences, -5-19:119-139, 1997.
12

---

[33] J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: statistical view of boosting (with discussion and rejoinder by the authors). Annals of Statistics, 28(2):337-407, 2000.
[34] H. Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, pages 1189-1232, 2001.
[35] Y. Gal and Z. Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, pages 1050-1059. PMLR, 2016.
[36] L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In Proceedings of Advances in Neural Information Processing Systems, volume 35, pages 507-520, 2022.
[37] L. Hancox-Li. Robustness in machine learning explanations: does it matter? In Proceedings of the Conference on Fairness, Accountability, and Transpurency. ACM, 2020.
[38] M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In Proceedings of Advances in Neural Information Processing Systems, volume 29, 2016.
[39] M. G. M. M. Hasan and D. Talbert. Mitigating the rashomon effect in counterfactual explanation: game-theoretic approach. In Proceedings of the International FLAIRS Conference, 2022.
[40] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the Conference Computer Vision and Pattern Recognition. IEEE, 2016.
[41] W. Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409-426, 1994.
[42] H. Hsu and Calmon. Rashomon capacity: A metric for predictive multiplicity in classification. In Proceedings of Advances in Neural Information Processing Systems, 2022.
[43] H. Hsu, G. Li, Hu, and C.-F. R. Chen. Dropout-basee rashomon exploration for efficient predictive multiplicity estimation. In Proceedings of the International Conference on Learning Representations, 2024.
[44] X. Hu, L. Chu, J. Pei, W. Liu, and J. Bian. Model complexity of deep learning: A survey. Knowledge and Information Systems, 63:2585-2619, 2021.
[45] F. Kamiran, A. Karim, and X. Zhang. Decision theory for discrimination-awanl classification. In Proceedings of the International Conference Data Mining, pages 924-929, IEEE, 2012.
[46] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master Thesis, Department of Computer Science, University of Toronto, 2009.
[47] S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1 797 1951.
[48] B. Kulynych, H. Hsu, C. Troncoso, and F. P. Calmon. Arbitrary decisions are hidden cost of differentially private training. In Proceedings of the Conference on Fairness, Accountabiliry, and Transparency. ACM, 2023.
[49] V. Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. In Proceedings of Advances in Neural Information Processing Systems, volume 27, 2014.
[50] K. Lene and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 991-999. IEEE, 2015.
[51] S. Li, R. Wang, Q Deng, and A. S. Barnard. Exploring the cloud of feature interaction scores in a rashomon set. In Proceedings of the International Conference on Learning Representations, 2024.
13

---

[52] C. X. Long, H. Hsu, W. Alghamdi, and F. P. Calmon. Arbitrariness lies beyond the fairness- accuracy frontier. In Proceedings of Advances in Neural Information Processing Systems, 2023.
[53] M. Long and A. Servedio. Random classification noise defeats all convex potential boosters. In Proceedings of the international conference on Machine learning, pages 608-615, 2008.
[54] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations, 2019.
[55] A. Malinin, L. Prokhorenkova, and A. Ustimenko. Uncertainty in gradient boosting via ensembles. In Proceedings of the International Conference on Learning Representations, 2021.
[56] C. Marx, F. Calmon, and B. Ustun, Predictive multiplicity in classification. In Proceedings of the International Conference on Machine Learning. PMLR, 2020.
[57] L. Mason, J. Baxter, P. Bartlett, and M. Frean. Boosting algorithms as gradient descent. In Proceedings of Advances in Neural Information Processing Systems, volume 12, 1999.
[58] L. Mason, J. Baxter, L. Bartlett, M. Frean, et al. Functional gradient techniques for combining hypotheses. In Proceedings of Advances in Neural Information Processing Systems, pages 221-246, 1999.
[59] M. Milani Fard, Q Cormier, K. Canini, and M. Gupta. Launch and iterate: Reducing prediction churn. In Proceedings of Advances Neural Information Processing Systems, volume 29, 2016.
[60] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56 (11):5847-5861, 2010.
[61] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, volume 32, 2019.
[62] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, Dubourg, al. Scikit-learn: Machine learning python. the Journal of Machine Learning research, 12:2825-2833 2011.
[63] A. Roth, A. Tolbert, and S. Weinstein. Reconciling individual probability forecasts. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2023.
[64] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22) 215.0 2019.
[65] R. E. Schapire. The strength of weak learnability. Machine learning, 5:197-227, 1990.
[66] R. Schapire et al. A brief introduction boosting. In Proceedings of the International Joint Conference Artificial Intelligence, volume 99, pages 1401-1406. Citeseer, 1999.
[67] I.. Semenova, C. Rudin, and R. Parr. A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint 1rXiv:1908.01755. 2019.
(68] S. Shalev-Shwartz and S. Bev-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[69] Y. Song and C. Zhang. New boosting methods Gaussian processes for regression. In Proceedings of the International Joint Conference Neural Networks, volume 2, pages 1142- 1147. JEEE, 2005.
[70] O. Sprangers, S. Schelter, and M. de Rijke. Probabilistic gradient boosting machines for large-scale probabilistic regression. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021.
14

---

[71] M. Telgarsky. The fast convergence of boosting. In Proceedings of Advances in Neural Information Processing Systems, 2011.
[72] R. Vershynin. Hiigh-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
[73] M. Wainwright. High-dimenssonal statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[74] J. Watson-Daniels, D. C. Parkes, and B. Ustun. Predictive multiplicity in probabilistic classifi- cation. Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[75] Watson- Watson-Damiels, churn with the set of good models. arXiv preprint arXiv:2402.07745, 2024.
Calmon, D' Amour, Long, D. C. Parkes, and Ustun. Predictive
[76] R. Xin, C. Zhong, Z. Chen, T. Takagi. M. Seltzer, and C. Rudin. Exploring the whole rashomon set of sparse decision trees. In Proceedings of Advances in Neural Information Processing Systems, 2022.
[77] A. Xu and M. Raginsky. algorithms. In Proceedings of Advances Neural Information Processing Systems, volume 30, 2017.
analysis of generalization capability of learning
[78] I.-C. Ych and C.-h. Lien, The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2):2473- 2480, 2009.
[79] Zhang and B. Yu. Boosting with early stopping: Convergence and consistency. Annals of Statistics, 33(4):1538-155) 2005.
15

---

- The appendix is divided into the following parts. Appendix A: Omitted proof' and theoretical results; Appendix B: Discussion on predictive multiplicity metrics; Appendix C: discussion on the Rashomon set with Rademacher complexity; Appendix D: Details on the experimental setup; and Appendix E: Additional empirical results and ablation studies.
A Omitted proofs and theoretical results
We first introduce the following useful lemmas to facilitate the proofs of the propositions. The first lemma a fundamental property of sub-Gaussian random variables (Proposition 2.5.2 from Vershynin [72]), and the second lemma is slight extension of the Theorem 3 in Xu and Raginsky [77].
Lemma A.1. If random variable X is o-sub-Gaussian then
Pr[ 2 7) S 3
(A.1)
Lemma A.2. Suppose ((h, o-sub-Gaussian under the data-generating distribution Ps with S n. for all h E H. then
Pr (r (I) Ls(H) > a) 3,
(A.2)
if 80² (1(5.n) log 3)) for
any any 0 and <B 1.
A.1 Proof of Proposition! 1
By the definition of the Rashomon set in Eq. (1), we know that E S, if and only if the loss deviation is upper bounded by e; that is,
Trg (h) (1)) Se.
(A.3)
We can decompose the loss deviation in Eq. (A.3) by
Lps (l) Lps (h²) Lps (h) Ls(h) + Ls(h) - Lps (hi)
(A.4)
(2)
By picking = - in Lemma A.2, we have with probability at least Z.
802
(2I(S: H)
1
(1) = (h) Ls(h) <
+
(A.5)
P
Moreover, since the loss function E(h, is o-sub-Gaussian the risk (s(h) is a Gausubaa By Lemma A. and the non-negativity of the loss functions, we have
Pr (Ls(h) Lr, (h*) 2 7) < Pr (Ls(h) Pr
7) 2e
(A.6)
2n2 Let 2e .2 = and solve for 7, we have with probability at least
P.
0² In P
4
(2) = Is(h)) L.p (1,") <
(A.7)
Therefore, combining Eq. (A.4), Eq. (A.5) and Eq. (A.7) with probability union bounds, we have with probability at least - P.
802
(21(S:H)
4
0
4
(P)(h) - (1e) S V
In
In
(A.8)
P
p
and hence
802 (2I(S; H)
4
)
Pr he BER R H,S,h", V
In
+
In p
>1
-1-0
(A.9)
p
p
16

---

A.2 Proof of Proposition 2
We start with the Rashomon set of fitting models to the pseudo-residuale T-th boosting iteration.
With the MSE loss, we have Tu (e(((-1as))
2(y; (x-1(x;)) We neglect the factor just constant. From Proposition 1, we know that for hr E H. with probability at least 1 - P.
(x)
(xx) rTT)" + cr:
(A.10)
72
where 6T Sa²
+ In + In is the Rashomon parameter. By plugging in the recursive relation of the pseudo-residuals: we have
21(Sr:H)
P
(x:) -T:)) + eT
(1g (x.) - (y: Σ
hi (xx)))))
1=1
h (x,))) y1))² E2
(A.11)
t=
(x,) - ui)² + Ee.
On the other hand, for the left-handed term in Eq. (A.10), we have for hr E H. with probability at
least 1 P
(NT((x) r+ti2
(x;) -(T--1) + 11 (x:))
(x)((x)) TCT- + 10T 1 -T-1
(A.12)
T
he(x;) - r(T 1()) ET-1,
(=T-1
where the inequality in Eq. (A.12) comes again from Proposition By repeating the decomposition of the pseudo-residuals in Eq. (A.12), we have for fr E F, with probability at least (T' - 1)p.
(xr(x;;
rt) 26)22
(x,)); 11)²
7
71
(A.13)
T-1 4
Σ
(xr(x)) - y1))
Σ
n
Finally, combining Eq. (0,, Eq. (A.11), and Eq. (A.13) with probability union bounds, we have for fr E F, with probability at least 1 - TTp
n Σ (T(xx)
n
v1)3 -.:)3
Σ (ff(x:)
y1)
Et
(A.14)
where the overall Rashomon parameter DT= 4, T
02 In
Sa2
2I(S,;H)
In 1 p
P
A.3 Proof of Proposition 3
With loss of generality, let t and to t. If the loss function l is the MSE loss, the pseudo-residuals of the f-th boosting iteration have the form
[8(()(-1,8:)
Tu
2(y: - fe-1(x,) = 2(y - fx-2(xx) - hx-1(x;)) (A.15)
0f:-1
r(t-1)i 2h2 (x:)-
17

---

Therefore, we have the recursive relation of the residual variables as R₃ = -R-11 2H. The mutual information then follows
[H;; Rx T(;; 1, HIX) T(H; (H: + + ( Ri-1) R XX),
(H;
(A.16)
since mutual information I(H;; XX, R2-1) is non-negative.
Similarly, the loss function is the binary loss, the pseudo-residuals of the 1-th boosting iteration have the form
ae de(f
V1
- Vi
Y: (x:)
Tts
0h1-1
[x-1(x;)
ft (xi)
-
(A.17)
(x))
+c= ((--((-1))++ + +c
St 2x-2 st-2(xi))
where e cs function of 14-1 (x:):
31) h2 hx=1(x;))
fx-2 ft- + 1411 xx) (1 2f 2 (x:) (x;))
(A.18)
Therefore, consider the corresponding random variables of Eq. (A.17), have Rg Rt-1 C' and the desired result follows from Eq. (A.16).
18

---

B
Additional discussions
We provide addition background and discussions on gradient boosting, predictive multiplicity, the Rashomon gradient boosting algorithm, and group fairness.
B.1 More on gradient boosting
The minimization of Ls(fr) can be viewed functional gradient descent (FGD), which is com- putationally infeasible optimization problem in general [58]. A common alternative to approximate FGD is applying steepest descent to find a local minimum of the loss function by iterating on given (x). i.e., (x) = fx-1(x) - '/ ft LS((i11) with a learning rate 1) E IR. This way, forward stage-wise manner, we can
basis function h2 E H. that closest to the functional gradient V y. Ls subject to a distance measure d with h2 = argmin, d(V L h). Varying the choices of and d recovers most widely-used boosting algorithmss. For instance, E(hy, 8t) e (xh and d( (fi-1), he) -VLLs(fi-l) (fi-1)-hs,, /1,, the FGD recovers adaptive boost- ing (AdaBoost) [32], and if (h,, s:) log(1 + e -vv.h(x))), it recovers LogitBoost [33]. Finally, if d is the f2-norm || - V (s((tt)) - hell instead of the inner product, it recovers the gradient boosting [34]. One of the most popular variants of gradient boosting is by further applying LASSO and ridge regularizations, called the eXtreme Gradient Boosting (XGB) [15].
B.2 More on predictive multiplicity and the empirical Rashomon sets We summarize existing predictive multiplicity metries, from their background, mathematical for- mulation, operational meanings, to computational details. Predictive multiplicity metrics can be categorized into two groups: score-based and decision-based, where a decision is thresholded score or the score vector after argmax. Precisely, consider a binary classification, if we have a score g. then the decision can be obtained by [[s r]. where is a threshold. For c-class classification problem where > 2, the score is vector, say q Ac, and the decision can be obtained by argmax [q)- In the following, we start with the decision-based metrics, see Table B.I.
Decision-based predictive multiplicity metrics essentially measure the "conflictions" of the decisions either for the whole dataset or per sample. Marx et al. [56] propose two metrics: ambiguity and discrepancy; both of them measure the fraction of conflicting decisions across dataset. Ambiguity the proportion of samples in a dataset that can be assigned conflicting predictions by competing classifiers in the Rashomon set. Discrepancy is the maximum number of predictions that could change in dataset if we were to switch between models within the Rashomon set. More precisely, given a pre-trained model hw,, the ambiguity (D) and the discrepancy 6(D) are respectively defined in [56, Definitions 3 and 4]. Both ambiguity and discrepancy can be estimated by mixed integer program [56, Section 3]. The implementation of estimating ambiguity and discrepancy can be accessed https:/ thub. cwwccharllccccccooooococcccca charl iemarx/ ppwpoos...o......
Table B.1: Decision-based predictive multiplicity metrics.
Metrics
Definitions
Amhiguity [56]
(D) A
max 1 (argmax (xi) * argmax hw (xx)) how ER
P
X;
Discrepancy [56]
8(D)
max fow E
Σ 1 Jargmax how (x.) o argmax how- (x:)] X:ED
Disagreement [10, [8]
(xx.) A 2 Pr (argmaxxa hw (x.) o argmax Rew (x.); low, Koy E R2
Table B.2: Score-based predictive multiplicity metrics.
Metrics
Definitions
Std./ Var: of scores [52, 17, 9]
s(x;) A Enw (x; Eno Pro. (> (xx v(x.) 2
Viable Prediction Range (VPR) [74]
max /xw(x.)
min Hew (xc. lowk
hw
Rashomon Capacity (RC) [42]
c(x:) 2 sup inf EnwwPa DKL (hw(hw (x:))l())
PR q
19

---

Instead of computing the empirical fraction of conflicting decision over dataset D, disagreement directly using the probability of the occurrence of conflicting decisions per sample Black et al. [10. Section A.1] and Kulynych et al. [48, Eq. (4)]. The factor 2 in the definition of disagreement ensures that (x(x;) in the [0, 1] range for the ease of interpretation. Kulynych et al. [48] further proposed plug-in estimator to estimate disagreement for binary classification with sample complexity bound on the number of models obtained by re-training. The implementation of directly estimating disagreement from the empirical Rashomon set along with the plug-in estimator can be accessed at https:// //gi thub. cow cwm/soriinggiaecce spring-epfl/dd http//dppmmmliiiiii cuuutiplai tipicittyy
On the other hand, score- based metrics focus on the spread of the output scores; see Table B.2. The most straightforward metric to compute the standard deviation (std.) s(x,) (and the variance (var.) of the scores of a sample by all models in the Rashomon set Long et al. [52. Definition 2]. see https: / //githut thub. com/Carol-Lonn Looooll--on//- /angg//aii Faairness
rness, and_aandddttaa Arbitrariness for the implementation. However,
score std. or var. fails to capture large score spreads that concentrate on small subset of models. To precisely capture the largest possible spread of scores, Watson- Danielss et al. [74, Definition 2] proposed Viable Prediction Range (VPR) vxx)). which is the largest score deviation of sample that can be achieved by models in the Rashomon set. The VPR can be computed using similar mixed integer programs in Marx et al. [56] for binary classification with linear classifiers. However, Watson Daniels et al. [74] did not release their codes.
Borrowing from information theory, Hsu and Calmon [42, Definition 2] measures the spread of output scores for c-class classification problems in the probability simplex A, by an analog of channel capacity, termed the Rashomon Capacity. Note that the infimum inf qea. Eh DKL (hw (x; )|()) measures (in the sense of KI. divergence) the spread of the scores of a sample x; given a distribution PR over all the models hw in the Rashomon set, where the minimizing acts as "centroid" for the outputs of the classifiers. The supremum picks the worst-case distribution P'R over all possible distributions in the Rashomon set. They proposed the adversarial weight perturbation (AWP), which perturbs the weights of a pre-trained model such that the output scores of sample are thrust toward all possible classes. The outputs of the perturbed models can then be used to compute RC by the algorithm [11, 4]. The implementation of AWP and the Aaohut-Aromoto algorithm can be accessed at https: / htt///gttttttt , com/Hs anggisinggguagg http://raaaoooooooaaaaee
The estimation of all these predictive multiplicity metrics is, practice, computed with the empirical Rashomon set, as the true Rashomon set in Eq. (1) is computationally infeasible. Note that, however, the size of the empirical Rashomon set is not a proxy for multiplicity. For instance, an empirical Rashomon set with 100 globally diverse (e.g., obtained by re-training with different seeds) models might exhibit a higher predictive multiplicity metric (e.g.. VPR) compared to another empirical Rashomon set containing 1000 models that differ only locally. The size of the true Rashomon set, on the other hand, representing an ideal scenario achievable with unlimited computational and storage resources, can indeed act a proxy for predictive multiplicity. In this context, predictive multiplicity metrics are non-decreasing with a larger size of the true Rashomon set (i.c., a larger e).
B.3 More on RashomonGB
We provide visualizations of RashomonGB in Figure B.6. For the sake of illustration, we pick T 3 and m = 2: however, T and m could be arbitrary numbers. The left hand side shows 2 re-trraae trainngn of gradient boosting, where the final models are M31 and M32. and there are m T = 2 x
Iteration
Mys
Mip
M22
lteration 2
WORe Mor
Mus
Mus
My May
MareMorr
Ma
lteration 3
Myn
Maa
M
MM
Ma
Ma MMy
Mc
Mar
Mar
Figure B.6: Re-training and RashomonGB with the same training costs produce difference numbers of models in the Rashomon set.
20

---

training in total. The right hand side is RashomonGB that generates m = 2 = 8 models, i.c., {M,. + M22 [Ma1, M32). In essence, building the Rashomon set with RashomonGB is similar to expanding the tree in each iteration (i.e., layers) until it is complete m-ary tree.
B.4 Group-fairness metrics and fairness interventions
Consider triple of sample (x, a, y), where and y are the usual feature and label pair, and is an additional attribute, such as race or sex, that can be used to split the datasets into different groups (e.g.. female vs. male if a is sex). Group fairness is measured by the gap of a certain quantity evaluated on different groups. For example, statistical parity (SP) considers the the probability of the model outputting class 1, and mean equalized odds (MEO) considers the sum of True Positive Rate (TPR) and False Positive Rate (FPR). For simplicity, denote Y arggaax h(X) the prediction label from the model h, and assume that there are B groups. Then the equalized odds (EO) and (SP) can be formally defined as:
Definition B.1 (SP (29]). Pr (8 = 1|A a) Pr (8- = 1|A = ad). Va, a' E {
.').Va,a'
Definition B.2 (EO [38]). Pr Y 1|A a, - Pr (r 1A d' vv) Va, a' E [B] and Vy [c].
The operational meaning of the two group fairesss metrics is as follows: Statistical Parity (SP) requires that the predicted label Y be independent of the group attribute A, while Equalized Odds (EO) conditions both the group attribute and the true label for independence. EO improves upon SP by allowing for perfect classifiers when the true label Y is correlated with the group attribute A. In practice, we measure fairness by quantifying the level of EO and SP violations, as reported in Figure 4. The violation of EO is also referred to as the mean EO.
MEO A max a,a' a,a'E[B] EB]
([TPRA=a -TRR + APR FPR
2
(B.19)
1 2
y 1 Pr (v 1|A= =d)
SP violation 0.a0 [
max
Pr
Fairness intervention algorithms aim to make the outputs of a machine learning model satisfy either MEO or SP violation smaller than given budget. The faimess interventions can be categorized into three categories: pre-processing, in-processing and post-prose postt-rocessing. Pre-processing mechanisms, such the one proposed by Calmon et al. [14], transform the dataset using a random mapping to reduce group fairness metrics while preserving utility. This approach is the most flexible within the data science pipeline, as it is independent of the modeling algorithm and can be integrated with data release and publishing mechanisms. In in-processing mechanisms such as Reduction incorporate fairness constraints directly into the training process. This typically involves adding fairness constraint to the loss function, resulting in fair classifier. Post-processing mechanisms such as Eqodds and rejection treat the model as a black box and adjust its predictions (by, e.g., tilting) to meet the desired fairness constraints.
Here, we provide more details on the faimess intervention baselines used in Section 4.2. Reduction [1], short for exponentiated gradient reduction, is an in-processing technique that converts fair classi- fication into sequence of cost-sensitive classification problems. It produces randomized classifier that achieves the lowest empirical error while satisfying the desired fairness constraints. Although this technique efffectively achieves fairness with minimal accuracy loss, it is computationally expensive due to the need for re- re-training multiple models. Rejection [45] is a post-processing technique that achieves fairness constraints by adjusting the outcomes of samples within confidence band around the decision boundary. It assigns favorable outcomes to unprivileged groups and unfavorable outcomes to privileged groups, resulting in thresholded predictions rather than probabilities over binary labels. EqOdds [38] is poost-prcessing technique that addresses empirical risk minimization with fairness constraint by formulating it as a linear program. It adjusts predictions based on the derived probabilities to achieve equalized odds. FaiRS [20] develops a framework for characterizing predictive fairness properties across models in the Rashomon set. They also propose a variant of FaiRS that addresses the issue of selective labels, achieving the same guarantees with oracle access to the outcome regression function.
21

---

C Characterizing Rashomon sets with Rademacher complexity
Besides the information-theorrtiic analysis the Rashomon sets in Section 3, here, further provide an analysis on the size the Rashomon sets based on Rademacher complexity [68]. Similar to the Sapnik-Chervooenkis (VC) dimension, Rademacher complexity measures the richness of of functions with respect to probability distribution, reflecting its capacity to fit random noise, and is widely-adopted in modern analysis of machine learning generalizability. Following the notations in Section 2, the definition of Rademacher complexity is as follows.
Definition C.3 (Rademacher Complexity). For a hypothesis space H. samples and loss function e, the Rademacher complexity of H with respect to S is defined as
Rad(e ( H o o
[11)
sup
(,l(h,ss))
(C.20)
71
KCW
where the random variable in a are i.i.d. distributed according to the Rademacher distribution, i.e., P(o; 1) P(a, 1) 0.5.
The following lemma is one of the fundamental results of the Rademacher Complexity. Lemma C.3 (168, Lemma 26.5]). Assume for all s; S and H we have (é(h, s;)] S M, i.e. the loss function is bounded by M; then with probability of at least - for all h E H.
(m/pp/)
Lps (h) - Lhh)) < 2Rad(( - H o - 4M 4M
(C.21)
The following proposition states that the size of a Rashomon set is controlled by the Rademacher complexity. Proposition C.4. Let y 2 sup Ls(h) Ls(h), we have with probability of at least p.
WAEEH
H.S. h², .7 H 5M 2ln(4/p)
h E R
(C.22)
n
Proof. Recall the constrain in the definition of the Rashomon set in Eq. (1), i..,
(h) Lps (h*) = Lps - Ls(h) + Ls(h) Ls(h") - Ls(h") - (h*)
(C.23)
())
(2)
(3)
Term (1) follows directly from Lemma C.3 and let (2) S sup Ls(h) Ls(h") S 7 by assumption.
be7t
By the Hoeffding's inequality [41], and denote 2; = =(hh, s;), we have for any function h E H,
n
(- 2nt
P (h) - (p)( < t) = P
24
26
<t
< 2exp
n
M2
(C.24)
Therefore by taking = h², with probability of at least - 2/2, 2, we have
In(4/p)
3) Ls(h") Lp, (h*) s My
(C.25)
2n
Combining all together with the union bound, we have with probability of least 1 p
2 lm(/p)
5M LPs (h) (t²") S 7 2Rad( Ho +5M
V
(C.26)
If we pick the loss function to be the p-norm, i.e., {(h(x;), = [h(x;;) vil. then the results in Proposition C.4 become
21m(4/p)
)) 21-p. (C.27)
P E H.S.h", , 2pM'- Rad(E o H. S) + 5M"
n
22

---

Despite that we did not use Proposition C.4 in the main text, Eq. (C.22) stills provide us insights on controlling the size of Rashomon sets. For example. y and M reflect the boundedness of the loss function, implying that CE loss, which is in general unbounded, could be a cause of a large Rashomon set. Moreover, the Rademacher complexity Rad(EoHos)) suggests that larger hypothesis space leaves the models bigger "wiggle room" for multiplicity, Finally, note that as -> oc, the Rashomon parameter in Eq. (C.22) does no converge to O.
23

---

D Details on the experimental setup
We summarize the dataset descriptions and training setups in Section 4.
D.1 Dataset description and pre-processing
In this paper, we use 18 tabular datasets from the UCI machine learning repository [5], the ACS Income dataset [24] and the COMPAS recidivism dataset [3]. We summarize the descriptions of all tabular datasets, including the number of features, training/test split (seed = 42), and the label description in Table D.3.
The UCI machine learning repository (accessible at DuBois [28]; license: CC BY 4.0) is well- known and widely used collection of 650 datasets for machine learning research and experimentation, and contains a diverse and extensive collection of datasets across various domains. We select 18 datasets in specific domains, including medicine, economics, society, etc.,, that may possess critical consequences if predictive multiplicity is not accounted for. For these UCI datasets, we remove samples with missing values, one-hot encoded nominal features, re- re-seale numeric features, and set the target label name to be 1 and the rest to be O.
The ACS Income dataset (accessible at https: httt:::/github.... http://ggthuuu..ooaar/r////////a
license: MIT license) and the UCI Adult dataset are collected from the United States Census Bureau. The goal of both datasets is to predict the income with demographic features of people. The UCI Adult dataset has smaller size and a binary record of the income label (Y = 1 if income >> S50K and Y 0 otherwise). The ACS Income dataset has more than 1.6 million samples. The income attribute is real-valued rather than a binary label. We transform the income to a binary labels by Y 1 if income > $39K (the median income) and Y otherwise.
The COMPAS Recidivism dataset (accessible at https://wwa wwwp:pppoppbic..ccmm org/ wrwpiiiia.grg/atttttrtrro datastore/
dataset/ cowpees/coossseeeeeeee cttp://ooooa--rriiiiiiiit www.iiriiiiiiiiiiiiiiie satee------ddd------aea and- license: CC-BY-4.0) con- tains the prior criminal history for criminal defendants and the demographic makeup of prisoners in Brower County, Florida from 2013-2014, we select gender, age, number of prior crimes, length of custody and likelihood of recidivism to be the features. We pre-process the dataset by dropping missing/incomplete records, and convert categorical variables by one-hot encoding. For the group fairness experiments in Section 4.2, we only keep two races, African American (S = 0) and Caucasian (S
html; license: MIT License) contains 60,000 images and equally distributed to 10 classes, such cars, cats, etc. The dataset is split into 50,000 and 10,000 images for training and validation, respectively. Each image is colored image with size of 32x32. For the pre-processing, we normalize each channel of image by the mean and standard deviation of the whole training set.
CIFAR- 10 dataset (accessible at https:/ / /www./wwwcccoo// cs. toronto. . edu/
D.2 Training setups and results
The weak learners used for all tabular datasets are decision trees with the Python Scikit Learn package [62]. The decision trees are expanded up to a maximal depth of 2, and the criterion for splitting is the squared loss with impurity gain greater than 10-7. The minimum number of samples required to split an internal node and the maximum number leaf nodes is The number of iterations for re-training and RashomonGB is T 10. In each iteration we train models with different random seeds for the splitting at each depth, and use a filtering process in place that screens out models with MSE loss greater than 0.1 (i.e., 6, 0.1) and retains models with an MSE loss smaller than 0.01 until m = 10 models are collected at each iteration. For RashomonGE, we randomly pick 2 out of m models in each iteration. For all experiments, we set the learning rate to be a = 0.8, and the loss function is the binary cross-entropy loss For experiments in Section 4.2, we the AIF360 [7] and Fairleam [8] packages implement the faimess intervention algorithms.
(h) A 1 -.=il- log softmax(h(xx)) - (1 - y:) log(1 - softmax(h(xx)
n
The weaker learner used for CIFAR-10 experiment is 3-stage convolutional neural network with Python Pytorch package [61], each stage composed by a convolutional layer with kernel size 3x3 and output channel 64, a batch normalization layer and ReLU layer, and the residual shortcut is
24

---

added before ReLU for stage 2 and 3; at the end of each stage, average pooling layer reduces the spatial dimension by 2 with kernel size After stage a global average pooling applied to reduce the spatial dimension to 1x following by linear layer for regression the pseudo-residual of each class. For each weaker learner, we train with batch size 256 for 50 epochs, and we use the AdamW [54] optimizer with fixed learning rate, 10-3 and weight decay, 10-1. Due to its complexity, we use T and m 50 for both re-training and RashomonGB. During inference for RashomonGB, we random pick 3 out of models for each boosting iteration but for the last iterations, which results 972 models. We adopt the same learning rate (0.8) used the experiments for the tabular datasets.
Table D.3: Tabular dataset descriptions.
of features
Training set size
Test
Dataset
Label (# of classes)
set size
ACS Income
10
1331k
332k
income larger than median not (2)
UCI Adult
104
22621
7541
income >50K (2)
AIDS-175
26
1283
856
patient death within the study period (2)
Bank Marketing
63
30891
10297
has deposit (2)
Cardiotocography (ctg)
84
1275
851
normal or not (2)
COMPAS
6
4222
1056
commit a crime again or not (2)
Contraception
9
1104
369
long or short term (2)
Credit Approval
51
414
276
credit card application approval not (2)
Credit Card
23
24000
6000
default payment not
Cylinder bands
39
324
216
band no band (2)
Dropout
36
2654
1770
student drops out of school or not (2)
Epileptic seizure
178
6900
4600
subject has seizure not (2)
German credit
20
600
400
good/bad credit risk (2)
Heart Disease (Cleveland)
13
181
122
absence/presence of heart discase (2)
ILPD
10
349
234
patient with with/without liver disease (2)
Mammography
5
622
208
benign or malignant (2)
Mushroom Secondary
20
36641
24428
poisonous not (2)
Qualitative Bankruptcy
18
150
100
had bankruptcy
(2)
Taiwan Credit
23
18000
12000
borrower defaults on payment or not (2)
Wine
13
106
72
Wine type 2 vs. rest (2)
Wine Quality
13
3898
2599
Quality > 5 (2)
25

---

E Additional results and experiments
We include an illustration on how interpret Figure additional experiments on other UCI datasets (UCI Adult, Bank Marketing, Mammography), computational time comparison between re-training and RashomonGB, ablation studies on different types of weak learner, number of boosting iterations T, number of models in each iteration m, comparison with predictive uncertainty estimation methods, and hyperparameters for mitigation predictive multiplicity (E, k, and X).
E.1 How to interpret Figure
We provide an explanation on how to properly interpret Figure 3 in Figure E.7. We first perform re-training with different random seeds and perform RashomonGB by using the same weak learners, i.e, the training cost of RashomonGB and Re-training are the same and hence the comparison presented the paper is fair. Note that the c we report here is the overall Rashomon parameter after T = 10 iterations, i.e., T X Moreover, we do not compare models with different as it is clearly unfair. For the experiments reporting predictive multiplicity in Section 4.1, we report the Rashomon parameter in the vertical axis (leftmost column) in Figure 3. For the experiments of fair model selection, we report the in the caption of Figure 4. For the experiments of mitigating predictive multiplicity by model averaging, we report the (in terms of the improvement of accuracy) in the vertical axis in Figure
Re: Me-training
RashomanGB
CE Love
0.50
0.00
0.25
0.50
1.0
14 Rashomon Capacity
1.2
0.0
0.0
0.3
Accuracy
VPR
Disagreement
Discrepancy
This figure is part of the Figure 3. For illustration, we draw a red dashed line at CE Loss = 0.542. In the leftmost column, each marker represents model for Re Re-traiming (cross) and RashomenGB (square). and the shaded area covers all models from either Re ee-training or RashomonGB that has CE Loss smaller building by
Figure E.7:
than 0.542. In other words, cross and square markers the shaded area form the Rashomon Re training and RashomonGB respectively. The values of predictive multiplicity metrics on the rightmost 4 columns at the intersection of the red dashed line are the predictive multiplicity metrics estimated with the Rashomon set of Lrg (f+) + Te 0.542. Scanning the red dashed line upward leads to larger Rashomon We would like to emphasize that our comparison between Re-training and RashomonGB fair those models are obtained from the same training cost.
26

---

1.75 E.2 Estimating predictive multiplicity metrics of other UCI datasets
We include additional results in Figure E.8 on comparing the effectiveness of estimation prediction multiplicity metrics between re-training and RashomonGB (similar to Figure 3) for UCI Adult, Bank marketing, and Mammography datasets [5]. The UCI Adult dataset aims to predict whether the income of an individual exceeds 50,000 per year based on 1994 census data. The Bank Marketing dataset related with direct marketing campaigns of a Portuguese banking institution based on phone calls in order to predict if the client will subscribe a bank term deposit or not. The Mam- mography dataset aims to discriminate between benign and malignant mammographic masses based on BI-RADS attributes and the patient's s age. Our observation is that for decision-based predictive multiplicity metrics such as disagreement and discrepancy, RashomonGB outperforms re-training in terms of the effectiveness. For score-based predictive multiplicity metrics, when the CE loss constraint is small, RashomonGE performs better for all three datasets. When CE loss constraint is large, the size of the Rashomon set grows fast, and re- training captures more diverse models in the Rashomon set. These observations are consistent with the results shown in the main text.
Re-tra inin
RashomonGB
GA22
UCI Adutt
-44
6-433
0.40
0.99
0.38
0:414
0.35
6.351
1.75
0.393
Bank Markating
0.40
- 8..0
CE Lass
0.292
6-251
0.399
0.39
0.90
0.00
0.25
0.50
0.75
1.0
1.6
0.2
0.00
0.05
0.10
Mammagraphhy
DE Loos
0.00
6-522
0.40
0.40
0.302
3.75 0.80 Accuracy
0.85
0.0
0.5 VPR
1.00 1.25 1.50
0.0
0.5
1.0
0.0
0.1
0.2
Rashomon Capacity
Disagreement
Discrepancy
Figure E.8: Re- ge-training vs. RashomonGB in exploring the Rashomon set for predictive multiplicity metrics estimation. In the leftmost column, each marker represents model. The rightmost 4 figures in row share the same y-axis for the loss difference (values shown at the right), i.c., Lrg (h e Eq. (1), Higher predictive multiplicity values mean better estimate.
27

---

E.3
Computational time comparison
We compare the training time, and the time to obtained one model for re-training and RashomonGB with T = 10, m = 10, a 0.8 and depth-2 decision tree as weak learners. We repeated the experiments times with different random seeds and report the time (in seconds) in Table E.4. Note that re- re-training and RashomonGB share the same training time, and the values we show for the columns re- trriiinng and RashomonGB are the inference time. It clear that the time cost to obtain model from RashomonGB is consistently and significantly smaller than that from re- re-training. All the runtimes we reported here are computed on the same machine in an Amazon EC2 Exlare instance.
Table E.4: Comparisons of training time, and the time to obtained a model for re- ve-training and RashomonGB. All values are in seconds, and are repeated with 3 experiments with different seeds.
Datasets
Training
retraining per model
RashomonGB per model
UCI Adult
2.6667 0.4714
0.0010 = 0.0000
0,0000 = 0.0000
ACS Income
54.6667 + 0.4714
0.4000 + 0.0000
0.0273 = 0.0000
Bank Marketing
2.6607 = 0.4714
0.0333 + 0.0471
0.0013 = 0.0005
Mammography
0.0000 = 0.0000
0.0010 = 0.0000
0.0000 = 0.0000
Contraception
0.0000 + 0.0000
0.0667 + 0.0471
0.0003 - 0.0005
Credit Card
0.6667 + 0.4714
0.0007 + 0.0005
0.0000 0.0000
COMPAS
0.3333 + 0.4714
0.0667 + 0.0471
0.0003 + 0.0005
E,4 Supporting experiments for Section 3.3
We validate our discussion in Section 3.3 that more boosting iterations could lead to larger Rashomon set. We perform RashomonGB on the UCI Adult dataset with m 100, a 0.8, and decision trees as weak learners with different numbers of boosting iterations T [1..... 10] in Figure E.9. On the left side, we show the CE Loss vs. accuracy of the models in each iteration. CE losses decreases and the accuracy increases with more boost iterations. On the right side, we show the percentage of models in the Rashomon set each boosting iteration under different loss constraints. For example, when the loss constraint is around 0.40, more than 90% of the models in iteration T' 10 are in the Rashomon set but only 50% of the models in itcration = 4.
clear that the
0.
Far
rarsentinn Moceet the
har
Exr
2:31
0:30
Figure E.9: Fraction models the Rashomon set different number of boosting iterations CE loss constraints.
28

---

E.5 Ablation study on different types of weak learners
We compare re-training and RashomonCB on the UCI Adult dataset with T 10, m 10. = 0.8, and different weak learners (depth-2 decision trees, feedforward neural networks (one- hidden layer with 10 neurons, ReIU activations, Adam optimizer, trained with 10 epochs with constant learning rate 0.001), and linear regression) in Figure 1. RashomonGB outperforms re-training on decision tree and neural network regressors as these two types of weak learners are more complicated. Linear regression is too simple and therefore RashomonGB is not able to find diverse models in the Rashomon set. Neural network weak learners achieves the highest accuracy (1% higher) than linear regression and decision trees.
Hae traiiigi
Reshomontad
0:43
0.40
0:30
0.84
0.2
0.6
14
1.2
1.3
0.5
1.0
0.0
0.37
0:35
0,835
1,50
0.5
0.05
0.15
Retworks
0.34
C1 Loace
0.33
C
C.84250 04500 84750 55000 0 Accuracy
0.2
1.00
1.05
3.10
0.0
0.5
0.00
0.05
0.10
0.15
VPR
Rashomon Capacity
Disagreement
Discrepancy
Figure 1.10: Re-training vs. RashomonGB exploring the Rashomon set for predictive multiplicity metrics estimation with different weak learners.
E.6 Ablation study on different depths of decision tree regressors
We compare re-training and RashomonCB on the UCI Adult dataset with T 10, m = 10, = 0.8, and decision trees of different depths as weak learners in Figure 11 As the tree depth increases, i.e., weak learners are more complicated, and both re- trainiin and RashomonGB achieve higher accuracy and lower CE losses. For the same loss constraints, e.g., CE loss s 0.370, a more complicated weak learner will lead to more predictive multiplicity compared to simpler weak learners, since the hypothesis space is larger, leading to a larger Rashomon set.
RashomanGB
0.45
Depth
V
0.38
0.50
2.82
0,540.00
0.73
0.50
0.75
10
12
1.4
2.0
0.5
1.0
0.0
0.1
0.2
0.45
Dopth3
Decitin
-0211
0.25
0.30
1.0
1.2
1.4
3.0
0.2
Decarre
0.38
K039
0.26
XXXX
60.00
0.82
3.83
0.64
0.00
0.25
0.50
0.75
3.0
144
0.5 Disagreement
1.0.
0.0
0.1 Discrepancy
Accuracy
VPR
Rashomon Capacity
Figure 1.11: Re-training vs. RashemonGB in exploring the Rashomon set for predictive multiplicity metrics estimation with decision tree regressors of different depths weak learners.
29

---

E.7 Ablation study on number of boosting iterations
We compare re- pe-training and RashomonGB on the UCI Adult dataset with m 10, 0.8. and decision trees as weak learnrr with different numbers of boosting iterations T = [2. 5, 10] in Figure E.12. For 5, RashomonGB finds models with higher losses and does not perform as good as trraiinnnnng. For T = 2 and 7 = 10, RashomonGB outperforms re-training as usual. Note that for the same size of Rashomon set with CE losses Ri 0.370, more boosting iterations (T 10 vs. T = 2) could lead to larger effect of predictive multiplicity, as discussed in Section 3.3.
Re-iraining
ReshomonC8
Dechule
T=2
2311 II
0:38
0.0
0.2
0:38
Decision
T-3
3
0.36
0202
0.03
0.04
0.0
0.2
0.4
0.6
10
11
12
10
0.0
0.5
1.00000 0.05 010 0.15
Decision
T-10
0.38
2320 4
0.36
1he
Accuracy
Rashomon Capacity
Disagreement
Discrepancy
Figure E.12 Re-training vs. RashemenGB in exploring the Rashomon set for predictive multiplicity metrics estimation with different boosting iterations T.
E.8 Ablation study on the number of model in each iteration
We compare re-training and RashomonCB on the UCI Adult dataset with T = 10, a = 0.8, decision trees weak learners, and different number of model in each iteration Figure E.1 When m = 2, re-rreeeee so-ttraiinng only obtains models whereas RashomonGB is still able to find more than 1000 models in the Rashomon set. Therefore, when is smaller, i.e., less budget on computational resources, RashomonGB consistently performs better than re-training
[2.5.. 10 in
Re-treining
RechomonCB
1419
Deciston Trees me
0:40
110 3
0:38
0:35
0.00
1.
233
Decision
175
0.34
0.82
0.33
0.54
0.0
02
0.4
0.6
1.0
1.2
1.4
0.0
0.5
1,0
0.0
0.1
1M
Devition
m=-30
2377 1
0.36
146 E
0.83
0.80
0.00
0.25
0.50
0:75
1.0
12
3.4
0.5
1.0.000
0.05 0.10 0:35
Rashoman Capacity
Disagreement
Discrepancy
Figure E.13: Re- Re-training vs. RashomonG3 in exploring the Rashomon set for predictive multiplicity metrics estimation with different number of models m in each iteration.
30

---

E.9 Comparison with predictive uncertainty estimation in gradient boosting
Prediction uncertainty [35, 2] indeed differs fundamentally from predictive multiplicctyy Prediction uncertainty. derived from Bayesian perspective, seeks to reconstruct the distribution p(Y|z) for given sample and assess metrics such as variance or negative ligellikohoodd of Y. typically involving only one model without specific loss constraints. Conversely, predictive multiplicity involves evaluating multiple models within the Rashomon set that exhibit similar loss, thereby reflecting a variety of potential outcomes for the same inputs. To elucidate these distinctions, we have compared our re-training strategy, RashomonGB, with the prediction uncertainty methods- [27], PGBM [70]. and IBUG [13] the UCI Contraception dataset. For rigorous comparison, in Figure 1.14, we applied these prediction uncertainty methods to estimate p(Y (z) (parameterized as Gaussian), sampled 1024 values of y, and computed the corresponding Rashomon set and predictive multiplicity metrics. The results demonstrate that RashomonGB encompasses the widest range of models, thereby providing consistently higher and more robust estimates of predictive multiplicity metrics. This comparison highlights the unique capabilities of RashomonGB in capturing a broader spectrum of potential model behaviors within the dataset.
0.625
0.575
CT
0.675
0.70
0.725
00
022
04
04
13
0.0
0.1
CE Loxs
0.00
0.75
0.575
0.100
0.725
0.0
0.2
04
0.8
10
11
12
1.3
00
0.
00
011
0.3
0.855
0.000
CE (Sts
0.625
0.75
0.675
3.700
0.725
0.0
0.2
0.4
0.6
10
1.1
12
1.3
00
0.3
1.0
0.0
0.1
0.2
03
0.600
CE 000
0.55
0.50
0.66
0.4
1.20.30
Figure 141 Re- Do-trrining (Ist row) vs. RashomonGB (2nd row) vs, prediction uncertainty methods (NGBoost (3rd row). PGBM (4th row), and IBUG (5th row)) in exploring the Rashomon set for predictive multiplicity metrics estimation the UCT Contraception dataset. For the prediction uncertainty methods, following the assumption of Gaussianity
P(Y|az) (ar) and estimate the mean and covariance from NGBoost, PGBM, and IBUG. Using the estimated mean 0 and covariance Σ. we sample 1024 from the distribution P(Y (x) Pe (0, )). In the leftmost column, each marker represents model for Re- Re-training and RashomonGB, or realization of Y sampled from prediction uncertainty methods The rightmost 4 figures in row share the for loss difference (values shown the right). i.e., (fT) + Te. Higher predictive multiplicity values mean a better estimate. For the leftmost column, it is clear that RashomonGB covers widest range models, leading to consistently higher estimates of VPR and the Rashomon Capacity. For decision- metrics such as disagreement, RashomonGB has the better estimate when CE Loss is higher; however, for discrepancy IBUG has the better estimate.
y-axis
31

---

E.10
Hyperparameter sensitivity results related to mitigation methods in Section 4.3
Figure E.15 illustrates the effect of model hyperparameters on our two measures of interest: 0- disagreement and accuracy. We report mean of each measure over 20 random train/test splits. We evaluate several datasets with larger AO-disagreement in Figure 5. In (a), varying the ensemble size E tends to reduce disagreement on the 5 datasets evaluated. From this result, we use E = 20 when fixing in our experiments. In (b), 0-disagreement increases over increasing We select the weak learners from the candidate set in ascending order of loss. Therefore, increasing k tends to add models of decreasing quality. Furthermore, the D-disagreement measure is strict; it requiring every additional model to have the same prediction on the sample. In (c), O-disagreement tends to decrease to around = 5. From this result, we tune on E [0, .0,,..3 5] in our experiments. Finally, the bottom row demonstrates that accuracy is not sensitive to changes in any of these hyperparameters.
835
025
000
000
a
(b)
(c)
SOAEAAY wayin
EAEPNY wapg 10-25
000
850
042
042
800
100
(d)
(e)
(I)
Figure E.15: Comparison varying hyperparameters the ensemble size (E) loss reweighting penalty (A),
and the total number of trained models k. The first row reports the O-disagreement (Lower better), the second reports the accuracy. We report average statistics over 20 random train-test splits.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We include a list of contributions the end of the introduction, where each contribution and the claims made therein are specifically referred to section in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include discussion of limitations regarding theoretical extensions and computational overhead in the last section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: We clearly state all the assumptions and lemmas that we have used with citations in our theoretical results. We also provide the sketch of proof/intuition in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details descriptions of the proposed methodology in the main text and in the appendix. Our methodology is tested on over 20 datasets with consistent and explainable results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
34

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Due intellectual property protection and anonymity requirements, we choose to release our codes upon decision. We provide details scripts on how to access the datasets, implement methodology, and reproduce the empirical results. We may be able release the codes during the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
Seriides/CodeSubiil
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/
guw.e////ggi/////eeeeeee//esssee /oode/oomiessssaeeeeoee wwwtebbmmssiiinooo aassonoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed information data pre-processing, train/test split with random seeds, hyper-parameter settings, etc. the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results we reported in the main text are from experiments repeated 3 20 times with different random seeds for initialization. We report the numbers with mean and standard deviation (error bars). Our methodology shows statistically significant improvements.
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification; We provide the specific type of machine we used for the experiments, especially for the runtime comparison.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : //w.riuu...../. cc/ CoubliicEEhicsuuiiei
Answer: [Yes]
Justification:
We have reviewed and complied the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We cover the positive and negative impacts of the Rashomon effect and predictive multiplicity in the introduction and related work
36

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide proper citations for all datasets including licenses and how to access, and programming packages (Python, Pytorch, Scikit Learn, AIF360, and Fairlearn) used in this paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
37

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "8Elq8CwQT8", "reviewer": "Reviewer_HS14", "summary": "The paper explores the concept of predictive multiplicity in gradient boosting models. The Rashomon effect refers to the existence of multiple models that perform similarly well on a given dataset. The authors formalize this effect in the context of gradient boosting, introduce a new method called RashomonGB to efficiently explore this multiplicity, and demonstrate its application on various datasets. The paper aims to improve the estimation of predictive multiplicity and model selection, especially with considerations for group fairness.", "questions": "1. Can the method be extended or adapted for other types of machine learning models beyond gradient boosting?\n2. How does the choice of hyperparameters in RashomonGB affect the stability and reliability of the results?\n3. What are the practical challenges faced during the implementation of RashomonGB, and how can they be addressed to facilitate broader adoption?", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of RashomonGB represents a novel method for exploring the Rashomon set in gradient boosting, offering an exponential search space as opposed to traditional linear methods.\n2. The paper provides a robust theoretical foundation using statistical learning and information theory to analyze the Rashomon effect, enhancing the understanding of this phenomenon in gradient boosting.\n3. The authors demonstrate the practical utility of RashomonGB on a wide range of real-world datasets, including tabular and image data, showcasing its versatility and effectiveness.", "weaknesses": "1. While the paper discusses the positive societal impacts of RashomonGB, it lacks a thorough exploration of potential negative impacts or misuse of the method.\n2. The theoretical analysis relies on several assumptions that may not hold in all practical scenarios, potentially limiting the generalizability of the findings.\n3. The paper mentions the intention to release code post-review, but the lack of immediate open access to code and data can hinder reproducibility and independent validation by other researchers.\n4. Implementing RashomonGB might be complex for practitioners without a strong background in the theoretical aspects of machine learning and gradient boosting, potentially limiting its adoption in the industry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and questions. \n\nFor Weakness 1, in the Introduction (Lines 26-29), we discuss the beneficial aspects of the Rashomon effect within the framework of responsible machine learning, highlighting its role in fairness by imposing additional constraints on models. This can be seen as searching for a fair model within the Rashomon set, which is further elaborated in Section 4.2 and [19].\nOn the other hand, the negative implications of the Rashomon effect (Lines 30-35), suggest that predictive multiplicity may occur, leading to decisions for certain individuals being arbitrarily based on randomness in the training process rather than on learned knowledge.\nMoreover, the RashomonGB, along with other methods designed to monitor predictive multiplicity, could lead to negative societal impacts. For example, it might be exploited by service providers to identify models within the Rashomon set that disadvantageously affect the benefits (e.g., loan approvals) of certain populations, without showing significant statistical differences from non-discriminatory models. This could contribute to what we term an \"Algorithmic Leviathan\" [21], where discrimination and bias are concealed under the guise of algorithmic arbitrariness. \n\nFor Weakness 2, specifically, we treat the loss function as a random variable, and assume that this loss function behaves as a sub-Gaussian random variable. Sub-Gaussianity is a practical assumption, as it can be easily achieved by clipping the loss (Lines 184-188).\nIt is also a common assumption in theoretical analysis as it effectively generalizes the concept of boundedness [71, 75]. Adopting the sub-Gaussian assumption enables us to conduct a more expansive analysis.\nAdditionally, we would like to highlight the novelty of our approach in establishing dataset-related bounds on the Rashomon set (i.e., Proposition 1). To the best of our knowledge, previous studies on the Rashomon effect have primarily focused on characterizing the Rashomon set through its hypothesis space. Our approach provides a fresh perspective and a significant contribution to the understanding of the Rashomon effect and predictive multiplicity in machine learning. We will enhance the clarity of these points in the revisions to Section 3.2.\n\nFor Weakness 3, we plan to release the code, but only after the review process is complete for two reasons: First, releasing the code during the review phase could compromise the anonymity required by the double-blind review process in the NeurIPS guidelines. Second, early release could potentially infringe upon the protection of our intellectual property. Despite not releasing the code at this stage, we have provided a detailed, step-by-step procedure in the paper on how to construct the RashomonGB and reproduce our findings, specifically in Figure 1, Sections 4, and Appendices B.3 and D (Lines 266-270). This should enable reviewers and readers to understand and evaluate our methodology thoroughly. Moreover, the datasets used in this paper are all public available, and detailed descriptions including preprocessing can be found in Appendix D.\n\nFor Weakness 4, it's important to clarify that the RashomonGB method can be straightforwardly implemented by training multiple models (i.e., weak learners) at each iteration, selecting $m$ models that meet the loss constraints defined by the Rashomon set, contrary to training only one model as illustrated in Figure 1. Additionally, during the inference phase, RashomonGB employs the method expansion outlined in Section 3.1 and depicted in Figure B.6 of Appendix B.3. This adaptability makes RashomonGB particularly suitable for industry applications, especially given that tabular datasets are still prevalent. Furthermore, although the training and inference processes of RashomonGB are simple and user-friendly, the effectiveness of the approach is underpinned by rigorous and innovative propositions detailed in Section 3. These factors collectively ensure that RashomonGB is not only practical but also robust, enhancing its applicability in various real-world scenarios. \n\nFor Question 1, yes, we demonstrate the utility of RashomonGB beyond beyond decision trees as weak learners by incorporating alternative weak learners, such as convolutional neural networks (shown in the results for CIFAR-10 in the lowest row of Figure 3 and the GrowNet in reference [6]), and linear regression (illustrated in Figure E.9 of Appendix E.4). This flexibility confirms that the methodology developed in our study is adaptable to a variety of settings beyond conventional gradient boosting.\n\nFor Question 2, the key hyper-parameters of RashomonGB include the number of iterations $T$ and the number of models per iteration $m$, which ablation studies are included in Appendix E.6 and E.7. With a constant probability $\\rho$ as per Proposition 3, increasing $T$ accumulates the Rashomon effect at each iteration, which in turn increases $\\epsilon$.  Additionally, increasing the number of models $m$ per iteration allows RashomonGB to explore a broader range of models, thereby enhancing the reliability of model selection and the estimation of predictive multiplicity.\n\nFor Question 3, current implementations of gradient boosting, such as those available in Scikit-Learn or XGBoost, do not support training multiple models within a single iteration. Additionally, RashomonGB requires a unique filtering process during each boosting iteration to regulate the loss deviation, which is critical for constructing the Rashomon sets. Please also refer to our reply for Weakness 4.\n\nThanks again for the comment! We are happy to provide more information or answer any follow-up questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ouYKGioKR8", "reviewer": "Reviewer_JGsC", "summary": "The paper studies the Rashomon effect in gradient boosting, a commonly used algorithm for tabular datasets, but something that has not received enough attention in multiplicity literature. The paper provides several theoretical discussions on the size of the Rashomon set and the impact of the number of iterations on multiplicity in GBRTs. Furthermore, the paper proposes RashomonGB, a method to create an exponential number of ‘near-optimal models’ by training only a polynomial number of models. With more models in the Rashomon set, the use of RashomonGB can create several downstream benefits without any extra cost of training, shown empirically by the authors.", "questions": "My questions and comments mostly revolve around the RashomonGB formulation.\n- I don’t believe equation 4 is correct. A model formed from residual models that are present in their Rashomon sets at every step does not necessarily make a model that will be present in the Rashomon set overall. That’s because the composition of GBRTs occurs at the prediction level, while Rashomon sets are defined by the authors at the loss level. Equation 4 probably would have been true if the loss function had a linear relationship with the model predictions, which is not an assumption I see being made anywhere in the paper. This also makes me question the empirical results, because if the RashomonGB formulation isn’t precise, do the models across which the authors calculate multiplicity even belong to the same Rashomon set? Edit after rebuttal: Acknowledged and corrected.\n- Can the authors comment on why they compare two situations with different Rashomon parameters and make claims on their multiplicity? For example, Proposition 3 and the following paragraph. A Rashomon set would of course be bigger with a larger value of epsilon, and having that variability when talking about other trends doesn’t seem convincing to me. Edit after rebuttal: Confusion clarified.\n- What was the exact epsilon value used for the experiment? I couldn’t find it anywhere in the paper. Moreover, I hope that given the Rashomon sets for the RashomonGB setup were defined with T*epsilon as the new epsilon value, the same freedom was also given to retraining. Again, if the comparison was done across methods with different epsilon values (which might not be the case, but I don’t know the details), that does not make sense to me. Edit after rebuttal: Appropriate information added.", "limitations": "- A central piece of the paper is their method RashomonGB. While the authors do try to emphasize the importance of this method by highlighting the number of models that can be created using their method, just the number alone is not enough to imply a better method for measuring multiplicity. Even assuming that the comparisons are indeed fair (see Questions), the differences in multiplicity are not very severe, and that makes me wonder if combining pieces of various residual models actually gives us new interesting models or do we just end up with similar models as already seen during retraining. The authors acknowledge this briefly in their limitations paragraph. Edit after rebuttal: Appropriate details added and clarified.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Multiplicity in GBRTs, or generally any gradient-boosting algorithm, has not been studied in the literature, and so the authors provided a novel discussion, especially given the importance of these algorithms in tabular settings.\n- The paper provides several theoretical discussions backed by empirical support. The insights on the growing Rashomon set with iterations were quite interesting, although I have concerns about the validity of these insights (see Weaknesses).\n- Multiplicity quantification can be quite costly, and various methods in pursuit of reducing this cost can significantly benefit further auditing. The use of RashomonGB, as proposed by the authors, can be an important step in that direction for gradient-boosted algorithms.", "weaknesses": "- While the presentation of the rest of the concepts and the theoretical discussion were easy to follow, important details about the RashomonGB method and the details of the empirical setup were either missing (even from the Appendix) or imprecise. For instance, the Rashomon set of the gradient boosting algorithm isn’t going to simply be the iterative extension of Rashomon sets at every residual level, i.e., equation 4 is imprecise. Similarly, it seems that the epsilon value of the Rashomon set increases with more iterations, and thus it is confusing to me whether the insight that more iterations create bigger Rashomon sets is a result of multiple iterations or simply a result of bigger epsilon. See the section ‘Questions’ for more detailed comments and some follow-up questions. Edit after rebuttal: Acknowledged, correct and clarified.\n- There are other methods to measure predictive uncertainty in gradient-boosted algorithms. Some examples based on a cursory search (there might be more, as I’m not too familiar with GBRTs) - https://arxiv.org/abs/2205.11412 https://arxiv.org/pdf/1910.03225 https://arxiv.org/abs/2106.01682 - While I understand that prediction uncertainty is not the same as predictive multiplicity, the two are closely related, and when proposing a better method to measure multiplicity, the paper should compare itself with other stronger baselines than just retraining. Just as previous works have proposed using Monte Carlo Dropout (which was initially created as a method to measure uncertainty) as a measure of multiplicity, uncertainty measurement baselines for GBRTs could have been adopted to create reasonable baselines, and would have made the results a lot stronger. Edit after rebuttal: Acknowledged and added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s constructive feedback. We address the weaknesses, questions, and limitations point-by-point below.\n\nFor Weakness 1, please refer to our responses to Question 1 and Question 2.\n\nFor Weakness 2: as indicated, prediction uncertainty indeed differs fundamentally from predictive multiplicity. Prediction uncertainty, derived from a Bayesian perspective, seeks to reconstruct the distribution $p(Y|x)$ for a given sample $x$ and assess metrics such as variance or negative log-likelihood of $Y$, typically involving only one model without specific loss constraints. Conversely, predictive multiplicity involves evaluating multiple models within the Rashomon set that exhibit similar loss, thereby reflecting a variety of potential outcomes for the same inputs. To elucidate these distinctions, we have compared our re-training strategy, RashomonGB, with the prediction uncertainty methods cited by the reviewer—NGBoost [R1], PGBM [R2], and IBUG [R3]—using the UCI Contraception dataset. This comparison is detailed in Figure R.2 of the attached one-page PDF. For a rigorous comparison, we applied these prediction uncertainty methods to estimate $p(Y|x)$ (parameterized as Gaussian), sampled 1024 values of $y$, and computed the corresponding Rashomon set and predictive multiplicity metrics. The results demonstrate that RashomonGB encompasses the widest range of models, thereby providing consistently higher and more robust estimates of predictive multiplicity metrics. This comparison highlights the unique capabilities of RashomonGB in capturing a broader spectrum of potential model behaviors within the dataset.\n\nFor Question 1: when utilizing GBRT for classification tasks, the method actually performs a regression on the log-likelihood using the MSE loss (Lines 140-143). The MSE loss qualifies as sub-Gaussian, aligning with the assumption set up in Proposition 1. Additionally, the pseudo-residual of the MSE loss exhibits a linear relationship between the prediction and the output of each iteration (Line 597). This allows us to apply Proposition 1 to aggregate the losses across iterations, leading to the formulation of Proposition 2, which is a detailed extension of Equation 4.\nEquation 4 delivers the concept of constructing the overall Rashomon set by the Rashomon sets in each iteration. We thank the reviewer for noting the error and will change the equality to $\\supseteq$.\nWe will ensure that these assumptions and their implications are clearly articulated in the revised manuscript to eliminate any ambiguity.\n\nFor Question 2, we thank the reviewer for highlighting the potential confusion regarding the direction of reasoning related to $\\epsilon$. To clarify, we do not start with the assumption that $\\epsilon$ increases with each iteration; rather, this conclusion emerges from Proposition 3. Proposition 3 and the discussions in Section 3.3 indicate that with a constant $\\rho$—as defined in Proposition 1—additional iterations result in increased conditional mutual information, which in turn necessitates a larger $\\epsilon$, as detailed in Line 243. This dynamic is visually supported by Figure 2, where the conditional entropy—and consequently the mutual information—escalates as the boosting process progresses. This is because the Rashomon effect accumulates over the sequential learning problems tackled in each iteration, emphasizing the cumulative impact on the diversity within the model space. The Ablation study in Appendix E.3 further clarifies the selection of $\\epsilon$ through its iterations. Figure E.8 demonstrates that fixing $\\epsilon$ while re-training with different random seeds results in a decreasing percentage of models ($\\rho$ from Proposition 1) in the Rashomon set. This implies that to maintain a consistent $\\rho$, the chosen $\\epsilon$ must increase. This observation corroborates Proposition 3's findings on the relationship between $\\epsilon$ and $\\rho$. \n\nFor Question 3, for the experiments of reporting predictive multiplicity in Section 4.1, we report the Rashomon parameter $\\epsilon$ in the vertical axis (leftmost column) in Figure 3. For the experiments of fair model selection, we report the $\\epsilon$ in the caption of Figure 4. For the experiments of mitigating predictive multiplicity by model averaging, we report the $\\epsilon$ (in terms of the improvement of accuracy) in the vertical axis in Figure 5. Note that the $\\epsilon$ we report here is the overall Rashomon parameter after $T = 10$ iterations, i.e., T\\*$\\epsilon$. Moreover, we do not compare models with different $\\epsilon$ as it is clearly unfair. We provide an explanation on how to interpret the results of Figure 3 in Figure R.1 in the one-page PDF, please check!\n\nFinally for the limitation, indeed, as discussed in Section 5, while re-training with different random seeds offers a \"global\" exploration of models within the Rashomon set, RashomonGB conducts a \"local\" exploration. However, RashomonGB demonstrates greater efficiency than the re-training strategy, highlighting a trade-off between the efficiency of exploring the Rashomon set and the effectiveness of capturing model diversity. For instance, in Figure 3, numerous instances across different datasets show that the re-training strategy significantly underestimates predictive multiplicity metrics (e.g., VPR and Rashomon Capacity are reported as zero), particularly when $\\epsilon$ is small. This underestimation often occurs because re-training fails to gather a sufficient number of models. To the best of the authors' knowledge, finding the (sub-)optimal strategy for exploring the Rashomon set remains an active area of research. We will incorporate this additional discussion into the revised Section 5 to provide a comprehensive understanding of the trade-offs involved and the current state of research in this field.\n\nThanks again and we would be happy to provide more clarifications and answer any follow-up questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pO4zAVBB0", "reviewer": "Reviewer_XEPy", "summary": "This paper presents an approach that compute Rashomon set for gradient boosting algorithm where the set can be obtained through products over weak learners at each step rather than sampling them through retraining. The authors further proposed a dataset related Rashomon bound through sub-Gaussian assumption, where mutual information between hypothesis space and dataset shows the predictive multiplicity, which can further decomposed into model uncertainty and quality of data. Experiments show the proposed solution offers more models in Rashomon set than retraining given the same computation budget.", "questions": "Why boosting algorithms? \nFurther justification about the dataset related Rashomon set bound?", "limitations": "No hard limitation I can see.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The rough idea of the proposed approach is straightforward since decomposing Rashomon set search on boosting algorithm can be a \"standard\" operation given the unique residual learning property of boosting algorithms. The novelty of the proposed approach is probably more from \"our work is the first to explore the Rashomon effect for gradient boosting\".\n\nThe dataset related Rashomon set bound seems an interesting point. But it needs some justification for the key assumption of it (sub-Gaussian). Proposition 2 seems make sense given the positive relation between number of boosting iterations and Rashomon set (also for dataset size).\n\nExperiments in 4.2 seem interesting. I would love to see more experiments like it.", "weaknesses": "I got some difficult time to understand the introduction and abstract of this paper even I have read some literatures about Rashomon effect and predictive multiplicity. It is simply hard to read given the narrative there. Especially the second paragraph of introduction; it gets me confused and self-questioning my understanding of Rashomon effect from other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback! We clarify the weakness and answer the reviewer's question below.\n\nTo address the weakness pointed out, it would be helpful if the reviewer could specify which parts of the second paragraph in the Introduction are unclear or difficult to understand during the author-reviewer discussion phase. To enhance clarity, we will expand on the concepts introduced in both the Abstract and the Introduction.\nIn this paper, our objective is to explore both the positive (e.g., improved model selection as discussed in Section 4.2) and negative (e.g., predictive multiplicity as detailed in Section 4.1) impacts of the Rashomon effect using gradient boosting algorithms. These algorithms uniquely employ a sequential training procedure that focuses on learning the residuals of the data.\nThe Rashomon effect articulates that within a given hypothesis space, numerous models can achieve similar performance levels (such as 99\\% accuracy). These similarly performing models can be grouped into what is known as the Rashomon set. The positive aspect of the Rashomon effect is particularly significant in tasks associated with responsible machine learning, which often requires models to possess additional properties (such as group fairness) without significantly sacrificing performance. In essence, the pursuit of responsible machine learning is about finding models within the Rashomon set that meet these extra constraints (as exemplified by the fairness considerations in Section 4.2). The Rashomon effect thus provides assurance of finding viable solutions when optimizing for responsible machine learning goals.\nConversely, the negative aspect, termed predictive multiplicity, occurs when a model selected at random from the Rashomon set leads to inconsistent decisions for some individuals (e.g., affecting 5\\% of the samples as illustrated in Figure 3 of Section 4.1). This unpredictability can undermine the reliability of the machine learning process.\nBy elaborating on these concepts, we aim to resolve any confusion and reinforce the significance of our investigation into the dual implications of the Rashomon effect within gradient boosting frameworks.\n\nIn response to the Question, boosting algorithms are prevalently utilized for tabular datasets, particularly in the realm of trustworthy machine learning, such as fairness interventions detailed in Section 4.2. Notably, boosting algorithms have been shown to surpass deep learning on tabular datasets, as referenced in [35]. Despite this, existing theoretical frameworks that explore the Rashomon effect and predictive multiplicity have primarily focused on linear classifiers [55, 72], generalized additive models [15], sparse decision trees [74], and neural networks [42]. The unique sequential training procedure of boosting and its influence on the characterization of the Rashomon set remain poorly understood. This paper aims to bridge this gap, providing mathematical tools that could also extend to other sequential residual training schemes. We initially discuss our motivation for focusing on boosting algorithms in Lines 75-90 and 117-127. In the revision, we will reorganize this content to better highlight our motivation, as suggested by the reviewer.\n\nRegarding the assumption of sub-Gaussianity of the loss function, this represents a generalization beyond mere boundedness. While it is feasible to assume boundedness of the loss function—a common and practical approach readily achieved by clipping the loss, as mentioned in Lines 185-187—we opt for the sub-Gaussian assumption in Proposition 1 to allow for a broader analysis. This paper emphasizes the novelty of dataset-related bounds on the Rashomon set, where previous studies have largely concentrated on the hypothesis space. This perspective underscores our novel contribution to the understanding of the Rashomon effect and predictive multiplicity in machine learning.\n    \nWe will include additional explanations regarding the sub-Gaussian assumption in the revised Section 3. We welcome any follow-up questions and are happy to provide further clarification. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcA0QhNUXj", "reviewer": "Reviewer_E8Ec", "summary": "This paper proposes a method (RashomonGB ) to estimate the Rashomon sets/predictive multiplicity of gradient boosting models. It estimates multiple ($m$) models at each stage (effectively performing a local exploration) and then combine all such models in the end to construct $m^T$ models for Rashomon set computation, where $T$ is the number of iterations of the boosting. On several datasets the paper shows that RashomonGB performs better than re-training with $m$ seeds, in that at the fix $\\epsilon$ (loss difference) level, RashomonGB tends to show more predictive multiplicity.", "questions": "1. What's $\\epsilon_{t_1}$ (and $\\epsilon_{t_2}$) in L243-L244? Isn't epsilon a quantity set by the user? \n\n\n2. In L282-283, do we construct 10 final models and 1024 for re-training and RashomonGB, respectively? If only 2 out of $m$ models are used why train $m$ of them (L282-283) for RashomonGB? \n\n3. Related to the above, I originally thought there is a model \"filtering\" step in each iteration $t$, and wonder how $\\epsilon_t$ is set for each iteration. However, from L282-283 it seems like we just randomly pick a few models and brute-force combine all weak models for the final Rashomon set exploration. Could the authors clarify?\n\n4. Are Fig 4 measured on the test set? If so, then it's not clear how useful this is as we cannot choose models basing on test performance - did the authors try picking models on the frontier basing on the validation set and then plot this on the test set? Right now, due to the sheer number of final models generated by RashomonGB, it's unclear if the models with better trade-off are just lucky.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Predictive multiplicity is an important topic. The paper is generally clear and well-written. The proposed method is a sensible first method for boosting algorithms, which was previously underexplored. I think the proposed method is likely adopted by people who care about this problem as it's intuitive and easy to implement.", "weaknesses": "1. The current exploration strategy is fast to compute, but I'm not sure if this follows the motivation of Rashomon set very well. While the authors mention one example on the Contraception dataset where re-training underestimates the predictive multiplicity, in general RashomonGB might create models that are more correlated than normal (because the \"backbone\" is the same GB model), thus underestimating the predictive multiplicity. Right now, the conclusion shows otherwise probably because the number of re-training is too small. \n\n2. Regarding the experiment, if I read this correctly, currently we use more compute for RashomonGB as well (by combining different weak models), so it is also not quite a fair comparison in my opinion. I would be very interested to see some estimate of how much compute RashomonGB saves against re-training, by running more re-training and see when are the metrics in Fig3 in the two methods become comparable.\n\n\n\nminor: one \"RashomonGB\" in L290 should be \"re-training\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback and encouragement. Below, we systematically address each weakness and question raised in the review. \n\nFor Weakness 1, the estimates in Figure 3, derived from both re-training and RashomonGB, utilized the same training cost. Specifically, each method required training T\\*m = 10\\*10 = 100 weak learners (except CIFAR-10; cf. Line 282). The re-training approach, using different random seeds, explores the Rashomon set in a \"global\" fashion, potentially yielding a more diverse set of models within this space. Conversely, RashomonGB explores the set in a more \"local\" manner, focusing on models that incorporate the same weak learners. Despite this, under identical training costs, RashomonGB can explore exponentially more models than the re-training strategy. It's important to note that for complex hypothesis spaces, like Gradient Boosting (GB) used here, any method, including re-training, will likely underestimate predictive multiplicity due to the sheer computational impracticality of fully exploring the Rashomon set. This introduces an inevitable trade-off between the efficiency (training cost) and the effectiveness (degree of underestimation) in estimating predictive multiplicity. In Figure 3, our goal was to highlight the efficiency and effectiveness of RashomonGB under equal training costs. However, as the number of re-training increases—thereby increasing the training cost—the re-training method may surpass RashomonGB in effectively estimating the diversity within the Rashomon set. \n\nRegarding Weakness 2, although the computational costs for re-training and RashomonGB are equivalent, RashomonGB significantly reduces the time required to obtain a model. This is because RashomonGB explores an exponential search space, making it much more efficient. This efficiency is compellingly demonstrated in the comparative analysis of computational times shown in Table E.4 of Appendix E.2, where RashomonGB's model generation speed significantly outpaces that of re-training. For instance, for the ACSIncome dataset, both methods recorded a training time of 54.67 seconds. However, while the inference time per model for re-training is 0.4 seconds, it is only 0.02 seconds for RashomonGB. This means that, under the same training cost, RashomonGB is 20 times more efficient in terms of generating models from the Rashomon set compared to the re-training strategy. We will address this comment by clearly directing readers in the revised main text to the additional experiments detailed in the Appendix. This will help ensure that the relevant information is easily accessible and comprehensible.\n\nFor Question 1, indeed, $\\epsilon$ is a parameter configured by the user. As stated in Lines 243-244, maintaining the same $\\rho$—defined as the probability in Proposition 1—results in more iterations increasing the conditional mutual information, which in turn leads to a larger $\\epsilon$. This insight serves as a practical guideline advising users against choosing smaller values of $\\epsilon$ in subsequent boosting iterations. \nThis effect is further illustrated in the Ablation study detailed in Appendix E.3. Figure E.8 demonstrates this by fixing $\\epsilon$ for each iteration, re-training with different random seeds, and computing the percentage of models (i.e., $\\rho$ in Proposition 1) in the Rashomon set for each iteration. It is observable that, with the same percentage (i.e., the same $1-\\rho$), $\\epsilon$ increases. This observation reinforces the results suggested by Proposition 3, validating the relationship between $\\epsilon$ and $\\rho$ under consistent conditions.\n\nFor Question 2, indeed, we implemented the re-training strategy using 10 different random seeds for the Gradient Boosting with 10 iterations. This strategy involved training 100 weak learners (10 per iteration). Ideally, RashomonGB can generate up to $10^{10}$ final models. However, as noted in footnote 5, even selecting 3 models per iteration for RashomonGB would yield over 59,000 final models, which exceeds our storage capabilities. Consequently, we chose to select 2 models per iteration for RashomonGB. Comparatively, if we train only 2 models in each iteration, the re-training strategy results in just 2 final models, whereas RashomonGB still generates 1024 models. This underscores the superior efficiency of RashomonGB in generating a higher number of models under the same training cost, thereby providing a broader exploration of the model space. \n\nFor Question 3, indeed, there is a filtering process in place that screens out models with an MSE loss greater than 0.1 (i.e., $\\epsilon_t = 0.1$) and retains models with an MSE loss smaller than 0.01 until $m=10$ models are collected at each iteration. \n\nFor Question 4, the models situated at the accuracy-fairness trade-off frontier were selected using a hold-out validation set post-training, and the results depicted in Figure 4 were evaluated using the test set. Compared to the re-training strategy, the lower cost of obtaining models through RashomonGB enhances the likelihood of identifying a model with a more optimal operation point. The advantage of RashomonGB becomes even more pronounced when dealing with larger datasets. In such scenarios, re-training and re-training-based fairness intervention algorithms, such as Reduction (and FaiRS) and Rejection, may incur significantly higher training costs. \n\nWe will add the additional explanation and results, and fix the typo (e.g., \"RashomonGB\" in L290) in the revision. Thanks again and we would be happy to provide more clarifications and answer any follow-up questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their time and effort in reading and commenting on the manuscript. We appreciate that the reviewers found that the paper **“study a novel problem”** (Reviewer HS14, JGsC, XEPy, and E8Ec), **“has robust and interesting analysis on dataset-related Rashomon set bound”** (Reviewer HS14, JGsC and XEPy), and **"propose RashomonGB which is easy to implement and its practical utility validated with real-world datasets "** (Reviewer E8Ec and HS14). Below, we outline the specific enhancements and changes that will be incorporated into the revised version of our paper.

- We clarify the contributions of this paper. The Rashomon effect and predictive multiplicity are not yet studied for gradient boosting algorithms, which is widely used for tabular datasets, especially in the field of responsible machine learning. With empirical studies, we present both the positive (fair model selection in Section 4.2) and negative (predictive multiplicity in Section 4.1) impacts of the Rashomon effect for gradient boosting, along with two algorithms to mitigate predictive multiplicity (Section 4.3). The analysis we developed here is not limited to gradient boosting, i.e., decision tree weak learners, but can also be applied to other sequential learning schemes with neural network (GrowNet) or linear regression weak learners (CIFAR-10 results in Figure 3 and Figure E.9 of Appendix E.4). 

- We clarify and add more discussions on the theoretical results regarding the dataset-related Rashomon set bound. We explain that the sub-Gaussian assumption for the loss random variable is a generalization of boundedness, and is a pratical and common assumption for information-theoretic bounds. Moreover, we explain that the Rashomon effect on the residual cumulates with the number of iterations and increase the Rashomon parameter under the same $\rho$. We also refer the reviewers to Figure E.8 in the Appendix, which demonstrates that fixing $\epsilon$ while
re-training with different random seeds results in a decreasing percentage of
models ($\rho$ from Proposition 1) in the Rashomon set. This observation corroborates
Proposition 3’s findings on the relationship between the mutual information, $\epsilon$ and $\rho$.

- We clarify the comparison between RashomonGB and Re-training for estimating predictive multiplicity metrics and fair model select, and how to properly interpret Figure 3 in the main text in Rebuttal Figure R.1. We first perform re-training with different random seeds and perform RashomonGB by using the same weak learners, i.e., the training cost of RashomonGB and Re-training are the same and hence the comparison presented in the paper is fair. 

- We provide an additional experiments to compare RashomonGB with three baselines in estimating prediction uncertainty for gradient boosting regression trees, including the NGBoost [R1], PGBM [R2], and IBUG [R3] in Rebuttal Figure R.2. Note that prediction uncertainty aims to estimate $P(Y|x)$ given a sample $x$ while predictive multiplicity aims to construct a Rashomon set with multiple models with similar losses. As observed in Figure R.2, RashomonGB is able to cover more diverse models than NGBoost, PGBM, and IBUG, and hence less under-estimates predictive multiplicity metrics.


Finally, we would like to point out that the core problem in Rashomon effect research is to explore models in the Rashomon set. For a general hypothesis space it is computationally infeasible. Therefore, there is a fundamental trade-off between the efficiency and effectiveness of exploring diverse models in the Rashomon set. Re-training is the de facto method that gives the most diverse models in the Rashomon set while also the most inefficient. The RashomonGB trades in the diversity by searching for "local" models, in return of improving the efficiency.
Please feel free to follow up! We very much welcome further discussions.






[R1] Duan, Tony, Avati Anand, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Ng, and Alejandro Schuler. "Ngboost: Natural gradient boosting for probabilistic prediction." In International conference on machine learning, 2020.

[R2] Sprangers, Olivier, Sebastian Schelter, and Maarten de Rijke. "Probabilistic gradient boosting machines for large-scale probabilistic regression." In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining, 2021.

[R3] Brophy, Jonathan, and Daniel Lowd. "Instance-based uncertainty estimation for gradient-boosted regression trees." Advances in Neural Information Processing Systems, 2022.

Meta-review: The authors introduce a novel approach to characterizing predictive multiplicity for gradient boosting. The paper also explores the implications for finding a group fair model with a sufficient level of accuracy. Reviewers appreciated the information theoretical analysis of the Rashomon effect in this setting, and saw a potential for impact of the proposed method given the prevalence of GB in tabular learning, along with the importance of fairness in tabular settings. Some concerns around the presentation and attention to detail seem to have been resolved during the rebuttal period.
Decision: Accept (poster)