Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation
Authors: Hsiang Hsu
Guihong Li
Shaohan Hu
Chun-Fu Chen
Venue: ICLR 2024 poster (2024)
Keywords: Rashomon Effect
Rashomon Set
Predictive Multiplicity
Dropout
Speedup.
Primary area: societal considerations including fairness, safety, privacy

Abstract: Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-equally-optimal performance, yet generate conflicting outputs for individual samples.
This presents significant concerns, as it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness in practical applications.
Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need to explore all such almost-equally-optimal models, known as the Rashomon set, in potentially huge hypothesis spaces. 
To address this challenge, we propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set.
We provide rigorous theoretical derivations to connect the dropout parameters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation.
Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation, with runtime speedup up to $20\times \sim 5000\times$.
With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.

Published as a conference paper at ICLR 2024
DROPOUT-BASED RASHOMON SET EXPLORATION FOR EFFICIENT PREDICTIVE MULTIPLICITY ESTIMATION
Hsiang Hsu', Guihong Shaohan Hu', and Chun-Fu (Richard) Chen'
JPMorgan Chase Bank, N.A., USA
The University of Texas at Austin, TX, USA
(hsiang. hsu, thaohan.hy, richard. of chen www.chhennneeccceeeeooooo ighoutexas.edu
ABSTRACT
Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-cqualll--opimm per- formance, yet generate conflicting outputs for individual samples. This presents significant concerns, it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness practical applications. Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need explore all such dimost-equully-optimaa models, known as the Rashomon set, in potentially huge hypothesis spaces. To address this challenge, we propose novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We provide rigorous theoretical derivations to connect the dropout parame- ters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation. Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive mul- tiplicity metric estimation, with runtime speedup up to 20x 5000x. With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.
INTRODUCTION
The Rashomon effect, first introduced in Breiman [2001], describes the phenomenon that there not just "best" explanation for the data, but many almost-equally-optinne models. The deliberation of Rashomon effect has recently prevailed due to the increasingly complicated learning architectures that exacerbate the snder-specification of optimization problems [D' Amour al., 2022]. The col- lection of these dimost-equalll-opiinie Marx al., 2020]. The existence of Rashomon sets facilitates finding -umost-equall-opptine els that further carry practically-desires properties for practitioners, such as interpretability [Rudin, 2019], epistemic patterns (e.g. causality) [Hancox-Li, 2020], fairness [Coston al., 2021], or com- pliance with domain knowledge [Hasan & Talbert, 2022].
models is termed the Rashomon set [Semenova et al., 2019;
mod-
Despite the newfound potential, the Rashomon effect introduces a burgeoning challenge in classifi- cation problems, known predictive multiplicity-tth phenomenon where almost-cqualll-oppinn classification models assign conflicting predictions to individual samples, i.e., the predictions given to individual could be determined by an arbitrary model in the Rashomon set. When predictive multiplicity left out account, an arbitrary choice single model in the Rashomon set may lead to systemic exclusion from critical opportunities, unexplainable discrimination, and unfairness, to individuals [Creel & Hellman, 2021]. Moreover, recent studies suggest predictive multiplicity additional dimension to evaluate the trade-off's of privacy-assuring mechanisms such dif- ferential privacy [Kulynych et al., 2023] and fairness interventions [Long et al., 2023]. Therefore, measuring predictive multiplicity has been increasingly recognized key aspect when reporting model performance aside from its accuracy, for example, in Model Cards [Mitchell et al., 2019].
Several metrics have been proposed to measure predictive multiplicity by quantifying either the spread of output scores from models in the Rashomon set such as viable prediction range [Watson-
Work done during internship at JPMorgan Chase Bank, N.A.
Undde--pecificatton means there is no unique solution optimization problem, e.g. the empirical risk minimization that widely used in modern machine learning [TTnny et al., 2022].

---

Published as a conference paper at ICLR 2024
Daniels et al., 2023] and Rashomon Capacity [Hsu & Calmon, 2022], or the inconsistency of deci- sion such as ambiguity/diserepancy [Marx et al., 2020] and disagreement [Kulynych et al., 2023], to name a few. The exact computation of these metrics relies on full access to models in the Rashomon set, which is computationally infeasible and memory inefficient when the hypothesis space of can- didate models is large and complex. Therefore, current strategies to compute predictive multiplicity metrics either focus on special hypothesis spaces (c.g., linear classifiers [Marx et al., 2020], sparse decision trees [Xin et al., 2022], and generalized additive models [Chen al., 2023],, or by re- peatedly re-training models with different random seeds to explore the Rashomon set when there no special structures to relieve the computational overheads. However, re- re-training is still time- consuming when learning with large datasets. Thus, how to efficiently estimate predictive multiplic- ity metrics for highly complicated models, esp. neural networks, still remains an open problem.
In this paper, we propose an efficient dropout-based Rashomon set exploration framework that is easily applicable neural networks without arrenuously-eppeatee model re-training. The dropout technique was first found useful to reduce over-fitting [Srivastava et al., 2014], and is later con- nected with Bayesian approximation Ghahramani, 2016]. However, Gal & Ghahramani [2016] do not consider prediction uncertainty with the notion of the Rashomon effect-they consider prediction uncertainty among all possible models in the hypothesis space instead of models in the Rashomon set (see Appendix C for further comparisons). To the best of our knowledge, this work is the first in using dropout techniques for model exploration in Rashomon sets and efficient estimation of predictive multiplicity metrics. We provide a rigorous theory on how hyper-paramette parameters of dropout and network architectures control the loss deviations of Rashomon sets for ferd-forward neural networks (FFNNs). Through meticulous parameter selection, a dropout model is highly likely to belong to a Rashomon set as the number of model parameters approaches infinity. Besides theoretical discussions, we conduct empirical stud- ies of measuring predictive multiplicity metrics using dropout with multiple datasets from diverse domains, including financial analytics, medical prediction, large-scale images classification, and human detection. Exploring models in Rashomon sets with our framework attains 20 5000 speedup compared with the baseline methods such as re-training. Finally, we demonstrate the mit- igation of predictive multiplicity using the proposed dropout- based Rashomon exploration with (i) ensemble methods and (ii) model selection with smaller predictive multiplicity.
neural networks to measure prediction uncertainty [Gal &
Omitted proofs, additional explanations and discussions, details experiment setups and training, and additional experiments are included in the Appendix. Code to reproduce our experiments can be accessed
2 BACKGROUND AND RELATED WORK
We consider a dataset D {(), where each pair (x) consists of feature vector x; [x...... , x T E X S Rdd and a target yi E y. We denote by H. a hypothesis space, set candidate models parameterized by w E W. ie., H & {lt : X' - y : w E W}. The loss function used to evaluate model performance is denoted by e: D/=2 x y + IR+ and L(w, D) A [D] E(x,v))DD Eh (x:).y:) y:) denotes the empirical risk evaluated with dataset D. When the context is clear, we simply write T(w) instead of I(w,, D) for the risk, and use w to denote a model. For c-class classification problems, i.e., y = [c] and í is the c-dimensional probability simplex A. & {(r).- rr) E [0, 1]": Σ Ei T2 = 1), the loss function could be the cross-entropy loss or the Brier score loss [Brier, 1950] Les (w) A + -. /// (x:)- yvlla- For regression problems, y = - IR, and the loss function is mean square error (MSE) loss LMSE (w) & 1 Ei= (x; - -y In this paper, we mainly focus on predictive multiplicity for classification problems. Finally, for a vector v, diag(v) is matrix with v diagonals and 0 on off-diagonals, and for a matrix V, diag(V (V) is matrix that only keeps the diagonal entries.
The Rashomon set. We define the Rashomon set as the set of all models in the hypothesis space H whose empirical risks are similar to that of a given empirical risk minimizer w" argmin L(w). Formally, given a Rashomon parameter e 2 0, the Rashomon set2 is defined as [Semenova et al., 2019]:
R(H, D. w*,e) & [low E H; H( D) < L(h ; D) + E).
(1)
"The Rashomon set is defined regarding any given dataset D, even for out-of-distribution data.
2

---

Published as a conference paper at ICLR 2024
Here, determines the size of the set. We omit the arguments of R(H, D, Dw"","" when they are clearly implied from context. The Rashomon set is at the core of measuring predictive multiplicity, as introduced next.
c) later in this paper
Measuring predictive multiplicity. There are various metrics to quantify predictive multiplic- ity across models in R. by either considering their decisions (thresholded predictions/scores after argmax) or output scores. Due to space limit, we summarize the mathematical formulations of existing predictive multiplicity metrics in Appendix B.
Decision- based metrics include ambiguity and discrepancy, which measure the proportion of sam- ples in a dataset that can cause models in the Rashomon set to make conflicting decisions [Marx et al., 2020], and disagreement, which is the probability that any two models in the Rashomon set output different decisions for given sample [Kulynych et al., 2023]. On the other hand, a finer- grained characterization of predictive multiplicity examines the output scores (i.e., vectors in Ac). For example, Long et al. [2023], Cooper et al. [2023] and Watson-Daniels et al. [2023] quantify predictive multiplicity by the standard deviation, variance and the largest possible difference of the scores (termed viable prediction range (VPR) therein) respectively. The Rashomon Capacity (RC) measures score variations in the probability simplex with information-theoreed quantities that can be generalized beyond binary classification. Despite score conflicts can not be directly translated to decision conflicts, score-based metries avoid potential over-estimation of predictive multiplicity compared to decision-based metrics (see [Hsu & Calmon, 2022, Fig. 2] for a detailed discussion).
Exploring models in the Rashomon set. The computation of predictive multiplicity metrics (cf. Appendix B) requires an exact characterization' of a Rashomon set, i.e., having full access to all of its models. When H is large (e.g., a neural network architecture), exhaustively finding all models in the Rashomon set is computationally infeasible. Therefore, to approximate a full Rashomon set (H. D, w", <), we define an empirical Rashomon set with m models, as follows.
(2) The empirical Rashomon set RH,, w", e) is a subset of the Rashomon set defined in (I). Pre- dictive multiplicity metries can then be evaluated with the empirical Rashomon set instead of the Rashomon set. Note that any estimation based on the empirical Rashomon set is an under-estimate of the true predictive multiplicity. In consequence, an empirical Rashomon set is "better" approx- imate of the true Rashomon set than another empirical Rashomon set if it leads to higher estimates of predictive multiplicity metrics. As m increases, the empirical Rashomon set better recovers the Rashomon set, leading to more precise estimation of predictive multiplicity metrics. Therefore, how to efficiently acquire vast amount of models in a Rashomon set becomes the core problem of measuring predictive multiplicity.
RT D,,,D,"",, A {lwa -. ",w E Hi Llb., D) < L(t D) + Vi [m]]-
In practice, the m models in the empirical Rashomon set can be obtained either by re-training with different initializations (e.g., different random seeds to initialize the model weights, different data shuffling for stochastic gradient descent, etc.), or by adversarial weight perturbation (AWP) [Hsu& & Calmon, 2022, Section 4]. The re-training strategy views training procedure T to be randomized. Accordingly, we can denote a random variable T(D) that outputs all possible models trained with procedure T on D. Different models in T(D) can be induced by using different random initialization'. By re- ve-training models and rejecting those that disobey the loss deviation constraint in (2), we are able to collect m models for the empirical Rashomon set. The AWP strategy, on the other hand, aims to perturb the weights of pre-trained model such that the output scores of a sample are thrust toward all possible classes, again under the loss deviation constraint (cf. Appendix B for more details). Both re-training and AWP require repeatedly training models, which is time-consuming and hinders practitioners from efficiently estimating predictive multiplicity metrics.
Another strategy besides re- re- training and AWP is to explore sparse models in the Rashomon set. For example, Xin al. [2022] prove that large portions of the decision tree in the hypothesis space do not contain any members of the Rashomon set and can safely be excluded, and provide an algorithm and data structure to completely enumerate and store all models in the Rashomon set for sparse decision trees. More recently, Chen et al. [2023] study an interpretable, predictive model called generalized
It is possible for simple models such as an exact characterization of the Rashomon for ridge regression [Semenova et al., 2019, Section 5.1] or an exact computation of ambiguity/discrepaney for linear classifiers hy mixed integer programming [Marx et al., 2020, Section 3).
*See Kulynych et al. [2023] and Semenova et al. [2019] for more details about the re-training strategy.
3

---

Published as a conference paper at ICLR 2024
additive models (GAMs), and use high-dimensional ellipsoids to approximate the Rashomon sets of sparse GAMs. Despite these progress, how to explore Rashomon sets and measure predictive multiplicity for more complicated models, especially neural networks, still remain unclear. This paper intends to bridge this gap using dropout techniques, as discussed next.
3 EXPLORING THE RASHOMON SET WITH DROPOUT
The dropout technique, originated from the concept of dilution [Hertz et al., 1991], is family of stochastic techniques for regularization to prevent over-fitting [Hinton et al., 2012]. As implied by its name, the dropout technique randomly removes neurons time, and can be interpreted as implicitly averaging over an ensemble of sparse neural networks with different configurations during training [Srivastava et al., 2014].
neural network each training
In addition to applying dropout in training, Gal & Ghahramani [2016] apply dropout at inference time to estimate output uncertainty of model. They interpret dropout as a variational approximation of a deep Gaussian process, which is Bayesian framework that produces a probability distribution of model outputs. The underlying distribution can then be used to estimate the variance for a cer- tain input, indicating the uncertainty of the model. However, the dropout inference studied in Gal & Ghahramani [2016] has not taken the Rashomon effect into consideration, i.e., they consider the out- put uncertainty of all possible models in the hypothesis space instead of all almost-equally-opiinn models in the Rashomon set; see Appendix C for a more thorough discussion. In this case, predictive multiplicity and output uncertainty are over-estimated, since models without comparable accuracy are also counted in, and therefore dropout inference is not directly applicable to measuring predic- tive multiplicity. To address this issue, here, we illustrate how dropout parameters control the loss deviations of Rashomon sets for linear models, and eventually for FFNNs.
3.1 FORMULATION OF DROPOUT TECHNIQUES
We start with the mathematical formulation. Consider model weights W E Rd, and define dropout random variables as [Z,. Zz, Za]. Let D2 = diag(z) Raxd the weights after dropout can be denoted as w = D,w. There are two common choices of the random variables Z₁- Bernoulli dropout (also called standard dropout), adopts Zí to be i.i.d. Bernoulli(1 p) random variables, where E [0, 1] is the probability of dropping out a weight (i.e., the dropout rate). On the other hand, Gaussian dropout applies i.i.d. Gaussian multiplicative noise, Gaussian a). on the weights [Wang& & Mamning, 2013; Kingma et al., 2015]. Gaussian dropout can also be interpreted as a variational information bottleneck layer [Rey Mnih, 2021].
Suppose we have m realizations of the dropout matrix, De...... .D... At the inference time, we may apply each dropout realization Dz, to a fixed model weight w", where each realization is a new, sparse model İID., w'. We would like to note that even though applying dropout is much more computationally efficient than re-training and AWP to obtain a vast amount of models, if, however, the dropout method leads to a huge loss deviation T.(hD., w-) [(hw-) ), dropout models are not likely to be in the Rashomon set unless the Rashomon parameter is set to be large. Therefore, in the next section, we investigate the connection among dropout parameters (i.e., rate p for Bernoulli dropout and variance a for Gaussian dropout), loss deviations, and the Rashomon set.
3.2 THE RASHOMON SET ON RIDGE REGRESSION WITH DROPOUT
We start the analysis with a special case of Bernoulli dropout on ridge regression and its connection to the notion of Rashomon sets. Consider parametric hypothesis space of linear models H = {hw (x)= = w T x: w E (Rd), ridge regression aims to minimizes the penalized sum of squared error (SSE) loss, T w) Xw 1| for a dataset (X,y) ) E R xR", x= i.e., min (ssS(())+ -/|w w|2 wEW
where À is the regularization strength. The solution of ridge regression is given by w* = (X X + ALa) X y [Hastie et al., 2009]. Denoting the weight solution after dropout to be w) the loss LSSE (w") becomes a random variable. Choosing Z, ~ bernoulli(1 - p). the goal is to characterize the deviation between LSSE (IsSE(w)) and LSSE (w") ). In fact, if we define the loss deviation as LSSE(w>)) LSSE (ssE( where w' و (1 p)w", then is random variable well, and its expectation can be computed, as shown in the following proposition.
D
'The difference is that dilution randomly removes connect of neuron while dropout randomly removes the entire neuron (and all its connections to other neurons). For variants of dilution and dropout, see Hertz [2018, Chapter 3] and Labach et al. [2019].
4

---

Published as a conference paper at ICLR 2024
Proposition 1.
Consider Bernoulli dropout with rate p. and denote the dropout weights wh
Dew". The loss deviation e LSSE(WD) - LsSS(w'") satisfies
pdagx X)w*. Bx--bamoul((1-p)11 = Ex--bernoull(( 1 1pp) [LSSE( - LSSE( (o's((w")) p(1 -p)w" diag(X
(3)
Moreover, the features of data matrix X are linearly independent and normalized, i.e., x It, have Bg--bemouli(( (1) p(1-p) (1+A)² lyyl..
Proposition illustrates that a dropout model Wwp. expectation, belongs to the Rashomon set R(w"", pp1 p)w" (
We provide Monte Carlo sim- ulation with Gaussian synthetic dataset with different dimension in Figure I. It clear that the expected sim- ulated losses follow the theory, and sheds light the pos- sibility to control loss deviations of Rashomon sets using dropout. Proposition can be further improved. First, (3) could be defined regarding the original weights w' instead of w' Second, (3) only specifies that dropout models belongs to Rashomon set in expectation. In the rest of this section, we aim to improve this two limita- tions, and show that the probability is controlled by the feature dimension
200 033 010
211 0.23 6.T%
-.15 1355 D.0 3.53 0090
Figure 1: Proposition 1 with 20k models and dimension dl {10, 50, 100, 2003. creases, the variance of loss enlarges, while the mean still matches the theory in (3).
3.3 THE RASHOMON SET OF LINEAR MODELS WITH DROPOUT
We start with ridge regression, which Rashomon set Rridge (w".e, e) has an analytical form; in fact, it an d-dimensional ellipsoid centered at w [Semenova al., 2019, Theorem 16], i.c.
Rridge (w" c) (w E W; (w -w"""" "x"T""Xxx
(L)(( w") S c).
(4)
With the exact form of the Rashomon set, we can further compute the probability that dropout model D2 w is in the Rashomon set, i.e., Pr (D,w* RRidge (w0r)))
Proposition 2. Without loss of generality, assume X (otherwise we can always "whiten" the data matrix X) and the weights are bounded, i.e., 1|w122 < M, then the probability that the model w after dropout is in the Rashomon set is lower bounded by
-
[1 (1+
PM
Z Lid. bernoulli -1 p);
Pr (Dzw" E Riigel (w", wo, .0) 2 (1+ aM if Zz Gaussian(1,
(5)
i.d.d
Proposition 2 suggests that the dropout parameters, rate p and variance are important for con- trolling the probability. Indeed, a trivial case when p = a = 0 (i.e., no dropout), the probability Pr (Dew* E Rrigg (w", } } is always 1. Morcover, let 5 > 0. if p ((d=)) for Bernoulli dropout Zi ~ Bernoulli(1 p). and Odd-6) for Gaussian dropout Z: Gaussian(1, a), both dropout mechanisms leads to lim Pr DDw" w Rridge (w", .c)] 1 (cf. Appendix A.6 for details).
Similar results also hold if we pass the outputs (x,)) = w x; through a softmax function softmax(t) = 1/(1 exp(-t)) to get output scores, and use the Brier score loss (ws(w) to minimize the average difference between labels and scores for classification problem. Using the 1-Lipschitzness of the softmax function, the next proposition shows the probability that dropout models belong to a Rashomon set RBrier (ooier(w",. (w* {w E W; Les(w)- Les (w") ) c}.
T 1|xx1112 and suppose the weights bounded, i.e., ||wl| < M, for both Bernoulli dropout Zx ~ Bernoulli(1 - p) and Gaussian dropout Z, ~ Gaussian(1 (1, at), we have
Proposition 3. Let [|
(1 2) P
Zf i.idd.
bernoull -
Pr (Dzw* E R w ,c)
(6)
(1 A) Mddli
if Gaussian(1,a))
s.i.d.
The bound in Proposition 3 is different from that Proposition 2 with merely a factor of There- fore, the asymptotic behaviors of the probability hold again d approaches infinity. Precisely,
5

---

Published as a conference paper at ICLR 2024
as long p and of order ((+)()+)) and the norm of inputs is finite, 6 > 0, have for both Bernoulli and Gaussian dropout mechanisms.
lim Pr (Dew" Rerier(w", .)] = d=500
Proposition 2 and together show that for regression and classification problems, as long as the dropout parameters carefully controlled with respect to the feature dimension d. dropout leads to models in the Rashomon set with high probability.
3.4 EXTENSION TO FEED-FORWARD NEURAL NETWORKS In the previous section, we focus on simple linear models. Here, we include the discussion between dropout and the Rashomon set for FFNNs with hidden layers". We use the fact that activation functions (-)) commonly used in neural networks such as ReLU, softmax, and tanh, and non-linear functions such as max-pooling, are Lipschitz with constant 1 (cf. Virmaux & Scaman [2018]).
Consider K K-hidden-layer neural network as follows:
bw (x:) WK (Wk-I-
(W]x))...))
where We 1 E [K] are the weight matrices, where mg = d and mæ c. Let W (W1,-. W...... W be the collection of all the weights and W [W is empirical minimizer. Now, consider i.i.d. dropout matrices D₃ for each weight matrix We respectively, the output after dropout is then
ww D (xx) (Dx Wk) (
(D) x:) x.)...)
where W (Dx W is the collection of all dropout weights. Next, we prove that by care- fully controlling variance of Gaussian dropout, the deviation of losses before and after dropout can also be controlled.
Proposition 4. Consider K-hidden-layer neural network (x:) defined in (7) with [| S M for all & E [K], and Gaussian dropout, i.e., all diagonal entries Dr itd ~ Gaussian(1, a.). For
both MSE loss L(-)- = LMSS(.) and Brier score loss L(-) = LBs(-), with probability at least p.
L(W;) - L(W") S p- M | (Ima + +1)* -1).
(9)
where m max my and X 1 = 7 KE[K]
Σ C
Proposition 4 indicates that both Bernoulli and Gaussian dropouts on neural networks lead to models
in the Rashomon set R (W", pp M
|
p (ma + 1)* -1)) with probability 1 -p.
MEASURING PREDICTIVE MULTIPLICITY WITH DROPOUT MODELS
In previous sections, we have shown that the probability of dropout models in Rashomon set could be controlled by the hyper-parameters of neural network architectures and dropout. These dropout models can then be used to construct the empirical Rashomon set defined in (2), and es- timate predictive multiplicity metrics. Note that not all estimators of predictive multiplicity metrics carry a theoretical analysis of its statistical properties such as consistency and sample complexity (cf. Appendix B for details). In Appendix A.7, we provide additional theoretical analysis regarding a sample complexity bound of estimating (a surrogate metric of) score variance for a single and mul- tiple samples, using Hoeffding's 's inequality [Hoeffding, 1994] and Gaussian annulus theorem [Blum et al., 2020, Theorem 2.9].
Next, we show empirical results of estimating predictive multiplicity metrics with the empirical Rashomon set obtained from the dropout techniques. We estimate the predictive multiplicity metrics with 6 UCI datasets [Asuncion & Newman, 2007], including three from the financial domain (Adult Income, Bank Marketing, and Credit Approval). and the rest from the medical domain (Dermatol- ogy, Mammography, and Contraception). These domains are selected since predictive multiplicity therein could cause critical consequences of injustice. See Appendix D. for detailed descriptions and processennn of the datasets, and Appendix D.2 for detailed training setups. The baseline methods we adopt to compare with dropout are re-training with different seeds [Semenova et al.,
'The analysis can also be applied to convolutional neural networks as they are FFNNs that use filters and
pooling layers [Goodfellow al., 2016].

---

Published as a conference paper at ICLR 2024
Demoulliitate
Adolt
0.023
0.000
0.0000 0,0055
00
0.01
0.0
Bank
0.000
0.0000
122
0.02 0.0
0.03
0.00
0.5
0.00
0.00
0.55
08
03
0.0
Contraccation Nommegraphy Dermatat
0.05
0.00
0.3
0.25
2.50
0.00
0.01
1.0
1.2
0.200.500.2
0.05 0.15
0.82
GausslanVar.
VPR
ScoreVar.
RC
Disagreement
Ambiquity
Discrepancy
The Tta-traningg
Berralll Brepout
GounoNat Dropout
AMNT
Figure 2: Loss vs. dropout parameters and the corresponding predictive multiplicity metrics of the baselines with UCI datasets. The figures in row share the same y-axis for the loss difference e, i.c. the Rashomon parameter in (2). Both Bernoulli and Gaussian dropouts give higher multiplicity estimates than t--trinnng under the same loss deviation constraints. In other words, dropout is much more effective than re- re-training. Despite AWP outperforming all the other methods, it the most computationally expensive.
2019], and the adversarial weight perturbation algorithm [Hsu Calmon, 2022]. All the models here have the same architecture- neural network with single hidden layer of 1 neurons.
Figure summarizes the estimation of the 6 predictive multiplicity metrics, introduced Section 2, using the empirical Rashomon set obtained with dropout models. For Bernoulli dropout, we pick the dropout rates p E [0.0, 0.2), and for Gaussian dropout, the variance is set be in [0.0, 0.6] : we obtain 100 models per dropout parameters. The leftmost column shows how the loss deviations (Loss diff. in the figure) are controlled by varying the dropout parameters, as suggested by the analysis in Section 3. Note that since the Adult Income and Bank Marketing datasets have much more samples than rest of the datasets, their variances of loss difference are much smaller com- pared to the other datasets. Moreover, for all datasets, the accuracy deviations caused by the loss differences are all within 1% (see Appendix E.
For predictive multiplicity metrics that are de- fined per sample (e.g., VPR and RC), for the sake of demonstration, we plot the values of 50% or 90% quantile, depending on which quantile value best shows the difference be- tween dropout and the baseline methods. We report the values of those metries for all sam- ples in Appendix E.I. As observed in Figure 2, both Bernoulli and Gaussian dropouts mostly outperform the re-training strategy in terms of exploring models the Rashomon set for es- timating predictive multiplicity. For example, in both Bank Marketing and Mammography datasets, the VPR and Ambiguity given by re- training are close to zero, whereas dropouts provide diverse models with high multiplicity that within the loss deviation regime (i.e.,
Table 1: Average runtime speedup model on UCT datasets, evaluated with the computational plat- form. For the raw values of runtime, see Table
Gaussian Dropout
Dataset
Speedup over
Re-training
AWP
Adult Income
305.53x
Bank Deposit
35.36x
Credit Approval
267.75x
5345.45x
Dermatology
28.88x
1121.59x
Mammography
130.20x
4238.18x
Contraception
78.40x
2952.73x

---

Published as a conference paper at ICLR 2024
in the same Rashomon set). On the other hand, AWP outperforms both dropouts and re-training, since it adversarially searches the models that mostly flip the decisions toward all possible classes for each sample. Despite that AWP best explores the Rashomon set, it comes at the cost of high time complexity, as shown in Table 1, where we record the average runtime per model for re-training, Bernoulli/Gaussian dropouts, and AWP. Note that for all methods, a pre-trained an empirical mini- mizer w' is given and its training time is not included in Table 1. The good performance of AWP comes at the cost of it being more than 1000 times slower than dropout methods. Also, the dropout methods are tens to hundreds times faster than re- re-training. We refer the reader to Table E.7 for an additional comparison of the efficiency obtain models from the Rashomon set between the re-training and dropout strategies.
We study a more complicated and practical use case by using the Microsoft COCO hu- man detection dataset [Lin et al., 2014]. The goal of human detection is to determine, in image or video sequence that contain multi- ple objects, the smallest rectangular bounding boxes that enclose humans. Predictive mul- tiplicity therein could lower the trustworthi- ness of applications involving human detection such as autonomous driving or AR/VR systems INguyen ct al.. 2016]. Figure 3 shows the pre- dictive multiplicity of determining the human bounding boxes with the Y3 detector [Red- mon & Farhadi, 2018]. We use the Gaussian dropout to explore models in the Rashomon set, whose average precisions (APs) are within 0.5% to the pre-trained model. Despite that the APs of the models are similar, the bound- ing boxes exhibit high predictive multiplicity in terms of the coverage and confidence of the re- gions. For the corresponding predictive multi- plicity metrics with the YoloV3 and Mask R- CNN [He et al., 2017] detectors, see Appendix E.2.
0.0430
0.574P
0,2123
2ion
0,0160
puseno
0,6274
0,7399
Human detection on MS COCO dataset. The leftest column shows the ground truth of the bounding boxes, and the rest of columns are the bounding boxes found by 4 models in the dropout-basee based Rashomon set. The green values denote the confidence of the bounding boxes larger than 0.5, and red otherwise. The detectors of the bounding boxes suffer from predictive multiplic- ity in terms of the coverage and confidence.
Figure 3:
For more extensive experiments with CIFAR- 10/-100 datasets [Krizhevsky & Hinton, 2009] using VGGI6 [Simonyan & Zisserman, 2014] and ResNet50 [He et al., 2016], along with their runtime comparisons, see Appendix E.3. For ablation studies on the depth, width, and architectures of the neural network models, and model calibration [Platt et al., 1999], see Appendix E.4.
Note that either baselines such as re-training and AWP, or the proposed dropout method, can only explore partially of the entire (true) Rashomon set when the hypothesis space is large, and there- fore all estimates obtained from the three strategies are under-estimates of the multiplicity metrics. In Figure 1. in Appendix E. we demonstrate that when the hypothesis space small (e.g., lo- gistic regression), the re-training and dropout strategies have similar performance of exploring the Rashomon set. Despite that the proposed dropout method may only search local models in the Rashomon set, it provides fast lower bounds for multiplicity metric estimates.
APPLICATIONS USING EFFICIENT EXPLORATION OF THE RASHOMON SET
The efficient dropout technique makes applications regarding the exploration of the Rashomon set computationally feasible. Here, we show two applications of the Rashomon set: (i) mitigating predictive multiplicity using the ensemble method with dropout, and (ii) selecting models that have the smallest multiplicity subject to potential loss deviations. We use the UCI Adult Income dataset [Asuncion & Newman, 2007] following the same training settings in Section 4 for both applications.
Dropout ensembles to mitigate predictive multiplicity. Long et al. [2023] have shown that the outputs of ensemble models have less variance and hence suffer less from predictive multiplicity. The main challenge of getting an ensemble model is to obtain multiple models in the Rashomon set. Using dropout, we can efficiently obtain a vast amount of models to compose an ensemble model by averaging their outputs. As shown in Figure 4a, the number of models in ensemble
8

---

Published as a conference paper at ICLR 2024
1.25
1,00
Bernoulli Drooout
1.2
1.0
0.307
0.00126
1.0146
0.04
0.20725
0,775
0.75
0.00124
1.0145
Ensevele Predtctuue
0..50
0.50
Soen
Irb.
0.105
1.0144
0.02
0.044
0.00675
0.25
6.00
20'
17'
0.304
0.03120
1,0142
1.0141
-0.02
0.00
0.042
0.00550
0.00625
0.000
0.05
0.00600
0.815
Ensomble (Log)
0.04
Ensomble Size (Log)
0.301
0.00116
1.0139
0.00575
(a) Predictive multiplicity metries vs. different num- (b) Predictive multiplicity metries 5 different pre- ber of models in an ensemble. A larger ensemble can trained models using bernoulli1 0.01) dropout. effectively reduce multiplicity. Each value is aver- Model 2 is preferred since it has smallest estimate of aged over 100 ensemble models.
multiplicity metrics over the rest of the models.
Figure 4: Applications of Rashomon set using dropout with the Adult Income dataset.
(Ensemble Size in the figure) increases, both the score and decision-based predictive multiplicity metries shrink. Therefore, whenever given a pre-trained model, we can efficiently construct an ensemble with dropout models in order to reduce predictive multiplicity.
Selecting models with smaller predictive multiplicity. We can also use our efficient estimation framework to perform model selection if presented with multiple pre-trained models with simi- lar performance. By performing dropout on each pre-trained model to construct the empirical Rashomon set, we can pick the model that yields the smallest estimate of predictive multiplicity metries. For example, Figure 4b shows the spread of predictive multiplicity metrics for 5 models, trained with different weight initializations and same architectures. Model 2 has the smallest VPR, RC, Ambiguity, and Discrepancy compared to the rest of the models. Conversely, Model 4 has the highest estimate of predictive multiplicity metrics. Therefore, Model 2 would be a more favorable choice for deployment over Model 4.
6 DISCUSSION
Here we reflect on the limitations and highlight interesting avenues for future work.
Limitations. First, in order to apply dropout the model weights, our proposed framework re- quires full access to the weights of the pre-trained model as it multiplies them with the dropout random variables, i.e., the pre-trained model has to be white-box model. However, white-box pre-trained models are often unavailable due to practical reasons such as data security and IP pro- tection. Second, discussed in E.3, when the hypothesis space is extremely large, dropout method could possibly only explore models that are "close" to (and dependent on) the pre-trained model and therefore under-perform the re-training strategy. A potential solution to this issue is combine the re-training strategy and the dropout methods we first re-train a few models to ensure a sufficient exploration of diverse local minima, and then apply dropout on these models to further explore the Rashomon set -see Figure E.22 and E.23 in Appendix E.3 for a demonstration of such strategy.
Future directions. First, our analysis of the connection between dropout and the Rashomon set is primarily on FFNNs and CNNs. The analysis could be generalized to other neural network archi- tectures with feedback loops (i.e., recurrent neural networks [Yu al., 2019]) in order investigate predictive multiplicity for other applications such as time series or natural languages. Second, our analysis assumes equal dropout probabilities for all weights on the first few layers (or some unfrozen layers) may better explore the Rashomon set, some lay- ers carry more semantics information [Yosinski et al., 2014]. Third, disagreements among models in the empirical Rashomon set can be used to falsify improve at least one (with the lowest loss) of the models by reconciling the conflicting decisions [Roth et al., 2023]. Given models with different dropout parameters, a possible procedure to reconcile the conflicting decisions could be assigning different weights to those decisions, depending on the dropout parameters and neural network prop- erties.
neural network. However, dropout

---

Published as a conference paper at ICLR 2024
Ethics statement. The phenomenon of the Rashomon effect and predictive multiplicity can be maliciously exploited by adversaries to make harmful decisions towards targeted users. For example, an adversary can measure predictive multiplicity across different training procedures, e.g., different neural network architectures or order of batches the training dataset, and select a combination of settings that produces the most "unfair" decisions against certain group of population.
Reproducibility statement. Our codes are based on PyTorch [Paszke et al., 2017], where the Bernoulli and Gaussian dropouts could be implemented by intermediate activations with forward hooks. Note that our implementation is different from Gal & Ghahramani [2016] in https:
//gi com/ /aar/arriaa iigan Drrppout Dropout.nnceta. Uncerta intyEpps// ree /master. The estima- tion of the predictive multiplicity metrics follows directly from either the corresponding mathemati- cal definitions in the papers or the GitHub repository therein. See Appendix B for the mathematical formulations, corresponding definitions in each reference and the GitHub repositories.
Disclaimer. This paper was prepared for informational purposes by the Global Technology Ap- plied Research center of JPMorgan Chase & Co. This paper is not product of the Research De- partment of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit bility in connection with this paper, including, without limitation, with respect to the completeness, accuracy, reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended as investment research or investment advice, or as a recommendation, offer, or solicitation for the purchase or sale of any security, finan- cial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction.
implied representation warranty and none of them accept any lia-
REFERENCES
Suguru Arimoto. An algorithm for computing the capacity of arbitrary discrete memoryless chan- nels. IEEE Transactions on Information Theory, 18(1):14-20, 1972.
Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2022.
Richard Blahut. Computation of channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460-473 1972.
Avrim Blum, John Hoperoft, and Ravindran Kannan. Foundations of data science. Cambridge University Press, 2020.
Leo Breiman. Statistical modeling: The two cultures. Statistical science, 16(3):199-231, 2001.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprini arXiv: /0t.::/900001555.. 2019.
Zhi Chen, Chudi Zhong, Margo Seltzer, and Cynthia Rudin. Understanding and exploring the whole set of good sparse generalized additive models. arXiv preprint 233.:22333./644,, 2023.
A Feder Cooper, Solon Barocas, Christopher De Sa, and Siddhartha Sen. Variance, self-consistency, and arbitrariness in fair classification. arXiv preprint arXiv:2301.11563 2023.
10

---

Published as a conference paper at ICLR 2024
Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing faimess the set of good models under selective labels. In Proceedings of the International Conference on Machine Learning. PMLR, 2021.
Kathleen Creel and Deborah Hellman. The algorithmic leviathan: Arbitrariness, fairness, and op- portunity in algorithmic decision making systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2021.
Andreas Damianou and Neil Lawrence. Deep Gaussian processes. In Artificial intelligence and statistics, pp. 207-215, PMLR, 2013.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, al. Underspecifi- cation presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23(1):10237-1229) 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the International Conference on Machine Learning, pp. 1050-1059. PMLR, 2016.
Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Leif Hancox-Li. Robustness in machine learning explanations: does it matter? Proceedings of the Conference Fairness, Accountability, and Transparency. ACM, 2020.
Md Golam Moula Mchedi Hasan and Douglas Talbert. Mitigating the rashomon effect in coun- terfactual explanation: A game-theoretic approach. In Proceedings of the International FLAIRS Conference, 2022.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
John Hertz, Anders Krogh, Richard G Palmer, and Heinz Horner. Introduction to the theory of neural computation, 1991.
John A Hertz. Introduction to the theory of neural computation. CRC Press, 2018.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing ro-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp. 409-426, 1994.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators, Neural networks, 265):359-366, 1989.
Hsiang Hsu and Flavio Calmon. Rashomon capacity: A metric for predictive multiplicity in classi- fication. Proceedings of Advances in Neural Information Processing Systems, 2022.
JHB Kemperman. On the shannon capacity of an arbitrary channel. In Indagationes Mathematicae (Proceedings), volume 77, pp. 101-115. North-Hollani 1974.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. Proceedings of Advances in Neural Information Processing Systems, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- ter Thesis, Department Computer Science, University of Toronto, 2009.
11

---

Published as a conference paper at ICLR 2024
Bogdan Kulynych, Hsiang Hsu, Carmela Troncoso, and Flavio P Calmon. Arbitrary decisions a hidden cost of differentially private training. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2023.
Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee. Survey of dropout methods for deep neural networks. arXiv preprint arXiv: 13310, 2019.
Jachoon Lec, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. In Proceedings of the International Conference on Learning Representations, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision. Springer, 2014.
Carol Xuan Long, Hsiang Hsu, Wael Alghamdi, and Flavio P Calmon. Arbitrariness lies beyond the fairness-accuracy frontier. arXiv preprini urXiv:2306.09425. 2023.
Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Proceed- ings of International Conference on Machine Learning. PMLR, 2020.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2019.
Duc Thanh Nguyen, Wanqing Li, and Philip O Ogunbona. Human detection from images and videos: A survey. Pattern Recognition, 51: 1111148-177 175, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in Py- Torch. In Proceedings of Advances in Neural Information Processing Systems, Autodiff Workshop, 2017.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Joseph Redmon and Ali Farhadi. Yolov3:
An incremental improvement. arXiv preprint
arXiv: 2018.
Mélanie Rey and Andriy Mnih. Gaussian dropout as an information bottleneck layer. Proceedings of Advances in Neural Information Processing Systems, Bayesian Deep Learning Workshop, 2021.
Aaron Roth, Alexander Tolbert, and Scott Weinstein. Reconciling individual probability forecasts. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 2023.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215 2019.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint /90.00/90800/1 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint xrxiv:1409.180 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-199 1958, 2014.
Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting is not understanding: Recog- nizing and addressing underspecification in machine learning. In Proceedings of the European Conference on Computer Vision. Springer, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
12

---

Published as a conference paper at ICLR 2024
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Proceedings of Advances in Neural Information Processing Systems, 2018.
Sida Wang and Christopher Manning. Fast dropout training. In Proceedings of the International Conference on Machine Learning. PMLR, 2013.
Jamelle Watson-Daniels, David Parkes, and Berk Ustun. Predictive multiplicity in probabilistic classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. In Proceedings of Advances in Neural Information Processing Systems, 2022.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Proceedings of Advances in Neural Information Processing Systems, 2014.
Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation, 2257):1235-1270, 2019.
13

---

Published as a conference paper at ICLR 2024
The appendix is divided into the following parts. Appendix A: Omitted proofs and theoretical sults; Appendix B: Discussion on predictive multiplicity metrics; Appendix C: Additional discus- sion prediction uncertainty; Appendix D: Details on the experimental setup; and Appendix E: Additional empirical results.
A OMITTED PROOFS AND THEORETICAL RESULTS
A.1 USEFUL LEMMAS
We first introduce (and prove) the following useful lemmas to facilitate the proofs of the proposi- tions. The first lemma is about the expectation of the dropout matrix Dz. Lemma A.1. Let D2 diag Z1. Zz. Za) E Raxd where Ze are i.i.d. random variables, and given a vector a E R then
E a D2 - 12) (D, Lt)a] = ||l|i i a.d
bernoulli(1 - p);
(A.1)
||l|aa if ~
i.i.d
Gaussian (1,
Moreover, if a is the all-one d-dimensional vector, (A.1) can be alternatively expressed as
E [ Iall2] dp, if Zi Ltd. bernoulli(1 p);
(A.2)
da, if Z 1.d. Gaussian(1, a).
Proof. By direct computation of the expectation, we have
d
d
E a (D=, (D= [.)a] E
Σ (a. (Z, 1)2
T. [Zz 1) 2 (a.)
(A.3)
= E [Z, 1)²] ||a||2-
Now, we discuss different distributions for the random variable Zz-
Bernoulli random variable. If Zi bernoulli(1 p). then Z₁ 2bernoulli(p) and (Z, - 1)3 ~ bernoulliyy). Therefore,
E
[Z, 1) = EZ p, where Z bernoulli(p).
(A.4)
Gaussian random variable. Z, ~ Gaussian a)), then Z; - Vas., where S Gaussian(O, 1) follows the standard Gaussian distribution. Therefore,
IE (Z: 1) [Vaas))* aE [$?] a (var(S,) + E(S,3) a.
(A.5)
The desired result follows from combining (A.3), (A.4) and (A.5). Finally, if a is the all-one d-dimensional vector,
d [Z
E a DDz -
-2) 1a) (D2 L2)a] = E
Σ
1)2
= E
D2 L.ke
(A.6)
i=1
E D2
1a
and the rest follows by directly plugging ||a||
The second lemma is on the upper bound of the neural network outputs after dropout.
Lemma A.2. Given the dropout matrices Di.... DK. weight matrices W1,... K. input x and the activation function with Lipschitz constant have
K
K [II ( - Ialla + +1)) ||x13
la (DR W WKO o (D, (1))..)....)
DD&WRoL
I
&
(A.7)
14

---

Published as a conference paper at ICLR 2024
Proof. We prove this lemma by induction the number of layers k. First, when is have
||o (D, Wix) 11 = |o ((D, W1) x) (Wi x) + (W] x) l
S lo (D1 W ) [x) (w] x) |1 + 11 (W l2
(A.8)
S W WW X (I]D Lalli + +1)
Now, let lw_ (x) follows the definition in (8), and suppose (A.7) holds for k: K i....
K
K-1
|1o (NW (x)) 112 =
W
II
(I
Iall + +11) ||x |xx
(A.9)
have
l|a (bW, o (x)) 112 = l|o ((D W To (WWD (hw-1(x)) (x))) 12
I l|o (DDx W k) TJ (h (x)) a W (xWW(())) 112
+ l|o (W ko (l (x)) 11
< [|Dx - || 1 Illlo -x)=
11 + || | (1w (x)) 11
(I -all 1) W k 1/2h |llo (1---11 (x)) 11
K-1
KK1
II (I - Ial + +1) -1) |x
< ([DDx - Iall + 1) W x 1
W & 11:
II 1
π
[I) Lalle + +1) |xx|3-
(A.10)
The third lemma essentially says that the distance of points, sampled from a d-dimensional Gaussian distribution, is tightly concentrated around the distance vd. The quantity vd is also called natural scale or radius of d-dimensional Gaussian distributions.
Lemma A.3 (Gaussian annulus theorem [Blum et al., 2020, Theorem 2.9]). For a d-dimensional Gaussion with mean zero and variance I for any 8 S ovd, all but at most 3e probability mass lies within the annulus avd S x S 3 ovd.
42 of the
The fourth lemma is direct application of Lemma A.3 by choosing 3 = ovd.
Lemma A.4. Consider d-dimensional Gaussian distribution PN,(v) with mean E Rdd and variance o² IdI Rdxd, and unit ball B centered at w with radius 20 i.e., B(w, 20 (v E TR.d 0 S |v| S 20, we have the following integral
J (VBV(
2a and
[
(va(v)dv S 3e da
(vadv)dv 2 - 3e
(A.1I)
JVEBs(w.2ovā
where the latter integral could be viewed as the tail probability bound on the d-dimensional ball with radius 20 d.
A.2 PROOF OF PROPOSITION
For any weight vector w W. denote the dropout weights WD D2w, the mean square error (MSE) is
XwD - y12 = (XWD - yy (XWD - =y 2w 2w y wIx XWD
(A.12)
- - 2w D x + D XD,w.
15

---

Published as a conference paper at ICLR 2024
Taking the expectation, we have
E [| y112 =y 2w BE + w E x w
=yTy - 2(1 - p)w x y E x
E 2D) w
[[(1 p)xw - -y12 (1-p) X Xw w xXD= 1 E XD2) w
(( Xw y112 + w T [x E [ (1 p) 2 x x)
= [111 - Xw - y112 + p(1 - p)w diag X X)w
(A.13)
where the last equality comes from the fact that the off-diagonal entries are cancelled out, and that for Bermoulli(1 p) random variables, the second moment is p(1 p) + (1 p)2 = - p, and therefore (1 - p) (1 p)² = p(1 p).
Since the equality above holds for all weights, let w" and WD D2 w", have
E Be--bemoull(1-p)] = Ezz-bernouli(1-p))
[ww D -y [1 - Xw - -y1
XXD -yl II(1 X -yll (A.14)
Be--benoull((1pp)
p(1 - p)w T di X)w";
which is the desired result.
If the features of data matrix X are linearly independent and normalized, i.e., X La we have diag(X' XX) = La and w = (x + (La) y 14X The expectation of can then be further simplified as
X_y
(iag((X X)w" = p(1 pp w- pp(1 - p) (1 1)2
E Bxw-bernouli(1ppppp = p(1 (p) T
xx
p(1 - p)
(1 A)2 1y1
(A.15)
A.3 PROOF OF PROPOSITION 2
Following the definition of the Rashomon set for ridge regression in (4) and the assumption X X
Ia. we have
Pr (Dew" E ER(w", ,c) = Prr =Pr ( -xx
x + (D, - -w") sc
=Pr ((D( -I) A)L,(D, (D, Lajw* Se
Pr w T(22 (Dz La) TD= -Law" S It\
(A.16)
E (D2 ID) Ia)w'
2 1 - (1+
where the last inequality comes from Markov inequality.
Now, using Lemma A.1 for different dropout random variables Z;, the assumption ||w'|| S M, and (A.16), we have
E W (D, La) - -L))w*)
Pr (r ER(w", 2 - (1 A)
(A.17)
(1 [1+ X) RM
if Zi
i..d bernoulli -p);
- 11 add if Z, Gaussian a).
sid.
If p O(d-)) and 0dd~)) with 5 0. have
lim (1+ + X) pM lim (1 X) aM d-+00
0.
(A.18)
e d-v00
Therefore, the desired results follow for both Bernoulli and Gaussian dropout mechanisms, i.c.,
lim Pr (D.w" E R(w", .)} =.
(A.19)
d
16

---

Published as a conference paper at ICLR 2024
A.4 PROOF OF PROPOSITION 3
Follow the definition of the Rashomon and using tiangle and Cauchy-Schwarz inequalities, we have
LBS (B2w") LBS LBs(w')
(<(( - y:)² 1 Σ
(o(w'Tx xi) -v:)
==1
((Daw") xx) - -o(w Txi)
w TrT(D, (Dz In] Ix)]xx 2
(A.20)
D 1
= M D2
La 1
where |
Σニ 1 and || 112 S M. Therefore, the probability that the dropout weights
lead to models in the Rashomon set is
Pr (BB((Dww") LBS(w") 5 Pr
M
Pr
D2 Iell v
(A.21)
M M 13
M x
D2
Ia 12:
21
where the inequality comes from the Markov inequality. Using Lemma A.1 and (A.21), we have
ME D2 Ial
Pr (Igs(Dzw") - LBS(w") 50)21 21
C
1 (1+ 11++))MMa M
if Z, i..dd
bernoulli(1 p);
1 1 (1+ add
if 2, (aussian(1..))
iid
(A.22)
Therefore, as long as p and a are of order (dd-()+0)). 8 > 0. we have
pMd x
aM 0,
lim (1 + A)
lim (1 + A) d
(A.23)
d=200
and lim Pr (TBs(D)w" - lBB(w*)) { = for both Bernoulli and Gaussian dropout mecha- d-000
nisms.
A.5 PROOF OF PROPOSITION 4
We prove this proposition by the induction method on the number of layers & E [K]. We first consider a neural network byy and derive bound for the deviation of the outputs (in f--norm)
17

---

Published as a conference paper at ICLR 2024
Σ -i=1 1,w, (x:) - how (xx) before and after dropout. By triangle inequality, have
1
Ww (x:) Nww (x,)
((D. W2 (W, x.) l2 ((2:22) To (DD: - W (D) w1) x,) 11
71
W ((D) x.) W (WI x:) 13
71 1=1
(A.24)
By Lemma A.2 and let X
=
1
E
|x we can bound the first term (i) (A.24) as
(i)
I((D2 L)W,) (DD W1) x.) 11
n 1=1
1
(A.25)
DD Lali W | (D, W,) _ xx) 1/
71
D2
Ia
W
| W / (|D: Iall + +) /
Similarly, we have the bound on the second term (ii) in (A.24) as
n (ii) W.) x; w 12 DD Lal 12
(A.26)
Combining (A.24), (A.25) and (A.26), we have
lo (xx) (x;) S (i) + (ii)
71
< D2 - I. W W ( Di - +1) I
||D Tall W / |
W w.l (1) - Lall + D) +1) - (A.27)
Now, suppose the upper bound holds for a (K 1) -iiddennle feed-forward neural network, i.e..
12
K-1
(
(K-1
(x.)
1x: S
II
[ II
DI - Iall + +1) -1) |
(A.28)
18

---

Published as a conference paper at ICLR 2024
then again by applying Lemma A.2 and (A.28), for L layers, the upper bound is
W
(x:)
-n (x:))
1
(Dx Wx) W WKO (WW-1(x)) 112
(1W
71
71
i=1
(x:)) [D&W (x: W (NWw) (xx))
11
1=1
W (x.W
WKO (nwv (xx)) 13
12
n2 (1~0 (1 (x:)) 11
$ Dx Iall W
|1
+ +|
a [x: - a ( 11
K-1
< Dx - Lall?
II
(IID - -11B++1
W X
k=1
K-1 (I|D: Iall? + +1) -1 |
K
KK
W
II (ID: - Lall +1) -1) |
(A.29)
Since the weights are bounded, i.e., | < M for all k E [K]. the probability that the output deviation before and after dropout is smaller than is given by
II [w.li P (fi (IID: Lall +1) -1
-1)
Pr (1lWW (x.) - Ax se 2 Pr
| Se
1=1
K
(IDD Lall +1) -1) 5
e
Pr
[TK W 113 X 1
[I (IID, - Lall: +1) +1) -1
2 Pr
M X
2 1 E
(IID, +1)
M
1
π E [(ID, Lall +1)] 1)]
-1)
M | 2 (A.30)
where the last equation comes from the fact that the Dj for different layers are independent.
19

---

Published as a conference paper at ICLR 2024
Let m max my and since Z (E Lall-
Gaussian (1, a), the product in (A.30) becomes
K
k
m
K -
m
(Z (2 - 1)2 +1 ]
[[ID;
+ 1)]
II
Σ j=1
(Z,
- 1)² 12)=1)++1
Σ
i=1
i=1
m
Σ
(Zg 1)²) +1 +1)
(ma + 1)
i=1
(ma 1)*
(A.31)
Combining (A.30) and (A.31), have
Pr
[1l (xi) //
K
(x; (x:)11 sc}
K
II E [ - I +1]] -1)
2 1
(A.32)
M I
21- (Ima 1)k-11 MK X
In other words, with probability least 1- p. the deviation bounded by
Ilw (x,) 1 (x, 11 S p- MM |XI (ma + 1)x -1).
(A.33)
Moreover, as long as a = Omm (m 8 0,
lim
M
X ( - 1) - 11) 0.
(A.34)
We finally connect the deviation of the outputs to the loss functions. For regression tasks, the deviation MSE losses becomes
TMSE (D) MSSE (W*)
Iww (x:) - -vil
K (x:) y
n
(A.35)
Wp (x:) hx
For classification task, the deviation of the Brier scores is
softmax(1 (x:)) yi1l 1
LBS(wD) LES W
|softmax (1W ysl12
softmax(11 ( (x:)) (
| (x,) 1 (x;))1
n
(A.36)
Therefore, combining, for both MSE loss and Brier score, with at least probability at least P.
LMSE (W (Wp) L (W*) I p K | (ma + 1)
1)) -1)= and
(A.37)
Las (W,, p) Las(W') < p -
X (ma + (1) -1)
20

---

Published as a conference paper at ICLR 2024
A.6 ADDITIONAL NOTES ON THEORETICAL RESULTS
Note that the condition in Proposition and 3, d + xc, is a regime commonly known in the asymp- totic analysis of learning models such as pereetrratttiniioonnn [Cao & Gu, 2020J, and the universal approximator theorem [Hornik et al., 1989].
The goal explore the Rashomon set find models with diverse outputs while satisfying the loss deviation constraint. If we are allowed to find models outside of the Rashomon set, the performance of the models could be much worse and are less likely to be selected practice. Note that the phenomenon of predictive multiplicity is only discussed within almost-equally optimal models (say models that all have 99% accuracy). In this sense, the concentration bound helps to characterize the probability that model after dropout will still be inside the Rashomon set, and is vital mathemat- ical tool used our proofs. Ideally, it more efficient to directly select models at the boundaries of the Rashomon set. It is possible when the Rashomon set could be explicitly expressed, c.g., for ridge regression. However, for general hypothesis space, how to search at the boundaries of a Rashomon set is still an open challenge, and re- re-training (and AWP) was the only strategy.
The Rashomon set in (2) is defined regarding the mean of the loss for a given dataset of a fixed size. In other words, the dataset is given parameter, not source of randomness in the definition of Rashomon set. Note that the randomness in the convergence in (6) and (9) is with respect to the dropout matrix, not like the vanilla concentration bound that consider drawing samples from data distribution. it makes sense that those convergence results does not rely on the number of samples. In (5), the loss we used is the sum of the loss for each samples as we w.l.o.g. assume the data matrix is whitened, In (6) and (9) the losses are defined with the mean of the loss for each sample.
A.7 ADDITIONAL THEORETICAL RESULTS
We show the sample complexity for estimating a surrogate metric for score variance. Using score variance a predictive multiplicity metric is adopted in Cooper et al. [2023] and Long et al. [2023]; however, they only consider the case of binary classification. Here, we generalize the notion to multi-class classification problems. Since (w(x;) is a c-dimensional vector, can model the distribution of the score [hw (x:)lp, p. of the correct label E [c] as a beta distribution beta 3). In this case, the variance of the beta distribution
111 1) a+ 3+ S p(1 -1),
(A.38)
(a + () + 3
where y is the population mean.
We assume that the models around W' are uniformly distributed in a d-dimensional ball with center W and radius 8, i.e., 15( B(W*, 8). Accordingly, we may assume that the population mean f for sample can be expressed as
p(x,) = un .5)
[/w (xx)),
(A.39)
The assumption of the uniform distribution around W may not reflect the true underlying distribu- tions of models in the Rashomon set (c.g.. when there are exponentially many local minima), but is an ideal assumption that facilitates mathematical analysis that was also adopted in existing liter- ature [Kulynych et al., 2023]. How to characterize the distribution of models in the Rashomon set still remains an open and active challenge in the field. The estimator for (u(x;) by using T Gaussian dropout models is given by
1 T
(xx;))
A T
Σ
lowb, (x.)) where W (De, W and Da.eli ~ Gaussian o).
(A.40)
We pick v(x:) A u(x;)(1 - p(x:)) as a surrogate metric for the upper bound of the score variance in (A.38), and the plug-in estimator is 6(x;) é ((xx) - ((x;)).
In the following proposition, the number T of models in the empirical Rashomon set (2) by concentration bounds and Gaussian annulus theorem (cf. Lemma A.3).
show that (xx,) can be used to estimate v(x,) reliably in terms of
21

---

Published as a conference paper at ICLR 2024
Proposition 5. For TGaussian(1. 1, a) dropout models, hapa. hworr in the empirical Rashomon set, with probability at least 1 p. p E (0,1), the deviation {xx,) - v(x+) satisfies
diw
In 2
(1 (I+ be
adw
In 2 27 p)
- u(x)) -
(A.41)
27
where Wmax
max ke(K)]ie(m ilme((m))
Wil and d = E m x m..
Proof. Since (xx,) is continuous transformation of fx(x).. we could bound the deviation (o(x,)
exx)]l by (ā(xx)) - ( (x,)). Suppose j(xx)) - (xx,) = and v [-1), 17], we have
(0(x,) vxx,)l (1 - p(xx)) ((xx)(1 x(xx))1
= [(((x;) + v1 - (xx,) - v) - -1(x)) - (xx:))]
- (1=((x))))) + 1(1 - (((x)) - -u((x;) - -22- -2-=m(xx)(( -x(xx)(1--(xx)))
I |/ |1 - -2u(x,) - vl
|e|| |1 + v/-
(A.42)
Therefore, only need to bound the deviation (ā(x;) (xxx)). Here, have
[ī(x:) p(x:)l = [ā(x:) [x]( + Eff(x;)) - mxx:)
I 11(xx;) [(A(x;)] [x]] (xx;)
(A.43)
(
where the expectation is taken over the distribution of all dropout matrices Dj. For (i) in (A.43), using Cheroff-Hoeffding inequality [Hoeffding, 1994]. we have
Pr (lū(x:) - Blj((x;) < n 2 1 eex((-2n27)
(A.44)
In order to bound (ii) in (A.43), let Wiax =
ke k max (mm))
W d Ek= m x mk.
8 Wmax v ad and P. be the probability density of Gaussian dropout, using Lemma A.4, we have
[[(((x;)] - sxxx)1 = [[:))((x))ol)) -
(W (W WdWW-
J (5)
[/w (xw(x;lp vol(B( W .)) dW
Satw (xw(x:)l (DN(W)- vol(B(W> dW
1
Veerwrss (lw (Mw(x))l
BBW (DN(W)- - vol(B(W* 8)) dW + +oo PW(WWddW
J JaCw...) PW)WW (B(W.5) vol(B(W- dw
1// PAr(
I J .5)
PN(W)dW / /B PN(W)dW
PWWWWaW
(W..5)
6e
(A.45)
22

---

Published as a conference paper at ICLR 2024
Combining (A.43), (A.44) and (A.45), have
Pr (xx) - µ(x)) S n) 2 Pr (xxxxx) - [lu(x))] + [[[](x)]] (xxxs1 S n
adwinu
5n} }
2 Pr
(xx:) [[( + 6e
Pr
[A(x)) [[A(x+I11 5 - 6e
adw
(A.46)
-2 n-
2 1 - exxp
T
Let 2exp
-2(1166-
-2 1- 6e
T
have with probability at least 1 P. E (0,1).
adw?
2
li(x;)) - m(xx)) S Ge
(A.47)
2T
P
Therefore, by plugging in the deviation lá((x)) - m(xx)l (A.42), we have with probability at least
1 1-p
16(x;) - oxx)) S + |
6E
adu
2T (1+
In
p /
2
(1
Ge
adwm
In 2T
I
2
P
(A.48)
Proposition 5 shows the sample complexity of estimating vxx,) with T models for single sample. In practice, one might need to estimate the score variance for multiple samples, e.g., computing average score variance
test dataset. Naïvely, we need nT models estimate v(x;) for dataset with n samples. contrast, we can easily generalize the results Proposition 5 to n samples by union bounds, and show that in such cases sample complexity grows only logarithmically under mild assumptions. To be precise, since the samples X1, X2, Xn, are i.i.d,, we have the following union bound for the concentration of sample mean, i.e..
Pr U (1((x.) -Bi(x))) 2 m}
II Pr (lā(x:) - BlA(x)) n) S 2n (mexp(22nn"))
(A.49)
Therefore, with probability - p. for all E [n]. [A(x;) - E[ū(x;)) S
2n
By following
27
the proof of Proposition 5, with probability p. for all [n].
2n
(1+ (1+ One
addi
2n
(0(x.) v(x:)) S 6ne
2T In
In
(A.50)
P
27
0
Since the term e 2
could be made arbitrarily small, it can compensate the linear growth of n.
23

---

Published as a conference paper at ICLR 2024
B DISCUSSION ON PREDICTIVE MULTIPLICITY METRICS
We give a thorough introduction of predictive multiplicity metrics, including their mathematical formulation, operational meanings, and computational details. Predictive multiplicity metrics can be categorized into two groups: score-based and decision-based, where a decision is a thresholded score or the score vector after argmax. Precisely, consider binary classification, if we have a score s. then the decision can be obtained by L[s > 7]. where T is threshold and [[-] is the indicator function. For a c-class classification problem where e > 2, the score a vector, say s Ae. and the decision can be obtained by argmax (s): In the following, we start with the recision-based metries.
=E(e!
First, consider Rashomon set R. the pattern Rashomon ratio r(D) of a dataset is defined as the ratio of the count of all possible binary predicted classes given by the functions in the Rashomon set to that given by the functions in the hypothesis space [Semenova et al., 2019, Defn. 12], i.e
1= 1 /w E R, [argmax hw (x1), ... argmax hw (xn)) = binary (i)
(D) A
(B.51)
=0 1 Nw E H. [argmax Nw (x1)," argmax Aw (x,) = binary(1))
where binary(+) denotes the vector of the binary representation of an integer, e.g., 5 = [0, ... ,0, 1, Computing pattern Rashomon ratio involves summation with the number of terms grows exponentially fast with the number of samples n. Therefore, pattern Rashomon ratio an "expensive" metric for predictive multiplicity when evaluated on large datasets.
There are several decision-based predictive multiplicity metrics that use the disagreement among de- cisions given by models in the Rashomon set under difference disguise [Black et al., 2022; D' Amour et al., 2022; Marx et al., 2020]. For example, Marx et al. [2020] propose two metrics: ambiguity and discrepancy. Ambiguity is the proportion of samples in dataset that can be assigned conflicting predictions by competing classifiers in the Rashomon set. Discrepancy is the maximum number of predictions that could change in dataset if we were to switch between models within the Rashomon set. More precisely, given a pre-trained model hw., the ambiguity (xD) and the discrepancy o(D) respectively defined as [Marx et al., 2020, Definitions 3 and 4]
a
Σ
max 1 [argmax hww (xx) + argmaax ht (x:)]
D
ho ER
x, X
(B.52)
8 (D)
max
Σ
1 [argmax (x.) # argmax ho (xx)]
ER
D
x;ED
Both ambiguity and discrepancy can be estimated by a mixed integer program [Marx et al., 2020, Section 3].
Moreover, instead of computing the empirical mean of decision disagreement over a dataset D, we may define disagreement directly using the notion of probability. Precisely, Black et al. [2022, Section A. 1] and Kulynych et al. [2023, Eq. (4)) propose the following quantity to measure disagree- ment:
p(x;)= A 2
> r] A I1 (x.) > T]i fw Mw E RJ.
(B.53)
where bw and My are any two models in the Rashomon set and the factor ensures that v(x;) is in the [0, 1] range for the ease of interpretation. Kulynych et al. [2023] further proposed plug-in estimator to estimate disagreement for binary classification with a sample complexity bound on the number of models obtained by re- re-training.
On the other hand, score-based metrics focus on the spread of the output scores [Hsu & Calmon, 2022: Watson-Danich et al., 2023; Long et al., 2023]. Borrowing from information theory, Hsu & Calmon [2022, Definition 2] measures the spread of output scores for c-class classification problems in the probability simplex Ae by an analog of channel capacity, termed the Rashomon Capacity, i.e.,
c(x;) A sup inf Eng-PP DKL (h (x;))lq); Pe qEa:
(B.54)
where PR is the probability distribution over the models in the Rashomon set, and DKL (-l-) is the Kullback-Lebble (KL) divergence. The infimum inf E DKL (x:)llq) measures (in
qEA.
the sense of KI. divergence) the spread of the scores of a sample x; given distribution PR over
24

---

Published as a conference paper at ICLR 2024
Table B.2: Predictive multiplicity metrics implemented in this paper.
Metrics
GitHub
Viable Prediction Range
not included
Score /ta./var.
not included
Rashomon Capacity
httpps:/ggitnn cwm
Disagreement
https://github: com/ wttphuuu...o/ppooaaaeeeeee ctw/:/prinng-ge///dd...ttttaaaa
Ambiguity
Discrepancy
all all the models hw in the Rashomon set, where the minimizing acts "centroid" for the outputs the classifiers. The supremum picks the worst-case distribution PR. over all possible distributions in the Rashomon set. The Rashomon Capacity is closely connected the information diameter, a metric to measure the diameter of set of probability distributions [Kemperman, 1974]. Hsu & Calmon [2022] proved that the Rashomon Capacity be estimated using models in the Rashomon set, obtained by the adversarial weight perturbation (AWP) algorithm, and the Blahut- Aromoto algorithm [Blahut, 1972; Arimoto, 1972].
Based on ambiguity and discrepancy, Watson-Danies et al. [2023, Definition 2] proposed Viable Prediction Range (VPR) v(x)). which is the largest score deviation sample that can be achieved by models in the Rashomon set:
(x;;)
A max (x,)), - min hw (x.). fow E.R
(B.55)
E.ER.
The VPR can be computed using similar mixed integer programs in Marx al. [2020] for binary classification with linear classifiers.
Instead of computing the largest score deviation of a sample, Long et al. [2023, Definition 2] mea- sures the standard deviation (std.) sxx;) (and the variance (var.)) of the scores of a sample by all models in the Rashomon set:
8 (x;))
Enn (1/w((x)) - TE Pre (x:)))
(B.56)
The score standard deviation can be estimated using the re-training strategy.
Computation. There are some theoretical analyses of estimating predictive multiplicity metrics. For example, the exact estimation of ambiguity and discrepancy for linear classifiers are studied in Marx et al. [2020]. Similarly, the theoretical analysis of estimating VPR with logistic regression also provided in Watson-Daniels et [2023]. Moreover, the estimation of disagreement is also discussed in Kulynych et al. [2023]. However, the statistical properties of estimating other metrics such as the Rashomon Capacity, or with general hypothesis space, still remain an open challenge. Finally, summarize the GitHub implementation for computing the predictive multiplicity metrics in Table B.2. For metrics where the implementations are not included, we implement them directly following the mathematical definitions.
The AWP algorithm. The Adversarial Weight Perturbation algorithm, as proposed in Hsu & Cal- mon [2022, Eq. 9], will adversarially perturb the output scores of given sample to each class. For example, if there are classes, AWP maximizes each entries of h_w(x), i.e., [=_ for each i [c], under the constraint that the model after perturbation is still inside the Rashomon set. Given sample X,, we obtain models with output predictions Pı by approximately solving the following optimization problem which maximizes the output score for class l:
Ps = hx(xx), where θ argmaa [xo(x;)]k, Vk 11:==12.-.. 1,2,...,C.. hgER(HH.r)
(B.57)
To solve equation B.57, for each k we set the objective to be minsge -[ha compute the gradients, and update the parameter 9 until L(ha) > e. Therefore, for one sample, AWP requires to perform one perturbations (adversarial training) via SGD for each class (i.e., adversarial training in total for one sample), and if there is a dataset with n samples, we need n x c adversarial training to estimate multiplicity metries for the entire dataset. We can imagine that each of these
25

---

Published as a conference paper at ICLR 2024
perturbations leads to model in the Rashomon set that outputs the most inconsistent score for given class per sample. On the other hand, the dropout method does not require to iterate over all samples and does not require gradient computations and updating model weights. In this sense, AWP is very different from the proposed dropout strategy.
26

---

Published as a conference paper at ICLR 2024
C ADDITIONAL DISCUSSION ON DROPOUT AND PREDICTION UNCERTAINTY
We introduce dropout inference for prediction uncertainty proposed in Gal & Ghahramani [2016], and discuss its difference from the notion of Rashomon and predictive multiplicity.
Gal & Ghahramani [2016] proposed a theoretical understanding of dropout at inference with Bayesian approximation. In the Bayesian framework, the conditional probability p(y|x, D) of the output for an unseen imput x is viewed as deep Gaussian process [Damianou & Lawrence, 2013], and can be expressed as
/ p(v|x, w)p(wwD)ddw))
plu1x, D)
(C.58)
weW
Here, p(w[D) is the distribution of the weights given the training set D, and is usually intractable in practice. By substituting the true distribution (D) with an approximation q(w), the posterior x D) can be computed as
pluxx, D) ~
p(ulx, w)g(w)dw
(C.59)
wEW
q(w) is assumed to follow distribution parameterized by 0, and suitable approximation q(w) can be found by minimizing the KL divergence min DKL (p(
Gal & Ghahramani [2016] used dropout for q(w) with mongle-hideen-eeee networks, and interpret dropout inference as as Monte Carlo sampling that is equivalent estimate characteristics of the underlying distribution p(y) x, D) in (C.59). Therefore, the variance of the prediction y can be estimated using p(y)x, x, D), and be taken to indicate the prediction uncertainty of the model for a particular input. Later on, Lee et al. [2017] generalized the analysis in Gal & Ghahramani [2016] multi- multi-hiddee-layy hidden-layer neural networks.
Note that the integral in (C.59) is computed over the set of all possible models p(y(x, w) w) in the hypothesis space W; however, not all models provide satisfactory performance in terms of loss and accuracy, i.e., not all models are in the Rashomon set. In other words, the posterior we are interested is
[
ply D)
(WE((W*..) ply|x, w (o))p(/(DDss)
(C.60)
The difference between the integrals in (C.58) and (C.60) is the main difference between the con- sideration of prediction uncertainty and predictive multiplicity. In Table C.3, Approval dataset to compare the predictive uncertainty, i.c., score variance evaluated with (C.58), against score variance evaluated with (C.60), where the Rashomon parameter is the mean loss de- viation under each dropout parameters p for Bernoulli dropout and a for Gaussian dropout. For the sake of illustration, we show the value of the 80% quantile of the score variances across all samples. It is clear that without the consideration of the Rashomon set, both dropout techniques could include models that not in the Rashomon set (i.e., models that have loss deviations larger than e), and hence evereestimatin the score variance.
the UCI Credit
Table C.3: Comparison between the score variances computed with (C.58) and (C.60) the Credit Approval dataset. is clear that using (C.58), i.e., without the consideration the Rashomon set, score variance over-estimated.
Bernoulli Dropout
Gaussian Dropout
Dropout Parameters
p = 0.020
p 0.040
0.060
a 0.120
a 0.210
a 0.330
Rashomon Parameters
c= 0.003
0.006
0.007
< 0.002
< 0.005
< 0.011
Score Var. with (C.58)
0.001131
0.002051
0.003280
0.000385
0.001211
0.003000
Score Var. with (C.60)
0,000010
0.000114 14
0.001699
0.000067
0.000285
0,000858
27

---

Published as a conference paper at ICLR 2024
D DETAILS ON THE EXPERIMENTAL SETUP
We summarize the dataset descriptions, training setups and the results.
D.1 DATASET DESCRIPTION AND PRE-PROCESSING
The datasets used in this paper are the datasets from the UCI machine learning repository [Asun- cion & Newman, 2007] and the Microsoft COCO dataset [Lin et al., 2014]. The UCI machine learning repository is a well-known and widely used collection of 650 datasets for machine learn- ing research and experimentation, and contains diverse and extensive collection of datasets across various domains. We select 6 datasets that may possess critical consequences if predictive multi- plicity is not accounted for. The first three datasets related financial applications, including Adult Income, Bank Marketing and Credit Approval, and the other three are related to medical, including Dermatology. Mammography and Contraception. The Adult Income dataset aims to pre- dict whether the income of an individual exceeds 50,000 per year based on 1994 census data. The Bank Marketing dataset related with direct marketing campaigns of Portuguese banking insti- tution based on phone calls in order to predict if the client will subscribe bank term deposit or not. The Credit Approval dataset concerns the prediction of successful credit card application with anonymized feature names, The Dermatology dataset aims to determine the differential diagnosis of erythemato- rrythemato-squamouss diseases in dermatology. The Mammography dataset aims to discriminate be- tween benign and malignant mammographic masses based BI-RADS attributes and the patient's age. The Contraception dataset contains samples from married women who were either not pregnant do not know if they were at the time of interview of the 1987 National Indonesia Contraceptive Prevalence Survey, and the goal is to predict the current contraceptive method choice (no use, long- term methods, or short-term methods) of woman based on her demographic and socio-economic characteristics. For these datasets, we remove samples with missing values, one-hot encoded nomi- nal features, re-scale numeric features, and set the target label name be and the rest to be For the number of features, training/test split and the label names, see Table D.4.
The Microsoft COCO (Common Objects in Context) dataset benchmark dataset with 200,000 images spanning 80 object categories, comprising vast collection of high-quality images annotated with rich and detailed object segmentation, captioning, and keypoint information. For the image datasets, we follow standard normalization procedures normalize the values of each pixel in each color channel.
D.2 TRAINING SETUPS AND RESULTS.
For the UCI datasets, we use a single-hidden-layer neural network with 1000 hidden neu- rons, and train for 100 epochs with learning rate 0.001 and batch size 100. The perfor- mance of the base models, including training and test losses and accuracy, is included in Ta- ble D.5. The re-training strategy follows the same setting but with fewer epochs. For Adult Income dataset, we re-train 100 times for each epoch 20, 25, 30, 35, 40, 45, 50, 55. For Bank Marketing dataset, re- re-train 100 times for each epoch 2, 5, 6, 9. For Credit Ap- proval dataset, we re-train 100 times for each epoch 35, 40,40,44, 45, 50, 55, 60. For Dermatol- ogy dataset, re-train 100 times for each epoch 2, 3,5,,,6,,
For Mammography dataset, we re-train 100 times for each epoch 11, 12, 13, 14, 15, 16, 17, 18, 19. For Contracep- tion dataset, re-train 100 times for each epoch 11, 12, 13, 14, 15, 16, 17, 18, 19. Similarly,
Table D.4: UCT dataset deseriptions.
Datasct
# of features
Training set size
Test set size
Label (# of classes)
Adult Income
104
22621
7551
income >50K (2)
Bank Marketing
63
30891
10297
has deposit (2)
Credit Approval
46
489
164
approved (2)
Dermatology
34
268
has erythema (2)
Mammography
5
622
208
benign or malignant (2)
Contraception
9
1104
369
long or short term (2)
28

---

Published as a conference paper at ICLR 2024
Table D.5: UCI dataset base model results.
Training
Test
Datasct
Loss
Accuracy
Loss
Accuracy
Adult Income
0.3026
85.96%
0.3190
85.00%
Bank Marketing
0.1962
91.27%
0.2013
91.06%
Credit Approval
0.3149
88.14%
0.2975
87.80%
Dermatology 0.1138
99.25%
0.1248
98.89%
Mammography
0.3604
84.24%
0,4920
79.33%
Contraception 0.5658
73.64%
0.5986
69.38%
for Bernoulli and Gaussian dropouts, obtain 100 models each with 21 values of p and spread evenly between 0.2 and 0.6 respectively. Finally, we perform adversarial weight pertur- bation on four datasets (Credit Approval, Dermatology. Mammography and Contraception), with e E (0.000, 0.004, 0.008, 0.012, 0.016, 0.020, 0.024, 0.028, 0.032, 0.036, 0.040) and the perturba- tion learning rate is 0.001.
For MS COCO, we adopt two pre-trained models (the YoloV3 [Redmon Farhadi, 2018] and Mask R-CNN [He al., 2017] detectors) in MMDetection [Chen et al., 2019] vali- date the Rashomon effect on the detection models with the proposed dropout inference. For the YoloV3 detector, we use p ranged from 5e-5 le-4 for Bernoulli dropout and use a ranged from 0.03 to 0.1 for Gaussian dropout; while for the Mask R-CNN detector, we use ranged from 5e-5 to 3e-4 for Bernoulli dropout and use a ranged from 0.01 to 0.07 for Gaus- sian dropout. Then, we choose the models whose average precision on person class is within (0.005, 0.006, 0.007, 0. 0.008, 0.009, 0.010, 0.011, 0.011,0.012) 0.012).
29

---

Published as a conference paper at ICLR 2024
E ADDITIONAL RESULTS AND EXPERIMENTS
We include (i) the deviations of loss and accuracy versus different dropout parameters, for Bernoulli dropout and a for Gaussian dropout, (ii) the complete values of the estimated predictive multiplicity metrics, (iiithee comparison of the efficiency between re-training and dropout-based Rashomon set exploration, (iv) addition experiments on CIFAR-I0/-100 datasets, and (v) ablation studies.
E.I ADDITIONAL RESULTS ON UCI DATASETS
In Figure E.5, we show the deviation of loss and accuracy versus different dropout parameters for both Bernuuli (p E [0.0,0.2) 0.2)) and Gaussian (a E [0.0, 0.6)) dropouts. As
see, the dropout parameters can stably control the loss deviation and the according accuracy changes. Note that for all dropout parameters, the accuracy change is 1 1%. Moreover, the standard deviations of the loss difference and the accuracy are determined by the number of samples in the datasets-not necessarily the accuracy of the base models. For example, the Adult Income and Bank Marketing datasets have the most number of samples > 20000, and hence has the smallest standard deviations. On the other hand, despite that the Dermatology dataset also has 98% accuracy, it suffers from a relatively large standard deviation in terms of loss difference and accuracy.
Figure E.6, Figure E.7, Figure E.8, Figure E.9, Figure E.10, and Figure E.11 summarize the cu- mulative distributions of the predictive multiplicity metrics that are defined across all samples, and Figure 12 shows the ambiguity and discrepancy, for UCI Adult Income, Bank Marketing, Credit Approval, Dermatology, Mammography, and Contraception datasets [Asuncion & Newman, 2007]. Since all estimates of predictive multiplicity metrics are under-estimate of their true values, and therefore higher values indicate more effective method for estimation. Observing viable predic- tion range, score variance, and the Rashomon Capacity, it is clear that both Bernoulli and Gaussian dropouts more effective in terms of estimating predictive multiplicity metries compared training. Moreover, Bernoulli and Gaussian dropouts are also comparable to the adversarial weight perturbation algorithm. Note that we do not perform adversarial weight perturbation algorithm on Adult Income and Bank Marketing datasets since the number of samples is too large, which causes extreme time complexity.
Table E.6 summarizes the raw runtime values of four strategies, and the speedups are reported in Table 1. The runtimes for Bernoulli and Gaussian dropouts are in similar order, and therefore in Table 1 we only report the speedup of Gaussian dropout. The adversarial weight perturbation algorithm costs the most time since has to scan over all samples and perturb (i.e., train) the weights.
Finally, Table E.7 shows the efficiency of obtaining models from the Rashomon set with differ- ent Rashomon parameters e, compared with the re-training strategy. Note that it is hard to control the eventual loss value with the re- re-training strategy, and therefore we perform the re-training strat- egy will different number of epochs, in order to obtain models corresponding to different e in the Rashomon set. On the other hand, it is easier to control using dropout techniques, as supported in Figure E.5. For example, when c = 0.004, there are only less than 3% of the models obtained by the re- re-training strategy that are in the Rashomon set; however, by carefully controlling dropout parame- ters, there are more than 70% of the models obtained by Gaussian dropout that are in the Rashomon set. Moreover, Gaussian dropout is more "benign" compare to Bernoulli dropout in terms of chang- ing the model weights, since Bernoulli dropout directly removes the weights while Gaussian dropout only scales them.
In Figure E.13, we show the model weights of logistic regression learnt with the UCI Adult Income dataset, explored by the re-training and dropout strategies. Since the model weights (features) is digh-dimensional, we apply t-SNE visualization [Van der Maaten & Hinton, 2008] on the model weights for the sake of illustration. As observed, the model weights obtained by he re-training and dropout strategies largely overlapped with each other, indicating that they have similar performance in terms of exploring the Rashomon set when the hypothesis space is small. Note that there is no closed-fome characterization of the Rashomon set for logistic regression.
30

---

Published as a conference paper at ICLR 2024
0.0
0.00
600
0.0
-00
6.90
1ate N4.44
0390
1o1e
10.30
1440 Mtade
203
3.11
300
0.01
(a) Adult Income dataset.
(b) Bank Marketing dataset.
(c) Credit Approval dataset.
0.00
6.9
0.00
6.0
0.22
3.2
0.00
0C2
9.2
O.N
0.00
0.0
6.2
Lost Firrnse
LosL TeTICE
Lost
(d) Dermatology dataset.
(e) Mammography dataset.
(I) Contraception dataset.
Figure E.5; Loss and accuracy deviations versus Bernoulli and Gaussian dropouts on the UCI datasets.
Rashemon Parameter
6.002
< 0.002
ca 0.024
0.032
< 0.040
0.004
0.012
8.020
0.028
0.038
3.0
1.0
3.0
0.5
0.5
0.5
0.5
coe
0.2
0.4
000000 0000500 03300 0375
1.00
1.05
1.10
0.5
1.0
Burnallt
0.5
cDr
0.
0.000
0.010
1.3
0.5
1.0
CDT Gaustien
2.5
1.0
1.0
1.0
0.5
000
0.005
0.010
2,0
0.5
1.2
Viable Range
Score
Reshomon Cepacity
Ditg Deeggeemeent
Figure E.6: Cumulative distributions of the predictive multiplicity metrics that are defined across all samples the UCI Adult Income dataset.
Rashemon Parameter
6,002
< 0,003
0.0
<= 5.024
0.032 0.838
< 0.040
0.004
0.012
0.000
0.009
3.0
1.0
3.0
0.5
0.5
0.5
0.5
COP
0.0
0.2
0.4
0.6
020
0.005
0.230
0.015
1.0
1.1
1.2
0.0
0.5
1.0
cor Semples
Bernoull
0.2
0.25
3.50
0.75
1.2
0.5
100
1,5
1.0
1.0
1.0
Caussaan
0.5
0.5
0.0
000
8.055
0.00
2.0
0.5 Disigroemnt
16
Viable Prodiction
Range
Score Verianes
Rashemon Copactty by
Figure F.7: Cumulative distributions of the predictive multiplicity metries that defined across all samples the UCI Bank Marketing dataset.
31

---

Published as a conference paper at ICLR 2024
Reshamon
Parameter
0.000
0.008
0.016
0.024
0.032
0.040
0.020
Re training
cur
0.0
0.1
0.2
0.3
0.000
0.031
0.002
0.003
1.00
1,02
1.04
1.06
0.3
0.5
1.0
1.0
1.0
1,0
1.0
Bernoulti
0.5
0.5
0.5
0.5
0.
0.0
0.0
0.0
0.0050
1.0
1.2
100
1.0
1.0
1.0
1.0
Gaussian
0.5
0.5
CDP
0.0
0.0
0.25
0.50
0.75
0.000
0.002
0.004
1.0
1.2
144
16
0.5
1.0
AWP Samples
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
0.00
0.25
0.50
0.75
0.00
0.05
0.10
0.15
1.0
0.5
1.0
Viable Prediction Range
Score Variance
Rashoman Capacity
Disagreement
Figure E.8:
Cumulative distributions of the predictive multiplicity metries that are defined across all samples the UCI Credit approval dataset.
Rashomon Parameter
0.000
0.008
<= 0.016
0.024
= 0.032
< 0.040
Re training
CDF
0.0
0.1
0.2
0.3
0.000
0.002
3.004
1.000
1.025
1,050
1,075
0.0
0.3
1.0
1.0
1.0
1.0
1.0
Bemoulli
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.034
0.25
1.0
1.0
1.0
1.0
Gaussian
0.5
0.5
0.5
0.5
CDP
0.0
0.0
0.0
0.0
0.00
0.25
0.50
0.75
0.001
0.002 0.003
1.4
0.0
0.2
0.4
0.6
1.0
1.0
1.0
2.0
COP AWP
0.5
0.5
0.0
0.00
0.25
0.50
0.75
0.00
0.05
0.10
0.25
1.0
1.2
1.4
0.3
1.0
Viable Prediction Range
Score Variance
Rachomon Capacity
Disagreement
Figure E.9:
Cumulative distributions of the predictive multiplicity metrics that defined across all samples the UCI Dermatology dataset.
32

---

Published as a conference paper at ICLR 2024
0.000
0.008
0.016
0.024
<= 0.032
<= 0.040
Re -training
CD
0.0
0.2
0.000
0.002
0.004
1.000
1.025
1020
10755
0.0
0.3
10
1.0
1.0
1.0
1.0
Bernulli
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.2
0.010
1.0
0.0
0.5
1.0
1.0
1.0
1.0
Gaussian
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0000 3.0025 0.0050
0.0
CDF Samples
AWP
0.00
0.25
0,50
0.75
0.0
0.1
0.2
1.0
1.2
1.4
1.6
0.0
0.5 Disagreement
1.0
Viable Prediction
Score
Rashomon Capacity
Figure 10 Cumulative distributions the predictive multiplicity metries that defined across all samples the UCI Mammography dataset.
E 0.000
F- 0.008
<- 0.016
C 0.024
E- 0.032
-- 0.040
CDF training
1.0
0.2
0.4
0.6
0.000
0.005
0.010
1.0
1.1
1.2
0.0
0.5
1.0
1.0
1.0
1.0
1.0
Bemoulli
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.2
0.6
0.000
0.005
1.0
Gaussian
0.0
0.25
0.50
0.0000
0.0025 0.0050
CDF Samples
AWP
0.00
0.25
0.50
0.75
0.00
0.05
0.10
0.15
1.0
1.2
1.4
0.0
0.5
1.0
Viabin Prodiction Range
Variance
Rasheman Capacity
Disagreement
Figure E.11: Cumulative distributions the predictive multiplicity metrics that defined across all samples the UCI Contraception dataset.
33

---

Published as a conference paper at ICLR 2024
Ambiggtt
0.01
0.50
0.03
ning
2.00
Re-trardnng
9.02
Remaill
0.01
Remoull
Gai55
Gaussinn
Parameters
(a) Adult Income dataset.
(b) Bank Marketing dataset.
3.6
U.15
2.00
parsout
010
2.00
2.02
0.00
0.00
0.01
0.01
3.00
0.03
3.00
0.00
0.1
0.04
3.00
0.01
0.22
0.03
0.34
Rashamen Parameter
Rashomon
Paramerer
Rashomon Parameter
(c) Credit Approval dataset.
(d) Dermatology dataset.
3.8
0.4
Diccnpancc
0.4
0.50
0.05
347330
0.00
0.01
0.02
0.03
2.04
0.00
0.01
3.02
0.03
3.34
0.60
0.01
0.02
0.03
0.04
0.00
0.01
3,02
0.03
0.04
Reshamon Parameter
Rashomon Parameten
Rashomon Parameter
Rashommon Parameter
(e) Mammography dataset.
(f) Contraception dataset.
Figure E. Ambiguity and discrepancy of the UCI datascts.
Table E.6: Raw runtime values the UCI datasets.
Runtime (seconds per model)
Dataset
Bernoulli Dropout
Gaussian
Re-training
AWP
Dropout
Adult Income
19.35 + 0.68
0.0614 = 0.24
0.0633 + 0.24
Bank Deposit
7.78 2.94
0.2243 0.42
0.2200 0.41
Credit Approval
0.51 + 0.56
10.18 + 2.29
0.0019 = 0.04
0.0019 0.04
Dermatology
0.11 1 0.34
4.27 + 0.96
0.0038 1 0.06
0.0038 1 0.06
Mammography
0.31 + 0.52
10.09 +: 1.62
0.0024 +: 0.05
0.0024 4: 0.05
Contraception
0.56 0.67
21.09 + 3.78
0.0071 + 0.08
0.0071 4 0.08
34

---

Published as a conference paper at ICLR 2024
Table E.7: The efficiency of obtaining models from the Rashomon set with different Rashomon parameters the UCT datasets. Each number is the percentage of the obtained models that are in the Rashomon set by different methods. For most datasets and dropout has higher percentage compared re-training.
Rashomon
Methods
Dataset
Parameter
Bernoulli Dropout
Gaussian
Re-training
Dropout
0.001
29.50%
0.00%
75. 75.00%
0.002
55.25%
50.00%
82.67%
Adult Income
0.003
71.00%
66.67%
85.00%
0.004
81.25%
75.00%
87.38%
0.001
2.89%
0.00%
75.00%
0.002
9.89%
50.00%
83. 1.%%
Bank Marketing
0.003
18.44%
66.67%
85.43%
0.004
24.67%
75.00%
87.50%
0.001
0.33%
0.00%
63.00%
0.002
0.67%
50.00%
71.25%
Credit Approval
0.003
2.00%
50.00%
71.00%
0.004
2.67%
59.00%
72.00%
0.001
0.78%
0.00%
66. 75%
0.002
1.33%
50.00%
74.80%
Dermatoooyy
0.003
1.89%
50.00%
78.50%
0.004
2.11%
61.33%
79. 14%
0.001
20.78%
0.00%
57.50%
0.002
25.22%
50.00%
59.50%
Mammography
0.003
30.78%
52.33%
65.38%
0.004
38.11% %
57.75%
69.38%
0.007
92.00%
0.00%
66.00%
0.008
95.56%
59.00%
69. 69.43%
Contraception
0.010
97.89%
65.50%
73. 73.75%
0.011
08.89%
64.43%
75, 75.40%
0OOLD
0.520
0.030
0.040
0.C50
0.060
0.076
080
0.000
0,100
0.110
Exploration Strategies Reiralting Bemo: Deopout Gaussien Descout
Figure E.1 Comparisons model weights logistic regression obtained from re-training dropout strategics on UCI Adult Income datasct with t-SNE visualization [an der Maaten Hinton, 2008].
35

---

Published as a conference paper at ICLR 2024
E.2 ADDITIONAL RESULTS ON THE MS COCO DATASET
In Figure E.14 and Figure E. 16a, we report the estimation of all predictive multiplicity metrics that corresponds the example shown in Figure for the MS COCO dataset. There significant portion of samples (30% ~ 50%) that disagree with each other on the human bounding boxes found by the Yolov3 detector. Due to extremely high time complexity, we do include the re-training strategy and the adversarial weight perturbation algorithm.
In Figure E.15 and Figure 16b, we implement another detector Mask R-CNN [He et al., 2017], which is state-of-the-an model for instance segmentation, developed on top of a region-bases convolutional neural networks called Faster R-CNN. The Mask R- R-CNN detector makes a larger portion of the samples have higher predictive multiplicity, in terms of viable prediction range and Rashomon Capacity, compared against the Yolov3 detector.
0.015
0.005
Bernoull CDr Strrekes
1.0
1.0
1.0
3.0
0.5
0.5
1.0
3.0
Couosien of StStolle
0.0
0.55
0.50
0.75 1.00
0.3
0.2
1.00
1.25
0.00
0.25
0.50
0.75
Viable Pradiction Range
Score Variance
Rashoman Capacity
Dhaggeement
Figure 1414: Cumulative distributions of the predictive multiplicity metrics that are defined across all samples the MS COCO dataset with the V3 detector.
Parameter
5.025
0.005
0.010
1.0
1.0
1.0
3.0
0.5
CoT Semples
Gauzsias
0.0
0.
0.0
0.25
0.50
0.75
1.00
3.00
0.5 0.10 n15
0.20
1.00
125
3.50
0.00
0.25
0.50
0.75
Viable hrdiction Runge
Scare Variance
Ranhhoman Capacily
Dhageemment
Figure E. Cumulative distributions of the predictive multiplicity metrics that are defined across all samples the MS COCO dataset with the Mask R- R-CNN detector.
3.63
5,85b
0.400
5.00
3,400
0.211
355
448b
Cencier
Brow
8252
0.400
0.35
5.005
0.028
0.200
0.012
5.009
bidee
0820
0.02
0.000
0.008
0.20
0014
5,00
0.008
0.000
0.02
Holenmce Nataratte
Rashoman Paummeers
(a) The Yo detector.
(b) The Mask R-CNN detector.
Figure E. Ambiguity and discrepancy of the MS COCO dataset with the YoloV3 and Mask R-CNN detec- tors.
36

---

Published as a conference paper at ICLR 2024
E.3 ADDITIONAL EXPERIMENTS ON CIFAR- 100-100 DATASETS
We measure the predictive multiplicity metrics on more challenging image datasets CIFAR-10/-100
(Krizhevsky & Hinton, 2009] with convolutional neural network architectures including VGG16
monyan & Zisserman, 2014] and ResNet50 [He et al., 2016]. CIFAR-10 consists 60,000 32 x 32 colored tiny images divided into ten classes, with each class representing distinct object category such as airplanes, dogs, and cars. CIFAR-100 offers a more challenging dataset containing 100 classes with 600 images each, providing broader range of object categories, including fne-grained distinctions like various types of birds and flowers. We train CIFAR- 10 with VGGI6 for 7 epochs with learning rate 0.001, and CIFAR-100 with ResNet50 for 40 epochs with learning rate 0 performance of the bases models is summarized in Table E.8. We obtain 50 Bernoulli and Gaussian dropout models for each values p and a spread evenly in [0, 0.008 and [0, 0.1] respectively for CIFAR-10. 10, and similarly for CIFAR- 100 where p and spread evenly in [0, 0.002] and [0, 0.05]. For the re-training results, we re-train 20 times for each epoch 4,5,6,7,8,9 for CIFAR-10 10, and 20 times for each epoch 00,25,30.33 for CIFAR-100. The estimates of predictive multiplicity metrics are summarized in Figure E.18, and runtime/speedup Table E.9. Since VGG16 and ResNet50 are large-scale models that contain many local minima, re-training ima that have similar performance. On the other hand, the dropout methods can only search models at the neighborhood of the given pre-trate rre-trained model. Therefore, re-training outperforms dropout in terms of exploring diverse models the Rashomon set, at the expense of a much larger runtime. Finally, Figure 1.19 and Figure E.20 summarize the cumulative distributions of the predictive mul- tiplicity metrics that are defined across all samples for CIFAR-10 and CIFAR- 100 respectively, and Figure E.21 shows the ambiguity and discrepancy.
easily find different local min-
Table F.8: CIFAR-10/-100 dataset base model performance.
Training
Test
Dataset
Architecture
Loss
Accuracy
Loss
Accuracy
CIFAR-10 CIFAR-100
VGG-16 ResNet-50
0.2692 0.0014
90.85% 99.96%
0.5743
81.63%
2.1591
59.53%
1990
0003
600
112
2011
0002
2000 Vucce
5:37
3.51
537
CIFAR-I0 dataset.
(b) CIFAR- dataset.
Figure E.17: Loss and accuracy deviations Bernoulli and Gaussian dropouts CIFAR-10/-100 datasets.
BernouliRatte
Lo06 EM
OFAR-
9.00
0.0
0.0
0.000
0,002
OIFRR 160
Lose DR.
0,025
05 Ambiguity
GaussianVar.
VPR
ScoreVar.
Disagreement
Discrepancy
Re-training
Gaussian Dropout
Figure E.18 Loss dropout parameters and the corresponding predictive multiplicity metrics of the baselines with CIFAR-IO/-11 datasets.
37

---

Published as a conference paper at ICLR 2024
Table E.9: Raw runtime values and speedup CIFAR-10/-100 datasets. All runtimes evaluated with the
computational platform.
Runtime (seconds per model)
Speedup
Dataset
Bernoulli Dropout
Gaussian Dropout
Gaussian Dropout over Re-training
Re-training
CIFAR-IO (VGG16)
485.10 10.29
1.8720 = 0.33 1.0880 0.28
4.1040 0.93 1.3200 0.47
118.20x 3019.28> x
CIFAR-100 (ResNet50)
3985.45 15 1626.27
0.010
0.020
0.053
CoT training
1.0
1.0
1.0
1.0
Bemoultl
0.5
0.
0.5
cer Samples
AL0
0.5
1.0
0.000
0.005
0.050
2,03
1.0
1.5
20
25
0.8
0.2
0.4
Vible Prediction
Score
Rashomon
Figure E. Cumulative distributions of the predictive multiplicity metrics that are defined across all samples CIFAR- 10 dataset.
Rachomen
3.5
1.0
1.8
1.0
1.8
cat Bermoulti
0.
35
0.5
0.5
36
0.0
0.0
3.0
0.5
1.0
0.00
0.05
3:10
2
3
0.00
0.25
0.50
0.75
0.0
Vable Prediction
Rashomon
Figure E.20: Cumulative distributions of the predictive multiplicity metrics that are defined across all samples in CIFAR- 100 dataset.
3.20
0.6
0.35
Re traininngg Bemoulli Coussion
0.6
Decoepanc
Reirrining Bemoulii Goussion
0.4
5:10
0.2
66
2.00
0.00
0.02
0.04
3.20
0.02
0.34
0.00
0.32
0.06
0.00
0.02
0.04
Rashoron Parameter 7
Rastomon Penameter&
deveomoon Pamammeer
Rashoman Penameter
(a) CIFAR-I0 dataset.
(b) CIFAR dataset.
Figure E.21: Ambuguity dsicrepancy of CIFAR- 100----0--100000 datasets.
38

---

Published as a conference paper at ICLR 2024
Finally, implement the re-training dropout strategy, mentioned in Section 6, to better explore the Rashomon set. In Figure E.22 and E.23, the predictive multiplicity metrics presented in each row all estimated from 100 models in the Rashomon set. The first is 100 dropout models obtained from model via re-training. The second row 50 dropout models obtained separately from 2 different model via re-training (again 100 models in total). The third row is 25 dropout models obtained separately from 4 different model via re-training, and the fourth is 20 dropout models obtained separately from 5 different model via re-training. Observing multiplicity metrics such as the viable prediction range, the score variance and the Rashomon Capacity, it is clear that with more diverse model via re-training, the estimates of the multiplicity metrics have a higher cumulant. Take the viable prediction range value 0.5 if we apply dropout on only one re-training model (first row). However, there are 50% of the samples has up to value 0.5 if we apply dropout on 5 diverse re-training models (last row).
example, around 20% of the samples has up to
Samhomer Paramaber
Re- training:3 fe briiiing :4 Ra training Ra training
1.0
1.2
1.0
1,0
03
02
0.00
0.62
05
1.3
0.50
0:52
0.34
0.00
0.07
0.04
1.0
1.2
1.0
1.8
5.35
01
0100
10
0.50
0.34
0.01
0..0
1.0
1.0
1.0
0.185
0.800
0.
2.020
0.1
10
1.5
20
0.50
0.24
0.02
0.04
1.0
1.2
1.0
1.0
0.185
0.000
0.
6135
0.0
0.000
05
1.2
0.00
0.24
0.00
0.02
0.04
Wibre Prodicion Rango
Score Narlance
Bashomon
Rashomen Paromitor
Rashomon Paromete
Figure E.22: Re-training Bernoulli dropout to explore the Rashomon set on CIFAR- 10 dataset with VGG- 16.
0:000
0.210
5.035
5.032
0.040
3.053
Artiiiutt
Disceporccy
Re- training: Re- training: fe trainngg2 Rt training:
0.300
0.5
0137
0.
0.132
0.
16
1.5
1.3
0.50
0.34
0.00
0.00
3180
45
0.135
0.0
60
0.56
16
3.5
0.80
0.34
0.00
0.185
0.100
45
2.22
0.0
0.0
8.1
0.2
1.5
2.3
25
3.3
0.50
0.34
0.00
0.07
0.5
0.0
6.9
8/
0.5
1.2
0.00
0:52
0.34
0.00
0.02
0.00
Score
Capacky
Ossagicemett
Parameter
Figure E.23: Re-training Gaussian dropout to explore the Rashomon set the CIFAR- 10 dataset with VGG- 16..
39

---

Published as a conference paper at ICLR 2024
E.4 ABLATION STUDY
We use the UCI Adult Income dataset to perform ablation studies on the width and depth of the feed-forward neural networks, and use CIFAR-10/-100 for different neural network architectures.
We train the UCT Adult Income dataset on a neural network with different numbers of layers K (1, 5), and the number of neurons in each layers is 200. Figure E.24 shows the loss and accuracy deviations caused by Bernoulli and Gaussian dropouts. As the number of layers increases, the loss deviation also increases, which is consistent with the theoretical bound derived in (9). On the other hand, Figure E.25 shows that for a neural network with only one hidden layer and differ- ent number of neurons {200, 400, 600, 800, 1000). does not have a significant effect on the loss deviation under the same dropout parameter. The reason is in (9), the effect of number of layers K polynomial while the effect of the number of neurons m is linear.
Finally, we discuss the Rashomon effect with different neural network architectures on CIFAR-10/- 100 datasets. Note that VGG-16, ResNet-18 and ResNet-50 have 138M, 11M and 25.6M param- eters. We compare the loss deviation using Figure E.17 for VGG-16 and Figure E.26 for ResNet- 18/-50. For CIFAR-10, Bernoulli dropout on different architectures has similar performance, while Gaussian dropout leads to the largest loss deviation ResNet- 18. It is because ResNet- 18 has the least number of parameters. Moreover, ResNet-18 18 and ResNet-50 have similar loss deviations on CIFAR-100.
0.345
8:000
0.33
0:30 6m
$10 321
142
110
0111
0202 90% 0/20 2.05
0.35
111
20
511
0.002
0.02
0.600
182
0.30
0.30
200
0.32
36
Difle
Lose Cofferece
Difeereace
Figure E.24: Loss and accuracy deviations versus Bernoulli and Gaussian dropouts on Adult Income dataset with different numbers of layers.
Nexrrenn 200
of Naturess 400
Seureen E00
of Seurten 500
of Norrona 1002
220
5
0,7500
0.320
0.4315
0,5175
0.545
0.3000
0,3000
0,050
0.200
0.350
EdRS
60/00
0
0.310
0.10
0.30
3.23
2.24
0:30
6.00
1.4
0.39
0,37
0.30
6.00
E.04
2.66
0,2
Figure F.25: Loss and accuracy deviations versus Bernoulli and Gaussian dropouts on on Adult Income dataset with different numbers of neurons.
60M
3.00
00030
1200
C544
0.85
6331
1200
[540
10005
0.00
000000
612
004
6345
0.7
Made
Bata Malee
600
CO
C.4
330
6.22
(a) CIFAR-10 with ResNec--1.
(b) CIFAR-10 with ResNet.50.
(c) CIFAR-100 with ResNet- 18.
Figure E.26: Loss and accuracy deviations versus Bernoulli and Gaussian dropouts on CIFAR- 10000000000000 datascts with different neural network architectures.
40

---

Published as a conference paper at ICLR 2024
Rashomon Parameter
0.022
<- 0.013
2- 0.015
0.025
<- 0.031
2 0.037
Re treining CDF of Sample"
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0000
1.00
1.01
0.0
training (Calibrated)
CDF of Sample
1.0
1.0
1.0
0.0
3.1
0.2
0.000
0.001
0.002
1.00
1.01
1.02
0.00
0.25
0.50
0.75
Viable Prediction Range
Score Variance
Rashoman Capacity
Disagreement
Figure E.27: Score-based predictive multiplicity metries of the Credit Approval dataset with and without model calibration. Calibration, as score manipulation on average performance, will not affect predictive multiplic- ity.
Moreover, discuss whether model calibration could either reduce exacerbate predictive mul- tiplicity. a perfectly calibrated classifier assigns a 50% score to a sample (c.g.. in binary classifi- cation), it does not necessarily mean that this sample has high multiplicity. A perfectly calibrated classifier is one whose predicted classes matches the true classes on average across samples (e.g., samples predicted be 50% of one class have true outcomes matching that class 50% of the time). However, this does not necessarily translate to a (in)consistent set of predictions for a single target sample across equally calibrated classifiers. It may be the case that all calibrated models drawn from the Rashomon set assign the same 50% probability for that sample (no multiplicity). Conversely, some models may assign higher and lower confidence for that sample (high multiplicity) yet, on average, still be well-calibrated. Again, this happens because calibration (like accuracy and loss) is an average metric across all samples.
We implement Platt scaling [Platt et al., 1999] to calibrate the scores produced by the neural net- works for the Credit Approval dataset, and compare the score-based predictive multiplicity metrics with and without calibration in Figure E.27. We observe that the score-based predictive multiplicity metrics are not impacted with calibration, for the exact reasons we have provided above.
41


Reviews:
[{"review_id": "Lri9gAkvBT", "reviewer": "Reviewer_7265", "summary": "The paper studies the problem of how to measure and mitigate predictive multiplicity.\nTo achieve them, the authors utilize the dropout technique to explore the models in the Rashomon set.\nRigorous theoretical analysis is provided to connect dropout and Rashomon set.\nNumerical results demonstrate the effectiveness of the proposed method.", "questions": "1. It seems the proposed method is only evaluated on in-distribution scenarios. Can it be applied to out-of-distribution data?\n2. Are the uncertainty scores calibrated? In other words, are the confidence scores reliable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is simple, straightforward, and well-motivated. Utilizing the dropout technique to explore the models in the Rashomon set is interesting.\n2. Rigorous theoretical analysis is provided to connect dropout and Rashomon set.\n3. The paper is well-written and well-organized. The authors first show the implementations on linear models and extend them to feedforward neural networks. \n4. The limitations and potential solutions are also discussed in the paper.", "weaknesses": "1. In the experiments, the authors mentioned that \"On the other hand, AWP outperforms both dropouts and re-training, since it adversarially searches the models that mostly flip the decisions toward all possible classes for each sample.\" I may miss some details of the method part, how can the proposed method to adversarially search the models since the dropout is random?\n2. As mentioned by the authors, good performance comes at the cost of efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGsXt2gnyM", "reviewer": "Reviewer_pYox", "summary": "The paper described a Rashomon set exploration method through Drop-out with probabilistic bound. The paper starts with fairly well-covered literature of Rashomon set research and motivates its proposal by pointing out the computation cost of existing empirical solution (re-training, AWP). The solution is fairly simple by adopting Drop-out where Rashomon set likely rests. Probably the most significant part of this paper (theoretically) would be pointing out the probabilistic bound of Rashomon set under Drop-out. Empirical results show the proposed method is computationally efficient than previous solutions and even showing better divergence metric than retraining.", "questions": "The proposition 1 uses deviation between  $L_{SSE}(\\mathbf{w}_D^*)$ and $L_{SSE}(\\mathbf{w}')$ but not original model parameter $L_{SSE}(\\mathbf{w}')$. How to make the connection ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Predictive multiplicity itself is an interesting topic that worths more investigation. The method proposed in this work is a great complement of existing literature in this field.\n2. The paper is well written and motivated. It comes with sufficient background knowledge to understand the gap in the literature.\n3. Potential application of this approach is covered in Section 5, which is good since I was concerning where people can use this innovation in their work.", "weaknesses": "1. Model augmented by Dropout could result in a fairly small search space of Rashomon set. I am not very convinced that this is a good idea in practice if our goal is to look for a better model that can address various reliability problem of predictive model. e.g. fairness etc. It maybe inspirational to see the movement of predictive multiplicity measurement, but I am wondering what is the practical meaning of it.\n2. The paper demonstrates the effectiveness of the proposed method on toy datasets that were used for decades. As the paper concerns the efficiency of existing methods, I am wondering if the authors can introduce more realistic tasks to show the effectiveness of the proposed method quantitatively.  While COCO is good example, it is very qualitative without much statistic support. \n3. There is a descriptive gap in section 3.2 where transforming deviation between $L_{SSE}(\\mathbf{w}_D^*)$ and $L_{SSE}(\\mathbf{w})$ suddenly become  deviation between  $L_{SSE}(\\mathbf{w}_D^*)$ and $L_{SSE}(\\mathbf{w}')$. I don't quite see why they are aligned or if the model works correctly under $L_{SSE}(\\mathbf{w}')$ if it is not trained with such dropout rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jG7KzS4g5I", "reviewer": "Reviewer_kvbj", "summary": "This paper studies the possibility of using Dropout to explore the Rashomon set. It proves that for a FFNN, we could bound the probability that a Dropout realization is in a certain Rashomon set. In experiments it shows that the proposed the method does not explore the Rashomon set as effectively as AWP (as measured by several predictive multiplicity metrics), but is much faster as it does not retrain any model.", "questions": "1. AWP is slower due to re-training, but the models are trained only once. Therefore, doesn't it run *faster* than Dropout (because it uses fewer samples/models to explore the Rashomon set) with a reasonably large test data?\n2. What does \"5 models\" mean in Figure 4b? 5 different base weights, or 5 different architectures?\n3. Is $\\epsilon$ and the $L$ in Eq.(5), (7) and (10) related to the \"sum\" of loss or the \"mean\" of loss? It seems like it's the sum? If so, by changing $\\epsilon$ to some offset on the mean loss, we can probably get a convergence basing on the sample size, which is much more meaningful than dimension of the model's hidden layers.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper establishes some theoretical bounds (although seemingly loose) on the probability that FFNNs with Dropout are in the corresponding Rashomon Set.\n2. The proposes method is easy to implement.", "weaknesses": "1. It is unclear what's the practical use of the propose method. It is fast, but it does not explore the Rashomon set well. For this reason, we can only mitigate predictive multiplicity *as estimated by Dropout* but not in general. \n2. Following 1, it seems like additional experiments on whether the mitigation via Dropout also transfers to, say, AWP, is interesting. \n3. The bounds in Proposition 2 and 3 only converge to 1 when $d\\to\\infty$, which does not seem like useful. See Q3 as well.\n4. It is not clear why a concentration bound helps. Notably, in applications, we want the models that are in the Rashomon set but closer to the boundary. In fact, it seems like in practice we need to sample a few weights and empirically verify that they have low loss (?). If so, a concentrated distribution, especially one that's more concentrated when the dim of the model increases, seems like a bad feature. A method that samples very diverse model that potentially has a higher probability of falling outside the Rashomon set seems more desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGVg9HH2dT", "reviewer": "Reviewer_BakJ", "summary": "This paper’s main goal is to explore Rashomon sets of feed-forward neural networks with the help of dropout. Both Gaussian and Bernoulli dropouts are considered, the former involving the addition of noise to the weights of the networks. The approach is used to estimate the Rashomon set, and proof of the consistency of the approach is provided. Empirical results show the effectiveness of the approach on various datasets.", "questions": "1 – It is said that « not all estimators of predictive multiplicity metrics carry a theoretical analysis of its statistical properties such as consistency and sample complex » Could you provide some citation supporting this claim?\n\n2 – What justifies fixing a single Bernoulli or a Gaussian dropout parameter for all layers simultaneously? Shouldn’t the layers be treated independently?\n\n3 – Concerning the quantification of predictive multiplicity, it is said that « [f]or example, Long et al. [2023], Cooper et al. [2023] and Watson-Daniels et al. [2023] quantify predictive multiplicity by the standard deviation, variance and the largest possible difference of the scores (termed viable prediction range (VPR) therein) respectively » So, what definition between those three is retained in the article?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea is intuitive. It leads to impressive computation time saving compared to other approaches from the literature. The article is well-written and clear. The experiments are directly in line with the motivations of the work (ethical concerns).", "weaknesses": "**Major**\n\n1.1 – The biggest weakness of the approach concerns its limitations. As honestly discussed by the authors in Section 6 – Limitations, the fact that when the hypothesis space to explore is huge (that is when the predictor has many parameters to tune), the exploration that is done by the dropout approach is fairly limited. This is clearly related to Proposition 4, where an important value of $M$ is necessary for layers having hundreds of neurons and an important value of $k$ is necessary with complex neural networks.\n\n1.2 – Two scenarios could occur: the first one is that the hypothesis space to explore is relatively small. That is, it is fairly explored by the dropout method. But, even though a 30x to 5000x speedup over other approaches is seen, it is never defended that those other approaches are not scalable with small hypothesis space. Even though there is a huge speedup gain, if the other approaches are relatively fast (do not take hours to compute), then why favourising dropout? The second scenario is that the hypothesis space to explore is large. The time gain is then undermined by the limitation in the exploration. Plus, when it comes to large models, it is common to retrain only the classification head of the predictor, or to fix many layers; doing so could really fasten the retraining scheme, thus undermining the potential advantage of the dropout approach.\n\n2 – It seems to me that both the depiction in Figure 2 and the speedup reported in Table 1 are lacking important details. For example: How many different models are sought? How many reruns were done for retraining VS how many dropouts were computed? What was the total time for each individual method? To my understanding, many reruns are already needed in the first place, no matter the approach, in order to ensure that the reference model is an « empirical minimizer »; was that taken into account when comparing the time for building the empirical Rashomon sets in Table 1? What was the size of the predictor used on these different UCI tasks (this kind of information is necessary in the main article, not the supplementary material)?\n\n3 – I feel like something is conceptually wrong with the comparison between retraining and the current dropout scheme. Retraining makes it such that the validation loss is the highest possible. Therefore, it makes sense that many runs are needed in order to find models close to the « empirical minimizer». With the dropout scheme, an empirical minimizer is found, and then dropout is applied while making sure the training loss does not diminish too much. The two approaches do not have the same objectives.\n\n4.1 - The dropout leads to a scheme where each new model depends on the initial model. All of the models are thus dependent. Therefore, the estimation of the Rashomon metrics is biased. And while « not all estimators of predictive multiplicity metrics carry a theoretical analysis of its statistical properties such as consistency and sample complex », I feel like it is a property of interest. Indeed, one of the motivations of the work is the need for ethics and, more specifically, fairness. I see the goal in exploring the Rashomon sets to find many predictors giving different predictions to people for different reasons. But having all of the models interconnected makes it such that the reasons for the predictions are all linked and just a few are explored with the dropout scheme.\n\n4.2 – Proposition 5 aims at proving that the approach is not biased, but relies on the assumption that «  the models around W∗ are uniformly distributed in a d-dimensional ball with center $\\mathbf{W}^*$ and radius $\\delta$, i.e., $B(\\mathbf{W}^*, \\delta)$. Accordingly, we may assume that the population means $\\mu$ for a sample can be expressed as [...] ». The method explicitly does that (especially the Gaussian dropout), exploring around the « population mean », that is, the empirical minimizer. Therefore, assuming the uniform distribution of the Rashomon set around a center trivially leads to the unbiasedness of the dropout scheme, but is unreasonable.\n\n**Minor**\n\n1 – Typo: « Moreover, as lone as »", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)