SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lesia Semenova Harry Chen Ronald Parr Cynthia Rudin
Department of Computer Science, Duke University
{lesia.semenova, harry.chen084, ronald.parr, cynthia.rudin}@duke.edu
The Rashomon set is the set of models that perform approximately equally well on a given dataset, and the Rashomon ratio is the fraction of all models in a given hypothesis space that are in the Rashomon set. Rashomon ratios are often large for tabular datasets in criminal justice, healthcare, lending, education, and in other areas, which has practical implications about whether simpler models can attain the same level of accuracy as more complex models. An open question is why Rashomon ratios often tend to be large. In this work, we propose and study a mechanism of the data generation process, coupled with choices usually made by the analyst during the learning process, that determines the size of the Rashomon ratio. Specifically, we demonstrate that noisier datasets lead to larger Rashomon ratios through the way that practitioners train models. Additionally, we introduce a measure called pattern diversity, which captures the average difference in predictions between distinct classification patterns in the Rashomon set, and motivate why it tends to increase with label noise. Our results explain a key aspect of why simpler models often tend to perform as well as black box models on complex, noisier datasets.
It is possible that for many datasets, a simple predictive model can perform as well as the best black box model – we simply have not found the simple model yet. Interestingly, however, we may be able to infer whether such a simple model exists without finding it first, and we may be able to determine conditions under which such simple models are likely to exist.
We already know that many datasets exhibit the "Rashomon effect" [7], which is that for many real-world tabular datasets, many models can describe the data equally well. If there are many good models, it is more likely that at least one of them is simple (e.g., sparse) [39]. Thus, a key question in determining the existence of simpler models is to understand why and when the Rashomon effect happens. This is a difficult question, and there has been little study of it. The literature on the Rashomon effect has generally been more practical, showing either that the Rashomon effect often exists in practice [39, 13, 45], showing how to compute or visualize the set of good models for a given dataset [52, 14, 15, 1, 32, 53, 54, 48], or trying to reduce underspecification by learning a diverse ensemble of models [25, 37]. However, no prior works have focused on understanding what causes this phenomenon in the first place.
Our thesis is that noise is both a theoretical and practical motivator for the adoption of simpler models. Specifically, in this work, we refer to noise in the generation process that determines the labels. In noisy problems, the label is more difficult to predict. Data about humans, such as medical data or criminal justice data, are often noisy because many things worth predicting (such as whether someone will commit a crime within 2 years of release from prison, or whether someone will experience a medical condition within the next year) have inherent randomness that is tied to random processes in the world (Will the person get a new job? How will their genetics interact with their diet?). It might sound intuitive that noisy data would lead to simpler models being useful, but this is not
something most machine learning practitioners have internalized – even on noisy datasets, they often use complicated, black box models, to which post-hoc explanations are added. Our work shows how practitioners who understand the bias-variance trade-off naturally gravitate towards more interpretable modes in the presence of noise.
We propose a path which begins with noise, is followed by decisions made by human analysts to compensate for that noise, and that ultimately leads to simpler models. In more detail, our path follows these steps: 1) Noise in the world leads to increased variance of the labels. 2) Higher label variance leads to worse generalization (larger differences between training and test/Validation performance). 3) Poor generalization from the training set to the validation set is detected by analysts on the dataset using techniques such as cross-validation. As a result, the analyst compensates for anticipated poor test performance in a way that follows statistical learning theory. Specifically, they choose a simpler hypothesis space, either through soft constraints (i.e., increasing regulation), hard constraints (explicit limits on model complexity, or model sparsification), or by switching to a simpler function class. Here, the analyst may lose performance on the training set but gain validation and test performance. 4) After reducing the complexity of the hypothesis space, the analyst's new hypothesis space has a larger Rashomon ratio than their original hypothesis space. The Rashomon ratio is the fraction of models in the function class that perform close to the empirical risk minimizer. It is the fraction of functions that performs approximately-equally-well to the best one. This set of "good" functions is called the Rashomon set, and the Rashomon ratio measures the size of the Rashomon set relative to the function class. This argument (that lower complexity function classes leads to larger Rashomon ratios) is not necessarily intuitive, but we show it empirically for 19 datasets. Additionally, we prove this holds for decision trees of various depths under natural assumptions. The argument boils down to showing that the set of non-Rashomon set models grows exponentially faster than the set of models inside the Rashomon set. As a result, since the analyst's hypothesis space now has a large Rashomon ratio, a relatively large fraction of models that are left in the simpler hypothesis are good, meaning they perform approximately as well as the best models in that hypothesis space. From that large set, the analyst may be able to find even a simpler model from a smaller space that also performs well, following the argument of Semenova et al. [39]. As a reminder, in Step 3 the analysts discovered that using a simpler model class improves test performance. This means that these simple models attain test performance that is at least that of the more complex (often black box) models from the larger function class they used initially.
In this work, we provide the mathematics and empirical evidence needed to establish this path, focusing on Steps 1, 2, and 4 because Step 3 follows directly (however, we provide empirical evidence for Step 3 as well). Moreover, for the case of ridge regression with additive attribute noise, we prove directly that adding noise to the dataset results in an increased Rashomon ratio. Specifically, the additive noise acts as  $\ell_2$ -regularization, thus it reduces the complexity of the hypothesis space (Step 3) and causes the Rashomon ratio to grow (Step 4).
Even if the analyst does not reduce the hypothesis space in Step 3, noise still gives us larger Rashomon sets. We show this by introducing pattern diversity, the average Hamming distance between all classification patterns produced by models in the Rashomon set. We show that under increased label noise, the pattern diversity tends to increase, which implies that when there is more noise, there are more differences in model predictions, and thus, there could be more models in the Rashomon set. Hence, a much shorter version of the path also works: Noise in the world causes an increase in pattern diversity, which means there are more diverse models in the Rashomon set, including simple ones.
It is becoming increasingly common to demand interpretable models for high-stakes decision domains (criminal justice, healthcare, etc.) for policy reasons such as fairness or transparency. Our work is possibly the first to show that the noise inherent in many such domains leads to technical justifications for demanding such models.
Rashomon set. The Rashomon set, named after the Rashomon effect coined by Leo Breiman [7], is based on the observation that often there are many equally good explanations of the data. When these are contradictory, the Rashomon effect gives rise to predictive multiplicity [30, 6, 18]. Rashomon sets have been used to study variable importance [15, 14, 42], for characterizing fairness [40, 9, 2], to improve robustness and generalization, especially under distributional shifts [37, 25], to study
connections between multiplicity and counterfactual explanations [36, 53, 8], and to help in robust decision making [46]. Some works focused on trying to compute the Rashomon set for specific hypothesis spaces, such as sparse decision trees [52], generalized additive models [54], and decision lists [32]. Other works focus on near-optimality to find a diverse set of solutions to mixed integer problems [1], a set of targeted predictions under a Bayesian model [23], or estimate the Rashomon volume via approximating model in Reproducing Kernel Hilbert Space [31]. Black et al. [6] shows that the predictive multiplicity metric defined as expected pairwise disagreement increases with expected variance over the models in the Rashomon set. On the contrary, we focus on probabilistic variance in labels in the presence of noise.
Metrics of the Rashomon set. To characterize the Rashomon set, multiple metrics have been proposed [39, 38, 30, 49, 18, 6]. The Rashomon ratio [39], and the pattern Rashomon ratio [38] measure the Rashomon set as a fraction of models or predictions within the hypothesis space; ambiguity and discrepancy [30, 49] indicate the number of samples that received conflicting estimates from models in the Rashomon set; Rashomon capacity [18] measures the Rashomon set for probabilistic outputs. Here, we focus on the Rashomon ratio and pattern Rashomon ratio. We also introduce pattern diversity. Pattern diversity is close to expected pairwise disagreement (as in Black et al. [6]), however, it uses unique classification patterns (see Appendix G).
Learning with noise. Learning with noisy labels has been extensively studied [34], especially for linear regression [5] and, more recently, for neural networks [44] to understand and model effects of noise. Stochastic gradient descent with label noise acts as an implicit regularizer [12] and noise has been added to hidden units [35], labels [41], or covariances [50] to prevent overfitting in deep learning. When the labels are noisy, constructing robust loss [16], adding a slack variable for each training sample [19], or early stopping [27] also helps to improve generalization. In this work, we study why simpler models are often suitable for noisier datasets from the perspective of the Rashomon effect.
3 Notation and Definitions
Consider a training set of  $n$  data points  $S = \{z_{1}, z_{2}, \dots, z_{n}\}$ , such that each  $z_{i} = (x_{i}, y_{i})$  is drawn i.i.d. from an unknown distribution  $\mathcal{D}$ , where  $\mathcal{X} \subset \mathbb{R}^{m}$ , and we have binary labels  $\mathcal{Y} \in \{-1, 1\}$ . Denote  $\mathcal{F}$  as a hypothesis space, where  $f \in \mathcal{F}$  obeys  $f: \mathcal{X} \to \mathcal{Y}$ . Let  $\phi: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}^{+}$  be a 0-1 loss function, where for point  $z = (x, y)$  and hypothesis  $f$ , the loss function is  $\phi(f(x), y) = \mathbb{1}_{[f(x) \neq y]}$ . Finally, let  $\hat{L}(f)$  be an empirical risk  $\hat{L}(f) = \frac{1}{n} \sum_{i=1}^{n} \phi(f(x_i), y_i)$ , and let  $\hat{f}$  be an empirical risk minimizer:  $\hat{f} \in \arg \min_{f \in \mathcal{F}} \hat{L}(f)$ . If we want to specify the dataset on which  $\hat{f}$  was computed, we will indicate it by an index,  $\hat{f}_S$ .
The Rashomon set contains all models that achieve near-optimal performance and can be defined as:
Definition 1 (Rashomon set). For dataset  $S$ , a hypothesis space  $\mathcal{F}$ , and a loss function  $\phi$ , given  $\theta \geq 0$ , the Rashomon set  $\hat{R}_{set}(\mathcal{F},\theta)$  is:
$$
\hat {R} _ {s e t} (\mathcal {F}, \theta) := \{f \in \mathcal {F}: \hat {L} (f) \leq \hat {L} (\hat {f}) + \theta \},
$$
where  $\hat{f}$  is an empirical risk minimizer for the training data  $S$  with respect to loss function  $\phi$ :  $\hat{f} \in \arg \min_{f \in \mathcal{F}} \hat{L}(f)$ , and  $\theta$  is the Rashomon parameter.
Rashomon parameter  $\theta$  determines the risk threshold  $(\hat{L} (\hat{f}) + \theta)$ , such that all models with risk lower than this threshold are inside the set. For instance, if we stay within  $1\%$  of the accuracy of the best model, then  $\theta = 0.01$ . Given parameter  $\gamma >0$ , we extend the definition to the true risk by defining the true Rashomon set, containing all models with a bound on true risk  $R_{set}(\mathcal{F},\gamma) = \{f\in \mathcal{F}: L(f)\leq L(f^{*}) + \gamma \}$ , where  $f^{*}$  is optimal model,  $f^{*} = \arg \min_{f\in \mathcal{F}}L(f)$ .
In this work, we study how noise influences the Rashomon set and choices that practitioners make in the presence of noise. We measure the Rashomon set in different ways, including the Rashomon ratio, pattern Rashomon ratio, and pattern diversity (defined in Section 6). For a discrete hypothesis space, the Rashomon ratio [39] is the ratio of the number of models in the Rashomon set compared to the hypothesis space,  $\hat{R}_{\text{ratio}}(\mathcal{F}, \theta) = \frac{|\hat{R}_{\text{set}}(\mathcal{F}, \theta)|}{|\mathcal{F}|}$ , where  $|\cdot|$  denotes cardinality. It is possible to weight the hypothesis space by a prior to define a weighted Rashomon ratio if desired.
Given a hypothesis  $f$  and a dataset  $S$ , a predictive pattern (or pattern)  $p$  is the collection of outcomes from applying  $f$  to each sample from  $S$ :  $p^f = [f(x_1),\dots,f(x_i),\dots,f(x_n)]$ . We say that pattern  $p$  is achievable on the Rashomon set if there exists  $f \in \hat{R}_{set}(\mathcal{F},\theta)$  such that  $p^f = p$ . Let pattern Rashomon set  $\pi (\mathcal{F},\theta) = \{p^f : f \in \hat{R}_{set}(\mathcal{F},\theta), p^f = [f(x_i)]_{i=1}^n\}$  be all unique patterns achievable by functions from  $\hat{R}_{set}(\mathcal{F},\theta)$  on dataset  $S$ . Finally, let  $\Psi(\mathcal{F})$  be the pattern hypothesis set, meaning that it contains all patterns achievable by models in the hypothesis space,  $\Psi(\mathcal{F}) = \{p^f : f \in \mathcal{F}, p^f = [f(x_i)]_{i=1}^n\}$ . The pattern Rashomon ratio is the ratio of patterns in the pattern Rashomon set to the pattern hypothesis set:  $\hat{R}_{ratio}^{pat}(\mathcal{F},\theta) = \frac{|\pi(\mathcal{F},\theta)|}{|\Psi(\mathcal{F})|}$ .
In the following sections, we walk along the steps of our proposed path. Rather than trying to prove these points for every possible situation (which would be volumes beyond what we can handle here), we aim to find at least some way to illustrate that each step is reasonable in a natural setting.
4 Increase in Variance due to Noise Leads to Larger Rashomon Ratios
4.1 Step 1. Noise Increases Variance
One would think that something as simple as uniform label noise would not really affect anything in the learning process. In fact, we would expect that adding such noise would just uniformly increase the losses of all functions, and the Rashomon set would stay the same. However, this conclusion is (surprisingly) not true. Instead, noise adds variance to the loss, which, in turn, prevents us from generalizing.
For infinite data distribution  $\mathcal{D}$  consider uniform label noise, where each label is flipped independently with probability  $\rho < \frac{1}{2}$ . If  $\tilde{y}$  is a flipped label,  $P(\tilde{y} \neq y) = \rho$ . If the empirical risk of  $f$  is over  $\frac{1}{2}$  after adding noise, we transform  $f$  to  $-f$ . For a given model  $f \in \mathcal{F}$  let  $\sigma^2(f, \mathcal{D})$  be the variance of the loss, meaning that  $\sigma^2(f, \mathcal{D}) = \mathrm{Var}_{z \sim \mathcal{D}} l(f, z)$ . We show in the following theorem that, for a given  $f \in \mathcal{F}$ , label noise increases the variance of the loss.
Theorem 2 (Variance increases with label noise). Consider infinite true data distribution  $\mathcal{D}$ , and uniform label noise, where each label is flipped independently with probability  $\rho$ . Let  $\mathcal{D}_{\rho}$  denote the noisy version of  $\mathcal{D}$ . Consider 0-1 loss  $l$ , and assume that there exists at least one function  $\bar{f} \in \mathcal{F}$  such that  $L_{\mathcal{D}}(\bar{f}) < \frac{1}{2} - \gamma$ . For a fixed  $f \in \mathcal{F}$ , let  $\sigma^2(f, \mathcal{D}_{\rho})$  be the variance of the loss,  $\sigma^2(f, \mathcal{D}_{\rho}) = Var_{z \sim \mathcal{D}_{\rho}} l(f, z)$  on data distribution  $\mathcal{D}_{\rho}$ . For any  $0 < \rho_1 < \rho_2 < \frac{1}{2}$ ,
$$
\sigma^ {2} (f, \mathcal {D} _ {\rho_ {1}}) <   \sigma^ {2} (f, \mathcal {D} _ {\rho_ {2}}).
$$
The proof of Theorem 2 is in Appendix A. This covers the uniform noise case, but variance increases more generally, and we prove this for several other common cases in Appendix A. More specifically, we show that the variance increases with other types of label noise, such as non-uniform label noise (see Theorem 12 in Appendix A) and margin noise (see Theorem 15 in Appendix A). For non-uniform label noise, for a sample  $z = (x,y)$ , each label  $y$  is flipped independently with probability  $\rho_{x}$ , meaning that noise can depend on  $x$ . This noise model is more realistic than uniform label noise and allows modeling of cases when one sub-population has much more noise than another. We model margin noise such as that which arises from high-dimensional Gaussians. Because of the central limit theorem, data often follow Gaussian distributions, therefore this noise is realistic and models mistakes near decision boundary. Label noise in datasets is common. In fact, real-world datasets reportedly have between  $8.0\%$  and  $38.5\%$  label noise [44, 43, 24, 28, 51]. We hypothesize that a significant amount of label noise in real-world datasets is a combination of Gaussian (due to the central limit theorem) and random noise (for example, because of clerical errors causing label noise).
For the true Rashomon set  $R_{set}(\mathcal{F},\gamma)$ , we consider the maximum variance for all models in the true set:  $\sigma^2 = \sup_{f\in R_{set}(\mathcal{F},\gamma)}\mathrm{Var}_{z\sim \mathcal{D}}l(f,z)$ . Then, from Theorem 2 we have that maximum expected variance over the Rashomon set increases with noise.
Corollary 3 (Maximum variance increases with label noise). Under the same assumptions as in Theorem 2, we have that
$$
\sup  _ {f \in R _ {s e t} _ {\mathcal {D} _ {\rho_ {1}}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {1}}) <   \sup  _ {f \in R _ {s e t} _ {\mathcal {D} _ {\rho_ {2}}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {2}}).
$$
The next step is to show that this increased maximum variance leads to worse generalization.
4.2 Step 2. Higher Variance Leads to Worse Generalization
Here we use an argument based on generalization bounds. Generalization bounds have been the key theoretical motivation for much of machine learning, including support vector machines (SVMs), because the margin that SVMs optimize appears in a bound. While bounds themselves are not directly used in practice, the terms in the bounds tend to be important quantities in practice. Our bound cannot be calculated in practice because it uses population information on the right side, but it still provides insight and motivation.
Unlike standard bounds, we will use the fact that the user is using empirical risk minimization, and cross-validation to assess overfitting. Thus, for  $\hat{f}$ , there are two possibilities:  $\hat{f}$  is in the true Rashomon set, or it is not. If it is not, then for the empirical risk minimizer  $\hat{f}$ , the difference between the true and the empirical risk must be at least  $\gamma$ , which will be detected with high probability in cross-validation [20, 33]. In that case, the user will reduce their hypothesis space and we move to Step 3. If  $\hat{f}$  is in the true Rashomon set, it obeys the following bound.
Theorem 4 (Variance-based "generalization bound"). Consider dataset  $S$ , 0-1 loss  $l$ , and finite hypothesis space  $\mathcal{F}$ . With probability at least  $1 - \delta$ , we have that for every  $f \in R_{\text{set}}(\mathcal{F}, \gamma)$ :
$$
L (f) - \hat {L} (f) \leq \frac {2}{3 n} \log \left(\frac {\left| R _ {s e t} (\mathcal {F} , \gamma) \right|}{\delta}\right) + \sqrt {\frac {2 \sigma^ {2}}{n} \log \left(\frac {\left| R _ {s e t} (\mathcal {F} , \gamma) \right|}{\delta}\right)}, \tag {1}
$$
where  $\sigma^2 = \sup_{f\in R_{set}(\mathcal{F},\gamma)}\operatorname{Var}_{z\sim \mathcal{D}}l(f,z)$ , and  $n$  is number of samples in  $S = \{z_i\}_{i = 1}^n\sim \mathcal{D}$ .
The proof of Theorem 4 is in Appendix C. Note that generalization bounds are usually based on Hoeffding's inequality (see Lemma 17), which is a special case of Bernstein's inequality (see Lemma 16). In fact, we show in Appendix B that Bernstein's inequality, which we used to prove Theorem 4, can be sharper than Hoeffding's when the variance is less than  $\sigma_f^2 < \frac{1}{12}$  for a given  $f$ . Theorem 4 is easily generalized to continuous hypothesis spaces through a covering argument over the true Rashomon set (as an example, see Theorem 20), where the complexity is measured as the size of the cover over the true Rashomon set instead of the number of models in the true Rashomon set.
Let  $c(\mathcal{F}, n) = \frac{2}{3n} \log \left( \frac{|R_{\text{set}}(\mathcal{F}, \gamma)|}{\delta} \right)$ , which is the first term in the bound (1) in Theorem 4. According to Theorem 8 in Semenova et al. [39] under random label noise, the true Rashomon set does not decrease in size. Therefore,  $c(\mathcal{F}, n)$  at least does not decrease with more noise as it depends only on complexity. However, the second term  $\sqrt{3\sigma^2 c(\mathcal{F}, n)}$  depends on the maximum loss variance, which increases with label noise, as motivated in the previous section. This means with more noise in the labels, we would expect worse generalization, which would generally lead practitioners who are using a validation set to reduce the complexity of the hypothesis space.
As discussed earlier, we use cross-validation to assess whether  $\hat{f}$  overfits. From Proposition 5 (proved in Appendix D) we infer that if the empirical risk minimizer (ERM) does not highly overfit, it has a high chance to be in the true Rashomon set and thus Theorem 4 applies.
Proposition 5 (ERM can be close to the true Rashomon set). Assume that through the cross-validation process, we can assess  $\xi$  such that  $L(\hat{f}) - \hat{L} (\hat{f})\leq \xi$  with high probability (at least  $1 - \epsilon_{\xi}$ ) with respect to the random draw of data. Then, for any  $\epsilon >0$ , with probability at least  $1 - e^{-2n\epsilon^2} - \epsilon_{\xi}$  with respect to the random draw of training data, when  $\xi +\epsilon \leq \gamma$ , then  $\hat{f}\in R_{set}(\mathcal{F},\gamma)$ .
4.3 Step 3. Practitioner Chooses a Simpler Hypothesis Space
We have shown earlier that noisier datasets lead to higher variance and worse generalization. The question we consider here is whether one can see the results of these bounds in practice and would actually reduce the hypothesis space. For example, considered four real-world datasets and the hypothesis space of decision trees of various depths. In Figure 1 (a) we show that as label noise increases, so does the gap between risks (1 - accuracy) evaluated on the training set and on a hold out dataset for a fixed depth tree, and thus, as a result, during the validation process, the smaller depth would be chosen by a reasonable analyst. We simulated this by using cross validation to select the optimal tree depth in CART, as shown in Figure 1 (b). As predicted, the optimal tree depth decreases as noise increases. We also show that a similar trend happens for the gradient boosted trees in Figure