SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recall that  $a_{i}$  is sample agreement over the pattern Rashomon set. Then we can compute the pattern diversity based on sample agreement as in Theorem 9.
Theorem 9 (Pattern diversity via sample agreement). For 0-1 loss, dataset  $S$ , and pattern Rashomon set  $\pi(\mathcal{F},\theta)$ , pattern diversity can be computed as  $\text{div}(\hat{R}_{\text{set}}(\mathcal{F},\theta)) = \frac{2}{n}\sum_{i=1}^{n}a_i(1 - a_i)$ , where  $a_i = \frac{1}{\Pi}\sum_{k=1}^{\Pi}\mathbb{1}_{[p_k^i=y_i]}$  is sample agreement over the pattern Rashomon set.
Proof. Let  $y \in \{0,1\}$ . We can transform  $y \in \{-1,1\}$  to  $\{0,1\}$ , simply by adding one and dividing by two.
Recall that Hamming distance  $H(p_j, p_k) = \sum_{i=1}^{n} \mathbb{1}_{[p_j^i \neq p_k^i]}$ . Alternatively, we can rewrite logical XOR as  $\mathbb{1}_{[p_j^i \neq p_k^i]} = p_j^i(1 - p_k^i) + p_k^i(1 - p_j^i)$ . Denote  $b_i = \frac{1}{\Pi} \sum_{j=1}^{\Pi} p_j^i$ , then from the pattern diversity definition:
$$
\begin{array}{l} d i v (\hat {R} _ {s e t} (\mathcal {F}, \theta)) = \frac {1}{n \Pi \Pi} \sum_ {j = 1} ^ {\Pi} \sum_ {k = 1} ^ {\Pi} \sum_ {i = 1} ^ {n} \mathbb {1} _ {[ p _ {j} ^ {i} \neq p _ {k} ^ {i} ]} = \\ = \frac {1}{n \Pi \Pi} \sum_ {j = 1} ^ {\Pi} \sum_ {k = 1} ^ {\Pi} \sum_ {i = 1} ^ {n} \left[ p _ {j} ^ {i} \left(1 - p _ {k} ^ {i}\right) + p _ {k} ^ {i} \left(1 - p _ {j} ^ {i}\right) \right] \\ = \frac {1}{n \Pi \Pi} \sum_ {j = 1} ^ {\Pi} \sum_ {k = 1} ^ {\Pi} \sum_ {i = 1} ^ {n} \left[ p _ {j} ^ {i} + p _ {k} ^ {i} - 2 p _ {k} ^ {i} p _ {j} ^ {i} \right] \\ = \frac {1}{n \Pi} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {\Pi} \left[ \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} p _ {j} ^ {i} + \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} p _ {k} ^ {i} - 2 \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} p _ {k} ^ {i} p _ {j} ^ {i} \right] \\ \end{array}
$$
(c)
Figure 6: Illustration of how reparameterization changes pairwise disagreement metric. Consider a separable dataset of four data points with a real-valued feature in one dimension:  $S = \{(1,0), (2,0), (3,1), (4,1)\}$  and a hypothesis space of linear models. Let the Rashomon parameter be  $\theta = 0.25$ . There are three patterns in the Rashomon set: 0111, 0011, and 0001. The pattern diversity (c) is 0.444. Consider two different parameterizations for the hypothesis space of linear models:  $ax \pm 1$  and  $\pm x + b$ . These two parameterizations produce the same decision boundaries for the dataset  $S$ . For the parameterization  $\pm x + b$  (b), each pattern is achieved with the same number of models. For the parameterization  $ax \pm 1$  (a), more models will support patterns that are closer to the origin. The support of each pattern is shown in a different color. The pairwise disagreement metric is 0.321 for  $ax \pm 1$  and 0.444 for  $\pm x + b$ . (For the parameterization  $ax \pm 1$ , we see that the pattern 0001 occurs when  $a \in (1, \frac{1}{2})$ , the pattern 0011 occurs when  $a \in (\frac{1}{2}, \frac{1}{3})$ , and the pattern 0111 occurs when  $a \in (\frac{1}{3}, \frac{1}{4})$ . Therefore, the pattern 0001 has probability  $w_1 = \frac{1 - \frac{1}{2}}{\frac{3}{4}} = 0.666$ , the pattern 0011 has probability  $w_2 = \frac{1}{2} - \frac{1}{3} = 0.222$ , and the pattern 0111 has probability  $w_3 = \frac{1}{3} - \frac{1}{4} = 0.111$ . Recall that  $H(c\mathrm{dot},\cdot)$  is the Hamming distance, then the pairwise disagreement metric is  $w_1w_2H(0001,0011) + w_1w_3H(0001,0111) + w_2w_3H(0011,0111) = w_1w_2 + 2w_1w_3 + w_2w_3 = 0.321$ . For the parameterization  $\pm x + b$ , each pattern has equal probability  $\frac{1}{3}$ . We can then similarly calculate that the pairwise disagreement metric is 0.444). Note that if the data points are shifted together to the left, the difference in pairwise disagreement metrics for parameterizations in (a) and (b) will only grow.
$$
\begin{array}{l} = \frac {1}{n \Pi} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {\Pi} \left[ p _ {j} ^ {i} + b _ {i} - 2 b _ {i} p _ {j} ^ {i} \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \frac {1}{\Pi} \sum_ {j = 1} ^ {\Pi} p _ {j} ^ {i} + \frac {1}{\Pi} \sum_ {j = 1} ^ {\Pi} b _ {i} - 2 \frac {1}{\Pi} \sum_ {j = 1} ^ {\Pi} b _ {i} p _ {j} ^ {i} \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ b _ {i} + b _ {i} - 2 b _ {i} ^ {2} \right] \\ = \frac {2}{n} \sum_ {i = 1} ^ {n} b _ {i} (1 - b _ {i}). \\ \end{array}
$$
On the other hand, according to logical XNOR, we have that  $\mathbb{1}_{[p_k^i = y_i]} = p_k^i y_i + (1 - p_k^i)(1 - y_i)$ , therefore we can rewrite  $a_{i}$  as:
$$
\begin{array}{l} a _ {i} = \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \mathbb {1} _ {[ p _ {k} ^ {i} = y _ {i} ]} \\ = \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \left[ p _ {k} ^ {i} y _ {i} + \left(1 - p _ {k} ^ {i}\right) \left(1 - y _ {i}\right) \right] \\ = \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \left[ 2 p _ {k} ^ {i} y _ {i} + 1 - y _ {i} - p _ {k} ^ {i} \right] \\ = 2 y _ {i} \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} p _ {k} ^ {i} + 1 - y _ {i} - \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} p _ {k} ^ {i} \\ \end{array}
$$
$$
= 2 y _ {i} b _ {i} + 1 - y _ {i} - b _ {i}.
$$
Since  $y_{i}\in \{0,1\}$ , then  $y_{i}^{2} = y_{i}$  and we have that:
$$
\begin{array}{l} \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} (1 - a _ {i}) = \frac {2}{n} \sum_ {i = 1} ^ {n} (2 y _ {i} b _ {i} + 1 - y _ {i} - b _ {i}) (- 2 y _ {i} b _ {i} + y _ {i} + b _ {i}) \\ = \frac {2}{n} \sum_ {i = 1} ^ {n} \left(- 4 y _ {i} b _ {i} ^ {2} + 2 y _ {i} b _ {i} + 2 y _ {i} b _ {i} ^ {2} - 2 y _ {i} b _ {i} + y _ {i} + b _ {i} \right. \\ + 2 y _ {i} b _ {i} - y _ {i} - y _ {i} b _ {i} + 2 y _ {i} b _ {i} ^ {2} - y _ {i} b _ {i} - b _ {i} ^ {2}) \\ = \frac {2}{n} \sum_ {i = 1} ^ {n} \left(b _ {i} - b _ {i} ^ {2}\right) \\ = \frac {2}{n} \sum_ {i = 1} ^ {n} b _ {i} (1 - b _ {i}). \\ \end{array}
$$
Therefore we get:
$$
d i v (\hat {R} _ {s e t} (\mathcal {F}, \theta)) = \frac {2}{n} \sum_ {i = 1} ^ {n} b _ {i} (1 - b _ {i}) = \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} (1 - a _ {i}).
$$
I Proof for Theorem 10
Before providing the proof for Theorem 10, we show that average sample agreement over hypotheses that realize patterns in the pattern Rashomon set is negatively proportional to the average loss of these hypotheses. We use this intuition to derive an upper bound for average sample agreement and then discuss the upper bound for pattern diversity.
Let hypothesis pattern set  $\mathcal{H}_{\pi(\mathcal{F},\theta)} \subset \hat{R}_{\text{set}}(\mathcal{F},\theta)$  be a set of unique hypotheses corresponding to each pattern in  $\pi(\mathcal{F},\theta)$ , meaning that there is no  $f_1^\pi, f_2^\pi \in \mathcal{H}_{\pi(\mathcal{F},\theta)}$ , such that  $f_1^\pi \neq f_2^\pi$ , yet  $p^{f_1^\pi} = p^{f_2^\pi}$ .
Theorem 23. Average sample agreement over the pattern Rashomon set is negatively proportional to the average loss of models in the hypothesis pattern Rashomon set  $\mathcal{H}_{\pi}(\mathcal{F},\theta)$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i} = 1 - \hat {L} _ {a v g} (\mathcal {H} _ {\pi} (\mathcal {F}, \theta)),
$$
where  $\hat{L}_{avg}(\mathcal{H}_{\pi}(\mathcal{F},\theta)) = \frac{1}{\Pi}\sum_{k = 1}^{\Pi}\hat{L}(f_k^\pi)$ . Moreover, when the Rashomon parameter  $\theta = 0$ , then
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i} = 1 - \hat {L} (\hat {f}).
$$
Proof. For a given  $(x_i, y_i)$ , when hypothesis  $f_k^\pi$  realizes pattern  $p^{f_k^\pi} = p_k$ , we have that  $p_k^i = f_k^\pi(x_i)$ . Consider average sample agreement:
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i} = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \mathbb {1} _ {[ p _ {k} ^ {i} = y _ {i} ]} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \mathbb {1} _ {[ p _ {k} ^ {i} \neq y _ {i} ]}\right) \\ \end{array}
$$
<sup>1</sup>Since there could be many hypotheses that achieve the same pattern,  $\mathcal{H}_{\pi(\mathcal{F},\theta)}$  is not unique. We can work with any of them, as  $\mathcal{H}_{\pi(\mathcal{F},\theta)}$  is simply a representation of the pattern set in the hypothesis space.
$$
\begin{array}{l} = 1 - \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \mathbb {1} _ {[ p _ {k} ^ {i} \neq y _ {i} ]} \\ = 1 - \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} _ {[ f _ {k} ^ {\pi} (x _ {i}) \neq y _ {i} ]} \\ = 1 - \frac {1}{\Pi} \sum_ {k = 1} ^ {\Pi} \hat {L} \left(f _ {k} ^ {\pi}\right) \\ = 1 - \hat {L} _ {a v g} (\mathcal {H} _ {\pi} (\mathcal {F}, \theta)). \\ \end{array}
$$
When  $\theta = 0$ , for any  $k$ ,  $\hat{L} (\hat{f}) = \hat{L} (f_k^\pi)$ , therefore  $\frac{1}{n}\sum_{i = 1}^{n}a_i = 1 - \hat{L} (\hat{f})$ .
Given the definition of models in the Rashomon set, we can derive an upper bound on average sample agreement in Corollary 24.
Corollary 24. For any parameter  $\theta >0$ , average sample agreement is upper and lower bounded by the empirical loss of the empirical risk minimizer,
$$
1 - \hat {L} (\hat {f}) - \theta \leq \frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i} \leq 1 - \hat {L} (\hat {f}).
$$
Proof. Proof follows directly from Theorem 23 and the fact that for every model  $f$  from the Rashomon set,  $\hat{L}(\hat{f}) \leq \hat{L}(f) \leq \hat{L}(\hat{f}) + \theta$ .
Finally, we provide proof for Theorem 10.
$$
d i v (\hat {R} _ {s e t} (\mathcal {F}, \theta)) \leq 2 (\hat {L} (\hat {f}) + \theta) (1 - (\hat {L} (\hat {f}) + \theta)) + 2 \theta .
$$
Proof. From the Cauchy-Schwarz inequality, we have that
$$
\left(\sum_ {i = 1} ^ {n} a _ {i}\right) ^ {2} \leq \sum_ {i = 1} ^ {n} 1 ^ {2} \sum_ {i = 1} ^ {n} a _ {i} ^ {2} = n \sum_ {i = 1} ^ {n} a _ {i} ^ {2}.
$$
Given this and from the definition of pattern diversity and Corollary 24 we get that:
$$
\begin{array}{l} d i v (\hat {R} _ {s e t} (\mathcal {F}, \theta)) = \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} (1 - a _ {i}) = \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} - \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} ^ {2} \\ \leq \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {i} - \frac {2}{n ^ {2}} \left(\sum_ {i = 1} ^ {n} a _ {i}\right) ^ {2} = 2 \left(\frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i}\right) - 2 \left(\frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i}\right) ^ {2} \\ \leq 2 (1 - \hat {L} (\hat {f})) - 2 (1 - \hat {L} (\hat {f}) - \theta) ^ {2} \\ = 2 - 2 \hat {L} (\hat {f}) - 2 + 4 (\hat {L} (\hat {f}) + \theta) - 2 (\hat {L} (\hat {f}) + \theta) ^ {2} \\ = 2 (\hat {L} (\hat {f}) + \theta - (\hat {L} (\hat {f}) + \theta) ^ {2} + \theta) \\ = 2 (\hat {L} (\hat {f}) + \theta) (1 - (\hat {L} (\hat {f}) + \theta)) + 2 \theta . \\ \end{array}
$$
When  $\theta = 0$ , then  $div(\hat{R}_{set}(\mathcal{F},0)) \leq 2\hat{L} (\hat{f})(1 - \hat{L} (\hat{f}))$ .
J Proof for Theorem 11
We state and prove Theorem 11 below.
Theorem 11. Consider a hypothesis space  $\mathcal{F}$ , 0-1 loss, and a dataset  $S$ . Let  $\rho \in (0, \frac{1}{2})$  be the probability with which each label  $y_i$  is flipped independently, and let  $S_{\rho} \sim \Omega(S_{\rho})$  denote a noisy version of  $S$ . For the Rashomon parameter  $\theta \geq 0$ , if  $\inf_{f \in \mathcal{F}} \mathbb{E}_{S_{\rho}} \hat{L}_{S_{\rho}}(f) < \frac{1}{2} - \theta$  and  $\hat{L}_S(\hat{f}_S) < \frac{1}{2}$ , then adding noise to the dataset increases the upper bound on pattern diversity of the expected Rashomon set:
Proof. Given the noise model, hypothesis  $f$  is in the expected Rashomon set if  $\mathbb{E}_{S_{\rho}}\hat{L}_{S_{\rho}}(f)\leq \inf_{f\in \mathcal{F}}\mathbb{E}_{S_{\rho}}\hat{L}_{S_{\rho}}(f) + \theta$ . Let  $\bar{f}\in \mathcal{F}$  be such that  $\bar{f}\in \arg \inf_{f\in F}\mathbb{E}_{S_{\rho}}\hat{L}_{S_{\rho}}(f)$ . Since  $\rho \in (0,\frac{1}{2})$ , and  $\hat{L}_S(\hat{f}_S) < \frac{1}{2}$  by assumption, from (3) and the definition of the empirical risk minimizer, we have that:
$$
\begin{array}{l} \inf  _ {f \in \mathcal {F}} \mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (f) = \mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (\bar {f}) \\ = (1 - 2 \rho) \hat {L} _ {S} (\bar {f}) + \rho \\ \geq (1 - 2 \rho) \hat {L} _ {S} (\hat {f} _ {S}) + \rho \\ > \hat {L} _ {S} (\hat {f} _ {S}). \\ \end{array}
$$
Consider  $g(x) = 2(x + \theta)(1 - x - \theta) + 2\theta$ . For  $x \in [0, \frac{1}{2} - \theta)$ ,  $g(x)$  is monotonically increasing, as  $g'(x) = 2(1 - x - \theta) - 2(x + \theta) = 4\left(\frac{1}{2} - x - \theta\right) > 0$ . Given monotonicity, assumption of the theorem  $\inf_{f \in \mathcal{F}} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f) < \frac{1}{2} - \theta$ , and since  $\hat{L}_S(\hat{f}_S) < \inf_{f \in \mathcal{F}} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f)$ , we have that
$$
\begin{array}{l} U _ {d i v} (\hat {R} _ {s e t _ {S}} (\mathcal {F}, \theta)) = 2 (\hat {L} _ {S} (\hat {f} _ {S}) + \theta) (1 - \hat {L} _ {S} (\hat {f} _ {S}) - \theta) + 2 \theta \\ <   2 \left(\inf _ {f \in \mathcal {F}} \mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (f) + \theta\right) \left(1 - \inf _ {f \in \mathcal {F}} \mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (f) - \theta\right) + 2 \theta \\ = U _ {d i v} (\hat {R} _ {s e t _ {\mathbb {E} _ {S _ {\rho}} S _ {\rho}}} (\mathcal {F}, \theta)). \\ \end{array}
$$
Interestingly, in the proof of Theorem 11, the empirical risk minimizer of dataset  $S$ ,  $\hat{f}_S$ , also minimizes the expected risks over noisy datasets, meaning that  $\hat{f}_S \in \arg \inf_{f \in F} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f)$ . To see this, assume that  $\hat{f}_S \notin \arg \inf_{f \in F} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f)$ , then there is  $\bar{f} \in \arg \inf_{f \in F} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f)$ , such that:
$$
\mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (\bar {f}) <   \mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (\hat {f} _ {S}).
$$
Applying (3) to both sides of the inequality above, we get that:
$$
(1 - 2 \rho) \hat {L} _ {S} (\bar {f}) + \rho <   (1 - 2 \rho) \hat {L} _ {S} (\hat {f} _ {S}) + \rho ,
$$
which after simplification becomes:
$$
\hat {L} _ {S} (\bar {f}) <   \hat {L} _ {S} (\hat {f} _ {S}).
$$
This is a clear contradiction, since  $\hat{f}_S$  is the empirical risk minimizer on  $S$ , and thus  $\hat{L}_S(\hat{f}_S) \leq \hat{L}_S(f)$  for any  $f \in \mathcal{F}$ , including  $\bar{f}$ . Therefore our assumption was incorrect, and  $\hat{f}_S \in \arg \inf_{f \in F} \mathbb{E}_{S_\rho} \hat{L}_{S_\rho}(f)$ .
K Setup for experiments
K.1 Datasets Description
Please see Table K.1 for the description of datasets used in the paper and all the processing steps. We normalize all real-valued features.
Table 1: Preprocessed datasets
Dataset	Number of Samples	Number of Features	Notes
Car Evaluation	1728	16	We use one-hot encoding for features
Breast Cancer Wisconsin	699	11	We use one-hot encoding for features
Monks 1	124	12	We use one-hot encoding for features
Monks 2	169	12	We use one-hot encoding for features
Monks 3	122	12	We use one-hot encoding for features
SPECT	267	23	We use one-hot encoding for features
COMPAS	6907	13	Processed in [52]
FICO	10459	18	Processed in [52]
Bar 7 (Coupon)	1913	15	Processed in [52]
Expensive Restaurant	1417	16	Processed in [52]
Carryout Takeaway	2280	16	Processed in [52]
Cheap Restaurant	2653	16	Processed in [52]
Coffee House	3816	16	Processed in [52]
Bar	1913	16	Processed in [52]
Telco Bin	7043	6	We use only the binary features
Iris	100	4	We consider classes Versicolour and Setosa
Wine	130	13
Wine 4	130	4	We use PCA to create 4 features
Seeds 4	140	4	We consider classes 1 and 2 and use PCA to create 4 features
Immunotherapy 4	90	4	[21, 22]. We use one-hot encoding for feature "type". We use PCA to create 4 features
Penguin 4	265	4	We use one-hot encoding for feature "island." We consider classes "adelie" and "gentoo" only and use PCA to create 4 features
Digits 0-4 4	359	4	We consider digit 0 and digit 4. We use PCA to create 4 features
K.2 Illustration of Cross-Validation Process in Step 3
We considered uniform label noise where each label is flipped independently with probability  $\rho$ . For each dataset, we performed five random splits into a train set and a validation set, where the validation set size is  $20\%$  of the number of samples. For the tree depth of CART, we considered the values  $d \in \{1, \dots, m\}$ , where  $m$  is the number of features for a given dataset.
For Figure 1(a), we tuned the parameters and then added noise to see what happens, which is that performance degrades. For every train/validation split we performed 5-fold cross-validation
Figure 7: Practitioner's validation process in the presence of noise for gradient boosted trees. For a fixed number of estimators, as we add noise, the gap between training and validation accuracy increases (Subfigure a). As we use cross-validation to select the number of estimators, the best number of estimators decreases with noise (Subfigure b).
on the training set and computed the best depth. We fixed this depth (and thus hypothesis space). Then, we start adding noise to the dataset. We considered six different noise levels,  $\rho \in \{0, 0.03, 0.05, 0.10, 0.15, 0.20, 0.25\}$ . For every level, we performed 25 draws of  $S_{\rho}$ . For every noise level, noise draw, and train/validation split, we evaluated train and validation performance and reported the average.
For Figure 1(b), we tuned the parameters for each noise level. We will see that noisier datasets lead us to use more regularization. We started adding noise to the dataset and then chose the best parameter based on cross-validation. More specifically, we considered six different noise levels,  $\rho \in \{0,0.03,0.05,0.10,0.15,0.20,0.25\}$ . For every level, we performed 25 draws of  $S_{\rho}$ . Then we performed 5-fold cross-validation on the training data to choose the best depth for CART. For every noise level, noise draw, and train/validation split, we report mean depth based on cross-validation results.
We performed a similar experiment to Figure 1 for the gradient boosting algorithm, where we varied the number of tree estimators. We observe similar behaviors, where, with more noise, the best number of estimators (according to cross-validation) decreases. We used the same level of noise and cross-validation procedure as discussed above. For the number of estimators, we considered values  $d \in \{5, 10, 20, \ldots, 150\}$ .
K.3 Branch and Bound Method to Compute Patterns in the Rashomon set
Here we describe a two-step method that allows us to compute all patterns in the pattern Rashomon set. In the first step, we reduce the complexity of the problem, by discarding points that have low sample agreement. In the second step, we use a branch-and-bound approach in order to enumerate patterns and discard prefixes of those patterns that will not be in the Rashomon set based on the Rashomon parameter and the empirical risk of the empirical risk minimizer.
Consider a dataset  $S = \{z_i\}_{i=1}^n$ . For every point  $z_i$  assume that we have sample agreement  $a_i$ . If  $a_i = 0$ , it means that all patterns in the pattern Rashomon set assign an incorrect label to sample  $z_i$ . On the other hand, if  $a_i = 1$ , all patterns assign the correct label. If we exclude all  $z_i$  such that  $a_i = 0$  or  $a_i = 1$  from the dataset, then the number of patterns will not change in the Rashomon set. Therefore, for a given point  $z_k$  ( $k = 1..n$ ) we will try to answer a question: is there a model in the Rashomon set such that it classifies  $\bar{z}_k = (x_k, -y_k)$  correctly and still stays in the Rashomon set. If there is no such model, then sample  $z_k$  has no influence on the pattern Rashomon set. Since it is harder to optimize for 0-1 loss, we instead consider exponential loss. If the problem is separable by 0-1 loss, then exponential loss will converge to a separable solution exponentially fast (which is known from the convergence analysis of AdaBoost [4]). Then given hypothesis space of linear models  $\mathcal{F} = \{w^T x\}$ , for every  $z_k$ , we aim to solve following optimization problem:
$$
\min  \frac {1}{n} \sum_ {i = 0} ^ {n} e ^ {- y _ {i} w ^ {T} x _ {i}} \tag {8}
$$
$$
y _ {k} w ^ {T} x _ {k} \leq 0, \tag {9}
$$
and then check if  $w^T x$  is in the Rashomon set defined by 0-1 loss.
Since we optimize exponential loss, it is fast to solve the optimization problem with gradient descent. More importantly, we can run the optimization in parallel for samples  $z_{k}$ . After, we consider dataset  $S_{\mathrm{inside}}$  that contains only those samples for which models were in the Rashomon set that could accommodate misclassified  $z_{k}$ . We formally define the dicard point procedure in procedure DISCART POINTS below:
procedure DISCARD POINTS(dataset S,ERM  $\hat{f}$  , the Rashomon parameter  $\theta$  Initialize  $S_{dp}$    
for every  $z = (x,y)\in S$  do Solve optimization problem (8)-(9). Let  $\bar{f}$  be a solution. if  $\hat{L} (\bar{f}) > \hat{L} (\hat{f}) + \theta$  then add  $z$  to  $S_{dp}$  (this point has a single predicted label for the entire Rashomon set). end if   
end for   
return  $S_{dp}$    
end procedure
In the second step, we build a search tree over the set of patterns that are formed by samples in  $S_{\mathrm{inside}}$ . We use breadth-first search over subsets of data. We "bound" (i.e., exclude part of the search space) when the prefix of the pattern (which is the part of the dataset we are working with) misclassifies more samples than the threshold to stay in the Rashomon set, which is  $\hat{L}(\hat{f}) + \theta$ . Since not all patterns can be realized by the model class. We "bound" if the prefix or pattern can not be achieved (the pattern is achievable when all points with labels matching the pattern are classified correctly by some model from the hypothesis space). In order to perform branch and bound more effectively, given an empirical risk minimizer (ERM), we sort points in the dataset based on their distance to the decision boundary of the ERM. More specifically, we split the points into four categories depending on whether the point is a true positive, false positive, true negative, or false negative. Then for every category, we compute the distances from each point to the decision boundary of the ERM and then sort points from least distance to greatest distance. Finally, we cyclically choose one point from each category until all samples have been considered. Conceptually, true positive and true negative samples that are closest to the decision boundary determine most of the patterns in the pattern Rashomon set. We add false positives and false negatives early to the order of points as they are more likely to be misclassified, allowing us to bound the prefixes sooner. We describe the branch and bound procedure in Algorithm 1. We use bit vectors to represent prefixes and patterns to speed up computations. Since we apply this approach to linear models, we use logistic regression without regularization to check the achievability of the patterns and their prefixes. However, the algorithm in general can be applied to other hypothesis spaces and losses (for example, hinge loss).