SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Branch and bound approach to find the pattern Rashomon set
Input: The Rashomon parameter  $\theta$ , dataset  $S = X \times Y$ ,  $\operatorname{ERM} \hat{f}$ , algorithm  $A$ .  
Output: Pattern Rashomon set  $\pi(\mathcal{F}, \theta)$ .  
1: Run DISCARD POINTS  $(S, \hat{f}, \theta)$  to exclude points that have the same predicted label for all models in the Rashomon set. Let  $S_{dp}$  be the set of discarded points, and  $S_{\text{inside}}$  be the rest of the points.  
2: Divide points in  $S_{\text{inside}}$  into four categories: true positive, false positive, true negative, and false negative.  
3: Compute the distance from the decision boundary of  $\hat{f}$  to every point for every category.  
4: Sort points in ascending order for every category.  
5: Create a new order of the points in  $S_{\text{inside}}$  by iteratively choosing points from each of the four categories until all points in  $S_{\text{inside}}$  are re-ordered.  
6: Concatenate  $S_{dp}$  and  $S_{\text{inside}}$  to form  $S = X \times Y$  based on the new order, where discarded points are followed by the sorted points.  
7: Initialize the prefix  $pinit$  of length  $|S_{dp}|$  based on the labels of the samples in  $S_{dp}$ .  
8: Initialize  $Q$  as the queue for the breadth-first search over the prefixes.  
9: while  $i \leq |S_{\text{inside}}|$  (loop over all points in  $S_{\text{inside}}$ ) do  
10: for every element in  $Q$  (loop over all prefixes in  $Q$ ) do  
11: for  $e \in [0, 1]$  (loop over possible labels; this is a "branch" step) do  
12:  $Y_a = Y_{S_{dp}} \cup e$  (consider potential prefix).  
13: Form the training data  $(X_a, Y_a)$  to check if the prefix is achievable by algorithm  $A$ .  $X_a$  consists of the first  $|S_{dp}| + i$  samples of sorted  $X$ .  
14: Fit algorithm  $A$  on  $(X_a, Y_a)$  and compute accuracy and loss.  
15: if accuracy = 1 and loss ≤  $\hat{L}(\hat{f}) + \theta$  then  
16: Q.append(elem  $\cup e$ ) (the prefix is achievable and the pattern has the potential to be achieved in the pattern Rashomon set, thus we add this element to the queue. This is a "bound" step).  
17: end if  
18: end for  
19: end for  
20: end while  
21: As we have now looped over all samples,  $Q$  contains all the achievable patterns that are in the Rashomon set, set  $\pi(\mathcal{F}, \theta) = Q$ .
K.4 Computation of Rashomon Ratio and Pattern Rashomon Ratio for Step 4
We considered the hypothesis space of sparse decision trees of various depths and the hypothesis space of linear models with a given number of non-zero coefficients.
For decision trees (Figure 2 (a)), we varied the depth of the maximum allowed decision tree from 1 to 7. To compute the numerator of the Rashomon ratio, we used TreeFARMS [52]. We set the Rashomon parameter to  $5\%$ . To compute the denominator of the Rashomon ratio, we considered all possible sparse decision trees up to a given depth  $d$  and used the following recursive formula to compute the size of hypothesis space with  $m$  features:
$$
C (d, m) = 2 + m C (d - 1, m - 1) ^ {2},
$$
where  $C(0, \cdot) = 2$ . In the base case, the only possible trees classify every point as 0, or every point as 1. Then for decision trees up to depth  $d$  with  $m \geq d$  features, there are two cases. The first case is when the tree has depth 0 which produces 2 possible trees. The other case is when the tree has depth at least 1. In this case, there are  $m$  possible features to initially split on, and then the left and right subtrees are of depth at most  $d - 1$  with  $m - 1$  features to choose from. The left and right subtrees can be chosen independently of each other, so we have  $mC(d - 1, m - 1)^2$  trees in this case, which proves the overall recursive formula.
Note that for decision trees of depth exactly equal to  $d$  for every tree path, following recursive formula holds
$$
C _ {\text {c o m p l e t e}} (d, m) = m C _ {\text {c o m p l e t e}} (d - 1, m - 1) ^ {2},
$$
which is equivalent to closed-form solution described in appendix E.
For the hierarchy of regularized linear models (Figure 2 (b)), we considered regularization for 1 non-zero coefficient, 2 non-zero coefficients, 3 non-zero coefficients, and 4 non-zero coefficients. To compute the numerator of the pattern Rashomon ratio, we used the approach described in Section K.3. We set the Rashomon parameter to  $3\%$ . To compute the denominator of the Rashomon ratio, we used the formula that gives the number of all possible patterns for the hypothesis space of linear models: if no  $m - 1$  points are coplanar,  $C(n,m) = 2\sum_{i=0}^{m}\binom{n-1}{i}$  [10].
K.5 Experiments for Pattern Diversity and Label Noise for Linear Models
(b)
Figure 8: Rashomon set characteristics such as the number of patterns in the Rashomon set (Subfigure a) and pattern diversity (Subfigure b) tend to increase with uniform label noise for hypothesis spaces of linear models.
For the hypothesis space of linear classifiers, we show the pattern diversity and the number of patterns in the Rashomon set for different datasets in the presence of noise in Figure 8. We considered uniform label noise, where each label is flipped independently with probability  $\rho$ . We set noise level  $\rho$  to values in  $\{0, 0.02, 0.04, 0.06, 0.08, 0.10, 0.15\}$  and performed five draws of  $S_{\rho}$  for every noise level. We then computed the pattern Rashomon set for each draw using the method described in Appendix K.3 and finally computed the pattern diversity. Both the number of patterns and pattern diversity tend to increase with label noise.
(b)
Figure 9: (a) Curve of the hypothesis space complexity vs. Rashomon ratio (as in Figure 2) stays the same shape for different Rashomon parameters for the Monks 3 dataset for the hypothesis space of sparse decision trees. (b, c) Rashomon characteristics tend to increase with uniform label noise for the hypothesis space of decision trees (as in Figure 3) for different Rashomon parameters for the Monks 3 dataset. For (b) and (c), we averaged over 25 iterations.
K.6 The Choice of the Rashomon Parameter does not Influence Results
For Figures 2(a) and 3, we set the Rashomon parameter to be  $5\%$ . In Figure 9, on the example of Monks 3 dataset, we show that the results in Figures 2(a) and 3 hold for different values of the Rashomon parameter.
K.7 Computation Resources
We performed experiments on Duke University's Computer Science Department cluster. We parallelized computations for the majority of the figures. It took up to 3 hours to compute the Rashomon sets for the hypothesis space of sparse decision trees for different noise levels and draws (Figures 3 and 9), and up to 48 hours for the hypothesis space of linear models (Figures 2 and 8).