SlowGuess/NeurIPS_2023_OCR huggingface dataset:
since as we showed above,  $\omega^T\mu > 0$ , and  $\phi$  is the PDF of normal distribution  $\mathcal{N}(\bar{0},I)$  which is always positive. Therefore,  $L_{\mu_k}(f)$  is monotonically decreasing with respect to  $k$ , and we have that  $L_{\mu}(f) < L_{\mu_k}(f)$  for all  $0 < k < 1$ .
For the true distribution  $\mathcal{D}$ , since  $l$  is 0-1 loss, then for a given model  $f$ ,  $l(f,z)$  is Bernoulli distributed with mean  $p_{Ber} = \mathbb{E}_{z\sim \mathcal{D}}l(f,z) = L_{\mathcal{D}}(f)$  and variance  $\sigma_f^2 = p_{Ber}(1 - p_{Ber}) = L_{\mathcal{D}}(f)(1 - L_{\mathcal{D}}(f))$ . Therefore, the expected variance for a given model  $f\in R_{set}(\mathcal{F},\gamma)$  on distributions  $\mathcal{D}_{\mu}$  and  $\mathcal{D}_{\mu_k}$  obeys:
$$
\sigma^ {2} (f, \mu) = L _ {\mu} (f) (1 - L _ {\mu} (f))
$$
$$
\begin{array}{l} <   L _ {\mu_ {k}} (f) \left(1 - L _ {\mu_ {k}} (f)\right) \\ = \sigma^ {2} (f, \mu_ {k}), \\ \end{array}
$$
where the inequality arises from the fact that the parabola  $x(1 - x)$  is monotonically increasing along the interval  $x \in [0, 0.5]$ , and  $\mu_k = k\mu$  is closer to  $\bar{0}$  than  $\mu$ .
Note, that we can generalize Theorem 15 to the case when  $\Sigma$  is any positive-definite matrix that is not necessarily diagonal (covariance matrices are always positive semi-definite, and we now additionally assume that  $\Sigma$  does not have zero eigenvalues). Since  $\Sigma$  is real and symmetric, by the spectral theorem, there exists an orthogonal matrix  $Q \in \mathbb{R}^{m \times m}$  such that  $D = Q\Sigma Q^T$  where  $D$  is diagonal and contains eigenvalues of  $\Sigma$ . The diagonal elements of  $D$  must be real and positive since  $\Sigma$  is positive-definite. Then, consider the data distribution  $(Q\mathcal{X}) \times \mathcal{Y}$ . From the properties of the Gaussian distribution,  $Q\mathcal{X}$  is Gaussian with mean  $Q\mu$  and covariance matrix  $Q\mathcal{X}Q^T = D$ . Thus, we can generalize the results of Theorem 15 to apply to positive-definite non-diagonal matrices  $\Sigma$ .
For a fixed model, we additionally verify the results of Theorem 15 empirically, by generating Gaussian distributions and introducing margin noise by moving the Gaussians closer together (see Figure 5(b).) The variance of losses increases with additive and uniform random attribute noise as well, as we show empirically in Figure 5(c)-(d).
While the results of Theorems 12, 15 are for a given and fixed model  $f$ , they hold for the  $f$  that achieves the maximum variance in the Rashomon set as well, meaning that Corollary 3 extends to Theorems 12, 15.
B Bernstein's and Hoeffding's inequalities
In this section, we compare Bernstein's and Hoeffding's inequalities and show that, under certain assumptions on variance, Bernstein's inequality is tighter than Hoeffding's inequality. We provide Bernstein's inequality in Lemma 16 and Hoeffding's inequality in Lemma 17.
Lemma 16 (Bernstein's inequality for loss class). Consider a hypothesis space  $\mathcal{F}$ . For a fixed  $f \in \mathcal{F}$ , let loss  $l$  be bounded by  $C > 0$  such that  $|l(f, z)| \leq C$  for every  $z \in \mathcal{Z}$ . For any  $\varepsilon > 0$ ,
$$
P \left(L (f) - \hat {L} (f) > \varepsilon\right) \leq e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma_ {f} ^ {2} + 2 C \varepsilon / 3}}, \tag {4}
$$
where  $\sigma_f^2 = \mathrm{Var}_{z\sim \mathcal{D}}l(f,z)$ , and  $n$  is number of samples in  $S = \{z_i\}_{i = 1}^n\sim \mathcal{D}$ .
Lemma 17 (Hoeffding's inequality for loss class). Consider a hypothesis space  $\mathcal{F}$ . For a fixed  $f\in \mathcal{F}$ , let loss  $l$  be bounded by  $a,b\geq 0$  such that  $a\leq l(f,z)\leq b$  for every  $z\in \mathcal{Z}$ . For any  $\varepsilon >0$
$$
P (L (f) - \hat {L} (f) > \varepsilon) \leq e ^ {\frac {- 2 n \varepsilon^ {2}}{(b - a) ^ {2}}}, \tag {5}
$$
where  $n$  is the number of samples in  $S = \{z_i\}_{i=1}^n \sim \mathcal{D}$ .
Note that for 0-1 loss in the lemmas above,  $a = 0$ ,  $b = 1$ , and  $C = 1$ . Now we show that Bernstein's inequality is stronger than Hoeffding's if variance is lower than  $\frac{(b - a)^2}{12}$ .
Theorem 18 (Bernstein's inequality is stronger than Hoeffding's for lower variance). For a fixed  $f \in \mathcal{F}$ , let loss  $l \in [a, b]$  so that  $a \leq l(f, z) \leq b$  for every  $z \in \mathcal{Z}$ . Then, Bernstein's inequality is stronger than Hoeffding's inequality for all  $\varepsilon \in (0, b - a)$  if  $\sigma_f^2 \leq \frac{(b - a)^2}{12}$  or if  $|L(f) - \frac{a + b}{2}| > \frac{b - a}{\sqrt{6}}$  where  $\sigma_f^2 = \operatorname{Var}_{z \sim \mathcal{D}} l(f, z)$ .
Note that since the true risk and empirical risk can only differ by at most  $b - a$ ,  $\epsilon$  is not meaningful if  $\epsilon \geq b - a$ .
Proof. According to Hoeffding's inequality (5), we have that
$$
P \left(\left| L (f) - \hat {L} (f) \right| > \varepsilon\right) \leq 2 e ^ {\frac {- 2 n \varepsilon^ {2}}{(b - a) ^ {2}}}.
$$
(d)
Figure 5: The variance of losses increases with margin (b) and additive attribute (c, d) noise. For (b) and (c) we generated data from Gaussians in 3, 5, 7, and 10 dimensions. For margin noise (b), as illustrated in (a), the negative class is generated from  $\mathcal{N}(\bar{\mu}_1,I)$  and positive from  $\mathcal{N}(\bar{\mu}_2,I)$ , where  $I$  is the identity matrix,  $\bar{\mu}_1 = -m/2 \times \bar{1}$ ,  $\bar{\mu}_2 = m/2 \times \bar{1}$ , and  $m$  controls the distance between Gaussians that determines the amount of margin noise. For additive noise, data is generated from  $\mathcal{N}(\bar{0},I)$  and  $\mathcal{N}(\bar{2},I)$ . The noise model is  $x' = x + \epsilon$ , where  $\epsilon \sim \mathcal{N}(\bar{0},\sigma I)$  is the noise vector added to every sample and  $\sigma$  determines how much noise is added to the data. For evaluation, as a fixed model we consider a random linear model from the Rashomon set. For (d), we chose 3 features with the highest AUC value and introduced uniform noise by negating the attribute values with probability  $\rho_a$ . As a fixed model, we consider a tree generated by the CART algorithm that uses at least one of the features to which noise was applied (this is because if the model does not use these features, the variance of losses for that model will not change). All plots are based on 0-1 loss and are averaged over 10 iterations.
Recall that Bernstein's inequality (4) states
$$
P \left(\left| L (f) - \hat {L} (f) \right| > \varepsilon\right) \leq 2 e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma_ {f} ^ {2} + 2 C \varepsilon / 3}}
$$
where  $C = \frac{b - a}{2}$ . Without loss of generality, let  $l'(f,z) = l(f,z) - \frac{a + b}{2}$  so that  $l' \in [-C,C]$ . Then, we get that  $L'(f) = L(f) - \frac{a + b}{2}$ ,  $\operatorname{Var}_{z \sim \mathcal{D}} l'(f,z) = \operatorname{Var}_{z \sim \mathcal{D}} l(f,z)$ , and  $\hat{L}'(f) = \hat{L}(f) - \frac{a + b}{2}$ . Therefore, we can rewrite Bernstein's inequality as
$$
P \left(\left| L (f) - \hat {L} (f) \right| > \varepsilon\right) = P \left(\left| L ^ {\prime} (f) - \hat {L} ^ {\prime} (f) \right| > \varepsilon\right) \leq 2 e ^ {\frac {- 2 n \varepsilon^ {2}}{4 \sigma_ {f} ^ {2} + 2 (b - a) \varepsilon / 3}}.
$$
Consider  $\sigma_f^2 \leq \frac{(b - a)^2}{12}$ . Then, we can upper-bound the right side of Bernstein's inequality by
$$
2 e ^ {- \frac {2 n \varepsilon^ {2}}{4 \sigma_ {f} ^ {2} + 2 (b - a) \varepsilon / 3}} <   2 e ^ {- \frac {2 n \varepsilon^ {2}}{(b - a) ^ {2} / 3 + 2 (b - a) ^ {2} / 3}} = 2 e ^ {\frac {- 2 n \varepsilon^ {2}}{(b - a) ^ {2}}},
$$
where  $2e^{\frac{-2n\varepsilon^2}{(b - a)^2}}$  is the bound given by Hoeffding's inequality. Therefore, we showed that, if  $\sigma_f^2 \leq \frac{(b - a)^2}{12}$ , then Bernstein's inequality is stronger than Hoeffding's inequality for all  $\varepsilon \in (0, b - a)$ .
We now consider  $\left|L(f) - \frac{a + b}{2}\right| > \frac{b - a}{\sqrt{6}}$ . Recall that  $L^{\prime}(f) = L(f) - \frac{a + b}{2}$ , so we can rewrite this as  $|L^{\prime}(f)| > \frac{b - a}{\sqrt{6}}$ . Since  $-C \leq l^{\prime}(f,z) \leq C$ , we know that
$$
\begin{array}{l} \operatorname {V a r} _ {z \sim \mathcal {D}} \left(l ^ {\prime} (f, z)\right) = E _ {z \sim \mathcal {D}} \left(\left(l ^ {\prime} (f, z)\right) ^ {2}\right) - \left(E _ {z \sim \mathcal {D}} \left(l ^ {\prime} (f, z)\right)\right) ^ {2} \\ \leq C ^ {2} - \left(L ^ {\prime} (f)\right) ^ {2} \\ \leq \frac {(b - a) ^ {2}}{4} - \frac {(b - a) ^ {2}}{6} \\ = \frac {(b - a) ^ {2}}{1 2}. \\ \end{array}
$$
Then, we can follow the same argument as in the previous case to conclude that Bernstein's inequality is stronger than Hoeffding's inequality for all  $\varepsilon \in (0,b - a)$ .
C Proof for Theorem 4
For a discrete hypothesis space, Theorem 4 bounds generalization error with a term that depends on the size of the true Rashomon set and maximum variance of the loss. Recall Theorem 4.
$$
L (f) - \hat {L} (f) \leq \frac {2}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) + \sqrt {\frac {2 \sigma^ {2}}{n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right)},
$$
where  $\sigma^2 = \sup_{f\in R_{set}(\mathcal{F},\gamma)}\mathrm{Var}_{z\sim \mathcal{D}}l(f,z)$ , and  $n$  is number of samples in  $S = \{z_i\}_{i = 1}^n\sim \mathcal{D}$ .
Proof. For each fixed model  $f \in R_{\text{set}}(\mathcal{F}, \gamma)$  in the true Rashomon set, from Bernstein's inequality, using that the maximum value for the 0-1 loss is 1, we have that
$$
P (L (f) - \hat {L} (f) > \varepsilon) \leq e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma_ {f} ^ {2} + 2 \varepsilon / 3}}.
$$
According to the union bound:
$$
\begin{array}{l} P \left(\exists f \in R _ {s e t} (\mathcal {F}, \gamma): L (f) - \hat {L} (f) > \varepsilon\right) \leq \sum_ {f \in R _ {s e t} (\mathcal {F}, \gamma)} P \left(L (f) - \hat {L} (f) > \varepsilon\right) \\ \leq \sum_ {f \in R _ {s e t} (\mathcal {F}, \gamma)} e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma_ {f} ^ {2} + 2 \varepsilon / 3}} \\ \leq \sum_ {f \in R _ {s e t} (\mathcal {F}, \gamma)} e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma^ {2} + 2 \varepsilon / 3}} \\ = | R _ {s e t} (\mathcal {F}, \gamma) | \cdot e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma^ {2} + 2 \varepsilon / 3}}, \\ \end{array}
$$
where we used the fact that  $e^{-\frac{1}{\sigma_f^2}} \leq e^{-\frac{1}{\sup_{f \in R_{set}(\mathcal{F}, \gamma)} \sigma_f^2}} = e^{-\frac{1}{\sigma^2}}$ , since the exponential function is monotonic.
Let  $\delta = |R_{set}(\mathcal{F},\gamma)|e^{\frac{-n\varepsilon^2}{2\sigma^2 + 2\varepsilon / 3}}$ , then we have the following quadratic equation to find  $\varepsilon$ :
$$
\varepsilon^ {2} - \frac {2}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) \varepsilon - \frac {2 \sigma^ {2}}{n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) = 0.
$$
Setting  $a = \frac{2}{n}\log \left(\frac{|R_{set}(\mathcal{F},\gamma)|}{\delta}\right)$ , we find that the roots of the quadratic equation with respect to  $\varepsilon$  are:
$$
\varepsilon = \frac {a}{2 \cdot 3} \pm \frac {1}{2} \sqrt {\left(\frac {a}{3}\right) ^ {2} + 4 a \sigma^ {2}}.
$$
Since  $4a\sigma^2 \geq 0$ , we see that  $\frac{a}{2\cdot 3} - \frac{1}{2}\sqrt{\left(\frac{a}{3}\right)^2 + 4a\sigma^2} < 0$  which is not a valid root as  $\varepsilon > 0$ . Thus,
$$
\begin{array}{l} \varepsilon = \frac {1}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) + \sqrt {\left(\frac {1}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right)\right) ^ {2} + \frac {2 \sigma^ {2}}{n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right)} \\ \leq \frac {2}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) + \sqrt {\frac {2 \sigma^ {2}}{n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right)}, \\ \end{array}
$$
where the latter inequality arises from the inequality  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ . Therefore, we get that with probability at least  $1 - \delta$ :
$$
\forall f \in R _ {s e t} (\mathcal {F}, \gamma): L (f) - \hat {L} (f) \leq \varepsilon = \frac {2}{3 n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right) + \sqrt {\frac {2 \sigma^ {2}}{n} \log \left(\frac {| R _ {s e t} (\mathcal {F} , \gamma) |}{\delta}\right)}.
$$
D Proof for Proposition 5
We recall and provide proof for Proposition 5 below.
Proof. For a fixed  $f \in \mathcal{F}$  for 0-1 loss by Hoeffding's inequality (5):
$$
P \left[ \hat {L} (f) - L (f) > \epsilon \right] \leq e ^ {- 2 n \epsilon^ {2}}.
$$
Therefore, with probability at least  $1 - e^{-2n\epsilon^2}$  with respect to the random draw of data,  $\hat{L}(f) - L(f) \leq \epsilon$ . This is true for the optimal model as well, thus with high probability  $\hat{L}(f^*) - L(f^*) \leq \epsilon$ .
Since  $\hat{f}$  is the empirical risk minimizer, and  $\epsilon + \xi \leq \gamma$  by assumption, we have that  $\hat{L}(\hat{f}) \leq \hat{L}(f^*)$ . We use that for two events  $A$  and  $B$ ,  $P(\neg (A \cup B)) = 1 - P(A \cup B) \geq 1 - P(A) - P(B)$ , where  $A$  is the event that cross-validation gives us an incorrect generalization bound, and  $B$  is the event that  $f^*$  does not generalize. Thus,  $P(A) \leq e^{2n\epsilon^2}$  and  $P(B) \leq \epsilon_{\xi}$ . Thus, with probability at least  $1 - e^{-2n\epsilon^2} - \epsilon_{\xi}$ ,
$$
L (\hat {f}) \leq \hat {L} (\hat {f}) + \xi \leq \hat {L} (f ^ {*}) + \xi \leq L (f ^ {*}) + \epsilon + \xi \leq L (f ^ {*}) + \gamma .
$$
Therefore  $\hat{f}\in R_{set}(\mathcal{F},\gamma)$
E Proof for Proposition 6
For the hypothesis space of decision trees, the number of possible decision trees in the hypothesis space grows exponentially fast with the depth of the tree and the number of features. In Proposition 6 we show that the Rashomon set growth rate is smaller than the growth rate of the hypothesis space, and thus this leads to larger Rashomon ratios for simpler hypothesis spaces. We recall Proposition 6 below.
Proof. The hypothesis space of fully-grown trees of depth  $d$  contains
$$
\left| \mathcal {F} _ {d} \right| = 2 ^ {2 ^ {d}} \prod_ {k = 1} ^ {d} (m - k + 1) ^ {2 ^ {k - 1}}
$$
trees, where 2 is the number of label options each leaf can have,  $2^{d}$  is the number of leaves we have,  $\prod_{k=1}^{d}$  is the product over all depth levels in a tree,  $2^{k-1}$  is the number of nodes we have at that level, and  $(m - (k - 1))$  is the number of options we have to choose from given that the previous features were used in the path from the root. We do not count symmetric trees, meaning that we always assume that split = 0 is on the left and = 1 is on the right.
Now let's compute the size of the Rashomon set. First, since each leaf of every tree in the Rashomon set has correctly classified  $\lceil \theta n\rceil$  points more than misclassified, flipping the label of this leaf will add more than  $\theta$  to the loss and thus will push the tree out of the Rashomon set. Therefore, for every tree, every leaf label is determined by the data.
Second, since the empirical risk minimizer of models using only the bad features  $m_{bad}$  is not in the Rashomon set, every tree that has only features from the set  $m_{bad}$  is not in the Rashomon set. Therefore, trees in the Rashomon set must have at least one "good" feature at some node, where good means that the feature is not in  $m_{bad}$ . The cardinality of the set of good features is  $\bar{m} = m - |m_{bad}|$ , then the cardinality of the Rashomon set is:
$$
\hat {R} _ {s e t} \left(\mathcal {F} _ {d}, \theta\right) = \prod_ {k = 1} ^ {d} (m - k + 1) ^ {2 ^ {k - 1}} - \prod_ {k = 1} ^ {d} (m - \bar {m} - k + 1) ^ {2 ^ {k - 1}},
$$
meaning that among all models, we do not consider those that consist of bad features only (since  $m_{bad} \geq d$ , there exists at least one such tree). Then the Rashomon ratio is:
$$
\begin{array}{l} \hat {R} _ {r a t i o} (\mathcal {F} _ {d}, \theta) = \frac {| \hat {R} _ {s e t} (\mathcal {F} _ {d} , \theta) |}{| \mathcal {F} _ {d} |} = \frac {\prod_ {k = 1} ^ {d} (m - k + 1) ^ {2 ^ {k - 1}} - \prod_ {k = 1} ^ {d} (m - \bar {m} - k + 1) ^ {2 ^ {k - 1}}}{2 ^ {2 ^ {d}} \prod_ {k = 1} ^ {d} (m - k + 1) ^ {2 ^ {k - 1}}} \\ = \frac {1}{2 ^ {2 ^ {d}}} \left(1 - \frac {\prod_ {k = 1} ^ {d} (m - \bar {m} - k + 1) ^ {2 ^ {k - 1}}}{\prod_ {k = 1} ^ {d} (m - k + 1) ^ {2 ^ {k - 1}}}\right) \\ = \frac {1}{2 ^ {2 ^ {d}}} \left(1 - \prod_ {k = 1} ^ {d} \left(1 - \frac {\bar {m}}{m - k + 1}\right) ^ {2 ^ {k - 1}}\right) \\ = \frac {1 - \alpha (d)}{2 ^ {2 ^ {d}}}, \\ \end{array}
$$
where  $\alpha(d) = \prod_{k=1}^{d}\left(1 - \frac{\bar{m}}{m-k+1}\right)^{2^{k-1}}$ . Since  $d > 1$ ,  $\bar{m} > 1$ , and  $\frac{\bar{m}}{m-k+1} > \frac{\bar{m}}{m}$  for  $k > 2$ , we get that
$$
\alpha (d) = \prod_ {k = 1} ^ {d} \left(1 - \frac {\bar {m}}{m - k + 1}\right) ^ {2 ^ {k - 1}} <   \prod_ {k = 1} ^ {d} \left(1 - \frac {\bar {m}}{m}\right) ^ {2 ^ {k - 1}} <   1 - \frac {\bar {m}}{m}.
$$
Note as well that  $\alpha(d) < 1$  for any  $d$ . Recall that  $m < 2^{2^d}$ , then for the ratio of ratios:
$$
\begin{array}{l} \frac {\hat {R} _ {r a t i o} (\mathcal {F} _ {d} , \theta)}{\hat {R} _ {r a t i o} (\mathcal {F} _ {d + 1} , \theta)} = \frac {| \hat {R} _ {s e t} (\mathcal {F} _ {d} , \theta) |}{| \mathcal {F} _ {d} |} \frac {| \mathcal {F} _ {d + 1} |}{| \hat {R} _ {s e t} (\mathcal {F} _ {d + 1} , \theta) |} \\ = \frac {1 - \alpha (d)}{2 ^ {2 ^ {d}}} \frac {2 ^ {2 ^ {d + 1}}}{1 - \alpha (d + 1)} = 2 ^ {2 ^ {d}} \frac {1 - \alpha (d)}{1 - \alpha (d + 1)} \\ > 2 ^ {2 ^ {d}} (1 - \alpha (d)) > 2 ^ {2 ^ {d}} \left(1 - \left(1 - \frac {\bar {m}}{m}\right)\right) \\ = 2 ^ {2 ^ {d}} \frac {\bar {m}}{m} > 2 ^ {2 ^ {d}} \frac {1}{2 ^ {2 ^ {d}}} = 1. \\ \end{array}
$$
Thus we showed that  $\hat{R}_{ratio}(\mathcal{F}_d,\theta) > \hat{R}_{ratio}(\mathcal{F}_{d + 1},\theta)$ , meaning that the Rashomon ratio grows as we consider less deep trees.
F Proof for Theorem 7
Recall Theorem 7:
Theorem 7 (Rashomon ratio increases with noise for ridge regression). Consider dataset  $S = X \times Y$ ,  $X$  is a non-zero matrix, and a hypothesis space of linear models  $\mathcal{F} = \{f = \omega^T x, \omega \in \mathbb{R}^m, \omega^T \omega \leq \hat{L}_{\max} / C\}$ . Let  $\epsilon_i$ , such that  $\epsilon_i \sim \mathcal{N}(\bar{0}, \lambda I)$  ( $\lambda > 0$ ,  $I$  is identity matrix), be i.i.d. noise vectors added to every sample:  $x_i' = x_i + \epsilon_i$ . Consider options  $\lambda_1 > 0$  and  $\lambda_2 > 0$  that control how much noise we add to the dataset. For ridge regression, if  $\lambda_1 < \lambda_2$ , then the Rashomon ratios obey  $\hat{R}_{ratio_{\lambda_1}}(\mathcal{F}, \theta)) < \hat{R}_{ratio_{\lambda_2}}(\mathcal{F}, \theta)$ .
Proof. For simplicity denote  $\mathbb{E}_{\epsilon_1,\dots ,\epsilon_n}\sim \mathcal{N}(\bar{0},\lambda I)$  as  $\mathbb{E}_{\epsilon}$ . To find the optimal solution, under added noise, we would like to minimize expected regularized least squares:
$$
\begin{array}{l} \mathbb {E} _ {\epsilon} \hat {L} (\omega) = \mathbb {E} _ {\epsilon} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left(x _ {i} + \epsilon_ {i}\right) ^ {T} \omega - y _ {i}\right) ^ {2} + C \omega^ {T} \omega \right] \\ = \mathbb {E} _ {\epsilon} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left((x _ {i} ^ {T} \omega - y _ {i}) ^ {2} + 2 \epsilon^ {T} \omega (x _ {i} ^ {T} \omega - y _ {i}) + \omega^ {T} \epsilon_ {i} \epsilon_ {i} ^ {T} \omega\right) \right] + C \omega^ {T} \omega \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left(x _ {i} ^ {T} \omega - y _ {i}\right) ^ {2} + 2 \mathbb {E} _ {\epsilon} \left[ \epsilon_ {i} \right] ^ {T} \omega \left(x _ {i} ^ {T} \omega - y _ {i}\right) + \omega^ {T} \mathbb {E} _ {\epsilon} \left[ \epsilon_ {i} \epsilon_ {i} ^ {T} \right] \omega\right) + C \omega^ {T} \omega \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left(x _ {i} ^ {T} \omega - y _ {i}\right) ^ {2} + \omega^ {T} (\lambda I) \omega\right) + C \omega^ {T} \omega \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} (x _ {i} ^ {T} \omega - y _ {i}) ^ {2} + (C + \lambda) \omega^ {T} \omega , \\ \end{array}
$$
where  $\mathbb{E}_{\epsilon}\left[\epsilon_{i}\epsilon_{i}^{T}\right] = \lambda I, I$  is identity matrix, and  $E_{\epsilon}\left[\epsilon_{i}\right] = \bar{0}$ .
Therefore, adding attribute noise to the training data becomes equivalent to  $\ell_2$ -regularization, and the new regularization parameter is  $C + \lambda$ . According to Theorem 10 in Semenova et al. [39], the Rashomon volume can be computed as:
$$
\mathcal {V} (\hat {R} _ {s e t _ {\lambda}} (\mathcal {F}, \theta)) = \frac {(\pi \theta) ^ {\frac {m}{2}}}{\Gamma (\frac {m}{2} + 1)} \prod_ {i = 1} ^ {m} \frac {1}{\sqrt {\sigma_ {i} ^ {2} + C + \lambda}},
$$
where  $\sigma_{i}$  are singular values of matrix  $X$ , and  $\Gamma (\cdot)$  is the Gamma-function.
On the other hand, for the regularization parameter  $C + \lambda$ , the hypothesis space is defined as  $(C + \lambda)w^{T}w\leq \hat{L}_{\max}$ , meaning that  $w^{T}w\leq \frac{\hat{L}_{\max}}{C + \lambda}$ . The volume of the ball defined by the  $\ell_2$ -norm in  $m$ -dimensional space with radius  $R$ ,  $\| x\| _2 = \left(\sum_{i = 1}^m |x_i|^2\right)^{\frac{1}{2}}\leq R$ , can be computed as:
$$
\mathcal {V} _ {m} ^ {2} (R) = \frac {\pi^ {\frac {m}{2}}}{\Gamma (\frac {m}{2} + 1)} R ^ {m}.
$$
Since for  $\| \omega \|_2^2 = w^T w \leq \frac{\hat{L}_{\max}}{C + \lambda}$ , we have radius  $R_{\lambda} = \sqrt{\frac{\hat{L}_{\max}}{C + \lambda}}$ , we get that the Rashomon ratio obeys:
$$
\begin{array}{l} \hat {R} _ {r a t i o _ {\lambda}} (\mathcal {F}, \theta)) = \frac {\mathcal {V} (\hat {R} _ {s e t _ {\lambda}} (\mathcal {F} , \theta))}{\mathcal {V} _ {m} ^ {2} (R _ {\lambda})} \\ = \frac {(\pi \theta) ^ {\frac {m}{2}}}{\Gamma (\frac {m}{2} + 1)} \left[ \prod_ {i = 1} ^ {m} \frac {1}{\sqrt {\sigma_ {i} ^ {2} + C + \lambda}} \right] \frac {\Gamma (\frac {m}{2} + 1)}{\pi^ {\frac {m}{2}}} \frac {(C + \lambda) ^ {\frac {m}{2}}}{(\hat {L} _ {\max}) ^ {\frac {m}{2}}} \\ = \left(\frac {\theta}{\hat {L} _ {\operatorname* {m a x}}}\right) ^ {\frac {m}{2}} \prod_ {i = 1} ^ {m} \sqrt {\frac {C + \lambda}{\sigma_ {i} ^ {2} + C + \lambda}}. \\ \end{array}
$$
Since  $0 < \lambda_1 < \lambda_2, C > 0$ , without loss of generality, let  $\lambda_C = \lambda_1 + C$ , and  $\lambda_C + \delta = \lambda_2 + C$  where  $\delta = \lambda_2 - \lambda_1 > 0$ . Consider function  $\frac{x}{a + x}$ , where  $a > 0$ . This function is monotonically increasing for all  $x > 0$ , since  $\frac{\partial}{\partial x}\left(\frac{x}{a + x}\right) = \frac{a}{(a + x)^2} > 0$ . Therefore, for all non-zero  $\sigma_i^2$ :
$$
\frac {\lambda_ {C}}{\sigma_ {i} ^ {2} + \lambda_ {C}} <   \frac {\lambda_ {C} + \delta}{\sigma_ {i} ^ {2} + \lambda_ {C} + \delta}.
$$
Since  $X$  is a non-zero matrix, there is at least one non-zero singular value  $\sigma_i^2$ . Given monotonicity of the square root function, we have that for the Rashomon ratios for noise levels  $\lambda_1$  and  $\lambda_2$ :
$$
\begin{array}{l} \hat {R} _ {r a t i o _ {\lambda_ {1}}} (\mathcal {F}, \theta)) = \hat {R} _ {r a t i o _ {\lambda_ {C}}} (\mathcal {F}, \theta)) = \left(\frac {\theta}{\hat {L} _ {\max}}\right) ^ {\frac {m}{2}} \prod_ {i = 1} ^ {m} \sqrt {\frac {\lambda_ {C}}{\sigma_ {i} ^ {2} + \lambda_ {C}}} \\ <   \left(\frac {\theta}{\hat {L} _ {\operatorname* {m a x}}}\right) ^ {\frac {m}{2}} \prod_ {i = 1} ^ {m} \sqrt {\frac {\lambda_ {C} + \delta}{\sigma_ {i} ^ {2} + \lambda_ {C} + \delta}} \\ = \hat {R} _ {r a t i o _ {\lambda_ {C} + \delta}} (\mathcal {F}, \theta)) = \hat {R} _ {r a t i o _ {\lambda_ {2}}} (\mathcal {F}, \theta)). \\ \end{array}
$$
Therefore we proved that with the additive attribute noise, the Rashomon ratio increases.
Compared to the Rashomon ratio, the relationship between the regularization parameter and the Rashomon volume is inverted: the stronger the regularization, the smaller the Rashomon volume. This means that adding more noise leads to stronger regularization and smaller Rashomon volume. In some ways, this is consistent with what we saw in Figure 1(b), where CART preferred shorter trees in the presence of noise.