SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next we show that the variance of losses (recall notation  $\sigma^2(f, \mathcal{D}) = \mathrm{Var}_{z \sim \mathcal{D}} l(f, z)$ ) increases for the least squares loss function under additive attribute noise, as in Step 1 of the path in Section 4:
Theorem 19 (Variance of least squares loss increases with noise). Consider dataset  $S = X \times Y$  where  $X \in \mathbb{R}^{n \times m}$ ,  $Y \in \mathbb{R}^n$ ,  $z_i = (x_i, y_i)$ . Let  $\epsilon_i = \{\epsilon_{ij}\}_{j=1}^m$ , such that  $\epsilon_{ij} \sim \mathcal{N}(0, \sigma_{\mathcal{N}}^2)$ , be i.i.d. noise vectors added to every sample:  $x_i' = x_i + \epsilon_i$ . Consider  $\sigma_{\mathcal{N}_1}^2 > 0$ ,  $\sigma_{\mathcal{N}_2}^2 > 0$  that control how much noise is added to the dataset. For the least squares loss  $l(z_i) = r_i^2 = (w^T x_i - y_i)^2$  and a fixed model  $f(x) = \omega^T x$ , where  $\omega \in \mathbb{R}^m$ ,  $\omega \neq \bar{0}$ , the variance of losses increases with more noise: if  $\sigma_{\mathcal{N}_1}^2 < \sigma_{\mathcal{N}_2}^2$ , then:  $\sigma^2(f, S_{\sigma_{\mathcal{N}_1}}) < \sigma^2(f, S_{\sigma_{\mathcal{N}_2}})$ .
Proof. For simplicity, denote  $\mathbb{E}_{\epsilon_{11},\ldots ,\epsilon_{1m},\ldots ,\epsilon_{n1},\ldots ,\epsilon_{nm}}$  as  $\mathbb{E}_{\bar{\epsilon}}$ , and  $\mathbb{E}_{x_i,y_i}$  as  $\mathbb{E}_z$ . The variance of losses for the least squares loss under the additive normal noise is:  $\sigma^2 (f,S_{\sigma_N}) = Var_{z,\bar{\epsilon}}\left[(x_i + \epsilon_i)^T\omega -y_i\right]^2$ . Also, for simplicity, we will omit index  $i$  over samples (but keep index  $j$  over the dimensions). Recall that  $r = x^{T}\omega -y$ . From the definition of the variance we have that:
$$
\begin{array}{l} V a r _ {z, \bar {\epsilon}} \left[ \left(\left(x + \epsilon\right) ^ {T} \omega - y\right) ^ {2} \right] = V a r _ {z, \bar {\epsilon}} \left[ \left(\left(x ^ {T} \omega - y\right) + \epsilon^ {T} \omega\right) ^ {2} \right] = V a r _ {z, \bar {\epsilon}} \left[ \left(r + \epsilon^ {T} \omega\right) ^ {2} \right] \\ = \mathbb {E} _ {z, \bar {\epsilon}} \left[ (r + \epsilon^ {T} \omega) ^ {4} \right] - \left(\mathbb {E} _ {z, \bar {\epsilon}} \left[ (r + \epsilon^ {T} \omega) ^ {2} \right]\right) ^ {2}. \\ \end{array}
$$
Since  $\epsilon_{j} \sim \mathcal{N}(0, \sigma_{\mathcal{N}}^{2})$ , we have that  $\mathbb{E}_{\epsilon_j}[\epsilon_j] = 0$ ,  $\mathbb{E}_{\epsilon_j}[(\epsilon_j)^2] = \sigma_{\mathcal{N}}^2$ ,  $\mathbb{E}_{\epsilon_j}[(\epsilon_j)^3] = 0$  (this is a property of Gaussian random variables), and  $\mathbb{E}_{\epsilon_j}[(\epsilon_j)^4] = 3\sigma_{\mathcal{N}}^4$ . Also recall that the multinomial theorem states:
$$
\left(\sum_ {j = 1} ^ {m} a _ {j}\right) ^ {t} = \sum_ {k _ {1} + k _ {2} + \dots + k _ {m} = t} \frac {t !}{k _ {1} ! \cdot k _ {2} ! \cdot \dots \cdot k _ {m} !} \cdot a _ {1} ^ {k _ {1}} \cdot a _ {2} ^ {k _ {2}} \cdot \dots \cdot a _ {m} ^ {k _ {m}}.
$$
The multinomial theorem helps us to compute coefficients of the first four moments for  $\epsilon^T\omega$ . More specifically, for the first and second moment:
$$
\mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon^ {T} \omega \right] = 0,
$$
$$
\begin{array}{l} \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {2} \right] = \mathbb {E} _ {\bar {\epsilon}} \left[ \left(\sum_ {j = 1} ^ {m} \epsilon_ {j} \omega_ {j}\right) ^ {2} \right] = \mathbb {E} _ {\bar {\epsilon}} \left[ \sum_ {j = 1} ^ {m} (\epsilon_ {j} \omega_ {j}) ^ {2} + \sum_ {j = 1.. m, k = 1.. m, j \neq k} \epsilon_ {j} \omega_ {j} \epsilon_ {k} \omega_ {k} \right] \\ = \sum_ {j = 1} ^ {m} \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon_ {j} ^ {2} \right] \omega_ {j} ^ {2} = \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega . \\ \end{array}
$$
For the third moment, notice that from the multinomial theorem,  $k_{1} + \dots + k_{m} = 3$ . Then there are three possible combinations of the values of  $k_{j}$ : some  $k_{a} = 3$  and the rest are 0, some  $k_{a} = 2$ ,  $k_{b} = 1$ , and the rest are 0, and finally some  $k_{a} = 1$ ,  $k_{b} = 1$ ,  $k_{c} = 1$  and the rest are 0. All of these cases will lead to the presence of either  $\epsilon_{j}^{3}$  or  $\epsilon_{j}$  in the product. Since  $\mathbb{E}_{\epsilon_{j}}[\epsilon_{j}] = 0$ , and  $\mathbb{E}_{\epsilon_{j}}[\epsilon_{j}^{3}] = 0$ , we have that
$$
\mathbb {E} _ {\bar {\epsilon}} \left[ (\boldsymbol {\epsilon} ^ {T} \omega) ^ {3} \right] = 0.
$$
Similarly, for  $\mathbb{E}_{\bar{\epsilon}}\left[\left(\sum_{j = 1}^{m}\epsilon_{j}\omega_{j}\right)^{4}\right]$  we get non-zero terms for some of the combinations and the others are 0. In particular, non-zero terms arise when some  $k_{a} = 4$  and the rest are 0, and some  $k_{a} = 2, k_{b} = 2$ , and the rest are 0s. This gives us:
$$
\begin{array}{l} \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {4} \right] = \mathbb {E} _ {\bar {\epsilon}} \left[ \left(\sum_ {j = 1} ^ {m} \epsilon_ {j} \omega_ {j}\right) ^ {4} \right] \\ = \sum_ {j = 1} ^ {m} \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon_ {j} ^ {4} \right] \omega_ {j} ^ {4} + 6 \sum_ {j = 1.. m, k = 1.. m, j \neq k} \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon_ {j} ^ {2} \right] \omega_ {j} ^ {2} \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon_ {k} ^ {2} \right] \omega_ {k} ^ {2} \\ = 3 \sigma_ {\mathcal {N}} ^ {4} \sum_ {j = 1} ^ {m} \omega_ {j} ^ {4} + 6 \sigma_ {\mathcal {N}} ^ {4} \sum_ {j = 1.. m, k = 1.. m, j \neq k} \omega_ {j} ^ {2} \omega_ {k} ^ {2} \\ = 3 \sigma_ {\mathcal {N}} ^ {4} (\omega^ {T} \omega) ^ {2}. \\ \end{array}
$$
Let's focus on the first term of the variance equation (6):
$$
\begin{array}{l} \mathbb {E} _ {z, \bar {\epsilon}} \left[ (r + \epsilon^ {T} \omega) ^ {4} \right] = \mathbb {E} _ {z, \bar {\epsilon}} \left[ r ^ {4} + 4 r ^ {3} \epsilon^ {T} \omega + 6 r ^ {2} (\epsilon^ {T} \omega) ^ {2} + 4 r (\epsilon^ {T} \omega) ^ {3} + (\epsilon^ {T} \omega) ^ {4} \right] \\ = \mathbb {E} _ {z} \left[ r ^ {4} \right] + 4 \mathbb {E} _ {z} \left[ r ^ {3} \right] \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon^ {T} \omega \right] + 6 \mathbb {E} _ {z} \left[ r ^ {2} \right] \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {2} \right] \\ + 4 \mathbb {E} _ {z} [ r ] \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {3} \right] + \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {4} \right] \\ = \mathbb {E} _ {z} \left[ r ^ {4} \right] + 6 \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega \mathbb {E} _ {z} \left[ r ^ {2} \right] + 3 \sigma_ {\mathcal {N}} ^ {4} (\omega^ {T} \omega) ^ {2}. \\ \end{array}
$$
Now, we focus on the second term of the variance equation (6):
$$
\begin{array}{l} \left(\mathbb {E} _ {z, \bar {\epsilon}} \left[ (r + \epsilon^ {T} \omega) ^ {2} \right]\right) ^ {2} = \left(\mathbb {E} _ {z, \bar {\epsilon}} \left[ r ^ {2} + 2 r \epsilon^ {T} \omega + (\epsilon^ {T} \omega) ^ {2} \right]\right) ^ {2} \\ = \left(\mathbb {E} _ {z} \left[ r ^ {2} \right] + \mathbb {E} _ {z} \left[ 2 r \right] \mathbb {E} _ {\bar {\epsilon}} \left[ \epsilon^ {T} \omega \right] + \mathbb {E} _ {\bar {\epsilon}} \left[ (\epsilon^ {T} \omega) ^ {2} \right]\right) ^ {2} \\ = \left(\mathbb {E} _ {z} \left[ r ^ {2} \right] + \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega\right) ^ {2} \\ = \left(\mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2} + 2 \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega \mathbb {E} _ {z} \left[ r ^ {2} \right] + \sigma_ {\mathcal {N}} ^ {4} (\omega^ {T} \omega) ^ {2}. \\ \end{array}
$$
Therefore, for the variance we get that:
$$
V a r _ {z, \bar {\epsilon}} \left[ \left(r + \epsilon^ {T} \omega\right) ^ {2} \right] = \mathbb {E} _ {z} \left[ r ^ {4} \right] + 6 \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega \mathbb {E} _ {z} \left[ r ^ {2} \right] + 3 \sigma_ {\mathcal {N}} ^ {4} (\omega^ {T} \omega) ^ {2}
$$
$$
\begin{array}{l} - \left(\mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2} - 2 \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega \mathbb {E} _ {z} \left[ r ^ {2} \right] - \sigma_ {\mathcal {N}} ^ {4} \left(\omega^ {T} \omega\right) ^ {2} \\ = 2 \sigma_ {\mathcal {N}} ^ {4} \left(\omega^ {T} \omega\right) ^ {2} + 4 \sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega \mathbb {E} _ {z} \left[ r ^ {2} \right] + 2 \left(\mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2} + \mathbb {E} _ {z} \left[ r ^ {4} \right] - 3 \left(\mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2} \\ = 2 \left(\sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega + \mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2} + \mathbb {E} _ {z} \left[ r ^ {4} \right] - 3 \left(\mathbb {E} _ {z} \left[ r ^ {2} \right]\right) ^ {2}. \\ \end{array}
$$
Next, we will take the derivative of the variance with respect to  $\sigma_{\mathcal{N}}^2$ :
$$
\begin{array}{l} \frac {\partial}{\partial \sigma_ {\mathcal {N}} ^ {2}} \left(V a r _ {z, \bar {\epsilon}} \left[ (r + \epsilon^ {T} \omega) ^ {2} \right]\right) = \frac {\partial}{\partial \sigma_ {\mathcal {N}} ^ {2}} \left(2 \left(\sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega + \mathbb {E} _ {z} [ r ^ {2} ]\right) ^ {2} + \mathbb {E} _ {z} [ r ^ {4} ] - 3 (\mathbb {E} _ {z} [ r ^ {2} ]) ^ {2}\right) \\ = 4 \left(\sigma_ {\mathcal {N}} ^ {2} \omega^ {T} \omega + \mathbb {E} _ {z} \left[ r ^ {2} \right]\right) \omega^ {T} \omega > 0, \\ \end{array}
$$
since  $\sigma_{\mathcal{N}}^2 > 0$  by assumption,  $\omega^T\omega > 0$  since  $\omega \neq \bar{0}$ , and the risk of the least squares loss  $\mathbb{E}_z\left[r^2\right] \geq 0$ . Therefore, the variance of losses for a fixed model  $f = \omega^T x$  monotonically increases for  $\sigma_{\mathcal{N}}^2 > 0$ . Thus, for  $\sigma_{\mathcal{N}_1}^2 < \sigma_{\mathcal{N}_2}^2$  we have that:
$$
\sigma^ {2} (f, S _ {\sigma_ {\mathcal {N} _ {1}}}) <   \sigma^ {2} (f, S _ {\sigma_ {\mathcal {N} _ {2}}}).
$$
As before, Corollary 3 is easily extendable to the results of Theorem 19, meaning that the maximum variance of losses,  $\sigma^2 = \sup_{f\in \mathbb{R}_{set}(\mathcal{F},\gamma)}Var_{z\sim \mathcal{D}}l(f,z)$ , will also increase for the least squares loss under increasing additive attribute noise.
Next, we show that when the maximum variance  $\sigma^2$  increases, the generalization bound becomes worse for the least squares loss and the continuous hypothesis space. Cucker and Smale [11] proved the generalization bound based on Bernstein's inequality for the least squares loss (Theorem B). We state and provide proof of the theorem for the true Rashomon set. To derive the generalization bound, we use the covering number over the true Rashomon set. Recall that for the functional space  $\mathcal{F}$  and any  $\epsilon > 0$ , the  $\ell_{\infty}$  covering number  $N(\mathcal{F}, \epsilon)$  of  $\mathcal{F}$  is the minimum number of balls of radius  $\epsilon$ , such that they can cover  $\mathcal{F}$ , meaning that there exist  $h_1, \dots, h_{N(\mathcal{F}, \epsilon)} \in \mathcal{F}$ , such that for every  $f \in \mathcal{F}$  there is  $k \leq N(\mathcal{F}, \epsilon)$  such that  $\| f - h_k \|_{\infty} = \max_{x \in \mathcal{X}} |f(x) - h_k(x)| \leq \epsilon$ . Now we focus on the theorem.
Theorem 20 (Variance-based "generalization bound" for least squares loss). Consider data distribution  $\mathcal{D}$  over  $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$ , dataset  $S = \{z_i\}_{i=1}^n \sim \mathcal{D}$ , hypothesis space  $\mathcal{F}$ , and the least squares loss  $l(f, z) = (f(x) - y)^2$ . Let the loss be bounded by  $C^2 > 0$  such that  $l(f, z) \leq C^2$  for every  $z \in \mathcal{Z}$ . For any  $\epsilon > 0$ :
$$
P \left(\sup _ {f \in R _ {s e t} (\mathcal {F}, \gamma)} L (f) - \hat {L} (f) > \varepsilon\right) \leq N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\epsilon}{8 C}\right) e ^ {\frac {- n (\varepsilon / 2) ^ {2}}{2 \sigma^ {2} + C ^ {2} \varepsilon / 3}},
$$
where  $\sigma^2 = \sup_{R_{set}(\mathcal{F},\gamma)}Var_{z\in \mathcal{D}}l(f,z)$
Proof. For each fixed model  $f \in R_{\text{set}}(\mathcal{F}, \gamma)$  in the true Rashomon set, from Bernstein's inequality and given that loss is bounded by  $C^2$ , we have that
$$
P (L (f) - \hat {L} (f) > \varepsilon) \leq e ^ {\frac {- n \varepsilon^ {2}}{2 \sigma_ {f} ^ {2} + 2 C ^ {2} \varepsilon / 3}}.
$$
Let  $B_{1},\ldots B_{N\big(R_{set}(\mathcal{F},\gamma),\frac{\epsilon}{8C}\big)}$  be an  $\ell_{\infty}$  cover of radius  $\frac{\epsilon}{8C}$  of the true Rashomon set, meaning that  $R_{set}(\mathcal{F},\gamma)\subseteq \bigcup_{k = 1}^{N\big(R_{set}(\mathcal{F},\gamma),\frac{\epsilon}{8C}\big)}B_k$ , where  $N\left(R_{set}(\mathcal{F},\gamma),\frac{\epsilon}{8C}\right)$  is the covering number. Since the loss is bounded,  $l(f,z) = (f(x) - y)^2\leq C^2$ , then  $|f(x) - y|\leq C$ . For every  $f\in B_k$ , we have that  $\| f - h_k\|_{\infty}\leq \frac{\epsilon}{8C}$ , where  $h_k$  is the center of the ball  $B_{k}$ . Therefore:
$$
\begin{array}{l} (L (f) - \hat {L} (f)) - (L (h _ {k}) - \hat {L} (h _ {k})) = (L (f) - L (h _ {k})) + (\hat {L} (h _ {k}) - \hat {L} (f)) \\ = \mathbb {E} _ {z \sim \mathcal {D}} [ l (f, z) - l \left(h _ {k}, z\right) ] + \hat {\mathbb {E}} _ {z _ {i} \sim S} [ l (f, z _ {i}) - l \left(h _ {k}, z _ {i}\right) ] \\ = \mathbb {E} _ {z \sim \mathcal {D}} \left[ (f (x) - y) ^ {2} - \left(h _ {k} (x) - y\right) ^ {2} \right] + \hat {\mathbb {E}} _ {z _ {i} \sim S} \left[ \left(f \left(x _ {i}\right) - y _ {i}\right) ^ {2} - \left(h _ {k} \left(x _ {i}\right) - y _ {i}\right) ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \mathbb {E} _ {z \sim \mathcal {D}} \left[ \left(f (x) - h _ {k} (x)\right) \left(\left(f (x) - y\right) + \left(h _ {k} (x) - y\right)\right) \right] \\ + \hat {\mathbb {E}} _ {z _ {i} \sim S} \left[ \left(f \left(x _ {i}\right) - h _ {k} \left(x _ {i}\right)\right) \left(\left(f \left(x _ {i}\right) - y _ {i}\right) + \left(h _ {k} \left(x _ {i}\right) - y _ {i}\right)\right) \right] \\ \leq \mathbb {E} _ {z \sim \mathcal {D}} \left[ \| f - h _ {k} \| _ {\infty} (C + C) \right] + \hat {\mathbb {E}} _ {z _ {i} \sim S} \left[ \| f - h _ {k} \| _ {\infty} (C + C) \right] \\ = 4 C \| f - h _ {k} \| _ {\infty} \leq 4 C \frac {\epsilon}{8 C} = \frac {\epsilon}{2}. \\ \end{array}
$$
Therefore, if  $L(f) - \hat{L}(f) > \epsilon$ , we have  $L(h_k) - \hat{L}(h_k) \geq (L(f) - \hat{L}(f)) - \frac{\epsilon}{2} > \epsilon - \frac{\epsilon}{2} = \frac{\epsilon}{2}$ . This holds for every  $f \in B_k$ , and thus for  $\arg \sup_{f \in B_k}$  as well:
$$
P \left(\sup  _ {f \in B _ {k}} L (f) - \hat {L} (f) > \varepsilon\right) \leq P \left(L \left(h _ {k}\right) - \hat {L} \left(h _ {k}\right) > \frac {\varepsilon}{2}\right). \tag {7}
$$
Since the exponential function is monotonic,  $e^{-\frac{1}{\left(\sigma_{h_k}^2\right)}} \leq e^{-\frac{1}{\sigma^2}}$ . Based on the definition of the covering number, according to the union bound and (7) we have that:
$$
\begin{array}{l} P \left(\sup  _ {f \in R _ {\text {s e t}} (\mathcal {F}, \gamma)} L (f) - \hat {L} (f) > \varepsilon\right) = P \left(\exists f \in R _ {\text {s e t}} (\mathcal {F}, \gamma): L (f) - \hat {L} (f) > \varepsilon\right) \\ \leq P \left(\bigcup_ {k = 1} ^ {N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\varepsilon}{8 C}\right)} \exists f \in B _ {k}: L (f) - \hat {L} (f) > \varepsilon\right) \\ \leq \sum_ {k = 1} ^ {N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\epsilon}{8 C}\right)} P \left(\exists f \in B _ {k}: L (f) - \hat {L} (f) > \varepsilon\right) \\ \leq \sum_ {k = 1} ^ {N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\epsilon}{8 C}\right)} P \left(L (h _ {k}) - \hat {L} (h _ {k}) > \frac {\varepsilon}{2}\right) \\ \leq \sum_ {k = 1} ^ {N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\epsilon}{8 C}\right)} e ^ {\frac {- n (\varepsilon / 2) ^ {2}}{2 \sigma_ {h _ {k}} ^ {2} + C ^ {2} \varepsilon / 3}} \\ \leq \sum_ {k = 1} ^ {N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\varepsilon}{8 C}\right)} e ^ {\frac {- n (\varepsilon / 2) ^ {2}}{2 \sigma^ {2} + C ^ {2} \varepsilon / 3}} \\ = N \left(R _ {s e t} (\mathcal {F}, \gamma), \frac {\epsilon}{8 C}\right) e ^ {\frac {- n (\varepsilon / 2) ^ {2}}{2 \sigma^ {2} + C ^ {2} \varepsilon / 3}}. \\ \end{array}
$$
Therefore we obtained the desired bound.
Since  $e^{-1/x}$  monotonically increases for  $x > 0$ , in Theorem 20, as the maximum variance of losses increases, the bound on the right side increases as well, and thus the generalization bound becomes worse.
G Pattern Diversity and Other Metrics of the Rashomon set
In this appendix, we discuss similarities and differences between pattern diversity and pattern Rashomon ratio as well as expected pairwise disagreement (as in [5]).
Pattern Rashomon ratio. Pattern Rashomon ratio measures how expressive the Rashomon set is compared to the whole hypothesis space. Interestingly, for the hypothesis space of linear models, for different datasets with the same number of samples and attributes, as long as no  $m - 1$  points are collinear, the denominator of the pattern Rashomon ratio is the same and equal to  $2\sum_{i = 0}^{m}\binom{n - 1}{i}$  [10]. If we focus only on the numerator of the pattern Rashomon ratio, it is the number of distinct predictions, whereas the pattern diversity is the average Hamming distance between distinct predictions.
Intuitively, the more distinct prediction we have, the more different they are from each other, and the higher pattern diversity we should expect. However, it is not always the case, and there exists datasets such that we can have a large number of patterns with very small Hamming distance and a small number of patterns with larger Hamming distance.
Similarly to pattern diversity, we can upper-bound the number of patterns in the pattern Rashomon set by a bound that depends on the empirical risk of the empirical risk minimizer and the Rashomon parameter  $\theta$ . We discuss this bound in Lemma 21.
Lemma 21. Given the dataset  $S$  of size  $n$ , the pattern Rashomon set  $\pi(\mathcal{F}, \theta)$ , the empirical risk of the empirical risk minimizer  $\hat{L}(\hat{f})$ , and the Rashomon parameter  $\theta$ , the cardinality of the pattern Rashomon set obeys:
$$
| \pi (\mathcal {F}, \theta) | \leq \sum_ {k = 1} ^ {\lceil n \hat {L} (\hat {f}) + n \theta \rceil} \binom {n} {k}.
$$
Proof. For every model from the Rashomon set  $f$ ,  $\hat{L}(f) \leq \hat{L}(\hat{f}) + \theta$ , which means that, in the worst case, the Hamming distance between pattern  $p^{\hat{f}}$  and vector of true labels  $Y = [y_i]_{i=1}^n$  is  $\lceil n\hat{L}(\hat{f}) + n\theta \rceil$ . Thus, patterns in the pattern Rashomon set can make one mistake, two mistakes, and so on up to  $\lceil n\hat{L}(\hat{f}) + n\theta \rceil$  mistakes, which means there are at most  $\sum_{k=1}^{\lceil n\hat{L}(\hat{f}) + n\theta \rceil} \binom{n}{k}$  patterns in the pattern Rashomon set.
Expected pairwise disagreement (as in [5]). Following [6] and [30], empirical expected pairwise disagreement  $I(\hat{R}_{set}(\mathcal{F},\theta))$  over the Rashomon set can be defined as  $I(\hat{R}_{set}(\mathcal{F},\theta)) = \mathbb{E}_{f_1,f_2\sim \hat{R}_{set}(\mathcal{F},\theta)}\hat{\mathbb{E}}_{z\sim S}\mathbb{1}_{[f_1(x)\neq f_2(x)]}$ . Expected pairwise disagreement measures the average disagreement between every two hypotheses from the Rashomon set, while pattern diversity measures the average disagreement between two patterns from the Rashomon set. The expected pairwise disagreement is equivalent to pattern diversity when every pattern is achievable with the same probability by models from the Rashomon set. However, these metrics can be very different and we can have a small expected pairwise disagreement and larger pattern diversity as we show next.
Proposition 22 (Same pattern diversity but different expected pairwise disagreement). Consider finite Rashomon set  $\hat{R}_{set}(\mathcal{F},\theta)$  of size  $d\geq 2$ . Let  $\pi (\mathcal{F},\theta)$  be the pattern set of size  $\Pi$ ,  $2\leq \Pi \leq d$ . Assume that every pattern except  $p_1$  is achievable by only one hypothesis in the Rashomon set, and thus  $p_1$  is achievable by  $d - \Pi +1$  hypotheses. Let  $d^{*}$  be the current value of  $d$ , then as  $d\to \infty$  (for example, by replicating hypotheses that realize  $p_1$  an infinite number of times), expected pairwise disagreement converges to zero,  $I(\hat{R}_{set}(\mathcal{F}_d,\theta))\rightarrow 0$ , and pattern diversity does not change,  $\operatorname {div}(\hat{R}_{set}(\mathcal{F}_d,\theta)) = \operatorname {div}(\hat{R}_{set}(\mathcal{F}_{d^*},\theta))$ .
Proof. The proof proceeds in two steps.
Pattern diversity. As  $d$  increases, the pattern set does not change, therefore for any  $d \geq d^*$ ,  $\text{div}(\hat{R}_{\text{set}}(\mathcal{F}_d, \theta)) = \text{div}(\hat{R}_{\text{set}}(\mathcal{F}_{d^*}, \theta))$ .
Expected pairwise disagreement. Given a pattern  $p \in \pi(\mathcal{F}, \theta)$ , let  $P_{f \sim \hat{R}_{\text{set}}(\mathcal{F}, \theta)}[p = p^f]$  be a probability with which this pattern is achieved by models from the Rashomon set. Since support for all patterns except  $p_1$  is 1, then  $P_k = P_{f \sim \hat{R}_{\text{set}}(\mathcal{F}, \theta)}[p_k = p^f] = \frac{1}{d}$  for  $k = 2.. \Pi$ . And for  $p_1$  we have  $P_1 = P_{f \sim \hat{R}_{\text{set}}(\mathcal{F}, \theta)}[p_1 = p^f] = \frac{d - \Pi + 1}{d}$ . Then expected pairwise disagreement:
$$
\begin{array}{l} I \left(\hat {R} _ {s e t} \left(\mathcal {F} _ {d}, \theta\right)\right) = \mathbb {E} _ {f _ {1}, f _ {2} \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} \hat {\mathbb {E}} _ {x \sim S} \mathbb {1} _ {\left[ f _ {1} (x) \neq f _ {2} (x) \right]} \\ = \sum_ {k = 1} ^ {\Pi} \left[ P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} [ p _ {k} = p ^ {f} ] \sum_ {j = 1} ^ {\Pi} P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} [ p _ {j} = p ^ {f} ] \frac {1}{n} H (p _ {k}, p _ {j}) \right] \\ = \left(P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} [ p _ {1} = p ^ {f} ]\right) ^ {2} \frac {1}{n} H (p _ {1}, p _ {1}) \\ \end{array}
$$
$$
\begin{array}{l} + 2 P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} [ p _ {1} = p ^ {f} ] \sum_ {j = 2} ^ {\Pi} P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} [ p _ {j} = p ^ {f} ] \frac {1}{n} H (p _ {1}, p _ {j}) \\ + \sum_ {k = 2} ^ {\Pi} \left[ P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} \left[ p _ {k} = p ^ {f} \right] \sum_ {j = 2} ^ {\Pi} P _ {f \sim \hat {R} _ {s e t} (\mathcal {F}, \theta)} \left[ p _ {j} = p ^ {f} \right] \frac {1}{n} H (p _ {k}, p _ {j}) \right] \\ = 0 + 2 \frac {d - \Pi + 1}{d ^ {2}} \sum_ {j = 2} ^ {\Pi} \frac {1}{n} H \left(p _ {1}, p _ {j}\right) + \frac {1}{d ^ {2}} \sum_ {k = 2} ^ {\Pi} \sum_ {j = 2} ^ {\Pi} \frac {1}{n} H \left(p _ {k}, p _ {j}\right) \\ = \frac {1}{d} \left(2 \left(1 - \frac {\Pi - 1}{d}\right) \sum_ {j = 2} ^ {\Pi} \frac {1}{n} H \left(p _ {1}, p _ {j}\right) + \frac {1}{d} \sum_ {k = 2} ^ {\Pi} \sum_ {j = 2} ^ {\Pi} \frac {1}{n} H \left(p _ {k}, p _ {j}\right)\right). \\ \end{array}
$$
Therefore, as  $d\to \infty$ $I(\hat{R}_{set}(\mathcal{F}_d,\theta))\to 0$
As we see from Proposition 22, we can change expected pairwise disagreement, for example, by adding multiple copies of the same functions  $f$  to the hypothesis space. Expected pairwise disagreement measures predictive multiplicity [6], but it has the issue we showed above that it can depend on the weighting of hypotheses in the hypothesis space. In the case we described in Proposition 22, the multiplicity is small because one subset of hypotheses (which produce the same pattern) is weighted very heavily. Thus, expected pairwise disagreement can be influenced by overparameterization or poor choice of parameter space. We further illustrate the effect of reparameterization on pairwise disagreement on a simple one-dimensional example in Figure 6. Pattern diversity does not depend on the parameter space and is computed in the pattern space. It is not impacted by any probability distribution or weighting on the hypotheses. Moreover, we can compute the pattern diversity by enumerating all possible patterns of the given finite dataset as described in Appendix K.3. We cannot do the same for the pairwise disagreement metric without additional assumptions on the patterns' support.
H Proof for Theorem 9