SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[49] Jamelle Watson-Daniels, David C Parkes, and Berk Ustun. Predictive multiplicity in probabilistic classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10306-10314, 2023.
[50] Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of large-batch stochastic gradient descent with structured covariance noise. arXiv preprint arXiv:1902.08234, 2019.
[51] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691-2699, 2015.
[52] Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. In Advances in Neural Information Processing Systems, volume 35, pages 14071-14084, 2022.
[53] Tom Yan and Chicheng Zhang. Margin-distancing for safe model explanation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 5104-5134. PMLR, 28-30 Mar 2022.
[54] Chudi Zhong, Zhi Chen, Jiachang Liu, Margo Seltzer, and Cynthia Rudin. Exploring and interacting with the set of good sparse generalized additive models. Advances in Neural Information Processing Systems, 2023.
Supplemental materials A Path to Simpler Models Starts With Noise
A Proof for Theorem 2
We state and prove Theorem 2 below.
Proof. Recall that the true risk for 0-1 loss  $L_{\mathcal{D}}(f) = \mathbb{E}_{z = (x,y)\sim \mathcal{D}}[l(f,z)] = \mathbb{E}_{(x,y)\sim \mathcal{D}}[\mathbb{1}_{[f(x)\neq y]}]$ . Without loss of generality, let  $y\in \{0,1\}$ . Drawing from  $z_{\rho}\sim \mathcal{D}_{\rho}$  is equivalent to drawing  $z\sim \mathcal{D}$  and changing label  $y$  to  $1 - y$  with probability  $\rho$ . More explicitly, let  $\eta \sim \mathrm{Bernoulli}(\rho)$ , then the flipped label is  $XOR(y,\eta) = \mathbb{1}_{[y\neq \eta ]}$ . For any given  $f\in \mathcal{F}$  we have that:
$$
\begin{array}{l} L _ {\mathcal {D} _ {\rho}} (f) = \mathbb {E} _ {\eta \sim B e r (\rho)} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq X O R (y, \eta) ]} \right] \\ = \mathbb {E} _ {\eta \sim B e r (\rho)} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} (1 - \eta) \right] + \mathbb {E} _ {\eta \sim B e r (\rho)} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) = y ]} \eta \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \mathbb {E} _ {\eta \sim B e r (\rho)} \left[ \mathbb {1} _ {[ f (x) \neq y ]} (1 - \eta) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \mathbb {E} _ {\eta \sim B e r (\rho)} \left[ \mathbb {1} _ {[ f (x) = y ]} \eta \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho)} \left[ (1 - \eta) \right] \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) = y ]} \mathbb {E} _ {\eta \sim B e r (\rho)} \left[ \eta \right] \right] \\ = (1 - \rho) \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \right] + \rho \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) = y ]} \right] \\ = \left(1 - \rho\right) \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \right] + \rho \left(1 - \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \right]\right) \\ = (1 - \rho) L _ {\mathcal {D}} (f) + \rho (1 - L _ {\mathcal {D}} (f)) \\ = (1 - 2 \rho) L _ {\mathcal {D}} (f) + \rho . \\ \end{array}
$$
Note, following the technique above, a similar statement is true about dataset  $S$  instead of true distribution  $\mathcal{D}$ , meaning that for a given  $f \in \mathcal{F}$ ,
$$
\mathbb {E} _ {S _ {\rho}} \hat {L} _ {S _ {\rho}} (f) = (1 - 2 \rho) \hat {L} _ {S} (f) + \rho . \tag {3}
$$
Recall that we take expectation with respect to different ways of adding noise to labels, therefore  $S_{\rho}$  and  $S$  have the same  $x$ , but different  $y$ . We do not use (3) for the proof of Theorem 2, but use it in Appendix J.
For true distribution  $\mathcal{D}$ , since  $l$  is 0-1 loss, then for a given model  $f$ ,  $l(f,z)$  is Bernoulli distributed with mean  $p_{Ber} = \mathbb{E}_{z\sim \mathcal{D}}l(f,z) = L_{\mathcal{D}}(f)$  and variance  $\sigma_f^2 = p_{Ber}(1 - p_{Ber}) = L_{\mathcal{D}}(f)(1 - L_{\mathcal{D}}(f))$ . Therefore, the expected variance for a given model  $f\in R_{set}(\mathcal{F},\gamma)$  on distribution  $\mathcal{D}_{\rho}$  is:
$$
\begin{array}{l} V a r _ {z \sim \mathcal {D} _ {\rho}} [ l (f, z) ] = \mathbb {E} _ {\mathcal {D} _ {\rho}} L _ {\mathcal {D} _ {\rho}} (f) (1 - L _ {\mathcal {D} _ {\rho}} (f)) \\ = \mathbb {E} _ {\mathcal {D} _ {\rho}} L _ {\mathcal {D} _ {\rho}} (f) - \mathbb {E} _ {\mathcal {D} _ {\rho}} (L _ {\mathcal {D} _ {\rho}} (f)) ^ {2} \\ = \mathbb {E} _ {\mathcal {D} _ {\rho}} L _ {\mathcal {D} _ {\rho}} (f) - \mathbb {E} _ {\mathcal {D} _ {\rho}} \left(L _ {\mathcal {D} _ {\rho}} (f)\right) ^ {2} \\ = \mathbb {E} _ {\mathcal {D} _ {\rho}} L _ {\mathcal {D} _ {\rho}} (f) - \left(\mathbb {E} _ {\mathcal {D} _ {\rho}} L _ {\mathcal {D} _ {\rho}} (f)\right) ^ {2} - V a r _ {\mathcal {D} _ {\rho}} \left[ L _ {\mathcal {D} _ {\rho} (f)} \right] \\ = L _ {\mathcal {D}} (f) (1 - 2 \rho) + \rho - (L _ {\mathcal {D}} (f) (1 - 2 \rho) + \rho) ^ {2} - V a r _ {\mathcal {D} _ {\rho}} [ L _ {\mathcal {D} _ {\rho} (f)} ] \\ = L _ {\mathcal {D}} (f) \left((1 - 2 \rho) - 2 \rho (1 - 2 \rho)\right) - L _ {\mathcal {D}} ^ {2} (f) (1 - 2 \rho) ^ {2} + \rho - \rho^ {2} - V a r _ {\mathcal {D} _ {\rho}} \left[ L _ {\mathcal {D} _ {\rho} (f)} \right] \\ = (1 - 2 \rho) ^ {2} (L _ {\mathcal {D}} (f) - L _ {\mathcal {D}} ^ {2} (f)) + \rho - \rho^ {2} - V a r _ {\mathcal {D} _ {\rho}} [ L _ {\mathcal {D} _ {\rho} (f)} ] \\ = (1 - 2 \rho) ^ {2} \left(L _ {\mathcal {D}} (f) (1 - L _ {\mathcal {D}} (f))\right) + \rho (1 - \rho) - V a r _ {\mathcal {D} _ {\rho}} \left[ L _ {\mathcal {D} _ {\rho} (f)} \right] \\ = (1 - 2 \rho) ^ {2} \left(L _ {\mathcal {D}} (f) \left(1 - L _ {\mathcal {D}} (f)\right)\right) + \rho (1 - \rho), \\ \end{array}
$$
Note that, by our assumption, there exists  $\bar{f}$  such that  $L_{\mathcal{D}}(\bar{f}) < \frac{1}{2} - \gamma$ , so  $L_{\mathcal{D}}(f^{*}) < \frac{1}{2} - \gamma$ , where  $f^{*}$  is optimal model. Then for any fixed  $f \in \mathcal{F}$ , we get  $L_{\mathcal{D}}(f) \leq L_{\mathcal{D}}(f^{*}) + \gamma < \frac{1}{2}$  which implies that  $L_{\mathcal{D}}(f)(1 - L_{\mathcal{D}}(f)) < \frac{1}{4}$ .
For  $\rho \in (0,\frac{1}{2})$ ,  $Var_{z\sim \mathcal{D}_{\rho}}[l(f,z)]$  is monotonically increasing in  $\rho$ , since:
$$
\begin{array}{l} \frac {\partial}{\partial \rho} \left[ V a r _ {z \sim \mathcal {D} _ {\rho}} [ l (f, z) ] \right] = \frac {\partial}{\partial \rho} \left[ (1 - 2 \rho) ^ {2} \left(L _ {\mathcal {D}} (f) \left(1 - L _ {\mathcal {D}} (f)\right)\right) + \rho (1 - \rho) \right] \\ = - 4 (1 - 2 \rho) \left(L _ {\mathcal {D}} (f) (1 - L _ {\mathcal {D}} (f))\right) + (1 - 2 \rho) \\ = (1 - 2 \rho) (1 - 4 L _ {\mathcal {D}} (f) (1 - L _ {\mathcal {D}} (f))) \\ > \left(1 - 2 \times \frac {1}{2}\right) \left(1 - 4 \times \frac {1}{4}\right) = 0. \\ \end{array}
$$
Consider  $\rho_1 < \rho_2$ . Since  $Var_{z \sim \mathcal{D}_{\rho}}[l(f, z)]$  is monotonically increasing in  $\rho$  for a fixed  $f$ , then  $\sigma^2(f, \mathcal{D}_{\rho_1}) < \sigma^2(f, \mathcal{D}_{\rho_2})$ , and we proved that variance increases with random uniform label noise.
In Theorem 2, the statement of the theorem is correct for any fixed  $f \in \mathcal{F}$ . Corollary 3 follows directly from Theorem 2. Here, instead of a fixed model  $f \in \mathcal{F}$ , we consider models in the Rashomon sets that maximize expected variance.
$$
\sup _ {f \in R _ {s e t} _ {\mathcal {D} _ {\rho_ {1}}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {1}}) <   \sup _ {f \in R _ {s e t} _ {\mathcal {D} _ {\rho_ {2}}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {2}}).
$$
Proof. Let  $f_1^{\mathrm{sup}}$  and  $f_2^{\mathrm{sup}}$  be maximizers of the variance of the loss in their respective Rashomon sets:
$$
f_{1}^{\sup}\in \operatorname *{arg  }\sup_{f\in R_{set\mathcal{D}_{\rho_{1}}}\left(\mathcal{F},\gamma\right)}Var_{z\sim \mathcal{D}_{\rho_{1}}}\left[l(f,z)\right],
$$
$$
f _ {2} ^ {\sup } \in \arg \operatorname * {s u p} _ {f \in R _ {s e t \mathcal {D} _ {\rho_ {2}}}} (F, \gamma) V a r _ {z \sim \mathcal {D} _ {\rho_ {2}}} [ l (f, z) ].
$$
Given that for any  $f \in R_{\text{set}_{\mathcal{D}_{\rho_2}}}(\mathcal{F}, \gamma)$ ,  $Var_{z \sim \mathcal{D}_{\rho_2}}[l(f, z)] \leq Var_{z \sim \mathcal{D}_{\rho_2}}[l(f_2^{\sup}, z)]$  and since  $Var_{z \sim \mathcal{D}_{\rho}}[l(f, z)]$  is monotonically increasing in  $\rho$ , we have that:
$$
\begin{array}{l} \sup  _ {f \in R _ {\text {s e t}} \mathcal {D} _ {\rho_ {1}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {1}}) = V a r _ {z \sim \mathcal {D} _ {\rho_ {1}}} [ l (f _ {1} ^ {\sup }, z) ] \\ <   \operatorname {V a r} _ {z \sim \mathcal {D} _ {\rho_ {2}}} [ l (f _ {1} ^ {\sup }, z) ] \\ \leq \operatorname {V a r} _ {z \sim \mathcal {D} _ {\rho_ {2}}} \left[ l \left(f _ {2} ^ {\sup }, z\right) \right] \\ = \sup  _ {f \in R _ {s e t} _ {\mathcal {D} _ {\rho_ {2}}} (\mathcal {F}, \gamma)} \sigma^ {2} (f, \mathcal {D} _ {\rho_ {2}}). \\ \end{array}
$$
Next, we generalize the statement of Theorem 2 to the non-uniform label noise case, where each sample  $z = (x,y)$  is flipped with probability  $\rho_{x}$  that depends on  $x$ . We show that under this non-uniform label noise, the variance of the loss increases in Theorem 12.
Theorem 12 (Variance increases with non-uniform label noise). Consider 0-1 loss  $l$ , infinite true data distribution  $\mathcal{D}$ , and a hypothesis space  $\mathcal{F}$ . Assume that there exists at least one function  $\bar{f} \in \mathcal{F}$  such that  $L_{\mathcal{D}}(\bar{f}) < \frac{1}{2} - \gamma$ . For a fixed  $f \in \mathcal{F}$ , let  $\sigma^2(f, \mathcal{D})$  be the variance of the loss:  $\sigma^2(f, \mathcal{D}) = \text{Var}_{z \sim \mathcal{D}} l(f, z)$  on data distribution  $\mathcal{D}$ . Consider non-uniform label noise, where each label  $y$  is flipped independently with probability  $\rho_x$ ,  $(x, y) \sim \mathcal{D}$ . Let  $\mathcal{D}_\rho$  denote the noisy version of  $\mathcal{D}$ . For any  $\delta > 0$ , let  $\mathcal{D}_{\rho^\delta}$  be a noisier data distribution than  $\mathcal{D}_\rho$ , meaning that for every sample
$(x,y)$  the probabilities of labels being flipped are higher by  $\delta$ :  $\rho_x^\delta = \rho_x + \delta$ . If for a fixed model  $f \in \mathcal{F}$ ,  $L_{\mathcal{D}_{\rho^{\delta}}}(f) < 0.5$ , then
$$
\sigma^ {2} (f, \mathcal {D} _ {\rho}) <   \sigma^ {2} (f, \mathcal {D} _ {\rho^ {\delta}}).
$$
Proof. Recall that the true risk for 0-1 loss  $L_{\mathcal{D}}(f) = \mathbb{E}_{z = (x,y)\sim \mathcal{D}}[l(f,z)] = \mathbb{E}_{(x,y)\sim \mathcal{D}}[\mathbb{1}_{[f(x)\neq y]}$ . Without loss of generality, let  $y\in \{0,1\}$ . Drawing from  $z_{\rho}\sim \mathcal{D}_{\rho}$  is equivalent to drawing  $z\sim \mathcal{D}$  and changing label  $y$  to  $1 - y$  with probability  $\rho_{x}$ . More explicitly, let  $\eta \sim$  Bernoulli  $(\rho_x)$ , then the flipped label is  $XOR(y,\eta) = \mathbb{1}_{[y\neq \eta ]}$ . For any given  $f\in \mathcal{F}$  we have that:
$$
\begin{array}{l} L _ {\mathcal {D} _ {\rho}} (f) = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \mathbb {1} _ {[ f (x) \neq X O R (y, \eta) ]} \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \mathbb {1} _ {[ f (x) \neq y ]} (1 - \eta) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \mathbb {1} _ {[ f (x) = y ]} \eta \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ (1 - \eta) \right] \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) = y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \eta \right] \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} [ (1 - \eta) ] \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \left(1 - \mathbb {1} _ {[ f (x) \neq y ]}\right) \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} [ \eta ] \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ (1 - \eta) \right] \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \eta \right] - \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \eta \right] \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ (1 - 2 \eta) \right] \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {E} _ {\eta \sim B e r (\rho_ {x})} \left[ \eta \right] \right] \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \left(1 - 2 \rho_ {x}\right) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \rho_ {x}. \\ \end{array}
$$
Now we will show that  $L_{\mathcal{D}_{\rho^{\delta}}}(f) > L_{\mathcal{D}_{\rho}}(f)$ :
$$
\begin{array}{l} L _ {\mathcal {D} _ {\rho^ {\delta}}} (f) - L _ {\mathcal {D} _ {\rho}} (f) = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \left(1 - 2 \rho_ {x} ^ {\delta}\right) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \rho_ {x} ^ {\delta} \\ - \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} (1 - 2 \rho_ {x}) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \rho_ {x} \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \left(- 2 \rho_ {x} ^ {\delta} + 2 \rho_ {x}\right) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left(\rho_ {x} ^ {\delta} - \rho_ {x}\right) \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} (- 2 \delta) \right] + \mathbb {E} _ {(x, y) \sim \mathcal {D}} (\delta) \\ = (- 2 \delta) \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbb {1} _ {[ f (x) \neq y ]} \right] + \delta \\ = \delta (1 - 2 L _ {\mathcal {D}} (f)) > 0. \\ \end{array}
$$
Note that, by our assumption, there exists  $\bar{f}$  such that  $L_{\mathcal{D}}(\bar{f}) < \frac{1}{2} - \gamma$ , so  $L_{\mathcal{D}}(f^{*}) < \frac{1}{2} - \gamma$ , where  $f^{*}$  is an optimal model. Then for any fixed  $f \in R_{\text{set}}(\mathcal{F}, \gamma)$ , we get  $L_{\mathcal{D}}(f) \leq L_{\mathcal{D}}(f^{*}) + \gamma < \frac{1}{2}$ , and then  $1 - 2L_{\mathcal{D}}(f) > 0$ . Since  $\delta > 0$ , we have shown that  $L_{\mathcal{D}_{\rho^{\delta}}}(\bar{f}) > L_{\mathcal{D}_{\rho}}(\bar{f})$ .
For true distribution  $\mathcal{D}$ , since  $l$  is 0-1 loss, then for a given model  $f$ ,  $l(f,z)$  is Bernoulli distributed with mean  $p_{Ber} = \mathbb{E}_{z\sim \mathcal{D}}l(f,z) = L_{\mathcal{D}}(f)$  and variance  $\sigma_f^2 = p_{Ber}(1 - p_{Ber}) = L_{\mathcal{D}}(f)(1 - L_{\mathcal{D}}(f))$ . Therefore, the expected variance for a given model  $f\in \mathcal{F}$  on distributions  $\mathcal{D}_{\rho}$  and  $\mathcal{D}_{\rho^{\delta}}$  is:
$$
\begin{array}{l} V a r _ {z \sim \mathcal {D} _ {\rho}} [ l (f, z) ] = L _ {\mathcal {D} _ {\rho}} (f) (1 - L _ {\mathcal {D} _ {\rho}} (f)) \\ <   L _ {\mathcal {D} _ {\rho^ {\delta}}} (f) (1 - L _ {\mathcal {D} _ {\rho^ {\delta}}} (f)) \\ = V a r _ {z \sim \mathcal {D} _ {\rho^ {\delta}}} [ l (f, z) ], \\ \end{array}
$$
where the inequality arises from the fact that the parabola  $x(1 - x)$  is monotonic along the interval  $x \in [0, 0.5]$ . This implies that  $\sigma^2(f, \mathcal{D}_{\rho}) < \sigma^2(f, \mathcal{D}_{\rho^\delta})$ .
We also show that the variance of losses increases under margin noise for data that come from Gaussian distributions (in Theorem 15). We model margin noise by moving two Gaussians closer together along the vector that connects the two means. Before stating and proving the theorem we discuss two lemmas that are helpful for the proof.
Lemma 13. Consider distribution  $\mathcal{X} \in \mathbb{R}^m$  and a linear model  $f = \omega^T x + b$ , where  $\omega \in \mathbb{R}^m$ ,  $\omega \neq \bar{0}$  and  $b \in \mathbb{R}$ . Let  $x \mapsto Ax + c$  be a bijective affine transformation, where  $A \in \mathbb{R}^{m \times m}$  and  $c \in \mathbb{R}^m$ . For the linear model  $g(x) = f(A^{-1}(x - c))$  and the distribution  $\mathcal{Z} = A\mathcal{X} + c$ , we have that:
$$
P _ {x \sim \mathcal {X}} (f (x) > 0) = P _ {z \sim \mathcal {Z}} (g (z) > 0).
$$
Proof. The proof follows from the lemma's statement and the assumption that  $A$  is a bijective affine transformation, and thus is invertible:
$$
P _ {z \sim \mathcal {Z}} (g (z) > 0) = P _ {x \sim \mathcal {X}} (g (A x + c) > 0) = P _ {x \sim \mathcal {X}} (f (A ^ {- 1} (A x + c - c)) > 0) = P _ {x \sim \mathcal {X}} (f (x) > 0).
$$
Lemma 14. Consider a Gaussian distribution  $\mathcal{X} \sim \mathcal{N}(\mu, I)$ , where  $\mu \in \mathbb{R}^m$ , and a linear model  $f = \omega^T x + b$ , where  $\omega \in \mathbb{R}^m$ ,  $\omega \neq \bar{0}$  and  $b \in \mathbb{R}$ . Let  $r = \frac{\omega^T\mu + b}{\|\omega\|}$  be the signed distance from  $\mu$  to the decision boundary of  $f$ . Then,
$$
P _ {x \sim \mathcal {X}} (f (x) > 0) = \Phi (r),
$$
where  $\Phi$  is the CDF of the univariate normal distribution  $\mathcal{N}(0,1)$ .
Proof. Let  $O \in \mathbb{R}^{m \times m}$  be a matrix with the first row equal to  $\frac{\omega}{\|\omega\|}$ , and let the other rows be chosen so that the rows of  $O$  form an orthonormal basis of  $\mathbb{R}^m$ . Note that  $O$  is an orthogonal matrix, so  $O$  is bijective and  $O^T O = O O^T = I$ . Let  $g(t) = f(O^{-1}(t + O\mu))$  and  $e_1$  be a unit vector  $e_1 = \{1, 0, \dots, 0\}$ , then:
$$
\begin{array}{l} g (t) = f \left(O ^ {- 1} (t + O \mu)\right) = f \left(O ^ {- 1} t + \mu\right) = f \left(O ^ {T} t + \mu\right) \\ = \omega^ {T} O ^ {T} t + \omega^ {T} \mu + b = \| \omega \| \left(e _ {1} ^ {T} t\right) + \omega^ {T} \mu + b \\ = \| \omega \| t _ {1} + \omega^ {T} \mu + b, \\ \end{array}
$$
where  $t_1$  is the first element of  $t$ , and  $\omega^T O^T = \| \omega \| e_1^T$  comes from the fact that  $\omega$  is orthogonal to every row of  $O$  except for the first row. Note that  $g(t) > 0$  when  $\| \omega \| t_1 + \omega^T \mu + b > 0$ , which leads to  $t_1 > -\frac{\omega^T \mu + b}{\|\omega\|} = -r$ . Correspondingly,  $g(t) < 0$  when  $t_1 < -r$ .
Now, let  $\mathcal{Z} = O(\mathcal{X} - \mu)$ . From the properties of the normal distribution,  $\mathcal{Z}\sim \mathcal{N}(\bar{0},I)$  since:
$$
\mathcal {Z} = O (\mathcal {X} - \mu) \sim \mathcal {N} (O (\mu - \mu), O I O ^ {T}) = \mathcal {N} (\bar {0}, I).
$$
Moreover, since the standard multivariate normal distribution is the joint distribution of independent univariate normal distributions,  $z_{1}\sim \mathcal{N}(0,1)$
From Lemma 13 and definitions of  $O$ ,  $g$ ,  $\mathcal{Z}$ , we get that  $P_{x \sim \mathcal{X}}(f(x) > 0) = P_{z \sim \mathcal{Z}}(g(z) > 0)$ . Therefore:
$$
\begin{array}{l} P _ {x \sim \mathcal {X}} (f (x) > 0) = P _ {z \sim \mathcal {Z}} (g (z) > 0) = P _ {z \sim \mathcal {Z}} (z _ {1} > - r) \\ = P _ {z _ {1} \sim \mathcal {N} (0, 1)} \left(z _ {1} > - r\right) = P _ {z _ {1} \sim \mathcal {N} (0, 1)} \left(z _ {1} \leq r\right) \\ = \Phi (r), \\ \end{array}
$$
where the strict inequality becomes non-strict since for the Gaussian distribution, the probability  $P_{z_1 \sim \mathcal{N}(0,1)}(z_i = r) = 0$ . Thus,  $P_{x \sim \mathcal{X}}(f(x) > 0) = \Phi(r)$  as desired.
Now, we show that the variance of losses increases under margin noise in the theorem below:
Theorem 15. Consider data distribution  $\mathcal{D} = \mathcal{X}\times \mathcal{Y}$ , where,  $\mathcal{X}\in \mathbb{R}^m$ ,  $\mathcal{Y}\in \{-1,1\}$ , classes are balanced  $P(Y = -1) = P(Y = 1)$  and generated by Gaussian distributions  $P(X|Y = -1) = \mathcal{N}(\bar{0},\Sigma)$ ,  $P(X|Y = 1) = \mathcal{N}(\mu ,\Sigma)$ , where  $\Sigma$  is a diagonal matrix with non-zero elements. Let the hypothesis space  $\mathcal{F}$  be the set of linear models,  $f = \omega^{T}x + b$ , where  $\omega \in \mathbb{R}^m$ .  $\omega \neq 0$  and  $b\in \mathbb{R}$ . We add margin noise by moving the means of the Gaussians towards each other by a factor of  $k$ , where  $0 < k < 1$ , meaning that the mean of the positive class becomes  $\mu_{k} = k\cdot \mu$ . For a fixed  $f\in \mathcal{F}$ , if  $L_{\mu}(f) < 0.5$ , we get that the variance of losses increases with more noise,
$$
\sigma (f, \mu) <   \sigma (f, \mu_ {k}).
$$
Proof. Without loss of generality, we will show that the variance of the losses increases for data generated from two Gaussian distributions  $P(X|Y = -1) = \mathcal{N}(\bar{0}, I)$  and  $P(X|Y = 1) = \mathcal{N}(\mu, I)$  (where  $I$  is the identity matrix) when we move them towards each other. More specifically, since normalization by variance  $\left( \frac{1}{\Sigma_{i,i}} \right)$  is a bijective linear transformation, by Lemma 13 we can work
Figure 4: An illustration of how Lemma 14 rotates each of the Gaussians  $\mathcal{N}(\mu_1, I), \mathcal{N}(\mu_2, I)$  and the decision boundary  $f(x)$  in order to compute loss as CDF of the signed distances  $(r_1, r_2)$  from means  $(\mu_1, \mu_2)$  to the rotated boundaries  $(g_1(z), g_2(z))$ . Note that we apply Lemma 14 separately to each Gaussian, thus there are two rotation operators  $O_1$  and  $O_2$ .
with  $P(X|Y = -1) = \mathcal{N}(\bar{0},I)$  and  $P(X|Y = 1) = \mathcal{N}(\mu ,I)$  instead of  $P(X|Y = -1) = \mathcal{N}(\bar{0},\Sigma)$  and  $P(X|Y = 1) = \mathcal{N}(\mu ,\Sigma)$ .
Let  $r_1 = \frac{b}{\|\omega\|}$  and  $r_2 = \frac{\omega^T\mu + b}{\|\omega\|}$  be the signed distances from the centers of the two Gaussians to the decision boundary. Then, from Lemma 14 (see illustration in Figure 4), the loss can be computed using the CDFs based on the signed distance:
$$
\begin{array}{l} L _ {\mu} (f) = P (f (x) > 0 | Y = - 1) P (Y = - 1) + P (f (x) \leq 0 | Y = 1) P (Y = 1) \\ = \frac {1}{2} P (f (x) > 0 | Y = - 1) + \frac {1}{2} (1 - P (f (x) > 0 | Y = 1)) \\ = \frac {1}{2} \left[ \left(\Phi \left(r _ {1}\right) + \left(1 - \Phi \left(r _ {2}\right)\right) \right] \right. \\ \end{array}
$$
Next, we will show that  $\omega^T\mu > 0$ . If  $L_{\mu}(f) < \frac{1}{2}$ , then we get that  $\frac{1}{2}[(\Phi(r_1) + (1 - \Phi(r_2)))] < \frac{1}{2}$ , which means that  $\Phi(r_2) > \Phi(r_1)$ . Since the CDF of the Gaussian distribution  $\mathcal{N}(\bar{0}, I)$  is strictly increasing, we have that  $r_2 > r_1$ , which means that  $\frac{\omega^T\mu + b}{\|\omega\|} > \frac{b}{\|\omega\|}$ , and so  $\omega^T\mu > 0$ .
Recall that we induce noise by moving the Gaussians towards each other by decreasing  $k$ . Now we will show that loss is monotonically decreasing with respect to increasing values of  $k$ , or equivalently that  $\frac{\partial}{\partial k} L_{\mu_k}(f) < 0$ :
$$
\begin{array}{l} \frac {\partial}{\partial k} L _ {\mu_ {k}} (f) = \frac {\partial}{\partial k} \left(\frac {1}{2} \left[ \left(\Phi \left(r _ {1}\right) + \left(1 - \Phi \left(r _ {2}\right)\right) \right] \right.\right) \\ = \frac {\partial}{\partial k} \left( \right.\frac {1}{2} \left[ \right.\left( \right.\Phi \left(\frac {b}{\| \omega \|}\right) + 1 - \Phi \left(\frac {k \omega^ {T} \mu + b}{\| \omega \|}\right)\left. \right]\left. \right) \\ = - \frac {1}{2} \left[ \frac {\partial}{\partial k} \Phi \left(\frac {k \omega^ {T} \mu + b}{\| \omega \|}\right) \right] = - \frac {1}{2} \frac {\omega^ {T} \mu}{\| \omega \|} \phi \left(\frac {k \omega^ {T} \mu + b}{\| \omega \|}\right) <   0, \\ \end{array}
$$