SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\mathbf {H}} \mathbb {E} \left[ \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {H} (\mathbf {x} + \mathbf {e} + \mathbf {z}) - \mathbf {x} \| _ {2} ^ {2} \right] \geq \min  _ {\mathbf {H} _ {\parallel}} \mathbb {E} \left[ \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} + \mathbf {e}) - \mathbf {c} \| _ {2} ^ {2} \right]. \tag {7}
$$
For simplicity of exposition, we drop the ||-notation. Thus, with a slight abuse of notation, the vectors on the left and right hand side have different dimensions. On the left hand side,  $\mathbf{z}$ ,  $\mathbf{e} \in \mathbb{R}^n$ , while on the right hand side  $\mathbf{z}$ ,  $\mathbf{e} \in \mathbb{R}^d$  and  $\mathbf{z}$  throughout has iid  $\mathcal{N}(0, \sigma_z^2 / n)$  entries.
Next, we'll apply the lemma below for characterizing the maximization inside of the expectation. The proof is in Section A.3. Similar computations as used to prove the lemma are on page 19-20 in the paper Lee et al. (2021) for deriving robustness-accuracy trade-off bounds.
Lemma A.1. For any  $\mathbf{z}$
$$
\max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {z} - \mathbf {H e} \| _ {2} ^ {2} = \min  _ {\lambda : \lambda \geq \lambda_ {\max } (\mathbf {H} ^ {T} \mathbf {H})} \lambda \epsilon^ {2} + \mathbf {z} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} \mathbf {z}. \tag {8}
$$
Using Lemma A.1 the term within the expectation is
$$
\begin{array}{l} \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| (\mathbf {H} - \mathbf {I}) \mathbf {c} + \mathbf {H z} + \mathbf {H e} \| _ {2} ^ {2} \\ = \min  _ {\lambda : \lambda \geq \lambda_ {\max } (\mathbf {H} ^ {T} \mathbf {H})} \lambda \epsilon^ {2} + \left(\left(\mathbf {H} - \mathbf {I}\right) \mathbf {c} + \mathbf {H} \mathbf {z}\right) ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} \left(\left(\mathbf {H} - \mathbf {I}\right) \mathbf {c} + \mathbf {H} \mathbf {z}\right) \\ = \min  _ {\lambda : \lambda \geq \lambda_ {\max } (\mathbf {H} ^ {T} \mathbf {H})} \lambda \epsilon^ {2} + \mathbf {c} ^ {T} (\mathbf {H} - \mathbf {I}) ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} (\mathbf {H} - \mathbf {I}) \mathbf {c} \\ + 2 \mathbf {z} ^ {T} \mathbf {H} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} (\mathbf {H} - \mathbf {I}) \mathbf {c} \\ + \mathbf {z} ^ {T} \mathbf {H} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} \mathbf {H} \mathbf {z}. \\ \end{array}
$$
Now let's rewrite using the singular value decomposition  $\mathbf{H}_{\parallel} = \mathbf{V}\pmb{\Sigma}\mathbf{W}^T$ , with orthogonal matrices  $\mathbf{V}$  and  $\mathbf{W}$ . With this notation, the terms above can be rewritten as
$$
\begin{array}{l} \mathbf {c} ^ {T} \left(\mathbf {V} \boldsymbol {\Sigma} \mathbf {W} ^ {T} - \mathbf {I}\right) ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {V} \boldsymbol {\Sigma} \mathbf {W} ^ {T} \left(\mathbf {V} \boldsymbol {\Sigma} \mathbf {W} ^ {T}\right) ^ {T}\right) ^ {- 1} \left(\mathbf {V} \boldsymbol {\Sigma} \mathbf {W} ^ {T} - \mathbf {I}\right) \mathbf {c} \\ = \mathbf {c} ^ {T} (\mathbf {W} \boldsymbol {\Sigma} \mathbf {V} ^ {T} - \mathbf {I}) \mathbf {V} \left(\mathbf {I} - \frac {1}{\lambda} \boldsymbol {\Sigma} ^ {2}\right) ^ {- 1} \mathbf {V} ^ {T} (\mathbf {V} \boldsymbol {\Sigma} \mathbf {W} ^ {T} - \mathbf {I}) \mathbf {c} \\ = \mathbf {c} ^ {T} (\mathbf {W} \boldsymbol {\Sigma} - \mathbf {V}) \left(\mathbf {I} - \frac {1}{\lambda} \boldsymbol {\Sigma} ^ {2}\right) ^ {- 1} \left(\boldsymbol {\Sigma} \mathbf {W} ^ {T} - \mathbf {V} ^ {T}\right) \mathbf {c} = \sum_ {i = 1} ^ {d} \frac {\left(\left(\sigma_ {i} \mathbf {w} _ {i} - \mathbf {v} _ {i}\right) ^ {T} \mathbf {c}\right) ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}}, \\ \end{array}
$$
where  $\mathbf{w}_i$  and  $\mathbf{v}_i$  are the  $i$ -th column vectors of the orthogonal matrices  $\mathbf{W}$  and  $\mathbf{V}$ , respectively. Using similar calculations for the other summands, the optimization problem on the right hand side of inequality (7) is
$$
\begin{array}{l} \min  _ {\mathbf {H} _ {\parallel}} \mathbb {E} \left[ \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} + \mathbf {e}) - \mathbf {c} \right\| _ {2} ^ {2} \right] \\ = \min _ {\mathbf {V}, \mathbf {W}, \sigma_ {i}} \mathbb {E} _ {\mathbf {c}, \mathbf {z}} \left[ \min _ {\lambda \colon \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {\left(\left(\sigma_ {i} \mathbf {w} _ {i} - \mathbf {v} _ {i}\right) ^ {T} \mathbf {c}\right) ^ {2} + 2 \left(\left(\sigma_ {i} \mathbf {w} _ {i} - \mathbf {v} _ {i}\right) ^ {T} \mathbf {c}\right) \left(\sigma_ {i} \mathbf {w} _ {i} ^ {T} \mathbf {z}\right) + \left(\sigma_ {i} \mathbf {w} _ {i} ^ {T} \mathbf {z}\right) ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right] \\ = \min _ {\mathbf {V}, \mathbf {W}, \sigma_ {i}} \mathbb {E} _ {\mathbf {c}, \mathbf {z}} \left[ \min _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {\left(\left(\sigma_ {i} \mathbf {w} _ {i} - \mathbf {v} _ {i}\right) ^ {T} \mathbf {c} + \sigma_ {i} \mathbf {w} _ {i} ^ {T} \mathbf {z}\right) ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right] \\ \stackrel {\mathrm {i}} {=} \underset {\mathbf {V}, \mathbf {W}, \sigma_ {i}} {\min } \mathbb {E} _ {\mathbf {g}} \left[ \underset {\lambda : \lambda \geq \sigma_ {i} ^ {2}} {\min } \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \left(\left(\sigma_ {i} ^ {2} + 1 - 2 \sigma_ {i} \mathbf {v} _ {i} ^ {T} \mathbf {w} _ {i}\right) \frac {\sigma_ {c} ^ {2}}{d} + \sigma_ {i} ^ {2} \frac {\sigma_ {z} ^ {2}}{n}\right)}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right] \\ \stackrel {\mathrm {i i}} {\geq} \min  _ {\mathbf {V}, \mathbf {W}, \sigma_ {i}} \mathbb {E} _ {\mathbf {g}} \left[ \min  _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \left(\left(\sigma_ {i} ^ {2} + 1 - 2 \sigma_ {i}\right) \frac {\sigma_ {c} ^ {2}}{d} + \sigma_ {i} ^ {2} \frac {\sigma_ {z} ^ {2}}{n}\right)}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right] \\ = \min _ {\sigma_ {i}} \mathbb {E} _ {\mathbf {g}} \left[ \min _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \left(\frac {\sigma_ {c} ^ {2}}{d} (\sigma_ {i} - 1) ^ {2} + \frac {\sigma_ {z} ^ {2}}{n} \sigma_ {i} ^ {2}\right)}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right], \\ \end{array}
$$
where minimization above is over the orthonormal matrices  $\mathbf{V},\mathbf{W}\in \mathbb{R}^{d\times d}$  and over the singular values  $\sigma_{i}$ . Equality (i) follows by noting that  $(\sigma_{i}\mathbf{w}_{i}^{T} - \mathbf{v}_{i}^{T})\mathbf{c}$  is Gaussian with variance  $(\sigma_i^2 +1 - 2\sigma_i(\mathbf{v}_i^T\mathbf{w}_i))\frac{\sigma_c^2}{d}$  and  $(\sigma_{i}\mathbf{w}_{i}^{T})\mathbf{z}$  is Gaussian with variance  $\sigma_i^2\frac{\sigma_z^2}{n}$  and therefore we can replace the expectation with respect to  $(\mathbf{c},\mathbf{z})$  with the expectation with respect to  $\mathbf{g}\sim \mathcal{N}(0,\mathbf{I})$  and suitable scaling. Moreover, inequality (ii) follows from the fact that  $\sigma_{i}\geq 0$  and  $g_{i}^{2}\geq 0$
Continuing, first note that the function  $\frac{x^2}{y}$  is convex in  $(x,y)$  when  $y > 0$ . Also the extended value function is increasing in the first input and decreasing in the second input. Furthermore the mappings  $(x,z) \mapsto x - 1$  and  $(x,z) \mapsto 1 - \frac{x^2}{z}$  are convex and concave. Thus by the composition rule of convex functions we conclude that the functions
$$
\frac {x ^ {2}}{1 - \frac {(x - 1) ^ {2}}{z}} \quad \text {a n d} \frac {(x - 1) ^ {2}}{1 - \frac {(x - 1) ^ {2}}{z}}
$$
are jointly convex in  $(x,z)$
Jensen's inequality states that for a convex function  $\psi$  we have  $\frac{\sum_{i}z_i^2\psi(x_i)}{\sum_{i}z_i^2}\geq \psi \left(\frac{\sum_{i}z_i^2\mathbf{x}_i}{\sum_{i}z_i^2}\right)$ . Thus by Jensen's inequality the sum in the expectation in the right-hand-side of the equation above can be lower-bounded as
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \frac {\sigma_ {c} ^ {2}}{d} (\sigma_ {i} - 1) ^ {2} + g _ {i} ^ {2} \frac {\sigma_ {z} ^ {2}}{n} \sigma_ {i} ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} = \| \mathbf {g} \| _ {2} ^ {2} \sum_ {i = 1} ^ {d} \frac {\frac {g _ {i} ^ {2}}{\| \mathbf {g} \| _ {2} ^ {2}} \frac {\sigma_ {c} ^ {2}}{d} (\sigma_ {i} - 1) ^ {2} + \frac {g _ {i} ^ {2}}{\| \mathbf {g} \| _ {2} ^ {2}} \frac {\sigma_ {z} ^ {2}}{n} \sigma_ {i} ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \\ \geq \| \mathbf {g} \| _ {2} ^ {2} \frac {\frac {\sigma_ {c} ^ {2}}{d} \left(\sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2}}{\| \mathbf {g} \| _ {2} ^ {2}} \sigma_ {i} - 1\right) ^ {2} + \frac {\sigma_ {z} ^ {2}}{n} \left(\sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2}}{\| \mathbf {g} \| _ {2} ^ {2}} \sigma_ {i}\right) ^ {2}}{1 - \frac {\left(\sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2}}{\| \mathbf {g} \| _ {2} ^ {2}} \sigma_ {i}\right) ^ {2}}{\lambda}} \\ = \| \mathbf {g} \| _ {2} ^ {2} \frac {\frac {\sigma_ {c} ^ {2}}{d} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \frac {\sigma_ {z} ^ {2}}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}} \\ \end{array}
$$
where  $\bar{\sigma} (\mathbf{g}) = \sum_{i = 1}^{d}\frac{g_i^2}{\|\mathbf{g}\|_2^2}\sigma_i$
Now consider the event  $\mathcal{E} = \{\mathbf{g}\colon \| \mathbf{g}\| _2^2\geq (1 - \delta)d\}$  which holds with probability at least  $1 - e^{-\frac{\delta^2}{2} d}$ . On this event, we have
$$
\lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \frac {\sigma_ {c} ^ {2}}{d} (\sigma_ {i} - 1) ^ {2} + g _ {i} ^ {2} \frac {\sigma_ {z} ^ {2}}{n} \sigma_ {i} ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \geq \lambda \epsilon^ {2} + (1 - \delta) \frac {\sigma_ {c} ^ {2} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}}.
$$
Using the same argument as before the right hand side of the above inequality is jointly convex in  $(\lambda, \bar{\sigma}(\mathbf{c}))$ . Since partial minimization of a jointly convex function preserves convexity we conclude that the function
$$
\psi (\bar {\sigma}) = \min  _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \mathbb {1} _ {\mathcal {E}} \lambda \epsilon^ {2} + \mathbb {1} _ {\mathcal {E}} (1 - \delta) \frac {\sigma_ {c} ^ {2} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}}
$$
is convex in  $\bar{\sigma}$ . Thus by using convexity in terms of  $\bar{\sigma}$ , applying Jensen's inequality (i.e.,  $\mathbb{E}[\psi(\bar{\sigma})] \geq \psi(\mathbb{E}[\bar{\sigma}])$ ) we have
$$
\begin{array}{l} \min  _ {\sigma_ {i}} \mathbb {E} _ {\mathbf {g}} \left[ \min  _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \| \mathbf {g} \| _ {2} ^ {2} \frac {1}{d} \frac {\sigma_ {c} ^ {2} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}} \right] \\ \geq \min _ {\sigma_ {i}} \mathbb {E} _ {\mathbf {g}} \left[ \min _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \mathbb {1} _ {\mathcal {E}} \lambda \epsilon^ {2} + \mathbb {1} _ {\mathcal {E}} (1 - \delta) \frac {\sigma_ {c} ^ {2} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}} \right] \\ \geq \min _ {\sigma_ {i}} \mathbb {E} _ {\mathbf {g} | \mathcal {E}} \left[ \min _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + (1 - \delta) \frac {\sigma_ {c} ^ {2} (\bar {\sigma} (\mathbf {g}) - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma} (\mathbf {g})) ^ {2}}{1 - \frac {(\bar {\sigma} (\mathbf {g})) ^ {2}}{\lambda}} \right] \\ \geq \min  _ {\sigma_ {i}} \min  _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + (1 - \delta) \frac {\sigma_ {c} ^ {2} (\mathbb {E} [ \bar {\sigma} (\mathbf {g}) ] - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\mathbb {E} [ \bar {\sigma} (\mathbf {g}) ]) ^ {2}}{1 - \frac {(\mathbb {E} [ \bar {\sigma} (\mathbf {g}) ]) ^ {2}}{\lambda}} \\ \geq \min_{\sigma_{i}}\min_{\lambda :  \lambda \geq \sigma_{i}^{2}}\lambda \epsilon^{2} + (1 - \delta)\frac{\sigma_{c}^{2}\left(\bar{\sigma} - 1\right)^{2} + \sigma_{z}^{2}\frac{d}{n}\left(\bar{\sigma}\right)^{2}}{1 - \frac{\left(\bar{\sigma}\right)^{2}}{\lambda}} \\ \end{array}
$$
where we defined  $\bar{\sigma} = \sum_{i=1}^{d} \mathbb{E}_{\mathbf{g}|\mathcal{E}}\left[\frac{g_i^2}{\|\mathbf{g}\|_2^2}\right] \sigma_i$ . Putting things together, we have shown that
$$
\begin{array}{l} \min _ {\mathbf {H}} R _ {\epsilon} (\mathbf {H}) \geq \min _ {\mathbf {H} _ {\parallel}} \mathbb {E} \left[ \max _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} + \mathbf {e}) - \mathbf {c} \right\| _ {2} ^ {2} \right] \\ = \min  _ {\sigma_ {i}} \mathbb {E} _ {\mathbf {g}} \left[ \min  _ {\lambda : \lambda \geq \sigma_ {i} ^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {g _ {i} ^ {2} \frac {\sigma_ {c} ^ {2}}{d} (\sigma_ {i} - 1) ^ {2} + g _ {i} ^ {2} \frac {\sigma_ {z} ^ {2}}{n} \sigma_ {i} ^ {2}}{1 - \frac {\sigma_ {i} ^ {2}}{\lambda}} \right] \\ \geq \min_{\bar{\sigma}}\min_{\lambda \colon \lambda \geq \sigma_{i}^{2}}\lambda \epsilon^{2} + (1 - \delta)\frac{\sigma_{c}^{2}\left(\bar{\sigma} - 1\right)^{2} + \sigma_{z}^{2}\frac{d}{n}\left(\bar{\sigma}\right)^{2}}{1 - \frac{(\bar{\sigma})^{2}}{\lambda}} \\ \geq \min _ {\bar {\sigma}} \min _ {\lambda : \lambda \geq \bar {\sigma} ^ {2}} \lambda \epsilon^ {2} + (1 - \delta) \frac {\sigma_ {c} ^ {2} (\bar {\sigma} - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} (\bar {\sigma}) ^ {2}}{1 - \frac {(\bar {\sigma}) ^ {2}}{\lambda}}, \\ \end{array}
$$
where the last inequality follows from  $\sigma_{\mathrm{max}} \geq \bar{\sigma}$ . For  $d \to \infty$ , we can choose  $\delta$  arbitrarily small, which yields
$$
\min  _ {\mathbf {H}} R _ {\epsilon} (\mathbf {H}) \geq \min  _ {\sigma} \min  _ {\lambda : \lambda \geq \sigma^ {2}} \lambda \epsilon^ {2} + \frac {\sigma_ {c} ^ {2} (\sigma - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} \sigma^ {2}}{1 - \frac {\sigma^ {2}}{\lambda}}. \tag {9}
$$
Solving the optimization problem (9): Consider the inner minimization problem in equation (9), i.e.,
$$
\min  _ {\lambda : \lambda \geq \sigma^ {2}} f (\lambda) \quad \text {w i t h} f (\lambda) = \lambda \epsilon^ {2} + \frac {c (\sigma)}{1 - \frac {\sigma^ {2}}{\lambda}},
$$
where  $c(\sigma) = \sigma_c^2 (\sigma - 1)^2 + \sigma_z^2 \frac{d}{n} \sigma^2$  for notational convenience. Since  $f$  is differentiable on  $(\sigma^2, \infty)$  we can calculate its critical point  $\lambda^*$  by setting the derivative with respect to  $\lambda$  to zero, which yields
$$
\lambda^ {*} = \frac {\sqrt {c (\sigma)}}{\epsilon} \sigma + \sigma^ {2}.
$$
From this expression, we see that the constraint  $\lambda^{*} > \sigma^{2}$  is satisfied and by convexity of  $f$  we know that  $\lambda^{*}$  is the unique minimizer. Hence, we have:
$$
\min  _ {\lambda : \lambda \geq \sigma^ {2}} f (\lambda) = f \left(\lambda^ {*}\right) = \left(\epsilon \sigma + \sqrt {c (\sigma)}\right) ^ {2}. \tag {10}
$$
$$
\min  _ {\mathbf {H}} R _ {\epsilon} (\mathbf {H}) \geq \min  _ {\sigma} g (\sigma), \quad g (\sigma) = \left(\epsilon \sigma + \sqrt {\sigma_ {c} ^ {2} (\sigma - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} \sigma^ {2}}\right) ^ {2}. \tag {11}
$$
Calculating the derivative of  $g(\sigma)$  and setting it to zero, we get:
$$
2 \left(\epsilon \sigma + \sqrt {\sigma_ {c} ^ {2} (\sigma - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} \sigma^ {2}}\right) \left(\epsilon + \frac {2 \sigma_ {c} ^ {2} (\sigma - 1) + \sigma_ {z} ^ {2} \frac {d}{n} 2 \sigma}{2 \sqrt {\sigma_ {c} ^ {2} (\sigma - 1) ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} \sigma^ {2}}}\right) = 0
$$
The left factor is non-negative, and the right factor is zero if  $\sigma_c^2 >\epsilon^2$  and if
$$
\sigma^ {*} = \frac {\sigma_ {c} ^ {2} - \frac {\epsilon \sigma_ {c} \sigma_ {z} \sqrt {\frac {d}{n}}}{\sqrt {\sigma_ {c} ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} - \epsilon^ {2}}}}{\sigma_ {c} ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n}}. \tag {12}
$$
If  $\sigma_c^2 \leq \epsilon^2$ , then the function  $g(\sigma)$  is monotonically increasing on  $[0, \infty)$  and hence  $\sigma_* = 0$  is the minimizer.
A.2 Upper bound for the risk of the estimator  $\mathbf{H} = \sigma \mathbf{U}\mathbf{U}^T$
We upper bound the risk of the estimator  $\mathbf{H} = \sigma \mathbf{U}\mathbf{U}^T$ . For  $\mathbf{H} = \sigma \mathbf{U}\mathbf{U}^T$  the risk (6) becomes
$$
\begin{array}{l} R _ {\epsilon} (\sigma \mathbf {U U} ^ {T}) = \mathbb {E} _ {\mathbf {x}, \mathbf {z}} \left[ \max _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {H} (\mathbf {x} + \mathbf {e} + \mathbf {z}) - \mathbf {x} \| _ {2} ^ {2} \right] \\ = \mathbb {E} _ {\mathbf {x}, \mathbf {z}} \left[ \max _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| \sigma \mathbf {U} \mathbf {U} ^ {T} (\mathbf {U} \mathbf {c} + \mathbf {e} + \mathbf {z}) - \mathbf {U} \mathbf {c} \right\| _ {2} ^ {2} \right] \\ = \mathbb {E} _ {\mathbf {c}, \mathbf {z}} \left[ \max  _ {\left\| \mathbf {e} _ {\parallel} \right\| _ {2} \leq \epsilon} \left\| (\sigma - 1) \mathbf {c} + \sigma \mathbf {e} _ {\parallel} + \sigma \mathbf {z} _ {\parallel} \right\| _ {2} ^ {2} \right]. \\ \end{array}
$$
With Lemma A.1,
$$
\begin{array}{l} R _ {\epsilon} (\sigma \mathbf {U U} ^ {T}) = \min _ {\mathbf {H}} \mathbb {E} _ {\mathbf {c}, \mathbf {z} _ {\parallel}} \left[ \min _ {\lambda : \lambda \geq \sigma^ {2}} \lambda \epsilon^ {2} + ((\sigma - 1) \mathbf {c} + \sigma \mathbf {z} _ {\parallel}) ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \sigma^ {2} \mathbf {I}\right) ^ {- 1} ((\sigma - 1) \mathbf {c} + \sigma \mathbf {z} _ {\parallel}) \right] \\ = \min  _ {\mathbf {H}} \mathbb {E} _ {\mathbf {c}, \mathbf {z}} \left[ \min  _ {\lambda : \lambda \geq \sigma^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {\left(\left(\sigma - 1\right) c _ {i} + \sigma z _ {i}\right) ^ {2}}{1 - \frac {\sigma^ {2}}{\lambda}} \right] \\ = \min _ {\mathbf {H}} \mathbb {E} _ {\mathbf {g}} \left[ \min _ {\lambda : \lambda \geq \sigma^ {2}} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} g _ {i} ^ {2} \frac {(\sigma - 1) ^ {2} \frac {\sigma_ {c} ^ {2}}{d} + \sigma^ {2} \frac {\sigma_ {z} ^ {2}}{n}}{1 - \frac {\sigma^ {2}}{\lambda}} \right]. \\ \end{array}
$$
Using the optimal  $\lambda^{*}$ , by equation 10, we get
$$
\begin{array}{l} R _ {\epsilon} (\sigma \mathbf {U U} ^ {T}) = \mathbb {E} \left[ \left(\epsilon \sigma + \| \mathbf {g} \| _ {2} \frac {1}{\sqrt {d}} \sqrt {(\sigma - 1) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}}\right) ^ {2} \right] \\ = \mathbb {E} \left[ (\epsilon \sigma) ^ {2} + 2 \epsilon \sigma \| \mathbf {g} \| _ {2} \frac {1}{\sqrt {d}} \sqrt {(\sigma - 1) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}} + \| \mathbf {g} \| _ {2} ^ {2} \frac {1}{d} \left((\sigma - 1) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}\right) \right] \\ \stackrel {\mathrm {i}} {\leq} \left(\epsilon \sigma\right) ^ {2} + 2 \epsilon \sigma \sqrt {\left(\sigma - 1\right) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}} + \left(\left(\sigma - 1\right) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}\right) \\ \stackrel {\mathrm {i i}} {=} \left(\epsilon \sigma + \sqrt {(\sigma - 1) ^ {2} \sigma_ {c} ^ {2} + \sigma^ {2} \frac {d}{n} \sigma_ {z} ^ {2}}\right) ^ {2}, \\ \end{array}
$$
where equation (i) follows by using Jensen's inequality once again (specifically, using  $(\mathbb{E}[\| \mathbf{g}\|_2])^2 \leq \mathbb{E}\left[\| \mathbf{g}\|_2^2\right] = d)$ . Noting that (ii) is equal to the lower bound of the risk for any symmetric  $\mathbf{H}$  in equation (11) shows that  $\mathbf{H} = \sigma \mathbf{U} \mathbf{U}^T$  with the optimal parameter  $\sigma^*$  derived above is optimal.
A.3 Proof of Lemma A.1
The optimization problem
$$
\max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {z} - \mathbf {H e} \| _ {2} ^ {2} \tag {13}
$$
can be written as
$$
\min _ {\mathbf {e}} - \left\| \mathbf {z} - \mathbf {H e} \right\| _ {2} ^ {2} \mathrm {s u b j e c t t o} \epsilon^ {2} - \mathbf {e} ^ {T} \mathbf {e} \geq 0.
$$
The corresponding Lagrangian is, for  $\lambda \geq 0$
$$
\begin{array}{l} L (\mathbf {e}, \lambda) = - (\mathbf {z} - \mathbf {H e}) ^ {T} (\mathbf {z} - \mathbf {H e}) - \lambda (\epsilon^ {2} - \mathbf {e} ^ {T} \mathbf {e}) \\ = - \mathbf {z} ^ {T} \mathbf {z} + 2 \mathbf {z} ^ {T} \mathbf {H e} - \mathbf {e} ^ {T} \mathbf {H} ^ {T} \mathbf {H e} - \lambda \epsilon^ {2} + \lambda \mathbf {e} ^ {T} \mathbf {e} \\ = \mathbf {e} ^ {T} (\lambda \mathbf {I} - \mathbf {H} ^ {T} \mathbf {H}) \mathbf {e} + 2 \mathbf {z} ^ {T} \mathbf {H} \mathbf {e} - \lambda \epsilon^ {2} - \mathbf {z} ^ {T} \mathbf {z} \\ \end{array}
$$
The Lagrange Dual Function is
$$
q (\lambda) = \inf  _ {\mathbf {e}} L (\mathbf {e}, \lambda).
$$
Using that  $\min_{\mathbf{e}} 2\mathbf{c}^T \mathbf{e} + \mathbf{e}^T \mathbf{B}\mathbf{e} = -\mathbf{c}^T \mathbf{B}^{-1} \mathbf{c}$ , we get, provided that  $\lambda \geq \lambda_{\max}(\mathbf{H}^T \mathbf{H})$
$$
\begin{array}{l} q (\lambda) = - \mathbf {z} ^ {T} \mathbf {H} ^ {T} \left(\lambda \mathbf {I} - \mathbf {H} ^ {T} \mathbf {H}\right) ^ {- 1} \mathbf {H} ^ {T} \mathbf {z} - \lambda \epsilon^ {2} - \mathbf {z} ^ {T} \mathbf {z} \\ = - \mathbf {z} ^ {T} \left(\mathbf {I} + \mathbf {H} (\lambda \mathbf {I} - \mathbf {H} ^ {T} \mathbf {H}) ^ {- 1} \mathbf {H}\right) \mathbf {z} - \lambda \epsilon^ {2} \\ \mathbf {\Sigma} = - \mathbf {z} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} ^ {T} \mathbf {H}\right) ^ {- 1} \mathbf {z} - \lambda \boldsymbol {\epsilon} ^ {2} \\ \end{array}
$$
where the last equality follows from the Woodbury Identity. Thus, the dual problem is
$$
\begin{array}{l} \max  _ {\lambda \geq 0} q (\lambda) = \min  _ {\lambda \geq 0} - q (\lambda) \\ = \min  _ {\lambda \geq \lambda_ {\max } (\mathbf {H} ^ {T} \mathbf {H})} \lambda \epsilon^ {2} + \mathbf {z} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} ^ {T} \mathbf {H}\right) ^ {- 1} \mathbf {z}. \\ \end{array}
$$
Even though the primal problem (13) is non-convex, strong duality holds, since the primal program is a quadratic program that is strictly feasible, see Boyd and Vandenberghe (2004, Appendix B.1). The primal problem is strictly feasible if there exists a vector  $\mathbf{e}$  such that  $\epsilon^2 -\mathbf{e}^T\mathbf{e} > 0$ . This is trivially satisfied as long as  $\epsilon >0$ .
B Additional proofs for denoising
We state two more proofs on optimal worst-case denoisers and jittering.
B.1 Proof of Corollary 3.2
In the main text it was stated that linear estimators  $f(\mathbf{y}) = \mathbf{H}\mathbf{y}$  that minimize the jittering risk  $J_{\sigma_w}$  with noise level chosen as a function of the desired noise level  $\epsilon$  as
$$
\sigma_ {w} (\epsilon) = \sqrt {\frac {\epsilon^ {2} \sigma_ {z} ^ {2} \frac {d}{n} + \sigma_ {z} \sqrt {\frac {d}{n}} \sigma_ {c} \epsilon \sqrt {\sigma_ {c} ^ {2} - \epsilon^ {2} + \sigma_ {z} ^ {2} \frac {d}{n}}}{d (\sigma_ {c} ^ {2} - \epsilon^ {2})}}
$$
also minimizes the worst-case risk  $R_{\epsilon}$ .
The result follows from Theorem 3.1. For that let  $f_{r}$  and  $f_{j}$  be linear estimators minimizing the robust risk  $R_{\epsilon}$  and the jittering risk  $J_{\sigma_w}$ , respectively. By Theorem 3.1, the two estimators are scaled projections onto the subspace, i.e.,  $f_{r}(\mathbf{y}) = \alpha_{r}\mathbf{U}\mathbf{U}^{T}$  and  $f_{j}(\mathbf{y}) = \alpha_{j}\mathbf{U}\mathbf{U}^{T}$  with