SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here,  $\sigma_{\mathrm{max}}$  is the largest singular value of the matrix  $\mathbf{H}$ . In order to find the optimal robust estimator we wish to solve the optimization problem  $\arg \min_{\mathbf{H}} R_{\epsilon}(\mathbf{H})$ . The difficulty in solving this optimization problem is that we can't solve the minimization problem within the expectation (3) in closed form. In order to prove Theorem 3.1 for denoising (i.e., for  $\mathbf{A} = \mathbf{I}$ ) we derived an upper and a matching lower bound of the risks using several unusual applications of Jensen's inequality. The proof does not generalize in a straightforward manner to the more general case where  $\mathbf{A} \neq \mathbf{I}$ . However, for large  $d$ , the random variable  $\mathbf{v}^T (\mathbf{I} - \frac{1}{\lambda} \mathbf{H} \mathbf{H}^T)^{-1} \mathbf{v}$  concentrates around its expectation, and thus we conjecture that for large  $d$ , we can exchange expectation and minimization, which yields:
$$
R _ {\epsilon} (\mathbf {H}) = \min  _ {\lambda \geq \sigma_ {\max } ^ {2}} \lambda \epsilon^ {2} + \mathbb {E} _ {\mathbf {v}} \left[ \mathbf {v} ^ {T} \left(\mathbf {I} - \frac {1}{\lambda} \mathbf {H} \mathbf {H} ^ {T}\right) ^ {- 1} \mathbf {v} \right]. \tag {4}
$$
The expectation in the risk expression (4) can be explicitly computed, which yields the following conjecture for the worst-case optimal estimator:
Conjecture 3.3. For  $d\to \infty$  the optimal worst-case estimator, i.e., the estimator minimizing the worst-case risk  $R_{\epsilon}(f)$  amongst all estimators of the form  $f(\mathbf{y}) = \mathbf{H}\mathbf{y}$ , is  $\mathbf{H} = \mathbf{U}\mathbf{V}\mathrm{diag}(\sigma_i)\mathbf{W}^T$  with
$$
\sigma_ {i} = \frac {1 + \lambda \lambda_ {i} ^ {2}}{2 \lambda_ {i}} + \frac {d}{m} \frac {\lambda}{2 \lambda_ {i}} \frac {\sigma_ {z} ^ {2}}{\sigma_ {c} ^ {2}} - \sqrt {\left(\frac {1 + \lambda \lambda_ {i} ^ {2}}{2 \lambda_ {i}} + \frac {d}{m} \frac {\lambda}{2 \lambda_ {i}} \frac {\sigma_ {z} ^ {2}}{\sigma_ {c} ^ {2}}\right) ^ {2} - \lambda},
$$
if  $\lambda_{i}\neq 0$  and  $\sigma_{i} = 0$  otherwise. Here, the parameter  $\lambda$  is a solution of:
$$
\operatorname * {a r g m i n} _ {\lambda \geq 0} \lambda \epsilon^ {2} + \sum_ {i = 1} ^ {d} \frac {1 - \lambda \lambda_ {i} ^ {2}}{2} \frac {\sigma_ {c} ^ {2}}{d} - \frac {\lambda}{2} \frac {\sigma_ {z} ^ {2}}{m} + \sqrt {\left(\frac {1 + \lambda \lambda_ {i} ^ {2}}{2} \frac {\sigma_ {c} ^ {2}}{d} + \frac {\lambda}{2} \frac {\sigma_ {z} ^ {2}}{m}\right) ^ {2} - \lambda \lambda_ {i} ^ {2} \frac {\sigma_ {c} ^ {4}}{d ^ {2}}}.
$$
The optimization problem involved is convex and box-constrained and can thus be solved numerically. Besides the argument above, we confirmed our conjecture with numerical simulations.
Optimal jittering estimator. Unlike for denoising, for general inverse problems, the jittering-risk minimizing estimator is in general not equal to the optimal worst-case estimator, but the two estimators are often close. The optimal estimator minimizing the jittering risk is given as (see Appendix C):
$$
\mathbf {H} _ {J} \left(\sigma_ {w}\right) = \mathbf {U V} \operatorname {d i a g} \left(\frac {\sigma_ {c} ^ {2} \lambda_ {i}}{\sigma_ {c} ^ {2} \lambda_ {i} ^ {2} + \sigma_ {z} ^ {2} \frac {d}{m} + \sigma_ {w} ^ {2} d}\right) \mathbf {W} ^ {T}, \tag {5}
$$
where as before  $\mathbf{A}\mathbf{U} = \mathbf{W}\boldsymbol{\Lambda}\mathbf{V}^T$  is the singular value decomposition with singular values  $\lambda_{i}$ . While the estimator (5) has the same form as the worst-case optimal estimator in Conjecture 3.3, the diagonal matrix in the two estimators is in general slightly different.
Numerical Simulation. The worst-case suboptimality of the jittering-risk optimal estimator (5) depends on the singular values of the matrix  $\mathbf{A}\mathbf{U}$ ; if they are equal the jittering-risk estimator is optimal, and if they are not equal there is typically a small gap. To illustrate the gap, we consider a forward operator  $\mathbf{A}$  with linearly decaying singular values  $\frac{i}{n}$ , for  $1 \leq i \leq n$  with signal energy  $\sigma_c^2 = 1$  and noise level  $\sigma_z = 0.2$ . We compare the (conjectured) optimal robust estimator specified by Conjecture 3.3 with the optimal Jittering estimator (5) at noise level  $\sigma_w$ , where  $\sigma_w$  is optimized such that one obtains minimal robust risk  $R_{\epsilon}$  at a given perturbation level  $\epsilon$ . The results in Figure 2, right panel, show a small gap in robust risk, which implies that Jittering is suboptimal for this case. However, simulations with varying forward operators and noise levels indicate that the gap is small relative to the robust risk of the standard estimator. Experiments on image deconvolution using U-Net presented in Section 4 show similar results.
In this section, we train standard convolutional neural-networks with standard training, adversarial training, and jittering for three inverse problems: denoising images, image deconvolution, and compressive sensing, and study their robustness. We find that Jittering yields well-performing robust denoisers at a computational cost similar to standard training, which is significantly cheaper than adversarial training. We also find that jittering yields robust estimators for deconvolution and compressive sensing. This indicates that training on real data which often contains slight measurement noise is robustness enhancing.
4.1 Problem setup
We start by describing the datasets, networks, and methodology.
Natural images. We consider denoising and deconvolution of natural images, where our goal is to reconstruct an image  $\mathbf{x} \in \mathbb{R}^n$  from a noisy measurement  $\mathbf{y} = \mathbf{A}\mathbf{x} + \mathbf{z}$ , where  $\mathbf{z} \sim \mathcal{N}(0, \sigma_z^2 1 / n\mathbf{I})$  is Gaussian noise and  $\mathbf{A}$  a measurement matrix, which is equal to identity for denoising, and implements
a convolution for deconvolution. For deconvolution we use a  $8 \times 8$ -sized discretization of the 2-dimensional Gaussian normal distribution with standard deviation 2. The kernel is visualized in Figure 3 in the appendix. We obtain train and validation datasets  $\{(\mathbf{x}_1, \mathbf{y}_1), \dots, (\mathbf{x}_N, \mathbf{y}_N)\}$  of sizes  $34k$  and  $4k$ , respectively, from colorized images of size  $n = 128 \cdot 128 \cdot 3$  generated by randomly cropping and flipping ImageNet images. The methods are tested on  $2k$  original-sized images.
Medical data. We also perform experiments on accelerated singlecoil magnetic resonance imaging (MRI) data, where the goal is to reconstruct an image  $\mathbf{x} \in \mathbb{R}^n$  from a noisy and subsampled measurement in the frequency domain  $\mathbf{y} = \mathbf{M}\mathbf{F}\mathbf{x} + \mathbf{z} \in \mathbb{R}^{2m}$ . We use the fastMRI singlecoil knee dataset (Zbontar et al., 2018), which contains the images  $\mathbf{x}$  and fully sampled measurements  $(\mathbf{M} = \mathbf{I})$ . We process it by random subsampling at acceleration factor 4 and obtain train, validation and test datasets with approximately  $31\mathrm{k}$ ,  $3.5\mathrm{k}$  and  $7\mathrm{k}$  slices, respectively. While perturbations are sought in frequency domain, the inverse Fourier transform is applied to the measurements before feeding the  $320 \times 320$  cropped and normalized images into the network.
Network architecture. We use the U-net architecture (Ronneberger et al., 2015) since it gives excellent performance for denoising (Brooks et al., 2019) and medical image reconstruction tasks, such as computed tomography (Jin et al., 2017) and is used as a building block for state-of-the-art methods for magnetic resonance imaging (Zbontar et al., 2018; Sriram et al., 2020). For natural images, we use a U-net with  $3 \times 3$  padded convolutions with ReLU activation functions,  $2 \times 2$  max-pooling layers for downscaling and transposed convolutions for upscaling. The network has  $120k$  parameters. For MRI reconstruction we use a U-Net architecture similar to Zbontar et al. (2018) with  $3 \times 3$  padded convolutions, leaky ReLU activation function,  $2 \times 2$  average pooling and transposed convolutions (480k learnable parameters). We denote the U-Net by the parameterized mapping  $f_{\theta} \colon \mathbb{R}^{m} \to \mathbb{R}^{n}$  in the following.
Evaluation. We evaluate networks by measuring its robustness via the empirical robust risk defined as  $\hat{R}_{\epsilon}(\pmb{\theta}) = \frac{1}{N}\sum_{i=1}^{N}\max_{\|\mathbf{e}_i\|_2 \leq \epsilon}\|f_{\pmb{\theta}}(\mathbf{y}_i + \mathbf{e}_i) - \mathbf{x}_i\|_2^2$ . For evaluation, the robust empirical risk is computed over the test set. We assess the accuracy by computing the standard empirical risk  $\hat{R}_0(\pmb{\theta})$ . Computing the robust empirical risk is non-trivial since it requires finding adversarial perturbations for solving the inner maximization problem. This is explained next. We also study the computational cost of the different methods, which we measure in terms of GPU cost and time.
Finding adversarial perturbations. To evaluate the empirical risk and for robust training, we need to compute adversarial perturbations  $\mathbf{e} = \arg \max_{\| \mathbf{e}\| _2\leq \epsilon}\| f_\theta (\mathbf{y} + \mathbf{e}) - \mathbf{x}\| _2^2$ . We find the perturbations by running  $N_{a}$  projected gradient ascent steps, starting with initial perturbation  $\mathbf{e}^0 = 0$  and iterate
$$
\mathbf {e} ^ {j + 1} = \mathcal {P} _ {B (0, \epsilon)} \left(\mathbf {e} ^ {j} + 2. 5 \frac {\epsilon}{N _ {a}} \frac {\Delta \mathbf {e} ^ {j}}{\| \Delta \mathbf {e} ^ {j} \| _ {2}}\right), \quad \text {w h e r e} \quad \Delta \mathbf {e} ^ {j} = \nabla_ {\mathbf {e} ^ {j}} \left\| f _ {\theta} (\mathbf {e} ^ {j} + \mathbf {y}) - \mathbf {x} \right\| _ {2} ^ {2}.
$$
Here,  $\mathcal{P}_{B(0,\epsilon)}$  is the projection into the  $\ell_2$ -ball  $B(0;\epsilon)$  of radius  $\epsilon$  around the origin. The gradient is normalized to facilitate step size optimization with multiplier 2.5 such that the iteration can reach and move along the boundary, as suggested by Madry et al. (2018).
Training methods. Standard training minimizes the standard empirical risk  $\hat{R}_0$ . Adversarial training minimizes the empirical robust risk  $\hat{R}_{\epsilon}$ . To minimize the empirical robust risk, we approximate the inner maximization,  $\max_{\| \mathbf{e}\| _2\leq \epsilon}\| f_\theta (\mathbf{y}_i + \mathbf{e}) - \mathbf{x}_i\| _2^2$ , with  $\| f_{\theta}(\tilde{\mathbf{y}}_i) - \mathbf{x}_i\| _2^2$ , where  $\tilde{\mathbf{y}}_i$  is the adversarially perturbed measurement computed as described above. Training via jittering minimizes
$$
\hat {J} _ {\sigma_ {w}} (\boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {w} \sim \mathcal {N} (0, \sigma_ {w} ^ {2} \mathbf {I})} \left[ \| f _ {\theta} (\mathbf {y} _ {i} + \mathbf {w}) - \mathbf {x} _ {i} \| _ {2} ^ {2} \right],
$$
where the jittering level  $\sigma_w$  is chosen depending on the desired robustness level. We treat the jittering noise level as a hyperparameter optimized using the validation dataset (shown in the appendix). Jittering is practically implemented via performing the SGD update rule  $\theta_{i+1} = \theta_i - \frac{\eta}{M} \sum_{i=1}^{M} \nabla_\theta \|f_\theta(\mathbf{y}_i + \mathbf{w}_i) - \mathbf{x}_i\|^2$ , with  $M$  samples per batch and learning rate  $\eta$ . For jittering, the network output is calculated on the noisy input  $\mathbf{y}_i + \mathbf{w}_i$ , instead of  $\mathbf{y}_i$  (standard training) or
Figure 3: Estimating the optimal jittering noise levels for the denoising task. The left panel shows the results of training networks via Jittering, at noise levels  $\sigma_w$ , and calculating the empirical robust risk  $\hat{R}_{\epsilon}$  of each model. Each green line corresponds to robust risks at one perturbation level. The optimal jittering noise levels are shown in the middle panel and follow well the prediction from theory (Cor. 3.2, details in appendix). The jittering estimators are similarly robust as adversarial training (Figure 1), but attain lower standard risks (right panel).
$\mathbf{y}_i + \mathbf{e}_i$  (adversarial training). To approximate the expectation in the jittering risk we draw independent jittering noise samples  $\mathbf{w}_i$  in each iteration of SGD.
Throughout, we use PyTorch's Adam optimizer with learning rate  $10^{-3}$  and batch size 50 for natural images, and  $10^{-2}$  and 1 for MRI data. As perturbation levels, we consider values within the practically interesting regime of  $\epsilon^2 /\mathbb{E}\left[\| \mathbf{A}\mathbf{x}\| _2^2\right] < 0.3$  for natural images and 0.03 for MRI data. Note that for  $\epsilon^2 >\mathbb{E}\left[\| \mathbf{x}\| _2^2\right]$ , Theorem 3.1 predicts for denoising  $(\mathbf{A} = \mathbf{I})$  that the optimal robust estimator is zero everywhere. Figure 7 in the appendix shows that for large perturbations  $\epsilon$  the trained U-net denoiser also maps to zero.
We now discuss the results of the denoising, deconvolution, and compressive sensing experiments.
Robust and standard performance. Figure 1, shows that the standard estimator is relatively robust for Gaussian denoising and increasingly sensitive for more ill-posed problems (deconvolution and compressive sensing). The experiments further show that jittering is effective for enhancing robustness, in particular relative to the sensitivity of the standard estimator. Nevertheless, as suggested by theory, we see a gap between the robust risk of adversarial training and jittering for image deconvolution and compressive sensing. For Gaussian denoising, however, Jittering is particularly effective and yields increasingly better performing networks in terms of standard risks for larger perturbations.
Choice of the jittering level. The results are based on choosing the jittering noise levels via hyperparameter search for each task. Figure 3 shows the results for Gaussian denoising: It can be seen that the choice of noise level is important for minimizing the robust risk. The estimated noise levels also aligns well the theoretical prediction. Details on this and the parameter choices for deconvolution and the compressive sensing experiments are in the appendix.
Computational complexity. We measured the GPU time until convergence and memory utilization of the methods and present the results in the Table 1 of the appendix. Performing adversarial training is by a factor of the projected gradient ascent steps more expensive than standard training. Moreover, training via jittering has similar computational cost as standard training, since it solely consists of drawing and adding Gaussian noise on the training data.
Figure 4: Example reconstructions using measurements (second column) and separately calculated perturbed measurements. The reconstructions are denoted as clean and perturbed, respectively. The perturbation levels are  $\epsilon^2 /\mathbb{E}[\| \mathbf{Ax}\| _2^2 ] = 0.03$  for denoising, 0.003 for compressive sensing and 0.001 for deconvolution. We can see that the standard estimator is visibly sensitive to perturbations. Jittering yields robust estimators, but at the same time yields smoother reconstructions.
Visual reconstructions. For the linear subspace setting adversarial training and jittering are equivalent. For Gaussian denoising with a neural network, however, they perform differently. For larger perturbations jittering tends to yield smoother images than networks trained adversarily, as can be seen in the example reconstructions shown in Figure 4. This effect is particularly noticeable for the Gaussian deconvolution task. We quantified the effect using the total variation norm and present the results in the appendix. Moreover, we discuss an approximation to jittering, Jacobian regularization, which similarly enhances robustness. It is computationally more expensive, but yields less smooth reconstructions.
In this paper, we characterized the optimal worst-case robust estimator for Gaussian subspace denoising and found that the optimal estimator can be provably learned with jittering. Our results for training neural networks for Gaussian denoising of images show that jittering enables the training of neural networks that are as robust as neural networks trained adversariably, but at a fraction of the computational cost, and without the hassle of having to find adversarial perturbations. While we demonstrated that jittering can yield suboptimal robust estimators in general, in practice, jittering is effective at improving the robustness for compressive sensing and image deconvolution. Moreover, our results imply that training on real data that contains slight measurement noise is robustness enhancing.
Reproducability The repository at https://github.com/MLI-lab/robust_reconstructors_via_jittering contains the code to reproduce all results in the main body of this paper.
Acknowledgments A.K. and R.H. are supported by the Institute of Advanced Studies at the Technical University of Munich, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 456465471, 464123524, the DAAD, and the German Federal Ministry of Education and Research, and the Bavarian State Ministry for Science and the Arts. M.S. is supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in Mathematics, an NSF-CAREER under award #1846369, DARPA FastNICS programs, and NSF-CIF awards #1813877 and #2008443.
A Proof of Theorem 3.1
In the main body we stated an analytical characterization of the optimal worst-case robust denoiser. We present the proof in the following and show that the risk
$$
\min  _ {\mathbf {H}} R _ {\epsilon} (\mathbf {H}) = \min  _ {\mathbf {H}} \mathbb {E} _ {\mathbf {x}, \mathbf {z}} \left[ \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {H} (\mathbf {x} + \mathbf {e} + \mathbf {z}) - \mathbf {x} \| _ {2} ^ {2} \right] \tag {6}
$$
is minimized by  $\mathbf{H} = \sigma \mathbf{U}\mathbf{U}^T$  with
$$
\sigma = \left\{ \begin{array}{l l} \sigma_ {c} ^ {2} - \frac {\epsilon \sigma_ {c} \sigma_ {z} \sqrt {\frac {d}{n}}}{\sqrt {\sigma_ {c} ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} - \epsilon^ {2}}} & \text {i f} \sigma_ {c} ^ {2} > \epsilon^ {2} \\ \hline \sigma_ {c} ^ {2} + \sigma_ {z} ^ {2} \frac {d}{n} & \\ 0 & \text {e l s e .} \end{array} \right.
$$
The scaling factor  $\sigma$  as a function of the noise level  $\epsilon$  of the optimal worst-case estimator is shown in Figure 2.
We start by proving a lower bound of the risk, which relies on a characterization of the maximization and many unexpected applications of Jensens inequality.
We then compute the risk for  $\mathbf{H} = \sigma \mathbf{U}\mathbf{U}^T$  and show that it is equivalent to the lower bound on the risk.
A.1 Lower bounding the risk
Towards lower bounding the risk we define for notational convenience
$$
\left[ \begin{array}{c c} \mathbf {H} _ {\parallel} & \mathbf {H} _ {0} \\ \mathbf {H} _ {1} & \mathbf {H} _ {\perp} \end{array} \right] = \left[ \begin{array}{c c} \mathbf {U} ^ {T} \mathbf {H} \mathbf {U} & \mathbf {U} ^ {T} \mathbf {H} \mathbf {U} _ {\perp} \\ \mathbf {U} _ {\perp} ^ {T} \mathbf {H} \mathbf {U} & \mathbf {U} _ {\perp} ^ {T} \mathbf {H} \mathbf {U} _ {\perp} \end{array} \right].
$$
$$
\mathbf {H} = \left[ \begin{array}{c c} \mathbf {U} & \mathbf {U} _ {\perp} \end{array} \right] \left[ \begin{array}{c c} \mathbf {H} _ {\parallel} & \mathbf {H} _ {0} \\ \mathbf {H} _ {1} & \mathbf {H} _ {\perp} \end{array} \right] \left[ \begin{array}{c} \mathbf {U} ^ {T} \\ \mathbf {U} _ {\perp} ^ {T} \end{array} \right]
$$
$$
\begin{array}{l} \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \| \mathbf {H} (\mathbf {x} + \mathbf {e} + \mathbf {z}) - \mathbf {x} \| _ {2} ^ {2} = \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| [ \mathbf {U} \quad \mathbf {U} _ {\perp} ] \left[ \begin{array}{c c} \mathbf {H} _ {\parallel} & \mathbf {H} _ {0} \\ \mathbf {H} _ {1} & \mathbf {H} _ {\perp} \end{array} \right] \left[ \begin{array}{c} \mathbf {U} ^ {T} \\ \mathbf {U} _ {\perp} ^ {T} \end{array} \right] (\mathbf {U} \mathbf {c} + \mathbf {e} + \mathbf {z}) - \mathbf {U} \mathbf {c} \right\| _ {2} ^ {2} \\ = \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| \left[ \begin{array}{l l} \mathbf {H} _ {\parallel} & \mathbf {H} _ {0} \\ \mathbf {H} _ {1} & \mathbf {H} _ {\perp} \end{array} \right] \left[ \begin{array}{c} \mathbf {c} + \mathbf {U} ^ {T} \mathbf {z} + \mathbf {U} ^ {T} \mathbf {e} \\ \mathbf {U} _ {\perp} ^ {T} \mathbf {z} + \mathbf {U} _ {\perp} ^ {T} \mathbf {e} \end{array} \right] - \left[ \begin{array}{l} \mathbf {c} \\ 0 \end{array} \right] \right\| _ {2} ^ {2} \\ \geq \max  _ {\| \mathbf {e} \| _ {2} \leq \epsilon} \left\| \left[ \begin{array}{c c} \mathbf {H} _ {\parallel} & \mathbf {H} _ {0} \end{array} \right] \left[ \begin{array}{c} \mathbf {c} + \mathbf {U} ^ {T} \mathbf {z} + \mathbf {U} ^ {T} \mathbf {e} \\ \mathbf {U} _ {\perp} ^ {T} \mathbf {z} + \mathbf {U} _ {\perp} ^ {T} \mathbf {e} \end{array} \right] - \mathbf {c} \right\| _ {2} ^ {2} \\ \geq \max  _ {\left\| \mathbf {e} _ {\parallel} \right\| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} \left(\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {\parallel}\right) - \mathbf {c} + \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2}. \\ \end{array}
$$
Here, we defined  $\mathbf{e}_{\parallel} = \mathbf{U}^T\mathbf{e}$  and  $\mathbf{z}_{\parallel} = \mathbf{U}^T\mathbf{z}$  for notational convenience, and the last inequality follows by adding  $\mathbf{U}_{\perp}^{T}\mathbf{e} = 0$  as a constraint to the maximization, which gives a lower bound.
Next, note that since  $\mathbf{z} = \begin{bmatrix} \mathbf{z}_{\parallel} \\ \mathbf{z}_{\perp} \end{bmatrix} \sim \mathcal{N}(0, \sigma_z^2 / n\mathbf{I})$  is Gaussian, the vector  $\begin{bmatrix} \mathbf{z}_{\parallel} \\ -\mathbf{z}_{\perp} \end{bmatrix}$  is equally Gaussian distributed  $\mathcal{N}(0, \sigma_z^2 / n\mathbf{I})$ . This implies that
$$
\mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {\parallel} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {\parallel}) - \mathbf {c} + \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} \right] = \mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {\parallel} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {\parallel}) - \mathbf {c} - \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} \right].
$$
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {\parallel} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {\parallel}) - \mathbf {c} + \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} \right] \\ = \frac {1}{2} \mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {1} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\|} (\mathbf {c} + \mathbf {z} _ {\|} + \mathbf {e} _ {1}) - \mathbf {c} + \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} + \max  _ {\| \mathbf {e} _ {2} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\|} (\mathbf {c} + \mathbf {z} _ {\|} + \mathbf {e} _ {2}) - \mathbf {c} - \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} \right] \\ \geq \mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {1} \| _ {2} \leq \epsilon} \frac {1}{2} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {1}) - \mathbf {c} + \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} + \frac {1}{2} \left\| \mathbf {H} _ {\parallel} (\mathbf {c} + \mathbf {z} _ {\parallel} + \mathbf {e} _ {1}) - \mathbf {c} - \mathbf {H} _ {0} \mathbf {z} _ {\perp} \right\| _ {2} ^ {2} \right] \\ \geq \mathbb {E} \left[ \max  _ {\| \mathbf {e} _ {1} \| _ {2} \leq \epsilon} \left\| \mathbf {H} _ {\|} (\mathbf {c} + \mathbf {z} _ {\|} + \mathbf {e} _ {1}) - \mathbf {c} \right\| _ {2} ^ {2} \right], \\ \end{array}
$$
where the last step follows from Jensens inequality. Thus, we have shown that