SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the first term is the risk of  $\mathcal{C}$  with respect to the uniform distribution on  $M$ . In words, eq. (19) is positive whenever  $\mathcal{C}$  performs better on the distribution  $p(X)$  than the uniform distribution, and this outcome is to be expected for an autoencoder with sufficient capacity trained on a non-uniform distribution  $p(X)$ .
Remark L.3. With notation as in the previous example, the result [4, Thm. 2] shows that the solution obtained with calculus of variations applied to an objective of the form
$$
\min  _ {\mathcal {C}} \int_ {M} \left(| \mathcal {C} (X) - X | _ {2} ^ {2} + \sigma^ {2} | \nabla_ {X} \mathcal {C} (X) | _ {2} ^ {2}\right) p (X) d x \tag {20}
$$
(i.e. mean squared error loss and an  $\ell^2$  penalty on gradient norms) satisfies
$$
\mathcal {C} (X) - X = \sigma^ {2} \nabla_ {X} \log p (X) + o \left(\sigma^ {2}\right) \text {a n d} \tag {21}
$$
$$
\nabla_ {X} \mathcal {C} (X) = I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X) + o \left(\sigma^ {2}\right) \text {a s} \sigma \rightarrow 0. \tag {22}
$$
In particular, the reconstruction error  $|\mathcal{C}(X) - X|_2$  is proportional to the gradient norm  $|\nabla_X\log p(X)|_2$  (up to higher order terms in  $\sigma^2$  as  $\sigma \to 0$ ). Since
$$
| \nabla_ {X} \log p (X) | _ {2} = \frac {1}{p (X)} | \nabla_ {X} p (X) | _ {2},
$$
this result sheds a more precise and quantitative light on  $(\ast)$ .
Equation (22) may also shed some light on robustness of  $\mathcal{C}$  as measured by  $\mathcal{R}$ : indeed, if we assume  $c(X) - X$  is small,
$$
\mathcal {C} (c (X)) - X = \mathcal {C} (X + c (X) - X) - X \approx \mathcal {C} (X) + \nabla_ {X} \mathcal {C} ^ {T} (c (X) - X) - X. \tag {23}
$$
Applying eqs. (21) and (22) gives the approximation
$$
\begin{array}{l} \mathcal {C} (X) + \nabla_ {X} \mathcal {C} ^ {T} (c (X) - X) - X \approx \sigma^ {2} \nabla_ {X} \log p (X) \\ + \left(I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X)\right) \left(c (X) - X\right) + o \left(\sigma^ {2}\right) \tag {24} \\ \end{array}
$$
as  $\sigma \to 0$ . Taking norms squared we get
$$
\begin{array}{l} \left| \mathcal {C} (c (X)) - X \right| ^ {2} \approx \sigma^ {2} \left| \nabla_ {X} \log p (X) \right| ^ {2} \\ + 2 \sigma^ {2} \nabla_ {X} \log p (X) ^ {T} \left(I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X)\right) (c (X) - X) \tag {25} \\ + \left| \left(I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X)\right) (c (X) - X) \right| ^ {2} + o \left(\sigma^ {2}\right) \\ \end{array}
$$
as  $\sigma \to 0$ . If we again make the simplifying assumption that  $c(X) - X$  is random and independent of  $X$ , then after taking the expectation over  $c$  (here denoted by  $E_{c}$ ) we obtain
$$
\begin{array}{l} E _ {c} \left[ \left| \mathcal {C} (c (X)) - X \right| ^ {2} \right] \approx \sigma^ {2} \left| \nabla_ {X} \log p (X) \right| ^ {2} \\ + E _ {c} \left[ \left| \left(I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X)\right) \left(c (X) - X\right) \right| ^ {2} \right] + o \left(\sigma^ {2}\right) \tag {26} \\ \end{array}
$$
as  $\sigma \to 0$ . If we let  $\alpha_{1} \geq \alpha_{2} \geq \dots \geq \alpha_{p}$  be the eigenvalues of  $\operatorname{Hess}_X \log p(X)$  and choose  $\sigma$  small enough that  $1 + \frac{\sigma^2}{\alpha_i} > 0$  for all  $i$ , we see that the norm
$$
\left| \left(I + \sigma^ {2} \operatorname {H e s s} _ {X} \log p (X)\right) (c (X) - X) \right| ^ {2} \tag {27}
$$
surpresses (resp. magnifies) contributions from components of  $c(X) - X$  along the eigenvectors associated with  $\alpha_{i}$  for large (resp. small)  $i$ . Note that it is reasonable as a first approximation to expect  $\mathrm{Hess}_X\log p(X)$  to be negative definite (this is precisely the case when  $p$  is a Gaussian distribution, and more generally the requirements  $p(X)\geq 0$  and  $\int p(X)dX = 1$  put restrictions on the extent to which  $\mathrm{Hess}_X\log p(X)$  can exhibit non-negative eigenvalues). In this case the large (resp. small) values of  $i$  correspond to the eigenvectors in directions of "sharpest" (resp. "shallowest") curvature of the graph of  $p(X)$ . Explicitly, if  $X$  is Gaussian with covariance  $\Sigma$ , then  $\mathrm{Hess}_X\log p(X) = \Sigma^{-1}$ , and if  $\lambda_1\geq \lambda_2\geq \dots \geq \lambda_p$  are the eigenvalues of  $\Sigma$  then  $\alpha_{i} = -\frac{1}{\lambda_{i}}$  for all  $i$ . In this case we see that the norm in eq. (27) surpasses (resp. magnifies) contributions from components of  $c(X) - X$  along principal components with low (resp. high) variance. Note that this provides a takeaway qualitatively similar to that of corollary 5.2.
L.4 A fundamental robustness-generalization inequality
The similarity between the  $\mathcal{R}$  heatmaps for glass blur in fig. 4 and the PSDs of the corruptions themselves appearing in fig. 1a has a simple explanation in terms of a fundamental relationship between  $\mathcal{D},\mathcal{G}$  and  $\mathcal{R}$ . One can show (via two applications of the triangle inequality) that
$$
\frac {1}{N} \sum_ {k = 1} ^ {N} P S D (X _ {k} - c (X _ {k})) - \mathcal {G} (\mathcal {C}, \mathcal {X}, c) \leq \mathcal {R} (\mathcal {C}, \mathcal {X}, c) \leq \frac {1}{N} \sum_ {k = 1} ^ {N} P S D (X _ {k} - c (X _ {k})) + \mathcal {G} (\mathcal {C}, \mathcal {X}, c). \tag {28}
$$
In the case of glass blur, generalization error is very low, so eq. (28) reduces to  $\mathcal{R} \approx \frac{1}{N}\sum_{k=1}^{N}PSD(X_k - c(X_k))$ , where the right-hand side is the average PSD of the corruption itself.
M Tables of classification accuracies
No Compression
clean	0.76
snow	0.34
frost	0.31
fog	0.46
brightness	0.69
contrast	0.44
motion blur	0.37
zoom blur	0.35
defocus blur	0.36
glass blur	0.16
elastic transform	0.53
jpeg compression	0.59
pixelate	0.50
gaussian noise	0.32
shot noise	0.29
impulse noise	0.32
Table 1: Classification accuracies without compression on the validation split of ImageNet-C.
SH NIC	ELIC	JPEG2000	JPEG
PSNR=36.93
clean	0.72	0.72	0.72	0.74
snow	0.33	0.31	0.30	0.33
frost	0.29	0.29	0.30	0.30
fog	0.28	0.28	0.45	0.44
brightness	0.65	0.65	0.65	0.68
contrast	0.26	0.26	0.44	0.42
motion blur	0.35	0.34	0.36	0.36
zoom blur	0.33	0.32	0.34	0.34
defocus blur	0.30	0.30	0.36	0.36
glass blur	0.19	0.19	0.17	0.16
elastic transform	0.48	0.49	0.52	0.53
jpeg compression	0.60	0.60	0.59	0.60
pixelate	0.55	0.53	0.51	0.50
gaussian noise	0.33	0.36	0.28	0.30
shot noise	0.29	0.32	0.25	0.28
impulse noise	0.31	0.34	0.23	0.30
PSNR=32.97
clean	0.64	0.64	0.67	0.70
snow	0.27	0.26	0.25	0.28
frost	0.25	0.25	0.28	0.28
fog	0.13	0.11	0.38	0.28
brightness	0.57	0.55	0.59	0.63
contrast	0.11	0.09	0.38	0.27
motion blur	0.30	0.29	0.34	0.32
zoom blur	0.29	0.28	0.32	0.30
defocus blur	0.26	0.24	0.34	0.30
glass blur	0.22	0.24	0.19	0.18
elastic transform	0.42	0.45	0.50	0.51
jpeg compression	0.57	0.57	0.57	0.59
pixelate	0.53	0.51	0.53	0.51
gaussian noise	0.28	0.30	0.25	0.37
shot noise	0.26	0.27	0.23	0.35
impulse noise	0.29	0.32	0.17	0.37
PSNR=31.17
clean	0.58	0.57	0.62	0.66
snow	0.22	0.21	0.21	0.26
frost	0.20	0.20	0.25	0.25
fog	0.08	0.07	0.32	0.17
brightness	0.51	0.48	0.54	0.60
contrast	0.07	0.05	0.34	0.16
motion blur	0.27	0.26	0.33	0.28
zoom blur	0.26	0.25	0.31	0.27
defocus blur	0.24	0.22	0.32	0.24
glass blur	0.23	0.24	0.21	0.19
elastic transform	0.39	0.40	0.48	0.50
jpeg compression	0.52	0.53	0.55	0.58
pixelate	0.49	0.47	0.53	0.48
gaussian noise	0.28	0.30	0.21	0.33
shot noise	0.28	0.31	0.21	0.32
impulse noise	0.29	0.31	0.16	0.33
Table 2: Classification accuracies from Figures 5 and 15
FR NIC	VR NIC	VR NIC pr=0.8	VR NIC pr=0.95	FR NIC opt. MS-SSIM	JPEG2000
bpp = 1.23
clean	0.74	0.73	0.73	0.73	0.74	0.72
snow	0.33	0.33	0.33	0.32	0.32	0.30
frost	0.30	0.30	0.29	0.28	0.30	0.30
fog	0.36	0.36	0.36	0.36	0.41	0.45
brightness	0.67	0.67	0.66	0.66	0.67	0.65
contrast	0.36	0.35	0.34	0.36	0.40	0.44
motion blur	0.36	0.36	0.35	0.36	0.37	0.36
zoom blur	0.34	0.34	0.33	0.34	0.35	0.34
defocus blur	0.32	0.32	0.32	0.33	0.34	0.36
glass blur	0.17	0.17	0.16	0.16	0.16	0.17
elastic transform	0.50	0.50	0.49	0.49	0.50	0.52
jpeg compression	0.60	0.60	0.61	0.61	0.60	0.59
pixelate	0.53	0.53	0.54	0.54	0.58	0.51
gaussian noise	0.34	0.33	0.33	0.35	0.36	0.28
shot noise	0.31	0.30	0.31	0.33	0.32	0.25
impulse noise	0.32	0.31	0.32	0.34	0.35	0.23
bpp = 0.77
clean	0.72	0.71	0.71	0.71	0.71	0.69
snow	0.33	0.32	0.32	0.32	0.29	0.27
frost	0.29	0.29	0.28	0.28	0.27	0.29
fog	0.28	0.27	0.27	0.25	0.36	0.41
brightness	0.65	0.65	0.65	0.64	0.65	0.62
contrast	0.26	0.25	0.26	0.24	0.37	0.41
motion blur	0.35	0.35	0.34	0.34	0.36	0.35
zoom blur	0.33	0.33	0.32	0.32	0.35	0.33
defocus blur	0.30	0.29	0.29	0.29	0.33	0.35
glass blur	0.19	0.19	0.17	0.17	0.17	0.18
elastic transform	0.48	0.48	0.48	0.47	0.48	0.51
jpeg compression	0.60	0.60	0.60	0.60	0.59	0.58
pixelate	0.55	0.55	0.54	0.54	0.60	0.52
gaussian noise	0.33	0.32	0.32	0.34	0.34	0.31
shot noise	0.29	0.29	0.29	0.32	0.31	0.28
impulse noise	0.31	0.30	0.31	0.33	0.33	0.26
bpp = 0.24
clean	0.58	0.56	0.58	0.56	0.60	0.52
snow	0.22	0.22	0.22	0.21	0.18	0.14
frost	0.20	0.20	0.20	0.20	0.20	0.19
fog	0.08	0.06	0.06	0.05	0.18	0.21
brightness	0.51	0.49	0.50	0.49	0.53	0.43
contrast	0.07	0.05	0.05	0.05	0.20	0.25
motion blur	0.27	0.25	0.26	0.25	0.30	0.29
zoom blur	0.26	0.25	0.26	0.25	0.29	0.27
defocus blur	0.24	0.22	0.22	0.21	0.27	0.28
glass blur	0.23	0.21	0.19	0.19	0.24	0.24
elastic transform	0.39	0.37	0.37	0.36	0.42	0.42
jpeg compression	0.52	0.51	0.52	0.51	0.53	0.47
pixelate	0.49	0.47	0.48	0.47	0.54	0.47
gaussian noise	0.28	0.28	0.25	0.25	0.27	0.27
shot noise	0.28	0.27	0.25	0.25	0.28	0.26
impulse noise	0.29	0.28	0.25	0.25	0.26	0.26
Table 3: Classification accuracies from Figure 9
FR NIC	VR NIC	VR NIC pr=0.8	VR NIC pr=0.95	FR NIC opt. MS-SSIM	JPEG2000
PSNR=36.93
clean	0.72	0.71	0.73	0.73	0.74	0.72
snow	0.33	0.32	0.33	0.32	0.33	0.30
frost	0.29	0.29	0.29	0.28	0.30	0.30
fog	0.28	0.27	0.33	0.35	0.42	0.45
brightness	0.65	0.65	0.66	0.66	0.67	0.65
contrast	0.26	0.25	0.31	0.34	0.41	0.44
motion blur	0.35	0.35	0.35	0.36	0.37	0.36
zoom blur	0.33	0.33	0.33	0.34	0.35	0.34
defocus blur	0.30	0.29	0.30	0.32	0.35	0.36
glass blur	0.19	0.19	0.16	0.16	0.16	0.17
elastic transform	0.48	0.48	0.49	0.49	0.50	0.52
jpeg compression	0.60	0.60	0.61	0.61	0.60	0.59
pixelate	0.55	0.55	0.54	0.54	0.56	0.51
gaussian noise	0.33	0.32	0.33	0.35	0.34	0.28
shot noise	0.29	0.29	0.30	0.33	0.31	0.25
impulse noise	0.31	0.30	0.31	0.34	0.33	0.23
PSNR=32.97
clean	0.64	0.64	0.66	0.66	0.71	0.67
snow	0.27	0.27	0.28	0.29	0.28	0.25
frost	0.25	0.25	0.26	0.26	0.27	0.28
fog	0.13	0.12	0.13	0.14	0.35	0.38
brightness	0.57	0.57	0.59	0.60	0.64	0.59
contrast	0.11	0.10	0.12	0.13	0.36	0.38
motion blur	0.30	0.30	0.30	0.31	0.36	0.34
zoom blur	0.29	0.29	0.29	0.29	0.35	0.32
defocus blur	0.26	0.25	0.24	0.25	0.32	0.34
glass blur	0.22	0.21	0.18	0.19	0.18	0.19
elastic transform	0.42	0.43	0.43	0.43	0.48	0.50
jpeg compression	0.57	0.57	0.58	0.58	0.59	0.57
pixelate	0.53	0.52	0.53	0.53	0.60	0.53
gaussian noise	0.28	0.29	0.28	0.31	0.33	0.25
shot noise	0.26	0.26	0.26	0.28	0.31	0.23
impulse noise	0.29	0.29	0.28	0.30	0.33	0.17
PSNR=31.17
clean	0.58	0.58	0.61	0.62	0.67	0.62
snow	0.22	0.23	0.24	0.25	0.25	0.21
frost	0.20	0.22	0.22	0.23	0.25	0.25
fog	0.08	0.07	0.08	0.08	0.29	0.32
brightness	0.51	0.51	0.54	0.55	0.60	0.54
contrast	0.07	0.06	0.07	0.08	0.30	0.34
motion blur	0.27	0.26	0.28	0.28	0.34	0.33
zoom blur	0.26	0.26	0.27	0.27	0.33	0.31
defocus blur	0.24	0.23	0.23	0.23	0.30	0.32
glass blur	0.23	0.21	0.19	0.19	0.20	0.21
elastic transform	0.39	0.39	0.39	0.40	0.46	0.48
jpeg compression	0.52	0.53	0.55	0.55	0.57	0.55
pixelate	0.49	0.48	0.50	0.50	0.59	0.53
gaussian noise	0.28	0.28	0.25	0.26	0.30	0.21
shot noise	0.28	0.26	0.25	0.25	0.30	0.21
impulse noise	0.29	0.28	0.26	0.27	0.30	0.16
Table 4: Classification accuracies from Figure 9