SlowGuess/NeurIPS_2023_OCR huggingface dataset:
terms of bpp (notice the bpp range for shot noise in Figure 12 is 0-17 versus a typical range of 0-2). Recall that in Section 4.2.1 we observed that NIC models adjusted their bpp usage depending on the type of corruption-- in particular, NIC models used more bpps than JPEG2000 on the shot noise corruption, even though it did not improve the PSNR. Figure 12 shows that VTM adjusts its bpp usage in a more extreme way than NIC models, but, unlike NIC models, this increase in bpp does result in an increase in PSNR with respect to  $c(\mathcal{X})$ . As expected from our results in Section 4.2.1, this increase in PSNR with respect to  $c(\mathcal{X})$  results in a decrease of PSNR with respect  $\mathcal{X}$ . Overall, VTM's highly variable bpp usage could be a real disadvantage to using VTM in the wild: in particular, if a practitioner selects a hyper-parameter from clean data and expects the model will use a certain amount of bpps, VTM might use a much larger bpp than expected on OOD data. Thus, our OOD benchmark dataset is a vital aspect to comprehensive image compression testing.
Figure 13 shows the  $\mathcal{G}$  and  $\mathcal{R}$  metrics on the three representative shifts. On the snow corruption, VTM has a smaller magnitude of errors than JPEG2000 across all frequencies; however, both models leave relatively even errors across all frequencies and both models distort low frequencies more than high frequencies (which is the opposite as FR NIC on the snow corruption in Figure 4). At both severities, the two models have similar plots of  $\mathcal{G}$ . VTM is highly effective at reconstructing the glass blur—the magnitude of  $\mathcal{G}$  is  $< 0.0001$  across all frequencies—which matches our findings on the other codecs in Section 4.2.2. Additionally, JPEG2000 and VTM have nearly identical plots of  $\mathcal{R}$  at both severities. Finally, on the shot noise corruption, both JPEG2000 and VTM display similar magnitudes and patterns of errors. Both models leave significantly more error on  $\mathcal{G}$  at severity=5 than severity=1; however, JPEG2000 maintains a bpp=0.8 at both severities while VTM increases its bpp from 0.05 to 1.02. Additionally, unlike NIC, both of these models leave their highest errors in the low-frequency domain of  $\mathcal{G}$ . Lastly, both models are not as effective at denoising high-frequency signals as NIC proved to be.
E Additional results: ELIC
Figure 14: More detailed plots of  $\mathcal{D}$  for ELIC. Equivalent to the ELIC row on the right side of Figure 2 with smaller colorbar ranges.
Note that we select one hyper-parameter for JPEG2000 (q=15); however, we vary the hyper-parameter choice for VTM across corruptions (q=32, 17, 47 for snow, glass blur, and shot noise respectively) so that VTM gives bpps in the ballpark of 0.8 for each corruption.
F Additional results: accuracies by individual corruptions
Figure 15: Effect of compressing corrupt images on classification accuracy. Specifically, let  $A(X)$  be the top-1 accuracy of the model on dataset  $X$ , measured in percentage points. We report  $A(\mathcal{C}(c(X))) - A(c(X))$  over all -C corruptions with severity 3 (or on clean ImageNet in the case of "clean"). Each subfigure shows results under a different fixed-PSNR constraint based on the PSNRs achieved by the compressors on the clean ImageNet dataset. Appendix M has tables with these accuracies.
G Additional results: Kodak and Kodak-C
This section contains Figures equivalent to those in the main body for the Kodak and Kodak-C dataset.
Figure 16: Figure equivalent to Figure 1a for Kodak-C dataset.
Figure 17: Figure equivalent to Figure 2 for Kodak dataset
Figure 18: Figure equivalent to Figure 3 for Kodak-C dataset
Figure 19: Figure equivalent to Figure 4 for Kodak-C dataset
H Additional results: other CLIC-C corruptions
Figures 20, 21, and 22 show the results of the other 12 corruptions from CLIC-C which were not included in the main body. Overall, we see that our three representative corruptions (snow, glass blur, and shot noise) represent the trends of the low, medium and high frequency corruptions well.
Figure 20: Average PSD of the difference between the corrupted images and the clean images for each given CLIC-C corruption  $c$ ,  $\frac{1}{N} \sum_{k=1}^{N} \text{PSD}(c(X_k) - X_k)$ . Severity=5.
Figure 21: RD Curves for corruptions not seen in the main body. Rows 1, 3, and 5 show PSNR w.r.t.  $c(\mathcal{X})$ . Rows 2, 4, and 6 show PSNR w.r.t.  $\mathcal{X}$ .
Figure 22: Generalization error  $\mathcal{G}$  on the other corruptions of CLIC-C. We plot  $\mathcal{G}$  for each corruption  $c$  at severities 1 and 5. The plots of  $\mathcal{G}$  are labeled with tuples of the model's (bpp, PSNR w.r.t.  $c(\mathcal{X})$ ).
Figure 23: Robustness error  $\mathcal{R}$  on the other corruptions of CLIC-C. We plot  $\mathcal{R}$  for each corruption  $c$  at severities 1 and 5. The plots of  $\mathcal{G}$  are labeled with tuples of the model's (bpp, PSNR w.r.t.  $\mathcal{X}$ ).
I Training setup and hyperparameters
We train NIC models using the train split of the CLIC 2020 dataset with batches of size 8 and random crops of size  $256 \times 256$ . For SH NIC, we set  $N = M = 192$  and train for 5,000 epochs (about 1M iterations). We trained 8 models with  $\lambda s$  0.0012, 0.005, 0.01, 0.03, 0.05, 0.1, 0.15, and 0.26. We used the pytorch model architecture in the compressai repository [10]. For ELIC, we use  $N = 192$ ,  $M = 320$  and train for 3,900 epochs. We trained 11 models with unique  $\lambda s$  (0.001, 0.0025, 0.003, 0.004, 0.006, 0.008, 0.016, 0.025, 0.032, 0.05, 0.15). We utilized this repository for the model architecture, but trained our own models on CLIC rather than experimenting with the publicly available checkpoints trained on ImageNet. Note that we trained more models for ELIC than FR NIC in order to obtain bpps and PSNRs which fit our fixed-bpp/PSNR constraint.
For the variable-rate models, we augment each convolutional layer with a FiLM layer which consists of two fully-connected layers and 128 hidden features. We train VR models for 10,000 epochs (about 2M iterations) and sample  $\lambda$  from a log-uniform distribution over [0.0012, 0.26]. For pruned models, we use global gradual magnitude pruning (GMP) between epochs 675 and 2400 and tune the remaining weights from epochs 2401-10,000. Fixed-rate MS-SSIM models are trained over several values of  $\lambda \in [1,1000]$ .
J Variable-rate model details
For our variable-rate NIC we implemented a model proposed by Dosovitskiy and Djolonga [17] which uses Loss Conditional Training (LCT) to train a variable-rate version of the scale-hyperprior NIC model from [9]. The model is trained over a continuous range of compression rates, optimizing the rate-distortion tradeoff at each. This reduces the computational redundancy involved in training separate models for the same task at different points along the rate-distortion curve. Additionally, once trained, this model can be adaptively used at any compression rate in the range, which makes it even more flexible than a discrete set of fixed-rate models.
In order to condition the compression rate via LCT, all convolutional layers in the model are augmented with Feature-wise Linear Modulation (FiLM) layers [49]. In this case, these are small neural networks that take a conditioning parameter  $\lambda$  as input and output a  $\mu$  and  $\sigma$  used to modulate the activations channel-wise based on the value of  $\lambda$ . More specifically, suppose a layer in the CNN has activations  $f$  of size  $W\times H\times C$ . In Loss Conditional Training (LCT), these activations are augmented by  $\mu$  and  $\sigma$  as follows,
$$
\tilde {f} = \sigma \mathbf {f} + \mu \tag {1}
$$
where both  $\mu$  and  $\sigma$  are vectors of size  $C$ .
Figure 24 shows a diagram of FiLM layers being applied to a small example network. Normally, without FiLM layers, the equation to calculate the features maps is
$$
\mathbf {x} ^ {(1)} = \rho \left(W ^ {(1)} \mathbf {x} ^ {(0)} + \mathbf {b} ^ {(1)}\right), \tag {2}
$$
where  $\rho$  is the activation function (e.g., ReLU),  $W$  are the weights and  $b$  are the biases in the layer. However, with FiLM layers, the weighted sum is first subject to augmentation with  $\mu$  and  $\sigma$ . Thus, the new calculation is
$$
\mathbf {x} ^ {(1)} = \rho \left(\boldsymbol {\sigma} ^ {(1)} \left(W ^ {(1)} \mathbf {x} ^ {(0)} + \mathbf {b} ^ {(1)}\right)\right) + \boldsymbol {\mu} ^ {(1)}. \tag {3}
$$
During training, a random  $\lambda$  is drawn with each mini-batch and used in two ways: 1) as input to the FiLM layers to modulate the feature maps and 2) in the mini-batch's loss function. During inference, the model takes as input a desired compression rate  $\lambda$  along with the input image. Notably, LCT and FiLM layers only affect the values of the activations (i.e., feature maps), but not the weights on the convolutional layers or activation functions. This is an essential observation for applying pruning to a network with LCT.
K Pruned model details
One limitation of using NIC in the wild is the size of the NIC models. These models require several million learned weights and efficient inference requires specialized GPUs. This limits the possibility
Figure 24: Cartoon of FiLM applied to a multi-layer perceptron. We consider pruning both the base layer weights (blue) and the film layer weights (red).
of using NIC on an edge device, like a drone or a cell phone. In this section, we demonstrate that NIC models themselves can be approximated (i.e., we can compress the compressor) and we propose a sparse versions of both fixed-rate and variable-rate NIC models.
To prune our NIC models, we propose gradual magnitude pruning (GMP) [70] based approach which has been demonstrated to effectively sparsify DNNs. At a high level, GMP gradually prunes DNN weights based on their magnitude over a fixed number of epochs then continues to train, or fine-tune, the remaining sparse network after the target sparsity is reached. The pruning can be done layer-wise (i.e., the mask must be updated such that each layer has the target sparsity) or globally (i.e., the overall network must have the target sparsity, but the sparsities can differ between layers). We utilized global pruning, which has been demonstrated to produce more performant sparse models.
To implement GMP, each layer is given binary mask of the same size as the original layer, which specifies which weights have been pruned (i.e., set to 0). Every  $\Delta t$  iterations, the mask is updated based on the magnitudes of the weights, such that the network has a desired sparsity. Suppose  $s_i$  is the initial sparsity (usually 0) at time  $t_0$  and  $s_t$  is the target sparsity (0.8 or 0.95 in our case) which we want to reach after  $n$  steps, then the target sparsity at time step  $t$  is
$$
s _ {t} = s _ {f} + \left(s _ {i} - s _ {f}\right) \left(1 - \frac {t - t _ {0}}{n \Delta t}\right) ^ {3} \tag {4}
$$
$$
\text {f o r} t \in \left\{t _ {0}, t _ {0} + \Delta t, \dots , t _ {0} + n \Delta t \right\}. \tag {5}
$$
The target sparsity equation is found to be especially effective because it causes weights to be pruned more quickly while the learning rate is still high and the weight-training on non-pruned weights can recover. After the desired sparsity is reached, the sparse network is trained for several more iterations. The pruning can be done layer-wise (i.e., the mask must be updated such that each layer has the target sparsity) or globally (i.e., the overall network must have the target sparsity, but the sparsities can differ between layers). Note that the network still performs weight training on the non-pruned weights, as usual, using an optimizer like stochastic gradient descent.
L Mathematical details
L.1 Derivation of lemma 5.1
Let  $\mathcal{X} \in \mathbb{R}^{N \times p}$  be a dataset (whose datapoints are the rows) with mean 0 and covariance  $\Sigma = \frac{1}{N} \mathcal{X}^T \mathcal{X}$ . Consider the problem of linearly auto-encoding  $\mathcal{X}$  with latent space  $\mathbb{R}^r$ , where  $r \leq p$ :
$$
\min  _ {W _ {1} \in M (r, p), W _ {2} \in M (p, r)} \frac {1}{2} \left| \mathcal {X} W _ {1} ^ {T} W _ {2} ^ {T} - \mathcal {X} \right| ^ {2}. \tag {6}
$$
It is a theorem of Eckart and Young [18] that for any solution  $W_{1}, W_{2}$  of eq. (6) the product  $\mathcal{C} = W_{2}W_{1}$  is an orthogonal projection onto the subspace spanned by the top  $r$  principal components of  $\mathcal{X}$  (see also [7] and for more recent developments and additional references, [50, 35]). Informally, this theorem says "linear autoencoders perform PCA." Formally, if  $\mathcal{X} = USV^{T}$  is an SVD and  $V[:r]$  is the first  $r$  rows of  $V$ , then  $W_{2}W_{1} = V[:r]^{T}V[:r]$ . It follows that the reconstruction error  $\mathcal{L}(X)$  on a new input  $X \in \mathbb{R}^{p}$  admits a simple closed form: let  $V_{1}, \ldots, V_{p}$  be the principal components of  $X$  (rows of  $V$  above), and write
$$
X = \sum_ {i} c _ {i} V _ {i}; \text {t h e n} \mathcal {L} (X) = \sum_ {i > r} c _ {i} ^ {2}. \tag {7}
$$
Suppose now that  $\mathcal{X}$  is a dataset of natural images. We will replace the single index  $i$  for  $X = (X_{1},\ldots ,X_{p})\in \mathbb{R}^{p}$  with 3 indices  $h,i,j$  for an image tensor  $X = (X_{h,i,j})\in \mathbb{R}^{KHW}$  (here  $h$  is viewed as the channel index and  $i,j$  as the spatial indices); we will also commit mild abuse of notation by letting the matrices  $W_{i}$  act on flattened (vectorized) tensors. Let  $F:\mathbb{R}^{KHW}\to \mathbb{R}^{KHW}$  denote the spatial discrete Fourier transform. We introduce the notation
$$
\hat {\mathcal {X}} = \mathcal {X} F ^ {T} \text {a n d} \hat {\mathcal {C}} = F \mathcal {C} F ^ {T}, \tag {8}
$$
and we will frequently use the shorthand "hat" decoration  $\widehat{.}$  to denote  $F$  as well, for ease of notation. Just as  $\mathcal{C}$  projects onto the subspace spanned by the top  $r$  principal components of  $\mathcal{X}$ ,  $\hat{\mathcal{C}}$  projects onto the subspace spanned by the top  $r$  principal components of  $\hat{\mathcal{X}}$ .
It has long been known that when transformed to (spatial) Fourier frequency space, there are reasonable (coarse) descriptions of the statistical distribution of natural images. Its variances in spatial frequencies  $(i,j)$  can be modelled as  $\frac{1}{|i|^{\alpha} + |j|^{\beta}}$  where  $\alpha ,\beta \approx 2$  and its covariances drop off rapidly away from the diagonal [12, 1, 36]. These two facts motivate the following assumptions:
Assuming  $\alpha = \beta = 2$ , the above would suggest that  $\hat{C}$  projects a Fourier transformed image  $\hat{X} = FX$  onto Fourier basis vectors corresponding to spatial frequencies  $(i,j)$  with  $i^2 + j^2 \leq \frac{r}{\pi K}$ . That is,
$$
(\hat {\mathcal {C}} \hat {X}) _ {:, i j} \approx \left\{ \begin{array}{l l} \hat {X} _ {:, i j} & \text {i f} i ^ {2} + j ^ {2} \leq \frac {r}{\pi K} \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {9}
$$
where  $\hat{X}_{:,ij}$  denotes the components of  $\hat{X}$  corresponding to spatial frequency  $(i,j)$  (this is a vector of dimension  $K$ , the number of channels in the image dataset). Together with the identity  $FCX = \hat{C}\hat{X}$ , eq. (9) yields lemma 5.1.
Remark L.1. Note that lemma 5.1 also suggests that such autoencoder compressors behave qualitatively similarly to classical codecs such as JPEG, which involves a discrete cosine transform (DCT) followed by removal of high-frequency components [53]. While design of hand-crafted codecs like JPEG was no doubt driven by empirical observations of the statistics of natural images, the NIC models in our experiments (and at least in a theoretical sense the autoencoder model appearing in lemma 5.1) are machine learned from natural image datasets.
Below, we use the above analysis to get some qualitative insights into the robustness and generalization metrics  $\mathcal{R}$  and  $\mathcal{G}$  introduced in section 3. In what follows let  $c: \mathbb{R}^{KHW} \to \mathbb{R}^{KHW}$  be a corruption transformation.
L.2 Derivations of corollaries 5.2 and 5.3
Robustness: here we are interested in  $\operatorname{PSD}(X - \mathcal{C}(c(X)))$ , which is simply the coordinatewise absolute value of  $\hat{X} - \widehat{\mathcal{C}(c(X))}$ . Using the fact that in our simple model  $\mathcal{C}$  is linear, we can expand like
$$
\hat {X} - \widehat {\mathcal {C} (c (X))} = \hat {X} - \hat {\mathcal {C}} (\widehat {c (X)}) = \hat {X} - \hat {\mathcal {C}} (\hat {X}) + \hat {\mathcal {C}} (\widehat {c (X)} - \hat {X}) \tag {10}
$$
although it is in general not the case that  $W_{1} = V[:r]$ ,  $W_{2} = V[:r]^{T}$ , see [50, §G] for discussion.
Here we assume that our Fourier transform  $F$  has the property that frequency  $(0,0)$  corresponds to constant images.
so that for any frequency  $(i,j)$
$$
\begin{array}{l} \left| \left(\hat {X} - \widehat {\left(c (X)\right)}\right) _ {:, i j} \right| ^ {2} = \left| \left(\hat {X} - \hat {\mathcal {C}} (\hat {X})\right) _ {:, i j} \right| ^ {2} + 2 \overline {{\left(\hat {X} - \hat {\mathcal {C}} (\hat {X})\right) _ {: , i j}}} \hat {\mathcal {C}} \left(\widehat {c (X)} - \hat {X}\right) _ {:, i j} \\ + \left| \hat {\mathcal {C}} \left(\widehat {c (X)} - \hat {X}\right) _ {:, i j} \right| ^ {2} \tag {11} \\ \approx \left\{ \begin{array}{l l} | (\widehat {c (X)} - \hat {X}) _ {:, i j} | ^ {2} & \text {w h e n} i ^ {2} + j ^ {2} \leq \frac {r}{\pi C} \\ | \hat {X} _ {:, i j} | ^ {2}. & \text {o t h e r w i s e .} \end{array} \right. \\ \end{array}
$$
Taking square roots, we see that
$$
\operatorname {P S D} (X - \mathcal {C} (c (X))) _ {:, i j} \approx \left\{ \begin{array}{l l} | (\widehat {c (X)} - \hat {X}) _ {:, i j} | & \text {w h e n} i ^ {2} + j ^ {2} \leq \frac {r}{\pi C} \\ | \hat {X} _ {:, i j} | & \text {o t h e r w i s e ;} \end{array} \right. \tag {12}
$$
averaging over the dataset  $\mathcal{X}$  gives corollary 5.2. In particular, the cross term  $2(\hat{X} -\widehat{\mathcal{C}} (\hat{X}))_{:,ij}\widehat{\mathcal{C}} (\widehat{c(X)} -\hat{X})_{:,ij}$  is identically 0. This suggests that autoencoder compressors trained on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that  $|\widehat{(c(X) - \hat{X})}_{:,ij}|^2$  is large for small values of  $ij$ ).
Generalization: here we are interested in  $\operatorname{PSD}(c(X) - \mathcal{C}(c(X)))$ , which is simply the coordinatewise absolute value of  $\widehat{c(X)} - \widehat{\mathcal{C}(c(X))}$ . Again using the fact that in our simple model  $\mathcal{C}$  is linear, we can expand like
$$
\begin{array}{l} \widehat {c (X)} - \mathcal {C} (\widehat {c (X)}) = \hat {X} + (\widehat {c (X)} - \hat {X}) - \hat {\mathcal {C}} (\hat {X} + (\widehat {c (X)} - \hat {X})) \tag {13} \\ = \left(\hat {X} - \hat {\mathcal {C}} (\hat {X})\right) + \left((\widehat {c (X)} - \hat {X}) - \hat {\mathcal {C}} (\widehat {c (X)} - \hat {X})\right). \\ \end{array}
$$
From this we obtain
$$
\begin{array}{l} \left| \widehat {(c (X) - \mathcal {C} (c (X)))} _ {:, i j} \right| ^ {2} \\ = \left| \left(\hat {X} - \hat {\mathcal {C}} (\hat {X})\right) _ {:, i j} \right| ^ {2} + 2 \overline {{\left(\hat {X} - \hat {\mathcal {C}} (\hat {X})\right) _ {: , i j}}} \left(\left(\widehat {c (X)} - \hat {X}\right) - \widehat {\mathcal {C} \left(c (X) - \hat {X}\right)}\right) _ {:, i j} \\ + \left| \left(\widehat {c (X)} - \hat {X}\right) - \hat {\mathcal {C}} (\widehat {c (X)} - \hat {X})\right) _ {:, i j} | ^ {2} \tag {14} \\ \approx \left\{ \begin{array}{l l} 0 & \text {w h e n} i ^ {2} + j ^ {2} \leq \frac {r}{\pi C} \\ | \hat {X} _ {:, i j} | ^ {2} + 2 \overline {{\hat {X} _ {: , i j}}} (\widehat {c (X)} - \hat {X}) _ {:, i j} + | (\widehat {c (X)} - \hat {X}) _ {:, i j} | ^ {2} & \text {o t h e r w i s e .} \end{array} \right. \\ \end{array}
$$
Taking square roots and relating back to power spectral density, this says
$$
\begin{array}{l} \operatorname {P S D} (c (X) - \mathcal {C} (c (X))) _ {:, i j} \\ \approx \left\{ \begin{array}{l l} 0 & \text {w h e n} i ^ {2} + j ^ {2} \leq \frac {r}{\pi C} \\ \sqrt {\left| \hat {X} _ {: , i j} \right| ^ {2} + 2 \overline {{\hat {X} _ {: , i j}}} \left(\widehat {c (X)} - \hat {X}\right) _ {: , i j} + \left| \left(\widehat {c (X)} - \hat {X}\right) _ {: , i j} \right| ^ {2}} & \text {o t h e r w i s e} \end{array} \right. \tag {15} \\ \end{array}
$$
and averaging over the dataset  $\mathcal{X}$  gives corollary 5.3. Equation (14) is more involved than eq. (11) - in particular, the "cross term"  $2\widehat{X}_{:,ij}(c(X) - \hat{X})_{:,ij}$  is in general non-zero. However, there are many cases where at least an expectation over the data set  $\hat{\mathcal{X}}$  the term  $2\widehat{X}_{:,ij}(c(X) - \hat{X})_{:,ij}$  vanishes (for example, when  $c$  is additive noise, or more generally when  $\widehat{c(X)} - \hat{X}$  is statistically independent of  $\hat{X}$ ). In such cases, we can see that autoencoder compressors trained on natural images generalize less successfully to corruptions with large amplitude in high frequencies.
L.3 Autoencoder reconstruction error and data density more generally
Our findings on generalization (both the experimental results and analysis with linear autoencoders) is closely related to the widely observed inverse correlation between autoencoder reconstruction error and probability density. We will take the recent [29] as a jumping-off point. However, it is worth noting that this observation (or at least something similar to it) has been around for some time, at least since [5], and forms the basis for widespread use of autoencoders in anomaly detection.
Formally, [29, Observation 2] states that:
(*) For an autoencoder  $\mathcal{C}$  the reconstruction error  $|\mathcal{C}(X) - X|_2$  is positively correlated with  $\frac{1}{p(X)}$ , the probability density of the input data at  $X$ .
If we assume this observation holds, then some of our experimental findings regarding generalization to corrupted data admit a simple explanation: as discussed above, the probability density of natural images is heavily concentrated along low (spatial) Fourier frequencies. Hence for a corruption  $c$  such that  $c(X) - X$  is concentrated in high frequencies, it is reasonable to suspect that  $p(c(X)) < p(X)$ , and given the assumed positive correlation of  $\frac{1}{p(X)}$  with  $|\mathcal{C}(X) - X|_2$ , that
$$
\left| \mathcal {C} (c (X)) - c (X) \right| _ {2} > \left| \mathcal {C} (X) - X \right| _ {2}, \tag {16}
$$
i.e. the reconstruction error of the corrupted datapoint, related to our generalization metric, is higher than that of the clean datapoint  $X$ .
Remark L.2. In some sense,  $(\ast)$  is a simple consequence of learning via risk minimization. Let  $\mathcal{C}:M\to M$  be an autoencoder on a manifold  $M$  with finite volume (for example, the image hypercube  $[0,1]^{CHW}$ ). Let  $\ell (\mathcal{C}(X),X)$  be a reconstruction loss on a point  $X\in M$  and let  $p(X)$  be the probability density of input data on  $M$ , assumed to be positive everywhere (otherwise, discussing correlation with  $\frac{1}{p(X)}$  is troublesome). The risk in this situation is
$$
\mathcal {L} (\mathcal {C}) := \int_ {M} \ell (\mathcal {C} (X), X) p (X) d x \tag {17}
$$
Note that  $\ell (\mathcal{C}(X),X)$  and  $\frac{1}{p(X)}$  will be positively correlated provided that their covariance is positive. By definition this covariance is
$$
\begin{array}{l} \int_ {M} \ell (\mathcal {C} (X), X) \frac {1}{p (X)} p (X) d x - \int_ {M} \ell (\mathcal {C} (X), X) p (X) d x \cdot \int_ {M} \frac {1}{p (X)} p (X) d x \tag {18} \\ = \int_ {M} \ell (\mathcal {C} (X), X) d x - \mathcal {L} (\mathcal {C}) \cdot \operatorname {v o l} M. \\ \end{array}
$$
Since at present we only care about the sign of the covariance, we can divide by  $\mathrm{vol}M$  to obtain
$$
\int_ {M} \ell (\mathcal {C} (X), X) \frac {d x}{\operatorname {v o l} M} - \mathcal {L} (\mathcal {C}); \tag {19}
$$