SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yibo Yang<sup>1</sup> Stephan Eckstein<sup>2</sup> Marcel Nutz<sup>3</sup> Stephan Mandt<sup>1</sup>
<sup>1</sup>University of California, Irvine 2ETH Zurich 3Columbia University
{yibo.yang, mandt}@uci.edu
stephan.eckstein@math.ethz.ch
mnutz@columbia.edu
In the theory of lossy compression, the rate-distortion (R-D) function  $R(D)$  describes how much a data source can be compressed (in bit-rate) at any given level of fidelity (distortion). Obtaining  $R(D)$  for a given data source establishes the fundamental performance limit for all compression algorithms. We propose a new method to estimate  $R(D)$  from the perspective of optimal transport. Unlike the classic Blahut-Arimoto algorithm which fixes the support of the reproduction distribution in advance, our Wasserstein gradient descent algorithm learns the support of the optimal reproduction distribution by moving particles. We prove its local convergence and analyze the sample complexity of our R-D estimator based on a connection to entropic optimal transport. Experimentally, we obtain comparable or tighter bounds than state-of-the-art neural network methods on low-rate sources while requiring considerably less tuning and computation effort. We also highlight a connection to maximum-likelihood deconvolution and introduce a new class of sources that can be used as test cases with known solutions to the R-D problem.
The rate-distortion (R-D) function  $R(D)$  occupies a central place in the theory of lossy compression. For a given data source and a fidelity (or distortion) criterion,  $R(D)$  characterizes the minimum possible communication cost needed to reproduce a source sample within an error threshold of  $D$ , by any compression algorithm [Shannon, 1959]. A basic scientific and practical question is therefore establishing  $R(D)$  for any given data source of interest, which helps assess the (sub)optimality of the compression algorithms and guide their development. The classic algorithm by Blahut [1972] and Arimoto [1972] assumes a known discrete source and computes its  $R(D)$  by an exhaustive optimization procedure. This often has limited applicability in practice, and a line of research has sought to instead estimate  $R(D)$  from data samples [Harrison and Kontoyiannis, 2008, Gibson, 2017], with recent methods [Yang and Mandt, 2022, Lei et al., 2023a] inspired by deep generative models.
In this work, we propose a new approach to R-D estimation from the perspective of optimal transport. Our starting point is the formulation of the R-D problem as the minimization of a certain rate functional [Harrison and Kontoyiannis, 2008] over the space of probability measures on the reproduction alphabet. Optimization over such an infinite-dimensional space has long been studied under gradient flows [Ambrosio et al., 2008], and we consider a concrete algorithmic implementation based on moving particles in space. This formulation of the R-D problem also suggests connections to entropic optimal transport and non-parametric statistics, each offering us new insight into the solution of the R-D problem under a quadratic distortion. More specifically, our contributions are three-fold:
First, we introduce a neural-network-free  $R(D)$  upper bound estimator for continuous alphabets. We implement the estimator by Wasserstein gradient descent (WGD) over the space of reproduction
distributions. Experimentally, we found the method to converge much more quickly than state-of-the-art neural methods with hand-tuned architectures, while offering comparable or tighter bounds.
Second, we theoretically characterize convergence of our WGD algorithm and the sample complexity of our estimator. The latter draws on a connection between the R-D problem and that of minimizing an entropic optimal transport (EOT) cost relative to the source measure, allowing us to turn statistical bounds for EOT [Mena and Niles-Weed, 2019] into finite-sample bounds for R-D estimation.
Finally, we introduce a new, rich class of sources with known ground truth, including Gaussian mixtures, as a benchmark for algorithms. While the literature relies on the Gaussian or Bernoulli for this purpose, we use the connection with maximum likelihood deconvolution to show that a Gaussian convolution of any distribution can serve as a source with a known solution to the R-D problem.
2 Lossy compression, entropic optimal transport, and MLE
This section introduces the R-D problem and its rich connections to entropic optimal transport and statistics, along with new insights into its solution. Sec. 2.1 sets the stage for our method (Sec. 4) by a known formulation of the standard R-D problem as an optimization problem over a space of probability measures. Sec. 2.2 discusses the equivalence between the R-D problem and a projection of the source distribution under entropic optimal transport; this is a key to our sample complexity results in Sec. 4.3. Lastly, Sec 2.3 gives a statistical interpretation of R-D as maximum-likelihood deconvolution and uses it to analytically derive a segment of the R-D curve for a new class of sources under quadratic distortion; this allows us to assess the optimality of algorithms in experiment Sec. 5.1.
2.1 Setup
For a memoryless data source  $X$  with distribution  $P_{X}$ , its rate-distortion (R-D) function describes the minimum possible number of bits per sample needed to reproduce the source within a prescribed distortion threshold  $D$ . Let the source and reproduction take values in two sets  $\mathcal{X}$  and  $\mathcal{Y}$ , known as the source and reproduction alphabets, and let  $\rho : (\mathcal{X}, \mathcal{Y}) \to [0, \infty)$  be a given distortion function. The R-D function is defined by the following optimization problem [Polyanskiy and Wu, 2022],
$$
R (D) = \inf  _ {Q _ {Y | X}: \mathbb {E} _ {P _ {X}} Q _ {Y | X} [ \rho (X, Y) ] \leq D} I (X; Y), \tag {1}
$$
where  $Q_{Y|X}$  is any Markov kernel from  $\mathcal{X}$  to  $\mathcal{Y}$  conceptually associated with a (possibly) stochastic compression algorithm, and  $I(X;Y)$  is the mutual information of the joint distribution  $P_{X}Q_{Y|X}$ .
For ease of presentation, we now switch to a more abstract notation without reference to random variables. We provide the precise definitions in the Supplementary Material. Let  $\mathcal{X}$  and  $\mathcal{Y}$  be standard Borel spaces; let  $\mu \in \mathcal{P}(\mathcal{X})$  be a fixed probability measure on  $\mathcal{X}$ , which should be thought of as the source distribution  $P_{X}$ . For a measure  $\pi$  on the product space  $\mathcal{X} \times \mathcal{Y}$ , the notation  $\pi_1$  (or  $\pi_2$ ) denotes the first (or second) marginal of  $\pi$ . For any  $\nu \in \mathcal{P}(\mathcal{Y})$ , we denote by  $\Pi(\mu, \nu)$  the set of couplings between  $\mu$  and  $\nu$  (i.e.,  $\pi_1 = \mu$  and  $\pi_2 = \nu$ ). Similarly,  $\Pi(\mu, \cdot)$  denotes the set of measures  $\pi$  with  $\pi_1 = \mu$ . Throughout the paper,  $K$  denotes a transition kernel (conditional distribution) from  $\mathcal{X}$  to  $\mathcal{Y}$ , and  $\mu \otimes K$  denotes the product measure formed by  $\mu$  and  $K$ . Then  $R(D)$  is equivalent to
$$
R (D) = \inf  _ {K: \int \rho d (\mu \otimes K) \leq D} H (\mu \otimes K | \mu \otimes (\mu \otimes K) _ {2}) = \inf  _ {\pi \in \Pi (\mu , \cdot): \int \rho d \pi \leq D} H (\pi | \pi_ {1} \otimes \pi_ {2}), \tag {2}
$$
where  $H$  denotes relative entropy, i.e., for two measures  $\alpha, \beta$  defined on a common measurable space,  $H(\alpha|\beta) \coloneqq \int \log\left(\frac{d\alpha}{d\beta}\right)d\alpha$  when  $\alpha$  is absolutely continuous w.r.t  $\beta$ , and infinite otherwise.
To make the problem more tractable, we follow the approach of the classic Blahut-Arimoto algorithm [Blahut, 1972, Arimoto, 1972] (to be discussed in Sec. 3.1) and work with an equivalent unconstrained Lagrangian problem as follows. Instead of parameterizing the R-D function via a distortion threshold  $D$ , we parameterize it via a Lagrange multiplier  $\lambda \geq 0$ . For each fixed  $\lambda$  (usually selected from a predefined grid), we aim to solve the following optimization problem,
$$
F _ {\lambda} (\mu) := \inf  _ {\nu \in \mathcal {P} (\mathcal {Y})} \inf  _ {\pi \in \Pi (\mu , \cdot)} \lambda \int \rho d \pi + H (\pi | \mu \otimes \nu). \tag {3}
$$
Geometrically,  $F_{\lambda}(\mu) \in \mathbb{R}$  is the y-axis intercept of a tangent line to the  $R(D)$  with slope  $-\lambda$ , and  $R(D)$  is determined by the convex envelope of all such tangent lines [Gray, 2011]. To simplify notation, we often drop the dependence on  $\lambda$  (e.g., we write  $F(\mu) = F_{\lambda}(\mu)$ ) whenever it is harmless.
To set the stage for our later developments, we write the unconstrained R-D problem as
$$
F _ {\lambda} (\mu) = \inf  _ {\nu \in \mathcal {P} (\mathcal {Y})} \mathcal {L} _ {B A} (\mu , \nu), \tag {4}
$$
$$
\mathcal {L} _ {B A} (\mu , \nu) := \inf  _ {\pi \in \Pi (\mu , \cdot)} \lambda \int \rho d \pi + H (\pi | \mu \otimes \nu) = \inf  _ {K} \lambda \int \rho d (\mu \otimes K) + H (\mu \otimes K | \mu \otimes \nu), \tag {5}
$$
where we refer to the optimization objective  $\mathcal{L}_{BA}$  as the rate function [Harrison and Kontoyiannis, 2008]. We abuse the notation to write  $\mathcal{L}_{BA}(\nu)\coloneqq \mathcal{L}_{BA}(\mu ,\nu)$  when it is viewed as a function of  $\nu$  only, and refer to it as the rate functional. The rate function characterizes a generalized Asymptotic Equipartition Property, where  $\mathcal{L}_{BA}(\mu ,\nu)$  is the asymptotically optimal cost of lossy compression of data  $X\sim \mu$  using a random codebook constructed from samples of  $\nu$  [Dembo and Kontoyiannis, 2002]. Notably, the optimization in (5) can be solved analytically [Csiszar, 1974a, Lemma 1.3], and  $\mathcal{L}_{BA}$  simplifies to
$$
\mathcal {L} _ {B A} (\mu , \nu) = \int_ {\mathcal {X}} - \log \left(\int_ {\mathcal {Y}} e ^ {- \lambda \rho (x, y)} \nu (d y)\right) \mu (d x). \tag {6}
$$
In practice, the source  $\mu$  is only accessible via independent samples, on the basis of which we propose to estimate its  $R(D)$ , or equivalently  $F(\mu)$ . Let  $\mu^m$  denote an  $m$ -sample empirical measure of  $\mu$ , i.e.,  $\mu^m = \sum_{i=1}^{m} \delta_{x_i}$  with  $x_1, \ldots, n$  being independent samples from  $\mu$ , which should be thought of as the "training data". Following Harrison and Kontoyiannis [2008], we consider two kinds of (plug-in) estimators for  $F(\mu)$ : (1) the non-parametric estimator  $F(\mu^m)$ , and (2) the parametric estimator  $F^{\mathcal{H}}(\mu^m) := \inf_{\nu \in \mathcal{H}} \mathcal{L}_{BA}(\mu^m, \nu)$ , where  $\mathcal{H}$  is a family of probability measures on  $\mathcal{V}$ . Harrison and Kontoyiannis [2008] showed that under rather broad conditions, both kinds of estimators are strongly consistent, i.e.,  $F(\mu^m)$  converges to  $F(\mu)$  (and respectively,  $F^{\mathcal{H}}(\mu^m)$  to  $F^{\mathcal{H}}(\mu)$ ) with probability one as  $m \to \infty$ . Our algorithm will implement the parametric estimator  $F^{\mathcal{H}}(\mu^m)$  with  $\mathcal{H}$  chosen to be the set of probability measures with finite support, and we will develop finite-sample convergence results for both kinds of estimators in the continuous setting (Proposition 4.3).
2.2 Connection to entropic optimal transport
The R-D problem turns out to have a close connection to entropic optimal transport (EOT) [Peyre and Cuturi, 2019], which we will exploit in Sec. 4.3 to obtain sample complexity results under our approach. For  $\epsilon > 0$ , the entropy-regularized optimal transport problem is given by
$$
\mathcal {L} _ {E O T} (\mu , \nu) := \inf  _ {\pi \in \Pi (\mu , \nu)} \int \rho d \pi + \epsilon H (\pi | \mu \otimes \nu). \tag {7}
$$
We now consider the problem of projecting  $\mu$  onto  $\mathcal{P}(\mathcal{V})$  under the cost  $\mathcal{L}_{EOT}$ :
$$
\inf  _ {\nu \in \mathcal {P} (\mathcal {Y})} \mathcal {L} _ {E O T} (\mu , \nu). \tag {8}
$$
In the OT literature this is known as the (regularized) Kantorovich estimator [Bassetti et al., 2006] for  $\mu$ , and can also be viewed as a Wasserstein barycenter problem [Agueh and Carlier, 2011].
With the identification  $\epsilon = \lambda^{-1}$ , problem (8) is in fact equivalent to the R-D problem (4): compared to  $\mathcal{L}_{BA}$  (5), the extra constraint on the second marginal of  $\pi$  in  $\mathcal{L}_{EOT}$  (7) is redundant at the optimal  $\nu$ . More precisely, Lemma 7.1 shows that (we omit the notational dependence on  $\mu$  when it is fixed):
$$
\inf  _ {\nu \in \mathcal {P} (\mathcal {Y})} \mathcal {L} _ {E O T} (\nu) = \inf  _ {\nu \in \mathcal {P} (\mathcal {Y})} \lambda^ {- 1} \mathcal {L} _ {B A} (\nu) \quad \text {a n d} \quad \underset {\nu \in \mathcal {P} (\mathcal {Y})} {\arg \min } \mathcal {L} _ {E O T} (\nu) = \underset {\nu \in \mathcal {P} (\mathcal {Y})} {\arg \min } \mathcal {L} _ {B A} (\nu). \tag {9}
$$
Existence of a minimizer holds under mild conditions, for instance if  $\mathcal{X} = \mathcal{Y} = \mathbb{R}^d$  and  $\rho (x,y)$  is a coercive lower semicontinuous function of  $y - x$  [Csiszár, 1974a, p.66].
2.3 Connection to maximum-likelihood deconvolution
The connection between R-D and maximum-likelihood estimation has been observed in the information theory, machine learning and compression literature [Harrison and Kontoyiannis, 2008, Alemi et al., 2018, Balle et al., 2017, Theis et al., 2017, Yang et al., 2020, Yang and Mandt, 2022]. Here, we bring attention to a basic equivalence between the R-D problem and maximum-likelihood deconvolution, where the connection is particularly natural under a quadratic distortion function. Also see [Rigollet and Weed, 2018] for a related discussion that inspired ours and extension to a non-quadratic distortion. We provide further insight from the view of variational learning and inference in Section 10.
Maximum-likelihood deconvolution is a classical problem of non-parametric statistics and mixture models [Carroll and Hall, 1988, Lindsay and Roeder, 1993]. The deconvolution problem is concerned with estimating an unknown distribution  $\alpha$  from noise-corrupted observations  $X_{1}, X_{2}, \ldots$ , where for each  $i \in \mathbb{N}$ , we have  $X_{i} = Y_{i} + N_{i}$ ,  $Y_{i} \stackrel{i.i.d.}{\sim} \alpha$ , and  $N_{i}$  are i.i.d. independent noise variables with a known distribution. For concreteness, suppose all variables are  $\mathbb{R}^d$  valued and the noise distribution is  $\mathcal{N}(0, \sigma^2 \mathrm{I}_d)$  with Lebesgue density  $\phi_{\sigma^2}$ . Denote the distribution of the observations  $X_{i}$  by  $\mu$ . Then  $\mu$  has a Lebesgue density given by the convolution  $\alpha * \phi_{\sigma^2}(x) \coloneqq \int \phi_{\sigma^2}(x - y) \alpha(dy)$ . Here, we consider the population-level (instead of the usual sample-based) maximum-likelihood estimator (MLE) for  $\alpha$ :
$$
\nu^ {*} = \underset {\nu \in \mathcal {P} (\mathbb {R} ^ {d})} {\arg \max } \int \log \left(\nu * \phi_ {\sigma^ {2}} (x)\right) \mu (d x), \tag {10}
$$
and observe that  $\nu^{*} = \alpha$ . Plugging in the density  $\phi_{\sigma^2}(x) \propto e^{-\frac{1}{2\sigma^2}\| x\|^2}$ , we see that the MLE problem (10) is equivalent to the R-D problem (4) with  $\rho(x,y) = \frac{1}{2}\| x - y\|^2, \lambda = \frac{1}{\sigma^2}$ , and  $\mathcal{L}_{BA}$  given by (6) in the form of a marginal log-likelihood. Thus the R-D problem has the interpretation of estimating a distribution from its noisy observations given through  $\mu$ , assuming a Gaussian noise with variance  $\frac{1}{\lambda}$ .
Figure 1: The  $R(D)$  of a Gaussian mixture source, and the estimated optimal reproduction distributions  $\nu^{*}$  (in bar plots) at varying R-D trade-offs. For any  $\lambda \in [\sigma^{-2},\infty)$ , the corresponding  $R(D)$  (yellow segment) is known analytically as is the optimal reproduction distribution  $\nu^{*}$  (whose density is plotted in gray). For  $\lambda \in (0,\sigma^{-2}]$ ,  $\nu^{*}$  becomes singular and concentrated on two points, collapsing to the source mean as  $\lambda \rightarrow 0$ .
This connection suggests analytical solutions to the R-D problem for a variety of sources that arise from convolving an underlying distribution with Gaussian noise. Consider an R-D problem (4) with  $\mathcal{X} = \mathcal{Y} = \mathbb{R}^d$ ,  $\rho(x, y) = \frac{1}{2}\|x - y\|^2$ , and let the source  $\mu$  be the convolution between an arbitrary measure  $\alpha \in \mathcal{P}(\mathcal{Y})$  and Gaussian noise with known variance  $\sigma^2$ . E.g., using a discrete measure for  $\alpha$  results in a Gaussian mixture source with equal covariance among its components. When  $\lambda = \frac{1}{\sigma^2}$ , we recover exactly the population-MLE problem (10) discussed earlier, which has the solution  $\nu^* = \alpha$ . While this allows us to obtain one point of  $R(D)$ , we can in fact extend this idea to any  $\lambda \geq \frac{1}{\sigma^2}$  and obtain the analytical form for the corresponding segment of the R-D curve. Specifically, for any  $\lambda \geq \frac{1}{\sigma^2}$ , applying the summation rule for independent Gaussians reveals the source distribution  $\mu$  as
$$
\mu = \alpha * \mathcal {N} (0, \sigma^ {2}) = \alpha * \mathcal {N} (0, \sigma^ {2} - \frac {1}{\lambda}) * \mathcal {N} (0, \frac {1}{\lambda}) = \alpha_ {\lambda} * \mathcal {N} (0, \frac {1}{\lambda}), \quad \alpha_ {\lambda} := \alpha * \mathcal {N} (0, \sigma^ {2} - \frac {1}{\lambda}),
$$
i.e., as the convolution between another underlying distribution  $\alpha_{\lambda}$  and independent noise with variance  $\frac{1}{\lambda}$ . A solution to the R-D problem (3) is then analogously given by  $\nu^{*} = \alpha_{\lambda}$ , with the corresponding optimal coupling given by  $\nu^{*} \otimes \tilde{K}, \tilde{K}(y, dx) = \mathcal{N}(y, \frac{1}{\lambda})$ . Evaluating the distortion and mutual information of the coupling then yields the  $R(D)$  point associated with  $\lambda$ . Fig. 1 illustrates the  $R(D)$  of a toy Gaussian mixture source, along with the  $\nu^{*}$  estimated by our proposed WGD algorithm (Sec. 4); note that  $\nu^{*}$  transitions from continuous (a Gaussian mixture with smaller component variances) to singular (a mixture of two Dirac's) at  $\lambda = \sigma^{-2}$ . See caption for more details.
3.1 Blahut-Arimoto
The Blahut-Arimoto (BA) algorithm [Blahut, 1972, Arimoto, 1972] is the default method for computing  $R(D)$  for a known and discrete case. For a fixed  $\lambda$ , BA carries out the optimization problem (3) via coordinate ascent. Starting from an initial measure  $\nu^{(0)} \in \mathcal{P}(\mathcal{Y})$ , the BA algorithm at step  $t$  computes an updated pair  $(\nu^{(t + 1)}, K^{(t + 1)})$  as follows
$$
\frac {d K ^ {(t + 1)} (x , \cdot)}{d \nu^ {(t)}} (y) = \frac {e ^ {- \lambda \rho (x , y)}}{\int e ^ {- \lambda \rho (x , y ^ {\prime})} \nu^ {(t)} \left(d y ^ {\prime}\right)}, \quad \forall x \in \mathcal {X}, \tag {11}
$$
$$
\nu^ {(t + 1)} = (\mu \otimes K ^ {(t + 1)}) _ {2}. \tag {12}
$$
Here  $\tilde{K}$  maps from the reproduction to the source alphabet, opposite to the kernel  $K$  elsewhere in the text.
When the alphabets are finite, the above computation can be carried out in matrix and vector operations, and the resulting sequence  $\{(\nu^{(t)},K^{(t)})\}_{t = 1}^{\infty}$  can be shown to converge to an optimum of (3); cf. [Csiszár, 1974b, Csiszár, 1984]. When the alphabets are not finite, e.g.,  $\mathcal{X} = \mathcal{Y} = \mathbb{R}^d$ , the BA algorithm no longer applies, as it is unclear how to digitally represent the measure  $\nu$  and kernel  $K$  and to tractably perform the integrals required by the algorithm. The common workaround is to perform a discretization step and then apply BA on the resulting discrete problem.
One standard discretization method is to tile up the alphabets with small bins [Gray and Neuhoff, 1998]. This quickly becomes infeasible as the number of dimensions increases. We therefore consider discretizing the data space  $\mathcal{X}$  to be the support of training data distribution  $\mu^m$ , i.e., the discretized alphabet is the set of training samples; this can be justified by the consistency of the parametric R-D estimator  $F^{\mathcal{H}}(\mu^m)$  [Harrison and Kontoyiannis, 2008]. It is less clear how to discretize the reproduction space  $\mathcal{V}$ , especially in high dimensions. Since we work with  $\mathcal{X} = \mathcal{Y}$ , we will discretize  $\mathcal{V}$  similarly and use an  $n$ -element random subset of the training samples, as also considered by Lei et al. [2023a]. As we will show, this rather arbitrary placement of the support of  $\nu$  results in poor performance, and can be significantly improved from our perspective of evolving particles.
3.2 Neural network-based methods for estimating  $R(D)$
RD-VAE ([Yang and Mandt, 2022]): To overcome the limitations of the BA algorithm, Yang and Mandt [2022] proposed to parameterize the transition kernel  $K$  and reproduction distribution  $\nu$  of the BA algorithm by neural density estimators [Papamakarios et al., 2021], and optimize the same objective (3) by (stochastic) gradient descent. They estimate (3) by Monte Carlo using joint samples  $(X_{i},Y_{i})\sim \mu \otimes K$ ; in particular, the relative entropy can be written as  $H(\mu \otimes K|\mu \otimes \nu) = \int \int \log \left(\frac{dK(x,\cdot)}{d\nu} (y)\right)K(x,dy)\mu (dx)$ , where the integrand is computed exactly via a density ratio. In practice, an alternative parameterization is often used where the neural density estimators are defined on a lower dimensional latent space than the reproduction alphabet, and the resulting approach is closely related to VAEs [Kingma and Welling, 2013]. Yang and Mandt [2022] additionally propose a neural estimator for a lower bound on  $R(D)$ , based on a dual representation due to Csiszar [1974a]. NERD [Lei et al., 2023a]: Instead of working with the transition kernel  $K$  as in the RD-VAE, Lei et al. [2023a] considered optimizing the form of the rate functional in (6), via gradient descent on the parameters of  $\nu$  parameterized by a neural network. Let  $\nu^{\mathcal{Z}}$  be a base distribution over  $\mathcal{Z} = \mathbb{R}^{K}$ , such as the standard Gaussian, and  $\omega : \mathcal{Z}\to \mathcal{V}$  be a decoder network. The variational measure  $\nu$  is then modeled as the image measure of  $\nu^{\mathcal{Z}}$  under  $\omega$ . To evaluate and optimize the objective (6), the intractable inner integral w.r.t.  $\nu$  is replaced with a plug-in estimator, so that for a given  $x\in \mathcal{X}$
$$
- \log \left(\int_ {\mathcal {Y}} e ^ {- \lambda \rho (x, y)} \nu (d y)\right) \approx - \log \left(\frac {1}{n} \sum_ {j = 1} ^ {n} e ^ {- \lambda \rho (x, Y _ {j})}\right), \quad Y _ {j} \sim \nu , j = 1, 2, \dots , n. \tag {13}
$$
After training, we estimate an R-D upper bound using  $n$  samples from  $\nu$  (to be discussed in Sec. 4.4).
3.3 Other related work