SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sadaf Salehkalaibar*
sadafs@ece.utoronto.ca
Buu Phan*
truong.phan@mail.utoronto.ca
Jun Chen
chenjun@mcmaster.ca
Wei Yu
weiyu@ece.utoronto.ca
Ashish Khisti
akhisti@ece.utoronto.ca
We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss to target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD, considers the joint distribution (JD) of all the video frames up to the current one, while the second metric, PLF-FMD, considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have a significant effect on the reconstruction, especially at low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion compared to PLF-FMD and further makes it more difficult to recover from errors made in the earlier output frames. Although the choice of PLF decisively affects reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. In particular, encoded representations generated by training a system to minimize the MSE (without requiring either PLF) can be near universal and can generate close to optimal reconstructions for either choice of PLF at the decoder. We validate our results using (one-shot) information-theoretic analysis, detailed study of the rate-distortion-perception tradeoff of the Gauss-Markov source model as well as deep-learning based experiments on moving MNIST and KTH datasets. Code will be available at https://github.com/truongbuu/URDP_flow.
There is an increasing demand for video compression algorithms that are able to generate visually pleasing videos at low bitrates. Most of the current video codecs use distortion measures such as PSNR [1-4], MSE and MS-SSIM [3-5] to generate reconstructions which tend to be blurry at extremely low bitrates. In recent years, there has been a growing interest (see e.g., [6-10]) in using deep generative models to make the reconstructions look more realistic. Such techniques introduce an additional perception loss function that measures a distance between distributions of the source and reconstruction, with perfect perception corresponding requiring that the two distributions be identical.
(a) Encoders and decoders for  $T = 3$  frames.
(b) Effects of different PLFs on reconstructions for Moving MNIST and KTH datasets (best view in the monitor).
(c) Error permanence on the UVG dataset. The PLF-JD reconstructions propagate the flaws in the color tone from the previous I-frame reconstruction while the decoder based on PLF-FMD is able to fix these flaws.
Figure 1: (a) Proposed System Model (b, c) Error permanence phenomenon under different PLFs. High fidelity but incorrect I-frame reconstruction propagates the error to subsequent P-frames in 0-PLF-JD reconstructions. The MMSE and 0-PLF-FMD reconstructions do not have this problem.
In compression systems, improving realism comes at the price of increasing distortion. The work of Blau and Michaeli [11] establishes the theoretical rate-distortion-perception (RDP) tradeoff which has also been validated in [12-15]. Furthermore universal encoded representations were proposed in [16] where the representation is fixed at the encoder and the decoder is adapted to achieve a performance near the optimal RDP tradeoff curve. The extension of these works to video compression involves many challenges. First, the compression system must not only account for spatial redundancy as in image compression, but also exploit the temporal redundancy across video frames, making the system design more complex. Secondly, unlike the case of image compression, there may be no clear choice of the perception loss function (PLF). Indeed, some prior works [7] consider PLF that preserves framewise marginal distribution (PLF-FMD) between the source and reconstruction, while other works consider joint distribution (PLF-JD) across multiple frames [9].
As illustrated in Fig. 1a, we study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss and either a PLF-JD or PLF-FMD metric for perception loss. Our contributions are as follows:
- Differences in reconstruction quality based on the choice of PLF: We demonstrate that the choice of PLF can decisively affect the reconstruction quality especially in the low bit-rate regime. We approximately characterize the operational RDP region on a per-frame basis for a first-order Markov source model and analyze the special case of Gauss-Markov sources in detail. We show that there is a significant penalty in distortion when using PLF-JD in the low-rate regime. On the experimental side, we demonstrate that while PLF-JD preserves better temporal consistency across video frames, it suffers from the permanence of error phenomenon in which the mistakes in reconstructions propagate to future frames. On the other hand, the PLF-FMD metric shows
more capability in correcting mistakes across frames (see Fig. 1b for visualizations involving three-frame videos). On the other hand, if the first frame is transmitted at high bit-rate, we demonstrate that PLF-JD performs better than PLF-FMD.
- **Universality of minimum mean square error (MMSE) reconstructions:** We demonstrate that encoded representations generated from an encoder trained to minimize MSE reconstruction (without considering any PLF) suffice to produce close-to-optimal reconstructions for either choice of PLF. For general sources, we show that when using PLF-FMD, the MMSE reconstruction can be transformed to a reconstruction satisfying perfect perceptual quality by increasing the distortion at most by a factor of two. While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. For the Gauss-Markov source model we demonstrate exact universality i.e., encoded representation for the MMSE reconstruction can be adapted to achieve any other reconstruction in the RDP region. We also use deep learning based experiments to provide experimental evidence of these results.
We note that the above notion of universal encoded representations based on MSE reconstruction can have significant advantages in practice. First, although the reconstructions associated with different choices of PLFs can be visually very different, universal representations delegate the choice of PLF to the decoder rather than requiring the encoder to commit to a specific PLF. Secondly, the universality of MMSE reconstructions is far more significant in the context of learned video compression. Given that perceptual reconstruction frequently generates novel details not present in the source frame, compressing motion flow vectors between the current frame and the prior perceptual reconstruction necessitates a higher bit allocation compared to utilizing the MMSE reconstruction. Hence, a recommended approach is to train end-to-end compression exclusively to minimize MSE and use our proposed scheme to achieve a (near-optimal) tradeoff between the distortion and perception losses as desired by the user.
The study of RDP region for learned video compression is considerably more challenging than the study of RDP function for a single frame in prior works ([11, 16]). This is because of the fact that the RDP region (for first-order Markov sources) involves a tradeoff between the compression rate assigned to each frame imposing a Markov structure on the reconstructions. For Gaussian sources, the proof of optimality of Gaussian reconstructions does not use the closed form of the RDP region due to its complexity. As a result, the study of RDP region for various operating regimes is quite more involved. Furthermore, for the proof of universality, one has to consider the achievability of the entire RDP region as opposed to just points on the boundary of RDP function in [16]. Finally the results on the fixed-encoder setup are more general than prior works ([11, 16]) that required a characterization of the information RDP region, which we do not require.
Perceptual Lossy Video Compression. Distribution preserving framework using Generative Adversarial Networks (GAN) has been widely adopted as a surrogate metric for perceptual quality in image [12, 17-19] and, recently, video compression [6-9]. Unlike image compression, where the choice of PLF is straightforward, there is currently no agreed-upon objective for lossy video compression. For instance, DVC-P [6] employs PLF-FMD to improve the visual quality per frame, ignoring the temporal coherence. Similarly, Mentzer et al. [7] utilize the per-frame metric with a conditional GAN model, and found no significant differences when using a GAN objective with multiple frames. Other works target temporal consistency by incorporating multiple frames in their GAN objective. This includes the work by Yang et al. [8], where every two consecutive frames are included, and by Veerabadran et al. [9], where they employ the PLF-JD metric in a non-causal setting. Unlike previous works, we study the impact of two different perception objectives, i.e. PLF-JD and PLF-FMD, on reconstructions in the causal setting, presenting theoretical properties that are verified by deep learning experiments. For the PLF-JD metric, we demonstrate the error permanence phenomenon, which, unlike the error propagation issue [18, 20], cannot be resolved by increasing the code rate assigned to the  $P$  frames.
RDP Tradeoff and Principle of Universality. Targeting the distribution preserving framework in lossy image compression, several theoretical works have shown the presence of RDP tradeoff [11, 21-23], where perfect perception comes at a cost of increasing distortion by at most a factor of 2. Furthermore, an encoder that generates universal representations exists [16, 24, 25], which enables the decoder to freely choose the level of distortion-perception tradeoff it desires. Our work explores these avenues
in the context of causal video compression. As discussed previously we show that the MMSE representation can be used as a universal representation for both perception metrics which has several advantages in the context of video compression..
Let  $(X_{1},\ldots ,X_{T})\in \mathcal{X}_{1}\times \ldots \times \mathcal{X}_{T}$  be  $T$  frames in a video (with each  $\mathcal{X}_i\subseteq \mathbb{R}^d$ ) distributed according to  $P_{X_1\dots X_T}$ . The frames are available for encoding sequentially;  $X_{1}$  is available first, then  $X_{2}$  arrives, followed by  $X_{3}$  and so on. There is a shared randomness  $K\in \mathcal{K}$  which is available at all encoders and decoders. The following (possibly stochastic) mappings define the encoding and decoding functions:
$$
f _ {j} \colon \mathcal {X} _ {1} \times \dots \times \mathcal {X} _ {j} \times \mathcal {K} \rightarrow \mathcal {M} _ {j}, \quad j = 1, \dots , T, \tag {1}
$$
$$
g _ {j} \colon \mathcal {M} _ {1} \times \mathcal {M} _ {2} \times \dots \times \mathcal {M} _ {j} \times \mathcal {K} \rightarrow \tilde {\mathcal {X}} _ {j}, \tag {2}
$$
where  $\mathcal{M}_j\in \{0,1\}^\star$  denotes the set of (variable-length) messages assigned by the  $j$ th encoder and  $\hat{\mathcal{X}}_j\subseteq \mathbb{R}^d$  is the  $j$ -th reconstruction alphabet (see Fig. 1a). Let  $P_{\hat{X}_1\dots \hat{X}_T|X_1\dots X_T}$  be the conditional distribution of the reconstructed video given the original video which is basically determined by the mappings  $\{f_j\}_{j = 1}^T$  and  $\{g_j\}_{j = 1}^T$ . The above setting is a one-shot setup as only a single source sample is compressed at a time. For each frame  $j$ , a distortion metric is imposed on the output, which we assume throughout is the mean squared-error (MSE) function i.e.  $d(x_{j},\hat{x}_{j}) = ||x - \hat{x}_{j}||^{2}$ , which is commonly used in many applications. From a perceptual point of view, for given probability distributions  $P_{X_1\dots X_j}$  and  $P_{\hat{X}_1\dots \hat{X}_j}$  on the original and reconstructed frame  $j$ , let  $\phi_j(P_{X_1\dots X_j},P_{\hat{X}_1\dots \hat{X}_j})$  be the perception function capturing the difference between them. Note that the function  $\phi_j$  is defined based on the joint distribution of all first  $j$  frames. We call this metric as perception loss function based on joint distribution (PLF-JD). Note that when  $\phi_j(P_{X_1\dots X_j},P_{\hat{X}_1\dots \hat{X}_j}) = 0$ , we have:
$$
P _ {X _ {1} \dots X _ {j}} = P _ {\hat {X} _ {1} \dots \hat {X} _ {j}}, \quad j = 1, \dots , T. \tag {3}
$$
We refer to this case as zero-perception loss function based on joint distribution (0-PLF-JD). Alternatively, the perception loss function based on frameworkwise marginal distribution (PLF-FMD) is denoted by  $\xi_{j}(P_{X_{j}},P_{\hat{X}_{j}})$  and is based on only the marginal distribution of the  $j$ -th frame. In particular, note that 0-PLF-FMD implies that  $P_{X_j} = P_{\hat{X}_j}$  for each  $j$ . In most of the paper, for simplicity of presentation, we provide some of our results for  $T = 3$  frames. In that case, we use the shorthand notation  $\mathbf{X}$  to denote the tuple  $(X_{1},X_{2},X_{3})$ , e.g.,  $\mathsf{M} \coloneqq (M_1,M_2,M_3)$ ,  $\mathsf{D} \coloneqq (D_1,D_2,D_3)$ ,  $\mathsf{f} \coloneqq (f_1,f_2,f_3)$ .
3 Distortion Analysis for a Fixed Encoder and Zero-perception Loss
In this section, we assume that the encoding functions  $f$  are fixed, but the decoding functions  $g$  can be optimized to generate different reconstructions. Equivalently, the distribution  $P_{\mathsf{M}|\mathsf{X}K} \coloneqq \mathbb{1}\{\mathsf{M} = \mathsf{f}(\mathsf{X}, K)\}$  is fixed, while by varying the reconstruction distribution  $P_{\hat{\mathsf{X}}|\mathsf{M}K} \coloneqq \mathbb{1}\{\hat{\mathsf{X}} = \mathsf{g}(\mathsf{M}, K)\}$ , one attains different reconstructions  $\hat{\mathsf{X}}$ , where  $\mathbb{1}\{.\}$  denotes the indicator function. Furthermore defining  $D_j \coloneqq \mathbb{E}_P[\| X_j - \hat{X}_j\|^2]$ , we denote  $\mathsf{D}$  as the achievable distortion tuple associated with  $P_{\hat{\mathsf{X}}|\mathsf{M}K}$ .
One natural choice of reconstructions is the minimum mean squared error (MMSE) reconstruction function. At step  $j$ , the reconstruction, which we denote in this case by  $\tilde{X}_j$ , is obtained by taking the conditional expectation of  $X_j$  given all information at the decoder up to time  $j$  i.e.,  $\tilde{X}_j := \mathbb{E}_P[X_j | M_1 \dots M_j, K]$  for each  $j = 1, 2, 3$ . It is well known that the MMSE reconstruction functions minimize the reconstruction distortion i.e., if we define the set
$$
\Phi_ {\mathrm {D} ^ {\min }} \left(P _ {\mathrm {M} | \mathrm {X} K}\right) = \{\mathrm {D}: D _ {j} \geq \mathbb {E} _ {P} [ \| X _ {j} - \tilde {X} _ {j} \| ^ {2} ], \quad j = 1, 2, 3 \} \tag {4}
$$
then the distortion tuple  $\mathsf{D}$  associated with any reconstruction  $P_{\hat{\mathsf{X}}|\mathsf{M}K}$  satisfies  $\mathsf{D} \in \Phi_{\mathsf{D}^{\min}}(P_{\mathsf{M}|\mathsf{X}K})$ .
The main result of this section is that assuming fixed encoder, the achievable distortions under 0-PLF-FMD is at most twice of that under the MMSE distortion loss alone. The same conclusion also holds for 0-PLF-JD for a class of encoders operating at low rate. We first consider the case of 0-PLF-FMD.
Definition 1 (0-PLF-FMD Distortion) For an encoder  $P_{\mathsf{M}|\mathsf{X}K}$ , the set  $\Phi_{\mathsf{D}^0}(P_{\mathsf{M}|\mathsf{X}K})$  denotes the set of all distortion tuples  $\mathsf{D}$  for which there exists a reconstruction  $P_{\hat{\mathsf{X}}|\mathsf{M}K}$  satisfying  $P_{X_j} = P_{\hat{X}_j}$  for each  $j \in \{1,2,3\}$ .
Theorem 1 The set  $\Phi_{\mathsf{D}^0}(P_{\mathsf{M}|\times K})$  is characterized as follows:
$$
\Phi_ {\mathsf {D} ^ {0}} \left(P _ {\mathsf {M} | \mathsf {X} K}\right) = \{\mathsf {D}: D _ {j} \geq \mathbb {E} _ {P} [ \| X _ {j} - \tilde {X} _ {j} \| ^ {2} ] + W _ {2} ^ {2} \left(P _ {\tilde {X} _ {j}}, P _ {X _ {j}}\right), j = 1, 2, 3 \}, \tag {5}
$$
where  $W_2^2 (P_{\hat{X}_j},P_{\hat{X}_j})$  denotes the Wasserstein-2 distance between the two distributions [26]. Furthermore, we also have that:
$$
\Phi_ {\mathrm {D} ^ {0}} \left(P _ {\mathrm {M} | \times K}\right) \supseteq \{\mathrm {D}: D _ {j} \geq 2 \mathbb {E} _ {P} [ \| X _ {j} - \tilde {X} _ {j} \| ^ {2} ], \quad j = 1, 2, 3 \}, \tag {6}
$$
i.e., minimum achievable distortion with 0-PLF-FMD is at most twice the MMSE distortion.
Proof: See Appendix A.
We remark that the proof of Theorem 1, operationally demonstrates that the MMSE reconstruction can be converted to another reconstruction satisfying 0-PLF-FMD with at-most a factor of 2 increase in distortion, generalizing the result in [16] for the single frame scenario (see also [21]).
We next consider the case when zero perception loss is satisfied under the PLF-JD metric. Analogous to  $\Phi_{\mathsf{D}^0}(P_{\mathsf{M}|\times K})$  in Definition 1, one can define  $\Phi_{\mathsf{D}^0}^{\mathrm{joint}}(P_{\mathsf{M}|\times K})$  to be the set of distortions associated with reconstruction functions that satisfy (3). The analysis of  $\Phi_{\mathsf{D}^0}^{\mathrm{joint}}(P_{\mathsf{M}|\times K})$  is discussed in Appendix B as it is more involved. In general, the factor of two bound as in Theorem 1 cannot be realized in this case as demonstrated by a counter-example in Appendix B. Nevertheless, for a special family of encoders we can obtain a counterpart of Theorem 1. In this family of encoders, the source  $X_{j}$  at time  $j$  is nearly independent of the encoder outputs up to and including time  $j$ , i.e., we can express:
$$
P _ {X _ {j} \mid M _ {1} \dots M _ {j} K} ^ {\text {n o i s y}} = (1 - \mu) P _ {X _ {j}} + \mu Q _ {X _ {j} \mid M _ {1} \dots M _ {j} K} ^ {\text {n o i s y}}, \quad j = 1, 2, 3. \tag {7}
$$
where  $\mu$  is a sufficiently small constant and the distribution  $Q^{\mathrm{noisy}}(\cdot)$  could be arbitrary conditional distribution with same marginal as  $P_{X_j}$ . We note that such encoders are studied in a variety of problems in information theory (see e.g., [27]) that correspond to the low rate operating regime. The following result states that the factor-two bound holds approximately for such encoders.
Theorem 2 For the class of encoders given by (7), we have
$$
\Phi_ {\mathsf {D} ^ {0}} ^ {j o i n t} \left(P _ {\mathsf {M} | \mathsf {X} K} ^ {n o i s y}\right) \supseteq \{\mathsf {D}: D _ {j} \geq 2 \mathbb {E} _ {P ^ {n o i s y}} [ \| X _ {j} - \tilde {X} _ {j} \| ^ {2} ] + O (\mu), \quad j = 1, 2, 3 \}. \tag {8}
$$
Proof: See Appendix C.
We note that the low-rate operating regime is practically important, as at higher rates MMSE based reconstructions can suffice and the use of PLF metrics may be less relevant.
4 Rate-Distortion-Perception Region
In this section, we assume that both the encoder  $P_{\mathsf{M}|\mathsf{X}K}$  as well as the reconstruction  $P_{\hat{\mathsf{X}} |\mathsf{M}K}$  can be optimized and study the associated rate-distortion-perception (RDP) tradeoff. We remind the reader that for PLF-JD and PLF-FMD, the PLFs are denoted by  $\phi_j(P_{X_1\dots X_j},P_{\hat{X}_1\dots \hat{X}_j})$  and  $\xi_j(P_{X_j},P_{\hat{X}_j})$  respectively. In this case, the operational RDP region in the one-shot setting is defined as follows.
Definition 2 (Operational RDP region) For a given  $P_{\mathsf{X}}$ , an RDP tuple  $(\mathsf{R}, \mathsf{D}, \mathsf{P})$  is said to be achievable for the one-shot setting if there exist an encoder  $P_{\mathsf{M}|\mathsf{X}K}$  and a reconstruction  $P_{\hat{\mathsf{X}}|\mathsf{MK}}$  satisfying:
$$
\mathbb {E} \left[ \ell \left(M _ {j}\right) \right] \leq R _ {j}, \quad \mathbb {E} \left[ \| X _ {j} - \hat {X} _ {j} \| ^ {2} \right] \leq D _ {j}, \quad \phi_ {j} \left(P _ {X _ {1} \dots X _ {j}}, P _ {\hat {X} _ {1} \dots \hat {X} _ {j}}\right) \leq P _ {j}, j = 1, 2, 3, \tag {9}
$$
where  $\ell(M_j)$  denotes the length of the message  $M_j$ . The closure of the set of all achievable tuples, denoted by  $\mathcal{C}_{\mathsf{RDP}}^o$ , is the operational RDP region. Moreover, for a given  $(\mathsf{D}, \mathsf{P})$ , the operational DP rate region, denoted by  $\mathcal{R}^o(\mathsf{D}, \mathsf{P})$ , is the closure of the set of all tuples  $\mathsf{R}$  such that  $(\mathsf{R}, \mathsf{D}, \mathsf{P}) \in \mathcal{C}_{\mathsf{RDP}}^o$ .
The region  $\mathcal{C}_{\mathrm{RDP}}^o$  cannot be directly computed as it involves all possible one-shot encoders/decoders. But for first-order Markov source, it has a tractable approximation in terms of mutual information.
4.1 RDP Region of First-Order Markov Sources
We first define the first-order Markov sources and then introduce an iRDP region which is computable.
Definition 3 We call  $\times$  as a first-order Markov source if the Markov chain  $X_{1} \to X_{2} \to X_{3}$  holds.
Definition 4 (Information RDP region) For first-order Markov sources, the information RDP (iRDP) region, denoted by  $\mathcal{C}_{\mathrm{RDP}}$ , is the set of all tuples  $(\mathsf{R},\mathsf{D},\mathsf{P})$  which satisfy the following
$$
R _ {1} \geq I \left(X _ {1}; X _ {r, 1}\right), \quad R _ {2} \geq I \left(X _ {2}; X _ {r, 2} \mid X _ {r, 1}\right), \quad R _ {3} \geq I \left(X _ {3}; X _ {r, 3} \mid X _ {r, 1}, X _ {r, 2}\right) \tag {10}
$$
$$
D _ {j} \geq \mathbb {E} [ \| X _ {j} - \hat {X} _ {j} \| ^ {2} ], \quad P _ {j} \geq \phi_ {j} \left(P _ {X _ {1} \dots X _ {j}}, P _ {\hat {X} _ {1} \dots \hat {X} _ {j}}\right), \quad j = 1, 2, 3, \tag {11}
$$
for auxiliary random variables  $(X_{r,1},X_{r,2},X_{r,3})$  and  $(\hat{X}_1,\hat{X}_2,\hat{X}_3)$  satisfying the following
$$
\hat {X} _ {1} = \eta_ {1} \left(X _ {r, 1}\right), \quad \hat {X} _ {2} = \eta_ {2} \left(X _ {r, 1}, X _ {r, 2}\right), \quad \hat {X} _ {3} = X _ {3, r}, \tag {12}
$$
$$
X _ {r, 1} \rightarrow X _ {1} \rightarrow \left(X _ {2}, X _ {3}\right), \quad X _ {r, 2} \rightarrow \left(X _ {2}, X _ {r, 1}\right)\rightarrow \left(X _ {1}, X _ {3}\right), \tag {13}
$$
$$
X _ {r, 3} \rightarrow \left(X _ {3}, X _ {r, 1}, X _ {r, 2}\right)\rightarrow \left(X _ {1}, X _ {2}\right), \tag {14}
$$
for some deterministic functions  $\eta_{1}(.)$  and  $\eta_{2}(.,.)$ . Moreover, for a given  $(\mathsf{D},\mathsf{P})$ , the information  $DP$  (iDP) rate region, denoted by  $\mathcal{R}(\mathsf{D},\mathsf{P})$ , is the closure of the set of all tuples  $\mathsf{R}$  that  $(\mathsf{R},\mathsf{D},\mathsf{P}) \in \mathcal{C}_{\mathsf{RDP}}$ .
The expression for the iRDP region involves a search over auxiliary random variables  $\mathsf{X}_r$  and  $\hat{\mathsf{X}}$  that satisfy (12)-(14) subject to the constraints in (10)-(11). For first-order Markov sources, the following theorem states that the operational DP rate region can be approximated by the iDP rate region.
Theorem 3 For first-order Markov sources, a given  $(\mathsf{D},\mathsf{P})$  and  $\mathsf{R}\in \mathcal{R}(\mathsf{D},\mathsf{P})$ , we have
$$
\mathrm {R} + \log (\mathrm {R} + 1) + 5 \in \mathcal {R} ^ {o} (\mathrm {D}, \mathrm {P}) \subseteq \mathcal {R} (\mathrm {D}, \mathrm {P}). \tag {15}
$$
Proof: See Appendix D.
From Theorem 3, it follows that  $\mathcal{R}(\mathrm{D},\mathrm{P})$  with overhead  $\log (\mathrm{R} + 1) + 5$  is an inner bound to  $\mathcal{R}^o (\mathrm{D},\mathrm{P})$ . On the other hand,  $\mathcal{R}(\mathrm{D},\mathrm{P})$  provides an outer bound to  $\mathcal{R}^o (\mathrm{D},\mathrm{P})$ . The two bounds match with each other in high rates. It can be shown that the overhead also vanishes in the large-blocklength setting where multiple symbols are encoded at a time. In the remainder of this paper, we will approximate  $\mathcal{R}^o (\mathrm{D},\mathrm{P})$  with  $\mathcal{R}(\mathrm{D},\mathrm{P})$  and use the latter region for our analysis.