Stochastic Codebook Regeneration for Sequential Compression of Continuous Alphabet Sources

This paper proposes an effective and asymptotically optimal framework for stochastic, adaptive codebook regeneration for sequential (“on the fly”) lossy coding of continuous alphabet sources. Earlier work has shown that the rate-distortion bound can be asymptotically achieved for discrete alphabet sources, by a “natural type selection” (NTS) algorithm. At each iteration <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n$</tex> , a maximum-likelihood framework is used to estimate the reproduction distribution most likely to generate the empirical types of a sequence of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> length- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l$</tex> codewords that respectively “d-match” (i.e., are within distortion <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$d$</tex> from) a sequence of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> length- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell$</tex> source words. The reproduction distribution estimated at iteration <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n$</tex> is used to regenerate the codebook for iteration <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n+1$</tex> . The sequence of reproduction distributions was shown to converge, asymptotically in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K, n$</tex> , and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell$</tex> , to the optimal distribution that achieves the rate-distortion bound for discrete alphabet sources. This work generalizes the NTS framework to handle sources over more general (e.g., continuous) alphabet spaces, which often preclude a natural interpretation of the concept of “type”. We show, for continuous alphabet sources and fixed block length <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell$</tex> , that as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K\rightarrow \infty$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n \rightarrow \infty$</tex> , the sequence of estimated reproduction distributions converges, in the weak convergence sense, to a distribution that achieves the rate-distortion bound, albeit for an auxiliary distortion measure introduced as subterfuge to effectively impose a maximum distortion constraint over <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> blocks. Leveraging this result, we establish that the sequence of reproduction distributions converges, asymptotically in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell$</tex> , to the optimal codebook reproduction distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Q^{\ast}$</tex> that achieves the rate-distortion bound, with respect to the original distortion measure.