# Predictive Quantization and Symbolic Dynamics
## 1 Introduction
Mathematically, this means that the goal of learning is to be able to approximately simulate the statistics of the data from an internal representation that the learning system constructs. Once such statistical representation is constructed, various operations can be performed directly on the model's parameters rather then on the data themselves. For instance, probabilistic clustering is done by an iterative procedure called expectation maximization (EM), which bears close relation to a variational method called evidence lower bound (ELBO), which was adopted using the reparametrization method in variational autoencoders (VAE). The VAE framework represents the data in terms of the multidimensional Gaussian distribution of latent factors, from which the data samples can be recovered. Although a detailed explanation of VAE architecture and its learning procedure by maximizing ELBO is beyond the scope of this paper, we show the general architecture of such a model in Figure 2.  
The important components of the VAE method is that it assumes a probabilistic representation of the latent space, which is learned by an encoder followed by estimating the mean and variance of the encodings as Gaussian distribution. Reconstruction of the data is done by sampling from a normal distribution and \"reparametrizing it\" by multiplying it with appropriate variance, adding a mean shift to the random sample, and passing it through a learned decoder. The learning process comprises the network parameters so as to achieve a minimization of two criteria, namely the reconstruction loss at the output of the decoder, and a statistical loss written in terms of Kullback-Leibler statistic divergence between the estimated Gaussian distribution of the latent vectors and a prior normal distribution. This second loss can also be viewed as a regularization penalty, trying to find the simplest random distribution of the latent encoding by trying to make it fit a normal distribution. Recently, VAE representation was further extended into a quantized version named VQ-VAE, becoming a powerful and popular alternative method for so-called \"discrete neural representation\". In Figure 3, an audio signal is encoded into a series of codes at three different resolutions as part of a larger system that uses transformers to generate high-fidelity and diverse musical materials that are several minutes long [4]. The input audio is segmented and encoded into latent vectors \(h_{t}\), which are then quantized to the closest codebook vector \(e_{k}\). The decoder then reconstructs the audio from a sequence of codebook vectors. In the following, we describe a different method of quantization and symbolization that is based on a lossy encoding of VAE latent codes. Unlike existing methods, where a quantization procedure is used to directly obtain a discrete or symbolized version of the data, the method proposed in this paper focuses on symbolization that maximizes the predictive information of the symbolic sequence. This predictive information, defined in terms of mutual information between the past and the present of the time series, is measured by \"information rate\". By taking the information dynamics into consideration, we hope to obtain a symbolization that better matches the dynamics of the original data. Limiting  
Figure 3: VQ-VAE discrete neural architecture that uses codes to represent and reconstruct audio. Figure 2: VAE architecture showing the encoder–decoder structure with estimation of the mean and variance using the normal distribution reparametrization trick, and the two loss functions components of the ELBO criterion. See text for more details. the amount of information in data representation based on predictive criteria is sometimes called a \"predictive bottleneck\". Moreover, prior to the symbolization step, the complexity of the continuous representation is reduced by applying a bit-rate reduction.