# Predictive Quantization and Symbolic Dynamics
## 4 Representation  
As mentioned earlier in the chapter, a suitable choice of time series representation is critical for the analysis of time series. Transforms, such as DFTs and wavelets, or adaptive transforms, such as SVD, are commonly used to find an embedding of the signal in another space that better captures some salient aspect of the data. In other cases, feature extractors are designed to represent the time series in terms of the special parameters of interest. The symbolization technique presented above is feature agnostic and can be applied to any sequence of features if an appropriate distance function is provided to measure feature or embedding similarity. To avoid the step of feature engineering or the choice of appropriate transforms, it would be desirable to have a method that allows learning a representation with sufficient quality and predictability according to the information dynamics objective that was defined in Section 2. Recently, generative deep neural models have become the primary tools for representation learning in multiple domains. The compelling idea of representation learning is that various aspects of modeling data from the world are captured using vector spaces that map statistical structures into mathematical space. The information dynamics objective states the goal of representation learning as a minimization of the problem of the mutual information between the data and a latent state, namely \(min_{p(Z|X)}I(X,Z)\), which can be subject to a distortion constraint \(D(X,Z)\). Such an approach is also known as a rate-distortion problem in information theory. The relation between variational encoding and rate distortion was made explicit in the \"Broken ELBO\" paper [13]. The authors show there that the mutual information between the input \(X\) and the latent code \(Z\) is bounded below and above by two factors \(D\) and \(R\) that comprise the ELBO, as follows  
\[H-D\leq I_{e}(X,Z)\leq R \tag{10}\]  
where \(H\) is the data entropy, \(D\) is the distortion measured as reconstruction log likelihood, and \(R\) is the model encoding rate (not the optimal Shannon rate, which is the mutual entropy) measured by KL divergence between the encoding distribution and the learned marginal approximation. The reader is referred to the appendix for a short summary of the results, and to Ref. [13] for more complete detail. The important point here is that this expression can be rewritten as a Lagrange optimization by adding \(D\) to all terms or the above inequality, giving  
\[H\leq I_{e}(X,Z)+D\leq R+D=-ELBO. \tag{11}\]  
Moreover, allowing for different weight of the distortion \(D\), we have  
\[I_{e}(X,Z)+\beta D\leq R+\beta D=-ELBO(\beta). \tag{12}\]  
where \(\beta=1\) gives the original VAE ELBO expression and \(ELBO(\beta)\) corresponds to \(\beta\)-VAE model. This expression is an unconstrained version of the rate distortion,  
\[\mathcal{L}=I(X,Z)+\beta\langle d(X,Z)\rangle \tag{13}\]  
where minimizing for \(\beta\) value gives a point on the rate-distortion curve. Changing \(\beta\) is important for dealing with two known problems in VAE encoding: Information Preference and Exploding Latent Space problems. The first refers to vanishing of the mutual information between \(Z\) and \(X\), or in other words \(Z\) and \(X\) becoming independent due to a powerful decoder. The second problem refers to over-fitting of the output likelihood by matching individually each data point when the training data is finite. #This also effectively causes the latent states to become irrelevant. One of the approaches proposed in this paper is applying the variational autoencoder (VAE) [14] as an embedding of the time series into a sequence of latent variables. Unlikestandard autoencoders that translate data into a more desirable latent representation found in the activation of the hidden layers, VAE explicitly constrains the latent variables \(Z\) so that they should be random variables distributed according to some prior \(p(z)\).