# Predictive Quantization and Symbolic Dynamics
## 1 Introduction
Changing the partitioning threshold effectively relabels the transitions between the possible words, changing the probabilities of transitioning between the vertices or even forbidding some words from appearing. This effect can be measured by considering the entropy over the word space and comparing it to theoretical entropy in the case when an analytic equation of the system dynamics is known. As shown in Refs. [2; 3], the entropy of such language is often lower than the exact system entropy. Moreover, the symbolic sequence entropy dependence on threshold displacement is a non-monotonic function because some words that disappear for some threshold values can reappear at other values. Figure 1: Transforming a real-valued continuous or sampled time function into SAX representation with alphabet a, b, c, d. Each time segment is turned into a symbol by taking the mean and looking up the corresponding symbol on y-axis. The complete signal is translated into a word eight letters long, as shown on the bottom of the graph. Rather than operating directly on the data values, some methods of signal or time series analysis perform transformations of the data into more convenient or meaningful vector spaces. It is evident that a suitable choice of such representation is critical for the success of downstream applications, such as classification, data mining, or generative models. When sequence dynamics is relatively simple, transforms, such as DFT, can be used to reveal periodic properties, even if slight deviations from the transform principal patterns occur in the signal. To address the time varying properties of a signal, transformation and change of representation is often done over short segments of data. In special cases, such as in audio signals, custom human-engineered features, such as mel frequency cepstral coefficients (MFCC) or Chroma are often used. When high-dimensional feature representation is efficiently translated into a low-dimensional space, the term \"embedding\" is used to denote such reduced representation. From a practical perspective, embedding serves as a unique identifier of a short data segment, represented as a vector in some space that maps similar or related data segments to nearby points. The situation is further complicated by longer-term signal dynamics that exhibit a-periodic repetitions with motifs of variable length with partial and approximate repetitions. Combining embeddings with symbolic discretization opens up the study of time series statistics to methods of information and dynamics theory, automata theory, and other formal tools coming from computational modeling. Among the common discretization methods are K-Means and a variety of so-called vector quantization (VQ) methods in general that assign the same index or letter symbol to samples falling within the same multi-dimensional partition. The K-Means algorithm is based on the iterative partitioning of points into regions that are then matched to their centroids and re-partitioned again, with the process repeating until convergence. Self-organizing maps (SOM) are another popular data analysis and visualization tool based on neural networks that bear close similarity to autoencoders (AE) that can learn a non-linear embedding of the data. Clustering can be performed on the SOM nodes or AE latent codes to identify groups of data points with similar metrics. Recently, generative deep neural models have become the primary tools for representation learning in multiple domains. The compelling idea of machine learning is that various modeling and inference tasks, including embeddings, representation, and encodings of data collected from the world, are captured using vector spaces that map statistical structures into mathematical space. In particular, the powerful idea of generative modeling is that in order to effectively represent the data, the learning system needs to be able to effectively reproduce similar data.