# Predictive Quantization and Symbolic Dynamics
## 2 Information Dynamics  
Information dynamics considers the information passing between the data past \(X\) and the present observation \(Y\), formally defined in terms of mutual information \(I(X,Y)\). In this paper, we extend the notion of information dynamics to aspects of data representation by providing a theoretical model and an analysis framework for exploring the relations between four factors in a time series: the data past \(X\) and a present observation \(Y\) versus their representation in terms of latent variables \(Z\) and \(T\), respectively. This relationship between the four factors \(X\), \(Y\), \(Z\), and \(T\) is illustrated in Figure 4, schematically shown for an example of musical observations, with latent states schematically drawn near a brain-like icon. It should be noted that this representation is schematic only--there is no assurance that the latent representation follows similar dynamics to the data or that later on its symbolized version induces correct generating partitions. Nevertheless, we assume that finding a representation that maximizes the information dynamics of the resulting sequence is intuitively a good approach to discover meaningful structures present in the original data. We further assume that the following Markov relations exist between the variables: \(Z\) is assumed to be a hidden cause of \(X\), which in turn causes \(Y\), which is further encoded into \(T\). In other words, the triplet \(X,Y,Z\) obeys the following generative Markov relation \(p(X,Y,Z)=p(Y|X)p(X|Z)p(Z)\), and similarly \(p(X,Y,Z,T)=P(T|Y)p(Y|X)p(X|Z)p(Z)\) includes the probability for the next encoding. The significance of the Markov relation is that it specifies the assumptions about how information in a time series evolves according to the manner in which probabilities of the different variables depend on each other. Based on the Markov relation between \(Z-X-Y\), our model comprises two competing factors: quality of data representation in terms of latent variables versus the prediction ability of the latent variables in time. Our model is defined as a minimization problem of the discrepancy between data prediction based on complete information about the past observation \(X\) versus prediction based on the latent encoding of the past \(Z\). This discrepancy is averaged over all possible encoding pairs \(X,Z\), which is stated as follows:  
**Lemma 1**. : _For Markov relation \(Z-X-Y\), the following relation holds_  
\[\langle D_{KL}(p(Y|X)||p(Y|Z))\rangle_{p(X,Z)}=I(X,Y|Z)=I(X,Y)-I(Z,Y) \tag{1}\]where, \(D_{KL}(\cdot,\cdot)\) is the Kullback-Liebler (KL) divergence and \(I(\cdot,\cdot)\) is mutual information._  
From the definitions of \(D_{KL}(\cdot,\cdot)\) as the Kullback-Liebler distance between different distributions and \(I(\cdot,\cdot)\) as the mutual information between their random variables, we write  
\[D_{KL}(P,Q)=\int p(x)\log\frac{p(X)}{q(X)}dX \tag{2}\]  
and  
\[I(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)=H(X)+H(Y)-H(X,Y) \tag{3}\]  
with signal entropy given by  
\[H(X)=-\int p(X)\log p(X)dX \tag{4}\]  
Using the following relation between KL distance and mutual information  
\[I(X,Y)=D_{KL}(p(X,Y),p(X)p(Y))=D_{KL}(p(Y|X)p(X),p(X)p(Y))\, \tag{5}\]  
and taking into account the Markov relations \(Z-X-Y\), we have the following conditional independence between the variables \(p(Y,X,Z)=p(Y|X,Z)p(X,Z)=p(Y|X)p(X,Z)\). Averaging over \(P(X,Z)\), we obtain  
\[\langle D_{KL}(p(Y|X)||p(Y|Z))\rangle_{p(X,Z)}\] \[=\int p(X,Z)(\int p(Y|X)\log\frac{p(Y|X)}{p(Y|Z)}dY)dX\;dZ\] \[=\int p(Y,X,Z)\log(\frac{p(Y|X)p(X)}{p(X)p(Y)}\frac{p(Y)p(Z)}{p( Y|Z)p(Z)})dYdX\;dZ\] \[=\int p(Y,X,Z)\log(\frac{p(Y,X)}{p(X)p(Y)}dYdXZ-\int p(Y,X,Z)\log (\frac{p(Y)p(Z)}{p(Y,Z)}dYdX\;dZ\] \[=I(X,Y)-I(Z,Y)\]  
To complete the proof, we need to show that \(I(X,Y|Z)=I(X,Y)-I(Z,Y)\). This can be shown by considering the definition of mutual information as  
\[D_{KL}(P,Q)=\int p(x)\log\frac{p(X)}{q(X)}dX \tag{2}\]0 Schematic graph of statistical dependencies between the different model variables.