# Predictive Quantization and Symbolic Dynamics
## 5 Transfer Entropy Estimation  
Transfer entropy between two sequences is the amount of information passing from the past of one sequence to another when the dependencies of the past of the other sequence (sequence's own dynamics) have been already taken into account. To distinguish the notation from that of previous sections, we will denote the two time series as data \(X\) and context \(C\), and their past as \(\bar{X}\) and \(\bar{C}\), respectively. This gives \(TE_{C\to X}=I(X;\bar{C}|\bar{X})\) Similarly \(TE_{X\to C}=I(C;\bar{X}|\bar{C})\). Writing mutual information in terms of entropy  
\[I(C;\bar{X})=H(C)-H(C|\bar{X})\] \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]  
Adding and subtracting \(H(C)\):  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})-H(C)+H(C)=I(C;\bar{X}, \bar{C})-I(C;\bar{C}) \tag{19}\]  
Additionally:  
\[I(X;\bar{C}|\bar{X})=I(X;\bar{C},\bar{X})-I(X;\bar{X}) \tag{20}\]  
We consider a sum of these expressions, calling it symmetrical transfer entropy (SymTE):  
\[SymTE=I(C;\bar{X}|\bar{C})+I(X;\bar{C}|\bar{X}) \tag{21}\]  
**Lemma 2**: _Given time series \(X\) and \(C\), the symmetrical transfer entropy is given by_  
\[SymTE=I((C,X);\overline{(C,X)})-I(C;X|\overline{(C,X)})+I(C,X)-I(X,\bar{X})-I( C,\bar{C})\, \tag{22}\]  
_where we used a notation for the past of the joint pair \((\bar{C},\bar{X})=\overline{(C,X)}\)._  
**Proof. ** Symmetric transfer entropy (SymTE) between two sequences \(X\) and \(C\),  
\[SymTE=I(C;\bar{X}|\bar{C})+I(X_{j};\bar{C}|\bar{X})\]equals to  
\[SymTE=I((C,X);\overline{(C,X)})-I(C;X|\overline{(C,X)})+I(C,X)-I(X_{i},\bar{X})- I(C,\bar{C})\,\]  
where we used a notation for the past of the joint pair \((\bar{C},\bar{X}_{i})=\overline{(C,X_{i})}\). Using the relation  
\[I(X;\bar{C}|\bar{X})=H(X|\bar{X})-H(X|\bar{X}\bar{C})=H(X|\bar{X})-H(X)+H(X)-H(X |\bar{X}\bar{C})=I(X;\bar{C}\bar{X})-I(X,\bar{X})\]  
and similarily  
\[I(C;\bar{X}|\bar{C})=I(C;\bar{X}\bar{C})-I(C,\bar{C})\]  
We sum both of the above expressions to derive the symmetrical TE:  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]0 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]1  
Continuing the derivation  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]2 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]3 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]4 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]5 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]6 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]7 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]8  
This gives the equality:  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})\]9  
Plugging back into SymTE gives:  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})-H(C)+H(C)=I(C;\bar{X}, \bar{C})-I(C;\bar{C}) \tag{19}\]0 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})-H(C)+H(C)=I(C;\bar{X}, \bar{C})-I(C;\bar{C}) \tag{19}\]1 \[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})-H(C)+H(C)=I(C;\bar{X}, \bar{C})-I(C;\bar{C}) \tag{19}\]2  
If we assume that the generation of \(X\) is independent of \(C\) given their joint past \(\overline{(C,X)}\), then \(I(C;X|\overline{(C,X)})=0\), resulting in  
\[I(C;\bar{X}|\bar{C})=H(C|\bar{C})-H(C|\bar{X},\bar{C})-H(C)+H(C)=I(C;\bar{X}, \bar{C})-I(C;\bar{C}) \tag{19}\]3  
which is a sum of the \(IR\) of the joint pair \((C,X)\) and the mutual information between \(C\) and \(X\) regardless of time, minus the \(IR\) of the separate time series. In other words, the symmetrical TE is a measure of surprisal present in the joint time series minus the surprisal of each of its components, plus the mutual information (lack of independence) between their individual components. This captures the difference between surprisal when considering the compound time series versus surprisal when considering them separately, with an added component of mutual information between the observations of the two time series regardless of time.