# Thermodynamic Formalism in Neuronal Dynamics and Spike Train Statistics
## 2 Mathematical Setting of Thermodynamic Formalism
### General Properties
#### 2.1.2 Entropy and the Variational Principle  
The Shannon entropy of a probability distribution quantifies the average level of \"uncertainty\" in its possible outcomes.  
\[h(p):=-\sum_{x\in A}p(x)\log p(x). \tag{3}\]  
More generally, let \(\
u\) be a shift-invariant probability measure on \(X\), we introduce the block entropy:  
\[H_{n}(\
u):=-\sum_{x_{0,n-1}\in A^{n}}\
u([x_{0,n-1}])\log\
u([x_{0,n-1}]).\]  
For finite alphabets and because \(\
u\) is shift-invariant, the following limit exists,  
\[h(\
u):=\lim_{n\to\infty}\frac{1}{n}H_{n}(\
u),\]  
and is called the Kolmogorov-Sinai entropy or simply entropy of \(\
u\)[76; 81]. The key is that the sequence \(H_{n}\) is sub-additive and the possible values lie in the compact set \([0,\log|A|]\) (see [76]).  
Another important quantity is the Kullback-Leibler (KL) divergence, which quantifies the difference between probability distributions. Consider two probability distributions \(p\) and \(q\) on the same probability space \(X\), the KL divergence is:  
\[D(p||q):=\sum_{x\in X}p(x)\log\frac{p(x)}{q(x)}.\]  
This divergence is finite whenever \(p\) is absolutely continuous with respect to \(q\), and it is only zero if \(p=q\).  
Following the analogy with systems in statistical mechanics, an equilibrium state is defined as the measure that satisfies the so-called variational principle, namely:  
\[\sup\left\{h(\
u)+\int\phi\mathrm{d}\
u\ :\ \
u\in\mathcal{M}_{\sigma}(X) \right\}=h(\mu_{\phi})+\int\phi\mathrm{d}\mu_{\phi}=F(\phi), \tag{4}\]  
where \(\int\phi\mathrm{d}\
u\) represents the expected value of \(\phi\) with respect to the measure \(\
u\) (it would be noted \(\mathbb{E}_{\
u}(\phi)\) in a probabilistic context). Let us comment on this result. The first equality establishes that, among all shift-invariant probability measures, there is a unique one, \(\mu_{\phi}\), which maximises \(h(\
u)+\int\phi\mathrm{d}\
u\). If \(\int\phi\mathrm{d}\
u\) is fixed to some value, this corresponds to maximising the entropy under constraints. This is the Maximum Entropy Principle, but (4) is more general. The second equality states that the maximum, \(h(\mu_{\phi})+\int\phi\mathrm{d}\mu_{\phi}\) is exactly the pressure defined in Equation (2). The equation \(F(\phi)=h(\mu_{\phi})+\int\phi\mathrm{d}\mu_{\phi}\) establishes a link with thermodynamics, as \(F(\phi)\) is equivalent to the free energy (In contrast to thermodynamics where free energy or pressure refer to different thermodynamic ensembles, we will not make such distinction here as it is irrelevant. Also, note that we do not keep the minus sign and take the Boltzmann constant equal to one. Thus, (4) coincides with the principle of free energy minimisation in statistical mechanics, but it corresponds to a maximising measure in our formalism because of a sign convention. Gibbs measures associated to potentials satisfying that \(\mathrm{var}_{k}(\phi)\leq b\theta^{k}\), as stated above, are equilibrium states. For more general potentials, the notion of Gibbs states and equilibrium states may not coincide.