## 2 The Information Bottleneck Method and Coarsely Quantized Information Bottleneck Signal Processing
### The Information Bottleneck Method  
The information bottleneck method is a quite generic information theoretical setup that was introduced by [PERSON] et al. in [2]. The basic setup consists of three discrete random variables \(X\), \(Y\) and \(T\) with realizations \(x\in\mathcal{X},y\in\mathcal{Y}\) and \(t\in\mathcal{T}\). These variables follow the Markov relation \(X\to Y\to T\) and interact as illustrated in Figure 1 and explained in the following. The random variable \(Y\) is considered to be observed and, therefore, termed the observed random variable in the information bottleneck problem setup. The baseline model is that observing the realizations \(y\in\mathcal{Y}\) of \(Y\) could deliver information that could be classified into relevant and irrelevant information. In order to define which features of \(Y\) are considered relevant, the so-called relevant random variable \(X\) is introduced. As a result, the relevance is defined through another variable, and the relevant information that \(X\) and \(Y\) share is  
\[I(X;Y)=\sum_{x\in X}\sum_{y\in\mathcal{Y}}p(x,y)\log\frac{p(x,y)}{p(x)p(y)}=D_{ KL}\{p(x,y)\mid p(x)p(y)\}, \tag{1}\]  
where \(p(x,y)\) is the joint distribution and \(p(x)\) and \(p(y)\) are the marginal distributions of \(X\) and \(Y\), respectively. \(D_{KL}\{\,\cdot\,|\,\cdot\}\) in Equation (1) is the Kullback-Leibler divergence. The aims of the information bottleneck method are now twofold:  
1. Conduct a lossy compression of the realizations \(y\in\mathcal{Y}\) to a compressed realization \(t\in\mathcal{T}\) to yield a compact compressed representation \(T\) of the observation \(Y\). The information theoretical notion of such a compression is the minimization of the compression information \(I(Y;T)\) (i.e., the transmission rate, relating to rate-distortion theory). 2. While conducting the compression mentioned above, preserve the relevant information \(I(X;T)\leq I(X;Y)\). The goals mentioned above are contradictory. Typically, a strong compression will limit the possibility to keep the preserved relevant information \(I(X;T)\) above a desired lower bound. Similarly, aiming for a certain minimum amount of \(I(X;T)\leq I(X;Y)\) typically provides a lower bound to \(I(Y;T)\). As a result, an optimum rule to compress \(Y\) onto \(T\) in the information bottleneck sense describes a trade-off between achieving a minimum possible compression information \(I(Y;T)\) that also allows keeping the preserved relevant information \(I(X;T)\) above a desired minimum level. In technical terms, the compression rule that maps \(Y\) onto \(T\) is described as a conditional probability distribution \(p(t|y)\). This conditional distribution provides the probabilities of a certain \(t\in\mathcal{T}\) for a given \(y\in\mathcal{Y}\) and hence describes a possibly stochastic mapping of \(y\) onto \(t\). The information bottleneck problem can be understood as the optimization problem of finding a suitable conditional probability distribution \(p(t|y)\) with the desired characteristics of minimizing \(I(Y;T)\) while preserving \(I(X;T)\) for the Markov chain \(X\to Y\to T\). [PERSON] et al. proposed finding an optimum mapping \(p(t|y)\) using the Lagrange method in [2] and introduced the Lagrangian  
\[\mathcal{L}(p(t|y))=I(Y;T)-\beta I(X;T) \tag{2}\]  
Figure 1: Illustration of the information bottleneck method. The variables \(X,Y\) and \(T\) form a Markov chain \(X\to Y\to T\) and are termed the relevant, observed and compressed variables, respectively. The fundamental principle is to minimize \(I(Y;T)\) while preserving \(I(X;T)\). which has to be minimized over the set of all valid conditional probability distributions \(p(t|y)\). The Lagrangian multiplier \(\beta\geq 0\) is a trade-off parameter that allows tuning the aforementioned trade-off between the compression and preservation of relevant information.