## 2 The Information Bottleneck Method and Coarsely Quantized Information Bottleneck Signal Processing
### The Information Bottleneck Method
For \(\beta=0\), the focus is only on compression, and the preservation of \(\mathrm{I}(\mathsf{X};\mathsf{T})\) is not taken into account, while \(\beta\to+\infty\) aims to maximize the preserved relevant information. In [2], [PERSON] et al. also derived a set of equations characterizing \(p(t|y)\), \(p(t)\) and \(p(x|t)\) such that  
\[p(t|y) = \frac{p(t)}{Z(y,\beta)}\exp(-\beta D_{\mathrm{KL}}\{p(x|y)\mid p (x|t)\}) \tag{3}\] \[p(t) = \sum_{y\in\mathcal{Y}}p(t|y)p(y)\] (4) \[p(x|t) = \frac{1}{p(t)}\sum_{y\in\mathcal{Y}}p(t|y)p(x,y), \tag{5}\]  
where \(Z(y,\beta)\) is a normalization function that guarantees that \(p(t|y)\) is a valid conditional distribution. In general, the optimization problem of finding a \(p(t|y)\) solution that minimizes the Lagrangian from Equation (2) is neither concave nor convex [2]. Nevertheless, the mentioned equations (Equations (3-5)) naturally suggest an iterative algorithm termed the iterative information bottleneck algorithm to obtain \(p(t|y)\) for a given joint distribution \(p(x,y)\), a trade-off parameter \(\beta\) and an intended cardinality \(|\mathcal{T}|\) of the compression variable \(\mathsf{T}\) that is also described in [2]. This algorithm provably converges to at least a local minimum of the Lagrangian in Equation (2). Details on its convergence are discussed in [3]. In addition to the iterative information bottleneck algorithm from [2], many other information bottleneck algorithms appeared in the literature (e.g., [57; 58]). These algorithms can be understood as the work horses of the information bottleneck method, as they can determine the compression mapping \(p(t|y)\) for a given \(p(x,y)\), \(\beta\) and a desired cardinality \(|\mathcal{T}|\). The information bottleneck algorithms also deliver the distributions \(p(x|t)\) and \(p(t)\), and thus \(p(x,t)=p(x|t)p(t)\) according to Equations (4) and (5) as side products. Figure 2 provides an overview of the inputs taken and the outputs delivered by an information bottleneck algorithm. An important notion of the resulting conditional probability distribution \(p(t|y)\) is that it clusters the event space \(\mathcal{Y}\) of the observed random variable \(\mathsf{Y}\) into clusters \(\mathcal{Y}_{t}\), \(t\in\{0,1,\ldots,|\mathcal{T}|-1\}\). This concept covers deterministic mappings of \(y\) onto \(t\) (i.e., \(p(t|y)\in\{0,1\}\forall(y,t)\)) that result in a hard clustering of \(\mathcal{Y}\) and also probabilistic mappings that describe a soft clustering. For the latter, the realizations \(y\) can be contained in various clusters \(\mathcal{Y}_{t}\) with different probabilities. In the very important special case of aiming to preserve a maximum desired amount of \(\mathrm{I}(\mathsf{X};\mathsf{T})\) (i.e., \(\beta\to+\infty\) for a given cardinality \(|\mathcal{T}|\)), the clustering of \(\mathcal{Y}\) described by \(p(t|y)\) becomes a hard clustering [58]. In this case, it is easy to limit the compression information \(\mathrm{I}(\mathsf{Y};\mathsf{T})\) by a proper choice of the cardinality \(|\mathcal{T}|\) of the compressed representation. In the context of this article, this cardinality determines the number of bits required to process the compressed realizations \(t\in\mathcal{T}\) in the digital hardware of a communication receiver. The remainder of this article will only focus on deterministic compression mappings, as these are most practical to be applied in digital communication receivers. Figure 2: Overview of the inputs taken and the outputs delivered by an information bottleneck algorithm. It will be important in the following that a deterministic mapping \(p(t|y)\) can also be interpreted as a deterministic input/output relation of a system with input \(y\in\mathcal{Y}\) and output \(t\in\mathcal{T}\). Such a system can be implemented, for example, as a lookup table that holds the respective \(t\in\mathcal{T}\) for each possible \(y\in\mathcal{Y}\) if the input space \(\mathcal{Y}\) has a manageable cardinality.