## 3 Parameter Learning of Trainable Functions to Maximize the Relevant Information
### Further Discussion, Other Approaches and Future Work  
The discussion of the lookup table approach, the computational domain approach and the neural network approach to information bottleneck signal processing reveals the key principles of information bottleneck signal processing. These are choosing appropriate functions \(f_{\mathbf{\theta}}(\mathbf{y})\) with trainable parameters \(\mathbf{\theta}\) and determining the values of these parameters that maximize the relevant information. The functions used should only use a small number of simple arithmetical operations, and the number of parameters should also be small for complexity reasons. As was discussed above, lookup tables have strengths regarding flexibility, as for discrete inputs \(\mathbf{y}\) and discrete outputs \(t\), they can implement arbitrary input/output relations \(t=f_{\mathbf{\theta}}(\mathbf{y})\). However, they often need too many parameters if the number of inputs is large. The studied computational domain approach requires much fewer parameters but needs a sophisticated choice for the reconstruction function \(\phi(. )\) and the function \(\Phi(. )\) for the signal processing problem of interest, (e.g., LDPC decoding). Neural network approaches offer an almost endless choice of network architectures. These are determined by the number of layers, the types of the layers and their respective activation functions. All these aspects influence their implementation complexity and the number of parameters. However, neural networks enable realizing almost arbitrary mappings with proper architectures and a reasonable number of parameters. Moreover, they can be trained efficiently using gradient-based algorithms. In future work, it will be interesting to further study flexible, trainable functions for information bottleneck signal processing units. To illustrate that many more options exist, we briefly mention a quite different approach from [64; 65] as an example. This approach is based on efficient nearest neighbor search algorithms in graphs. These search algorithms are used to implement trainable functions \(f_{\mathbf{\theta}}(\mathbf{y})\) that aim to maximize \(\mathrm{I}(\mathrm{X};\mathsf{T})\) in [64; 65]. The mappings \(t=f_{\mathbf{\theta}}(\mathbf{y})\) in [64; 65] use a small number of distance calculations between the incoming vector \(\mathbf{y}\) and some trained parameter vectors \(\mathbf{\theta}_{t}\) to determine the system output \(t\) such that \(\mathrm{I}(\mathrm{X};\mathsf{T})\rightarrow\max\). An exemplary nearest neighbor search is illustrated in Figure 16.  
In the figure, the parameters \(\mathbf{\theta}=[\mathbf{\theta}_{0},\mathbf{\theta}_{1},\dots,\mathbf{\theta}_{7}]\) reflect the positions of the points labeled \(\mathbf{\theta}_{t}\) in the two-dimensional plain. In addition, an exemplary two-dimensional input vector \(\mathbf{y}\) is shown as a red dot. The outcome of the operation \(t=f_{\mathbf{\theta}}(\mathbf{y})\) is the integer index \(t\) of an approximate or exact nearest neighbor \(\mathbf{\theta}_{t}\) of the input vector \(\mathbf{y}\) under some arbitrary distance measure \(d(\mathbf{y},\mathbf{\theta}_{t})\). In Figure 16, the distance is the Euclidean distance, and the exact nearest neighbor is considered for illustration purposes. Please note that it is not required to calculate all possible distances \(d(\mathbf{y},\mathbf{\theta}_{t})\) to determine the system output \(t\). Instead, either an exact or approximate nearest neighbor search using very efficient search algorithms in graph structures of the parameter vectors \(\mathbf{\theta}_{t}\) can be applied [64; 65]. The parameters involved in nearest neighbor search-based mappings were trained using genetic algorithms in [64; 65]. Genetic algorithms are parameter optimization algorithms which are inspired by the natural evolution of the species [66]. Figure 16: Illustration of a mapping \(t=f_{\mathbf{\theta}}(\mathbf{y})\) based on the nearest neighbor search.