The Information Bottleneck : Theory and Applications

This thesis introduces the first comprehensive review of the Information Bottleneck (IB) method along with its recent extension, the multivariate IB. The IB method was originally suggested in [82] as a new information-theoretic approach for data analysis. The bas ic idea is surprisingly simple: Given a joint distributionp(x; y), find a compressed representation of X, denoted byT , that is as informative as possible about Y . This idea can be formulated as a variational principle of mi ni zing the mutual information, I(T ;X) (which controls the compactness of the representation T ), under some constraint on the minimal level of mutual information that T preserves about Y , given byI(T ;Y ) . Hence, the fundamental trade-off between the complexity of the model and its precision is expressed he re in an entirely symmetric form, where the exact same concept of information controls both its sides. I ndeed, an equivalent posing of the IB principle would be to maximize the information T maintains about Y , where the (compression) information I(T ;X) is constrained to some maximal level. As further shown in [82], this constrained optimization pro blem can be considered analogous to rate distortion theory, but with an important distinction: the distort i n measure does not need to be defined in advance, but rather naturally emerges from the joint statistics, p(x; y). Moreover, it leads to a tractable mathematical analysis which provides a formal characterization of the op timal solution to this problem. As an immediate implication, the IB method formulates a well defined info rmation-theoretic framework for unsupervised clustering problems, which is the main focus of this thesis. Nonetheless, it is important to keep in mind that the same underlying principle of a trade-off between in formation terms may have further implications in other related fields, as recently suggested in [37]. After the introduction in Part I, in Part II we provide a detai led description of the IB method and its relations to rate distortion theory. We explicitly define so me of the ideas that were briefly mentioned in [82], and further suggest new algorithmic approaches to construc t solutions to the IB problem. Moreover, we provide empirical results that demonstrate the method’s us efulness in a variety of applications, and discuss several related issues which are left for further research. In Part III we suggest a general principled framework for mul tivariate extensions of the IB method. While the original principle suggested compressing one variable while preserving the information about another, this extension allows us to consider any finite number of rand om variables under the same framework. In particular, this extended formulation defines a novel famil y of optimization problems in which the original IB problem constitutes a special (important) case. These pr oblems suggest novel approaches to data analysis, that to the best of our knowledge have not been treated or defin ed elsewhere. Specifically, we suggest considering multiple systems of data partitions that are in t rrelated, where Bayesian networks are utilized to specify the systems of clusters and which information ter ms should be maintained. Analogous to the original IB problem, we characterize the fo rm of the optimal solution to this general multivariate principle. That is, we are not satisfied with de fining novel problems but also (formally) solve all of them at once. We further show how to extend all the algor ithmic approaches suggested to the original IB problem, and apply the extended algorithms to solve three diff rent IB variations with respect to several real world datasets. Nonetheless, we emphasize that additi onal applications of this multivariate framework still need be explored, and we expect that future research wi ll elucidate such examples. In the remainder of this abstract we provide a concise descri ption of the chapters and appendices that constitute this thesis. Chapter 1 forms a basic introduction to the remaining chapte rs. We start with a high level description of the fundamental precision-complexity trade-off and explain h ow the IB principle suggests a purely statistical and symmetric formulation to this trade-off. We further introd uce the basic concepts that will be used throughout this thesis, and in particular the concepts of mutual and mul ti information. In Chapter 2 we formally present the IB principle and its rela tionships to rate distortion theory. We define the concept of the relevance-compression function as a char a teristic function for a given joint distribution, p(x; y), and argue that this function can be considered as a natural e xtension to the well known rate-distortion