Learning Information Propagation in the Dynamical Systems via Information Bottleneck Hierarchy

1 INTRODUCTION . The use of machine learning for making inference and prediction from the real-world data has shown unprecedented growth . There exist a plethora of approaches for complex system ( CS ) modeling ( e.g. , multi-input multi-output state space identification ( Stoica & Jansson , 2000 ) , expectation maximization ( EM ) ( Martens , 2010 ) , regularization ( Chiuso , 2016 ) , graphical models ( Meinshausen & Buhlmann , 2006 ) , combined regularization and Bayesian learning ( Fox et al. , 2008 ; 2011 ; Bonettini et al. , 2015 ) , kernel-based regularization ( Pillonetto & Chiuso , 2015 ) ) . With the increase in the size of data , the complexity of the accurate models also increases , making inference and predictions slower . The major challenges of the upcoming era , hence , are likely to deal with the massive and diverse data sources , and still making quick decisions . Therefore , the compact modeling of time-varying complex systems1 is a challenging task and appealing for more investigation . The real-world data has complex inter-dependencies across spatial and temporal dimensions . We aim to identify such dependencies and carefully construct a compact representation of the given CS model while still ensuring accurate predictions . We do so by performing a soft-clustering of such inter-dependencies to preserve only the relevant information . For the CS model in the form of a dynamical system , we additionally argue that similar to the data , the most relevant information also gets transformed at each hop in an alternate dynamical system . From a bird ’ s-eye view , we track 1In this work , we interpret complex systems ( networks ) ( Barrat et al. , 2008 ; Gao et al. , 2014 ) as graphs comprising of nodes interacting spatially and temporally , i.e . both inter and intra dependence , with node activities available in the form of time-series . how the most relevant information propagates across the given dynamical system . We represent this propagation via an alternate dynamical system ( compact model ) and develop an unsupervised learning technique of such process . The most relevant work in this regard is information bottleneck ( IB ) principle ( Tishby et al. , 2000 ) . For fixed two random variables , it performs a soft-clustering to compress one variable while predicting another , given the joint probability distribution . The IB has been successfully applied to speech recognition ( Hecht & Tishby , 2005 ) , document classification ( Slonim & Tishby , 2000 ) , gene expression ( Friedman et al. , 2001 ) and deep learning ( Tishby & Zaslavsky , 2015 ) , etc. , and it has shown good performance . In contrast , we aim to learn a dynamics of the soft-clustering across the given dynamical system , and propose a general optimization framework to study the trade-offs between compactness and the resulting accuracies . The problem statement addressed in this work is : Given a dynamical system , we aim to develop a compact model by learning the dynamics of the soft-clustering in an unsupervised manner , or alternate dynamical process , through Information Bottleneck hierarchy ( IBH ) . The main contributions of the present work are as follows : ( i ) By learning the dynamics of the soft-clustering , we propose an alternate compact dynamical system of the given process , with emphasis on the prediction accuracies . ( ii ) We formulate a novel optimization setup , compact perception problem , and characterize general solution to the information theoretic problem . ( iii ) We quantify how most relevant information about future gets transformed at each hop in the alternatively designed dynamical system . A brief mention of the mathematical notations is provided in the next part . 1.1 DEFINITIONS AND NOTATIONS . In this manuscript , we use capital letters to denote random variables ( RVs ) , and lowercase letters are used for the realizations . The bold letters are used for multi-variate RVs . For a RV X , with little abuse of notation , we denote the probability mass function pX ( x ) as p ( X ) , unless specified otherwise . The expectation operator is denoted as E [ . ] . A Gaussian distributed multi-variate RV is denoted as X ∼ N ( µX , ΣX ) , where µX and ΣX are the mean vector and covariance matrix , respectively . Next , we present a few information theoretic definitions relevant to this work . Definition 1 . The Kullback-Leibler ( KL ) divergence ( Cover & Thomas , 1991 ) between two probability mass functions p ( · ) and q ( · ) is written as DKL ( p||q ) = ∑ x p ( x ) log p ( x ) q ( x ) . ( 1 ) The KL divergence is in general not symmetric , and DKL ≥ 0 with DKL ( p||q ) = 0 if and only if p = q . Using ( 1 ) , the mutual information between two RVs X and Y is defined as I ( X ; Y ) = DKL ( p ( X , Y ) ||p ( X ) p ( Y ) ) . Next , we state the problem statement addressed in this work . 2 PROBLEM FORMULATION . Complex systems consist of a large number of interacting dynamic components . In practical situations , we have high-dimensional time-series ( node activities ) having complex spatial and temporal correlations . The challenge lies in identifying such complicated inter-dependencies and recover the true dimensionality of the system in an unsupervised manner . We equivalently call it as identifying the compact model while preserving the relevant information across spatio-temporal states . Under stationary assumptions , ( Tishby et al. , 2000 ) proposed the IB to process information compactly . For the sake of completeness , we provide a brief overview of the IB principle . 2.1 INFORMATION BOTTLENECK APPROACH . The IB compresses a variableX into a new stochastic variableB via soft-clustering , while maintaining as much information as possible about another variable of interest Y . The variableB operates to minimize the information compression task and to maximize the relevant information . Provided that the relevance of one RV to another is measurable by an information-theoretic metric ( i.e. , mutual information ) , the trade-off can be written as the following variational problem min p ( B|X ) I ( X ; B ) − βI ( B ; Y ) , ( 2 ) where β controls the trade-off between the tasks mentioned above . Hence , the variable B , solving the minimization problem ( 2 ) , encodes the most informative part from the input X about output Y . Inspired by the IB concept and its internal predictive coding , we propose a causal inference framework for discrete-time stochastic dynamical systems . More precisely , we label two consecutive dynamic states Xk and Xk+1 to be the input and the output of the bottleneck Bk , respectively . Hence , Bk carries the most informative part ( relevant information ) from the past about the future . Next , we formalize this idea to create a sequence of bottlenecks ( or dynamics of soft-clustering ) to compactly and accurately represent the given dynamical system . 2.2 LEARNING INFORMATION PROPAGATION : SETUP . We consider a stochastic dynamical system , involving large number n of random processes X = [ X1 , X2 , .... Xn ] , interacting and evolving in time . A fundamental problem for decision making or prediction is to learn a compressed representation of the system dynamics from high-dimensional data ( i.e. , system identification ) . The higher-order dependencies hiding behind the high-dimensions make this task challenging , and it needs sophisticated techniques to extract meaningful information for an appropriate application . In this paper , we propose a framework that focuses only on the dynamics of the relevant information , by learning an alternate representation of the given process . Adopting an information theoretic representation , we aim to determine Bk and Bk+1 jointly since they provide compressed predictive information about the system dynamics . Figure 1 summarizes our objectives , and we study the shaded region ( 3 states ) of the dynamical system in isolation , i.e . without the influence of any other RVs in the complete system . An argument to generalize this study to any N -length is provided later in Section 3.1 . We determine the stochastic variable Bk that not only compresses Xk−1 as much as possible while preserving the relevant information about Xk , but also delivers this information to Bk+1 . The variable Bk+1 quantifies the meaningful information about the state of the system at time k + 1 , building upon the compressed information received via Bk , thus forming a dynamics of the relevant information . By construction , the new mapping Bk is designed from Xk−1 to preserve the consistent information about Xk , therefore , given Xk−1 , Bk and Xk are independent . Similarly , the mapping Bk+1 is independent of Xk+1 given Bk . Since , Bk+1 carries information from Bk which is compressed representation of Xk−1 , then given Bk , Xk−1 and Bk+1 are also independent . With this framework , the following Markov chains are considered : Xk–Xk−1–Bk , Xk+1–Bk–Bk+1 , Xk−1–Bk–Bk+1 and Bk–Xk−1–Xk+1 . The trade-off between the compression and preservation of relevant information is defined as the minimum achievable rate I ( Xk−1 ; Bk ) subject to constraints on the information processing . We call it a compact perception problem which determines a trade-off between compression representation and predictive characteristics . Formally , this can be written as the following optimization . min p ( Bk|Xk−1 ) , p ( Bk+1|Bk ) I ( Xk−1 ; Bk ) subject to I ( Xk−1 ; Xk ) − I ( Bk ; Xk ) ≤ 1 I ( Bk ; Bk+1 ) − I ( Xk−1 ; Xk ) ≤ 2 I ( Xk−1 ; Xk+1 ) − I ( Xk+1 ; Bk+1 ) ≤ 3 ( 3 ) where the constraints characterize the bounds on the desired prediction/compression at each step . We wish to lower bound ( to guarantee ) the prediction accuracies via lower bounding I ( Bk ; Xk ) and I ( Xk+1 ; Bk+1 ) , and upper bound ( to limit ) the compression level across hop via I ( Bk ; Bk+1 ) . For example , 1 bounds the accuracy of the prediction of Xk by Bk . The information flow across alternate dynamical process is controlled by 2 . Lastly , 3 defines closeness of prediction of Xk+1 by Bk+1 . We show in the Section 3.1 , and results in Section 4 , that such trade-off can be alternatively studied by choice of the Lagrange parameters . 3 LEARNING INFORMATION PROCESSING FOR DYNAMICAL SYSTEMS . This section provides the main results concerning solving the compact perception optimization problem in equation ( 3 ) under the most general case . Next , this general result is used to study a highdimensional continuous Gaussian distribution . Lastly , we describe an iterative method to update the corresponding parameters . 3.1 LEARNING INFORMATION PROPAGATION : SOLUTION . Finding the alternate dynamical representation , or Bk and Bk+1 , as stated in ( 3 ) requires to solve a variational problem . To solve this problem , we introduce the Lagrange multipliers β , λ and γ for the information processing constraints . Hence , we find the alternative representation ( or IBH ) by minimizing the following functional written using ( 3 ) : F [ p ( Bk|Xk−1 ) , p ( Bk ) , p ( Xk|Bk ) , p ( Xk+1|Bk ) , p ( Bk+1|Bk ) , p ( Bk+1 ) ] = I ( Xk−1 ; Bk ) − βI ( Bk ; Xk ) + λ ( I ( Bk ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) ) . ( 4 ) We can argue the following regarding the optimal solution which minimizes the equation ( 4 ) . Theorem 1 . The optimal solution that minimizes functional F in ( 4 ) satisfy the following selfconsistent equations : p ( Bk|Xk−1 ) = p ( Bk ) Z1 × exp { −βDKL ( p ( Xk|Xk−1 ) ||p ( Xk|Bk ) ) −λDKL ( p ( Bk+1|Bk ) ||p ( Bk+1 ) ) −λγEBk+1|Bk [ DKL ( p ( Xk+1|Xk−1 ) ||p ( Xk+1|Bk+1 ) ) ] } , ( 5 ) p ( Bk+1|Bk ) = p ( Bk+1 ) Z2 exp { −γDKL ( p ( Xk+1|Bk ) ||p ( Xk+1|Bk+1 ) ) } , ( 6 ) where Z1 and Z2 are normalizing partition functions . The functional F in ( 4 ) may not be convex in the product space of the associated probability simplexes . Hence , it is difficult to obtain a global optimum , however , a stationary point ( and locally optimal solution , in most of the cases ) can be obtained using the following result . Corollary 1 . The self-consistent equations in Theorem 1 can be used to write an iterative procedure to update the associated probabilities as equations ( 14 ) - ( 20 ) . The iterative approach of Corollary 1 is detailed in the Appendix B . The idea is similar to the BlahutArimoto algorithm ( Arimoto , 1972 ; Blahut , 1972 ) , also observed in ( Tishby et al. , 2000 ) . Finally , it remains to show the existence of a stationary point of the functional in ( 4 ) . The convergence of the iterations in Corollary 1 is established through the following result . Lemma 1 . The iterative procedure in Corollary 1 to minimize the functional F in ( 4 ) is convergent to a stationary point . The idea of proof ( in Appendix C ) is also somewhat similar to the EM McLachlan & Krishnan ( 1996 ) but with minimization of the functional . As in standard EM , in most of the cases , the stationary convergence point is local minimum ( maximum in EM ) of the functional . Corollary 2 . The IBH solution in Theorem 1 reduces to IB in ( 2 ) upon setting Xk−1 = Xk , and β →∞ . Proof . With Xk−1 = Xk , the functional of IBH in equation ( 4 ) can be written as F = ( 1− β ) I ( Xk−1 ; Bk ) + λ ( I ( Bk ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) ) . The functional F minimization in the limit of β → ∞ has one solution of p ( Bk|Xk−1 ) = 1Bk=Xk−1 ( such that I ( Xk−1 ; Bk ) > 0 ) , and the problem reduces to minimize the following F̂ = I ( Xk−1 ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) , where λ can be dropped , as λ ≥ 0 . An advantage of using the optimization framework in ( 3 ) is that it can be generalized to any length of the given input dynamical process by properly repeating the second and third constraint . In this work , we have studied the case of length three of the input process . However , the same principles can be used to write the solution for any length N of the dynamical system ( the complete Figure 1 ) , as will be presented in the future work .

Summary: This paper studied an extension of the Information Bottleneck Principle called Information Bottleneck Hierarchy (IBH).  The goal of IBH is to extract meaningful information from a Markov Chain. Then the authors studied case of the Gaussian linear dynamic and proposed an algorithm for computing the IBH. Then an experiment was conducted to show the usage of IBH to practical problems.