Learning Information Propagation in the Dynamical Systems via Information Bottleneck Hierarchy

1 INTRODUCTION . The use of machine learning for making inference and prediction from the real-world data has shown unprecedented growth . There exist a plethora of approaches for complex system ( CS ) modeling ( e.g. , multi-input multi-output state space identification ( Stoica & Jansson , 2000 ) , expectation maximization ( EM ) ( Martens , 2010 ) , regularization ( Chiuso , 2016 ) , graphical models ( Meinshausen & Buhlmann , 2006 ) , combined regularization and Bayesian learning ( Fox et al. , 2008 ; 2011 ; Bonettini et al. , 2015 ) , kernel-based regularization ( Pillonetto & Chiuso , 2015 ) ) . With the increase in the size of data , the complexity of the accurate models also increases , making inference and predictions slower . The major challenges of the upcoming era , hence , are likely to deal with the massive and diverse data sources , and still making quick decisions . Therefore , the compact modeling of time-varying complex systems1 is a challenging task and appealing for more investigation . The real-world data has complex inter-dependencies across spatial and temporal dimensions . We aim to identify such dependencies and carefully construct a compact representation of the given CS model while still ensuring accurate predictions . We do so by performing a soft-clustering of such inter-dependencies to preserve only the relevant information . For the CS model in the form of a dynamical system , we additionally argue that similar to the data , the most relevant information also gets transformed at each hop in an alternate dynamical system . From a bird ’ s-eye view , we track 1In this work , we interpret complex systems ( networks ) ( Barrat et al. , 2008 ; Gao et al. , 2014 ) as graphs comprising of nodes interacting spatially and temporally , i.e . both inter and intra dependence , with node activities available in the form of time-series . how the most relevant information propagates across the given dynamical system . We represent this propagation via an alternate dynamical system ( compact model ) and develop an unsupervised learning technique of such process . The most relevant work in this regard is information bottleneck ( IB ) principle ( Tishby et al. , 2000 ) . For fixed two random variables , it performs a soft-clustering to compress one variable while predicting another , given the joint probability distribution . The IB has been successfully applied to speech recognition ( Hecht & Tishby , 2005 ) , document classification ( Slonim & Tishby , 2000 ) , gene expression ( Friedman et al. , 2001 ) and deep learning ( Tishby & Zaslavsky , 2015 ) , etc. , and it has shown good performance . In contrast , we aim to learn a dynamics of the soft-clustering across the given dynamical system , and propose a general optimization framework to study the trade-offs between compactness and the resulting accuracies . The problem statement addressed in this work is : Given a dynamical system , we aim to develop a compact model by learning the dynamics of the soft-clustering in an unsupervised manner , or alternate dynamical process , through Information Bottleneck hierarchy ( IBH ) . The main contributions of the present work are as follows : ( i ) By learning the dynamics of the soft-clustering , we propose an alternate compact dynamical system of the given process , with emphasis on the prediction accuracies . ( ii ) We formulate a novel optimization setup , compact perception problem , and characterize general solution to the information theoretic problem . ( iii ) We quantify how most relevant information about future gets transformed at each hop in the alternatively designed dynamical system . A brief mention of the mathematical notations is provided in the next part . 1.1 DEFINITIONS AND NOTATIONS . In this manuscript , we use capital letters to denote random variables ( RVs ) , and lowercase letters are used for the realizations . The bold letters are used for multi-variate RVs . For a RV X , with little abuse of notation , we denote the probability mass function pX ( x ) as p ( X ) , unless specified otherwise . The expectation operator is denoted as E [ . ] . A Gaussian distributed multi-variate RV is denoted as X ∼ N ( µX , ΣX ) , where µX and ΣX are the mean vector and covariance matrix , respectively . Next , we present a few information theoretic definitions relevant to this work . Definition 1 . The Kullback-Leibler ( KL ) divergence ( Cover & Thomas , 1991 ) between two probability mass functions p ( · ) and q ( · ) is written as DKL ( p||q ) = ∑ x p ( x ) log p ( x ) q ( x ) . ( 1 ) The KL divergence is in general not symmetric , and DKL ≥ 0 with DKL ( p||q ) = 0 if and only if p = q . Using ( 1 ) , the mutual information between two RVs X and Y is defined as I ( X ; Y ) = DKL ( p ( X , Y ) ||p ( X ) p ( Y ) ) . Next , we state the problem statement addressed in this work . 2 PROBLEM FORMULATION . Complex systems consist of a large number of interacting dynamic components . In practical situations , we have high-dimensional time-series ( node activities ) having complex spatial and temporal correlations . The challenge lies in identifying such complicated inter-dependencies and recover the true dimensionality of the system in an unsupervised manner . We equivalently call it as identifying the compact model while preserving the relevant information across spatio-temporal states . Under stationary assumptions , ( Tishby et al. , 2000 ) proposed the IB to process information compactly . For the sake of completeness , we provide a brief overview of the IB principle . 2.1 INFORMATION BOTTLENECK APPROACH . The IB compresses a variableX into a new stochastic variableB via soft-clustering , while maintaining as much information as possible about another variable of interest Y . The variableB operates to minimize the information compression task and to maximize the relevant information . Provided that the relevance of one RV to another is measurable by an information-theoretic metric ( i.e. , mutual information ) , the trade-off can be written as the following variational problem min p ( B|X ) I ( X ; B ) − βI ( B ; Y ) , ( 2 ) where β controls the trade-off between the tasks mentioned above . Hence , the variable B , solving the minimization problem ( 2 ) , encodes the most informative part from the input X about output Y . Inspired by the IB concept and its internal predictive coding , we propose a causal inference framework for discrete-time stochastic dynamical systems . More precisely , we label two consecutive dynamic states Xk and Xk+1 to be the input and the output of the bottleneck Bk , respectively . Hence , Bk carries the most informative part ( relevant information ) from the past about the future . Next , we formalize this idea to create a sequence of bottlenecks ( or dynamics of soft-clustering ) to compactly and accurately represent the given dynamical system . 2.2 LEARNING INFORMATION PROPAGATION : SETUP . We consider a stochastic dynamical system , involving large number n of random processes X = [ X1 , X2 , .... Xn ] , interacting and evolving in time . A fundamental problem for decision making or prediction is to learn a compressed representation of the system dynamics from high-dimensional data ( i.e. , system identification ) . The higher-order dependencies hiding behind the high-dimensions make this task challenging , and it needs sophisticated techniques to extract meaningful information for an appropriate application . In this paper , we propose a framework that focuses only on the dynamics of the relevant information , by learning an alternate representation of the given process . Adopting an information theoretic representation , we aim to determine Bk and Bk+1 jointly since they provide compressed predictive information about the system dynamics . Figure 1 summarizes our objectives , and we study the shaded region ( 3 states ) of the dynamical system in isolation , i.e . without the influence of any other RVs in the complete system . An argument to generalize this study to any N -length is provided later in Section 3.1 . We determine the stochastic variable Bk that not only compresses Xk−1 as much as possible while preserving the relevant information about Xk , but also delivers this information to Bk+1 . The variable Bk+1 quantifies the meaningful information about the state of the system at time k + 1 , building upon the compressed information received via Bk , thus forming a dynamics of the relevant information . By construction , the new mapping Bk is designed from Xk−1 to preserve the consistent information about Xk , therefore , given Xk−1 , Bk and Xk are independent . Similarly , the mapping Bk+1 is independent of Xk+1 given Bk . Since , Bk+1 carries information from Bk which is compressed representation of Xk−1 , then given Bk , Xk−1 and Bk+1 are also independent . With this framework , the following Markov chains are considered : Xk–Xk−1–Bk , Xk+1–Bk–Bk+1 , Xk−1–Bk–Bk+1 and Bk–Xk−1–Xk+1 . The trade-off between the compression and preservation of relevant information is defined as the minimum achievable rate I ( Xk−1 ; Bk ) subject to constraints on the information processing . We call it a compact perception problem which determines a trade-off between compression representation and predictive characteristics . Formally , this can be written as the following optimization . min p ( Bk|Xk−1 ) , p ( Bk+1|Bk ) I ( Xk−1 ; Bk ) subject to I ( Xk−1 ; Xk ) − I ( Bk ; Xk ) ≤ 1 I ( Bk ; Bk+1 ) − I ( Xk−1 ; Xk ) ≤ 2 I ( Xk−1 ; Xk+1 ) − I ( Xk+1 ; Bk+1 ) ≤ 3 ( 3 ) where the constraints characterize the bounds on the desired prediction/compression at each step . We wish to lower bound ( to guarantee ) the prediction accuracies via lower bounding I ( Bk ; Xk ) and I ( Xk+1 ; Bk+1 ) , and upper bound ( to limit ) the compression level across hop via I ( Bk ; Bk+1 ) . For example , 1 bounds the accuracy of the prediction of Xk by Bk . The information flow across alternate dynamical process is controlled by 2 . Lastly , 3 defines closeness of prediction of Xk+1 by Bk+1 . We show in the Section 3.1 , and results in Section 4 , that such trade-off can be alternatively studied by choice of the Lagrange parameters . 3 LEARNING INFORMATION PROCESSING FOR DYNAMICAL SYSTEMS . This section provides the main results concerning solving the compact perception optimization problem in equation ( 3 ) under the most general case . Next , this general result is used to study a highdimensional continuous Gaussian distribution . Lastly , we describe an iterative method to update the corresponding parameters . 3.1 LEARNING INFORMATION PROPAGATION : SOLUTION . Finding the alternate dynamical representation , or Bk and Bk+1 , as stated in ( 3 ) requires to solve a variational problem . To solve this problem , we introduce the Lagrange multipliers β , λ and γ for the information processing constraints . Hence , we find the alternative representation ( or IBH ) by minimizing the following functional written using ( 3 ) : F [ p ( Bk|Xk−1 ) , p ( Bk ) , p ( Xk|Bk ) , p ( Xk+1|Bk ) , p ( Bk+1|Bk ) , p ( Bk+1 ) ] = I ( Xk−1 ; Bk ) − βI ( Bk ; Xk ) + λ ( I ( Bk ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) ) . ( 4 ) We can argue the following regarding the optimal solution which minimizes the equation ( 4 ) . Theorem 1 . The optimal solution that minimizes functional F in ( 4 ) satisfy the following selfconsistent equations : p ( Bk|Xk−1 ) = p ( Bk ) Z1 × exp { −βDKL ( p ( Xk|Xk−1 ) ||p ( Xk|Bk ) ) −λDKL ( p ( Bk+1|Bk ) ||p ( Bk+1 ) ) −λγEBk+1|Bk [ DKL ( p ( Xk+1|Xk−1 ) ||p ( Xk+1|Bk+1 ) ) ] } , ( 5 ) p ( Bk+1|Bk ) = p ( Bk+1 ) Z2 exp { −γDKL ( p ( Xk+1|Bk ) ||p ( Xk+1|Bk+1 ) ) } , ( 6 ) where Z1 and Z2 are normalizing partition functions . The functional F in ( 4 ) may not be convex in the product space of the associated probability simplexes . Hence , it is difficult to obtain a global optimum , however , a stationary point ( and locally optimal solution , in most of the cases ) can be obtained using the following result . Corollary 1 . The self-consistent equations in Theorem 1 can be used to write an iterative procedure to update the associated probabilities as equations ( 14 ) - ( 20 ) . The iterative approach of Corollary 1 is detailed in the Appendix B . The idea is similar to the BlahutArimoto algorithm ( Arimoto , 1972 ; Blahut , 1972 ) , also observed in ( Tishby et al. , 2000 ) . Finally , it remains to show the existence of a stationary point of the functional in ( 4 ) . The convergence of the iterations in Corollary 1 is established through the following result . Lemma 1 . The iterative procedure in Corollary 1 to minimize the functional F in ( 4 ) is convergent to a stationary point . The idea of proof ( in Appendix C ) is also somewhat similar to the EM McLachlan & Krishnan ( 1996 ) but with minimization of the functional . As in standard EM , in most of the cases , the stationary convergence point is local minimum ( maximum in EM ) of the functional . Corollary 2 . The IBH solution in Theorem 1 reduces to IB in ( 2 ) upon setting Xk−1 = Xk , and β →∞ . Proof . With Xk−1 = Xk , the functional of IBH in equation ( 4 ) can be written as F = ( 1− β ) I ( Xk−1 ; Bk ) + λ ( I ( Bk ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) ) . The functional F minimization in the limit of β → ∞ has one solution of p ( Bk|Xk−1 ) = 1Bk=Xk−1 ( such that I ( Xk−1 ; Bk ) > 0 ) , and the problem reduces to minimize the following F̂ = I ( Xk−1 ; Bk+1 ) − γI ( Xk+1 ; Bk+1 ) , where λ can be dropped , as λ ≥ 0 . An advantage of using the optimization framework in ( 3 ) is that it can be generalized to any length of the given input dynamical process by properly repeating the second and third constraint . In this work , we have studied the case of length three of the input process . However , the same principles can be used to write the solution for any length N of the dynamical system ( the complete Figure 1 ) , as will be presented in the future work .

Summary: This paper studies the problem of compactly represent the model of a complex dynamic system while preserving information. The method is based on the information bottleneck method. Basically, for a dynamic system whose states changing from X_{k-1}, X_k to X_{k+1}, the "information bottleneck hierarchy" method learns a variable B_k and B_{k+1} such that B_k predicts B_{k+1} well, B_k predicts X_k well, and B_{k+1} predicts X_{k+1} well, while minimizing the information of X_{k-1} contained in B_k.