A Koopman Approach to Understanding Sequence Neural Models

1 INTRODUCTION . Understanding the inner workings of predictive models is an essential requirement in many fields across science and engineering . This need is even more important nowadays with the emergence of neural networks whose visualization and interpretation is inherently challenging . Indeed , modern computational neural models often lack a commonly accepted knowledge regarding their governing mathematical principles . Consequently , while deep neural networks may achieve remarkable results on various complex tasks , explaining their underlying decision mechanisms remains a challenge . The goal of this paper is to help bridge this gap by proposing a new framework for the approximation , reasoning , and understanding of sequence neural models . Sequence models are designed to handle time series data originating from images , text , audio , and other sources of information . One approach to analyzing sequence neural networks is through the theory and practice of dynamical systems ( Doya , 1993a ; Pascanu et al. , 2013 ) . For instance , the temporal asymptotic behavior of a dynamical system can be described using the local analysis of its attractor states ( Strogatz , 2018 ) . Similarly , recurrent models have been investigated in the neighborhood of their fixed points ( Sussillo & Barak , 2013 ) , leading to work that interprets trained RNNs for tasks such as sentiment analysis ( Maheswaranathan et al. , 2019 ) . However , the local nature of these methods is a limiting factor which may lead to inconsistent results . Specifically , their approach is based on fixed-point analysis which allows to study the dynamical system in the neighborhood of a fixed-point . In contrast , our approach is global—it does not depend on a set of fixed-points , and it facilitates the exploration of the dynamics near and further away from fixed points . Over the past few years , a family of data-driven methods was developed , allowing to analyze complex dynamical systems based on Koopman ( 1931 ) theory . These methods exploit a novel observation by which nonlinear systems may be globally encoded using infinite-dimensional but linear Koopman operators . In practice , Koopman-based approaches are lossy as they compute a finite-dimensional approximation of the full operator . Nevertheless , it has been shown in the fluid dynamics ( Azencot et al. , 2020 ; Mezić , 2005 ) and geometry processing ( Sharma & Ovsjanikov , 2020 ; Ovsjanikov et al. , 2012 ) communities that the dominant features of general nonlinear dynamical systems can be captured via a single matrix per system , allowing e.g. , to align time series data ( Rahamim & Talmon , 2021 ) . Thus , we pose the following research question : can we design and employ a Koopman-based approach to analyze and develop a fundamental understanding of deep neural models ? Given a trained sequence neural network and a procedure to extract its hidden states , our Koopmanbased method generates a moderate size matrix which faithfully describes the dynamics in the latent space . Unlike existing work , our approach is global and independent of a particular latent sample , and thus it can be virtually applied to any hidden state . A key advantage of our framework is that we can directly employ linear analysis tools on the approximate Koopman operator to reason about the associated neural network . In particular , we show that the eigenvectors and eigenvalues of the Koopman matrix are instrumental for understanding the decision mechanisms of the model . For instance , we show in our results that the dominant eigenvectors carry crucial semantic knowledge related to the problem at hand . Moreover , the eigenvalues represent the memory of the network as they provide a timestamp for the temporal span of the respective eigenvectors . Finally and most importantly , Koopman-based methods such as ours are backed by rich theory and practice , allowing us to exploit the recent advances in Koopman inspired techniques for the purpose of developing a comprehensive understanding of sequence neural networks . Thus , the key contribution in this work is the novel application of Koopman-based methods for understanding sequential models , and the extraction of high-level interpretable and insightful understandings on the trained networks . We focus our investigation on two learning tasks : sentiment analysis and electrocardiogram ( ECG ) classification . We will identify four eigenvectors in the sentiment analysis model whose roles are to highlight : positive words , negative words , positive pairs ( e.g. , “ not bad ” ) , and negative pairs . In addition , we demonstrate that the eigenvectors in the ECG classification task naturally identify dominant features in normal beat signals and encode them . Specifically , we show that four Koopman eigenvectors accurately capture the local extrema points of normal beat signals . These extrema points are fundamental in deciding whether a signal is normal or anomalous . Our results reinforce that the network indeed learns a robust representation of normal beat signals . Then , we will verify that the main components of the nonlinear network dynamics can be described using our Koopman matrices by measuring the difference in accuracy results , and the relative error in predicted states . Further , we provide additional results and comparisons in the supplementary material . Given the versatility of our framework and its ease of use , we advocate its utility in the analysis and understanding of neural networks , and we believe it may also affect the design and training of deep models in the future . 2 RELATED WORK . Recurrent Neural Networks ( RNN ) and Dynamical Systems ( DS ) . Fully connected recurrent neural networks are universal approximators of arbitrary dynamical systems ( Doya , 1993b ) . Unfortunately , RNNs are well-known to be difficult to train ( Bengio et al. , 1993 ; Pascanu et al. , 2013 ) , and several methods adopt a DS perspective to improve training via gradient clipping ( Pascanu et al. , 2013 ) , and constraining the weights ( Erichson et al. , 2021 ) , among other approaches . Overall , it is clear that dynamical systems are fundamental in investigating and developing recurrent networks . Understanding RNN . Establishing a deeper understanding of recurrent networks is a long standing challenge in machine learning . To this end , Karpathy et al . ( 2015 ) follow the outputs of the model to identify units which track brackets , line lengths , and quotes . Recently , Chefer et al . ( 2020 ) proposed an approach for computing relevance scores of transformer networks . Perhaps mostly related to our approach is the analysis of recurrent models around their fixed points ( Sussillo & Barak , 2013 ) . This approach revealed low-dimensional attractors in the sentiment analysis task ( Maheswaranathan et al. , 2019 ) , which allowed to deduce simple explanations of the decision mechanisms of the associated models . Our work generalizes the approach of Sussillo & Barak ( 2013 ) in that it yields global results about the dynamics , and it introduces several novel features . We provide a more detailed comparison between our method and theirs in Sec . 4 . Koopman-based Neural Networks . Recently , several techniques that combine neural networks and Koopman theory were proposed , mostly in the context of predicting nonlinear dynamics . For example , Takeishi et al . ( 2017 ) ; Morton et al . ( 2018 ) optimize the residual sum of squares of the predictions the operator makes , Lusch et al . ( 2018 ) ; Erichson et al . ( 2019 ) ; Azencot et al . ( 2020 ) design dynamic autoencoders whose central component is linear and may be structured , Li et al . ( 2020 ) employ graph networks , and Mardt et al . ( 2018 ) use a variational approach on Markov processes . A recent line of work aims at exploiting tools from Koopman theory to analyze and improve the training process of neural networks ( Dietrich et al. , 2020 ; Dogra & Redman , 2020 ; Manojlović et al. , 2020 ) . To the best of our knowledge , our work is first to employ a Koopman-based method towards the analysis and understanding of trained neural networks . 3 METHOD . In what follows , we present our method for analyzing and understanding sequence neural models . Importantly , while we mostly discuss and experiment with recurrent neural networks , our approach is quite general and applicable to any model whose inner representation is a time series . We consider neural models that take input instances xt ∈ Rm at time t and compute ht = F ( ht−1 , xt ) , t = 1 , 2 , ... , ( 1 ) where ht ∈ Rk is a ( hidden ) state that represents the latent dynamics , and F is some nonlinear function that pushes states through time . In our analysis , we use only the hidden states set and discard the time series input . Thus , our method is a “ white-box ” approach as we assume access to { ht } , which is typically possible in most day-to-day scenarios . Importantly , all recurrent models including vanilla RNN ( Elman , 1990 ) , LSTM ( Hochreiter & Schmidhuber , 1997 ) , and GRU ( Cho et al. , 2014 ) , as well as Attention Models ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , and Residual neural networks ( He et al. , 2016 ) exhibit the structure of Eq . ( 1 ) . 3.1 ESSENTIALS OF KOOPMAN THEORY . Our approach is based on Koopman ( 1931 ) theory which was developed for dynamical systems . The key observation of Koopman was that a finite-dimensional nonlinear dynamics can be fully represented using an infinite-dimensional but linear operator . While the theoretical background is essential for developing a deep understanding of Koopman-based approaches , the practical aspects are more important to this work . Thus , we briefly recall the definition of the Koopman operator , and we refer the reader to other , comprehensive works on the subject ( Singh & Manhas , 1993 ; Eisner et al. , 2015 ) . Formally , we assume a discrete-time dynamical system ϕ acting on a compact , inner-product spaceM⊂ Rm , zt+1 = ϕ ( zt ) , zt ∈M , t = 1 , 2 , ... , ( 2 ) where t is an integer index representing discrete time . The dynamics ϕ induces a linear operator Kϕ which we call the Koopman operator , and it is given by Kϕf ( zt ) : = f ( zt+1 ) = f ◦ ϕ ( zt ) , ( 3 ) where f : M→ R is a scalar function in a bounded inner product space F . It is easy to show that Kϕ is linear due to the linearity of composition , i.e. , given α , β ∈ R and f , g ∈ F , we obtain that Kϕ ( αf +βg ) = ( αf +βg ) ◦ϕ = αf ◦ϕ+βg ◦ϕ = αKϕ ( f ) +βKϕ ( g ) . We emphasize that while ϕ describes the system evolution , Kϕ is a transformation on the space of functions . From a practical viewpoint , these functions may be interpreted as observations of the system , such as velocity , sea level , temperature , or hidden states in our setup . To justify our use of Koopman theory and practice in the context of neural networks , we propose the following . We interpret the input sequence { xt } as governed by some complex and unknown dynamics ϕ , i.e. , xt+1 = ϕ ( xt ) for every t. We emphasize that ϕ is different from F in Eq . ( 1 ) by its definition of domain and range . Then , the hidden states ht are finite samplings of observations of the system , namely , ht ≈ ft where ft : M→ R is the true observation . For instance , ft may be the smooth function cos ( tz ) , whereas ht ∈ Rk is its sampling at a finite set of points { z1 , . . . , zk } . It follows that { ht } is subject to an approximate Koopman representation . However , a fundamental challenge in facilitating Koopman theory in practice is the infinite-dimensionality of Kϕ . Recently , several data-driven methods were developed to produce a better approximate Kϕ using a moderatesize matrix C ( Schmid , 2010 ; Ovsjanikov et al. , 2012 ) . In particular , Koopman-based approaches have been proven instrumental in the analysis of fluid dynamics data ( Brunton et al. , 2021 ) , and for computing complex nonrigid isometric maps between shapes ( Ovsjanikov et al. , 2016 ) . Motivated by these empirical examples and their success , we will compute in this work approximate Koopman operator matrices C such that they encode the evolution of latent states { ht } .

Summary: The authors propose a method for analyzing NN applied to temporal signals (e.g., dynamical systems). The proposed method relies on the Koopman operator, which is widely used for studying dynamical systems. The authors use the eigenvectors of the Koopman operator to interpret predictions made by NN. The proposed method is evaluated on two tasks, namely: sentiment analysis and electrocardiogram (ECG) classification. The authors demonstrate that the eigenvectors of the Koopman operator do encode the latent information in these examples.