A Koopman Approach to Understanding Sequence Neural Models

1 INTRODUCTION . Understanding the inner workings of predictive models is an essential requirement in many fields across science and engineering . This need is even more important nowadays with the emergence of neural networks whose visualization and interpretation is inherently challenging . Indeed , modern computational neural models often lack a commonly accepted knowledge regarding their governing mathematical principles . Consequently , while deep neural networks may achieve remarkable results on various complex tasks , explaining their underlying decision mechanisms remains a challenge . The goal of this paper is to help bridge this gap by proposing a new framework for the approximation , reasoning , and understanding of sequence neural models . Sequence models are designed to handle time series data originating from images , text , audio , and other sources of information . One approach to analyzing sequence neural networks is through the theory and practice of dynamical systems ( Doya , 1993a ; Pascanu et al. , 2013 ) . For instance , the temporal asymptotic behavior of a dynamical system can be described using the local analysis of its attractor states ( Strogatz , 2018 ) . Similarly , recurrent models have been investigated in the neighborhood of their fixed points ( Sussillo & Barak , 2013 ) , leading to work that interprets trained RNNs for tasks such as sentiment analysis ( Maheswaranathan et al. , 2019 ) . However , the local nature of these methods is a limiting factor which may lead to inconsistent results . Specifically , their approach is based on fixed-point analysis which allows to study the dynamical system in the neighborhood of a fixed-point . In contrast , our approach is global—it does not depend on a set of fixed-points , and it facilitates the exploration of the dynamics near and further away from fixed points . Over the past few years , a family of data-driven methods was developed , allowing to analyze complex dynamical systems based on Koopman ( 1931 ) theory . These methods exploit a novel observation by which nonlinear systems may be globally encoded using infinite-dimensional but linear Koopman operators . In practice , Koopman-based approaches are lossy as they compute a finite-dimensional approximation of the full operator . Nevertheless , it has been shown in the fluid dynamics ( Azencot et al. , 2020 ; Mezić , 2005 ) and geometry processing ( Sharma & Ovsjanikov , 2020 ; Ovsjanikov et al. , 2012 ) communities that the dominant features of general nonlinear dynamical systems can be captured via a single matrix per system , allowing e.g. , to align time series data ( Rahamim & Talmon , 2021 ) . Thus , we pose the following research question : can we design and employ a Koopman-based approach to analyze and develop a fundamental understanding of deep neural models ? Given a trained sequence neural network and a procedure to extract its hidden states , our Koopmanbased method generates a moderate size matrix which faithfully describes the dynamics in the latent space . Unlike existing work , our approach is global and independent of a particular latent sample , and thus it can be virtually applied to any hidden state . A key advantage of our framework is that we can directly employ linear analysis tools on the approximate Koopman operator to reason about the associated neural network . In particular , we show that the eigenvectors and eigenvalues of the Koopman matrix are instrumental for understanding the decision mechanisms of the model . For instance , we show in our results that the dominant eigenvectors carry crucial semantic knowledge related to the problem at hand . Moreover , the eigenvalues represent the memory of the network as they provide a timestamp for the temporal span of the respective eigenvectors . Finally and most importantly , Koopman-based methods such as ours are backed by rich theory and practice , allowing us to exploit the recent advances in Koopman inspired techniques for the purpose of developing a comprehensive understanding of sequence neural networks . Thus , the key contribution in this work is the novel application of Koopman-based methods for understanding sequential models , and the extraction of high-level interpretable and insightful understandings on the trained networks . We focus our investigation on two learning tasks : sentiment analysis and electrocardiogram ( ECG ) classification . We will identify four eigenvectors in the sentiment analysis model whose roles are to highlight : positive words , negative words , positive pairs ( e.g. , “ not bad ” ) , and negative pairs . In addition , we demonstrate that the eigenvectors in the ECG classification task naturally identify dominant features in normal beat signals and encode them . Specifically , we show that four Koopman eigenvectors accurately capture the local extrema points of normal beat signals . These extrema points are fundamental in deciding whether a signal is normal or anomalous . Our results reinforce that the network indeed learns a robust representation of normal beat signals . Then , we will verify that the main components of the nonlinear network dynamics can be described using our Koopman matrices by measuring the difference in accuracy results , and the relative error in predicted states . Further , we provide additional results and comparisons in the supplementary material . Given the versatility of our framework and its ease of use , we advocate its utility in the analysis and understanding of neural networks , and we believe it may also affect the design and training of deep models in the future . 2 RELATED WORK . Recurrent Neural Networks ( RNN ) and Dynamical Systems ( DS ) . Fully connected recurrent neural networks are universal approximators of arbitrary dynamical systems ( Doya , 1993b ) . Unfortunately , RNNs are well-known to be difficult to train ( Bengio et al. , 1993 ; Pascanu et al. , 2013 ) , and several methods adopt a DS perspective to improve training via gradient clipping ( Pascanu et al. , 2013 ) , and constraining the weights ( Erichson et al. , 2021 ) , among other approaches . Overall , it is clear that dynamical systems are fundamental in investigating and developing recurrent networks . Understanding RNN . Establishing a deeper understanding of recurrent networks is a long standing challenge in machine learning . To this end , Karpathy et al . ( 2015 ) follow the outputs of the model to identify units which track brackets , line lengths , and quotes . Recently , Chefer et al . ( 2020 ) proposed an approach for computing relevance scores of transformer networks . Perhaps mostly related to our approach is the analysis of recurrent models around their fixed points ( Sussillo & Barak , 2013 ) . This approach revealed low-dimensional attractors in the sentiment analysis task ( Maheswaranathan et al. , 2019 ) , which allowed to deduce simple explanations of the decision mechanisms of the associated models . Our work generalizes the approach of Sussillo & Barak ( 2013 ) in that it yields global results about the dynamics , and it introduces several novel features . We provide a more detailed comparison between our method and theirs in Sec . 4 . Koopman-based Neural Networks . Recently , several techniques that combine neural networks and Koopman theory were proposed , mostly in the context of predicting nonlinear dynamics . For example , Takeishi et al . ( 2017 ) ; Morton et al . ( 2018 ) optimize the residual sum of squares of the predictions the operator makes , Lusch et al . ( 2018 ) ; Erichson et al . ( 2019 ) ; Azencot et al . ( 2020 ) design dynamic autoencoders whose central component is linear and may be structured , Li et al . ( 2020 ) employ graph networks , and Mardt et al . ( 2018 ) use a variational approach on Markov processes . A recent line of work aims at exploiting tools from Koopman theory to analyze and improve the training process of neural networks ( Dietrich et al. , 2020 ; Dogra & Redman , 2020 ; Manojlović et al. , 2020 ) . To the best of our knowledge , our work is first to employ a Koopman-based method towards the analysis and understanding of trained neural networks . 3 METHOD . In what follows , we present our method for analyzing and understanding sequence neural models . Importantly , while we mostly discuss and experiment with recurrent neural networks , our approach is quite general and applicable to any model whose inner representation is a time series . We consider neural models that take input instances xt ∈ Rm at time t and compute ht = F ( ht−1 , xt ) , t = 1 , 2 , ... , ( 1 ) where ht ∈ Rk is a ( hidden ) state that represents the latent dynamics , and F is some nonlinear function that pushes states through time . In our analysis , we use only the hidden states set and discard the time series input . Thus , our method is a “ white-box ” approach as we assume access to { ht } , which is typically possible in most day-to-day scenarios . Importantly , all recurrent models including vanilla RNN ( Elman , 1990 ) , LSTM ( Hochreiter & Schmidhuber , 1997 ) , and GRU ( Cho et al. , 2014 ) , as well as Attention Models ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , and Residual neural networks ( He et al. , 2016 ) exhibit the structure of Eq . ( 1 ) . 3.1 ESSENTIALS OF KOOPMAN THEORY . Our approach is based on Koopman ( 1931 ) theory which was developed for dynamical systems . The key observation of Koopman was that a finite-dimensional nonlinear dynamics can be fully represented using an infinite-dimensional but linear operator . While the theoretical background is essential for developing a deep understanding of Koopman-based approaches , the practical aspects are more important to this work . Thus , we briefly recall the definition of the Koopman operator , and we refer the reader to other , comprehensive works on the subject ( Singh & Manhas , 1993 ; Eisner et al. , 2015 ) . Formally , we assume a discrete-time dynamical system ϕ acting on a compact , inner-product spaceM⊂ Rm , zt+1 = ϕ ( zt ) , zt ∈M , t = 1 , 2 , ... , ( 2 ) where t is an integer index representing discrete time . The dynamics ϕ induces a linear operator Kϕ which we call the Koopman operator , and it is given by Kϕf ( zt ) : = f ( zt+1 ) = f ◦ ϕ ( zt ) , ( 3 ) where f : M→ R is a scalar function in a bounded inner product space F . It is easy to show that Kϕ is linear due to the linearity of composition , i.e. , given α , β ∈ R and f , g ∈ F , we obtain that Kϕ ( αf +βg ) = ( αf +βg ) ◦ϕ = αf ◦ϕ+βg ◦ϕ = αKϕ ( f ) +βKϕ ( g ) . We emphasize that while ϕ describes the system evolution , Kϕ is a transformation on the space of functions . From a practical viewpoint , these functions may be interpreted as observations of the system , such as velocity , sea level , temperature , or hidden states in our setup . To justify our use of Koopman theory and practice in the context of neural networks , we propose the following . We interpret the input sequence { xt } as governed by some complex and unknown dynamics ϕ , i.e. , xt+1 = ϕ ( xt ) for every t. We emphasize that ϕ is different from F in Eq . ( 1 ) by its definition of domain and range . Then , the hidden states ht are finite samplings of observations of the system , namely , ht ≈ ft where ft : M→ R is the true observation . For instance , ft may be the smooth function cos ( tz ) , whereas ht ∈ Rk is its sampling at a finite set of points { z1 , . . . , zk } . It follows that { ht } is subject to an approximate Koopman representation . However , a fundamental challenge in facilitating Koopman theory in practice is the infinite-dimensionality of Kϕ . Recently , several data-driven methods were developed to produce a better approximate Kϕ using a moderatesize matrix C ( Schmid , 2010 ; Ovsjanikov et al. , 2012 ) . In particular , Koopman-based approaches have been proven instrumental in the analysis of fluid dynamics data ( Brunton et al. , 2021 ) , and for computing complex nonrigid isometric maps between shapes ( Ovsjanikov et al. , 2016 ) . Motivated by these empirical examples and their success , we will compute in this work approximate Koopman operator matrices C such that they encode the evolution of latent states { ht } .

Summary: This paper proposes using Koopman analysis to study the dynamics of neural sequence models. The proposal involves fitting linear approximations to the hidden state dynamics of a sequence model.  The paper applies this idea to two applications: RNNs trained on IMDB reviews for sentiment classification, and RNN autoencoders trained on ECG traces used for arrhythmia classification.