A KOOPMAN APPROACH TO UNDERSTANDING SEQUENCE NEURAL MODELS
Year: 2021

Abstract:
Deep learning models are often treated as “black boxes”. Existing approaches for understanding the decision mechanisms of neural networks provide limited explanations or depend on local theories, such as fixed-point analysis. Recently, a data-driven framework based on Koopman theory was developed for the analysis of nonlinear dynamical systems. In this paper, we introduce a new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks (KANN) method. At the core of our method lies the Koopman operator, which is a linear operator that encodes the dominant features of the network latent dynamics. In practice, we compute this operator by representing the hidden states of the network in a basis, and the operator is defined to be the linear fit in that new space. Since it is a linear operator, we can study its eigenvectors and eigenvalues, and we observe they facilitate understanding: in the sentiment analysis problem, the eigenvectors highlight positive and negative n-grams; and, in the ECG classification challenge, the eigenvectors capture the salient features of normal beat signals.

heading: 1 INTRODUCTION
text: Understanding the inner workings of predictive models is an essential requirement in many fields across science and engineering. This need is even more important nowadays with the emergence of neural networks whose visualization and interpretation is inherently challenging. Indeed, modern computational neural models often lack a commonly accepted knowledge regarding their governing mathematical principles. Consequently, while deep neural networks may achieve remarkable results on various complex tasks, explaining their underlying decision mechanisms remains a challenge. The goal of this paper is to help bridge this gap by proposing a new framework for the approximation, reasoning, and understanding of sequence neural models.
Sequence models are designed to handle time series data originating from images, text, audio, and other sources of information. One approach to analyzing sequence neural networks is through the theory and practice of dynamical systems (Doya, 1993a; Pascanu et al., 2013). For instance, the temporal asymptotic behavior of a dynamical system can be described using the local analysis of its attractor states (Strogatz, 2018). Similarly, recurrent models have been investigated in the neighborhood of their fixed points (Sussillo & Barak, 2013), leading to work that interprets trained RNNs for tasks such as sentiment analysis (Maheswaranathan et al., 2019). However, the local nature of these methods is a limiting factor which may lead to inconsistent results. Specifically, their approach is based on fixed-point analysis which allows to study the dynamical system in the neighborhood of a fixed-point. In contrast, our approach is global—it does not depend on a set of fixed-points, and it facilitates the exploration of the dynamics near and further away from fixed points.
Over the past few years, a family of data-driven methods was developed, allowing to analyze complex dynamical systems based on Koopman (1931) theory. These methods exploit a novel observation by which nonlinear systems may be globally encoded using infinite-dimensional but linear Koopman operators. In practice, Koopman-based approaches are lossy as they compute a finite-dimensional approximation of the full operator. Nevertheless, it has been shown in the fluid dynamics (Azencot et al., 2020; Mezić, 2005) and geometry processing (Sharma & Ovsjanikov, 2020; Ovsjanikov et al., 2012) communities that the dominant features of general nonlinear dynamical systems can be captured via a single matrix per system, allowing e.g., to align time series data (Rahamim & Talmon, 2021). Thus, we pose the following research question: can we design and employ a Koopman-based approach to analyze and develop a fundamental understanding of deep neural models?
Given a trained sequence neural network and a procedure to extract its hidden states, our Koopmanbased method generates a moderate size matrix which faithfully describes the dynamics in the latent space. Unlike existing work, our approach is global and independent of a particular latent sample, and thus it can be virtually applied to any hidden state. A key advantage of our framework is that we can directly employ linear analysis tools on the approximate Koopman operator to reason about the associated neural network. In particular, we show that the eigenvectors and eigenvalues of the Koopman matrix are instrumental for understanding the decision mechanisms of the model. For instance, we show in our results that the dominant eigenvectors carry crucial semantic knowledge related to the problem at hand. Moreover, the eigenvalues represent the memory of the network as they provide a timestamp for the temporal span of the respective eigenvectors. Finally and most importantly, Koopman-based methods such as ours are backed by rich theory and practice, allowing us to exploit the recent advances in Koopman inspired techniques for the purpose of developing a comprehensive understanding of sequence neural networks. Thus, the key contribution in this work is the novel application of Koopman-based methods for understanding sequential models, and the extraction of high-level interpretable and insightful understandings on the trained networks.
We focus our investigation on two learning tasks: sentiment analysis and electrocardiogram (ECG) classification. We will identify four eigenvectors in the sentiment analysis model whose roles are to highlight: positive words, negative words, positive pairs (e.g., “not bad”), and negative pairs. In addition, we demonstrate that the eigenvectors in the ECG classification task naturally identify dominant features in normal beat signals and encode them. Specifically, we show that four Koopman eigenvectors accurately capture the local extrema points of normal beat signals. These extrema points are fundamental in deciding whether a signal is normal or anomalous. Our results reinforce that the network indeed learns a robust representation of normal beat signals. Then, we will verify that the main components of the nonlinear network dynamics can be described using our Koopman matrices by measuring the difference in accuracy results, and the relative error in predicted states. Further, we provide additional results and comparisons in the supplementary material. Given the versatility of our framework and its ease of use, we advocate its utility in the analysis and understanding of neural networks, and we believe it may also affect the design and training of deep models in the future.

heading: 2 RELATED WORK
text: Recurrent Neural Networks (RNN) and Dynamical Systems (DS). Fully connected recurrent neural networks are universal approximators of arbitrary dynamical systems (Doya, 1993b). Unfortunately, RNNs are well-known to be difficult to train (Bengio et al., 1993; Pascanu et al., 2013), and several methods adopt a DS perspective to improve training via gradient clipping (Pascanu et al., 2013), and constraining the weights (Erichson et al., 2021), among other approaches. Overall, it is clear that dynamical systems are fundamental in investigating and developing recurrent networks.
Understanding RNN. Establishing a deeper understanding of recurrent networks is a long standing challenge in machine learning. To this end, Karpathy et al. (2015) follow the outputs of the model to identify units which track brackets, line lengths, and quotes. Recently, Chefer et al. (2020) proposed an approach for computing relevance scores of transformer networks. Perhaps mostly related to our approach is the analysis of recurrent models around their fixed points (Sussillo & Barak, 2013). This approach revealed low-dimensional attractors in the sentiment analysis task (Maheswaranathan et al., 2019), which allowed to deduce simple explanations of the decision mechanisms of the associated models. Our work generalizes the approach of Sussillo & Barak (2013) in that it yields global results about the dynamics, and it introduces several novel features. We provide a more detailed comparison between our method and theirs in Sec. 4.
Koopman-based Neural Networks. Recently, several techniques that combine neural networks and Koopman theory were proposed, mostly in the context of predicting nonlinear dynamics. For example, Takeishi et al. (2017); Morton et al. (2018) optimize the residual sum of squares of the predictions the operator makes, Lusch et al. (2018); Erichson et al. (2019); Azencot et al. (2020) design dynamic autoencoders whose central component is linear and may be structured, Li et al. (2020) employ graph networks, and Mardt et al. (2018) use a variational approach on Markov processes. A recent line of work aims at exploiting tools from Koopman theory to analyze and improve the training process of neural networks (Dietrich et al., 2020; Dogra & Redman, 2020;
Manojlović et al., 2020). To the best of our knowledge, our work is first to employ a Koopman-based method towards the analysis and understanding of trained neural networks.

heading: 3 METHOD
text: In what follows, we present our method for analyzing and understanding sequence neural models. Importantly, while we mostly discuss and experiment with recurrent neural networks, our approach is quite general and applicable to any model whose inner representation is a time series. We consider neural models that take input instances xt ∈ Rm at time t and compute
ht = F (ht−1, xt) , t = 1, 2, ... , (1)
where ht ∈ Rk is a (hidden) state that represents the latent dynamics, and F is some nonlinear function that pushes states through time. In our analysis, we use only the hidden states set and discard the time series input. Thus, our method is a “white-box” approach as we assume access to {ht}, which is typically possible in most day-to-day scenarios. Importantly, all recurrent models including vanilla RNN (Elman, 1990), LSTM (Hochreiter & Schmidhuber, 1997), and GRU (Cho et al., 2014), as well as Attention Models (Bahdanau et al., 2015; Vaswani et al., 2017), and Residual neural networks (He et al., 2016) exhibit the structure of Eq. (1).

heading: 3.1 ESSENTIALS OF KOOPMAN THEORY
text: Our approach is based on Koopman (1931) theory which was developed for dynamical systems. The key observation of Koopman was that a finite-dimensional nonlinear dynamics can be fully represented using an infinite-dimensional but linear operator. While the theoretical background is essential for developing a deep understanding of Koopman-based approaches, the practical aspects are more important to this work. Thus, we briefly recall the definition of the Koopman operator, and we refer the reader to other, comprehensive works on the subject (Singh & Manhas, 1993; Eisner et al., 2015). Formally, we assume a discrete-time dynamical system ϕ acting on a compact, inner-product spaceM⊂ Rm,
zt+1 = ϕ(zt) , zt ∈M , t = 1, 2, ... , (2)
where t is an integer index representing discrete time. The dynamics ϕ induces a linear operator Kϕ which we call the Koopman operator, and it is given by
Kϕf(zt) := f(zt+1) = f ◦ ϕ(zt) , (3)
where f :M→ R is a scalar function in a bounded inner product space F . It is easy to show that Kϕ is linear due to the linearity of composition, i.e., given α, β ∈ R and f, g ∈ F , we obtain that Kϕ(αf +βg) = (αf +βg) ◦ϕ = αf ◦ϕ+βg ◦ϕ = αKϕ(f) +βKϕ(g). We emphasize that while ϕ describes the system evolution, Kϕ is a transformation on the space of functions. From a practical viewpoint, these functions may be interpreted as observations of the system, such as velocity, sea level, temperature, or hidden states in our setup.
To justify our use of Koopman theory and practice in the context of neural networks, we propose the following. We interpret the input sequence {xt} as governed by some complex and unknown dynamics ϕ, i.e., xt+1 = ϕ(xt) for every t. We emphasize that ϕ is different from F in Eq. (1) by its definition of domain and range. Then, the hidden states ht are finite samplings of observations of the system, namely, ht ≈ ft where ft :M→ R is the true observation. For instance, ft may be the smooth function cos(tz), whereas ht ∈ Rk is its sampling at a finite set of points {z1, . . . , zk}. It follows that {ht} is subject to an approximate Koopman representation. However, a fundamental challenge in facilitating Koopman theory in practice is the infinite-dimensionality of Kϕ. Recently, several data-driven methods were developed to produce a better approximate Kϕ using a moderatesize matrix C (Schmid, 2010; Ovsjanikov et al., 2012). In particular, Koopman-based approaches have been proven instrumental in the analysis of fluid dynamics data (Brunton et al., 2021), and for computing complex nonrigid isometric maps between shapes (Ovsjanikov et al., 2016). Motivated by these empirical examples and their success, we will compute in this work approximate Koopman operator matrices C such that they encode the evolution of latent states {ht}.

heading: 3.2 A KOOPMAN-BASED METHOD
text: We denote by H ∈ Rs×n×k a tensor of hidden state sequences, where s is the batch size, n is the sequence length and k is the hidden dimension. The method we employ for computing the matrix C follows two simple steps: 1. Represent the states using a basis B, and denote the resulting collection of spectral coefficients by H̃ . 2. Find the best linear transformation C which maps H̃t to H̃t+1 in the spectral domain, where H̃τ ∈ Rs×k denotes the tensor of coefficients from H̃ at time τ . To give a specific example of the general procedure we just described, we can choose the principal components bj , j = 1, 2, ... of the truncated SVD of the states H to be the basis in the first step. Then, the resulting basis elements are orthonormal, i.e., BTB = Id, where B = (bj) is the matrix of basis elements organized in its columns, and Id is the identity matrix. The matrix C is obtained by solving the following least squares minimization
C := arg min C̃ n−1∑ t=1 ∣∣∣H̃t · C̃ − H̃t+1∣∣∣2 F , (4)
H̃τ = Hτ ·B , ∀τ , (5)
where · is matrix multiplication. We note that the above scheme is a variant of the dynamic mode decomposition (Schmid, 2010), and the functional maps (Ovsjanikov et al., 2012) algorithms.

heading: 3.3 KOOPMAN-BASED PREDICTION
text: The infinite-dimensional Koopman operator in Eq. (3) describes the evolution of observable functions subject to the dynamics ϕ. Similarly, our C matrices allow us to predict a future hidden state ht+1 from a given current state ht simply by multiplying C with the spectral coefficients h̃t. Namely,
HKANNt+1 := Ht ·B · C ·BT . (6)
We will mostly use Eq. (6) to evaluate the validity of C in encoding the underlying dynamics based on the differences |HKANNt −Ht|2F /|Ht|2F for every admissible t, see Sec. 4.

heading: 3.4 KOOPMAN-BASED ANALYSIS
text: The key advantage of Koopman theory and practice is that linear analysis tools can be directly applied to study the behavior of the underlying dynamical system. The tools we describe next form the backbone of our analysis framework, and our results are heavily based on these tools.
Separable dynamics. If C ∈ Rk×k admits an eigendecomposition, then the dynamics can be represented in a fully separable manner, where the eigenvectors of C propagate along the dynamics independently of the other eigenvectors, scaled by their respective eigenvalues. Formally, we consider the eigenvalues λj ∈ C and eigenvectors vj ∈ Ck of C, i.e., it holds that C vj = λjvj . We assume that C is full-rank and thus V = (vj) forms a basis of Rk, and similarly, U = V −1 is also a spanning basis. In our setting, we call the rows of U the Koopman eigenvectors, and we represent any hidden state ht in this basis, similarly to Eq. (5). The projection of H onto U reads
Ĥτ := Hτ ·B · V = H̃τ · V . (7)
Then, re-writing Eq. (6) using the eigendecomposition of C = V ·Λ ·U yields the temporal trajectory of Ht via Ĥt+1 = Ht+1 ·B · V ≈ Ht ·B ·C · V = Ht ·B · V ·Λ = Ĥt ·Λ, where Λ is the diagonal matrix of eigenvalues, and the approximation is due to Eq. (6). The latter derivation yields
Ĥt+1 ≈ Ĥt · Λ , (8)
i.e., the linear dynamics matrix represented in the basis U is simply the diagonal matrix Λ, and thus U may be viewed as a “natural” basis for encoding the dynamics. Further, it directly follows that Ĥt+l ≈ Ĥt · Λl, that is, the number of steps forward is determined by the eigenvalues power.
Memory horizon. In addition to their role in describing the evolution of the hidden states, the eigenvalues naturally encode a memory horizon “timestamp” associated with each of the eigenvectors. Specifically, if |λj | < 1, then its powers decay exponentially to zero. We define the memory horizon of uj to be the decay time τj( ) of its associated eigenvalue, i.e.,
τj( ) = log( )
log(|λj |) , (9)
where 0 < 1 is a small threshold parameter. We note that both log( ) and log(|λj |) are negative, and thus τj( ) > 0. Given a choice of , the eigenvector uj becomes almost insignificant to the dynamics in Eq. (9) after τj( ) steps as it is scaled by = |λj |τj( ). In this context, eigenvectors whose eigenvalues satisfy |λj | = 1 have infinite memory, as powers of their eigenvalues simply rotate over the unit circle. Finally, if |λj | > 1, its respective eigenvector leads to an unstable behavior since liml→∞ |λlj | =∞.

heading: 4 RESULTS
text: In this study, we focus our exploration on the sentiment analysis and the ECG classification problems. Unless noted otherwise, we always compute C using the method in Sec 3.2, where the basis is given by the truncated SVD modes of the input hidden states, and C is the least squares estimation obtained from solving (4). We first provide our qualitative analysis in 4.1, and 4.2. Then, we include in 4.3 a quantitative evaluation of KANN and its ability to encode the dynamics. In Apps. A, D, and E, we provide additional results, and we show that our method is robust to the choice of basis and network architecture. Finally, we further use KANN to analyze the copy problem in App. F, where our results outperform the baseline approach (Maheswaranathan et al., 2019).

heading: 4.1 SENTIMENT ANALYSIS
text: We begin our qualitative study by considering the sentiment analysis task which was extensively explored in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020). Determining the sentiment of a document is an important problem which may be viewed as a binary classification task. We will use the IMDB reviews dataset, and we will embed the corpus of words to obtain a vector representation of text. Given a review, the role of the network is to output whether it reflects a positive or negative opinion. Adopting the setup of Maheswaranathan et al. (2019), we use a word embedding of size 128, and a GRU recurrent layer with a hidden size of 256. We train the model for 5 epochs during which it reaches an accuracy of ≈ 92%, 87%, 87% on the train, validation and test sets, respectively. For analysis, we extract a random test batch of 64 reviews and its states H ∈ R64×1000×256, where 1000 is the review length when padded with zeros. One of the main results in (Maheswaranathan et al., 2019) was the observation that the dynamics of the network span a line attractor. That is, the hidden states of the network are dominantly attracted to a one dimensional manifold, splitting the domain into positive and negative sentiments. Additionally, Maheswaranathan & Sussillo (2020) study inputs with contextual relations (e.g., the phrase “not bad”), and their effect on the network dynamics. Our results align with the observations in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020). Moreover, we generalize their results by showing that the attracting manifold is in fact of a higher dimension, and that the manifold can be decomposed to semantically understandable components using KANN. Specifically, we demonstrate that several Koopman eigenvectors are important in the dynamics, and we can link each of these eigenvectors to a semantically meaningful action. Thus, in comparison to the framework proposed in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020), our method naturally decomposes the latent manifold into interpretable attracting components. In addition, we provide a unified framework for reasoning and understanding by drawing conclusions directly from the separable building blocks of the latent dynamics.
Most of our results for the sentiment analysis problem are based on the eigendecomposition of C, its resulting eigenvalues {λj ∈ C} and corresponding eigenvectors {uj ∈ Ck}. For the random states batch H specified above, we obtain an operator C whose spectrum consists of four eigenvalues with modulus greater than 0.99, i.e., |λj | > 0.99. In comparison, Maheswaranathan et al. (2019) identify only a single dominant component. The values of our λj read λ1 = 0.9999. λ2 = 0.9965, and
A positive review projected onto {u1, u2}:
A negative review projected onto {u3, u4}:
Two reviews with contextual information projected onto {u8}:
Figure 1: We display reviews where each word is shaded based on
∑ j s(j, ht). Projecting onto U12
shows an increase in magnitude for several positive words (blue), whereas projecting onto U34 shows jumps in magnitude around negative words (red). See e.g., amazing, special (blue), mess, waste, muddled (red). We also show two reviews with contextual information which is naturally highlighted due to u8.
λ3,4 = 0.9942± i0.0035. Consequently, their respective eigenvectors have long memory horizons. For instance, if we set = 1e−2 in Eq. (9), then we get a time stamp τ > 800 for all four eigenvectors. Namely, these eigenvectors carry information across word sequences of length up to 800, and only < 2% of the reviews in the IMDB dataset are that long.
In their analysis, the authors of (Maheswaranathan et al., 2019) observe that the network mainly counts positive vs. negative words in a review along a line attractor. We hypothesize that in our setting, the dominant eigenvectors {u1, ..., u4} are responsible for this action. We expect that complex conjugates such as u3 and u4 share the same role, e.g., counting the negative words, and we expect that U12 = {u1, u2} take the role of counting the positive words. To verify our hypothesis, we use the readout layer of the model to generate the logits of the state when projected to U12 and U34. We denote by ỹ12 and ỹ34 the logits for H̃ · V12 and H̃ · V34, respectively, where Vij denote the i and j columns of V . For the above test batch, we get perfect correspondence, i.e., ỹ12 < .5 and ỹ34 > .5 on all samples. In addition to encoding a certain sentiment, the Koopman eigenvectors are advantageous in comparison to a single line attractor as they allow for a direct visualization of the importance of words in a review. Specifically, we define the projection magnitude of a hidden state as follows
s(j, ht) := ∣∣∣ĥt(j)∣∣∣ = ∣∣∣h̃Tt Vj∣∣∣ . (10)
We show in Fig. 1 two examples where the magnitude of projection onto U12 and U34 clearly highlights positive and negative words, respectively. In particular, as the network “reads” the review and identifies e.g., a negative word, it increases s(·). For instance, see mess and muddled in Fig. 1. Importantly, there may be occurrences of positive/negative n-grams which are not highlighted, such as the word good in the positive example. We show in App. A that the above results extend to the entire test set. Thus, we conclude that {u1, ..., u4} track positive and negative words. In addition to {u1, ..., u4}, we also want to understand how other eigenvectors affect the latent dynamics. We hypothesize that other vectors are responsible to track contextual information such as amplifiers (“extremely good”) and negations (“not bad”). We collected all reviews that include the phrases “not bad” and “not good” into a single batch, yielding a states tensor with 256 samples. One way to check our hypothesis is to employ the former visualization using other eigenvectors. We show two such examples in Fig. 1 in green, where phrases such as “not bad”, “terribly wrong”, and
“quite ok” are highlighted when projected onto u8. We provide additional results and analysis on the identification of amplifier words and negations using KANN in App. A. In addition to unigram and bigram highlighting, in App. B we also consider KANN in the general case of n-grams where n > 2.

heading: 4.2 ECG CLASSIFICATION
text: Electrocardiogram (ECG) tests track the electrical activity in the heart, and they help detect various abnormalities in a non-invasive way. Classifying whether a beat is normal or not is a challenging task which lacks descriptive neural models. A common approach for solving the classification problem using neural networks trains an autoencoder model with an L1 loss over the normal beats. Classification is performed by measuring the loss between the original and reconstructed signals; thus, while it is a classification task, ECG classification is solved via a regression model. In particular, high loss values indicate anomalous beats, whereas low values are attributed to normal signals. Typically, a threshold is set during the training phase, allowing automatic classification on the test set. We fix the threshold to be 26. Our network is composed of a single layer LSTM encoder Fenc with a hidden size of 64, and an LSTM decoder Fdec with one layer as well. We use a publicly available subset of the MIT-BIH arrhythmia database (Goldberger et al., 2000) for our data, named ECG50001. This dataset includes 5000 sample heartbeats with a sequence length of 140. Around 60% of the sequences are classified as normal and the rest are various anomalous signals. The model is trained for 150 epochs, yielding an accuracy of 97.1%, 97.6%, 98.6% on the train, validation and test sets, respectively.
Similarly to the sentiment analysis problem, we expect the Koopman eigenvectors to take a significant role in encoding the latent dynamics. Given that the network is generative as it is an autoencoder, we hypothesize that the eigenvectors {uj} capture dominant features of normal beats. Thus, we project normal beats onto pairs of dominant eigenvectors, and decode the resulting hidden states using the decoder to study the obtained signals. For example, say U58,59 are dominant, then we project onto the space spanned by this pair, then project back and decode. Using a test batch of 64 normal beats we collect the last hidden state of every sample in a matrix Hno ∈ R64×64 and we compute the following.
R64×140 3 X̄ij = Fdec(Hno ·B · |Vij · Uij | ·BT ) , (11) where | · | is the element-wise modulus of complex numbers. To determine which eigenvectors are dominant, we employ Eq. (10).
To visualize the results, we take the set of reconstructed signals X̄ij of a particular pair ij, and we compute its median µij(t) for t ∈ [1, . . . , n]. We plot these graphs in Fig. 2 using colors orange (pair 58–59), green (pair 25–26), red (pair 9–10), and brown (pair 50–51). In addition, the original signals’
1http://timeseriesclassification.com/description.php?Dataset=ECG5000
median µ (purple) and the median of the signals reconstruction µr (blue) are shown in each of the subplots for comparison. Indeed, µ and µr are almost indistinguishable, and can be differentiated only when zooming in. Each median graph is wrapped in its median absolute deviation envelope. We preferred median-based quantities over the common mean and standard deviation since the latter produce cluttered plots in our setting due to outliers. The plots in Fig. 2 clearly show that each conjugate pair captures a different feature of the time series as marked by the vertical dashed lines. Specifically, µ58,59 captures the minimum around t = 3, and µ25,26 encodes the part of the signal in t ∈ [35, 75]. Moreover, µ9,10 attains the maximum at t = 103, and µ50,51 is approximating the lower peak at t = 133 and we consider these t values to be the salient features. Importantly, the other Koopman eigenvectors beyond the ones we consider above are less important in the reconstruction, and are mostly helpful in fixing minor variations. Finally, we provide a similar computation in App. D using the dominant PCA modes and KernelPCA eigenvectors, where we show that PCA components and KerenelPCA eigenvectors are not useful in identifying the dominant features of beat signals. Also, we provide a quantitative comparison between the methods.
In addition to identifying principal features of beat signals, we show in what follows that the Koopman eigenvectors are also instrumental in analyzing the latent structure of the LSTM autoencoder. We begin by showing in Fig. 3 (left) the median values over time of a normal batch and its reconstruction (as in Fig. 2), and similarly for a batch of anomalous signals (orange and brown). From this data, the task of ECG classification may be viewed as a binary classification problem, separating normal from anomalous signals via reconstruction. However, we will now show that this is actually not the case. Instead, the network essentially encodes inputs, whether normal or anomalous, in a representation that is closer to the manifold of normal signals. To demonstrate and analyze this phenomenon, we consider hidden state tensors Hno and Han of normal and anomalous beats, respectively, and we concatenate these tensors over the samples yielding H = (Hno, Han) ∈ R128×140×64. We would like to study the decision boundary separating between different signals in the latent space.
To this end, we employ a standard measure known as the silhouette score (Rousseeuw, 1987) to quantify the class separation quality. The silhouette score σ is a real value in [−1, 1], where scores close to 1 mean the latent states are well separated. In contrast, values closer to zero indicate that samples are located on or close to the decision boundary. We compute the silhouette score estimates on H̃ = H · B averaged over samples, and cumulatively averaged over time. Namely, σ(t) = 1128·tΣs,tσ(h̃s,t), where σ(h̃s,t) is the silhouette score of the vector h̃s,t with s ∈ [1, . . . , 128] and t ∈ [1, . . . , 140]. We compare three silhouette score estimates denoted by: σ for the original H̃ , σPCA for the projection of H̃ onto its first five principal components, and σKoopman for the projection of H̃ onto the first five dominant Koopman eigenvectors. The results are shown in Fig. 3 (right), where Koopman’s embedding attains low scores compared to PCA and the original states. Namely, embedding the hidden states using Koopman eigenvectors reveals that the decision boundary between normal and anomalous signals is somewhat blurred, in contrast to the numerical results provided by the reconstructed signals and other embeddings. This understanding provides a rather straightforward interpretation of the model: it simply encodes the dominant components of all signals as being normal, allowing to easily identify anomalous signals later by measuring their reconstruction error. Finally, our analysis shows that Koopman eigenvectors successfully identify the salient features of normal beat signals. We conclude from this observation that the network focuses on identifying these features and reconstructing them accurately. A correct reconstruction of these salient features allows to subsequently identify using a simple loss check whether a signal is normal or anomalous. Importantly,
we show in Fig. 3 that the network indeed successfully reconstructs normal and anomalous signals. The understanding that we obtain using KANN is that the network mainly focuses on these salient features during the reconstruction.
4.3 KANN REPRODUCES THE LATENT DYNAMICS
We now perform a quantitative study of the ability of C to truly capture the latent dynamics. We will show that indeed, KANN is able to reproduce the nonlinear dynamics of the network in Eq. (1) to a high degree of precision, and thus we achieve the empirical justification to replace F with C. To this end, we consider the following two metrics: 1. Relative error of hidden states: let {hs,t} be a collection of states over samples s = 1, ..., S and across time t = 1, ..., T . We generate the predicted collection {hKANNs,t } using Eq. (6), and we compute
erel({hKANNs,t }, {hs,t}) = 1 T · S ∑ s,t |hKANNs,t − hs,t|22 / |hs,t|22 . (12)
2. Accuracy error: let G be the neural network component that takes a state and produces the output of the model, i.e., G(ht) = ỹt. We denote by c̃t the category predicted by ỹt, for instance c̃t = arg max(ỹt). We compare the difference between c̃t and c̃KANNt , obtained from G(h KANN t ) = ỹ KANN t .
We show in Fig. 4 the results of our quantitative study. For the sentiment analysis problem (Fig. 4, left), we obtain > 99% correspondence with the classification of the network over all the test set, as is shown for the True Positive (TP) and True Negative (TN) columns vs. the False Positive (FP) and False Negative (FN) columns. In the ECG classification task (Fig. 4, right), we reconstruct 145 signals of the normal test set and compute their loss. There is a noticeable yet small shift in the loss histogram between the network reconstruction (blue) in comparison to our reconstruction (orange). However, the threshold for this problem set at 26 during training (black dashed line) yields > 97% agreement in classification. In particular, the false classification of normal signals (around loss 90) appear both in the network output and in ours. Finally, we also computed the relative error of the hidden states, obtaining erel = 0.095 on a batch of size 64 for the sentiment analysis task, and erel = 0.0056 on a batch of size 145 for the ECG classification problem. Overall, the above results demonstrate that KANN faithfully represents the latent dynamics.

heading: 5 DISCUSSION
text: In this work we presented a novel framework for studying sequence neural models based on Koopman theory and practice. Our method involves a dimensionality reduction representation of the states, and the computation of a linear map between the current state and the next state. Key to our approach is the wealth of tools we can exploit from linear analysis and Koopman-related work. In particular, we compute linear approximations of the state paths via simple matrix-vector multiplications. Moreover, we identify dominant features of the dynamical system and their effect on inference and prediction. Our results on the sentiment analysis problem, and the ECG classification challenge provide simple yet accurate descriptions of the underlying dynamics and behavior of the recurrent models. Our work lays the foundations to further develop application-based descriptive frameworks, towards an improved understanding of neural networks. In the future, we plan to explore our framework during the training of the model, where in this work we focused only on trained models.

Review summaries:
The paper proposes a framework for studying sequence neural models based on Koopman theory. Particularly, the authors compute linear approximations of the state paths via simple matrix-vector multiplications. Also, the dominant features of the dynamical system and their effect on inference and prediction is determined. Their results on the sentiment analysis problem, and the ECG classification challenge provide simple yet precise descriptions of the underlying dynamics and behavior of the recurrent models. 

This paper proposes using Koopman analysis to study the dynamics of neural sequence models. The proposal involves fitting linear approximations to the hidden state dynamics of a sequence model.  The paper applies this idea to two applications: RNNs trained on IMDB reviews for sentiment classification, and RNN autoencoders trained on ECG traces used for arrhythmia classification.

The authors propose a method for analyzing NN applied to temporal signals (e.g., dynamical systems). The proposed method relies on the Koopman operator, which is widely used for studying dynamical systems. The authors use the eigenvectors of the Koopman operator to interpret predictions made by NN. The proposed method is evaluated on two tasks, namely: sentiment analysis and electrocardiogram (ECG) classification. The authors demonstrate that the eigenvectors of the Koopman operator do encode the latent information in these examples.