A Koopman Approach to Understanding Sequence Neural Models
Authors: Ilan Naiman
Omri Azencot
Venue: ICLR 2022 Submitted (2022)
Keywords: Koopman methods
sequence neural models
understanding deep learning

Abstract: Deep learning models are often treated as "black boxes". Existing approaches for understanding the decision mechanisms of neural networks provide limited explanations or depend on local theories. Recently, a data-driven framework based on Koopman theory was developed for the analysis of nonlinear dynamical systems. In this paper, we introduce a new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks (KANN) method. At the core of our method lies the Koopman operator, which is linear, yet it encodes the dominant features of the network latent dynamics. Moreover, its eigenvectors and eigenvalues facilitate understanding: in the sentiment analysis problem, the eigenvectors highlight positive and negative n-grams; and, in the ECG classification challenge, the eigenvectors capture the dominant features of the normal beat signal.

Under review as a conference paper at ICLR 2022
A KOOPMAN APPROACH TO UNDERSTANDING SEQUENCE NEURAL MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning models are often treated as "black boxes". Existing approaches for understanding the decision mechanisms of neural networks provide limited explanations or depend on local theories, such as fixed-poin analysis. Recently, a data-driven framework based on Koopman theory was developed for the analysis of nonlinear dynamical systems. In this paper, we introduce new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks (KANN) method. At the core of our method lies the Koopman operator, which is a linear operator that encodes the dominant features of the network latent dynamics. In practice, we compute this operator by representing the hidden states of the network in a basis, and the operator defined to be the linear fit in that new space. Since it is linear operator, we can study its eigenvectors and eigenvalues, and we observe they facilitate understanding: in the sentiment analysis problem, the eigenvectors highlight positive and negative n-grams; and, in the ECG elassification challenge, the cigenvectors capture the salient features of normal beat signals.
INTRODUCTION
Understanding the inner workings of predictive models is essential requirement in many fields across science and engineering. This need is even more important nowadays with the emergence of neural networks whose visualization and interpretation is inherently challenging. Indeed, modern computational neural models often lack a commonly accepted knowledge regarding their governing mathematical principles. Consequently, while deep neural networks may achieve remarkable results on various complex tasks, explaining their underlying decision mechanisms remains a challenge. The goal of this paper is help bridge this gap by proposing new framework for the approximation, reasoning, and understanding of sequence neural models.
Sequence models are designed handle time series data originating from images, text, audio, and other sources of information. One approach to analyzing sequence neural networks is through the theory and practice of dynamical systems (Doya) 1993a, Pascanu et al., 2013). For instance, the temporal asymptotic behavior of dynamical system be described using the local analysis of its attractor states (Strogatz, 2018). Similarly, recurrent models have been investigated in the neighborhood of their fixed points (Sussillo & Barak (013)), leading to work that interprets trained RNNs for tasks such as sentiment analysis (Maheswaranathan et al.) 2019). However, the local nature of these methods is limiting factor which may lead inconsistent results. Specifically, their approach is based on fixed-point analysis which allows to study the dynamical system in the neighborhood of a fixed-poiis point. In contrast, our approach global-it does not depend on set of fixed-points and it facilitates the exploration of the dynamics near and further away from fixed points.
Over the past few years, a family of data-driver methods was developed, allowing to analyze complex dynamical systems based on Koopman (I93T) theory. These methods exploit a novel observation by which nonlinear systems may be globally encoded using dimensioalll but linear Koopman operators. In practice, Koopman-basee
approaches are lossy as they compute a finite-dimensional approximation of the full operator. Nevertheless, it has been shown in the fluid dynamics (Azencot et al. 2020; Mezić, 2005) and geometry processing (Sharma & Ovsjanikov 2020; Ovsjanikov et al. 2012) communities that the dominant features of general nonlinear dynamical systems can be captured via single matrix per system, allowing e.g., to align time series data (Rahamim & Talmon 202II. Thus, we pose the following research question: can we design and employ a Koopman- basedd approach to analyze and develop a fundamental understanding of deep neural models?

---

Under review as a conference paper at ICLR 2022
Given trained sequence neural network and a procedure to extract its hidden states, our Koopman- based method generates a moderate size matrix which faithfully describes the dynamics in the latent space. Unlike existing work, our approach is global and independent of particular latent sample, and thus it can be virtually applied to any hidden state. A key advantage of our framework that we can directly employ linear analysis tools the approximate Koopman operator to reason about the associated neural network. In particular, we show that the eigenvectors and eigenvalues of the Koopman matrix are instrumental for understanding the decision mechanisms of the model. For instance, we show in our results that the dominant eigenvectors carry crucial semantic knowledge related the problem at hand. Moreover, the eigenvalues represent the memory of the network as they provide a timestamp for the temporal span of the respective eigenvectors. Finally and most importantly, Koopman-bane -based methods such as ours are backed by rich theory and practice, allowing us to exploit the recent advances in Koopman inspired techniques for the purpose of developing comprehensive understanding of sequence neural networks. Thus, the key contribution in this work is the novel application of Koopman-bir based methods for understanding sequential models, and the extraction of high-level interpretable and insightful understandings on the trained networks.
We focus investigation two learning tasks: sentiment analysis and electrocardiogram (ECG) classification. We will identify four eigenvectors in the sentiment analysis model whose roles are to highlight: positive words, negative words, positive pairs (e.g.. "not bad"), and negative pairs. In addition, we demonstrate that the eigenvectors in the ECG classification task naturally identify dominant features in normal beat signals and encode them. Specifically, we show that four Koopman cigenvectors accurately capture the local extrema points of normal beat signals. These extrema points are fundamental in deciding whether signal is normal anomalous. Our results reinforce that the network indeed learns a robust representation of normal beat signals. Then, we will verify that the main components of the nonlinear network dynamics can be described using our Koopman matrices by measuring the difference in accuracy results, and the relative error in predicted states. Further, we provide additional results and comparisons in the supplementary material. Given the versatility of our framework and its ease of use, we advocate its utility in the analysis and understanding of neural networks, and we believe it may also affect the design and training of deep models in the future.
2 RELATED WORK
Recurrent Neural Networks (RNN) and Dynamical Systems (DS). Fully connected recurrent
neural networks are universal approximators of arbitrary dynamical systems (Doya] 1993b). Unfor- tunately, RNNs are well-known to be difficult to train (Bengio et al. 1993: Pascanu et al. 2013). and several methods adopt DS perspective to improve training via gradient clipping (Pascanu et al.) 2013), and constraining the weights (Erichson et al. 202I), among other approaches. Overall, it is clear that dynamical systems fundamental in investigating and developing recurrent networks.
Understanding RNN. Establishing a deeper understanding of recurrent networks a long standing challenge in machine learning. To this end, Karpathy et al. (2015) follow the outputs of the model to identify units which track brackets, line lengths, and quotes. Recently, [Chffer al. (2020) proposed an approach for computing relevance scores of transformer networks. Perhaps mostly related to our approach is the analysis of recurrent models around their fixed points (Sussillo & Barak] 2013). This approach revealed fow-dimensional attractors in the sentiment analysis task (Maheswaranathar et al., 2019), which allowed to deduce simple explanations of the decision mechanisms of the associated models. Our work generalizes the approach of Sussillo & Barak (2013) in that it yields global results about the dynamics, and it introduces several novel features. We provide a more detailed comparison between our method and theirs in Sec. 4
Keeeeery,,Chefer
Koopman-based Neural Networks. Recently, several techniques that combine neural networks and Koopman theory were proposed, mostly in the context of predicting nonlinear dynamics. For example, Takeishi et al. (2017); Morton et al. (2018) optimize the residual sum of squares of the predictions the operator makes, Lusch et al. (2018); Erichson et al. (2019): Azencot et al. (2020) design dynamic autoencoders whose central component is linear and may be structured, Li et al. (2020) employ graph networks, and |ardt et al. (018) use variational approach on Markoy processes. A recent line of work aims at exploiting tools from Koopman theory to analyze and improve the training process of neural networks (Dietrich et al. 2020) Dogra & Redman 2020
2

---

Under review as a conference paper at ICLR 2022
Manojlović et al. 2020). To the best of our knowledge, work is first employ Koopman- based method towards the analysis and understanding of trained neural networks.
3 METHOD
In what follows, we present our method for analyzing and understanding sequence neural models. Importantly, while we mostly discuss and experiment with recurrent neural networks, our approach is quite general and applicable to any model whose inner representation is a time series. We consider neural models that take input instances E R at time t and compute
he = F(h-1,,
t 1,,,...
(I)
where h4 E IR is a (hidden) state that represents the latent dynamics, and F is some nonlinear function that pushes states through time. In our analysis, we use only the hidden states set and discard the time series input. Thus, our method is a "white-box" approach as we assume access to {he), which typically possible in most day-to-day scenarios. Importantly, including vanilla RNN (Elman) 1990), LSTM (Hochreiter & Schmidhuber,
recurrent models and GRU (Cho ctt al., 2014), as well as Attention Models (Bahdanau et al. [ Vaswani et al. 2017). and Residual neural networks (He et al. 2016) exhibit the structure of Eq. (
3.1 ESSENTIALS OF KOOPMAN THEORY
Our approach is based on Koopman [931] theory which was developed for dynamical systems. The key observation of Koopman was that a finite-dimensional nonlinear dynamics can be fully represented using an infinite-dimensional but linear operator. While the theoretical background is essential for developing a deep understanding of Koopman-basee based approaches, the practical aspects are more important to this work. Thus, we briefly recall the definition of the Koopman operator, and we refer the reader to other, comprehensive works on the subject (Singh & Manhas 1993, Eisner et al.) 2015). Formally, we assume a discrete-time dynamical system &p acting on a compact, inner-product space M C R
- = p(()),
21 E M, =
(2)
where is integer index representing discrete time. The dynamics &p induces a linear operator Kup which we call the Koopman operator, and it is given by
(xx) = f(++1) = p(z),
(3)
where f M + R is scalar function in bounded inner product space F. It is easy to show that K.,, is linear due to the linearity of composition, i.e., given a, 3 E R and f,g E F, we obtain that (.p 33) (a + (9) o af o + (f) (g). We emphasize that while x describes the system evolution, K.p is a transformation on the space of functions. From a practical viewpoint, these functions may be interpreted as observations of the system, such as velocity, sea level, temperature, or hidden states in our setup.
To justify our use of Koopman theory and practice in the context of neural networks, we propose the following. We interpret the input sequence {æ+} } as governed by some complex and unknown dynamics yp, i.e., Iz+11 1 = (xxx)) for every We emphasize that şP is different from F in Eq. ① by its definition of domain and range. Then, the hidden states h2 are finite samplings of observations of the system, namely, h4 fo where fe : M + R is the true observation. For instance, f. may be the smooth function cos(tz). whereas h E I is its sampling at finite set of points {2I...., } It follows that (h+) is subject to an approximate Koopman representation. However, fundamental challenge in facilitating Koopman theory in practice is the diniiitedddmmiiionaltty of K.p. Recently, several data-driven methods were developed to produce a better approximate Krp using a moderate- size matrix C' (Schmid) 2010 Ovsjanikov et al. 2012). In particular, Koopman-basse have been proven instrumental in the analysis of fluid dynamics data (Brunton et al.) 2021), and for computing complex nonrigid isometric maps between shapes (Ovsjanikov et al. 2016). Motivated by these empirical examples and their success, we will compute in this work approximate Koopman operator matrices C' such that they encode the evolution of latent states {he).
approaches
3

---

Under review as a conference paper at ICLR 2022
3.2 A KOOPMAN-BASER METHOD
We denote by H R tensor of hidden state sequences, where is the batch size, is the sequence length and & is the hidden dimension. The method we employ for computing the matrix C follows two simple steps: 1. Represent the states using basis B, and denote the resulting collection of spectral coefficients by H. 2. Find the best linear transformation C' which maps Hç to H in the spectral domain, where H, E Rexk denotes the tensor of coefficients from H at time т. To give a specific example of the general procedure we just described, we can choose the principal components j 1, 2.. of the truncated SVD of the states H be the basis in the first step. Then, the resulting basis elements are orthonormal, i.e. B B Id, where B (6,) is the matrix of basis elements organized in its columns, and Id is the identity matrix. The matrix C is obtained by solving the following least squares minimization
C arg min Σ
H..
C Hep
2
(4)
H = H, B, VT.
(5)
where is matrix multiplication. We note that the above scheme is variant of the dynamic mode decomposition (Schmid) 2010). and the functional maps (Ovsjanikvv et al. 2012) algorithms.
3.3 KOOPMAN-BASED PREDICTION
The dimenioadimensional Koopman operator in Eq. 3 describes the evolution of observable functions subject to the dynamics yp Similarly, our C' matrices allow us to predict future hidden state h from a given current state hr simply by multiplying C with the spectral coefficients ht, Namely,
H := H.H B. CB.C.
(6)
We will mostly use Eq. (6) to evaluate the validity of C' in encoding the underlying dynamics based on the differences |Haaam / for every admissible t, see Sec. 4
3.4 KOOPMAN-BASE ANALYSIS
The key advantage of Koopman theory and practice is that linear analysis tools can be directly applied to study the behavior of the underlying dynamical system. The tools deseribe next form the backbone of our analysis framework, and our results are heavily based on these tools.
Separable dynamics. If e gixk admits an cigendecomposition, then the dynamics can be represented in a fully separable manner, where the eigenvectors of C propagate along the dynamics independently of the other eigenvectors, scaled by their respective eigenvalues. Formally, we consider the eigenvalues A C and eigenvectors U/- C of C, i.e., it holds that Cu, Ay We assume that C is full-rank and thus V (wy) forms basis of & and similarly, U V-1 is also spanning basis. In our setting, we call the rows of U the Koopman eigenvectors, and we represent any hidden
state h, in this basis, similarly to Eq. (5) The projection of H onto U reads
A, - B H. ..
Then, re-writing Eq. 66 using the eigendecomposstionn of C V- U yields the temporal trajectory of H. via Ae+ H B V ~ H, B- B.C.V CV H, V.V.A. A, A, where A is the diagonal matrix of eigenvalues, and the approximation due to Eq. (6). The latter derivation yields
M ~ H, A
(8)
ie.. the linear dynamics matrix represented in the basis U is simply the diagonal matrix A. and thus U may be viewed as a "natural" basis for encoding the dynamics. Further, it directly follows that Hist Hr Al, that is, the number of steps forward is determined by the eigenvalues power.
4

---

Under review as a conference paper at ICLR 2022
Memory horizon. In addition to their role in describing the evolution of the hidden states, the eigenvalues naturally encode a memory horizon "timestamp" associated with each of the eigenvectors. Specifically, if 1Ayl < 1, then its powers decay exponentially to zero. We define the memory horizon of my to be the decay time (c) of its associated eigenvalue, i.e.,
log(c) (og((>pl)*
where 0 < < 1 is a small threshold parameter. We note that both log(e) and log((Ayl) are negative, and thus TE) > 0. Given a choice of e, the cigenvector uj becomes almost insignificant to the dynamics in Eq. ( after (e) steps as it is scaled by = =\)) In this context, eigenvectors whose eigenvalues satisty [Ayl = have infinite memory, as powers of their eigenvalues simply rotate
over the unit circle. Finally, if PAyl > 1, its respective eigenvector leads to an unstable behavior since
lim 1211 00.
4 RESULTS
In this study, we focus our exploration on the sentiment analysis and the ECG classification problems.
Unless noted otherwise, we always compute C' using the method in Sec 3.2 where the basis is given by the truncated SVD modes of the input hidden states, and C is the least squares estimation obtained
from solving 4 We first provide our qualitative analysis 1. and 4.2 Then, we include in 4.3
quantitative evaluation of KANN and its ability to encode the dynamics. In Apps. [ D and E we
provide additional results, and we show that our method is robust to the choice of basis and network architecture. Finally, we further use KANN to analyze the copy problem in App. where our results outperform the baseline approach (Maheswaranathan al. 2019).
4.1 SENTIMENT ANALYSIS
We begin our qualitative study by considering the sentiment analysis task which was extensively explored in (Maheswaranathan et al. [2019] Mahessaaannthan & Sussillo (020). Determining the sentiment of document is an important problem which may be viewed
binary classification
task. We will use the IMDB reviews dataset, and we will embed the corpus of words to obtain a vector representation of text. Given a review, the role of the network is output whether reflects a positive or negative opinion. Adopting the setup of Maheswaranathan et al. (2019), we use a word embedding of size 128, and GRU recurrent layer with a hidden size of 256. We train the model for 5 epochs during which it reaches an accuracy of ~ 92%, 87%, 87% on the train, validation and test sets, respectively. For analysis, we extract a random test batch of 64 reviews and its states H E 666 1000 where 1000 is the review length when padded with zeros.
One of the main results in (Maheswaranathan et al.) 2019) was the observation that the dynamics of the network span line attractor. That is, the hidden states of the network are dominantly attracted to one dimensional manifold, splitting the domain into positive and negative sentiments. Additionally, Maheswaranathar & Sussillo (2020) study inputs with contextual relations (e.g., the phrase "not bad"), and their effect on the network dynamics. Our results align with the observations in (Maheswaranathan et al., 2019) Maheswaranathan & Sussillo 2020). Moreover, we generalize their results by showing that the attracting manifold is in fact of higher dimension, and that the manifold can be decomposed to semantically understandable components using KANN. Specifically, we demonstrate that several Koopman eigenvectors are important in the dynamics, and we can link each of these eigenvectors to a semantically meaningful action. Thus, in comparison to the framework proposed in (Maheswaranathanl et al.) 2019; Maheswaranathan & Sussillo 2020), our method naturally decomposes the latent manifold into interpretable attracting components. addition, we provide unified framework for reasoning and understanding by drawing conclusions directly from the separable building blocks of the latent dynamics.
Most of our results for the sentiment analysis problem are based the eigendecomposition of C, its resulting eigenvalues {Λ₃ E C} and corresponding eigenvectors {u, E C*) J. For the random states batch H specified above, we obtain an operator C' whose spectrum consists of four eigenvalues with modulus greater than 0.99, i.e., |Ayl > 0.99. In comparison, Maheswaranathhn et al. 2019) identify only single dominant component. The values of our >, read λ₁ 0.9999. Aa 0.9965, and
5

---

Under review as a conference paper at ICLR 2022
A positive review projected onto {ui,
watchee
recall on
comment ng amazi
one ting TRe That
negative review projected onto {uz, un}:
W Wha PRiR mobrardon 2x porformarce 2I01 phony nooee Jan corcer do from
noviol
For robarta and
mould be tott
pewnore
Two reviews with contextual information projected onto {ua}:
DaY george Cloorer
h Foundtran track ithe nan constant sorFON esl
eSCOM
man ead this movie to everysody. greetings bert
tait
and rip.ntt Alrnady STAPPy hally StOry clasS
what YOU
lost Summer
path
comething
the tne time. not good
NOT happen
Fore happeas KERrY the aC aectrre ard R 95 IT. 05 and the 9Irls AIria ATe COIR iate er INCY rC DSi3n 821 50 2N1N
movie trios to
give INO OUT OF
AND
We display reviews where each word is shaded based on E, s(j, hz). Projecting onto U12 shows an increase in magnitude for several positive words (blue), whereas projecting onto Usi shows jumps in magnitude around negative words (red). See e.g., amaz aing special (blue), mess, waste, muddled (red). We also show two reviews with contextual information which is naturally highlighted due Us.
Figure I:
Aa.4 0.9942 + i0.0035. Consequently, their respective eigenvectors have long memory horizons. For instance, if we set e = 1e-2 in Eq. (9). then we get a time stamp r > 800 for all four eigenvectors. Namely, these eigenvectors carry information across word sequences of length up to 800, and only < 2% of the reviews in the IMDB dataset are that long.
In their analysis, the authors of (Mahesaaranathan et al. 2019) observe that the network mainly counts positive vs. negative words in a review along a line attractor. We hypothesize that in our setting, the dominant eigenvectors (ui, .... u4} are responsible for this action. We expect that complex conjugates such as v.3 and u4 share the same role, e.g., counting the negative words, and we expect that U12 [u1, (22 take the role of counting the positive words. To verify our hypothesis, the readout layer of the model to generate the logits of the state when projected to U12 and U31. We denote by 212 and 1/44 the logits for V12 and H V34. respectively, where Vij denote the i and columns of V. For the above test batch, we get perfect correspondence, i.e., U12 < .5 and Usa > on all samples. In addition to encoding certain sentiment, the Koopman eigenvectors are advantageous in comparison to a single line attractor as they allow for direct visualization of the importance of words in a review. Specifically, we define the projection magnitude of hidden state as follows
$(j, he) (n) K Vg
(10)
We show in Fig. two examples where the magnitude of projection onto U12 and Usa clearly highlights positive and negative words, respectively. In particular, as the network "reads" the review and identifies e.g., a negative word, increases s(-). For instance, see mes s and muddled Fig. [
Importantly, there may be occurrences of positive/negative n-grams which are not highlighted, such as the word good in the positive example. We show in App. A that the above results extend the entire test set. Thus, we conclude that {ui,..., } track positive and negative words.
In addition to (uI,...,u.k we also want to understand how other eigenvectors affect the latent dynamics. We hypothesize that other vectors are responsible to track contextual information such as amplifiers ("extremely good") and negations ("not bad"). We collected reviews that include the phrases "not bad"" and "not good" into single batch, yielding states tensor with 256 samples. One way to check our hypothesis is to employ the former visualization using other eigenvectors. We show such examples in Fig. green, where phrases such "not bad", "terribly wrong", and

---

Under review as a conference paper at ICLR 2022
-2-
Moa 29
Po5.25
Time
Time
Figure 2:
We show the median of reconstructions of normal beats when projected to each of the first four dominant conjugate pairs of Koopman eigenvectors. The medians (orange, green, red, brown) are plotted on top of the original signals and their reconstruction medians. The dashed black lines indicate important features of the signals which are well captured by the Koopman eigenvectors.
"quite ok" are highlighted when projected onto ug. We provide additional results and analysis on the identification of amplifier words and negations using XANN in App. A In addition to unigram and bigram highlighting, in App. B we also consider KANN in the general case of n-grams where >> 2.
4.2 ECG CLASSIFICATION
Electrocardiogram (ECG) tests track the electrical activity in the heart, and they help detect various abnormalities in a non-invasive way. Classifying whether a beat is normal or not is challenging task which lacks descriptive neural models. A common approach for solving the classification problem using neural networks trains an autoencoder model with an L loss over the normal beats. Classification performed by measuring the loss between the original and reconstructed signals; thus, while it is a classification task, ECG classification is solved via a regression model. In particular, high loss values indicate anomalous beats, whereas low values are attributed to normal signals. Typically, threshold is set during the training phase, allowing automatic classification on the test set. We fix the threshold to be 26. Our network is composed of single layer LSTM encoder Fene with hidden size of 64, and an LSTM decoder Fase with one laver as well. We use publicly available subset of the MIT-BIH arrhythmia database (Goldberger et al., 2000) for our data, named ECG500 This dataset includes 5000 sample heartbeats with a sequence length of 140. Around 60% of the sequences classified as normal and the rest are various anomalous signals. The model trained for 150 epochs, yielding an accuracy of 97. 1 %, 97.6%, 98.6% the train, validation and test sets, respectively.
Similarly to the sentiment analysis problem, we expect the Koopman eigenvectors to take a significant role in encoding the latent dynamics. Given that the network is generative it is autoencoder, hypothesize that the eigenvectors {uz} capture dominant features of normal beats. Thus, we project normal beats onto pairs of dominant eigenvectors, and decode the resulting hidden states using the decoder to study the obtained signals. For example, say U38.59 are dominant, then we project onto the space spanned by this pair, then project back and decode. Using test batch of 64 normal beats we collect the last hidden state of every sample in a matrix Hoo E R4 64 and we compute the following.
R 64x140
X Facc B- --IV Ung UbllBB)),
(11)
where | is the element-wise modulus of complex numbers. To determine which eigenvectors are dominant, we employ Eq. (10)
To visualize the results, we take the set of reconstructed signals X of particular pair ij, and we compute its median My (t) for t E [1, [I....,7, n). We plot these graphs in Fig, 2 using colors orange (pair
58 58-59), green (pair 25 25-26), 26), red (pair 9-10), and brown (pair 50-51). In addition, the original signals'

---

Under review as a conference paper at ICLR 2022
Aug: Siloouttte Score
0.6
deca
decepman
Mi
0.5
0.4
Time
Time
Figure 3: The network reconstructs relatively well both normal and anomalous signals (left), implying its inner representation is binary. However, the silhouette scores of the Koopman embedding (right) imply that only single class is being learnt in practice.
median μ (purple) and the median of the signals reconstruction μ4, (blue) are shown in each of the subplots for comparison. Indeed, μ and pr are almost indistinguishable, and can be differentiated only when zooming in. Each median graph is wrapped in its median absolute deviation envelope. We preferred median-based quantities over the common mean and standard deviation since the latter produce cluttered plots in our setting due to outliers. The plots in Fig. clearly show that each conjugate pair captures a different feature of the time series as marked by the vertical dashed lines. Specifically, (488.00 .99 captures the minimum around = 3, and 225,20 encodes the part of the signal in t E [35, 75]. Moreover, 19,10 attains the maximum at t = 103, and /430,31 is approximating the lower peak at 133 and we consider these values to be the salient features. Importantly, the other Koopman eigenvectors beyond the ones we consider above are less important in the reconstruction, and are mostly helpful in fixing minor variations. Finally, we provide similar computation in App. D using the dominant PCA modes and Kernel PCA eigenvectors, where we show that PCA. components and KerenelPCA cigenvectors are not useful in identifying the dominant features of beat signals. Also, we provide a quantitative comparison between the methods.
In addition to identifying principal features of beat signals, we show in what follows that the Koopman eigenvectors are also instrumental in analyzing the latent structure of the LSTM autoencoder. We begin by showing in Fig. B (left) the median values over time of normal batch and its reconstruction (as in Fig. (), task of ECG classification may be viewed as a binary classification problem, separating normal from anomalous signals via reconstruction. However, we will now show that this is actually not the case. Instead, the network essentially encodes inputs, whether normal or anomalous, in a representation that is closer to the manifold of normal signals. To demonstrate and analyze this phenomenon, we consider hidden state tensors H and Han of normal and anomalous beats, respectively, and we concatenate these tensors over the samples yielding H (H Ham) R. 128 x 140x64 We would like to study the decision boundary separating between different signals in the latent space.
2 and similarly for a batch of anomalous signals (orange and brown). From this data, the
To this end, we employ a standard measure known as the silhouette score (Rousseeuw 1987) to quantify the class separation quality. The silhouette score a is real value in 11, 1J. where scores close to 1 mean the latent states are well separated. In contrast, values closer to zero indicate that samples are located on close to the decision boundary. We compute the silhouette score estimates on H H- B averaged over samples, and cumulatively averaged over time. Namely, (t) 1 >x i (h..t). where (hh,,) is the silhouette score of the vector ha,t with E [L...., .., 128] and / E (I, 140). We compare three silhouette score estimates denoted by: a for the original H, OPCA for the projection of H onto its first five principal components, and Koopman for the projection of H onto the first five dominant Koopman eigenvectors. The results are shown in Fig. 3 (right), where Koopman's 's embedding attains low scores compared to PCA and the original states. Namely, embedding the hidden states using Koopman cigenvectors reveals that the decision boundary between normal and anomalous signals somewhat blurred, in contrast to the numerical results provided by the reconstructed signals and other embeddings. This understanding provides rather straightforward interpretation the model: it simply encodes the dominant components of all signals as being normal, allowing to easily identify anomalous signals later by measuring their reconstruction error. Finally, our analysis shows that Koopman eigenvectors successfully identify the salient features of normal beat signals. We conclude from this observation that the network focuses on identifying these features and reconstructing them accurately. A correct reconstruction of these salient features allows to subsequently identify using simple loss check whether signal is normal or anomalous. Importantly,
8

---

Under review as a conference paper at ICLR 2022
Sentiment analysis
ECG Clossification
met
Moncriig Marttinng percetlage
1:0l
20
60
300
loss
Figure 4: On the tasks we consider, our approach approximately reproduces the network classification outputs. In particular, we obtain > 99% and 97% agreement on the sentiment analysis and ECG classification problems, respectively. See the text for additional details.
we show in Fig. that the network indeed successfully reconstructs normal and anomalous signals. The understanding that we obtain using KANN is that the network mainly focuses on these salient features during the reconstruction.
4.3 KANN REPRODUCES THE LATENT DYNAMICS
We now perform quantitative study of the ability of C. to truly capture the latent dynamics. We will show that indeed, KANN is able to reproduce the nonlinear dynamics of the network in Eq. ① to high degree precision, and thus we achieve the empirical justification replace F' with C. To this end, we consider the following two metrics:
Relative error of hidden states: let {ha.}} be collection of states over samples L..., S and across time t = I,..., T. We generate the predicted collection (h using Eq. (6). and we compute
Grel ( (AEAN J. (ha,t))
1
Σ
1n MANNN
h 12 1
(12)
T T
2. Accuracy error: let G be the neural network component that takes state and produces the output of the model, i.e., G(h+) Dt. We denote by & the category predicted by Øt. for instance & arg max()).. We compare the difference between & and of atAMs obtained from G(GāAN)) J
We show in Fig. 4 the results of our quantitative study. For the sentiment analysis problem (Fig. 4 left), obtain 99% correspondence with the classification of the network over all the test set, as is shown for the True Positive (TP) and True Negative (TN) columns vs. the False Positive (FP) and False Negative (FN) columns. In the ECG classification task (Fig. right), we reconstruct 145 signals of the normal test set and compute their loss. There is a noticeable yet small shift in the loss histogram between the network reconstruction (blue) in comparison to our reconstruction (orange). However, the threshold for this problem set at 26 during training (black dashed line) yields particular, the false classification of normal signals (around also computed the relative error of
97% agreement in classification. loss 90) appear both in the network output and in ours. Finally, the hidden states, obtaining e = 0.095 on a batch of size 64 for the sentiment analysis task, and evel = 0.0056 on batch of size 145 for the ECG classification problem. Overall, the above results demonstrate that KANN faithfully represents the latent dynamics.
5 DISCUSSION
In this work we presented a novel framework for studying sequence neural models based on Koopman theory and practice. Our method involves dimensionality reduction representation of the states, and the computation of a linear map between the current state and the next state. Key to our approach is the wealth of tools we can exploit from linear analysis and Koopman-related work. In particular, we compute linear approximations of the state paths via simple matrix-vector multiplications. Moreover, we identify dominant features of the dynamical system and their effect on inference and prediction. Our results on the sentiment analysis problem, and the ECG classification challenge provide simple yet accurate descriptions of the underlying dynamics and behavior of the recurrent models. Our work lays the foundations to further develop application-basee descriptive frameworks, towards an improved understanding of neural networks. the future, we plan to explore our framework during the training of the model, where in this work focused only trained models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael W. Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning. volume 119, pp. 775-485,25 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Yoshua Bengio, Paolo Frasconi, and Patrice Simard. The problem of learning long-term dependencies in recurrent networks. In IEEE international conference on neural networks, pp. 1183-1188, IEEE, 1993.
Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz. Modern Koopman theory for dynamical systems. arXiv preprint arXiv:2102.12085 2021.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. arXiv preprint (rXiv:2012.09838, 2020.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pp. 1724-1734, 2014.
Felix Dietrich, Thomas N Thiem, and Ioannis G Kevrekidis. On the Koopman operator of algorithms. SIAM Journal on Applied Dynamical Systems, 19(2):80-885,, 2020.
Akshunna S. Dogra and William T. Redman. Optimizing neural networks via Koopman operator theory. In Advances Neural Information Processing Systems, 2020.
Kenji Doya. Bifurcations of recurrent neural networks in gradient descent learning. IEEE Transactions on neural networks, 1755):118, 1993a.
Kenji Doya. Universality of fully connected recurrent neural networks. Dept. of Biology, UCSD, Tech. Rep, 1993b.
Tanja Fisner, Bálint Farkas, Markus Haase, and Rainer Nagel. Operator theoretic aspects of ergodic theory, volume 272. Springer, 2015.
Jeffrey I. Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
Benjamin Erichson, Michael Muchlebach, and Michael W Mahoney. Physics-informed autoen- coders for lyapunov-stable fluid flow prediction. arXiv preprint arX::: 2019.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Represen- tations. 2021.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George B Moody, Chung-Karn Peng, and H Eugene Stanley. Physiobank. physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23)ee21--e22) 2000.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv: arXiv: 1506.02076 2015.
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931.
10

---

Under review as a conference paper at ICLR 2022
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional Koopman operators for model-based control. In International Conference on Learning Representations. 2020.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 91)::-11,, 2018.
Niru Maheswaranathan and David Sussillo. How recurrent networks implement contextual processing in sentiment analysis. arXiv preprint arXiv:2004.0800 00000,, 2020.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems, pp. 15696-15705, 2019.
Iva Manojjović, Maria Fonoberova, Ryan Mohr, Aleksandr Andrejeuk, Zlatko Drmač, Yannis Kevrekidis, and Igor Mezić. Applications of Koopman mode analysis to neural networks, arXiv preprint arXiv:2006. 1765, 2020.
Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. VAMPnets for deep learning of molecular kinetics. Nature communications, 9()):1111, 2018.
Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlin- ear Dynamics, 41(1-3):309-322 2005.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems, pp. 9258-9268, 2018.
Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Func- tional maps: flexible representation of maps between shapes. ACM Transactions on Graphics (TOG), 31(4):1-11, 2012.
Maks Ovsjanikov, Etienne Corman, Michael Bronstein, Emanuele Rodolå, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, and Alex Bronstein. Computing and processing correspon- dences with functional maps. In SIGGRAPH ASIA 2016 Courses, pp. 1-60. 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318, 2013.
Ohad Rahamim and Ronen Talmon. Aligning sets of temporal signals with Riemannian geometry and Koopman operator. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5310-5314, IEEE, 2021.
Peter J Roussceuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
Abhishek Sharma and Maks Ovsjanikov. Weakly supervised deep functional maps for shape matching. In Advances in Neural Information Processing Systems, 2020.
Raj Kishor Singh and Jasbir Singh Manhas. Composition operators on function spaces. Elsevier, 1993.
Steven H Strogatz. Nonlinear dynamics and chaos: With applications to physics, biology, chemistry, and engineering. CRC press, 2018.
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high-
dimensional recurrent neural networks. Neural computation, 25(3):626-649 2013.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, pp. 1 1130-11400 2017.
11

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
12


Reviews:
[{"review_id": "z9ezjXoPaB", "reviewer": "Reviewer_Ts2w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a new scheme for understanding the representations captured by sequence neural models. They build upon the Koopman operator and provide a framework for studying dynamical NNs. The paper is well written, and the method is clearly explained. The results demonstrate that the Koopman operator can capture the essential information behind the dynamics captured by supervised sequence-based NNs. \nIn what follows, I will detail my comments that could help improve the manuscript.\n-The abstract is not very clear, specifically “local theories” are not defined at this point, so this statement is confusing. \n-On the same point, the abstract does not provide intuition on the method; please expand.\n-” However, the local nature”- this statement is again not clear.\n-Last paragraph of the intro: “dominant features in normal beat signal”- this is not clear at this point what you mean. I had to read the experimental part to understand. Would you please make the description in the introduction self-explanatory?\n-” which is typically possible in most day-to-day” -can you add examples here?\n-P3 “moderate size matric C” - the matrix C is not yet defined at this point.\n-4.1 what are the sizes of the train/test/valid, also? How are the parameters of the network tuned?\n-4.2 same questions as above.\n-4.2 how was the threshold 26 selected?\n -”the network is generative” in what sense is it generative?\n-the comparison to PCA does not completely convince me; could you quantify the advantage over PCA? Also, can you try to use the eigenvectors of Kernel-PCA?\n-References from the main text to some of the appendix parts are missing, for example, Appendix C.\n", "summary_of_the_paper": "The authors propose a method for analyzing NN applied to temporal signals (e.g., dynamical systems). The proposed method relies on the Koopman operator, which is widely used for studying dynamical systems. The authors use the eigenvectors of the Koopman operator to interpret predictions made by NN. The proposed method is evaluated on two tasks, namely: sentiment analysis and electrocardiogram (ECG) classification. The authors demonstrate that the eigenvectors of the Koopman operator do encode the latent information in these examples. \n", "summary_of_the_review": "To summarize, the proposed method is interesting and could lead to several extensions. The English level is OK, and the writing is mostly clear. However, I believe that some of the descriptions (mainly in the results section) should be strengthened before the paper can be published. Moreover, I would expect a comparison to more baselines in the main text, like PCA or Kernel PCA. For these reasons, my recommendation is: marginally above the acceptance threshold.\nI would be happy to raise my score if the authors could address the points I raised.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "yCD6E-20ky", "reviewer": "Reviewer_XyzC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Koopman analysis is an interesting technique, and to date, I have not seen it used to try and understand the dynamics of trained neural networks. However, I have a number of concerns about how this paper goes about conducting Koopman analysis, outlined below. I am hoping that the authors can clarify my confusion.\n\n## Major concerns\n\nMy understanding of Koopman analysis is that if you want to understand a nonlinear dynamical system, you can find a set of nonlinear, transformed coordinates of the state variables such that there is a _linear_ operator (known as the Koopman operator) that advances the coordinates in the transformed space. This means that we can then use powerful tools from linear dynamical systems analysis to understand the system. The main downside is that we may require an infinite set of transformed coordinates, or that it may be hard to find the right transformation.\n\nGiven this understanding, I expected this paper to do something like the following: for a given neural sequence model, find a set of (nonlinear) transformed coordinates such that the nonlinear dynamics of the neural model can be approximated by linear dynamics in the transformed coordinates. Then, study the (linear) dynamics of the transformed coordinates, and show how they relate to the (nonlinear) computations in the original coordinates. If this were the case, I would be very excited about this paper! However, as far as I can tell, this is _not_ what the authors actually did.\n\nInstead, it looks like the authors directly fit a _linear_ dynamical system to the hidden state dynamics. For example, the optimization problem in equations (4) and (5) shows least squares fitting of a _linear_ operator to the hidden states (yes, they use a different linear basis, but it’s still linear, so the whole system $B C B^T$ is still linear as shown in equation (6).\n\nThis means that the authors are approximating the nonlinear dynamics with a _global_ linear operator. This strikes me as only being a good idea if the network solves the task using linear dynamics, which may be approximately true for simple tasks, but is most certainly not true for complex tasks. At a minimum, these drawbacks of the method need to be discussed in Section 3. It would be even better to see what happens if one attempts to fit a Koopman operator to a neural network that uses nonlinear dynamics to solve a task. Are the eigenmodes still interpretable? Are they misleading?\n\nFor example, if we look at the sentiment classification results, it looks like the network solves the class using a plane attractor, similar to the network analyzed in [Maheswaranathan et al, 2019](https://arxiv.org/abs/1906.10720). (As an aside, I suspect the reason for the plane, as opposed to line, attractor has to do with the amount of l2 regularization applied to the network weights during training. With low values of l2 regularization, networks often learn higher dimensional attractors than necessary. This is also discussed at the end of section 2 in [Aitken et al 2020](https://arxiv.org/abs/2010.15114). At this point, I have to admit the differences between KANN and RENN seem minor. Both identify a linear integration mechanism for solving the task, and both can be used to quantify the “valence” of particular words as seen by the network.\n\nHowever, I fail to see how the KANN approach is useful at identifying the computations that underlie contextual processing (phrases such as “not bad” or “very good”). The KANN approximation, being purely linear, implies by definition that the network’s interpretation of any word (such as “bad”) is _independent_ of surrounding context. This means that the phrases “not bad and very good” and “very bad and not good” should have the same predictions. Is this the case for the KANN approximation? It might not be true for the underlying (nonlinear) RNN, but must be true for the linear approximation. Is this not a drawback of the KANN approach? By assuming the global dynamics are linear, we hinder our ability to understand how the network processes contextual input.\n\nThe analysis done with eigenmode u8 (as an aside, what happened to eigenmodes u5-u7? Why was u8 chosen?) does not really reveal _how_ the network understands the phrase “not bad”. In fact, it does not even tell us what the effect of eigenmode u8 is, because we do not know what the projection of eigenmode u8 is onto the readout weights (output weights) of the network. It’s possible that due to the readout projection, u8 thinks “not bad” has _negative_ sentiment.\n\nI think the analysis in [Maheswaranathan et al 2020](https://arxiv.org/abs/2004.08013) more thoroughly breaks down _how_ the network correctly asses contextual phrases such as “not bad”--through _piecewise_ linear approximations of the dynamics. These piecewise linear approximations are sufficiently nonlinear to describe and understand contextual effects. By contrast, the method proposed here cannot identify these types of computations, because of the assumption of globally linear dynamics.\n\nI am less familiar with the ECG arrhythmia dataset, but I did not quite understand how the pictures in Figure 2 revealed much about the network’s behavior. The reconstruction (which is a linear approximation of the dynamics) will be a linear combination of the projections along each eigenmode, so by definition don’t the eigenmodes have to capture the important features in the signal? That doesn’t seem surprising to me.\n\n## Minor comments\n\n- Section 3.3: Typo: “our C matrices allow _us_ to predict a future state …”\n- For any ECG figures, such as Figure 2, what are the units on the time axis, and what is the y-axis?\n- In Figure 4, should make it clear in the Figure label that the “percentage” is not accuracy on the test set, but agreement with the baseline model.\n", "summary_of_the_paper": "This paper proposes using Koopman analysis to study the dynamics of neural sequence models. The proposal involves fitting linear approximations to the hidden state dynamics of a sequence model.\n\nThe paper applies this idea to two applications: RNNs trained on IMDB reviews for sentiment classification, and RNN autoencoders trained on ECG traces used for arrhythmia classification.", "summary_of_the_review": "This work presents a new application of Koopman analysis for understanding neural sequence models.\n\nHowever, it does not adequately present drawbacks of the current method (assuming globally linear dynamics), nor does it test the approach in settings where nonlinear computations are required.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BS3wScEXDet", "reviewer": "Reviewer_Dka9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. Applies the Koopman operator framework to develop some insights into two sequence modeling problems. \n\nComments: \n\n1. My main issue with this paper is that this method is not new, and it doesn't really add anything from a methods standpoint. It's more that it applies an existing framework (I would not agree this is novel) to some model/prediction tasks which it has not been before. \n2. It seems like it would be a better fit for an applied conference focusing on the problems in question (e.g., sentiment analysis). However, even then, I think it would be necessary to apply the Koopman analysis to several datasets and have more developed insights using the Koopman analysis. \n", "summary_of_the_paper": "The paper applies the developed theory of the Koopman operator to analyzing neural sequential models empirically, on two tasks: ECG and sentiment analysis, and derives some insights on what the models are doing using the spectrum of the Koopman operator.\n", "summary_of_the_review": "Overall, I lean on the rejection side: the paper is not that novel techniques-wise in my opinion, and the insights on the applications need to be more developed and thorough.\n\n\nUPDATE:\n\nI read the other reviews and responses and my opinion is unchanged: The final response to my questions was not convincing overall in terms of the insights provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6R2gMZVGJjz", "reviewer": "Reviewer_6ZvH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-The general idea and the results of this work are interesting and can be extended in many directions\n\n-The problem considered is nice, and has a clear audience, but their approach doesn't seem to be novel enough\n\n-The paper is well written, but some parts are not clear enough. Especially, some math parts still need more clarification\n\n*Some issues that, in my view, could improve the paper are*:\n\n- On page 3, section 3.1, line 23, how can we write \"h_t ≈ f_t\" while two functions h_t and f_t have different of domain and range? In line 24, the function cos(tz) is not a general appropriate example for f_t: M ---> R (it is only true for f_t: R---> R)\n\n- In section 3.2 some notations and explanations are not so clear. In particular, there is no discussion on the size of the matrices B and C\n\n- There are some grammatical mistakes as well as typos ", "summary_of_the_paper": "The paper proposes a framework for studying sequence neural models based on Koopman theory. Particularly, the authors compute linear approximations of the state paths via simple matrix-vector multiplications. Also, the dominant features of the dynamical system and their effect on inference and prediction is determined. Their results on the sentiment analysis problem, and the ECG classification challenge provide simple yet precise descriptions of the underlying dynamics and behavior of the recurrent models. ", "summary_of_the_review": "Some parts of the paper are not so clear and require more clarification. Their approach doesn't seem to be novel enough. The experimental results and comparisons to other existing methods are inadequate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to apply the Koopman operator theory framework for analyzing sequence neural models. The authors considered two particular applications, namely sentiment analysis and ECG (electrocardiogram) classification.

Reviewers generally agree that the results obtained on the two tasks are interesting. However, there are concerns that the paper lacks methodological novelty (concerning the Koopman operator framework, which the authors agreed) and that the paper would be more suited for an applied conference and/or journal.
Decision: Reject