Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for studying sequence neural models based on Koopman theory. Particularly, the authors compute linear approximations of the state paths via simple matrix-vector multiplications. Also, the dominant features of the dynamical system and their effect on inference and prediction is determined. Their results on the sentiment analysis problem, and the ECG classification challenge provide simple yet precise descriptions of the underlying dynamics and behavior of the recurrent models. 
main_review: -The general idea and the results of this work are interesting and can be extended in many directions

-The problem considered is nice, and has a clear audience, but their approach doesn't seem to be novel enough

-The paper is well written, but some parts are not clear enough. Especially, some math parts still need more clarification

*Some issues that, in my view, could improve the paper are*:

- On page 3, section 3.1, line 23, how can we write "h_t ≈ f_t" while two functions h_t and f_t have different of domain and range? In line 24, the function cos(tz) is not a general appropriate example for f_t: M ---> R (it is only true for f_t: R---> R)

- In section 3.2 some notations and explanations are not so clear. In particular, there is no discussion on the size of the matrices B and C

- There are some grammatical mistakes as well as typos 
summary_of_the_review: Some parts of the paper are not so clear and require more clarification. Their approach doesn't seem to be novel enough. The experimental results and comparisons to other existing methods are inadequate.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using Koopman analysis to study the dynamics of neural sequence models. The proposal involves fitting linear approximations to the hidden state dynamics of a sequence model.

The paper applies this idea to two applications: RNNs trained on IMDB reviews for sentiment classification, and RNN autoencoders trained on ECG traces used for arrhythmia classification.
main_review: Koopman analysis is an interesting technique, and to date, I have not seen it used to try and understand the dynamics of trained neural networks. However, I have a number of concerns about how this paper goes about conducting Koopman analysis, outlined below. I am hoping that the authors can clarify my confusion.

## Major concerns

My understanding of Koopman analysis is that if you want to understand a nonlinear dynamical system, you can find a set of nonlinear, transformed coordinates of the state variables such that there is a _linear_ operator (known as the Koopman operator) that advances the coordinates in the transformed space. This means that we can then use powerful tools from linear dynamical systems analysis to understand the system. The main downside is that we may require an infinite set of transformed coordinates, or that it may be hard to find the right transformation.

Given this understanding, I expected this paper to do something like the following: for a given neural sequence model, find a set of (nonlinear) transformed coordinates such that the nonlinear dynamics of the neural model can be approximated by linear dynamics in the transformed coordinates. Then, study the (linear) dynamics of the transformed coordinates, and show how they relate to the (nonlinear) computations in the original coordinates. If this were the case, I would be very excited about this paper! However, as far as I can tell, this is _not_ what the authors actually did.

Instead, it looks like the authors directly fit a _linear_ dynamical system to the hidden state dynamics. For example, the optimization problem in equations (4) and (5) shows least squares fitting of a _linear_ operator to the hidden states (yes, they use a different linear basis, but it’s still linear, so the whole system $B C B^T$ is still linear as shown in equation (6).

This means that the authors are approximating the nonlinear dynamics with a _global_ linear operator. This strikes me as only being a good idea if the network solves the task using linear dynamics, which may be approximately true for simple tasks, but is most certainly not true for complex tasks. At a minimum, these drawbacks of the method need to be discussed in Section 3. It would be even better to see what happens if one attempts to fit a Koopman operator to a neural network that uses nonlinear dynamics to solve a task. Are the eigenmodes still interpretable? Are they misleading?

For example, if we look at the sentiment classification results, it looks like the network solves the class using a plane attractor, similar to the network analyzed in [Maheswaranathan et al, 2019](https://arxiv.org/abs/1906.10720). (As an aside, I suspect the reason for the plane, as opposed to line, attractor has to do with the amount of l2 regularization applied to the network weights during training. With low values of l2 regularization, networks often learn higher dimensional attractors than necessary. This is also discussed at the end of section 2 in [Aitken et al 2020](https://arxiv.org/abs/2010.15114). At this point, I have to admit the differences between KANN and RENN seem minor. Both identify a linear integration mechanism for solving the task, and both can be used to quantify the “valence” of particular words as seen by the network.

However, I fail to see how the KANN approach is useful at identifying the computations that underlie contextual processing (phrases such as “not bad” or “very good”). The KANN approximation, being purely linear, implies by definition that the network’s interpretation of any word (such as “bad”) is _independent_ of surrounding context. This means that the phrases “not bad and very good” and “very bad and not good” should have the same predictions. Is this the case for the KANN approximation? It might not be true for the underlying (nonlinear) RNN, but must be true for the linear approximation. Is this not a drawback of the KANN approach? By assuming the global dynamics are linear, we hinder our ability to understand how the network processes contextual input.

The analysis done with eigenmode u8 (as an aside, what happened to eigenmodes u5-u7? Why was u8 chosen?) does not really reveal _how_ the network understands the phrase “not bad”. In fact, it does not even tell us what the effect of eigenmode u8 is, because we do not know what the projection of eigenmode u8 is onto the readout weights (output weights) of the network. It’s possible that due to the readout projection, u8 thinks “not bad” has _negative_ sentiment.

I think the analysis in [Maheswaranathan et al 2020](https://arxiv.org/abs/2004.08013) more thoroughly breaks down _how_ the network correctly asses contextual phrases such as “not bad”--through _piecewise_ linear approximations of the dynamics. These piecewise linear approximations are sufficiently nonlinear to describe and understand contextual effects. By contrast, the method proposed here cannot identify these types of computations, because of the assumption of globally linear dynamics.

I am less familiar with the ECG arrhythmia dataset, but I did not quite understand how the pictures in Figure 2 revealed much about the network’s behavior. The reconstruction (which is a linear approximation of the dynamics) will be a linear combination of the projections along each eigenmode, so by definition don’t the eigenmodes have to capture the important features in the signal? That doesn’t seem surprising to me.

## Minor comments

- Section 3.3: Typo: “our C matrices allow _us_ to predict a future state …”
- For any ECG figures, such as Figure 2, what are the units on the time axis, and what is the y-axis?
- In Figure 4, should make it clear in the Figure label that the “percentage” is not accuracy on the test set, but agreement with the baseline model.

summary_of_the_review: This work presents a new application of Koopman analysis for understanding neural sequence models.

However, it does not adequately present drawbacks of the current method (assuming globally linear dynamics), nor does it test the approach in settings where nonlinear computations are required.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for analyzing NN applied to temporal signals (e.g., dynamical systems). The proposed method relies on the Koopman operator, which is widely used for studying dynamical systems. The authors use the eigenvectors of the Koopman operator to interpret predictions made by NN. The proposed method is evaluated on two tasks, namely: sentiment analysis and electrocardiogram (ECG) classification. The authors demonstrate that the eigenvectors of the Koopman operator do encode the latent information in these examples. 

main_review: The authors propose a new scheme for understanding the representations captured by sequence neural models. They build upon the Koopman operator and provide a framework for studying dynamical NNs. The paper is well written, and the method is clearly explained. The results demonstrate that the Koopman operator can capture the essential information behind the dynamics captured by supervised sequence-based NNs. 
In what follows, I will detail my comments that could help improve the manuscript.
-The abstract is not very clear, specifically “local theories” are not defined at this point, so this statement is confusing. 
-On the same point, the abstract does not provide intuition on the method; please expand.
-” However, the local nature”- this statement is again not clear.
-Last paragraph of the intro: “dominant features in normal beat signal”- this is not clear at this point what you mean. I had to read the experimental part to understand. Would you please make the description in the introduction self-explanatory?
-” which is typically possible in most day-to-day” -can you add examples here?
-P3 “moderate size matric C” - the matrix C is not yet defined at this point.
-4.1 what are the sizes of the train/test/valid, also? How are the parameters of the network tuned?
-4.2 same questions as above.
-4.2 how was the threshold 26 selected?
 -”the network is generative” in what sense is it generative?
-the comparison to PCA does not completely convince me; could you quantify the advantage over PCA? Also, can you try to use the eigenvectors of Kernel-PCA?
-References from the main text to some of the appendix parts are missing, for example, Appendix C.

summary_of_the_review: To summarize, the proposed method is interesting and could lead to several extensions. The English level is OK, and the writing is mostly clear. However, I believe that some of the descriptions (mainly in the results section) should be strengthened before the paper can be published. Moreover, I would expect a comparison to more baselines in the main text, like PCA or Kernel PCA. For these reasons, my recommendation is: marginally above the acceptance threshold.
I would be happy to raise my score if the authors could address the points I raised.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper applies the developed theory of the Koopman operator to analyzing neural sequential models empirically, on two tasks: ECG and sentiment analysis, and derives some insights on what the models are doing using the spectrum of the Koopman operator.

main_review: Strengths: 
1. Applies the Koopman operator framework to develop some insights into two sequence modeling problems. 

Comments: 

1. My main issue with this paper is that this method is not new, and it doesn't really add anything from a methods standpoint. It's more that it applies an existing framework (I would not agree this is novel) to some model/prediction tasks which it has not been before. 
2. It seems like it would be a better fit for an applied conference focusing on the problems in question (e.g., sentiment analysis). However, even then, I think it would be necessary to apply the Koopman analysis to several datasets and have more developed insights using the Koopman analysis. 

summary_of_the_review: Overall, I lean on the rejection side: the paper is not that novel techniques-wise in my opinion, and the insights on the applications need to be more developed and thorough.


UPDATE:

I read the other reviews and responses and my opinion is unchanged: The final response to my questions was not convincing overall in terms of the insights provided.
review_title: null

Meta-review: This paper proposes to apply the Koopman operator theory framework for analyzing sequence neural models. The authors considered two particular applications, namely sentiment analysis and ECG (electrocardiogram) classification.

Reviewers generally agree that the results obtained on the two tasks are interesting. However, there are concerns that the paper lacks methodological novelty (concerning the Koopman operator framework, which the authors agreed) and that the paper would be more suited for an applied conference and/or journal.
Decision: Reject