Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ### Paper Summary

This paper sheds light on how trained RNNs solve text classification problems by analyzing them from a dynamical systems perspective. It extends recent work where a similar analysis was applied to the simpler setting of binary sentiment classification. When projecting the RNN hidden states to principal dimensions that explain most of the variance, the authors find (N-1) dimensional simplex attractors for N-dimensional classification, 2D attractors for ordered classification, and N-dimensional hypercubes for multi-label classification. 

### Strengths

In general, the paper is clear and enjoyable to read.

The use of synthetic data strengthens the analysis by first showing what behavior one might expect from RNNs on "clean" data, and then comparing to what is observed on natural data.

The use of different categories of tasks not only makes the experimentation thorough, it also clarifies understanding by contrasting behavior across these categories.

Based on the presented dynamical systems analysis, I think practitioners can acquire valuable intuitions about how RNNs process information (at least for certain problems) that is "correct", i.e. it has been verified by sufficient experimentation.

Finally, I agree with the authors that the paper motivates further work into understanding the behavior of RNNs from this perspective.

### Weaknesses

I found the multi-label classification experiments on natural data to be limited, since the authors only used reduced 2- and 3-class versions of the GoEmotions dataset and the behavior was not as "clean" as that for synthetic data for the 3-class setting (App. E.5). This part of the analysis leaves me somewhat uncertain about general behavior to expect for multi-label classification in practice.

### Review Summary

I found this paper to be interesting, well-written and useful. I recommend an Accept, but the authors can improve the paper by strengthening the multi-label classification experiments.

### Post Author Response

I thank the authors for their hard work and I'm happy to see that the analysis has improved.

review_title: Interesting paper with useful analyses

final_score: rating: 7: Good paper, accept
review_content: ### Summary:
 
Weak reject. This paper might be a weak accept if revised to present its key contributions more clearly. 


### Reasons for score: 
 
This paper is an interesting extension of the work from “Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics” (Maheswaranathan et al, 2019, https://arxiv.org/abs/1906.10720). The work presents new insights into the dimensionality and workings of low-dimensional attractors in which RNNs accumulate “evidence” in their hidden layer. The authors extend the above-mentioned work by Maheswaranathan et al from sentiment classification to other classification settings, such as multi-label classification. The authors find intriguing differences in the topology of these attractors for different types of classification. This work will be of interest to many researchers in the ML community. 
 
My main gripe with the paper is that in many places, it does not clearly distinguish between existing work and the paper’s contributions. In my mind, the paper is an extension of work by Maheswaranathan et al (2019) and Maheswaranathan & Sussillo (2020). While the authors of the paper reviewed here cite both of these works, overall the paper reviewed here could be more clear about what its contributions are, and how they relate to earlier work. In particular, the abstract describes contributions from Maheswaranathan’s 2019 paper, but doesn’t mention that this is existing work. 



### Pros
 
- The paper presents work on one of the most exciting topics being researched in this community today: how should we think about how neural networks (RNNs, in this paper) process information. The paper presents novel and exciting insights on this topic, and is one of a small number of contributions on the topic not just at this conference, but in general. The paper was a great read. 
- The precise research question addressed (how do low-dimensional attractors in hidden layers of RNNs differ for different classification tasks) is a useful problem to investigate, and the author’s findings may inspire further research and insights into the mechanics of how RNNs make predictions.  


### Cons
 
The authors should be more clear about the paper’s key contributions, which are currently somewhat buried in material that has already been presented in the above-mentioned Maheswaranathan  papers. Being clear about a paper’s contributions and how they relate to existing work makes the paper more helpful for readers. I believe that the authors meant no harm here, and they did cite Maheswaranathan’s existing work appropriately. As a reader, I ask myself “what are the key claims that the papers makes, and how does the paper support these claims?” These questions are answered in the text, but the distinction between existing work and new contributions should be clearer and mentioned earlier. 

For instance, the first half of the abstract, up to and including "*Specifically, across architectures and datasets, RNNs accumulate evidence for each class as they process the text, using a low-dimensional attractor manifold as the underlying mechanism.*", could easily have been the abstract of Maheswaranathan’s 2019 paper. In this way, the abstract advertises material without pointing out that this material is in fact not novel. The last sentence of the abstract (“ *To the degree that integration of evidence towards a decision is a common computational primitive, this work lays the foundation for using dynamical systems techniques to study the inner workings of RNNs.* ”) also looks somewhat as if credit is claimed for work from Meheswaranathan’s 2019 paper, where they write ‘ *Input words with positive valence (“amazing”, “great”, etc.) incremented the hidden state towards a positive sentiment prediction, while words with negative valence (“bad”, “horrible”, etc.) pushed the hidden state in the opposite direction.* ‘. 


### Questions during rebuttal period:

Please address be more clear about the paper’s key contributions, and how they relate to existing work (see “Cons” section and “Reasons for score” section above). 
 

### Some suggestions:

I would recommend to include a statement like “we expand on earlier work \cite{Maheswaranathan 2019, Maheswaranathan 2020} on the  …. for sentiment analysis, and find that … “ right in the abstract. 

To improve the readability of the paper, and better highlight the paper’s contributions, I would moreover suggest to rewrite “We find the network dynamics in these text classification…” paragraph and instead use an “Our main contributions are <itemized list follows>” format here. This way the reader of a paper can quickly glean the paper’s key contributions/claims, but also more easily form an opinion about how well the paper’s arguments support the key claims. 

Small typos
- “withing: should be “within”
- “eigenvalues well” should be “eigenvalues are well”


review_title: Exciting work, but relation to earlier work should be presented more clearly. Weak reject, maybe accept with revisions. 

final_score: rating: 7: Good paper, accept
review_content: The submission furthers existing dynamical systems inspired analyses of recurrent neural networks with focus on text classification.
The motivation is well founded, and stems from the wide-spread applicability and success of RNNs in NLP tasks. The efficacy of this class of models, is nevertheless lacking a mechanistic understanding of the properties that have enabled their success.

The major contribution of this submission is an in-depth analysis of the types of dynamics that permit the **trained** networks to solve text classification tasks. The conclusions notably include:

1. Empirical evidence that the networks use low-dimensional attractors ($ N\ll d$ where n is the number of, potentially not mutually exclusive, classes and d is the hidden layer size). This is result is of twofold importance, in as much as it reveals a fundamental mechanism that models leverage but also because it could be used as motivation to study the compression of large recurrent models to lower-dimensional and hence more interpretable and more energy efficient equivalents.

1.  The geometry of the hidden-state representations is dictated by the task, and loss function. To the best of my knowledge, such a result hasn't been stated as clearly. It bears similarity to results from theoretical neuroscience, where the embedding dimensionality of dynamical systems has been shown to be upper-bound by the 'complexity' of the task (see Gao et al 2017).

Despite the strong, and convincing empirical contributions, the manuscript suffers from some shortcomings. I would like to emphasize at this point, that I do not believe they invalidate the empirical results and hence the essential contribution. That being said, I find the following most concerning:

The analysis is contingent on numerical root finding to locate the fixed points of a *inhomogeneous* map. The problem certainly presents a substantial theoretical challenge, but the specifics of the analysis require further motivation or a modification of the methods. 
1. Specifically, the analysis of the inhomogeneous system is performed by looking at the linearization of the dynamics around the terminal time point. However, the stability of the inhomogeneous system generally depends on the entire sample path of the input process $x_t\, \forall t \in [0,\ldots T]$.
1. Moreover, the authors further simplify the analysis by assuming that there exists a prototypical terminal $x_T$ which is given by the average input. This strikes me as a peculiar choice, given that the maps are potentially highly non-linear and averaging ought not to commute with the application of the map. I personally would prefer a Monte Carlo estimate of the `hidden deflections` using samples from the datasets. 
1. On a technical level, the linearization analysis depends on conditions analogous to those stipulated by the Hartman-Grobman theorem from differential equations. I.E. the stability of the system is only given if no generalized eigenvalues are purely imaginary. While it may seem like a pedantic distinction, it bears relevance upon the conclusion regarding the existence of ` plane attractors`.  If the integration is indeed performed by a linear submanifold that has zero 'flow', then it's stability cannot be analyzed using the Hartman-Grobman type approach. 
1. The previous difficulty is compounded by the fact that all the equilibria and the plane attractor are only inferred with finite numerical precision, which muddles the distinction between a 'true' integrator plane attractor with zero motion within it, and a slow linear submanifold. This problem could potentially be ameliorated by computing the linearizations with arbitrary floating point precision, rather than the 16/32/64 bit precision that are default in neural network libraries.

The following are minor comments:
1. Building on remark (4), it is entirely not clear to me whether the detected submanifold is indeed a center manifold of a single attractor or rather a union of topologically disconnected equilibria. This distinction is arguably inconsequential from a practical point of view, but additional experiments investigating this would benefit the theoretical understanding of the problem.
1. In Fig. 1 the initial points are only marked for the artificial data. Moreover, for the real data the trajectories seem to start from different locations.
1. In Fig. 2 the right-hand side y-axis is not explained clearly, neither in the main text nor in the caption.
1. The model specification for GRU's and UGRNN's leaves out what specific non-linearity was used.
1. The figures in the main text lack indication of what specific model generated the data. I assume this was done to emphasize the universality of the dynamics, but I find it makes it harder to reproduce the result should it be necessary.
1. The network hidden unit sizes were chosen to be different for different architectures -- it would be good to state why this was done, should the result depend on the hidden unit dimensionality.
1. The relation between task/loss and the arrangement of terminal hidden states is very striking, and I think the explanation in its current form is sufficient. At the same time, I'm curious to what extent does the loss necessitate the structure of the manifold. For instance, for dynamics with a non-compact phase space (e.g. linear systems) does the use of cross-entropy somehow manage to contain the hidden states to simplices? If not, to what extent does this depend on the choice of non-linearity for the RNN units?

##### Overall, I think the submission is an interesting contribution, with well thought out empirical experiments. Moreover, it is well written and clear. Pending the authors response, I am rating it as marginally below the threshold.

review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper examines RNNs trained to perform a range of text classification tasks, and demonstrates that they can be understood using a dynamical-systems analysis. Specifically, the paper shows that RNN dynamics explore low-dimensional subspaces determined by the categories to be distinguished. Within those low-dimensional subspaces, RNN dynamics integrate evidence towards each category. The approach is based on recent works by Maheswaranathan et al, but extends them to a large range of tasks. This is a big step towards interpreting RNNs trained on NLP tasks, and I strongly recommend it for ICLR 2021.

Strengths:
- a strong conceptual framework for analysing trained RNNs
- large battery of categorisation tasks on both natural and synthetic datasets
- clear and compelling interpretation of trained RNNs.


Concerns:
The results for ordered classification (Fig 4) are a little puzzling. One would expect that 1-star and 5-star reviews should be further away from each other in PC space than 1-star and 3-star reviews. Related to this, I don't understand why "sentiment" and "intensity" scores should be orthogonal - intuitively they do not seem to be orthogonal for the words used in the synthetic data. One possibility is that the specific 2d organisation in Fig 4 results from the manner in which the readouts were implemented. Presumably, the readouts were implemented as 5 independent categories as in other tasks? If that's the case, the readout structure does not necessarily take into account the ordering in the task. A different readout structure (eg 5 different levels on a single readout) might lead to a different low-d structure.

Other feedback:
- I did not quite understand what is plotted in Fig 4c and similar. The legend says "fixed points", but are these fixed-points in response to different inputs? Which inputs? What do coloured regions and coloured lines respectively represent?
- how important is non-linearity in these tasks? Eg how big is the difference in performance between RNNs and linear accumulation models based on LSA?

Related work:
- a recent paper by Schuessler et al (arXiv:2006.11036) suggests that the low-d dynamics in the sentiment-classification task can be traced to low-rank structure in connectivity, as in neuroscience tasks.
review_title: An important step towards interpreting RNNs trained on NLP tasks

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an analysis on the trained recurrent neural networks (RNN) especially for NLP classification problems. The analysis takes the dynamical systems point of view and investigates the dynamics by looking at the Jacobians around the fixed points. This work founds low dimensionalility and attractor dynamics in the RNNs which might lead to a better undertanding of RNNs.

After reading the manuscript, I am leaning on not accepting it. My comments are below.

Opening the blackbox of RNNs is an important topic in machine learning and neuroscience. This work uses the tools from dynamical systems and focus on the RNNs in NLP classification problems. It empirically shows the connection between the dimensionality of internal dynamics and the tasks. 

The weakness of this work is the lack of theoretical implications. This paper put a lot of work on showing low-D dynamics are embedded in the high-D RNN for low-D tasks that is already shown in many empirical studies. The question is if one can train a low-D RNN to perform the same task, and why or why not. 

On the integration of evidence, the possible mechanisms include, for example, fixed points, line attractors or sequences of stable fixed points (forming a path to the decisions). Further investigation is needed to discriminate between them beyond only analyzing the Jacobian at the final states.

Minor comments
- sec 3.1 do the synthetic data contain inconclusive phrases (all words are 'neutral')?

- Fig 2. The subfigures are wrong-labeled.

- A typo in 'As the position withing...' in page 5

- arxiv:1906.01005 on continuous-time GRU might be relevant.
review_title: Recommendation to reject

Meta-review: this paper adds onto the line of research in investigating the mechanism by which a recurrent network solves a supervised sequence classification problem, following the recent studies such as Maheswaranathan et al., 2019 and Maheswaranathan & Sussillo (2020). in doing so, this paper hypothesizes and confirms that the internal hidden state of a recurrent net, be it GRU or LSTM, evolves over a planar (approximate) attractor as it reads the input, amounting to integrating the evidence as it processes the input sequence, and demonstrates the existences of these attractors and integration dynamics on three types of problems (classification, ordered classification and multi-label classification.) 

there were some potentially misleading or confusing statements throughout the manuscript in the initial version, which were pointed out by the reviewers. the authors however did a commendable job of addressing these concerns by the reviewers to the point that most of them have revised their scores up. 

based on the reviewers' assessments, authors' response and their exchange, i strongly believe this work will enrich our understanding of recurrent nets further.
Decision: Accept (Poster)