The geometry of integration in text classification RNNs
Authors: Kyle Aitken
Vinay Venkatesh Ramasesh
Ankush Garg
Yuan Cao
David Sussillo
Niru Maheswaranathan
Venue: ICLR 2021 Poster (2021)
Keywords: Recurrent neural networks
dynamical systems
interpretability
document classification
reverse engineering

Abstract: Despite the widespread application of recurrent neural networks (RNNs), a unified understanding of how RNNs solve particular tasks remains elusive.  In particular, it is unclear what dynamical patterns arise in trained RNNs, and how those pat-terns depend on the training dataset or task.  This work addresses these questions in the context of text classification, building on earlier work studying the dynamics of binary sentiment-classification networks (Maheswaranathan et al., 2019).  We study text-classification tasks beyond the binary case, exploring the dynamics ofRNNs trained on both natural and synthetic datasets.  These dynamics, which we find to be both interpretable and low-dimensional, share a common mechanism across architectures and datasets:  specifically, these text-classification networks use low-dimensional attractor manifolds to accumulate evidence for each class as they process the text.  The dimensionality and geometry of the attractor manifold are determined by the structure of the training dataset, with the dimensionality reflecting the number of scalar quantities the network remembers in order to classify.In categorical classification, for example, we show that this dimensionality is one less than the number of classes. Correlations in the dataset, such as those induced by ordering, can further reduce the dimensionality of the attractor manifold; we show how to predict this reduction using simple word-count statistics computed on the training dataset. To the degree that integration of evidence towards a decision is a common computational primitive, this work continues to lay the foundation for using dynamical systems techniques to study the inner workings of RNNs.

THE GEOMETRY OF INTEGRATION IN TEXT CLASSIFICATION RNNS
Kyle Aitken," *-t
Vinay V. Ramasesh! Blueshift, Alphabet Mountain View, CA
Ankush Garg,
Department of Physics
Google Research
University of Washington
Mountain View, CA
Seattle, WA
Yuan Cao,
David Sussillo, Google Research
Niru Maheswaranathan Google Research
Google Research Mountain View, CA
Mountain View, CA
Mountain View, CA
ABSTRACT
Despite the widespread application of recurrent neural networks (RNNs), unified understanding of how RNNs solve particular tasks remains elusive. In particular, it is unclear what dynamical patterns arise in trained RNNs, and how those pat- terns depend on the training dataset or task. This work addresses these questions in the context of text classification, building on earlier work studying the dynamics of binary centiment-classifiaatee study lext-classification tasks beyond the binary case, exploring the dynamics of RNNs trained both natural and synthetic datasets. These dynamics, which we find to be both interpretable and low-dimensional, share a common mechanism across architectures and datasets: specifically, these cext-classification networks use cow-dimensional attractor manifolds to accumulate evidence for each class as they process the text. The dimensionality and geometry of the attractor manifold are determined by the structure of the training dataset, with the dimensionality re- flecting the number of scalar quantities the network remembers in order classify. In categorical classification, for example, we show that this dimensionality is one less than the number of classes. Correlations the dataset, such as those induced by ordering, can further reduce the dimensionality of the attractor manifold; we show how to predict this reduction using simple word-count statistics computed on the training dataset. To the degree that integration of evidence towards decision is a common computational primitive, this work continues to lay the foundation for using dynamical systems techniques to study the inner workings of RNNs.
networks (Maheswaranathan et al. 2019). We
INTRODUCTION
Modern recurrent neural networks (RNNs) can achieve strong performance in natural language pro- cessing (NLP) tasks such as sentiment analysis, document classification, language modeling, and machine translation. However, the inner workings of these networks remain largely mysterious.
As RNNs are parameterized dynamical systems tuned to perform specific tasks, natural way to understand them is to leverage tools from dynamical systems analysis. A challenge inherent to this approach is that the state space of modern RNN architectures-tth number of units comprising the hidden state-is often high- nigh-dimensional, with layers routinely comprising hundreds of neurons. This dimensionality renders the application of standard representation techniques, such as phase portraits, difficult. Another difficulty arises from the fact that RNNs are monolithic systems trained end-to- end. Instead of modular components with clearly delineated responsibilities that can be understood and tested independently, neural networks could learn an intertwined blend of different mechanisms needed solve a task, making understanding them that much harder.
Work started while an intern at Google.
I Equal contribution.

---

Recent work has shown that modern RNN architectures trained on binary sentiment classification learn low-dimensional, interpretable dynamical systems MMaheswaranathan et al.) 2019). These integration-like mechanism, moving their hidden states along line of stable fixed points to keep track of accumulated positive and negative tokens. Later, MMah eswaranathan & Sussillo (200)) showed that contextual processing mechanisms in these networks e.g. for handling phrases like not good-
RNNs were found to implement
on top of the line-integration mechanism, employing an additional subspace which the network enters upon encountering a modifier word. The under- standing achieved in those works suggests the potential of the dynamical systems perspective, but remained to be seen whether this perspective could shed light on RNNS in more complicated settings.
In this work, we take steps towards understanding RNN dynamics in more complicated language tasks, illustrating recurrent network dynamics in multiple text-classification tasks with more than two categories. The tasks we dtodyydocument classification, review score prediction (from one to five stars), and emotion binary sentiment case, we find integration of evidence to underlie the operations of these networks; however, in multi-class classification, the geometry and dimensionality of the integration manifold depend on the type of task and the structure of the training data. Understanding and precisely characterizing this dependence is the focus of the present work.
three distinct types of classification tasks. As in the
Our contributions
We study three distinct types of text-classification tasks cates-cateorical,, ordered, and multi- labeled-and find empirically that the resulting hidden state trajectories lie largely in a low- dimensional subspace of the full state space.
Within this low-dimensional subspace, we find manifold of approximately stable fixed pointsl near the network trajectories, and by linearizing the network dynamics, we show that this man- ifold enables the networks integrate evidence for each classification they processes the sequence.
We find (N - -)-dimmnsia simplex attractor for N-class categorical classification, planar attractors for ordered classification, and attractors resembling hypercubes for multi-label classi- fication, explaining these geometries in terms of the dataset statistics.
We show that the dimensionality and geometry of the manifold reflects characteristics of the training dataset, and demonstrate that simple word-count statistics of the dataset can explain the observed geometries.
We develop clean, simple synthetic datasets for each type of classification task. Networks trained on these synthetic datasets exhibit similar dynamics and manifold geometries to networks trained on corresponding natural datasets, furthering an understanding of the underlying mechanism.
Related work Our work builds directly on previous analyses of binary sentiment classification by Mabeswaranathan et al. (2019) and Mahes & Sussillo (2020). Apart from these works, the dynamical properties of continuous time RNNs have been extensively studied (Vyas et al. 2020).
largely for connections to neural computation in biological systems. Such analyses have recently begun yield insights on discrete-time RNNs: for example, [Schuessler et al. (2020) showed that training continuous-timg RNNs on low-dimensional tasks led to low-dimensional updates to the networks' weight matrices; this observation held empirically in binary sentiment LSTMs well. Similarly, by viewing the discrete-time GRU as a discretization of a continuous-time dynamical system, Jordan et al. (2019) demonstrated that the continuous- analogue could express wide variety of dynamical features, including essentially nonlinear features like limit cycles.
Understanding and interpreting learned neural networks is a sopidly-growing field. Specifically in the context of natural language processing, the body of work on interpretability of neural models is reviewed thoroughly in Belinkov & Glass (2018). Common methods of analysis include, for exam- ple, training auxiliary classifiers (e.g., part-of-speech) on RNN trajectories to probe the network's s
As will be discussed in more detail below, by fixed points mean hidden state locations that approx- imately fixed on time-scales of order of the average phrase length for the task hand. Throughout this work we will use the term fixed point manifold to he synonymous with manifolds of slow points.
A 1 -iimpleex is a line segment, 2-simplex triangle, a 3-simplex tetrahedron, etc. A simplex is regular if it has the highest degree of symmetry (e.g. equilateral triangle is regular 2-simplex).
2

---

representations; of challenge sets to capture wider language phenomena than seen in natural corpora; and visualization of hidden unit activations as in Karpathy et al. (2015) and Radford et al. (2017).
SETUP
Models We study three common RNN architectures: LSTMs (Hochreiter & Schmidhuber, 1997,
GRUs (Cho et al., 2014), and UGRNNs (Collins et al. 2016). We denote their n-dimensional hidden state and d-dimensional input at time f as h, and x,, respectively. The function that applies hidden state update for these networks will be denoted by F, so that h, = F, x,). The network's hidden state after the entire example processed, hr, is fed through a linear layer get N output logits for each label: y = Whr + b. We call the rows of W 'readout vectors' and denote the readout corresponding the 1th neuron by rs, for i L..... N. Throughout the main text, we will present results for the GRU architecture. Qualitative features of results were found to be constant across all architectures; additional results for LSTMM and UGRNNs are given in Appendix F
Tasks The classification tasks we study fall into three categories. In the categorical case, samples are classified into con-overlapping classes, for example "sports" or "politics". By contrast, in the ordered case, there is natural ordering among labels: for example, predicting numerical rating (say, out of five stars) accompanying a user's review. Like the categorical labels, ordered labels are exclusive. Some tasks, however, involve labels which may not be exclusive; an example of this multi-labeled case is tagging a document for the presence of one or more emotions. A detailed de- scription of the natural and synthetic datasets used is provided in Appendices C
and D respectively.
Linearization and eigenmodes Part of our analysis relies on lincarization to render the complex RNN dynamics tractable. This linearization is possible because, we will see, the RNN states visited during training and inference lie near approximate fixed points h of the dynamics points that the update equation leave (approximately) unchanged, i.e. for which h" as F(h", x) Near these points, the dynamics of the displacement Ah, == h; h" from the fixed point h is well approximated by the linearization
Ah, = J
| (hrx)) x²) Ab 1 + Jinp (h*xx*) (x: x"),
where we have defined the recurrent and input Jacobians J x)x) aF(b,)); respectively (see Appendix A for details). dzy
aF(hx): and J (h, x) 8h,
In the linear approximation, the spectrum of Juse plays key role in the resulting dynamics. Each eigenmode of f represents displacement whose magnitude either grows or shrinks exponentially in time, with a timescale Ta determined by the magnitude of the corresponding (complex) eigenvalue Aa via the relation Ta := [log Paal= Thus, eigenvalues within the unit circle thus represent stable (decaying) modes, while those outside represent unstable (growing) modes. The Jacobians find in practice almost exclusively have stable modes, most of which decay on very short timescales (a few tokens). Figenmodes near the unit circle have long timescales, and therefore facilitate the network's ' storage of information.
Latent semantic analysis For a given text classification task, one can summarize the data by building a matrix of word or token counts for each class (analogous to a document-term matrix (Manning & Schutze 1999), where the documents are classes). Here, the i, entry corresponds to the number of times the Td word in the vocabulary appears in examples belonging to the jth class. In effect, the column corresponding to given word forms an "evidence vector". i.e. a large entry in an particular row suggests strong evidence for the corresponding class. Latent semantic analysis (LSA) (Deerwester et al. 1990) looks for structure in this matrix via singular value decomposition (SVD); if the evidence vectors lie predominantly in -iwdenmeniional subspace, LSA will pick up this structure. The top singular modes define a "semantic space": the left singular vectors correspond to the projections of each class label into this space, and the right singular vectors correspond to how individual tokens are represented in this space.
Although the fixed point expression depends on the input throughout this text we will only study fixed points with the zero input. That is, we focus on the autonomous dynamical system given by h = F(b,, 0) (see Appendix A for details).
3

---

Synthetic
Var. Exp.
3 Cless Categorical
Predicted
Class AG News
(a)
Class
(b)
(c)
Class
(
Business
PC#2
APC#2
PC#2
Dim.
Component #2
evid
Cless
Class
-6
Class
PC#1
APCR1
#1
Component #1
Naturas
Var. Exp:
3 Cless AG News
(h)
Variance explained
(d)
World
(e)
World
Dim.
PC#2
Sports
PC#2
APC
Sports
Businesa
toankruptoy
Business
-3
55
Component
PC#1
APC
PC#1
Figure 1: Results from training GRUs 3-class categorical data. (a, Final hidden states, hr. of many test samples, colored by their label, initial state ho is shown as a black square, and the three thick solid lines are the three readouts, colored by their respective class. (b, e) The hidden deflections, h. h for t = ....., T. from various words in the vocabulary, with the average deflection of each shown as solid line. (c, f) Approximate fixed points, h" F(h",. = 0), colored by their predicted label (sec Appendix A. for details).. The insct shows the variance explained as a function of number of PC dimensions. As (a, d), the solid lines are readout vectors for each class. (g) The LSA score vectors projected into the top two variance dimensions. (h) Percentage of variance explained versus number of dimensions for LSA.
few example hidden states trajectories shown explicitly. The
Below, will show that RNNs trained classification tasks pick up the same structure in the dataset as LSA; the dimensionality and geometry of the semantic space predicts corresponding features of the RNNs.
Regularization While the main text focuses on the interaction between dataset statistics and result- ing network dimensionality, regularization also plays role in determining the dynamical structures. In particular, strongly regularizing the network can reduce the dimensionality of the resulting mani- folds, while weakly regularizing can increase the dimensionality. Focusing on fg-reuularization, document this effect for the synthetic and natural datasets in Appendicies D.I and F respectively.
3 RESULTS
3.1 CATEGORICAL CLASSIHCATION YIELDS SIMPLEX ATTRACTORS
We begin by analyzing networks trained on categorical classification datasets, with natural examples including news articles (AG News dataset) and encyclopedia entries (DBPedia Ontology dataset). We find dynamics in these networks which are largely low-dimensional and governed by integra- tion. Contrary to our initial expectations, however, the dimensionality of the network's integration manifolds are not simply equal to the number classes in the dataset. For example, rather than exploring three-dimensional cube, RNNs trained on 3-class categorical tasks exhibit largely swo- dimensional state space which resembles an equilateral triangle (Fig. Th. d). As we will see, this example of pattern that generalizes larger numbers of classes.
Synthetic categorical data To study how networks perform N-class categorical classification, we introduce toy language whose vocabulary includes N + 1 words: a single evidence word "evid," for each label i, and neutral word "neutral". Synthetic phrases, generated randomly, are labeled with the class for which they contain the most evidence words (see Appendix Dor more details). This is analogous to a simple mechanism which classifies documents as, e.g., "sports" or "finance" based whether they contain more instances of the word "football" or "dollar".
4

---

(a)
3-class News
(b)
4-class
(c)
5-class Yelp
(d)
3-class Yelp
10
Timescale
10
Overlon
Manifold
10a-
10'
10'
10
0
2
5
Eigenmode
Eigenmode
Eigenmode
Eigenmode
Figure 2: Evidence integration is accomplished via eigenmodes with long timescales, aligned with the fixed-point manifold. Each step along the x-axis represents particular eigenmode of the dynamics, with corresponding cigenvalue Aa; the black (left y-iis)) points show the time constant (in units of tokens) associated with cach mode, given by Ta == |log |A.l=", the blue (right y-axis) points show the fraction of the mode's (right) cigenvector which lies in the fixed-point plane. From these plots, it is apparent that only few modes (highlighted in gray) both: (i) do not decay appreciably on the time scale of the average document length, and (ii) aligned with, and create motion within, the fixed-point manifold. These are integration modes responsible for accumulating evidence. We see (a) two integration modes in three-class categorical tasks, (b) three integration modes in four-class categorical tasks, and two integration modes in both (c) five-class ordered and (d) chree-class ordered tasks. These plots characterize LSTMs, with other architectures shown in the Appendices.
The main features of the categorical networks' integration manifolds are clearly seen in the 3-class synthetic case. First, the dynamics are dow-dimensional: performing PCA on the set of hidden states explored from hundreds of test phrases reveals that more than 97% of its variance is contained in the top two dimensions. Projected onto these dimensions, the set of network trajectories takes the shape of an equilateral triangle (Fig. [b). Diving deeper into the dynamies of the trained network, we examine the deflections, or change in hidden state, Ah, induced by each word. The deflections due to evidence words "evid," " align with the corresponding readout vector r, all times (Fig. [D)). Meanwhile, the deflection caused by the "neutral" word is much smaller, and on average, nearly zero. This suggests that the RNN dynamics approximate that of two-dimensional integrator: as the network processes each example, evidence words move its hidden state within the triangle in manner that approximately constant across the phrase. The location of the hidden state within the triangle encodes the integrated, relative counts of evidence for each of the three classes. Since the readouts are of approximately equal magnitude and align with the triangle's vertices, the phrase ultimately classified by whichever vertex is closest to the final hidden state. This corresponds to the evidence word contained the most in the given phrase.
Natural categorical data Despite the simplicity of the synthetic categorical dataset, its working mechanism generalizes to networks trained on natural datasets. We focus here on the 3-class AG News dataset, with matching results for 4-class AG news and 3- and 4-class DBPedia Ontology in Appendix [ Hidden states of these networks, as in the synthetic case, fill out an approximate equilateral triangle whose vertices
again lie parallel to the readout vectors (Fig. [t). While these results bear a strong resemblance to their synthetic counterparts, the manifolds for natural datasets are, unsurprisingly, less symmetric.
Though the vocabulary in natural corpora is much larger than the synthetic vocabulary, the network still learns the same underlying mechanism: by suitably arranging its input Jacobian and embedding vectors, it aligns an input word's deflection in the direction that changes relative class scores appro- priately (Fig. [p.) Most words behave like the synthetic word "neutral". causing little movement within the plane; certain words, however, (like "football"") cause large shift toward a particular ver- tex (in this case, "Sports"). Again, the perturbation relatively uniform across the plane, indicating that the order of words does not strongly influence the network's 's prediction.
In both synthetic and natural cases, the two-dimensional integration mechanism enabled by man- ifold of approximate fixed points, or slow points, near the network's 's hidden state trajectories, which allow the network to maintain its position in the absence of new evidence for all t = 1,..., T. As the position within the plane encodes the network's 's integrated evidence, this maintenance is essen- tial. In all 3-class categorical networks, we find planar, approximately triangle-shape manifold of fixed points which lie near the network trajectories (Fig. [c. f); vertices of this manifold align with
5

---

(a)
(b)
10
(c)
5
1-Simplex
2-Simplex
3-Simplex
Dimensionality
8
PC #2
0
4
2
-5
2
10
-5
0
5
# of Classes (N)
PC #1
Figure 3: Higher-ddienniinn simplexes in N-class categorical classification. (a) Various simplex fixed- point manifolds seen in synthetic data for N 2, 3, 4, colored by predicted label. (b) In red, the dimensionality of fixed point manifolds from synthetic data a function of number classes, N. Here, dimensionality is the number of dimensions needed to capture more than 95% of the variance in the fixed-point space. Each datapoint is an average over 10 initializations. The black dotted line is the dimensionality of an (N - -immmpleex. (c) The fixed point manifold and readouts for 4-class AG news, colored by predicted label.
the readout vectors. PCA reveals the dimensionality of this manifold to be very similar to that of the hidden states.
Since find the network's hidden state trajectories always lie close the fixed point manifold, we can use the fixed points' stability as an approximate measure of the network's 's ability to store integrated evidence. We check for stability directly by linearizing the dynamics around each fixed point and examining the spectra the recurrent Jacobians. Almost all the Jacobian's 's eigenvalues perturbations which decay on the timescale of
are well within the unit circle, corresponding few tokens. Only two modes, which lie within the fixed-point plane, are capable of preserving information on timescales on the order of the mean document length (Fig. 2). This linear stability analysis confirms our picture of two-dimensional attractor manifold of fixed points; the network dynamics quickly suppress activity in dimensions outside of the fixed-poine plane. Integration, i.e. motion within the fixed-poine plane is enabled by two eigenmodes with long time constants (relative to the average phrase length).
LSA predictions Intuitively, the two-dimensional structure in this three-class classification task reflects the fact that the network tracks relative score between the three classes to make its prediction. To see this wooddimensional structure quantitatively in the dataset statistics, analysis (LSA) to the dataset, finding a low-rank approximation to the evidence vectors of all the words in the vocabulary. This analysis (Fig. (hig shows that two modes necessary to capture the variance, just as we observed in the RNNs. Moreover, the class vectors projected into this space (Fig. [g]) match exactly the structure observed in the RNN readouts. The network appears to pick up on the same structure in the dataset's 's class counts identified by LSA.
apply latent semantic
General -llass categorical networks The triangular structure seen in the 3-class networks above is an example of general pattern: N-class categorical classification tasks result in an (N - 1)- dimensional simplex attractor (Fig. (3)) We verify this with synthetic data consisting of up to 10 classes, analyzing the subspace of R.t explored by the resulting networks. More than 95% of the variance of the hidden states contained in N dimensions, with the subspace taking on the approximate shape of regular - -V)-1-simplex centered about the origin (Fig. [b). The readout vectors, which almost entirely within this (N-1)-dimension subspace, align with the simplex's vertices. Mirroring the 3-class case, dynamics occur shaped like an (N -immim.. This simplex geometry reflects the fact that to classify between N classes, the network must track N scalars: the relative scores for each class.
manifold of fixed points which also
As a natural example of this simplex, the full, 4-class AG News dataset results in networks whose trajectories explore approximate 3-simplex, tetrahedron. The fixed points also form 3- dimensional tetrahedral attractor (Fig. Be). Additional results for 4-class natural datasets, which also yield tetrahedron attractors, shown in Appendix F.

---

Synthetic
Var. Exp.
Class Ordered
Predicted
Class Amazon reviews
(a)-2
5+
(b)
Dim.
APC 82
APC42
okey
PC
Component
3
4
5+
-2
2
2
0
7
APC #1
APC#1
PC #1
Component
Natural Class
(h)
(d)
2
(e)
(f)
Veriance expleined
APC 82
groat
pcor hombiee
PC #2
5+
1+
-2
-2
APC #1
-2
Companent
Figure 4: Results from training GRUs on 5-class ordered data. (a, d) Final hidden states, lT, many test samples, colored by their label, with a few example hidden states trajectories shown explicitly. The initial state ho is shown as black square, and the three thick solid lines are the five readouts, colored by their respective class. (b, The hidden deflections, h h for
I..... T, from the various words in the vocabulary, with the average deflection each shown a solid line. (c, f) Approximate fixed points, h = F(h", x 0). colored by their predicted label (see Appendix [A.]] for details). The inset shows the variance explained as a function of number of PC dimensions. As in (a, d), the solid lines are readout vectors for each class. (g) The LSA score vectors projected into the top two variance dimensions. (h) Percentage of variance explained versus number of dimensions for LSA.
3.2 ORDERED CLASSIFICATION YIELDS PLANE ATTRACTORS
Having seen networks employ simplex attractors to integrate evidence in categorical classification, we turn to ordered datasets, with Yelp and Amazon review star prediction as natural examples. Star prediction is more finely-grained version of binary sentiment prediction that RNNs solve by integrating valence along one-dimensional line attractor (Maheswaranathan et al. 2019). This turns out not to be the case for either 3-class 5-class star prediction.
For network trained on the 5-class Yelp dataset, we plot the two-dimensional projection of RNN trajectories while processing test batch of reviews well as the readout vectors for each class (Fig. 4t). Similar results for 3-class Yelp and Amazon networks are in Appendix [ The top two dimensions capture more than 98% of the variance in the explored states: with categorical clas- sification tasks, the dynamics here are largely low-dimensional. A manifold of fixed points that is also planar exists nearby (Fig. 4 The label predicted by the network is determined almost entirely by the position within the plane. Additionally, eigenmodes of the lincarized dynamics around these fixed points show two slow modes with timescales comparable document length, separated by clear gap from the other, much faster, modes (Fig. 2 d). These two integration modes lie almost entirely within the fixed-point plane, while the others are nearly orthogonal to
These facts suggest that that-in in contrast to binary sentiment analysis - 5-class (and 3-class) ordered networks are tio-dnmmns. tracking two scalars associated with each token rather than simply a single sentiment score. As an initial clue to understanding what these two dimensions represent, we examine the deflections the plane caused by particular words (Fig. 4D. These deflections span two dimensions-in in contrast to one-dimensional integrator, the word 'horrible' has different effect than multiple instances of a weaker word like "poor." " These two dimensions of the deflection vector seem roughly correspond
word's "sentiment" (e.g. good vs. bad) and "intensity" (strong vs. neutral). In this two-dimensional integration, a word like "okay" is treated by the network as evidence of neutral (e.g., 3-star) review.
Inspired by these observations, we build a synthetic ordered dataset with word bank (amazing, good, okay, bad, awful, neutral), in which each word now has separate sentiment and

---

intensity score Labels are assigned to phrases based on both its total sentiment and intensity; e.g., phrases with low intensity and sentiment scores are classified as "3 stars". while phrases with high positive sentiment and high intensity are "5 stars" (see Appendix D.2 for full details). Networks trained on this dataset correspond very well to the networks trained on Amazon and Yelp datasets (Fig. 4p-c). Dynamics are largely two-dimensional, with readout vectors fanning out in the plane from five stars to one. Deflections from individual words correspond roughly to the sentiment and intensity scores, and the underlying fixed-point manifold is two-dimensional.
More generally, the appearance of a plane attractor in both 3-class and 5-class ordered classifica- tion shows that in integration models, relationships (such as order) between classes can change the dimensionality of the network's 's integration manifold. These relationships cause the LSA evidence vectors for each word to lie in a low-dimensional space. As in the previous section, we can see this low-dimensional in the dataset statistics itself using LSA, showing that two singular values explain more than 95% of the variance (Fig. M.. Thus, the planar structure of these networks, with dimen- sions tracking both (roughly) sentiment and intensity, is consequence of correlations present in the dataset itself.
3.3 MULTI-LABELED CLASSIFICATION YIELDS INDEPENDENT ATTRACTORS
So far, we have studied classification datasets where there is only a single label per example. This only requires networks to keep track of the relative evidence for each class, as the overall evidence does not affect the classification. Put another way, the softmax activation used in the final layer will normalize out the total evidence accumulated for given example. This results in networks that, for an N N-way classification task, need to integrate (or remember) at most N - 11 quantities as we have seen above. However, this is not true in multi-label classification. Here, individual class labels are assigned independently to each example (the task involves N independent binary decisions). Networks trained on this task do need to keep track of the overall evidence level.
To study how this changes the geometry of integration, we trained RNNs on a multi-label classifi- cation dataset, GoEmotions (Demszky al. (2020). Here, the labels are emotions and particular text may be labeled with multplle emotions. We trained networks on two reduced variants of the full dataset, only keeping two or three labels. The results for three labels are detailed Appendix E.S For the two-class version, we only kept the labels "admiration" and "approval". and additionally resampled the dataset so each of the 2² possible label combinations were equally likely. We found that RNNs learned -wo-dimensional integration manifold where the readout vectors span a two-dimensional subspace (Fig. 5)),, rather than a one-dimensional line as in binary classification. Across the fixed point manifold, there were consistently two slow eigenvalues (Fig. 5e), correspond- ing to the two integration modes. Similar to the previous datasets, increasing lg regularization would (eventually) compress the dimensionality, again measured using the participation ratio (Fig. (5. No- tably, GoEmotions is a per class was important to observe match between the synthetic and the natural dynamics.
dataset; we found that balancing the number of examples
The synthetic version of this task classifies a phrase as if it were composed of two independent sentiment analyses (detailed in Appendix D.3) This is meant to represent the presence/absence of given emotion in a given phrase, but ignores the possibility of correlations between certain emotions. After training network on this data, we find low-dimensional hidden-state and fixed-point space that both take on the shape of square (Fig. 5], c). The deflections of words affecting independent labels act along orthogonal directions (Fig. (5))
These results suggest that integration manifolds are also found in RNNs trained on labeie classification datasets. Moreover, the geometry of the corresponding fixed points and readouts is dif- ferent from the exclusive case; instead of an (N-1) -imme-mmiinsi simplex we get an N-dimensional hypercube. Again, this makes intuitive sense given that the networks must keep track of N indepen- dent quantities in order to solve these tasks.
'Interestingly, synthetic model that only tracks sentiment fails to match the dynamics of natural ordered data for N > 2. We take this as further evidence that natural ordered datasets classify based on two-dimensional integration. This simple model still produces surprisingly rich dynamics that we detail in Appendix D.2
8

---

Synthetic
Var. Exp.
2 Class Multillbbled
(a)
[N.
(b)
(c)
7
0
(N,
bad,
Dim.
PC82
APC#2
PG#2
-2
PC#1
APC#1
PC#1
Natural
Class GoEmotions
(d)
0.5
thanks
(f)
N)
perfect
Dm.
PC H2
PC
JRC
0.0
0
bed
wrong
-3
-3
0
0.5
0.0
0.5
-3
3
PC #1
PC #1
Figure 5: GRUs trained multi-labeled synthetic (a-c) and natural (d-f) GoEmotions dataset with only two labels. (a, d) Final hidden states, hr. a synthetic network, colored by their label. with a few example trajectories. The two thick solid lines are readout vectors. The network classifies examples using a 2D plane attractor. (b, e) The hidden deflections, h, - h for f = 1..... T, from the various words in the vocabulary, with the average deflection of each shown colored by their predicted label: these form square (ef. the triangle in Fig. [T]. The inset shows the variance explained (two dimensions are sufficient to capture nearly of the variance).
solid line. (c, f) Approximate fixed points, h" zs F(h",
0),
DISCUSSION
In this work we have studied text classification RNNs using dynamical systems analysis. We found integration via attractor manifolds to underlie these tasks, and showed how the dimension and ge- ometry of the manifolds were determined by statistics of the training dataset. As specific examples, we see (N-1)-dimensional simplexes in N-class categorical classification where the network needs to track relative class scores; 2-dimensional attractors in ordered classification, reflecting the need to track sentiment and intensity; and N -ddimensional hypercubes in -lllass multi-labed classification.
We hope this line of analysis- using dynamical systems tools to understand RNNs MONs--bulle builds to- ward deeper understanding of how neural language models perform more involved tasks in NLP, including language modeling translation. These tasks cannot be solved by pure integration mechanism, but it is plausible that integration serves as a useful computational primitive in RNNs more generally, similar how line attractor dynamics serve computational primitive top of which contextual processing occurs (Maheswaranathan & Sussillo 2020).
REFERENCES
Yonatan Belinkov and James R. Glass. Analysis methods in neural language processing: A survey. CoRR,
2018.
Francesco Camastra and Alessandro Vinciarelli. Estimating the intrinsic dimension of data with fractal-based method. IEEE Transactions on pattern analysis and machine intelligence, 24(10): 1404 1407, 2002.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406.1074 2014.
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability recurrent neural networks, 2016.

---

Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-407, 1990.
Dorotya Demszky, Dana Movshovitz-Attias Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine- fine-grained emotions. pp. 4040-4054, 01 2020. doi:
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-17888 1997.
lan D. Jordan, Piotr Aleksander Sokol, and Il Memming Park. Gated recurrent units viewed through the lens of continuous time dynamical systems, 2019.
Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks CoRR, abs/1 506.02078, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 12 2014.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in neural information processing systems, pp. 777-784, 2005.
Niru Maheswaranathan and David Sussillo. How recurrent networks implement contextual process- ing in sentiment analysis. arXiv preprint urXiv:2004.08013, 2020.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engincering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems 32, pp. 15696-15705, Curran Associates, Inc., 2019.
Christopher Manning and Hinrich Schutze. Foundations of statistical natural language processing. MIT press, 1999.
Itamar Procaccia and Peter Grassherger. Measuring the strangeness of strange attractors. Physica. D, (1-2):18892000 1983.
Alec Radford, Rafal Józefowicz, and llya Sutskever. Learning to generate reviews and discovering sentiment. CORR,
2017.
Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in mns, 2020.
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626 649, 2013.
Saurabh Vyas, Matthew D Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Annual Review of Neuroscience, 43:249-275, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classi- fication. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 649-657, Curran Associates, Inc., 2015.
A METHODS
A.1 FIXED- POIDTS AND LINEARIZATION
We study several RNN architectures and will generically denote their n-dimensional hidden state and d-dimensional imput at time t as h, and x,, respectively. The function that applies hidden state update for these networks will be denoted by F, so that h, = F(h,-1,x,) The N output logits a readout the final hidden state, y = Why + b. We will denote the readout corresponding to the ith neuron by r,, for i 1, 1,.... N.
10

---

We define fixed point of the hidden-state space to satisfy the expression h' F(h", ,)) This defi- nitions of fixed- bhedtpoints is inherently x-dependent. In this text, we focus on fixed points of the network in for zero input, ie. when that only satisfy this fixed point relation approximately, i.e. h" ~ F(h", x). The slowness of the approximate fixed points can be characterized by defining loss function q := 1 ||h - F(h, x) x) Throughout this text we use the term fixed point to include these approximate fixed points as well.
0. We will also be interested in finding points in hidden state space
Expanding around given hidden state and input, xx),, the first-order approximation of F is
h = F (h", x²) + J (hr,x*) (h - ha) + JumP/
(hs,,* xx)) (x) -x)x,
(2)
== where we have defined the recurrent and input Jacobians as J. (h, x) (F(hx)) and J (h, x) (F(h,x),
0h,
= dzy respectively. If we expand about a fixed point h ~ F(h", x = 0), the effect of an imput X;
on the hidden state hr can be approximated by (J Writing the eigendecomposition,
Jree = RAL, with L = R we have
(J P' = RA T "JJ = Σ ra TTAT J
where A is the diagonal matrix containing the (complex) eigenvalues, A 2 A2 sorted in order of decreasing magnitude; Fa are the columns of R; and ey are the rows of L.
2 An that
The magnitude of the eigenvalues of J correspond to a time constant Ta = logiAa The time constants, Ta approximately determine how long and what information the system remembers from a given input.
We find fixed points by minimizing a function which computes the magnitude of the displacement F(h,x = 0) - h resulting from applying the update rule at point h. That is, numerically solve
I h F(h, =
minh 2
We seed the minimization procedure with hidden states visited by the network while processing test examples. To better sample the region, we also add some isotropic Gaussian noise to the initial points.
A.2 DIMENSIONALITY MEASURES
Here we provide details regarding the measures used to determine both the dimensionality of our hidden-state and fixed-point manifolds. When discuss the dimensionality of a set of points, we will mean their intrinsic dimensionality. Roughly, this is the dimensionality of a manifold that summarizes the discrete data points, accounting for the fact said manifold could be embedded in higher-dimensional space in a non-linear fashion. For example, if the discrete points lie along a one- dimensional line that is non oon-einerrly embedded in some two-dimensional space, then the measure of intrinsic dimensionality should be close to 1.
Let X (X....., Xm) } be the of M points X, for = L.... M for which we wish to measure the dimensionality. In this text, X is either set of hidden-states or set of fixed points and each X, E R is point in hidden-state space. To determine an accurate measure of dimensionality, we use the following measures:
Variance explained threshold. Let 14 2 2.. Pn be the eigenvalues from PCA (i.e. the variances) on X. A simple measure of dimensionality is to threshold the number of PCA dimensions needed to reach a certain percentage of variance explained. For low number of classes, this threshold can simply be set at fixed values 90% or 95%. However, we would expect such threshold to breakdown as the number of classes increase, so we also use an N N-dependent threshold of N/(N + 1)%
Global participation ratio. Again using PCA on X as above, the participation ratio (PR) is defined to be a scalar function of the eigenvalues:
E 14)
PR
(5)
Intuitively, this is scalar measure of the number of "important" PCA dimensions.
11

---

Global PR
10
10
Local
Dimensionality Measure
Dimenstonality Measure
N/NN 1)% Var. Exp.
90% Var. Exp.
95% Var. Exp.
mi k=10
4-
mi for k=50
my for k=100
2-
2-
Corr. Dim.
(N-1)-Simplex Dim.
0-
10
4
10
Number of Classes (N)
Number of Classes (N)
Figure 6: Various dimensionality measures as function number of classes, N, for the categorical synthetic data. Specifically, Left: the hidden state space and Right: the fixed point space dimensionality, This is for an f2 of 5 x 10 and each datapoint is an average over 10 initializations. The dotted black line shows the predicted dimensionality of regular (NV- -im-implex.
Local participation ratio. Since PCA is linear mapping. both of the above measures will fail the manifold is highly non-linear. We thus implement a local PCA as follows: we choose a random point and compute its & nearest neighbors, then perform PCA on this subset of k + points. We then calculate the participation ratio on the eigenvalues of the local PCA using equation We repeat the process over several random points, and then average the results. This measure is dependent upon the hyperparameter k.
MLE measure of intrinsic dimension (Levina & Bickel 2005). This is a seeaestt-nigghte based measure of dimension. For point X,, let T3 (X,) be the Euclidean distance to its kth nearest neighbor. Define the scalar quantities
M Σ
log Tx(X,) T2(
my
(X1)
my (X,) =
(6)
M
h:
This measure is also dependent upon the number of nearest neighbors k. Correlation Dimension (Procaccia Grassherger Define the scalar quantity
1983 Camastra Vinciarelli 2002
N
N
Cw(r) = N(N -
Σ
Σ
- <
J=1+1
Then, plotting log (N(r) as function of log r, the correlation dimension is found by estimating the slope of the linear part of the plot.
We plot these dimensionality measures used on synthetic categorical data for class sizes N to 10 in Figure 6 Despite their simplicity, we find the 95% variance explained threshold and the global participation ratio to be the best match to what theoretically predicted, hence we use these measures in the main text and in what follows.
B MODELS AND TRAINING
The three architectures we study are specified below, with W and b respectively representing train- able weight matrices and bias parameters, and h, denoting the hidden state at timestep All other vectors (c, g, r, i, represent intermediate quantities; a(-) represents pointwise sigmoid nonlin- earity; and f(-) is the tanh nonlinearity.
Update-Gate RNN (UGRNN)
(Wec h W x bo).
h, = h + c.
(8)
[W W + by)
12

---

Gated Recurrent Unit (GRU)
c = f (W h1-1) )+ + W + +b))
h = h + (1-1 (1-g).- g..
g (w W x, bd)
(9)
r (W" W + b¹)
Long-Short-Term-Memme (LSTM)
h f(c,) o (W" w xx + +b").
C₁
C f. c--1 10.0 (Weh h, + W x; + b9)
h,
(10)
he
i = a (W W + -b),
f (W W + +b).
With the natural datasets, we form the imput vectors x; by using a (learned) 228-dimessional embed- ding layer. These UGRNNs and GRUs have hidden-state dimension = 256, while in the LSTMs, both the hidden-state h; and the memory č are 256-dimensional, yielding total hidden-state di- mension = 512. For the synthetic datasets, due to their small vocabulary size, we simply pass one-hot encoded inputs in the RNN architectures, i.e we use no embedding layer. For UGRNNs and GRU, we use hidden-state dimension of = 128, while for LSTMs we again use the same dimension for both h, and č₁ resulting in total hidden-state dimension of n = 256.
The model's predictions (logits for each class) are computed by passing the final hidden state hy through a linear layer. In the synthetic experiments, do not add bias term to this linear readout layer, chosen for simplicity and ease of interpretation.
We train the networks using the ADAM optimizer (Kingma & Ba 2014) with exponentially- decaying learning rate schedule. We train using cross-entropy loss with added & regularization, penalizing the squared &2 norm of the network parameters. Natural experiments use batch size of 64 with initial learning rate rate decays by 0.9984 every step. Synthetic experiments use a batch size of 128, initial learning rate 0.1, and gradient clip of 10; the learning rate decays by 0.9997 every step.
0.01, clipping gradients to a maximum value of 30; the learning
C NATURAL DATASET DETAILS
We use the following text classification datasets in this study:
The Yelp reviews dataset (Zhang et al. 2015) consists of Yelp reviews, labeled by the corresponding star rating (1 through 5). Each of the five classes features 130,000 training examples and 10,000 test examples. The mean length of a review is 143 words.
The Amazon reviews dataset (Zhang al., 2015) consists of reviews of products bought on Amazon. com over an 18-year period. As with the Yelp dataset, these reviews are labeled by the corresponding star rating (1 through 5). Each of the five classes features 600,000 training examples and 130,000 test examples. The mean length of review is 86 words.
The DBPedia ontology dataset (Zhang et al.) 2015) consists of titles and abstracts of Wikipedia articles in one of 14 non-overlapping categories, from DBPedia 2014. Cat- egories include: company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film, and written work. Each class contains 40,000 training examples and 5,000 testing examples. We use the ab- stract only for classification; mean abstract length is words.
The AG's news corpus (Zhang et al. 2015) contains titles and descriptions of news articles from the web, in the categories: world, sports, business, sci/tech. Each category features 30.000 training examples and 1,900 testing examples. We use only the descriptions for classification; the mean length of a description is 35 words.
The GoEmotions dataset (Demszky et al. 2020) contains text from 58,000 Reddit com- ments collected between 2005 and 2019. These comments labeled with the following 27
13

---

emotions: admiration, approval, annoyance, gratitude, disapproval, amusement, curiosity, love, optimism, disappointment, joy, realization, anger, sadness, confusion, caring, excite- ment, surprise, disgust, desire, fear, remorse, embarrassment, nervousness, pride, relief, grief. The mean length of a comment is 16 words.
Two main characteristics distinguish these datasets: the class labels, and (ii) whether labels are exclusive. The reviews datasets, Amazon and Yelp, are naturally ordered, while the labels in the other datasets unordered. All of the datasets besides GoEmotions feature exclusive labels; only in GoEmotions can two or more labels (e.g.. the emotions anger and disappointment) characterize the same example. In addition to the standard five-class versions of the ordered datasets, we form three-class subsets by collecting reviews with 1, 3, and stars (excluding reviews with and stars).
(i) whether there is notion of order among
We build vocabulary for each dataset by converting all characters to lowercase and extracting the 32,768 most common words in the training corpus. Tokenization done by TensorFlow TF.Text TorepieceeTokenizer.
D SYNTHETIC DATASET DETAILS
In this appendix we provide many additional details and results from our synthetic datasets. Al- though these datasets represent significantly simplified settings compared to their realistic coun- terparts, often the results from training RNNs the synthetic and natural datasets are strikingly similar.
D.1 CATEGORICAL. DATASET
For the categorical synthetic dataset used in Section 3.1 generate phrases of L words, draw- ing from a word bank consisting of N + words, W {evidl,... evidy, neutral). Each word W E W has N-dimensional vector of integers associated with it, w with w.W E for all i = I....,N. The word "evid," has score defined by u wivii: = and w = for + j. Meanwhile, the word "neutral" has wheuurall = for all j. Additionally, each phrase has corresponding score that also takes the form of N-dimensional vector of integers, 8 {81.-............... )..). A phrase's score is equal to the sum of scores of the words contained in said phrase, Ew W Ephrase W The phrase is then assigned a label corresponding to the class with the maximum score, argmax(s)
{w ..... w
In the main text analyze synthetic datasets where phrases drawn from a uniform distribution over all possible scores, s. To do so, we enumerate all possible scores a phrase of length L. can produce as well all possible word combinations that can generate given score. It is also possible to build phrases by drawing each word from a uniform distribution over all words in W. In practice, find all results on synthetic datasets have minor quantitative differences when comparing these two methods, but qualitatively the results are the same
As highlighted in the main text, after training on this synthetic data we find the explored hidden- state space to resemble regular (N - ))-simplex. This holds for large range & values relative to the natural datasets. Figure 7 we plot the (global) participation ratio, defined in equation 5 as function of the number of classes, N.
In addition to the hidden states forming a simplex, we observe the N readout vectors are approxi- mately equal magnitude and are aligned along the N vertices of said (N 1))-simplex. In Figure we plot several measures on the readout vectors that support this claim that we now discuss. We find the readout vectors to have very close to the same magnitude (Fig. g left panel). The angle (in degrees) between a pair of vectors that point from the center of a (N 1)-simplex to two of its
'For phrases with multiple occurrences of the maximum score, the phrase is laheled by the class with the smallest numerical index.
'We have also analyzed the same synthetically generated data for variable phrase lengths. The qualitative results focused in this text did change in this setting.
14

---

12
1.0
10
Participation Rabio
Test Accuracy
0.9
1e-5
5e-5
6
4
0.8
5e-4
1e-3
2
5e-3
1e-2
0
0.7
2
6
10
10
Number of Classes (N)
Number of Classes (N)
Figure 7: How the dimensionality and accuracy of categorical synthetic data changes as function of l regularization. Each datapoint is average over 10 initializations. Left: Hidden state space dimensionality (global participation ratio) and Right: test accuracy synthetic data for several different values of &g. The dotted black line shows predicted regular (N 1)- simplex dimensionality.
function of number of classes, N, for the unordered
2.6
180
1.000
RO Pairwise Angle
RO Magnitude
RO Subspace %
90
0.990
4
6
8
10
4 6
8
10
4
6
8
10
[# of Classes)
N (# of Classes)
(# of Classes)
Figure 8: Readout (RO) measures as a function the number of classes, for synthetic categorical data. All data collected over 10 random initializations at an & of 5 x 10 Left: Magnitude, with the line heing the median and filled in area being 10th and 90th percentile. Center: Pairwise angle, again with median being the solid line with filled in 10th and 90th percentiles (the variance in angles is very low, this is almost indistinguishable from the line itself). Black dotted line theoretical simplex angle. Right: Subspace percentage, A, as defined in equation] 12
vertices is
180
0 dbearry
X arccos
N N
(II)
For example, for regular 2-simplex, i.e. an equilateral triangle, this predicts an angle between readout vectors of 120 degrees. The distribution of pairwise angles between readout vectors is plotted the center panel of Figure { Lastly, the readouts lie entirely within the (N - 1)-sipplex,
all N of them should live in the same R--1 subspace. To measure this, define r to be projection of r; into the subspace formed by the other N readout vectors, i.e. the span of the set (r, |j 1,...,N; Nij # i}. We then define the subspace percentage, A, as follows,
(12)
r
If all the readouts lie within the same N- subspace, then A = 1. The right panel of Figure shows that in practice for the synthetic data with la regularization parameter of 10-4
Why Regular (N - 1)-Simplex? Here propose intuitive scenario that leads the network's hidden states to form regular (N simsimplex. To classify a given phrase correctly, the network must learn keep track of the value of the N-dimensional score vector One way this can be
15

---

T3
AT3
(1, 1, 1) Direction
T1
Project out the (1,1,1) direction
12
2
T1
Figure 9: Left: Full space of possible scores as subspace R Right: Two-dimensional space resulting
from projecting out the (1, 1, 1) direction of the Le subspace, forming a regular 2-simplex.
done is follows: Let the network's hidden state live in some RN dimensional subspace. Within this subspace, let the N readout vectors be orthogonal and have equal magnitude. Furthermore, define a Cartesian coordinate system to have basis vectors aligned with the readouts, with 2; the coordinate along the direction of readout r,. Then, the coordinates within this subspace encodes the components of the N-dimensional score vector the evidence word 'evid, moves you along you the coordinate direction i some fixed amount and so X Note the subspace of R-N explored by hidden states has finite extent, since the phrases are of finite length. This subspace can be further subdivided into regions corresponding to different class labels: if > 2, for all of then r, > h ry and the phrase is classified as Class i. The left panel of Figure 9 shows an example of the 3-dimensional subspace for N = 3.
The important step that gets us from a subspace of N to the regular (N - 11)ssiilsexxxx is the presence of the softmax layer used when calculating loss. Since this function normalizes the scores, it is only the relative size of the components of s that matters. Removing the dependence on the absolute score values corresponds ones vector, (1, 1,.., 1). This projection results in an (N 1)-sipplex with the readouts aligned with the vertices. A demonstration of this procedure for N 3 is shown in Figure
projecting onto the R N- subspace orthogonal the N-dimensional
D.2 ORDERED DATASET
As alluded to in the main text, try two renditions of ordered synthetic data. The details of both are given below. The first relies on a ground truth of only a sentiment score, while the second classifies based on both sentiment and neutrality. Although the first is simpler and still bears many resemblances to natural data (i.e. Yelp and Amazon), we find the second to be a better match overall.
Sentiment Only Synthetic Data The first synthetic data for ordered datasets is very similar into that of the categorical sets with minor difference in the word bank and how phrases assigned labels. For N -lasss ordered datasets, the word bank always consists of only three words W {good, bad, neutral). We now take the word and phrase scores to be I-dimensional and w good = +1, whad = 11 and pouttall = O. We then subdivide the range possible scores into N equal regions, and phrase is labeled by the region which its score fall into. Given the above definitions, the range of scores is [-L, L). and so for the N = 3, with labels (Positive, Negative, Neutral), we define some threshold SN T/3. Then label y is assigned as follows:
Positive
$ 2 SN
Neutral
|s] <
(13)
Negative
< -SN
Mcanwhile, for N 5. could draw the region divisions at the score values (-3L/5,- L/5, L/5, 3L/5). Similar to the categorical data above, in the main text we draw phrases from uniform distribution all possible scores.
16

---

Var. Exp.
1
PC #2
Positive
APC #2
neutral
PC #2
0
0
Dim.
Negative
good
bad
-8
1
-8
-8
-1
0
1
-8
0
8
PC #1
APC #1
PC #1
Figure 10: Synthetic ordered data for N = 2. Left: Final hidden states for 600 test samples, colored by their label, with few example hidden states trajectories shown explicitly. The initial state ho is shown as black square, and the three thick solid lines the three readouts, colored by their respective class. Center: The hidden deflections from three words the vocabulary, with the average deflection each shown as solid line. Right: Fixed points, colored by their predicted label. The inset shows the variance explained.
The N = 2 corresponds sentiment analysis, and its hidden state space, word deflections, and fixed point manifold are plotted in Figure 10 The dynamics of this system qualitatively match the natural dataset analyzed in Maheswaranathan al. 2019). Briefly, the sentiment score is encoded in the hidden state's 's position along a one-dimensional line, aligned with the readouts which point in opposite directions. The word "good' ('bad') moves you along this line along the 'Positive' (" ("Negative") readout, increasing the corresponding logit value.
The simplest ordered dataset beyond binary sentiment analysis is that of N 3, and plot showing the final hidden states, deflections, and fixed-point manifold is shown in top row of Figure In the bottom row, we show the same plots for
5. In both cases, away from he onto curve embedded in 2d plane, with the curve bent around the origin of said plane. The readout vectors are evenly fanned out the 2d plane, which subdivides the curve into N regions corresponding to each of the N classes. The curve subdivisions reflect the ordering of the score subdivisions, for N = 3 we see 'Neutral' lying in between 'Positive' and 'Negative' and for N the stars ordered from 1 to 5.
hidden-state trajectories move
In contrast to categorical data, the word deflection Ah, are highly varied and have a strong depen- dence state's location in hidden-state space. On average, the words 'good' and "bad' move the hidden state further left/right along the curve. Although Ah, for the word 'neutral' is on average smaller, it tends move the hidden state along the 'Neutral' or '3 Star' readout. These dynamics are how the network encodes the relative count of 'good' and 'bad' words in a phrase that ultimately determines the phrase's ' classification. We show the fixed points in the far right panel of Figure I1 For N = 3, the fixed point manifold mostly resembles that of a one-dimensional bent line attractor, with a small region that is wwo-dimensional along the 'Neutral' readout. For N 5, the fixed point manifold is much more planar. Thus, the N = 3 case exhibits very similar dynamics to that of the line attractor studied in Maheswaranathan et al. (2019), the attractor now simply subdivided into three regions due to the readout vector alignments.
Sentiment and Neutrality Synthetic Data Instead of classifying a phrase based off single senti- ment score, our second ordered synthetic model classifies phrase based off of two scores that track the sentiment and intensity of given phrase. We draw from enhanced word bank consisting of W = ( (awesome, good, okay, bad, awful, neutral}. We take the two-dimensional word score to have components corresponding (sentiment, intensity) where positive (negative) sentiment scores cor- respond to positive (negative) sentiment and positive (negative) intensity scores correspond to high
'The N 2 ordered dataset is equivalent to the N = 2 categorical dataset. Intuitively, 'good' and 'bad' can he though of evidence vectors for the classes 'Positive" and 'Negative', respectively. Just like the categorical classification, whichever of these evidence words appears the most given phrase will be the phrase's 's label.
17

---

Five Var. Exp.
Neutral
neutral
PC #2
APC #2
PC #2
1
5
0
0
Dim.
good
bad
Positive
Negative
-7
0
7
-7
0
7
PC #1
APC #
PC #1
7
1*
Var. Exp
1
5*
0
good
bad
PC #2
PC #2
0
0
Dim.
APC
4*
2*
neutral
3*
-7
-7
-7
0
7
-1
-7
0
7
PC #1
APC #1
PC #1
Figure 11: Synthetic ordered data for 3 (top row) N (bottom row). Left: Final hidden states for 600 test samples, colored by their label, with few example hidden states trajectories shown explicitly. The initial state ho is shown black square, and the three thick solid lines are the three readouts, colored by their respective class. Center: The hidden deflections from the three words in the vocabulary, with the average deflection of each shown as solid line. Right: Fixed points, colored by their predicted label. The inset shows the variance explained.
(low) emotion. The word score values we use are (2, 1).
w awesome
w good
11,-
okay (0,-2)
(14a)
bad (-11--1/2)
awful (-,
neutral (0,
(14b)
As with the other synthetic models, we sum all word scores across phrase to arrive at phrase's sentiment and intensity score, (s, i). We then assign the phrase a label y based off the following
criterion:
Three Star
i << and | |s| otherwise:
Five Star
i 2 and > 0,
Four Star
i << 0 and > 0,
(15)
Two Star
i and 0,
One Star
12 and 0.
Thus we see that scores with negative (low) intensity where the intensity magnitude is greater than the sentiment magnitude are classified as 'Three Star', i.e. it is neutral phrase. Otherwise, phrases with low intensity that are the less extreme reviews are classified as either 'Two Star' or Four Star' based their sentiment. Finally, phrases with high intensity labeled either "One Star' Star', again based on their sentiment.
D.3 MULTI-LABELED DATASET
Here provide details of the synthetic multi-labeled dataset, that corresponds to natural dataset GoEmotions in Section 3. of the main text. Let us introduce this by taking the = 2 as an explicit have up two labels. We draw from word bank consisting of W = (good, bad,, good,, bad2, neutral), where
example, where each phrase
w neutral
(0, 0),
8ood = =(1,0),
w bad= = (-1,0),
(16a)
800d
(0,
w whaa (0, -1).
(16b)
18

---

Var. Exp.
1
0
7
2
7
bad,
neutral
1
Dim.
PC #2
APC #2
goodg
badg
PC #2
0
good
badg
good
-2
-7
0
7
-2
0
2
-7
0
7
PC #1
APC #1
PC
Figure 12:
Synthetic multi-labeled example with N 3. Left: few example hidden states trajectories.
Inset: variance explained over 600 hidden state examples. Center: Deflections from given word input. Fixed points and variance explained over fixed points.
Right:
We then classify each phrase with two labels, individually based on the score vector components $1 and $2. Namely,
Positive
0, Negative, 81 0,
Positives
$2 0,
y1
1/2
Negative,
$2 < 0.
(17)
Thus there are four possible combinations labels. For this synthetic datasets, generate phrases by uniformly drawing words one-by-one from W. Generalization of the above construction to an arbitrary number of possible labels N straightforward: one simply adds additional N dimmennsion score vectors woodd, and had, for each possible label i = 1....,N N and then uses the N components of the score assign the N labels, ys, individually.
The results after training network on the N = 2 dataset are shown in the main text in Figure and results for N = 3 are shown in Figure 12 Again, we see the explored hidden-state space to be low-dimensional, but notably it now resembles three-dimensional cube. This is certainly large departure from the 7-simplex. Instead, what categorical dataset, from which we expect a (seven-dimensional) regular here is the "outer product" of three N 2 ordered datasets. That is, we expect a single N = 2 ordered dataset (i.e. binary sentiment analysis) to have a hidden-state space that resembles line attractor. As might expect, tasking the network with analyzing three such sentiments at once leads to three line attractors that are orthogonal to one another, forming a cube. This is supported in the center panel of Figure where the various sentiment evidences are orthogonal from one another.
E
ADDITIONAL RESULTS ON NATURAL DATASETS
E. AG NEWS
This subsection contains two figures: Figures 13 and 14 complement Figure [ in the main text; the main text figure showed the manifolds learned by an LSTM on both and 4-class AG News datasets; the figures in this appendix show corresponding manifolds learned by a GRU and UGRNN.
E.2 DBPEDIA 3-CLASS AND 4-CLASS CATEGORICAL. PREDICTION
Like AG News, DBPedia Ontology is a categorical classification dataset. We show results for net-
works trained 3- and 4-class subsets of this dataset in Figures 15 16 and 17
E.3
YELP 5-CLASS STAR PREDICTION
Figures 19 18 and 20 show the fixed- -oint manifolds associated with a GRU, LSTM and UGRNN, respectively, trained on 5-class and 3-class Yelp dataset. These reviews naturally five star;
19

---

AG News 4-class
Trajectories
Deflections
Fixed points
Var Exp
Modes
3
10a
PC #2
APC #2
Olm.
iimescare
0
PC #2
0
10
-6
-3
-5
-3
0
3
2
4 5
PC #1
APC #1
Eigenmode
3-class
3
5
APC #2
Dm.
10a
PC #2
#2
0
0
Timescale
dyvllo
10
-6
-5 -5
10a
-6
0
-3
3
5
PC #1
APC #1
PC #1
Eigenmode
Figure 13: Trajectorics, individual word deflections, fixed-point manifolds, and eigenmodes learned by LSTM the 3- and 4-class AG News task. Solid lines in the trajectories and fixed-points plots show readout vectors, Note similarity between this manifold that learned by the GRU, described main text Figure]] The words we use to generate the deflections are sox, bankruptcy, military, and computer.
Trajectories
Deflections
Fixed
Modes
points
AG News 4-class
Timescale
PC #2
APC #2
Dim.
101
PC
-3
-6
2
PC #1
APC #1
PC #1
Eigenmode
3-class
Var: Exp.
3
102
PC #2
APC #2
Dim.
Timescale
Overlag
PC
101
-6
-3
10
-6
-3
0
1
#1
4PC
PC #1
Eigenmode
Figure 14: Trajectories, deflections, fixed-point manifolds, and eigenmodes learned UGRNN 3- and 4-class News. Solid lines the trajectories and fixed-points plots show readout vectors. Notice that in 4-class case, unlike GRUs or LSTMs, the UGRNN seems to always learn square manifold. This is likely due to correlations in the input data (see Figure 29 for details).
20

---

DBPedia 4-class
Trajectories
Fixed points
Modes
5
5
10a
Timescale
Overlap
PC #2
102
0
PC
10
-5
-5
-5
5
-5
5
2
3
4
5
6
PC #1
PC
#1
Eigenmode
3-class
5
Var.
0
1
PC #2
PC #2
Dm
Timescale
102
Overlap
0
10
-5
-5
-5
5
-5
PC #1
1
2
3
PC #1
Eigenmode
Figure 15: LSTM on four-class subset of the DBPedia ontology dataset
DBPedia 4-class
Trajectories
Fixed
Modes
points
Var. Exp.
0
Timescale
Overlap
PC #2
PC #2
Dim.
10
-5
-5
-5
0
-5
0
5
PC #1
PC
# #1
Eigenmode
3-class
Ver bp
5
Timescale
Overlap
PC #2
Dim.
0
PC
0
102
-5
5
6
-5
0
5
PC #1
Eigenmode
PC
Figure 16: GRU on four-class subset of the DBPedia ontology datasct
create a 3-class subset by removing examples labeled with 2 and 4 stars. These figures complement Figure in in the main text.
21

---

DBPedia 4-class
Trajectories
Fixed
Modes
points
Ver. Exp.
5
10a
Timescale
PC #2
PC #2
Overlap
Dim.
0
0
10
-5
-5
5
-5
5
-5
0
Eigenmode
PC # 1
PC
# 1
3-class
Voc Elo.
0
Timescale
102
PC #2
1
5
Sverlar
PC #2
10
100
-5
-5
1
2
3
4
5
5
-5
0
Eigenmode
PC #
#1
Figure 17: UGRNN four-class subset of the DBPedia ontology dataset
Yelp 5-class
Trajectories
Deflections
Fixed points
Modes
2
2
PC #2
APC #2
Timescale
0
PC #2
Dm
10
-2
-1
-2
-2
-2
2
Elgenmode
3-class
PC
APC
PC
#1
3
10a
#2
5
Timescale
dejRno
àPC
0
10a
10a
-8
-3
-8
-3
Eigenmode
APC #1
PC
Figure 18: LSTM on five-class and three-class Yelp
E.4 AMAZON 5-CLASS AND 3-CLASS STAR PREDICTION
As another example of an ordered dataset, Figures 21 22 and show results for networks trained on a 3-class and 5-class subsets of Amazon reviews. These reviews are naturally five star; we create a 3-class subset by removing examples labeled with 2 and stars.
E.5 3- 3-CLASS GoEMOTIONS
In addition to the class variant presented in the main text, we also trained 3 class version of the GoEmootions dataset. We filtered the dataset to just include the following three classes: "admirattoo"..
22

---

Yelp 5-class
Trajectories
Deflections
Fixed points
Modes
2
2
Timescale
10a
PC #2
#2
PC #2
0
0
10a
-2
-2
0
2
0
-2
Figenmode
3-class
PC #1
APC #1
PC #1
7
Timescale
10?
PC #2
APC #2
PC #2
101
-7
-3
100
-3
0
-7
7
Elgenmode
PC #1
APC #1
PC
#1
Figure 19: GRU on five-class and three-class Yelp
Yelp 5-class
Trajectories
Deflections
Fixed points
Modes
Timescale
APC #2
10a
0
#2
PC
-4
-1
-4
0
-1
Eigenmode
3-class
PC #1
APC #1
PC
PC #2
APC #2
Timescale
10a
PC #2
0-
101
10a
-7
-3
-7
2
Eigenmode
-7
-3
-7
PC
APC #1
PC
#1
Figure 20: UGRNN on five-class and three-class Yelp
"approval", and "annoyance" (these were selected they were the classes with the largest number of examples). These results are presented in Figure 24 For this network, despite having three classes, we find that the fixed points are largely two dimensional (Fig. 24p). The timescales of the eigenvalues of the Jacobian computed at these fixed points have two slow modes (Fig. (24p), which overlap with the two modes (Fig. 24b); thus we have a roughly 2D plane attractor. However, the participation ratio (Fig. 24 indicates that the dimensionality of this attractor is slightly higher than the 2D case shown in Fig. We suspect that these differences are due to the strong degree of class imbalance present in the GoEmotions dataset. There very few examples with multiple labels, for any particular combination labels. In synthetic multi- labli-l data (which is class balanced), we much clearer 3D structure when training a 3 class network (Fig. 12
23

---

Amazon 5-class
Trajectories
Deflections
Fixed points
Modes
2
Dg.
10a
Timescale
PC #2
APC #2
depad
0
0
10
PC
-2
2
3
-2
0
2
Eigenmode
0
3-class
PC #1
APC #1
#1
3
ΔPC #2
Timescale
107
PC 42
PC
101
-5
-3
-5
10a
-5
-3
3
PC #1
APC #1
PC
#1
Eigenmode
Figure 21: LSTM on five-class and three-clane Amazon
Amazon 5-class 2
Trajectories
Deflections
Fixed points
Modes
Ver Exp
2
PC #2
APC #2
Timessale
0
Dm
107
10'
-2
Egenmode
-2
0
2
APC #1
-2
0
3-class
PC #1
PC #1
107
Timescale
PC #2
APC #2
PC #2
Dm.
10'
-7
-3
-7
Elgenmode
APC #1
Figure 22: GRU on five-class and three-class Amazon
F THE EFFECT OF l REGULARIZATION: COLLAPSE, CONTEXT, AND CORRELATIONS
Regularizing the parameters the network during training can have strong effect on the dimension of the resulting dynamics. We describe this effect first for the datasets with ordered labels, Yelp and Amazon reviews. We penalize squared f--m of the parameters, adding the term \||0||3 the cross-entropy prediction loss; À is the lg penalty and 0 are the network parameters.
Collapse:
Figure 25 shows the performance of the LSTM, GRU, and UGRNN as a function of the l2 penalty. As the l2 penalty is varied, the test accuracy usually decreases gradually; however, at a few values, the accuracy takes a large hit. The first two of these jumps correspond to decrease in the dimension of the integration manifold from 2D and ID and then ID to OD. The resulting ID manifold is shown, for the example GRU on the Amazon dataset Figures 26 The effects of collapse the other architectures for the ordered datasets are identical.
24

---

Amazon 5-class
Trajectories
Deflections
Fixed points
Modes
APC #2
#2
Timescale
10?
0
10*
-2
-1
-2
-2
-2
0
3-class
PC #1
APC #1
PC
Figenmode
5
PC #2
Vac
Timescale
10a
deparO
0
PC #2
APC
0
-5
100
-3
-5
2
4
-5
-3
-5
5
Elgenmode
PC
#1
Figure 23: UGRNN on five-class and three-class Amazon
(a)
(b)
(c)
(d)
Var. Exp.
3
Dimensionality
Timescale
Overlap
PC#2
2
102
Dim.
109
-2
20
20
0
-2
0
2
Eigenmode
Eigenmode
10-3
1
PC #1
Le penalty
Figure 24: Analyzing networks trained on 3-class version GoEmotions dataset. (a) Approximate fixed points (gray circles) and readout vectors. Inset shows the variance explained by the different principal components. The dynamics are (largely) 2D. (b) Across these fixed points, we see two slow time constants. (e) The eigenvectors aligned with the slow modes are aligned with the top PCA subspace. (d) Across networks trained with different amounts regularization, between and 3, which reduces to less than two as the regularization penalty increased.
see the dimensionality (measured by participation ratio)
When the regularization is sufficient to collapse the manifold to 1D line, the dynamics are quite similar the ID line attractors studied in Maheswaranathan al. (2019). A single accumulated valence score is tracked by the network as it moves along the line; this tracking occurs via a single eigenmode with time constant comparable to the average document length, aligned with the fixed- point manifold. The difference between the binary- and 5-class line-attractor networks are largely the way the final states are classified; in the 5-class case, the line attractor is divided into sections based largely on the angle the line makes with the readout vector of each class.
The collapse a OD manifold with higher & penalty most strikingly seen in the recurrent Jacobian spectra at the fixed points (Figure 27p. Here there are no modes which remember activity on the timescale the mean document length. Given this lack integration, it unclear how these networks are achieving accuracies above random chance.
While the focus of this study has been on how networks perform integration, it is clear from the plots in Figure 25 that the best-performing models doing more than just Bag-of-worde style integration. When the order of words the sentence is shuffled, these models take a hit in accuracy. Interestingly, when the & coefficient is increased from the smallest values use, the contextual effects are the first to be lost: the model's accuracy on shuffled and ordered examples becomes the same.
Context:
Understanding precisely how contextual processing is carried out by the network is an interesting di- rection for future work. It is important to show, however, that the basic two-dimensional integration
25

---

LSTM on Yelp Reviews
LSTM on Amazon Reviews
0.6
Accuracy
0.5
0.4
0.3
GRU on Yelp Reviews
GRU on Amazon Reviews
0.6
Accuracy
0.4
0.2
0.0
UGRNN Yelp Reviews
UGRNN on Amazon Reviews
0.6
Accuracy
0.4
0.2
Ordered
Shuffled
0.0
10-3
10-2
10
10
10-3
10 10-2
10
100
Regularization (L2 penalty)
Regularization (L2 penalty)
Performances of the I.TTM, GRU, and UGRNN on ordered five-class datasets (both Yelp and Amazon reviews) function of la regularization. Networks shown in the main text and appendix section are highlighted with line.
Figure 25:
Fixed-point manifold
Eigenmodes
Overlap
10-
Var
Time constant [tokens]
10
PC Coordinate #2
0.5
Dim
Mode
1
0.0
-0.5
10a
-1.
10a
-1.5
100
PC Coordinate #
PC Coordinate #1
Figure 26: Geometry GRU trained the five-class Amazon dataset, which due high & penalty (here = 0.72) has collapsed to a manifold, rather than 2D manifolds seen higher-performing models with lower la penalty. This 1D collapse is seen in all models, in both the Yelp and Amazon datasets. Note the similarity of this fixed- fixed-point manifold to the line-attractor dynamics identified (in binary sentiment classification) in Maheswaranathan et al. 2019).
26

---

LSTM spectrum
GRU spectrum
UGRNN spectrum
Imagleigenvaluer
0.5
-0.5 0
0.5 1 -1
-0.5
0.5
1 -1
-0.5
0.5 Real[eigenvalue]
Real[eigenvalue]
Real[eigenvalue]
Figure 27: Jacobian spectra at an arbitrary fixed point of networks, trained on Yelp, with sufficiently high l-- regularization penalty À to force the classification manifold to collapse to zero-dimensional (above, ALSTM 0.268, AGRU 0.1, AUGRNN 0.268). These spectra display no modes which integrate information the timescale document length, i.c. no modes with close to unit magnitude.
LSTM
GRU
UGRNN
PC
Var. EXP
Var. EXI
Var. EXP.
PC
PC
PC
Figure 28: Fixed-point manifolds and predictions for RNNs trained on five-class Yelp with lower la penalties than used in the main text. These networks outperform bag-of-word oo-dsrds models and suffer performance hit upon shuffling the test sentences. indicating ability process context (through mechanism which we have studied this paper). The key point is: though these manifolds clearly extend into more than two dimensions (as indicated by the PCA explained-variane variance insets, the classes are nearly separable just by using the top two principal components. Thus, the mechanisms we identify in the main text still seem to underlie the operations of these networks, with contextual processing occurring top of this integration.
27

---

LSTM
GRU
UGRNN
World
Business Sciftech
Sports
PC
Var. Exp.
Var. Exp.
Var
PC 1
PC 1
PC 1
Figure 29: Fixed-point manifolds of LSTM, GRU. and UGRNN trained on 4-class AG News, with sufficiently high E regularization penalty À collapse the manifold from tetrahedron to square (above, ALSTM 0.3, AGRU 0.1, AUGENN 0.3). Across 10 random seeds per architecture, we observe the vertex the fixed-point square corresponding to the Sports category always opposite to either the Business or Sci/Tech categories. This fact reflects correlations among the classes, shown in Figure (30 Insets: Variance explained by dimension after PCA projection, showing the 2D nature of the manifold.
mechanism we have presented in the main text still underlies the dynamics of the networks which capable of handling context. To show this, we plot in Figure 28e the fixed point manifold, colored by the predicted class. As with the models which are not order-sensitive, the classification of the fixed points depends largely their top two coordinates (after PCA projection). This is the even though the PCA explained variance clearly shows extension of the dynamics into higher di- mensions. It is thus likely that, similarly how Maheswaaanahhan & Sussillo (2020) found that the contextual-procese
mechanism was a perturbation on top of the integration dynamics for binary sentiment classification, the same true for more finely-grained sentiment classification.
Correlations: As might be expected, increasing &2 regularization also causes collapse in models trained on categorical classification tasks. For example, shown in Figure 29 the tetrahedral manifold seen in 4-class AG News networks becomes square at higher values of &g collapsing from three dimensions two. That is, instead of class labels corresponding to vertices of tetrahedron, when the &2 regularization is increased, these labels correspond to the vertices of a square.
Interestingly, in the the collapse to a square, we find that- regardeessss of architecture and across 10 random seeds per architecture- - the ordering of vertices around the square appears to reflect correlations between classes. Up to symmetries, the only possible ordering of vertices around the square are: (i) World + Sci/Tech + Business + Sports, (ii) World - Sci/Tech - Sports Business, and (iii) World + Sports + Sci/Tech - Business. In practice we observe that most of the time (26 out of 30 trials), order (iii) appears; otherwise, order (i) appears. We never observe order (ii).
To show how this ordering arises from correlations between class labels, train bag-of- bag-of-words model on the full 4-class dataset. Taking the most common 5000 words in the vocabulary, we plot, in Figure 330 the changes in each logit due to these words. As the figure shows, for most pairs of classes there is a weak negative correlation between the evidence for the pair. However, between the classes "Sports" and "Business", there is strong negative correlation (R=-0.81); between "Sports" and "Sci/Tech", there is a slightly weaker negative correlation (R=0061).. Stated another way, words which constitute positive evidence for "Sports" are likely to constitute negative evidence for "Busi- ness" and/or "Sci/Tech". This matches with the geometries we observe in practice, where "Sports" and "Business" readouts are 'repelled' most often, and otherwise "Sports" and "Sci/Tech" are re- pelled.
G A CLOSER LOOK AT THE SLOW ZONE
In the main text, we were interested in approximate fixed points, or points which were slow on the timescale given roughly by the average length of documents Tay in the training dataset. For practical purposes, points which are slow on this time scale can be treated as effectively fixed, since evolving the system for T will result in little motion. Whether any of the points in the slow zone exact
28

---

Sports
-1
R=-0.1
Business
0=10.16
R=-0.81
Sci/Tech
-1
R=-0.02
R=-0.61
R=0.09
11
1
-1
World
Sports
Business
Figure 30: Correlations between evidence, provided by individual tokens, for the four classes present in the AG News dataset. The most significant correlations, in the middle column, the negative correlations between category pairs (Sports, Business) and (Sports, Sci/Tech). These correlations influence geom- etry of networks trained with sufficiently high & penalty to collapse the manifold to a square (see Figure 29
29

---

1
5
5
0
PC
PC
PC
-1
-2
-5
-5
-5
-3
4
-5
5
-5
0
-5
5
PC 0
PC 1
PC
Figure 31: A view of the speed variation 3-class News (see Figure iin main text). Colors are given by the log the speed, see colorbar at right. Points within the outer contour line correspond to speeds less than the inverse of the average document length; points within the inner contour are ten times slower still. Vertical slices in the middle and rightmost plot show the planar character of the manifold. The three horizontal dotted lines in the leftmost pancls correspond slices plotted in Figure 32
the triangular manifold corresponding GRU trained
fixed points of the dynamics, or whether all of them are simply slow points, not something our numerical experiments are capable of resolving. However, we do find that there is structure to the slow zone that all the points are not uniformly slow.
We define the speed of point, S(h), as the distance traveled from that point after single application of the dynamical map F(h,0) 0) (Sussillo Barak 2013
S(h) h P(h, 0)1l2
(18)
Note that, in the main text, are characterizing the RNN with input (the autonomous system). In Figure 31] we plot three orthogonal slices this function for GRU network trained on the 3-class AG News dataset. All the points in the roughly triangular region identified in the main text have speeds less than a tenth of the inverse document length, and are thus slow. However, there a few neighborhoods, roughly located near the vertices of the triangle that are even slower. This structure can be seen further in Figure 32, in which we plot the speed as a function of the PC 0 coordinate for three values of the PC coordinate. Similar structures can be seen in fixed point manifolds for other networks and datasets. Fully exploring these structures and understanding how they arise from the equations governing the RNN is an interesting direction for future research.
30

---

PC
10
PC: 0
Deflection magnitude
10-1
10
PC -2
10
10
-6
-2
PC
Figure 32: Speed versus coordinate, plotted various values of corresponding to the three yellow lines in Figure [3 The dashed lines correspond to speed equal to the inverse document length, and one-tenth of that value.
31


Reviews:
[{"review_id": "xEPrpkjr0qZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission furthers existing dynamical systems inspired analyses of recurrent neural networks with focus on text classification.\nThe motivation is well founded, and stems from the wide-spread applicability and success of RNNs in NLP tasks. The efficacy of this class of models, is nevertheless lacking a mechanistic understanding of the properties that have enabled their success.\n\nThe major contribution of this submission is an in-depth analysis of the types of dynamics that permit the **trained** networks to solve text classification tasks. The conclusions notably include:\n\n1. Empirical evidence that the networks use low-dimensional attractors ($ N\\ll d$ where n is the number of, potentially not mutually exclusive, classes and d is the hidden layer size). This is result is of twofold importance, in as much as it reveals a fundamental mechanism that models leverage but also because it could be used as motivation to study the compression of large recurrent models to lower-dimensional and hence more interpretable and more energy efficient equivalents.\n\n1.  The geometry of the hidden-state representations is dictated by the task, and loss function. To the best of my knowledge, such a result hasn't been stated as clearly. It bears similarity to results from theoretical neuroscience, where the embedding dimensionality of dynamical systems has been shown to be upper-bound by the 'complexity' of the task (see Gao et al 2017).\n\nDespite the strong, and convincing empirical contributions, the manuscript suffers from some shortcomings. I would like to emphasize at this point, that I do not believe they invalidate the empirical results and hence the essential contribution. That being said, I find the following most concerning:\n\nThe analysis is contingent on numerical root finding to locate the fixed points of a *inhomogeneous* map. The problem certainly presents a substantial theoretical challenge, but the specifics of the analysis require further motivation or a modification of the methods. \n1. Specifically, the analysis of the inhomogeneous system is performed by looking at the linearization of the dynamics around the terminal time point. However, the stability of the inhomogeneous system generally depends on the entire sample path of the input process $x_t\\, \\forall t \\in [0,\\ldots T]$.\n1. Moreover, the authors further simplify the analysis by assuming that there exists a prototypical terminal $x_T$ which is given by the average input. This strikes me as a peculiar choice, given that the maps are potentially highly non-linear and averaging ought not to commute with the application of the map. I personally would prefer a Monte Carlo estimate of the `hidden deflections` using samples from the datasets. \n1. On a technical level, the linearization analysis depends on conditions analogous to those stipulated by the Hartman-Grobman theorem from differential equations. I.E. the stability of the system is only given if no generalized eigenvalues are purely imaginary. While it may seem like a pedantic distinction, it bears relevance upon the conclusion regarding the existence of ` plane attractors`.  If the integration is indeed performed by a linear submanifold that has zero 'flow', then it's stability cannot be analyzed using the Hartman-Grobman type approach. \n1. The previous difficulty is compounded by the fact that all the equilibria and the plane attractor are only inferred with finite numerical precision, which muddles the distinction between a 'true' integrator plane attractor with zero motion within it, and a slow linear submanifold. This problem could potentially be ameliorated by computing the linearizations with arbitrary floating point precision, rather than the 16/32/64 bit precision that are default in neural network libraries.\n\nThe following are minor comments:\n1. Building on remark (4), it is entirely not clear to me whether the detected submanifold is indeed a center manifold of a single attractor or rather a union of topologically disconnected equilibria. This distinction is arguably inconsequential from a practical point of view, but additional experiments investigating this would benefit the theoretical understanding of the problem.\n1. In Fig. 1 the initial points are only marked for the artificial data. Moreover, for the real data the trajectories seem to start from different locations.\n1. In Fig. 2 the right-hand side y-axis is not explained clearly, neither in the main text nor in the caption.\n1. The model specification for GRU's and UGRNN's leaves out what specific non-linearity was used.\n1. The figures in the main text lack indication of what specific model generated the data. I assume this was done to emphasize the universality of the dynamics, but I find it makes it harder to reproduce the result should it be necessary.\n1. The network hidden unit sizes were chosen to be different for different architectures -- it would be good to state why this was done, should the result depend on the hidden unit dimensionality.\n1. The relation between task/loss and the arrangement of terminal hidden states is very striking, and I think the explanation in its current form is sufficient. At the same time, I'm curious to what extent does the loss necessitate the structure of the manifold. For instance, for dynamics with a non-compact phase space (e.g. linear systems) does the use of cross-entropy somehow manage to contain the hidden states to simplices? If not, to what extent does this depend on the choice of non-linearity for the RNN units?\n\n##### Overall, I think the submission is an interesting contribution, with well thought out empirical experiments. Moreover, it is well written and clear. Pending the authors response, I am rating it as marginally below the threshold.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_ZIgoKSeZ3E", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\n \nWeak reject. This paper might be a weak accept if revised to present its key contributions more clearly. \n\n\n### Reasons for score: \n \nThis paper is an interesting extension of the work from “Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics” (Maheswaranathan et al, 2019, https://arxiv.org/abs/1906.10720). The work presents new insights into the dimensionality and workings of low-dimensional attractors in which RNNs accumulate “evidence” in their hidden layer. The authors extend the above-mentioned work by Maheswaranathan et al from sentiment classification to other classification settings, such as multi-label classification. The authors find intriguing differences in the topology of these attractors for different types of classification. This work will be of interest to many researchers in the ML community. \n \nMy main gripe with the paper is that in many places, it does not clearly distinguish between existing work and the paper’s contributions. In my mind, the paper is an extension of work by Maheswaranathan et al (2019) and Maheswaranathan & Sussillo (2020). While the authors of the paper reviewed here cite both of these works, overall the paper reviewed here could be more clear about what its contributions are, and how they relate to earlier work. In particular, the abstract describes contributions from Maheswaranathan’s 2019 paper, but doesn’t mention that this is existing work. \n\n\n\n### Pros\n \n- The paper presents work on one of the most exciting topics being researched in this community today: how should we think about how neural networks (RNNs, in this paper) process information. The paper presents novel and exciting insights on this topic, and is one of a small number of contributions on the topic not just at this conference, but in general. The paper was a great read. \n- The precise research question addressed (how do low-dimensional attractors in hidden layers of RNNs differ for different classification tasks) is a useful problem to investigate, and the author’s findings may inspire further research and insights into the mechanics of how RNNs make predictions.  \n\n\n### Cons\n \nThe authors should be more clear about the paper’s key contributions, which are currently somewhat buried in material that has already been presented in the above-mentioned Maheswaranathan  papers. Being clear about a paper’s contributions and how they relate to existing work makes the paper more helpful for readers. I believe that the authors meant no harm here, and they did cite Maheswaranathan’s existing work appropriately. As a reader, I ask myself “what are the key claims that the papers makes, and how does the paper support these claims?” These questions are answered in the text, but the distinction between existing work and new contributions should be clearer and mentioned earlier. \n\nFor instance, the first half of the abstract, up to and including \"*Specifically, across architectures and datasets, RNNs accumulate evidence for each class as they process the text, using a low-dimensional attractor manifold as the underlying mechanism.*\", could easily have been the abstract of Maheswaranathan’s 2019 paper. In this way, the abstract advertises material without pointing out that this material is in fact not novel. The last sentence of the abstract (“ *To the degree that integration of evidence towards a decision is a common computational primitive, this work lays the foundation for using dynamical systems techniques to study the inner workings of RNNs.* ”) also looks somewhat as if credit is claimed for work from Meheswaranathan’s 2019 paper, where they write ‘ *Input words with positive valence (“amazing”, “great”, etc.) incremented the hidden state towards a positive sentiment prediction, while words with negative valence (“bad”, “horrible”, etc.) pushed the hidden state in the opposite direction.* ‘. \n\n\n### Questions during rebuttal period:\n\nPlease address be more clear about the paper’s key contributions, and how they relate to existing work (see “Cons” section and “Reasons for score” section above). \n \n\n### Some suggestions:\n\nI would recommend to include a statement like “we expand on earlier work \\cite{Maheswaranathan 2019, Maheswaranathan 2020} on the  …. for sentiment analysis, and find that … “ right in the abstract. \n\nTo improve the readability of the paper, and better highlight the paper’s contributions, I would moreover suggest to rewrite “We find the network dynamics in these text classification…” paragraph and instead use an “Our main contributions are <itemized list follows>” format here. This way the reader of a paper can quickly glean the paper’s key contributions/claims, but also more easily form an opinion about how well the paper’s arguments support the key claims. \n\nSmall typos\n- “withing: should be “within”\n- “eigenvalues well” should be “eigenvalues are well”\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DdqU_i6pFnv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an analysis on the trained recurrent neural networks (RNN) especially for NLP classification problems. The analysis takes the dynamical systems point of view and investigates the dynamics by looking at the Jacobians around the fixed points. This work founds low dimensionalility and attractor dynamics in the RNNs which might lead to a better undertanding of RNNs.\n\nAfter reading the manuscript, I am leaning on not accepting it. My comments are below.\n\nOpening the blackbox of RNNs is an important topic in machine learning and neuroscience. This work uses the tools from dynamical systems and focus on the RNNs in NLP classification problems. It empirically shows the connection between the dimensionality of internal dynamics and the tasks. \n\nThe weakness of this work is the lack of theoretical implications. This paper put a lot of work on showing low-D dynamics are embedded in the high-D RNN for low-D tasks that is already shown in many empirical studies. The question is if one can train a low-D RNN to perform the same task, and why or why not. \n\nOn the integration of evidence, the possible mechanisms include, for example, fixed points, line attractors or sequences of stable fixed points (forming a path to the decisions). Further investigation is needed to discriminate between them beyond only analyzing the Jacobian at the final states.\n\nMinor comments\n- sec 3.1 do the synthetic data contain inconclusive phrases (all words are 'neutral')?\n\n- Fig 2. The subfigures are wrong-labeled.\n\n- A typo in 'As the position withing...' in page 5\n\n- arxiv:1906.01005 on continuous-time GRU might be relevant.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ApGsn-ElttW", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper Summary\n\nThis paper sheds light on how trained RNNs solve text classification problems by analyzing them from a dynamical systems perspective. It extends recent work where a similar analysis was applied to the simpler setting of binary sentiment classification. When projecting the RNN hidden states to principal dimensions that explain most of the variance, the authors find (N-1) dimensional simplex attractors for N-dimensional classification, 2D attractors for ordered classification, and N-dimensional hypercubes for multi-label classification. \n\n### Strengths\n\nIn general, the paper is clear and enjoyable to read.\n\nThe use of synthetic data strengthens the analysis by first showing what behavior one might expect from RNNs on \"clean\" data, and then comparing to what is observed on natural data.\n\nThe use of different categories of tasks not only makes the experimentation thorough, it also clarifies understanding by contrasting behavior across these categories.\n\nBased on the presented dynamical systems analysis, I think practitioners can acquire valuable intuitions about how RNNs process information (at least for certain problems) that is \"correct\", i.e. it has been verified by sufficient experimentation.\n\nFinally, I agree with the authors that the paper motivates further work into understanding the behavior of RNNs from this perspective.\n\n### Weaknesses\n\nI found the multi-label classification experiments on natural data to be limited, since the authors only used reduced 2- and 3-class versions of the GoEmotions dataset and the behavior was not as \"clean\" as that for synthetic data for the 3-class setting (App. E.5). This part of the analysis leaves me somewhat uncertain about general behavior to expect for multi-label classification in practice.\n\n### Review Summary\n\nI found this paper to be interesting, well-written and useful. I recommend an Accept, but the authors can improve the paper by strengthening the multi-label classification experiments.\n\n### Post Author Response\n\nI thank the authors for their hard work and I'm happy to see that the analysis has improved.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1TVbydG3W8d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines RNNs trained to perform a range of text classification tasks, and demonstrates that they can be understood using a dynamical-systems analysis. Specifically, the paper shows that RNN dynamics explore low-dimensional subspaces determined by the categories to be distinguished. Within those low-dimensional subspaces, RNN dynamics integrate evidence towards each category. The approach is based on recent works by Maheswaranathan et al, but extends them to a large range of tasks. This is a big step towards interpreting RNNs trained on NLP tasks, and I strongly recommend it for ICLR 2021.\n\nStrengths:\n- a strong conceptual framework for analysing trained RNNs\n- large battery of categorisation tasks on both natural and synthetic datasets\n- clear and compelling interpretation of trained RNNs.\n\n\nConcerns:\nThe results for ordered classification (Fig 4) are a little puzzling. One would expect that 1-star and 5-star reviews should be further away from each other in PC space than 1-star and 3-star reviews. Related to this, I don't understand why \"sentiment\" and \"intensity\" scores should be orthogonal - intuitively they do not seem to be orthogonal for the words used in the synthetic data. One possibility is that the specific 2d organisation in Fig 4 results from the manner in which the readouts were implemented. Presumably, the readouts were implemented as 5 independent categories as in other tasks? If that's the case, the readout structure does not necessarily take into account the ordering in the task. A different readout structure (eg 5 different levels on a single readout) might lead to a different low-d structure.\n\nOther feedback:\n- I did not quite understand what is plotted in Fig 4c and similar. The legend says \"fixed points\", but are these fixed-points in response to different inputs? Which inputs? What do coloured regions and coloured lines respectively represent?\n- how important is non-linearity in these tasks? Eg how big is the difference in performance between RNNs and linear accumulation models based on LSA?\n\nRelated work:\n- a recent paper by Schuessler et al (arXiv:2006.11036) suggests that the low-d dynamics in the sentiment-classification task can be traced to low-rank structure in connectivity, as in neuroscience tasks.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: this paper adds onto the line of research in investigating the mechanism by which a recurrent network solves a supervised sequence classification problem, following the recent studies such as Maheswaranathan et al., 2019 and Maheswaranathan & Sussillo (2020). in doing so, this paper hypothesizes and confirms that the internal hidden state of a recurrent net, be it GRU or LSTM, evolves over a planar (approximate) attractor as it reads the input, amounting to integrating the evidence as it processes the input sequence, and demonstrates the existences of these attractors and integration dynamics on three types of problems (classification, ordered classification and multi-label classification.) 

there were some potentially misleading or confusing statements throughout the manuscript in the initial version, which were pointed out by the reviewers. the authors however did a commendable job of addressing these concerns by the reviewers to the point that most of them have revised their scores up. 

based on the reviewers' assessments, authors' response and their exchange, i strongly believe this work will enrich our understanding of recurrent nets further.
Decision: Accept (Poster)