The geometry of integration in text classification RNNs

1 INTRODUCTION . Modern recurrent neural networks ( RNNs ) can achieve strong performance in natural language processing ( NLP ) tasks such as sentiment analysis , document classification , language modeling , and machine translation . However , the inner workings of these networks remain largely mysterious . As RNNs are parameterized dynamical systems tuned to perform specific tasks , a natural way to understand them is to leverage tools from dynamical systems analysis . A challenge inherent to this approach is that the state space of modern RNN architectures—the number of units comprising the hidden state—is often high-dimensional , with layers routinely comprising hundreds of neurons . This dimensionality renders the application of standard representation techniques , such as phase portraits , difficult . Another difficulty arises from the fact that RNNs are monolithic systems trained end-toend . Instead of modular components with clearly delineated responsibilities that can be understood and tested independently , neural networks could learn an intertwined blend of different mechanisms needed to solve a task , making understanding them that much harder . ∗Work started while an intern at Google . †Equal contribution . Recent work has shown that modern RNN architectures trained on binary sentiment classification learn low-dimensional , interpretable dynamical systems ( Maheswaranathan et al. , 2019 ) . These RNNs were found to implement an integration-like mechanism , moving their hidden states along a line of stable fixed points to keep track of accumulated positive and negative tokens . Later , Maheswaranathan & Sussillo ( 2020 ) showed that contextual processing mechanisms in these networks— e.g . for handling phrases like not good—build on top of the line-integration mechanism , employing an additional subspace which the network enters upon encountering a modifier word . The understanding achieved in those works suggests the potential of the dynamical systems perspective , but it remained to be seen whether this perspective could shed light on RNNs in more complicated settings . In this work , we take steps towards understanding RNN dynamics in more complicated language tasks , illustrating recurrent network dynamics in multiple text-classification tasks with more than two categories . The tasks we study—document classification , review score prediction ( from one to five stars ) , and emotion tagging—exemplify three distinct types of classification tasks . As in the binary sentiment case , we find integration of evidence to underlie the operations of these networks ; however , in multi-class classification , the geometry and dimensionality of the integration manifold depend on the type of task and the structure of the training data . Understanding and precisely characterizing this dependence is the focus of the present work . Our contributions . • We study three distinct types of text-classification tasks—categorical , ordered , and multilabeled—and find empirically that the resulting hidden state trajectories lie largely in a lowdimensional subspace of the full state space . • Within this low-dimensional subspace , we find a manifold of approximately stable fixed points1 near the network trajectories , and by linearizing the network dynamics , we show that this manifold enables the networks to integrate evidence for each classification as they processes the sequence . • We find ( N − 1 ) -dimensional simplex attractors2 for N -class categorical classification , planar attractors for ordered classification , and attractors resembling hypercubes for multi-label classification , explaining these geometries in terms of the dataset statistics . • We show that the dimensionality and geometry of the manifold reflects characteristics of the training dataset , and demonstrate that simple word-count statistics of the dataset can explain the observed geometries . • We develop clean , simple synthetic datasets for each type of classification task . Networks trained on these synthetic datasets exhibit similar dynamics and manifold geometries to networks trained on corresponding natural datasets , furthering an understanding of the underlying mechanism . Related work Our work builds directly on previous analyses of binary sentiment classification by Maheswaranathan et al . ( 2019 ) and Maheswaranathan & Sussillo ( 2020 ) . Apart from these works , the dynamical properties of continuous-time RNNs have been extensively studied ( Vyas et al. , 2020 ) , largely for connections to neural computation in biological systems . Such analyses have recently begun to yield insights on discrete-time RNNs : for example , Schuessler et al . ( 2020 ) showed that training continuous-time RNNs on low-dimensional tasks led to low-dimensional updates to the networks ’ weight matrices ; this observation held empirically in binary sentiment LSTMs as well . Similarly , by viewing the discrete-time GRU as a discretization of a continuous-time dynamical system , Jordan et al . ( 2019 ) demonstrated that the continuous-time analogue could express a wide variety of dynamical features , including essentially nonlinear features like limit cycles . Understanding and interpreting learned neural networks is a rapidly-growing field . Specifically in the context of natural language processing , the body of work on interpretability of neural models is reviewed thoroughly in Belinkov & Glass ( 2018 ) . Common methods of analysis include , for example , training auxiliary classifiers ( e.g. , part-of-speech ) on RNN trajectories to probe the network ’ s 1As will be discussed in more detail below , by fixed points we mean hidden state locations that are approximately fixed on time-scales of order of the average phrase length for the task at hand . Throughout this work we will use the term fixed point manifold to be synonymous with manifolds of slow points . 2A 1-simplex is a line segment , a 2-simplex a triangle , a 3-simplex a tetrahedron , etc . A simplex is regular if it has the highest degree of symmetry ( e.g . an equilateral triangle is a regular 2-simplex ) . representations ; use of challenge sets to capture wider language phenomena than seen in natural corpora ; and visualization of hidden unit activations as in Karpathy et al . ( 2015 ) and Radford et al . ( 2017 ) . 2 SETUP . Models We study three common RNN architectures : LSTMs ( Hochreiter & Schmidhuber , 1997 ) , GRUs ( Cho et al. , 2014 ) , and UGRNNs ( Collins et al. , 2016 ) . We denote their n-dimensional hidden state and d-dimensional input at time t as ht and xt , respectively . The function that applies hidden state update for these networks will be denoted by F , so that ht = F ( ht−1 , xt ) . The network ’ s hidden state after the entire example is processed , hT , is fed through a linear layer to get N output logits for each label : y = WhT + b . We call the rows of W ‘ readout vectors ’ and denote the readout corresponding to the ith neuron by ri , for i = 1 , . . . , N . Throughout the main text , we will present results for the GRU architecture . Qualitative features of results were found to be constant across all architectures ; additional results for LSTMs and UGRNNs are given in Appendix E. Tasks The classification tasks we study fall into three categories . In the categorical case , samples are classified into non-overlapping classes , for example “ sports ” or “ politics ” . By contrast , in the ordered case , there is a natural ordering among labels : for example , predicting a numerical rating ( say , out of five stars ) accompanying a user ’ s review . Like the categorical labels , ordered labels are exclusive . Some tasks , however , involve labels which may not be exclusive ; an example of this multi-labeled case is tagging a document for the presence of one or more emotions . A detailed description of the natural and synthetic datasets used is provided in Appendices C and D , respectively . Linearization and eigenmodes Part of our analysis relies on linearization to render the complex RNN dynamics tractable . This linearization is possible because , as we will see , the RNN states visited during training and inference lie near approximate fixed points h∗ of the dynamics—points that the update equation leave ( approximately ) unchanged , i.e . for which h∗ ≈ F ( h∗ , x ) .3 Near these points , the dynamics of the displacement ∆ht : = ht − h∗ from the fixed point h∗ is well approximated by the linearization ∆ht ≈ Jrec| ( h∗ , x∗ ) ∆ht−1 + J inp ∣∣ ( h∗ , x∗ ) ( xt − x∗ ) , ( 1 ) where we have defined the recurrent and input Jacobians J recij ( h , x ) : = ∂F ( h , x ) i ∂hj and J inpij ( h , x ) : = ∂F ( h , x ) i ∂xj , respectively ( see Appendix A for details ) . In the linear approximation , the spectrum of Jrec plays a key role in the resulting dynamics . Each eigenmode of Jrec represents a displacement whose magnitude either grows or shrinks exponentially in time , with a timescale τa determined by the magnitude of the corresponding ( complex ) eigenvalue λa via the relation τa : = |log |λa||−1 . Thus , eigenvalues within the unit circle thus represent stable ( decaying ) modes , while those outside represent unstable ( growing ) modes . The Jacobians we find in practice almost exclusively have stable modes , most of which decay on very short timescales ( a few tokens ) . Eigenmodes near the unit circle have long timescales , and therefore facilitate the network ’ s storage of information . Latent semantic analysis For a given text classification task , one can summarize the data by building a matrix of word or token counts for each class ( analogous to a document-term matrix ( Manning & Schutze , 1999 ) , where the documents are classes ) . Here , the i , j entry corresponds to the number of times the ith word in the vocabulary appears in examples belonging to the jth class . In effect , the column corresponding to a given word forms an “ evidence vector ” , i.e . a large entry in an particular row suggests strong evidence for the corresponding class . Latent semantic analysis ( LSA ) ( Deerwester et al. , 1990 ) looks for structure in this matrix via a singular value decomposition ( SVD ) ; if the evidence vectors lie predominantly in a low-dimensional subspace , LSA will pick up on this structure . The top singular modes define a “ semantic space ” : the left singular vectors correspond to the projections of each class label into this space , and the right singular vectors correspond to how individual tokens are represented in this space . 3Although the fixed point expression depends on the input x , throughout this text we will only study fixed points with the zero input . That is , we focus on the autonomous dynamical system given by ht+1 = F ( ht , 0 ) ( see Appendix A for details ) . Below , we will show that RNNs trained on classification tasks pick up on the same structure in the dataset as LSA ; the dimensionality and geometry of the semantic space predicts corresponding features of the RNNs . Regularization While the main text focuses on the interaction between dataset statistics and resulting network dimensionality , regularization also plays a role in determining the dynamical structures . In particular , strongly regularizing the network can reduce the dimensionality of the resulting manifolds , while weakly regularizing can increase the dimensionality . Focusing on ` 2-regularization , we document this effect for the synthetic and natural datasets in Appendicies D.1 and F , respectively .

Summary: This paper presents an analysis on the trained recurrent neural networks (RNN) especially for NLP classification problems. The analysis takes the dynamical systems point of view and investigates the dynamics by looking at the Jacobians around the fixed points. This work founds low dimensionalility and attractor dynamics in the RNNs which might lead to a better undertanding of RNNs.