Venue: NAACL 2022

# Implicit \(N\)-grams Induced by Recurrence

Xiaobing Sun and Wei Lu

StatNLP Research Group

Singapore University of Technology and Design

xiaobing_sun@mymail.sutd.edu.sg, liwei@sutd.edu.sg

###### Abstract

Although self-attention based models such as Transformers have achieved remarkable successes on natural language processing (NLP) tasks, recent studies reveal that they have limitations on modeling sequential transformations Hahn (2020), which may prompt re-examinations of recurrent neural networks (RNNs) that demonstrated impressive results on handling sequential data. Despite many prior attempts to interpret RNNs, their internal mechanisms have not been fully understood, and the question on how exactly they capture sequential features remains largely unclear. In this work, we present a study that shows there actually exist some explainable components that reside within the hidden states, which are reminiscent of the classical \(n\)-grams features. We evaluated such extracted explainable features from trained RNNs on downstream sentiment analysis tasks and found they could be used to model interesting linguistic phenomena such as negation and intensification. Furthermore, we examined the efficacy of using such \(n\)-gram components alone as encoders on tasks such as sentiment analysis and language modeling, revealing they could be playing important roles in contributing to the overall performance of RNNs. We hope our findings could add interpretability to RNN architectures, and also provide inspirations for proposing new architectures for sequential data.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Modern recurrent neural networks (RNNs), including Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997) and Gated Recurrent Units (GRU) Cho et al. (2014), have demonstrated impressive results on tasks involving sequential data. They have proven to be capable of modeling formal languages Weiss et al. (2018); Merrill (2019); Merrill et al. (2020) and capturing structural features Li et al. (2015a, b, 2016); Linzen et al. (2016); Belinkov et al. (2017); Liu et al. (2019) on NLP tasks. Although Transformers Vaswani et al. (2017) have achieved remarkable performances on NLP tasks such as machine translation, it is argued that they may have limitations on modeling hierarchical structure Tran et al. (2018); Hahn (2020) and cannot handle functions requiring sequential processing of input well Dehghani et al. (2019); Hao et al. (2019); Bhattimishra et al. (2020); Yao et al. (2021). Furthermore, a recent work shows that combining recurrence and attention Lei (2021) can result in strong modeling capacity. Another recent work incorporating recurrent cells into Transformers Hutchins et al. (2022) substantially improved performance on language modeling involving very long sequences, prompting re-investigations of RNNs. On the other hand, it was observed in prior work that RNNs were able to capture linguistic phenomena such as negation and intensification Li et al. (2016), but the question why they could achieve so still largely remains unanswered.

In this work, we focus on better understanding RNNs from a more theoretical perspective. We demonstrate that the recurrence mechanism of RNNs may induce a linear combination of interpretable components. These components reside in their hidden states in the form of the iterated matrix-vector multiplication that is based on the representations of tokens in the (reverse) order they appear in the sequence. Such components, solely depending on inputs and learned parameters, can be conveniently interpreted and are reminiscent of those compositional features used in classical \(n\)-gram models Jurafsky and Martin (2009). They may also provide us with insights on how RNNs compose semantics from basic linguistic units. Our analysis further shows that, the hidden state at each time step includes a weighted combination of components that represent all the "\(n\)-grams" ending at that specific position in the sequence as shown

Figure 1: An RNN hidden state may encode a linear combination of all the \(n\)-grams ending at the current time step.



in Figure 1. We gave specific representations for the \(n\)-gram components in Elman RNNs Elman (1990), GRUs and LSTMs.

We investigated the interpretability of those \(n\)-gram components on trained RNN models, and found they could explain phenomena such as negation and intensification and reflect the overall polarity on downstream sentiment analysis tasks, where such linguistic phenomena are prevalent. Our experiments also revealed that the GRU and LSTM models are able to yield better capabilities in modeling such linguistic phenomena than the Elman RNN model, partly attributed to the gating mechanisms they employed which resulted in more expressive \(n\)-gram components. We further show that the linear combination of such components yields effective context representations. We explored the effectiveness of such \(n\)-gram components (along with the corresponding context representations) as alternatives to standard RNNs, and found they can generally yield better results than the baseline compositional methods on several tasks, including sentiment analysis, relation classification, named entity recognition, and language modeling.

We hope that our work could give inspirations to our community, serving as a useful step towards proposing new architectures for capturing contextual information within sequences.1

Footnote 1: Our code is available at [https://github.com/richardsun-voyager/inhibr](https://github.com/richardsun-voyager/inhibr).

## 2 Related Work

Interpretability of RNNs:A line of work focuses on the relationship between RNNs and finite-state machines Weiss et al. (2018); Merrill (2019); Suzgun et al. (2019); Merrill et al. (2020); Eyraud and Ayache (2020); Rabusseau et al. (2019), providing explanation and prediction on the expressive power and limitations of RNNs on formal languages both empirically and theoretically. Kanai et al. (2017) investigated conditions that could prevent gradient explosions for GRU based on dynamics. Maheswaranathan et al. (2019) and Mashewaranathan and Sussillo (2020) linearized the dynamics of RNNs around fixed points of hidden states and elucidated contextual processing. Our work focuses on studying a possible mechanism of RNNs that handles exact linguistic features.

Another line of work aims to detect linguistic features captured by RNNs. Visualization approaches Karpathy et al. (2015); Li et al. (2016) were initially used to examine compositional information in RNN outputs. Linzen et al. (2016) assessed LSTMs' ability to learn syntactic structure and Emami et al. (2021) gave rigorous explanations on the standard RNNs' ability to capture long-range dependencies. Decomposition methods Murdoch and Szlam (2017); Murdoch et al. (2018); Singh et al. (2019); Arras et al. (2017, 2019); Chen et al. (2020) were proposed to produce importance scores for hierarchical interactions in RNN outputs. Our work can be viewed as an investigation on how those interaction came about.

Compositional Models:A variety of compositional functions based on vector spaces have been proposed in the literature to compose semantic meanings of phrases, including simple compositions of adjective-neum phrases represented as matrix-vector multiplication Mitchell and Lapata (2008); Baroni and Zamparelli (2010) and a matrix-space model Rudolph and Giesbrecht (2010); Yessenalina and Cardie (2011) based on matrix multiplication. Socher et al. (2012, 2013) introduced a recursive neural network model that assigns every word and longer phrase in a parse tree both a vector and a matrix, and represents composition of a non-terminal node with matrix-vector multiplication. Kalchbrenner and Blunsom (2013) employed convolutional and recurrent neural networks to model compositionality at the sentence and discourse levels respectively. Those models are designed in an intuitive manner based on the nature of languages thus being interpretable. We can show that RNNs may process contextual information in a way bearing a resemblance to those early models.

## 3 A Theory on \(N\)-gram Representation

First, let us spend some time to discuss how to represent \(n\)-grams. Various approaches to representing \(n\)-grams have been proposed in the literature Mitchell and Lapata (2008); Bengio et al. (2003); Mitchell and Lapata (2008); Mnih and Teh (2012); Ganguli et al. (2008); Orhan and Pitkow (2020); Emami et al. (2021); Rudoloph and Giesbrecht (2010); Yessenalina and Cardie (2011); Baroni and Zamparelli (2010). We summarize in Table 1 different approaches for representing \(n\)-grams.

Although empirically it has been shown that different approaches can lead to different levels of effectiveness, the rationales underlying many of the design choices remain unclear. In this section, we establish a small theory on representing \(n\)-grams, which leads to a new formulation on capturing the semantic information within \(n\)-grams.

Let us assume we have a vocabulary \({\mathbb{V}}\) that consists of all possible word tokens. The set of \(n\)-grams can be denoted as \({\mathbb{V}}^{*}\) (including the special 

\(n\)-gram which is the empty string \(\epsilon\)). Consider three \(n\)-grams \(a\), \(b\), and \(c\) from \({\mathbb{V}^{*}}\), with their semantic representations \(r(a)\), \(r(b)\), and \(r(c)\) respectively. Similarly, we may have \(r(ab)\) which return the semantic representations of the concatenated \(n\)-grams \(ab\). It is desirable for our representations to be compositional in some sense. Specifically, a longer \(n\)-gram may be semantically related to those shorter \(n\)-grams it contains in some way.

Under some mild compositional assumptions related to the _principle of compositionality_[14]2, it is reasonable to expect that there exists some sort of rule or operation that allows us to compose semantics of longer \(n\)-grams out of shorter ones. Let us use \(\otimes\) to denote such an operation. We believe a good representation system for \(n\)-grams shall satisfy several key properties. First, the semantics of the \(n\)-gram \(abc\) shall be determined through either combining the semantics of the two \(n\)-grams \(a\) and \(bc\) or through combining the semantics of \(ab\) and \(c\). The semantics of \(abc\) is unique, regardless of which of these two ways we use. Second, for the empty string \(\epsilon\), it should not convey any semantics. Formally, we can write them as:3

Footnote 2: The principle states that “the meaning of an expression is determined by the meanings of the sub-expressions it contains and the rules used to combine such sub-expressions”.

Footnote 3: Besides, another important property is that the order used for combining two \(n\)-grams does matter. In other words, \(r(a)\otimes r(b)\) usually may not be the same as \(r(b)\otimes r(a)\).

* **Associativity**: \(\forall a,b,c\in{\mathbb{V}^{*}}\), \((r(a)\otimes r(b))\otimes r(c)=r(a)\otimes(r(b)\otimes r(c))\)
* **Identity**: \(\forall a\in{\mathbb{V}^{*}}\), \(r(a)\otimes r(\epsilon)=r(a)\), and \(r(\epsilon)\otimes r(a)=r(a)\)

This essentially shows that the representation space for all \(n\)-grams under the operation \(\otimes\), denoted as \(({\mathbb{V}^{*}},\otimes)\), forms a _monoid_, an important concept in abstract algebra [9], with significance in theoretical computer science [50; 86].

On the other hand, it can be easily verified that the space of all \(d\times d\) (where \(d\) is an integer) real square matrices under matrix multiplication, denoted as \(({\mathbb{R}}^{d\times d},\cdot)\), also strictly forms a monoid (i.e., it is associative and has an identity, but is _not_ commutative). We can therefore establish a homomorphism from \({\mathbb{V}^{*}}\) to \({\mathbb{R}}^{d\times d}\), resulting in the function \(r(\cdot)\in{\mathbb{V}^{*}}\rightarrow{\mathbb{R}}^{d\times d}\).

This essentially means that we may be able to rely on a sub-space within \({\mathbb{R}}^{d\times d}\) as our mathematical object to represent the space of \(n\)-grams, where the matrix multiplication operation can be used to compose representations for longer \(n\)-grams from shorter ones. Thus, for a unigram \(x\) (a single word in the vocabulary), we have:

\[r(x):=\boldsymbol{A}_{x} \tag{1}\]

where \(\boldsymbol{A}_{x}\in{\mathbb{R}}^{d\times d}\) is the representation for the word \(x\) (how to learn such a matrix is a separate question to be discussed later). Note that the empty string \(\epsilon\) comes with a unique representation which is the \(d\times d\) identity matrix \(\boldsymbol{I}\).

We can either use matrix left-multiplication or right-multiplication as our operator \(\otimes\). Assume the language under consideration employs the left-to-right writing system. It is reasonable to believe that a human reader processes the text left-to-right, and the semantics of the text gets evolved each time the reader sees a new word. We may use the matrix left-multiplication as the preferred operator in this case. The system will left-multiply (modify) an existing \(n\)-gram representation with a matrix associated with the new word that appears right after the existing \(n\)-gram, forming the representation of the new \(n\)-gram. Such an operation essentially performs a transform that simulates the process of yielding new semantics when appending a new word at the end of an existing phrase. With this, for

\begin{table}
\begin{tabular}{l l c l l} \hline \hline
* \\ (MVW) & & & & \\ \hline Matrix-Vector Multiplicative-Additive (MVWA) & \({\mathbf{u}}_{ij}=\left(\prod_{i\neq j}^{d+1}A(x_{i})\right)g(x_{i})\) & \(\sum_{i=1}^{d}{\mathbf{v}}_{ii,i}\) & \(t\) & This work \\ \hline \hline \end{tabular}
\end{table}
Table 1: Different models for defining representations for \(n\)-grams within the phrase \(x_{1},x_{2},\ldots,x_{t-1},x_{t}\) and constructing the context representation out of the \(n\)-grams during learning. \(L\): the maximum length allowed for the context representation. \({\boldsymbol{C}}\) is a weight matrix, and \({\boldsymbol{C}}_{k}\) is a (relative) position-specific weight matrix. \(A\) and \(g\) are functions that return a matrix and a vector respectively.
 

a general \(n\)-gram \(x_{i},x_{i+1},\ldots,x_{t}\) (\(i\leq t\)), we have:

\[r(x_{i},x_{i+1},\ldots,x_{t})=\prod_{k=t}^{i}\mathbf{A}_{x_{k}} \tag{2}\]

However, the conventional wisdom in NLP has been to use vectors to represent basic linguistic units such as words, phrases or sentences Mikolov et al. (2013, 2013); Pennington et al. (2014); Kiros et al. (2015). This can be achieved by a transform:

\[\left(\prod_{k=t}^{i}\mathbf{A}_{x_{k}}\right)\mathbf{u} \tag{3}\]

where \(\mathbf{u}\in\mathbb{R}^{d}\) is a vector that maps the resulting matrix representation into a vector representation.

Next, we will embark on our journey to examine the internal representations of RNNs. As we will see, interestingly, our developed \(n\)-gram representations can emerge within such models.

## 4 Interpretable Components in RNNs

An RNN is a parameterized function whose hidden state can be written recursively as:

\[\mathbf{h}_{t}=f(x_{t},\mathbf{h}_{t-1}), \tag{4}\]

where \(x_{t}\) is the input token at time step \(t\) and \(\mathbf{h}_{t-1}\in\mathbb{R}^{d}\) is the previous hidden state. Assume \(f\) is differentiable at any point, with the Taylor expansion, \(\mathbf{h}_{t}\) can be rewritten as:

\[\mathbf{h}_{t}=f(x_{t},\mathbf{0})+\nabla f(x_{t},\mathbf{0})\mathbf{h}_{t-1}+o(\mathbf{ h}_{t-1}), \tag{5}\]

where \(\nabla f(x_{t},\mathbf{0})=\frac{\partial f}{\partial\mathbf{h}_{t-1}}|\mathbf{h}_{t- 1}=\mathbf{0}\) is the Jacobian matrix, and \(o\) is the remainder of the Taylor series.

Let \(g(x_{t})=f(x_{t},\mathbf{0})\) and \(A(x_{t})=\nabla f(x_{t},\mathbf{0})\). Note that \(g(x_{t})\in\mathbb{R}^{d}\) and \(A(x_{t})\in\mathbb{R}^{d\times d}\) are both functions of \(x_{t}\). Therefore, the equation above can be written as:

\[\mathbf{h}_{t}=g(x_{t})+A(x_{t})\mathbf{h}_{t-1}+o(\mathbf{h}_{t-1}). \tag{6}\]

If the hidden state has a sufficiently small norm, it can be approximated by the first-order Taylor expansion as follows4:

Footnote 4: There will be an “approximation gap” at each time step between the “approximated” hidden state and the actual standard hidden state. We may leverage regularization methods such as weight-decaying and the spectral normalization Miyato et al. (2018) to prevent the gap from growing unbounded.

\[\mathbf{h}_{t}\approx g(x_{t})+A(x_{t})\mathbf{h}_{t-1}. \tag{7}\]

Next we illustrate how this recurrence relation can help us identify some salient components.

### Emergence of \(N\)-grams

Consider the simplified RNN with the following recurrence relation,

\[\mathbf{h}_{t}=g(x_{t})+A(x_{t})\mathbf{h}_{t-1}, \tag{8}\]

where \(\mathbf{h}\in\mathbb{R}^{d}\), and \(g(x_{t})\in\mathbb{R}^{d}\) and \(A(x_{t})\in\mathbb{R}^{d\times d}\) are functions of \(x_{t}\). This recurrence relation can be expanded repeatedly as follows,

\[\begin{split}\mathbf{h}_{t}&=g(x_{t})+A(x_{t})g(x_{t-1 })+A(x_{t})A(x_{t-1})\mathbf{h}_{t-2}\\ &=\cdots=\sum_{i=1}^{t}A(x_{t})\dots A(x_{i+1})g(x_{i})\\ &=\hskip-9.0pt\sum_{i=1}^{t}\underbrace{\left(\prod_{j=t}^{i+1}A (x_{j})\right)g(x_{i})}_{\mathbf{v}_{i:t}},\end{split}\]

We can see that \(\mathbf{v}_{i:t}\) bear some resemblance to the term in Equation 3, which can be rewritten as:

\[\left(\prod_{j=t}^{i+1}\underbrace{\mathbf{A}_{x_{j}}}_{A(x_{j})}\right)\!\! \underbrace{\big{(}\mathbf{A}_{x_{i}}\mathbf{u}\big{)}}_{g(x_{i})}. \tag{9}\]

With the definition \(A(x_{j}):=\mathbf{A}_{x_{j}}\) and \(g(x_{i}):=A(x_{i})\mathbf{u}\), we can see \(\mathbf{v}_{i:t}\) can be interpreted as an "\(n\)-gram representation" that we developed in the previous section. It is important to note that, however, the use of function \(g(x_{i})\) in RNNs may lead to greater expressive power than the original formulation based on \(\mathbf{A}_{x_{i}}\mathbf{u}\).5

Footnote 5: This is because we can always construct \(g(x_{i})\) from any given \(\mathbf{A}_{x_{i}}\) and \(\mathbf{u}\), but in general we may not always be able to decompose \(g(x_{i})\) into the form \(\mathbf{A}_{x_{i}}\mathbf{u}\) (for all \(x_{i}\)).

This interesting result shows that the hidden state of a simple RNN (characterized by Equation 8) is the sum of the representations of all the \(n\)-grams ending at time step \(t\). Such salient components within RNN also show that the standard RNN may actually have a mechanism that is able to capture implicit \(n\)-gram information as described above. This leads to the following definition:

**Definition 1** (\(N\)-gram Representation): _For the \(n\)-gram \(x_{i},x_{i+1},\dots,x_{t}\), its representation is:_

\[\mathbf{v}_{i:t}=\left(\prod_{j=t}^{i+1}A(x_{j})\right)g(x_{i}), \tag{10}\]

_where \(A(x_{j})\in\mathbb{R}^{d\times d}\) and \(g(x_{i})\in\mathbb{R}^{d}\)._

### Context Representation

With the above definition, we may want to consider how to perform learning. The learning task involves identifying the functions \(A\) and \(g\) - in other words, learning representations for word tokens.

A typical learning setup that we may consider here is the task of language modeling. Such a task 

can be defined as predicting the next word \(x_{t+1}\) based on the representation of preceding words \(x_{1},x_{2},\ldots,x_{t}\) which serves as its left context. This is an unsupervised learning task, where the underlying assumption involved is the _distributional hypothesis_Harris (1954). Specifically, the model learns how to "reconstruct" the current word \(x_{t+1}\) out of \(x_{1},x_{2},\ldots,x_{t}\) which serves as its context.

Now the research question is how to define the representation for this specific context. As this left context is also an \(n\)-gram, it might be tempting to directly use its \(n\)-gram representation defined above to characterize such a left context. However, we show such an approach is not desirable.

The \(n\)-gram representation for this context can be written in the following alternative form:

\[\mathbf{v}_{1:t}=\begin{pmatrix}2\\ j=t\end{pmatrix}g(x_{i})=W(x_{2:t})g(x_{1}), \tag{11}\]

This shows that the \(n\)-gram representation of \(x_{1},x_{2},\ldots,x_{t}\) could be interpreted as a "weighted" representation of the word \(x_{1}\) (where the weight matrix is derived from the words between \(x_{1}\) and \(x_{t+1}\), measuring the strength of the connection between them). However, ideally, the context representation shall not just take \(x_{1}\) but other adjacent words preceding \(x_{t+1}\) into account, where each word contributes towards the final context representation based on the connection between them. This leads to the following way of defining the context:

\[\sum_{i=1}^{t}\mathbf{v}_{i:t}=\sum_{i=1}^{t}\left(\prod_{j=t}^{i+1}A( x_{j})\right)g(x_{i}) \tag{12}\] \[=\sum_{i=1}^{t}W(x_{i:t})g(x_{i}),\]

In fact, such an idea of defining the context as a weighted combination of surrounding words is not new - it recurs in the literature of language modeling Bengio et al. (2003); Mnih and Teh (2012), word embedding learning Mikolov et al. (2013a, b), and graph representation learning Cao et al. (2016).

Interestingly, the hidden states in the RNNs, as shown in Equation 9, also suggest exactly the same way of defining this left context. Indeed, when using RNNs for language modeling, each hidden state is exactly serving as the context representation for predicting the next word in the sequence.

The above gives rise to the following definition:

**Definition 2** (Context Representation): _For the \(n\)-gram \(x_{1},x_{2},\ldots,x_{t}\), its representation when serving as the (left) context is:_

\[\mathbf{c}_{1:t}=\sum_{i=1}^{t}\mathbf{v}_{i:t}=\sum_{i=1}^{t}\left(\prod_{y=t}^{i+1} A(x_{j})\right)g(x_{i}), \tag{13}\]

_where \(A(x_{j})\in\mathbb{R}^{d\times d}\) and \(g(x_{i})\in\mathbb{R}^{d}\)._

### Model Parameterization

With the above understandings on such salient components within RNNs, we can now look into how different variants of RNNs parameterize the functions \(A\) and \(g\). The definition of Elman RNN, GRU and LSTM together with the corresponding Jacobian matrix \(A(x_{t})\) and vector function \(g(x_{t})\) functions are listed in Table 26. We discuss how such different parameterizations may lead to different expressive power when they are used in practice.

Footnote 6: For brevity, we suppress biases following Merrill et al. (2020).

We can see the ways GRU or LSTM parameterize \(A(x_{t})\) and \(g(x_{t})\) appear to be more complex compared to Elman RNN. This can partially be attributed to their gating mechanisms. Although the original main motivation of introducing such mechanisms may be to alleviate the exploding gradient and vanishing gradient issues Hochreiter and Schmidhuber (1997); Cho et al. (2014), we could see such designs also result in terms describing gates and intermediate representations. \(A\) and \(g\) are then independently derived based on certain rich interactions between such terms. We believe such interactions may likely increase the expressive power of the resulting \(n\)-gram representations. We will validate these points and discuss more in our experiments.

## 5 Experiments

In our experiments, we focus on the following aspects: 1) understanding the effectiveness of the proposed \(n\)-gram (and context) representations when used in practice, as compared to baseline models; 2) examining the significance of the choice of context representation; 3) interpreting the proposed representations by examining how well they could be used to capture certain linguistic phenomena.

We employ the sentiment analysis, relation classification, named entity recognition (NER) and language modeling tasks as testbeds. The first task is often used in investigating \(n\)-gram phenomena Yessenalina and Cardie (2011); Li et al. (2016) while the others are often used in examining how capable an encoder is when extracting features from texts Grave et al. (2018); Zhou et al. (2016); Lample et al. (2016).

DatasetsFor sentiment analysis, we considered the Stanford Sentiment Treebank (SST) Socher et al. (2013), the IMDB Maas et al. (2011), and the AG-news topic classification7Zhang et al. ( 

2015) datasets. The first dataset has sufficient labels for phrase-level analysis, the second dataset has instances with relatively longer lengths, and the third one is multi-class. For relation classification and NER, we considered the SemEval 2010 Task 8 Hendrickx et al. (2010) and CoNLL-2003 Tjong Kim Sang and De Meulder (2003) datasets respectively. For language modeling, we considered the Penn Treebank (PTB) dataset Marcus et al. (1993), the Wikitext-2 (Wiki2) dataset and the Wikitext-103 Wiki103 dataset Merity et al. (2016). PTB is relatively small while Wiki103 is large. The statistics are shown in Tables 6 and 7 in the appendix.

BaselinesThe \(n\)-gram representations (together with their corresponding context representations) discussed in the literature are considered as baselines, which are listed in Table 1 along with the MVMA and MVM models. MVM(A)-G/L/E refers to the MVM(A) model created with the \(A\) and \(g\) functions derived from GRU/LSTM/Elman, but are trained directly from data. The \(A\) and \(g\) functions for GRU, LSTM and Elman are listed in Table 2.

Additionally, to understand whether the complexity of \(A\) affects the expressive power, we created a new model called MVMA-ME, which comes with an \(A\) function that is slightly more complex than that of MVMA-E but less complex than those of MVMA-G and MVMA-L: \(A(x_{t})=0.25\operatorname*{diag}[\tanh(\mathbf{W}\mathbf{x}_{t})]\mathbf{M}+0.5\mathbf{I}\) and \(g(x_{t})=\tanh(\mathbf{W^{\prime}}\mathbf{x}_{t})\) (here, \(\mathbf{W}\), \(\mathbf{M}\) and \(\mathbf{W^{\prime}}\) are learnable weight matrices). The \(g\) function is the same as that of MVMA-E.

SetupFor sentiment analysis, relation classification and language modeling, models consist of one embedding layer, one RNN layer, and one fully-connected layer. The Adagrad optimizer Duchi et al. (2011) was used along with dropout Srivastava et al. (2014) for sentiment analysis8 and relation classification. For language modeling, models were trained with the Adam optimizer Kingma and Ba (2014). We ran word-level models with truncated backpropagation through time Williams and Peng (1990) where the truncated length was set to 35. Adaptive softmax Joulin et al. (2017) was used for Wiki103. For NER, models consist of one embedding layer, one bidirectional RNN layer, one projection layer and one conditional random field (CRF) layer. The SGD optimizer was used. Final models were chosen based on the best validation results. More implementation details can be found in the appendix.

Footnote 8: We investigated the approximation between RNNs and their corresponding recurrence relations in Appendix B.2. The spectral normalization Miyato et al. (2018) was used on the weight matrices \(\mathbf{W}_{h}\), for standard RNNs.

### Comparison of Representation Models

We investigate how baseline \(n\)-gram representation models9, the MVM model, and the MVMA model perform on the aforementioned testbeds. We also compare with the standard RNN models.

Footnote 9: We excluded VM, which we found was hard to train. We also excluded MVM-R which only considers bigrams.

Sentiment AnalysisApart from the GRU and LSTM models, it can be observed that our MVMA-G and MVMA-L models are also able to achieve competitive results on three sentiment analysis datasets, as we can see from Table 3, demonstrating the efficacy of those recurrence-induced \(n\)-gram

\begin{table}
\begin{tabular}{c c l}  & **Definition** & **Parameterization** \\ \hline \multirow{4}{*}{
\begin{tabular}{l} **Model** \\ **MM** 

representations. Although Elman RNN and its corresponding MVMA-E and MVM-E models also have a mechanism for capturing \(n\)-gram information (similar to GRU and LSTM), they did not perform well, which may be attributed to a limited expressive power of their \(A\) and \(g\) functions when used for defining \(n\)-grams as described previously.

Both MM and VA-EW fail to converge on AGnews and IMDB, showing challenges for them to handle long instances. This may be explained by the lengthy matrix multiplication involved in their representations, which may result in vanishing/exploding gradient issues. Interestingly, MVM-G and MVM-L, which solely rely on the longest \(n\)-gram representation, are also able to achieve good results on SST-2, indicating a reasonable expressive power of such \(n\)-gram representations alone. However, they fail to catch up with MVMA-G and MVMA-L on IMDB which contains much longer instances, confirming the significance of the context representation, which captures \(n\)-grams of varying lengths.

Unlike MVMA-E, the MVMA-ME model does not suffer from loss stagnation on AG-news and IMDB but the performance on IMDB obviously falls behind MVMA-G and MVMA-L as shown in Table 3. This indicates a sufficiently expressive \(A(x_{t})\) (such as the Jacobian matrices of GRU and LSTM) may be needed to handle long instances.

Relation Classification & NERFor relation classification, context representations (or final hidden states) are used for classification. For NER, we use the concatenated context representations (or hidden states) at each position of bidirectional models to predict entities and their types. Table 4 shows that MVMA-G and MVMA-L outperform the MVM-G and MVM-L models respectively on both tasks, again confirming the effectiveness of the context representations. MVM(A)-E did not perform as well as MVM(A)-G and MVM(A)-L, which demonstrates the significance of expressive power for the \(A\) and \(g\) functions. Similar to the results in sentiment analysis, MVMA-ME did not perform as well as MVMA-G and MVMA-L. However, to our surprise, MVMA-ME did not outperform VA-EW on NER, suggesting that a delicate choice of \(A\) can be important for this task. The poor performance of VA-W on NER might be explained by a weak expressive power of its \(n\)-gram representations. MM fails to converge on the relation classification task, which implies it is not robust across different datasets. Interestingly, it is remarkable that MVMA-G, MVMA-L and MVMA-E could yield competitive results compared to GRU, LSTM and Elman on NER, implying such \(n\)-gram representations could be crucial for our NER task.

Language ModelingFor the language modeling task, we choose MVMA-G, MVMA-L, MVM-G and MVM-L for experiments. We also run MVMA-ME. As we can see from Table 5, there are performance gaps between the MVMA models and the standard RNNs - though the gaps often do not appear to be particularly large. This indicates there may be extra information within higher order terms of the standard RNN functions useful for such a task. Yet, such information cannot be captured by the MVMA models that employ simplified functions. The gaps between the MVM models and MVMA models are remarkable, which again indicates that the correct way of defining the left context representation can be crucial for the task of next word prediction. MVMA-ME did not perform well on the language modeling task, which might be attributed to the less expressive power of its functions \(A\) and \(g\).

### Interpretation Analysis

We conduct some further analysis to examine the interpretability of \(n\)-gram representations. Specifically, we examine whether the models are able to capture certain linguistic phenomena such as negation, which is important for sentiment analysis Ribeiro et al. (2020). We also additionally

\begin{table}
\begin{tabular}{l c r r r} \hline \multicolumn{1}{c}{**Model**} & \multicolumn{2}{c}{**Reliative Considerations**} & \multicolumn{2}{c}{**NER**} \\  & Av & test & drv & test \\ \hline MM & - & - & 39.8606 & 38.04 \\ VQW & 41:20:37 & 39:6.9 & 76.66 & 165.16 \\ VAE & 9:71:18 & 8:33.0 & 370.0 & 634.21 \\ MVMQ & 51:20:36 & 55:64.0 & 542.16 & 47.62 \\ MVMQ & 48:81.3 & 50:51.3 & 58.11 & 48.61 \\ MVMQ & - & - & - & 27.80 & 32.64 \\ MVMQ & 62:21:10 & 59:72.0 & 58.01 & 67.23 \\ MVMQ & 57:50.3 & 56:24.8 & 756.20 & 67.93 \\ MVMQ & 27:20.9 & 58.09 & 69.01 & 67.12 \\ MPMMSE & 48:31.0 & 42:66.6 & 67.01 & 57.64 \\ GMFV & 67:20.0 & 62:21.0 & 78.60 & 57.69 \\ ISTM & 63:25.0 & 63:11.4 & 86.30 & 63.63 \\ Human & 27:80.9 & 25:84.9 & 67.11 & 10.99 & 86.46 \\ \hline \end{tabular}
\end{table}
Table 4: F1 scores (\(\uparrow\)) (averaged over 3 runs) on the relation classification and NER tasks. “-” means the model failed to converge.

\begin{table}
\begin{tabular}{l l l l l} \hline \multicolumn{1}{c}{**Model**} & \multicolumn{2}{c}{**PFB**} & \multicolumn{2}{c}{**Wall**} & \multicolumn{2}{c}{**Waiss**} \\ \cline{2-5} \multirow{-2}{*}{GLU} & dev & 18:0.4 & 14:0.4 & 14:0.4 & 149:0.6 \\  & test & 110:1.0 & 14:0.4 & 136.80 & 11.33 \\ MVMQ & dev & 19:95.0 & 15:0.3 & 111.80 & 5.5 \\  & test & 111:10.2 & 12:0.2 & 12.01 & 11.52 \\ \hline MVMQ & dev & 14:55.1 & 170.12 & -2.8 & - \\ MVMQ & test & 18:38.0 & 160.02 & 6.2 & - \\ \hline LSMM & vabl & 118:68.0 & 166.06 & 160.68 \\  & test & 108:0.4 & 184.03 & 112.48 \\ MVMQ & dev & 121:50.5 & 150.5 & 150.5 & 10.16 \\ MVMQ+L & resa & 113:20.5 & 142.50 & 712.61 \\ MVMQ & dev & 124:13.5 & 156.60 & - \\ MVMQ & text & 170:10.0 & 13.24 & - \\ MVMQ & dev & 124:70.9 & 160.01 & 13.51 & 12.42 \\ MVMQ & dev & 140:01.9 & 158.41 & 14.57 & 44.43 \\ \hline \end{tabular}
\end{table}
Table 5: Perplexities (\(\downarrow\)) on language modeling (averaged over 5 runs). “–”: the model failed to converge.

 

made comparisons with the vanilla Transformer Vaswani et al. (2017) here10 despite the fact that it remains largely unclear how it precisely captures sequence features such as \(n\)-grams.

Footnote 10: The mean of output representations was treated as the context representation for Transformer during training. We also tried to use the concatenation of the first and last token, following Luan et al. (2019), which yielded similar results.

We could also obtain the \(n\)-gram representations and the corresponding context representations from the learned standard RNN models, based on their learned parameters. We denote such \(n\)-gram representations as RNN\({}_{n\text{-gram}}\), and the context representations as RNN\({}_{\text{context}}\), where "RNN" can be GRU, LSTM or Elman. As \(n\)-gram representations are vectors, a common approach is to transform them into scalars with learnable parameters Murdoch et al. (2018); Sun and Lu (2020). We define the \(n\)_-gram polarity score_ to quantify the polarity information as captured by an \(n\)-gram representation \(\mathbf{v}_{i:t}\) from time step \(i\) to \(t\), which is calculated as:

\[s^{\mathbf{v}}_{i:t}=\mathbf{w}^{\top}\mathbf{v}_{i:t}, \tag{14}\]

where \(\mathbf{w}\) is the learnable weight vector of the final fully-connected layer. We also define the _context polarity score_ for the context as \(\sum_{i=1}^{t}s^{\mathbf{v}}_{i:t}\).

We trained RNNs and baseline models on SST-2 and automatically extracted 73 positive adjectives (e.g., "_nice_" and "_enjoyable_") and 47 negative adjectives (e.g., "_bad_" and "_tedious_") from the vocabulary11. \(N\)-gram polarity scores were calculated for those adjective unigrams and their negation bigrams formed by prepending "_not_" to them. For VA-EW and VA-W, their \(n\)-gram representations do not involve tokens other than the last token. Such limitations prevent them from capturing any negation information. We therefore calculate the context polarity scores using their context representations instead (which in this case is a bigram). This also applies to Transformer for the same reason.

Footnote 11: Such adjectives and detailed automatic extraction process can be found in the appendix.

We observed that, for the GRU and LSTM models along with their corresponding MVMA models, the \(n\)-gram representations are generally able to learn the negation for both the adjective and their negation bigrams as shown in Figures 1(a) and 1(b)12, prepending "_not_" to an adjective will likely reverse the polarity. This might be a reason why they could achieve relatively higher accuracy on the sentiment analysis tasks. Interestingly, MVM-G could also capture negation as shown in Figure 1(c), again suggesting the impressive expressive power of such \(n\)-gram representations alone.

Footnote 12: Results of LSTM are similar to GRU, which can be found in the appendix.

However, as shown in Figure 2, models such as VA-W, MVMA-E, and MM are struggling to capture negation for negative adjectives, again implying a weaker expressive power of their \(n\)-gram representations. Specifically, MVMA-E fails to capture negation for negative adjectives, which may be attributed to a relatively weaker Jacobian matrix function \(A\) (as compared to those of GRU and LSTM) preventing them from pursuing optimal conditions.

Figure 1(e) shows that the MVMA-ME model, which has a function \(A\) less complex than the ones from MVMA-G and MVMA-L but more complex than the one from MVMA-E, still can generally learn negation of negative adjectives better than the MVMA-E model. This demonstrates the necessity of choosing more expressive \(A\) and \(g\) functions.

Interestingly, both VA-W and Transformer are struggling with capturing the negation phenomenon for negative adjectives in our experiments as shown in Figures 1(g) and 1(h), which suggests that they may have a weaker capability in modeling sequential features in our setup. However, we found they could still achieve good performances on the AGnews and IMDB datasets13. We hypothesize this is because the nature of SST-2 makes these two models suffer more on this dataset - it has rich linguistic phenomena such as negation cases while the other two datasets do not.

Footnote 13: We conducted additional experiments for Transformers on sentiment analysis. Results are in appendix.

Additionally, we examined the ability for GRU, LSTM, MVMA-G and MVMA-L to capture both the negation and intensification phenomena. For such experiments, instead of using SST-2, we trained the models on SST-5, which comes with polarity intensity information. Polarity intensities were mapped into values of \(\{-2,-1,0,+1,+2\}\), ranging from _extremely negative_ to _extremely positive_. We conducted some experiments based on the same setup above for capturing negation on SST-2. To our surprise, our preliminary results show that all models were performing substantially worse in terms of capturing intensification than capturing negations. We hypothesize that this is caused by the imbalance between negation phrases and intensification phrases. Specifically, the intensification word _"very"_ (1,729 times) was exposed less than the negation word _"not"_ (4,601 times) in the training set of SST-5.

One approach proposed in the literature for sentence classification is to consider all the hidden states of an RNN in an instance Bahdanau et al. (2015). We believe this may actually be able to al 

leviate the above issue as it allows more \(n\)-grams within an instance to be exposed to the label information. Thus, we followed their approach for training our MVMA and MVM models14.

Footnote 14: However, for simplicity, in this work we only used the mean context representations (or hidden states) instead of a weighted sum of them.

We can see that the negation and intensification phenomena can be explained by both the context representations in Figure 315. Specifically, prepending either positive or negative adjectives with "_very_" will likely strengthen their polarity while adding "_not_" will likely weaken their polarity. These results suggest that RNNs are able to capture information of linguistic significance within the sequence, and our identified \(n\)-gram representations within their hidden states appear to be playing a salient role.

Footnote 15: More results are in the appendix.

### Discussion

From the experiments above, we can see that our introduced \(n\)-gram representations, coupled with the corresponding context representations, are powerful in practice in capturing \(n\)-gram information better than the baseline compositional models introduced in the literature. We also found that RNNs can induce such representations due to their recurrence mechanism16.

Footnote 16: We also visualized the context representations and \(n\)-gram representations in the appendix, which provide intuitive understanding of them.

However, there can be several factors that affect the efficacy of different representations. First, through comparisons with different variants of MVMA, we can see that the precise way of parameterizing the functions \(A(x_{t})\) and \(g(x_{t})\) matter. Second, through the comparison between MVMA and MVM, we can see that defining an appropriate context representation that incorporates a correct set of \(n\)-grams is also important. Third, for models which do not capture such explicit \(n\)-gram features like ours, interestingly, they may still be able to yield good performances on certain tasks. For example, though VA-W and Transformer did not perform well on SST-2, they yielded results competitive to GRU and LSTM on AG-news and IMDB. This observation indicates there could be other useful features captured by such models that can contribute towards their overall modeling power.

Although in this work we did not aim to propose novel or more powerful architectures, we believe our work can be a step towards better understanding of RNN models. We also hope it can provide inspiration for our community to design more interpretable yet efficient architectures.

## 6 Conclusion

In this work, we focused on investigating the underlying mechanism of RNNs in terms of handling sequential information from a theoretical perspective. Our analysis reveals that RNNs contain a mechanism where each hidden state encodes a weighted combination of salient components, each of which can be interpreted as a representation of a classical \(n\)-gram. Through a series of comprehensive empirical studies on different tasks, we confirm our understandings on such interpretations of these components. With the analysis coupled with experiments, we provide findings on how RNNs learn to handle certain linguistic phenomena such as negation and intensification. Further investigations on understanding how the identified mechanism may capture a wider range of linguistic phenomena such as multiword expressions Schneider et al. (2014) could an interesting future direction.

Figure 3: Context polarity scores (MVMA-G, SST-5) for positive (L) and negative (R) adjectives along with their negation and intensification bigrams.

Figure 2: Distribution of polarity scores for adjectives and their negation bigrams on SST-2. _p-adj_ and _n-adj_ refer to the positive and negative adjectives respectively. [-] refers to the negation operation (prepending the word “not”). Circles refer to outliers. More results can be found in the appendix.

 



## Acknowledgements

We would like to thank the anonymous reviewers and our ARR action editor for their constructive comments. This research/project is supported by the Ministry of Education, Singapore, under its Tier 3 Programme (The Award No.: MOET32020-0004). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the Ministry of Education, Singapore.

## References

* Arras et al. (2019) Leila Arras, Jose Arjona-Medina, Michael Widrich, Gregoire Montavon, Michael Gillhofer, Klaus-Robert Muller, Sepp Hochreiter, and Wojciech Samek. 2019. Explaining and interpreting lstms. In _Explainable ai: Interpreting, explaining and visualizing deep learning_, pages 211-238. Springer.
* Arras et al. (2017) Leila Arras, Gregoire Montavon, Klaus-Robert Muller, and Wojciech Samek. 2017. Explaining recurrent neural network predictions in sentiment analysis. In _Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of ICLR_.
* Baroni and Zamparelli (2010) Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In _Proceedings of EMNLP_.
* Belinkov et al. (2017) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In _Proceedings of ACL_.
* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. _The journal of machine learning research_, 3:1137-1155.
* Bhattacharya et al. (2020) Satwik Bhattacharshra, Kabir Ahuja, and Navin Goyal. 2020. On the Ability and Limitations of Transformers to Recognize Formal Languages. In _Proceedings of EMNLP_.
* Cao et al. (2016) Shaosheng Cao, Wei Lu, and Qiongkai Xu. 2016. Deep neural networks for learning graph representations. In _Proceedings of AAAI_.
* Chen et al. (2020) Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. 2020. Generating hierarchical explanations on text classification via feature interaction detection. In _Proceedings of ACL_.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of EMNLP_.
* Dehghani et al. (2019) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In _Proceedings of ICLR_.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of machine learning research_, 12(Jul):2121-2159.
* Elman (1990) J. Elman. 1990. Finding structure in time. _Cogn. Sci._, 14:179-211.
* Emami et al. (2021) Melikasadat Emani, Mojtaba Sahraee-Ardakan, Parthe Pandit, Sundeep Rangan, and Alyson K Fletcher. 2021. Implicit bias of linear rnns. In _Proceedings of ICML_.
* Eyraud and Ayache (2020) Remi Eyraud and Stephane Ayache. 2020. Distillation of weighted automata from recurrent neural networks using a spectral approach. [https://arxiv.org/abs/2009.13101](https://arxiv.org/abs/2009.13101).
* Frege (1948) Gottlob Frege. 1948. Sense and reference. _The philosophical review_, 57(3):209-230.

* Grave et al. (2018) Edouard Grave, Armand Joulin, and Nicolas Usunier. 2018. Improving neural language models with a continuous cache. In _Proceedings of ICLR_.
* Gupta and Schutze (2018) Pankaj Gupta and Hinrich Schutze. 2018. LISA: Explaining recurrent neural network judgments via layer-wlsse semantic accumulation and example to pattern transformation. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_.
* Hahn (2020) Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. _Transactions of the Association for Computational Linguistics_.
* Hao et al. (2019) Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019. Modeling recurrence for transformer. In _Proceedings of NAACL_.
* Harris (1954) Zellig S Harris. 1954. Distributional structure. _Word_, 10(2-3):146-162.
* Hendrickx et al. (2010) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Pado, Marco Pennacchiotti, Lorenza Romano, and Stan Szapakovicz. 2010. SemEval-2010 task 8: Multiway classification of semantic relations between pairs of nominals. In _Proceedings of the 5th International Workshop on Semantic Evaluation_

* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hutchins et al. (2022) DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent transformers. _[https://arxiv.org/abs/2203.07852_](https://arxiv.org/abs/2203.07852_).
* Joulin et al. (2017) Armand Joulin, Moustapha Cisse, David Grangier, Herve Jegou, et al. 2017. Efficient softmax approximation for gpus. In _Proceedings of ICML_.
* Jurafsky and Martin (2009) Daniel Jurafsky and James H. Martin. 2009. _Speech and Language Processing (2nd Edition)_. Prentice-Hall, Inc., USA.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In _Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality_.
* Kanai et al. (2017) Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwamura. 2017. Preventing gradient explosions in gated recurrent units. In _Proceedings of NeurIPS_.
* Karpathy et al. (2015) Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. _[http://arxiv.org/abs/1506.02078_](http://arxiv.org/abs/1506.02078_).
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _Proceedings of ICLR_.
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Proceedings of NeurIPS_.
* Lallement (1979) Gerard Lallement. 1979. _Semigroups and combinatorial applications_. John Wiley & Sons, Inc.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of NAACL_.
* Lei (2021) Tao Lei. 2021. When attention meets fast recurrence: Training language models with reduced compute. In _Proceedings of EMNLP_.
* Li et al. (2016) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In _Proceedings of NAACL_.
* Li et al. (2015a) Jiwei Li, Thang Luong, and Dan Jurafsky. 2015a. A hierarchical neural autoencoder for paragraphs and documents. In _Proceedings of ACL_.
* Li et al. (2015b) Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015b. When are tree structures necessary for deep learning of representations? In _Proceedings of EMNLP_.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4.
* Liu et al. (2019) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In _Proceedings of NAACL_.
* Luan et al. (2019) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In _Proceedings of NAACL_.
* Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _Proceedings of ACL_.
* Maheswaranathan and Sussillo (2020) Niru Maheswaranathan and David Sussillo. 2020. How recurrent networks implement contextual processing in sentiment analysis. In _Proceedings of ICML_.
* Maheswaranathan et al. (2019) Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, S. Ganguli, and David Sussillo. 2019. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In _Proceedings of NeurIPS_.
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. _Computational Linguistics_, 19(2):313-330.
* Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. _[https://arxiv.org/abs/1609.07843_](https://arxiv.org/abs/1609.07843_).
* Merrill (2019) William Merrill. 2019. Sequential neural networks as automata. In _Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges_.
* Merrill et al. (2020) William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. 2020. A formal hierarchy of RNN architectures. In _Proceedings of ACL_.
* Meseguer and Montanari (1990) Jose Meseguer and Ugo Montanari. 1990. Petri nets are monoids. _Information and computation_, 88(2):105-155.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _[https://arxiv.org/abs/1301.3781_](https://arxiv.org/abs/1301.3781_).
* Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Proceedings of NeurIPS_.
* Mitchell and Lapata (2008) Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In _Proceedings of ACL_.
* Miyato et al. (2018) Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In _Proceedings of ICLR_ 

Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In _Proceedings of ICML_.
* Murdoch et al. (2018) W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In _Proceedings of ICLR_.
* Murdoch and Szlam (2017) W. James Murdoch and Arthur Szlam. 2017. Automatic rule extraction from long short term memory networks. In _Proceedings of ICLR_.
* Orhan and Pitkow (2020) Emin Orhan and Xaq Pitkow. 2020. Improved memory in recurrent neural networks with sequential non-normal dynamics. In _Proceedings of ICLR_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of EMNLP_.
* Rabusseau et al. (2019) Guillaume Rabusseau, Tianyu Li, and Doina Precup. 2019. Connecting weighted automata and recurrent neural networks through spectral learning. In _Proceedings of AISTATS_.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of ACL_.
* Rozenberg and Salomaa (2012) Grzegorz Rozenberg and Arto Salomaa. 2012. _Handbook of Formal Languages: Volume 3 Beyond Words_. Springer Science & Business Media.
* Rudolph and Giesbrecht (2010) Sebastian Rudolph and Eugenie Giesbrecht. 2010. Compositional matrix-space models of language. In _Proceedings of ACL_.
* Schneider et al. (2014) Nathan Schneider, Emily Danchik, Chris Dyer, and Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. _Transactions of the Association for Computational Linguistics_, 2:193-206.
* Singh et al. (2019) Chandan Singh, W James Murdoch, and Bin Yu. 2019. Hierarchical interpretations for neural network predictions. In _Proceedings of ICLR_.
* Socher et al. (2012) Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of EMNLP_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of ENMLP_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The journal of machine learning research_, 15(1):1929-1958.
* Sun and Lu (2020) Xiaobing Sun and Wei Lu. 2020. Understanding attention for text classification. In _Proceedings of ACL_.
* Suzgun et al. (2019) Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019. LSTM networks can perform dynamic counting. In _Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges_.
* Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In _Proceedings of CoNLL_.
* Tran et al. (2018) Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In _Proceedings of EMNLP_.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(86):2579-2605.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of NeurIPS_.
* Weiss et al. (2018) Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In _Proceedings of ACL_.
* Williams and Peng (1990) Ronald J. Williams and Jing Peng. 1990. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. _Neural Computation_, 2(4):490-501.
* Yao et al. (2021) Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. 2021. Self-attention networks can process bounded hierarchical languages. In _Proceedings of ACL-IJCNLP_.
* Yessenalina and Cardie (2011) Anirv Yessenalina and Claire Cardie. 2011. Compositional matrix-space models for sentiment analysis. In _Proceedings of EMNLP_.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Proceedings of NeurIPS_.
* Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In _Proceedings of ACL_ 

Dataset Statistics

The statistics of the sentiment analysis, relation classification and NER datasets are shown in Table 6. The language modeling datasets are obtained from Einstein.ai and the statistics are shown in Table 7.

We created the binary dataset SST-2 by extracting instances (including phrases) with polarity from the constituency parse trees in the original SST dataset Socher et al. [2013]. We merged the labels _extremely positive_ and _positive_ as _positive_ and the labels _extremely negative_ and _negative_ as _negative_. We also extracted all the phrases in the constituency parse trees from the original dataset and created the 5-class dataset SST-5. The labels _extremely positive_, _positive_, _neutral_, _negative_ and _extremely negative_ were mapped into +2, +1, 0, -1, and -2 respectively.

## Appendix B More Result from the SST datasets

### Negation and Intensification

Figure 4 shows that the \(n\)-gram representations from the LSTM model together with its corresponding MVMA-L and MVM-L models can also capture negation on the extracted adjectives from SST-2. However, VA-EW fails to capture the negation phenomenon for the negative adjectives, which may be explained by that: the \(n\)-gram representation of VA-EW solely involves the current token, thus being less expressive compared to the one from models such as MVMA-L and MVMA-G. Moreover, the MVMA-G model can also capture the negation and intensification phenomena on SST-5 as shown in Figure 5. The intensification token will generally strengthen the polarity of an adjective while the negation token will generally weaken the polarity of it.

We also visualized the polarity score of each \(n\)-gram within a sentence. Two examples are shown in Figures 5(a) and 5(b), where a warmer color indicates a higher polarity score (i.e., the \(n\)-gram is more positive). For example, "_never_" itself has a remarkably negative polarity score while "_loses_" has a remarkably positive one. Consequently, the \(n\)-grams starting from "_never_" (while ending with another word) generally have positiv

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Data** & **Train** & **Dev** & **Test** & **Value** & **Maxlen** & **Class** \\ \hline SST-2 & 98,794 & 872 & 1,821 & 17,404 & 54 & 2 \\ MDB & 17,212 & 4,304 & 4,363 & 63,311 & 437 & 2 \\ AO-news & 110,000 & 100,000 & 7,600 & 55,686 & 212 & 4 \\ SST-5 & 318,582 & 41,447 & 82,600 & 18,025 & 54 & 5 \\ SemEval & 7,000 & 1,000 & 2,717 & 27,115 & 91 & 10 \\ CALL-2003 & 14,897 & 3,466 & 3,684 & 26,873 & 113 & 20 \\ \end{tabular}
\end{table}
Table 6: Statistics of the sentiment analysis, relation classification and NER datasets. “V.size” indicates the vocabulary size and “Max.len” indicates the maximum length of the instances. “SemEval” refers to the SemEval 2010 Task 8 dataset for relation classification. For CoNLL-2003, “class” refers to the tag size.

\begin{table}
\begin{tabular}{c l r r r} \hline \hline  & **Dataset** & **Train** & **Dev** & **Test** \\ \hline PTB & Token Num & 887,521 & 70,390 & 78,669 \\  & Vocab Size & & 10,000 & \\ \hline Wiki2 & Token Num & 2,088,628 & 217,646 & 245,569 \\  & Vocab Size & & 33,278 & \\ \hline Wikii103 & Token Num & 103,227,021 & 217,646 & 245,569 \\  & Vocab Size & & 267,735 & \\ \hline \hline \end{tabular}
\end{table}
Table 7: Statistics of the language modeling datasets.

Figure 4: Distribution of polarity scores for adjectives and their negation bigrams. _p-adj_ and _n-adj_ refer to the positive and negative adjectives respectively. [-] refers to the negation operation (prepending the word ”not”). Circles refer to outliers.

Figure 5: Context polarity scores for positive adjectives (a) and negative adjectives (b) along with their corresponding negation and intensification bigrams from SST-5.

 

Such visualization results show that our identified representations defined over the linguistic units as captured by RNNs can be highly interpretable.

### First-order Approximation

To examine how well the recurrence relation in Equation 7 can approximate the standard RNNs, we followed the method in the work of Maheswaranathan and Sussillo (2020) and compared the hidden state of the standard RNNs (\({\mathbf{h}_{t}=RNN({\mathbf{x}_{t}},{\mathbf{h}_{t-1}})}\)) at each time step to the corresponding context representations \((\hat{{\mathbf{h}}_{t}}={\mathbf{g}}({\mathbf{x}_{t}})+{\mathbf{A}}({\mathbf{x}_{t}}){\mathbf{h}_{t-1}})\). The error at each time step is defined as

\[||{\mathbf{h}_{t}}-\hat{{\mathbf{h}}_{t}}||_{2}/||{\mathbf{h}_{t}}||_{2}. \tag{15}\]

We used the current standard hidden state to predict the next hidden state and the context representations on the SST-2 test set.

We noticed that the weight decaying coefficient has a remarkable impact on the error. Specifically, a larger coefficient can result in smaller errors. When the coefficient is \(1e-5\), the average errors on the Elman, GRU, and LSTM models were 26.2%, 21.7% and 46.6% and respectively. When the coefficient is \(3e-4\) the the average errors dropped to 17.1%, 15.1%, and 33.3% respectively. Note that since this is the single step error, the accumulated errors across many times steps can be large, particularly for LSTM, and thus the first-order approximation cannot fully replace standard RNNs. Despite this, the resulting context and \(n\)-gram representations can help us understand how RNNs process contextual information such as \(n\)-gram features.

## Appendix C T-sne Visualization

We visualized the context representations from the MVMA-G model using t-sne van der Maaten and Hinton (2008), which provides us with an intuitive understanding on the efficacy of our identified representations. We automatically extracted 2,188 phrases with less than 30 tokens from AG-news with 4 topics17 and projected their context representations to a two-dimension space. Figures 6(a) and 6(b) show there exist four major clusters corresponding to the four topics, indicating those representations can generally learn the topic information and explain the differences. Similar to the previous analysis, the MVM-G model is able to learn the topic information with the \(n\)-gram representations.

Footnote 17: Although SST-5 has 5 tables, most of its phrases are neutral, we therefore did not use this dataset for visualization.

## Appendix D Results on Transformer

We have also run the Transformer model on the sentiment analysis datasets and the results are listed in Table 8.

 



## Appendix E Implementation Details

### Sentiment Analysis

SettingsFor the SST-2, AG-news, and IMDB datasets, we used the cross-entropy as the loss function to train the models. Embeddings were randomly initialized and trainable during training. For the SST-5 dataset, we treated the classification as a regression problem as the labels are polarity intensity. The mean-squared error was used as the loss function during training. Note that we initialized embeddings with pre-trained GloVe (Pennington et al., 2014) and fixed them during training on SST-5 for the analysis of both the negation and intensification phenomena.

Furthermore, for the MM model, each token was represented as a matrix and the matrix size was set as 32\(\times\)32. For the other models, the embedding and hidden sizes were both set as 300.

Polarity AdjectivesWe automatically extracted adjectives with polarity (examples shown in Table 9) from SST-2 in two steps. In the first step, following the method of Sun and Lu (2020), we calculated a frequency ratio for each token (in the vocabulary) between the frequencies of the token seen in the positive and negative instances respectively. If a token has a frequency ratio either larger than 3 or less than 1/3, it will be extracted as an _positive token_ or an _negative token_. In the second step, we used the _textblob_ package 18 to detect positive and negative adjectives from those positive tokens and negative tokens respectively.

Footnote 18: [https://textblob.readthedocs.io/en/dev/](https://textblob.readthedocs.io/en/dev/)

### Relation Classification

Following the work of Gupta and Schutze (2018), we examined the RNN, baseline, MVMA and MVM models on SemEval 2010 Task 8 (Hendrickx et al., 2010) which has 9 directed relationships and an undirected _other_ type. We used the final hidden states of the standard RNNs (or context representations of the MVMA, MVM and baseline models) as the instance representations for classification. The cross-entropy loss was employed during training.

### Named Entity Recognition

At each time step, we concatenated the context representations (or hidden states) from both directions in a bidirectional model, fed them to a projection layer and then to a linear CRF layer. More details about the architecture can be referred to the biLSTM-CRF model in the work of Lample et al. (2016). We also referred to the code at [https://github.com/allanj/pytorch_neural_crf](https://github.com/allanj/pytorch_neural_crf) for the implementation of the linear CRF layer.

CoNLL-2003 contains four types of entities: persons (PER), organizations (ORG), locations (LOC) and miscellaneous names (MISC). The original dataset was labeled with the BIO (Beginnings-Inside-Outside) format. For example, "United Arab Emirates" are labeled as "B-LOC I-LOC I-LOC". We transform the tags into the IOES format where two prefixes "E-" and "S-" are added. Specifically, "E-" is used to label the last token of an entity span. The "S-" prefix is used for a single-token span. For example, "United Arab Emirates" are labeled as "B-LOC I-LOC E-LOC" in this format. There are 20 categories of tags in total including the starting, ending and padding tags. We trained the models to predict each entity.

The embedding size and hidden size were set to 300 and 200 respectively. The SGD optimizer was used to learn parameters.

### Language Modeling

The embedding size and hidden size were both 512 for PTB and Wiki2, and 256 and 512 respectively for Wiki103. The cross-entropy loss was used during training. For PTB and Wiki2, the output of the final fully-connected layer was fed to a _softmax_ function while the Adaptive softmax (Joulin et al., 2017) was used for Wiki103 (because of its large vocabulary size). We only considered the word-level models. We trained each model for 50 epochs, chose the model which had the best performance on the development set as the final model and evaluated the final model on the test set.

## Appendix F Jacobian matrix of LSTM

Unlike GRU and Elman RNN, LSTM has a memory cell apart from a hidden state. Here, we describe how to get their Jacobian matrices. An

\begin{table}
\begin{tabular}{l c c c c c} \hline  & **SST-2** & **AG-news** & **IMDB** \\ \hline dev & text & dev & test & dev & test \\
83.4\(\pm\)0.4 & 82.0\(\pm\)0.1 & 90.9\(\pm\)0.5 & 90.5\(\pm\)0.4 & 88.4\(\pm\)0.2 & 88.1\(\pm\)0.2 \\ \hline \end{tabular}
\end{table}
Table 8: Accuracy on sentiment analysis tasks. Transformer

LSTM cell can be written as

\[{\mathbf{i}_{t}=\sigma(\mathbf{W}_{ii}\mathbf{x}_{t}+\mathbf{W}_{hi}\mathbf{h}_{t-1}),} \tag{16}\] \[{\mathbf{f}_{t}=\sigma(\mathbf{W}_{if}\mathbf{x}_{t}+\mathbf{W}_{hf}\mathbf{h}_{t-1}),}\] \[{\mathbf{o}_{t}=\sigma(\mathbf{W}_{io}\mathbf{x}_{t}+\mathbf{W}_{ho}\mathbf{h}_{t-1}),}\] \[{\mathbf{c}^{m}_{t}=\tanh(\mathbf{W}_{ic}\mathbf{x}_{t}+\mathbf{W}_{hc}\mathbf{h}_{t-1 }),}\] \[{\mathbf{c}_{t}=\!\mathbf{f}_{t}\odot\mathbf{c}_{t-1}\!+\!{\mathbf{i}_{t}}\odot\bm {c}^{m}_{t},\mathbf{h}_{t}\!=\!\mathbf{o}_{t}\odot\tanh(\mathbf{c}_{t}),}\]

where \({\mathbf{i}_{t}}\), \({\mathbf{f}_{t}}\), \({\mathbf{o}_{t}\in\mathbb{R}^{d}}\) are the input gate, forget gate and output gate respectively. \(\mathbf{c}^{m}_{t}\in\mathbb{R}^{d}\) is the new memory, and \(\mathbf{c}_{t}\) is the final memory.

Let us expand the memory state and hidden state at time step \(t\) as

\[\begin{split}&\mathbf{c}_{t}=\mathbf{g}_{c}(\mathbf{x}_{t})+B(\mathbf{x}_{t}) \mathbf{c}_{t-1}\\ &+D(\mathbf{x}_{t})\mathbf{h}_{t-1}+\mathbf{o}_{c}(\mathbf{c}_{t-1},\mathbf{h}_{t-1}) ,\\ &\mathbf{h}_{t}=\mathbf{g}_{h}(\mathbf{x}_{t})+E(\mathbf{x}_{t})\mathbf{c}_{t-1}\\ &+F(\mathbf{x}_{t})\mathbf{h}_{t-1}+\mathbf{o}_{h}(\mathbf{c}_{t-1},\mathbf{h}_{t-1}) ,\end{split} \tag{17}\]

where \(B\), \(D\), \(E\) and \(F\in\mathbb{R}^{d\times d}\) are all Jacobian matrices. \(\mathbf{o}_{c}(\mathbf{h}_{t-1})\) and \(\mathbf{o}_{h}(\mathbf{h}_{t-1})\) are remainder terms of the Taylor expansion.

We concatenate the memory state and hidden state and view the concatenation as an "extended hidden state". The context representation for the "extended hidden state" at time step \(t\) (assuming of zero vectors as initial states) will be written as:

\[\left[\hat{\mathbf{c}_{t}}\atop{\hat{\mathbf{h}}_{t}}\right]\!=\!\sum_{i=1}^{t}\left[ \boldsymbol{v}^{c}_{ix}\atop{\mathbf{v}^{h}_{ix}}\right]\!=\!\sum_{i=1}^{t}\left[ \prod_{k=t}^{i+1}A(x_{k})\right]\left[g_{c}(\mathbf{x}_{i})\atop g_{h}(\mathbf{x}_{ i})\right], \tag{18}\]

where \(\hat{\mathbf{c}}_{t}\) and \(\hat{\mathbf{h}}_{t}\) refer to the context representations corresponding to the memory state and hidden state respectively. \(g_{c}\), \(g_{h}\in\mathbb{R}^{d}\), and \(A\in\mathbb{R}^{2d\times 2d}\) are all functions of inputs. \(A(x_{k})\) contains many interaction terms resulting from the gating mechanism, which may result in a strong expressive power. As the hidden state \(\mathbf{h}_{t}\) is commonly used for downstream tasks, we will only consider \(\mathbf{v}^{h}_{ict}\) as the \(n\)-gram representation on our tasks, and the context representation will be \(\sum_{i=1}^{t}\mathbf{v}^{h}_{i:t}\).

## Appendix B

\begin{table}
\begin{tabular}{l l} \hline
**Type** & **Size** \\ \hline Pos & outstanding, ecological, inventive, comfortable, nice, authentic, spontaneous, sympathetic, lowable, \\  & unadulterated, controversial, suitable, grand, happy, enthusaistic, adventurous, successful, noble, \\  & true, detailed, sophisticated, sensational, exotic, fantastic, remarkable, impressive, charismatic, \\  & good, effective, rich, popular, unforgettable, famous, comical, energetic, ingenious, extraordinary, ... \\ \hline Neg & bad, tedious, miserable, psychotic, didactic, inexplicable, feeble, sloppy, disastrous, stupid, \\  & amateurish, false, cynical, faracil, terrible, unhappy, horrible, atroccious, idiotic, wrong, pathetic, \\  & angry, uninspired, vicious, unfocused, unnecessary, artificial, troubled, questionable, arduous, \\  & stereotypical, ... \\ \hline \hline \end{tabular}
\end{table}
Table 9: Examples of the extracted adjectives from the SST-2 dataset. “Pos” refers to _positive_ adjectives and “Neg” refers to _negative_ adjectives.