SlowGuess/NAACL_2022_OCR huggingface dataset:
Implicit  $N$ -grams Induced by Recurrence
Although self-attention based models such as Transformers have achieved remarkable successes on natural language processing (NLP) tasks, recent studies reveal that they have limitations on modeling sequential transformations (Hahn, 2020), which may prompt re-examinations of recurrent neural networks (RNNs) that demonstrated impressive results on handling sequential data. Despite many prior attempts to interpret RNNs, their internal mechanisms have not been fully understood, and the question on how exactly they capture sequential features remains largely unclear. In this work, we present a study that shows there actually exist some explainable components that reside within the hidden states, which are reminiscent of the classical  $n$ -grams features. We evaluated such extracted explainable features from trained RNNs on downstream sentiment analysis tasks and found they could be used to model interesting linguistic phenomena such as negation and intensification. Furthermore, we examined the efficacy of using such  $n$ -gram components alone as encoders on tasks such as sentiment analysis and language modeling, revealing they could be playing important roles in contributing to the overall performance of RNNs. We hope our findings could add interpretability to RNN architectures, and also provide inspirations for proposing new architectures for sequential data.
Modern recurrent neural networks (RNNs), including Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014), have demonstrated impressive results on tasks involving sequential data. They have proven to be capable of modeling formal languages (Weiss et al., 2018; Merrill, 2019; Merrill et al., 2020) and capturing structural features (Li et al., 2015a,b, 2016; Linzen et al., 2016; Belinkov et al., 2017; Liu et al., 2019) on NLP tasks. Although Transformers (Vaswani et al., 2017) have achieved remarkable performances on
Figure 1: An RNN hidden state may encode a linear combination of all the  $n$ -grams ending at the current time step.
NLP tasks such as machine translation, it is argued that they may have limitations on modeling hierarchical structure (Tran et al., 2018; Hahn, 2020) and cannot handle functions requiring sequential processing of input well (Dehghani et al., 2019; Hao et al., 2019; Bhattamishra et al., 2020; Yao et al., 2021). Furthermore, a recent work shows that combining recurrence and attention (Lei, 2021) can result in strong modeling capacity. Another recent work incorporating recurrent cells into Transformers (Hutchins et al., 2022) substantially improved performance on language modeling involving very long sequences, prompting re-investigations of RNNs. On the other hand, it was observed in prior work that RNNs were able to capture linguistic phenomena such as negation and intensification (Li et al., 2016), but the question why they could achieve so still largely remains unanswered.
In this work, we focus on better understanding RNNs from a more theoretical perspective. We demonstrate that the recurrence mechanism of RNNs may induce a linear combination of interpretable components. These components reside in their hidden states in the form of the iterated matrix-vector multiplication that is based on the representations of tokens in the (reverse) order they appear in the sequence. Such components, solely depending on inputs and learned parameters, can be conveniently interpreted and are reminiscent of those compositional features used in classical  $n$ -gram models (Jurafsky and Martin, 2009). They may also provide us with insights on how RNNs compose semantics from basic linguistic units. Our analysis further shows that, the hidden state at each time step includes a weighted combination of components that represent all the " $n$ -grams" ending at that specific position in the sequence as shown
Human Language Technologies, pages 1624 - 1639
in Figure 1. We gave specific representations for the  $n$ -gram components in Elman RNNs (Elman, 1990), GRUs and LSTMs.
We investigated the interpretability of those  $n$ -gram components on trained RNN models, and found they could explain phenomena such as negation and intensification and reflect the overall polarity on downstream sentiment analysis tasks, where such linguistic phenomena are prevalent. Our experiments also revealed that the GRU and LSTM models are able to yield better capabilities in modeling such linguistic phenomena than the Elman RNN model, partly attributed to the gating mechanisms they employed which resulted in more expressive  $n$ -gram components. We further show that the linear combination of such components yields effective context representations. We explored the effectiveness of such  $n$ -gram components (along with the corresponding context representations) as alternatives to standard RNNs, and found they can generally yield better results than the baseline compositional methods on several tasks, including sentiment analysis, relation classification, named entity recognition, and language modeling.
We hope that our work could give inspirations to our community, serving as a useful step towards proposing new architectures for capturing contextual information within sequences.
Interpretability of RNNs: A line of work focuses on the relationship between RNNs and finite-state machines (Weiss et al., 2018; Merrill, 2019; Suzgun et al., 2019; Merrill et al., 2020; Eyraud and Ayache, 2020; Rabusseau et al., 2019), providing explanation and prediction on the expressive power and limitations of RNNs on formal languages both empirically and theoretically. Kanai et al. (2017) investigated conditions that could prevent gradient explosions for GRU based on dynamics. Maheswaranathan et al. (2019) and Maheswaranathan and Sussillo (2020) linearized the dynamics of RNNs around fixed points of hidden states and elucidated contextual processing. Our work focuses on studying a possible mechanism of RNNs that handles exact linguistic features.
Another line of work aims to detect linguistic features captured by RNNs. Visualization approaches (Karpathy et al., 2015; Li et al., 2016) were initially used to examine compositional information in RNN outputs. Linzen et al. (2016) assessed LSTMs' ability to learn syntactic structure and
Emami et al. (2021) gave rigorous explanations on the standard RNNs' ability to capture long-range dependencies. Decomposition methods (Murdoch and Szlam, 2017; Murdoch et al., 2018; Singh et al., 2019; Arras et al., 2017, 2019; Chen et al., 2020) were proposed to produce importance scores for hierarchical interactions in RNN outputs. Our work can be viewed as an investigation on how those interaction came about.
Compositional Models: A variety of compositional functions based on vector spaces have been proposed in the literature to compose semantic meanings of phrases, including simple compositions of adjective-noun phrases represented as matrix-vector multiplication (Mitchell and Lapata, 2008; Baroni and Zamparelli, 2010) and a matrix-space model (Rudolph and Giesbrecht, 2010; Yessenalina and Cardie, 2011) based on matrix multiplication. Socher et al. (2012, 2013) introduced a recursive neural network model that assigns every word and longer phrase in a parse tree both a vector and a matrix, and represents composition of a non-terminal node with matrix-vector multiplication. Kalchbrenner and Blunsom (2013) employed convolutional and recurrent neural networks to model compositionality at the sentence and discourse levels respectively. Those models are designed in an intuitive manner based on the nature of languages thus being interpretable. We can show that RNNs may process contextual information in a way bearing a resemblance to those early models.
3 A Theory on  $N$ -gram Representation
First, let us spend some time to discuss how to represent  $n$ -grams. Various approaches to representing  $n$ -grams have been proposed in the literature (Mitchell and Lapata, 2008; Bengio et al., 2003; Mitchell and Lapata, 2008; Mnih and Teh, 2012; Ganguli et al., 2008; Orhan and Pitkow, 2020; Emami et al., 2021; Rudolph and Giesbrecht, 2010; Yessenalina and Cardie, 2011; Baroni and Zamparelli, 2010). We summarize in Table 1 different approaches for representing  $n$ -grams.
Although empirically it has been shown that different approaches can lead to different levels of effectiveness, the rationales underlying many of the design choices remain unclear. In this section, we establish a small theory on representing  $n$ -grams, which leads to a new formulation on capturing the semantic information within  $n$ -grams.
Let us assume we have a vocabulary  $\mathbb{V}$  that consists of all possible word tokens. The set of  $n$ -grams can be denoted as  $\mathbb{V}^*$  (including the special
1Our code is available at https://github.com/richardsun-voyager/inibr.
Model	N-gram Representation	Context Representation	L	Representative Work
Vector Multiplicative (VM)	vi:j=g(xi) ⊙···g(xj)	v1:t	t	Mitchell and Lapata (2008)
Matrix Multiplicative (MM)	Mi:j=Πk=iA(xk)	M1:t	t	Yessenalina and Cardie (2011)
Vector Additive (weighted) (VA-W)	vi:j=Cj-i g(xi)	∑i=t-m+1tvi:t	m	Bengio et al. (2003)
Vector Additive (exponentially weighted) (VA-EW)	vi:j=Cj-i g(xi)	∑i=1tvi:t	t	Emami et al. (2021)
Matrix-Vector Multiplicative (restricted) (MVM-R)	vi-1:i=A(xi-1)g(xi)	vt-1:t	2	Baroni and Zamparelli (2010)
Matrix-Vector Multiplicative (MVM)	vi:j=(Πk=j i+1 A(xk)) g(xi)	v1:t	t	-
Matrix-Vector Multiplicative-Additive (MVMA)	vi:j=(Πk=j i+1 A(xk)) g(xi)	∑i=1tvi:t	t	This work
Table 1: Different models for defining representations for  $n$ -grams within the phrase  $x_{1},x_{2},\ldots ,x_{t - 1},x_{t}$  and constructing the context representation out of the  $n$ -grams during learning.  $L$ : the maximum length allowed for the context representation.  $C$  is a weight matrix, and  $C_k$  is a (relative) position-specific weight matrix.  $A$  and  $g$  are functions that return a matrix and a vector respectively.
$n$ -gram which is the empty string  $\epsilon$ ). Consider three  $n$ -grams  $a, b$ , and  $c$  from  $\mathbb{V}^*$ , with their semantic representations  $r(a), r(b)$ , and  $r(c)$  respectively. Similarly, we may have  $r(ab)$  which return the semantic representations of the concatenated  $n$ -grams  $ab$ . It is desirable for our representations to be compositional in some sense. Specifically, a longer  $n$ -gram may be semantically related to those shorter  $n$ -grams it contains in some way.
Under some mild compositional assumptions related to the principle of compositionality (Frege, 1948) $^2$ , it is reasonable to expect that there exists some sort of rule or operation that allows us to compose semantics of longer  $n$ -grams out of shorter ones. Let us use  $\otimes$  to denote such an operation. We believe a good representation system for  $n$ -grams shall satisfy several key properties. First, the semantics of the  $n$ -gram  $abc$  shall be determined through either combining the semantics of the two  $n$ -grams  $a$  and  $bc$  or through combining the semantics of  $ab$  and  $c$ . The semantics of  $abc$  is unique, regardless of which of these two ways we use. Second, for the empty string  $\epsilon$ , it should not convey any semantics. Formally, we can write them as: $^3$
- Associativity:  $\forall a, b, c \in \mathbb{V}^*$ ,  $(r(a) \otimes r(b)) \otimes r(c) = r(a) \otimes (r(b) \otimes r(c))$
- Identity:  $\forall a \in \mathbb{V}^*$ ,  $r(a) \otimes r(\epsilon) = r(a)$ , and  $r(\epsilon) \otimes r(a) = r(a)$
This essentially shows that the representation space for all  $n$ -grams under the operation  $\otimes$ , denoted as  $(\mathbb{V}^*, \otimes)$ , forms a monoid, an important concept in abstract algebra (Lallement, 1979), with significance in theoretical computer science
(Meseguer and Montanari, 1990; Rozenberg and Salomaa, 2012).
On the other hand, it can be easily verified that the space of all  $d \times d$  (where  $d$  is an integer) real square matrices under matrix multiplication, denoted as  $(\mathbb{R}^{d \times d}, \cdot)$ , also strictly forms a monoid (i.e., it is associative and has an identity, but is not commutative). We can therefore establish a homomorphism from  $\mathbb{V}^*$  to  $\mathbb{R}^{d \times d}$ , resulting in the function  $r(\cdot) \in \mathbb{V}^* \to \mathbb{R}^{d \times d}$ .
This essentially means that we may be able to rely on a sub-space within  $\mathbb{R}^{d\times d}$  as our mathematical object to represent the space of  $n$ -grams, where the matrix multiplication operation can be used to compose representations for longer  $n$ -grams from shorter ones. Thus, for a unigram  $x$  (a single word in the vocabulary), we have:
$$
r (x) := \boldsymbol {A} _ {x} \tag {1}
$$
where  $A_{x}\in \mathbb{R}^{d\times d}$  is the representation for the word  $x$  (how to learn such a matrix is a separate question to be discussed later). Note that the empty string  $\epsilon$  comes with a unique representation which is the  $d\times d$  identity matrix  $\pmb{I}$
We can either use matrix left-multiplication or right-multiplication as our operator  $\otimes$ . Assume the language under consideration employs the left-to-right writing system. It is reasonable to believe that a human reader processes the text left-to-right, and the semantics of the text gets evolved each time the reader sees a new word. We may use the matrix left-multiplication as the preferred operator in this case. The system will left-multiply (modify) an existing  $n$ -gram representation with a matrix associated with the new word that appears right after the existing  $n$ -gram, forming the representation of the new  $n$ -gram. Such an operation essentially performs a transform that simulates the process of yielding new semantics when appending a new word at the end of an existing phrase. With this, for
2The principle states that "the meaning of an expression is determined by the meanings of the sub-expressions it contains and the rules used to combine such sub-expressions".
3 Besides, another important property is that the order used for combining two  $n$ -grams does matter. In other words,  $r(a) \otimes r(b)$  usually may not be the same as  $r(b) \otimes r(a)$ .
a general  $n$ -gram  $x_{i}, x_{i+1}, \ldots, x_{t} (i \leq t)$ , we have:
$$
r \left(x _ {i}, x _ {i + 1}, \dots , x _ {t}\right) = \prod_ {k = t} ^ {i} \boldsymbol {A} _ {x _ {k}} \tag {2}
$$
However, the conventional wisdom in NLP has been to use vectors to represent basic linguistic units such as words, phrases or sentences (Mikolov et al., 2013a,b; Pennington et al., 2014; Kiros et al., 2015). This can be achieved by a transform:
$$
\left(\prod_ {k = t} ^ {i} \boldsymbol {A} _ {x _ {k}}\right) \boldsymbol {u} \tag {3}
$$
where  $\pmb{u} \in \mathbb{R}^d$  is a vector that maps the resulting matrix representation into a vector representation.
Next, we will embark on our journey to examine the internal representations of RNNs. As we will see, interestingly, our developed  $n$ -gram representations can emerge within such models.
4 Interpretable Components in RNNs
An RNN is a parameterized function whose hidden state can be written recursively as:
$$
\boldsymbol {h} _ {t} = f \left(x _ {t}, \boldsymbol {h} _ {t - 1}\right), \tag {4}
$$
where  $x_{t}$  is the input token at time step  $t$  and  $\pmb{h}_{t - 1}\in \mathbb{R}^d$  is the previous hidden state. Assume  $f$  is differentiable at any point, with the Taylor expansion,  $\pmb{h}_t$  can be rewritten as:
$$
\boldsymbol {h} _ {t} = f \left(x _ {t}, \mathbf {0}\right) + \nabla f \left(x _ {t}, \mathbf {0}\right) \boldsymbol {h} _ {t - 1} + o \left(\boldsymbol {h} _ {t - 1}\right), \tag {5}
$$
where  $\nabla f(x_{t},\mathbf{0}) = \frac{\partial f}{\partial h_{t - 1}} |_{h_{t - 1} = \mathbf{0}}$  is the Jacobian matrix, and  $o$  is the remainder of the Taylor series.
Let  $g(x_{t}) = f(x_{t},\mathbf{0})$  and  $A(x_{t}) = \nabla f(x_{t},\mathbf{0})$ . Note that  $g(x_{t})\in \mathbb{R}^{d}$  and  $A(x_{t})\in \mathbb{R}^{d\times d}$  are both functions of  $x_{t}$ . Therefore, the equation above can be written as:
$$
\boldsymbol {h} _ {t} = g \left(x _ {t}\right) + A \left(x _ {t}\right) \boldsymbol {h} _ {t - 1} + o \left(\boldsymbol {h} _ {t - 1}\right). \tag {6}
$$
If the hidden state has a sufficiently small norm, it can be approximated by the first-order Taylor expansion as follows $^4$ :
$$
\boldsymbol {h} _ {t} \approx g \left(x _ {t}\right) + A \left(x _ {t}\right) \boldsymbol {h} _ {t - 1}. \tag {7}
$$
Next we illustrate how this recurrence relation can help us identify some salient components.
4.1 Emergence of  $N$ -grams
Consider the simplified RNN with the following recurrence relation,
$$
\boldsymbol {h} _ {t} = g \left(x _ {t}\right) + A \left(x _ {t}\right) \boldsymbol {h} _ {t - 1}, \tag {8}
$$
where  $\pmb{h} \in \mathbb{R}^d$ , and  $g(x_{t}) \in \mathbb{R}^{d}$  and  $A(x_{t}) \in \mathbb{R}^{d\times d}$  are functions of  $x_{t}$ . This recurrence relation can be expanded repeatedly as follows,
$$
\begin{array}{l} \boldsymbol {h} _ {t} = g \left(x _ {t}\right) + A \left(x _ {t}\right) g \left(x _ {t - 1}\right) + A \left(x _ {t}\right) A \left(x _ {t - 1}\right) \boldsymbol {h} _ {t - 2} \\ = \dots = \sum_ {i = 1} ^ {t} A \left(x _ {t}\right) \dots A \left(x _ {i + 1}\right) g \left(x _ {i}\right) \\ = \sum_ {i = 1} ^ {t} \underbrace {\left(\prod_ {j = t} ^ {i + 1} A (x _ {j})\right) g (x _ {i})} _ {\boldsymbol {v} _ {i: t}}, \\ \end{array}
$$
We can see that  $\pmb{v}_{i:t}$  bear some resemblance to the term in Equation 3, which can be rewritten as:
$$
\left(\prod_ {j = t} ^ {i + 1} \underbrace {\boldsymbol {A} _ {x _ {j}}} _ {A (x _ {j})}\right) \underbrace {\left(\boldsymbol {A} _ {x _ {i}} \boldsymbol {u}\right)} _ {g (x _ {i})}, \tag {9}
$$
With the definition  $A(x_{j}) \coloneqq A_{x_{j}}$  and  $g(x_{i}) \coloneqq A(x_{i})u$ , we can see  $v_{i:t}$  can be interpreted as an "  $n$ -gram representation" that we developed in the previous section. It is important to note that, however, the use of function  $g(x_{i})$  in RNNs may lead to greater expressive power than the original formulation based on  $A_{x_{i}}u$ .<sup>5</sup>
This interesting result shows that the hidden state of a simple RNN (characterized by Equation 8) is the sum of the representations of all the  $n$ -grams ending at time step  $t$ . Such salient components within RNN also show that the standard RNN may actually have a mechanism that is able to capture implicit  $n$ -gram information as described above. This leads to the following definition:
Definition 1 (N-gram Representation) For the  $n$ -gram  $x_{i}, x_{i+1}, \ldots, x_{t}$ , its representation is:
$$
\boldsymbol {v} _ {i: t} = \left(\prod_ {j = t} ^ {i + 1} A \left(x _ {j}\right)\right) g \left(x _ {i}\right), \tag {10}
$$
where  $A(x_{j})\in \mathbb{R}^{d\times d}$  and  $g(x_{i})\in \mathbb{R}^{d}$
4.2 Context Representation
With the above definition, we may want to consider how to perform learning. The learning task involves identifying the functions  $A$  and  $g$  - in other words, learning representations for word tokens.
A typical learning setup that we may consider here is the task of language modeling. Such a task
There will be an "approximation gap" at each time step between the "approximated" hidden state and the actual standard hidden state. We may leverage regularization methods such as weight-decaying and the spectral normalization (Miyato et al., 2018) to prevent the gap from growing unbounded.
This is because we can always construct  $g(x_{i})$  from any given  $A_{x_i}$  and  $\pmb{u}$ , but in general we may not always be able to decompose  $g(x_{i})$  into the form  $A_{x_i}\pmb{u}$  (for all  $x_{i}$ ).
can be defined as predicting the next word  $x_{t+1}$  based on the representation of preceding words  $x_1, x_2, \ldots, x_t$  which serves as its left context. This is an unsupervised learning task, where the underlying assumption involved is the distributional hypothesis (Harris, 1954). Specifically, the model learns how to "reconstruct" the current word  $x_{t+1}$  out of  $x_1, x_2, \ldots, x_t$  which serves as its context.
Now the research question is how to define the representation for this specific context. As this left context is also an  $n$ -gram, it might be tempting to directly use its  $n$ -gram representation defined above to characterize such a left context. However, we show such an approach is not desirable.
The  $n$ -gram representation for this context can be written in the following alternative form:
$$
\boldsymbol {v} _ {1: t} = \left(\prod_ {j = t} ^ {2} A \left(x _ {j}\right)\right) g \left(x _ {i}\right) = W \left(x _ {2: t}\right) g \left(x _ {1}\right), \tag {11}
$$
This shows that the  $n$ -gram representation of  $x_{1}, x_{2}, \ldots, x_{t}$  could be interpreted as a "weighted" representation of the word  $x_{1}$  (where the weight matrix is derived from the words between  $x_{1}$  and  $x_{t+1}$ , measuring the strength of the connection between them). However, ideally, the context representation shall not just take  $x_{1}$  but other adjacent words preceding  $x_{t+1}$  into account, where each word contributes towards the final context representation based on the connection between them. This leads to the following way of defining the context:
$$
\begin{array}{l} \sum_ {i = 1} ^ {t} \boldsymbol {v} _ {i: t} = \sum_ {i = 1} ^ {t} \left(\prod_ {j = t} ^ {i + 1} A \left(x _ {j}\right)\right) g \left(x _ {i}\right) \tag {12} \\ = \sum_ {i = 1} ^ {t} W (x _ {i: t}) g (x _ {i}), \\ \end{array}
$$
In fact, such an idea of defining the context as a weighted combination of surrounding words is not new – it recurs in the literature of language modeling (Bengio et al., 2003; Mnih and Teh, 2012), word embedding learning (Mikolov et al., 2013a,b), and graph representation learning (Cao et al., 2016).
Interestingly, the hidden states in the RNNs, as shown in Equation 9, also suggest exactly the same way of defining this left context. Indeed, when using RNNs for language modeling, each hidden state is exactly serving as the context representation for predicting the next word in the sequence.
The above gives rise to the following definition: Definition 2 (Context Representation) For the  $n$ -gram  $x_{1}, x_{2}, \ldots, x_{t}$ , its representation when serving as the (left) context is:
$$
\boldsymbol {c} _ {1: t} = \sum_ {i = 1} ^ {t} \boldsymbol {v} _ {i: t} = \sum_ {i = 1} ^ {t} \left(\prod_ {j = t} ^ {i + 1} A \left(x _ {j}\right)\right) g \left(x _ {i}\right), \tag {13}
$$
4.3 Model Parameterization
With the above understandings on such salient components within RNNs, we can now look into how different variants of RNNs parameterize the functions  $A$  and  $g$ . The definition of Elman RNN, GRU and LSTM together with the corresponding Jacobian matrix  $A(x_{t})$  and vector function  $g(x_{t})$  functions are listed in Table 2<sup>6</sup>. We discuss how such different parameterizations may lead to different expressive power when they are used in practice.
We can see the ways GRU or LSTM parameterize  $A(x_{t})$  and  $g(x_{t})$  appear to be more complex compared to Elman RNN. This can partially be attributed to their gating mechanisms. Although the original main motivation of introducing such mechanisms may be to alleviate the exploding gradient and vanishing gradient issues (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), we could see such designs also result in terms describing gates and intermediate representations.  $A$  and  $g$  are then independently derived based on certain rich interactions between such terms. We believe such interactions may likely increase the expressive power of the resulting  $n$ -gram representations. We will validate these points and discuss more in our experiments.
In our experiments, we focus on the following aspects: 1) understanding the effectiveness of the proposed  $n$ -gram (and context) representations when used in practice, as compared to baseline models; 2) examining the significance of the choice of context representation; 3) interpreting the proposed representations by examining how well they could be used to capture certain linguistic phenomena.