SEQUENCE-LEVEL FEATURES: HOW GRU AND LSTM CELLS CAPTURE N-GRAMS
Authors: Xiaobing Sun
Wei Lu
Venue:  (2021)
Keywords: GRU
LSTM
Sequence-level
Features
N-grams

Abstract: Modern recurrent neural networks (RNN) such as Gated Recurrent Units (GRU) and Long Short-term Memory (LSTM) have demonstrated impressive results on tasks involving sequential data in practice. Despite continuous efforts on interpreting their behaviors, the exact mechanism underlying their successes in capturing sequence-level information have not been thoroughly understood. In this work, we present a study on understanding the essential features captured by GRU/LSTM cells by mathematically expanding and unrolling the hidden states. Based on the expanded and unrolled hidden states, we find there was a type of sequence-level representations brought in by the gating mechanism, which enables the cells to encode sequence-level features along with token-level features. Specifically, we show that the cells would consist of such sequence-level features similar to those of N-grams. Based on such a finding, we also found that replacing the hidden states of the standard cells with N-gram representations does not necessarily degrade performance on the sentiment analysis and language modeling tasks, indicating such features may play a significant role for GRU/LSTM cells.

Under review as a conference paper at ICLR 2021
SEQUENCE-LEVEL FEATURES: How GRU AND LSTM CELLS CAPTURE -GRAM
Anonymous authors Paper under double-blind review
ABSTRACT
Modern recurrent neural networks (RNN) such Gated Recurrent Units (GRU) and Long Short-term Memory (LSTM) have demonstrated impressive results on tasks involving sequential data in practice. Despite continuous efforts on interpret- ing their behaviors, the exact mechanism underlying their successes in capturing sequence-level information have not been thoroughly understood. In this work, we present a study on understanding the essential features captured by GRU/LSTM cells by mathematically expanding and unrolling the hidden states. Based the expanded and unrolled hidden states, we find there was a type of sequence-level representations brought in by the gating mechanism, which enables the cells to encode sequence-level features along with token-level features. Specifically, we show that the cells would consist of such sequence-level features similar to those of N N-grams. Based on such finding, we also found that replacing the hidden states of the standard cells with --gram representations does necessarily de- grade performance on the sentiment analysis and language modeling tasks, indi- cating such features may play a significant role for GRU/LSTM cells.
INTRODUCTION
Long Short-term Memory (LSTM) (Hochreiter Schmidhuber 1997) and Gated Recurrent Unit (GRU) (Chung et al., 2014) are widely used and investigated for tasks that involve sequential data. They are generally believed to be capable of capturing long-range dependencies while being able to alleviate gradient vanishing or explosion issues (Hochreiter & Schmidhuber) 1997, Karpathy et al.) 2015 Sutskever al. 2014). While such models were empirically shown to be successful across range of tasks, certain fundamental questions such as "what essential features are GRU or LSTM cells able to capture?" have not yet been fully addressed. Lacking answers to them may limit our ability in designing better architectures.
One obstacle can attributed to the non-linear activations used the cells that prevent us from obtaining explicit closed-form expressions for hidden states. A possible solution is to expand the non-linear functions using the Taylor series {Arfken & Mullin 1985) and represent hidden states with explicit input terms. Literally, each hidden state can be viewed as the combination of con- stituent terms capturing features of different levels of complexity. However, there is a prohibitively large number of polynomial terms involved and they can be difficult to manage. But it is possible that certain terms are more significant than others. Through a series of mathematical transforma- tion, we found there were sequence-level representations in form of matrix-vector multiplications among the expanded and unrolled hidden states of the GRU/LSTM cell. Such representations could represent sequence-level features that could theoretically be sensitive the order of tokens and able to differ from the token-level features of its tokens as well as the sequence-level features of its sub-sequences, thus making it able to represent N -rgamss
We assessed the significance of such sequence-level representations on sentiment analysis and lan- guage modeling tasks. We observed that the sequence-leve representations derived from a GRU or LSTM cell were able to reflect desired properties on sentiment analysis tasks. Furthermore, in both the sentiment analysis and language modeling tasks, we replaced the GRU or LSTM cell with cor- responding sequence-level representations (along with token-level representations) directly during training, and found that such models behaved similarly to the standard GRU or LSTM based models. This indicated that the sequence-level features might be significant for GRU LSTM cells.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
There have been plenty prior works aiming to explain the behaviors of RNNs along with the vari- ants. Early efforts were focused exploring the empirical behaviors of recurrent neural networks (RNNs). Li et al. 2015) proposed a visualization approach to analyze intermediate representations of the LSTM-based models where certain interesting patterns could be observed. However, it might not be easy to extend to models with high-dimension representations. Greff' et al. (2016) explored the performances of LSTM variants on representative tasks such as speech recognition, handwrit- ing recognition, and argued that none of the proposed variants could significantly improve upon the standard L.TTM architecture. KKarpathy et al. (2015) studied the existence of interpretable cells that could capture long-range dependencies such as line lengths, quotes and brackets. However, those works did not involve the internal mechanism of GRUs LSTMs.
Melis et al. (2020) and Krause et al. (011)) found that creating richer interaction between contexts and inputs on top of standard LSTMs could result in improvements. Their efforts actually pointed out the significance of rich interactions between inputs and contexts for LSTMs, but did not study what possible features such interactions could result for good performances, |rras et al. (2017) applied an extended technique Layer-wise Relevance Propagation (LRP) to a bidirectional L.STM for sentiment analysis and produced reliable explanations which words are responsible for attributing sentiment in individual text. Murdoch et al. (2018) leverage contextual decomposition methods to conduct analysis on the interactions of terms for LSTMS, which could produce importance scores for words, phrases and word interactions. A RNN unrolling technique was proposed by Shhrstin-- sky (2018) based on signal processing concepts, transforming the RNN into the "Vanilla LSTM" network through a series of logical arguments, and Kanai al. (017)) discussed the conditions that could prevent gradient explosions by looking into the dynamics of GRUs. Merrill et al. (2020) examined the properties of saturated RNNs and linked the update behaviors to weighted tinite-stat machines. Their ideas gave inspirations to explore internal behaviors of LSTM or GRU cells further. In this work, we sought to explore and study such significant underlying features.
MODEL DEFINITIONS
Vanilla RNN The representation of a vanilla RNN cell can be written as:
Wi,hi-1), where h, E Rdd, T; E Rdd are the hidden state and input at time step respectively, h is the hidden state of the layer time (t or the initial hidden state. W, and Wn. are weight matrices. Bias is suppressed here as well.
h; tanh(W,WWa
(I)
GRU The representation of GRU cell can be written as +
Tt = 5 a; + Wnrth-1),
2, o (W ac Whehe-1),
n1 tanh W in + Tt Whnhu=1),
h (1 - 2,) 3 nu 2; h
where h, E IRd, æ, E Rd are the hidden state and input at time step t respectively, h is the hidden state of the layer at time (t - 1) or the initial hidden state. rt E Rd 2, E RR n, E Rd are the reset, update, and the new gates respectively. W.. refers weight matrix. is the element- wise sigmoid function, and @ is the element-wise Hadamard product.
LSTM The representation of an LSTM cell be written as:
i = o W + Whahe-1), 1);
f. W W.yz, + Washi-ll
= a (W D0 + Whohy-1h,
(3)
d = lanh W v. + Whche-1), 1),
C = f, 3 C- i, O di.
h o, ① tanh(c:),
For brevity, suppressed the bias for both GRU and L.T cells here.
2

---

Under review as a conference paper at ICLR 2021
where he RR, Rd are the hidden state and input at time step respectively, ét fi, O2 R are the input gate, forget gate, output gate respectively. cf E IR is the new memory, C₁ is the final memory. W refers to weight matrix.
UNROLLING RNNS
Using the Taylor series, the activations tanh(z) and (Ix) be expanded (at () as:
tanh = + 0 (|) () )((
2 + + O(x²) (|r) < x))
(4)
In this work, we do not seek to approximate the GRU or LSTM cells precisely, but to explore what features the cells could capture.
4.1 VANILLA RNN
We can expand the vanilla RNN hidden state using the Taylor series
he xx + 1 O(x" + Whhh-1)3,
(S)
where x; = W,wr. Let unroll it as:
h o
Σ
W x (,1, 22,
Bt),
(6)
where C, is the unrolled representation produced by higher-order terms. It can be seen that the vanilla RNN cell can capture the input information at each time step.
4.2 GRU
Let write at W oi Wr, x. W Tr. h h h hー_ W Wr. 11 Plugging Equation in Equation 2 we can expand the hidden state at time step t, then combine like terms with respect the order of h and represent them as:
Whahi-1,
h;
x
2
Seroth-orde
1 n
1
2 h 1 L [of=xx)=0
1
h,-1
x h,11
4 1
h
o x
3
h
16
1 (hi h @ hi hi-1 hh-1++ +
1
oi 3 hi hう
oj
oh11
3 h
16
second aorded-order
16 h 2 h o hi +E(x,, h=:1),
Third-order
where ((v,, 11 refers to the higher-order terms of æt, h; well as their interactions. We will focus on the terms involving zeroth-order and first-order terms of h and explore the features they can possibly result in. Then the hidden state time step can be written as:
h
2 xx
x. ④ xz
1 (x; x; 2 4 xx @ h; +
(8)
h,-1
+ hi
+
4 27 o h
o xx)@@
h;
2
4
8
+ E((2,, hz-1),
3

---

Under review as a conference paper at ICLR 2021
where g' h4-1) refers to the higher-order terms of h plus f h4-1). Note that the Hadamard products can be transformed into matrix-vector multiplications (a @ b = diag(a)b) and can obtain the following:
h2 zi zij о xi
1
1
+ I +
Win + diag(a;) 7 diag( at Whx + diag(a;; o - x; =2a; @ x;)
Wnn
h=-1
2
4
4
+ (xt, h4-1)-
For brevity, let us define two functions of æy;
1
g(.,)) 2 i
④
ai
1 1
1
1
A(x;)
- I + Wnn + diag(a;)) 1 diag(a;)) W diag(æ; o दमं i @ W
2
8
(10)
Both g(x+)) and A(x,) are only functions of æs, Then
rewrite Equation as:
h (mx) + A(x:)h+-1 + (xx,, h.-1).
(11)
Throughout all the previous time steps (assuming the initial state are Os), the hidden state at time step & can be finally unrolled as:
he = g(.))
A A
(a))) A A g(())) c 22, 22, (0,,..,,))), 21:)
ME11a
(12)
=g(xx) +
Mo+1)ag(a;)) + (2, .t.)).
where M(+1)a = I (axx)) E Rdxd the matrix-matris product from time step to 1, (, 22, #2,,.., 2t) are the unrolled representations from the higher-order terms in Equation I1 The function gxx)) solely encodes current input, namely token-level features and thus we call it token- level representation. The matrix vector product ..:- M Mi+n): encodes the tokens starting from time step and ending at time step l. If the matrices are different and not diagonal, any change of the order will result in different product. Therefore, 4;:; able to capture the feature of the token sequence between time step i and in order-sensitive manner. We call it sequence-lever representation. Such representations are calculated sequentially from the left to the right through a sequence of vector/matrix multiplications, leading to features reminiscent of the classical N N-grams commonly used in natural language processing (NLP). Let us use h2 denote the first two terms in Equation 12 as:
h, = -=<() + Σ $ st-
(13)
h can be called as N-gram representations (N 2 1). At time step it is able to encode current token input and all the token sequences starting from time step 21, 1} and ending at time step given an instance. In other words, it linear combination current token-level input feature (can be understood as the unigram feature) and sequence-level features of all the possible N N-grams ending time step l. Bidirectional GRUs would be able to capture sequence-level features from both directions.
we make a comparison with the unrolled vanilla RNN cell as discussed above, we can see that the sequence-level representation
is more expressive than w (i 1, ..., 1) when capturing the sequence level features. Specifically, the sequence level representation in GRU explicitly models interactions among input tokens, while capturing the useful order information conveyed by them. This may also be reason why gating mechanism can bring in improved effectiveness over vanilla RNNs apart from alleviating the gradient vanishing or explosion problems.

---

Under review as a conference paper at ICLR 2021
4.3 LSTM
Let us write x; W. x. Wi 2; Wio कं Wt. Similarly, for an LSTM cell, expand the memory cell and the hidden state in similar way. We also focus on the terms that involve the zeroth-order and first-order ct-1 as well as h and write the final memory cell and hidden state together as:
Ci
(x)) gN (+t))
((as) D(x1) E(vr) F(at)
C--1
(a h4 c1-1) (2s, h4 1,-1,
+
+
(14)
he
ht-1
where:
9x(2))
(x) + 2) O xy ((x;)= = diag (z: + 2).
4
(xx+)
diag x + W he + diag (xr{ Wh::
(15)
(x,)
(x) + 2) @ gz)( E(x;) diag(æ + B(x)),
4
Fxx) 4
diag(x; + (x:)) + (xx W
(at, h C(-1) and (2 ct-1 the higher-order terms.
(x::)) E(x;)
((a)) F(x:))
Let us use the matrix Axx;) E gadxad to denote be written as:
then the equation above can
ct
ge(xz)) gh (xx
c--1 h
(a,, h 11-1, Ct-1)
he
+ A (xx)
+
E% h Cc-1)
(16)
And the final memory cell and hidden state can be unrolled as:
Ct
(x)))) (xx))
[++1
9c 9c(x); gh (xx
(
+ (2,,,,,
(17)
he
where E; #2, ..., Fe) are the unrolled representations from the higher-order terms. The matrix- vector product 2.r E 2d for the token sequence between time step and will be viewed as the sequence-level representation. Similar properties can be inferred. Analogously, we will use é and h, to denote the first two terms in Equation 17 respectively.
5 EXPERIMENTS
We would assess the significance of the sequence-level representations on downstream tasks. For N-grams, negation is a common linguistic phenomenon that negates part all of the meaning of a linguistic unit with negation words or phrases. Particularly in sentiment analysis, the polarity of be negated by adding negation words phrases. This task is good testing ground for us to verify the effectiveness of the learned sequence-level features. Thus, we would like to examine whether the sequence-lever representations could capture the negation information for N- --grams, which is crucial for the sentiment analysis tasks. Language modeling tasks are often used in examining how capable an encoder is when extracting features from texts. We would use them to verify whether the sequence-level representations along with token-level representations could capture sufficient features during training and produce performances on par with standard GRU or LSTM cells. The statistics of the datasets shown in Appendix A.I
certain N-grams
5.1 INTERPRET SEQUENCE-LEVER REPRESENTATIONS
We first trained the model with the standard GRU or LSTM cell with Adagrad optimizers (Duchi et al.) 20 then used the learned parameters to calculate and examine the token-level and sequence-level features on sentiment analysis tasks. Final hidden states were used for classifica- tion. L-2 regularization adopted. There three layers in the model: embedding layer, GRU/LSTM layer,and fully-connected layer with a sigmoid/softmax function for binary/multi-clast sentiment analysis.
5

---

Under review as a conference paper at ICLR 2021
5.1.1 POLARITY SCORE
We would like to use the metric polarity score (Sun & Lu 2020) to help us understand the properties of the token-level features and sequence-level features. We followed the work of Sun & Lu 2020), and defined two types of "polarity scores" to quantify such polarity information, token-level polarity score and sequence-level polarity score. Such scores able capture the degree of association between a token (a sequence) and a specific label. For binary sentiment analysis, each polarity score scalar. For multi-class sentiment analysis, each polarity score is vector corresponding to lahels.
For a GRU cell, the two types of scores will be calculated as:
g (z)), S1t w TT 1s
(18)
For LSTM cell, the sequence-level representation be split into two parts: a..
( Ets & E R.). The polarity scores will be calculated as:
= w (xt); Sit ur Φ 6:
(19)
And the overall polarity at time step be viewed the of the token-level polarity score, sequence-level polarity scores and other polarity scores:
8t w h of
jst
st.
(20)
where is the fully-connected layer weight, of is the token-level polarity score time step and 2 is the sequence-level polarity score for the sequence between time step and I, of is the polarity See
score produced by <(æ11 .2.,.., -... 2.). For binary sentiment analysis, w E Rdd sg E R, & E RR For multi-class sentiment analysis, w E Ba of E R + E IR and k is the label size. The overall polarity scores will be used to make decisions for sentiment analysis.
We examined sequence-leved representations on the binary and 3-class Stanford Sentiment Treebank (SST) dataset with subphrase labels (Socher et al.) 2013) respectively. Final models were selected based on validation performances with embeddings randomly initialized. The embedding size and hidden size were 300 and 1024 respectively.
5.1.2 SEPARATING PHRASES
We extracted all the short labeled phrases (2-5 words, 18490 positive/12199 negative) from the bi- nary training set, and calculated the sum of the token-level polarity score and sequence-level polarity scores for each phrase using the first two terms in Equation 20 We call the sum phrase polar- ity score. Figure [ shows that the two types phrases can be generally separated by the phrase polarity scores. This set of experiments show that the learned sequence-level features capture information useful in making discrimination between positive and negative phrases.
Postivue Phrwee
Posit Phrase Nogative Phrase
Nogative Phrase
0.04
0.05
0.04
0.02
0.01
0.00
0.00
-100
-75
-50
-25
50
75
1.00
-100
-75
-50
-25
75
100
Figure 1: Phrase polarity score distribution for short phrases in binary SST. Left, GRU; right, LSTM.

---

Under review as a conference paper at ICLR 2021
5.1.3 NEGATING ADJECTIVES
We extracted 65 positive adjectives and 42 negative adjective following the criterion in the work Sf & Lu (020) from the vocabulary of the binary SST training set. We calculated the token- level polarity scores for those adjectives and sequence-level polarity scores for their corresponding negation bigrams (adding the negation word "not" and "never"). It can be seen from Table [ that the model could likely learn to infer negations with respect to sequence-level polarity scores: the negation bigrams generally have sequence-level polarity scores of opposite signs to the correspond- ing adjectives. For example, "outstanding" has a large positive token-level polarity score while "not outstanding" has large negative sequence-level polarity score.
Table I: Statistics of token-level polarity scores for positive and negative adjectives and sequence- level polarity scores for negation bigrams. Negation words are "not" and "never". Models trained on the binary SST dataset.
Polarity Score
Cell
Adjective
Remark
mcan
std
word
8.7
4.3
token-level
positive adjectives
negation bigram (not + word)
-46.5
13.9
sequence-level
negation bigram (never word)
-39.3
13.9
sequence-level
LSTM
word
-8.4
3.6
token-level
negative adjectives
negation bigram (not + word)
14.3
11.8
sequence-eeee
negation bigram (never + word)
22.3
11.8
sequence-level
word
8.2
3.8
token-level
positive adjectives
negation higram (not + word)
-33.6
8.7
sequence-level
negation bigram (never + word)
-27.1
7.9
sequence-level
GRU
word
-7.7
3.2
token-level
negative adjectives
negation bigram (not + word)
9.2
9.5
sequence-level
negation bigram (never + word)
11.0
8.0
sequence-level
5.1.4 DISSENTING SUB-PHRASES
We also examined whether the sequence-level representations could play a dissenting role in negat- ing the polarity of sub-phrases. We searched for labeled phrases (3-6 tokens) that start with negation words "not", "never" and "hardly" and have corresponding labeled sub-phrases without negation words (those sub-phrases have opposite labels). For example, the phrase "hardly seems worth the effort" was labeled as "negative" while the sub-phrase "seems worth the effort" was labeled "pos- itive". Based on such conditions, we automatically extracted 14 positive phrases and 36 negative phrases along with their corresponding sub-phrases, then we calculated the polarity scores with pre- trained models. We would like to see if the polarity scores assigned to such linguistic units by our models consistent with the labels.
Table Dissenting Sub- -hrasehss "Polarity score" refers to the sequence-level polarity scores for the longest N-gram in the phrases. ">0" and "<0"rrferss to the number of positive polarity scores and number of negative polarity scores respectively. "S Ph" refers to the number of phrases. "Example" refers to the example phrase (with sequence-level polarity score) for each type of extracted phrases. The sub-phrases have opposite labels to the phrases. Models trained on the binary SST dataset.
Polarity score
Example
Type
Ph
>0
<0
Phrase
Polarity score
positive phrases
14
7
never becomes claustrophobic
32.3
LSTM
negative phrases
36
36
hardly an objective documentary
-15.77
positive phrases
14
6
never becomes claustrophobic
10.8
GRU
negative phrases
36
36 hardly an objective documentary -48.1
Ideally, based on our analysis the longest N-grams for the phrases will be assigned polarity scores consist with their labels to offset the impact of their sub-phrases. Table@]shows that the sequence- level representations of the N-grams could be generally assigned polarity scores with the signs opposite to the sub-phrases, likely playing dissenting role. Figure 2 shows the sequence-level
Thhe adjectives are listed Table 7 in the appendix.

---

Under review as a conference paper at ICLR 2021
polarity score of the four-gram "hardly objective documentary" negatively large that could help reverse the polarity of the sub- sub-phrase "an objective documentary" and make the overall polarity of the phrase negative. Such negation can be also observed on models with bidirectional GRU or LSTT cells in A.2
2.1
4.2
8.1
6.7
0.8
115.7
hardly
objective
documentary
objective
documentary
Figure 2: Example of dissenting a sub-phrase. Polarity scores are listed for N-grams in the phrase "hardly objective documentary". Each vertical bar represents either token or sequence that starts from its left side and ends with its right side. Left, GRU; right, LSTM.
We noticed that order for such cells as GRU/LSTM to learn complex compositional language structures, it may be essential for the model to have enough exposure to relevant structures during the training phase. We did simple controlled experiment on two sets of labeled instances. In the first set, the six training instances are "good", "not good", "not not good", "not not not not good" and "not not not not not good" with alternating labels "positive" and "negative". the training set only consists two labeled instances, the positive phrase "good" and the negative phrase "not not not not not good". We then trained the GRU model on these two training sets, and then applied these models additional phrases "nol nol not not not not good" and "nol not not not nol not not good". As we can see from Figure the model can infer the multiple negation correctly for the given cases when trained on the first set, and is able to generalize to unseen phrases well. However, it fails to do so for the second. This indicates that proper supervision would be needed for the models capture the compositional nature of the semantics as conveyed by N-grams.
good", "nor
In the second
dataset by extending the first training
with two
150
50
-10
-50
-20
100
200
000d
1-not
2-not
not
4-not
not
6-not
7-not good
1-not
2rrot
3inot
4-not
not
Ginot
7-not
Figure 3: Distributions of polarity scores for negation N-grams (N= 8) in the phrase "not not not not not good". Each box represents polarity score distribution for either the token "good" or times negation N N-grams (e.g., "3-not" refers to "nol not not good"). Circles refer to outliers. Left, model trained six labeled phrases. Right, model trained on two labeled phrases. Results from 30 trials with random initializations. GRU cell used.
5.2 TRAINING WITH SEQUENCE-LEVEL REPRESENTATIONS
We examined whether the sequence-level representations along with the token-level representations could capture sufficient features during training and perform on par with the standard cells. We trained models by replacing the standard GRU LSTM cell with the corresponding N-gram rep- resentations (h, for GRU LSTM cell). We evaluated them both sentiment analysis and
8

---

Under review as a conference paper at ICLR 2021
language modeling tasks and compared them with the standard models. Additionally, we created baseline model named "Simppified" by removing all the terms involving 2; from A(x,) in Equation
[1 and I6 The resulting representations do not capture sequence-level information. On the binary SST dataset (with sub-phrases) and Movie Review dataset (Pang & Lee 2004), we found both the standard cells and our N-gram representations behaved similarly, as shown in Table []] But the "Simplified" models did not perform well. Glove (Pennington et al., 2014) embeddings were used.
We ran language modeling tasks on the Penn Treebank (PTB) dataset (Marcus et et. [993), Wikitext- 2 dataset and Wikitext-105 103 dataset (Merity et al., 2016) respectively) The embedding size and hidden size were both set as 128 for PTB and Wikitext-2, and 256 for Wikitext- Adaptive softmax (Joulin et al., 2017) was used for Wikitext-103.
It can be seen that using such representations can yield comparable results as the standard Table 3: Accuracy (%) on sentiment analysis
GRU or LSTM cells, as shown in Table datasets
However, for the "simplified" representations, their performances dropped sharply which im- plies the significance of sequence-level repre- sentations. We noticed the intermediate outputs could grow to very large values on Wikitext- 103, therefore, we clamped the elements in the hidden states to the range (-3, at each time step. This demonstrated that the sequence-level features might be significant contributor to the performances of a GRU LSTM cell apart from the token-level features.
Cell
Type
SST
MR
Valid
Test
Valid
Test
Standard
85.8
87.9
81.6
78.7
LSTM
N N-gram
86.9
87.9
82.4
78.4
Simplified
83.0
84.8
81.0
76.2
Standard
85.9
88.1
81.2
76.7
GRU
N-gram
86.4
88.3
82.6
77.6
Simplified
82.9
83.7
80.6
76.1
Although using the N-gram representations perform well on both tasks, we cannot rule out the contributions of other underlying complex features possibly captured by the standard cells. This can be observed from the test perplexities obtained from the N-gram representations, which are generally slightly higher than those obtained from the standard GRU or LSTM cells. However, the N-gram representations even outperformed the standard GRU cell on Wikitext-103.
Table 4: Perplexities on language modeling datasets. All parameters were initialized randomly.
PTB
Wikitext-2
Wikitext-
103
Cell
Variants
Valid
Test
Valid
Test
Valid
Test
Standard
89.08
77.19
91.79
80.68
121.77
116.70
GRU
N-gram
90.46
78.32
94.53
82.69
109.95
106.23
Simplified
100.30
85.58
105.50
92.28
137.00
130.65
Standard
89.22
77.11
92.23
80.74
105.25
101.08
LSTM
N-gram
90.21
77.82
94.35
81.21
107.1
103.34
Simplified
101.06
85.64
105.67
92.13
135.69
130.07
CONCLUSION
In this work, explored the underlying features captured by GRU and LSTM cells. We expanded and unrolled the internal states of a GRU or LSTM cell, and found there were special representations among the terms through a series of mathematical transformations. Theoretically, we found those representations were able to encode sequence-level features, and found their close connection with the N-gram information captured by classical sequence models. Empirically, we examined the use of such representations based on our finding, and showed that they can be used to construct linguistic phenomenons such as negations on sentiment analysis tasks. We also found that models using such representations only can behave similarly to the standard GRU or LSTM models on both the sentiment analysis and language modeling tasks. Our results confirm the importance of sequence- level features as captured by GRU LSTM, but at the same time, we also note that could not rule out the contributions of other more complex features captured by the standard models. There some future directions that are worth exploring. One of them is to explore possible significant features captured by higher-order terms in a GRU/LSTM cell, and understand how they contribute the performances.
"Our model is word-level language model, we used the torchtext package obtain and process the data.

---

Under review as a conference paper at ICLR 2021
REFERENCES
George Arfken and Albert A Mullin. Mathematical Methods for Physicists, 3rd ed., chapter 5, pp- 303-313. Orlando, FL: Academic Press, edition, 1985.
Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Explaining recurrent neural network predictions in sentiment analysis. In Proceedings WASSA @ EMNLP, 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint vrXiv: 2014.
URL
wttp://///.a........//// 3555
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and
stochastic optimization. Journal of machine learning research, 12(jl):2121-2159, 2011. URL
http://www
Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm:
A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):
2222-2233 2016. UtLht:t: org/ abs 1503. 04069
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, et al. Efficient softmax approxima- tion for gpus. In International Conference Machine Learning, pp. 1302-1310. PMLR, 2017.
URL /ettps//aaxii..orr/ab///////////
04309
Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwamura. Preventing gradient explosions in gated
recurrent units. In Proceedings of 2017. URLhttp://paperrs nips cc cc/pprr
6647 preventing wra.ggggggggaaeeeeeeeeeeeeeeeeee oeearaaeeeeee....ooooooooooooeee pdf
in recurrent wnnteuuurrettuttts...... units.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing unddsssanngg recurren arXiv preprint
02078 recurrent networks.
2015. UKLL/ttt://aaxxx..rrr/bb//////// 02078
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of ICLR, 2014. htt.://p//aaav.ooooooo////////// wttp://..rrr//bb/1...00020
Ben Krause, Liang Lu, lain Murray, and S. Renals. Multiplicative Istm for sequence modelling. ArXiv, abs//609.0000 (.660999.99,.... 2017. /ttp:tt/::///xxx....r////////
07959
Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. CoRR, 2015. URL
01066
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated
corpus of English: The Penn Treebank. Computational Linguistics, 19(2):3133330, 1993. URL https: http::/wwwwwwwww.bbbbbb orq http:///bb.o..aatttooo/////----/
Gábor Melis. Tomáš Kočiský, and Phil Blunsom. Mogrifier Istm. In Proceedings of ICLR, 2020.
URL htt.httpss//peenevvieee
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 2016.
William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. A formal hierarchy of RNN architectures. In Proceedings ACI, pp. 443-459, July 2020. URL
htttps://www.ctt
org/
maan.43
W. James Murdoch, Peter Liu, and Bin Yu. Beyond word importance: Contextual decom- position to extract interactions from LSTMs. In Proceedings of ICER, 2018. URL https:
//openreview... wtw.:enneett.ooooooooooore.re
Oz
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity. In Pro- ceedings of ACI. pp. 271-278, 2004.
10

---

Under review as a conference paper at ICLR 2021
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of EMNLP, 2014. URL http://wwyt www.eb............ orrg
aatholooyyD1441112
Alex Sherstinsky. Deriving the recurrent neural network definition and mn unrolling using signal processing. In Proceedings of Critiquing and Correcting Trends in Machine Learning Workshop
at NeurlPS, volume 31, 2018. CtLLttt:// http://axxvvv..g.x..e. 1808. 03314
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. Proceedings of EMNLP, 2013. URL httpp:://www/ http:://wwwwwwweelle org/anthology/
D13-1170
Xiaobing Sun and Wei Lu. Understanding attention for text classification. In Proceedings of ACL, pp. 3418-3428, July 2020. URL https: http::/wwww.llllc. www./oottooogg////oo0/0
acl main. 312
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Proceedings of NewrlPS, pp. 1104-1112, 2014.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASET STATISTICS
We listed the statistics of the datasets used in our experiments.
Table 5: Statistics of sentiment analysis datasets. "Label" refers to the size of the positive la- bels and negative labels in the training set, "positive/negative" for binary classification, "posi- tive/neutrall/negative" for 3-class classification.
Data
Train/Valid/Test
Label
Vocab
Max.len
Remark
SST
98,794/872/1.821
56,187 /42,607
17404
54
sub-phrase labels
SST-Sampled
320/-/-
110/86/124
248
7
3-class
MR
9,500/500/662
4.729/4.771
18584
59
Synthetic
4,120/230/200
2,060/2,060
85
6
negation
Table 6: Statistics of language modeling dataset, quoted from Einstein.ai
PTB
Wikitext-2
Wikitext-103
Train
Valid
Test
Train
Valid
Test
Train
Valid
Test
Article Num
600
60
60
28.475
60
60
Token Num
887,521
70.390
78,669
2.088.628
217.646
245,569
103,227,021
217.646
245,569
Vocab Size
10,000
33,278
267.735
A.2 SST DATASET
We extracted adjectives (shown in Table 7 based on their frequency ratio in the positive and negative instances. If an adjective appeared mostly in positive (negative) instances, we would regard it as positive (negative) adjective. The textblob package was used to detect adjectives. The labeled phrases (shown in Table 8 would be selected from the SST dataset if they had labeled sub-phrases of opposite signs by removing the negation words.
Table 7: Extracted adjectives from the SST dataset
Type
Adjectives
Size
outstanding, ecological, inventive, comfortable. nice, authentic, spontaneous. sympathetic. lovable. unadulterated, controversial, suitable, grand, happy, enthusiastic, adventurous, successful, noble, true, detailed, sophisticated, sensational, exotic, fantastic, remarkable, impressive, charismatic, good, effective, rich, popular, unforgettable, famous, comical, energetic. ingenious, extraordinary. pleased, tremendous, marvelous, helievable, artistic, expressive, exceptional, fabulous, strong, humorous, gorgeous, available, incredible, memurable, likable, delicious, attractive, impeccuble, accessible, delighted, hilarious, inspirational, thoughtful, affable, creative, great, imaginative, enjoyable bad, tedious, miserable, psychotic, didactic, incxplicable, teeble. sloppy. disastrous. stupid, amateurish, false, cynical, farcical, terrible, unhappy. horrible, atrocious, idiotic, wrong, pathetic, angry, uninspired, vicious, unfocused, unnccessary, artificial, troubled, questionable, arduous, stereotypical, poor, unpleasant, forgettable, ridiculous, laughable, indecipherable, grievous, impassive, gratuitous, weak, simplistic
Positive
Negative
42
A.2.1 BIDIRECTIONAL GRU/LSTM MODELS
We also conducted experiments using bidirectional GRU and LSTM cells the binary SST dataset. Figures 4 and show that the models can capture such negation from both directions. For exam- ple, the four-gram "to feel contradictory things" has negative sequence-level polarity score while the five-gram "freedom to feel contradictory things" has a positive one. Similarly, in the backward direction, the bigram "to freedom" has a positive sequence-level polarity score while the five-gram "things contradictory feel to freedom" has a negative one. Glove (Pennington et al. 2014) embed- dings were used. Embedding size and hidden size were set as 300 and 1024 respectively.
12

---

Under review as a conference paper at ICLR 2021
Table 8: Selected labeled phrases from the SST dataset
Type
Phrase
never fails him, never fails fascinate not bad way, never wanted leave never feels derivative, not to be dismissed, never becomes claustrophobic, never fails to entertain, never feels draggy, never veers from its comic course, never growing old, not bad premise, not without merit, not mean spirited not life affirming, not for every taste, hardly an objective documentary, not must own, never rises to higher level, not exactly assured in its execution, not as good as the original, be a breakthrough filmmaking. not always for the better, not well acted, not very compelling or much fun, never reach satisfying conclusions, not as sharp, never rises above superficiality never seems firsh and vital not good movie, not movie make, not the great american comedy, not smart and, hardly seems worth the effort not enough of interest onscreen, not funny performers, not well enough, not in a good way, hardly a nuanced portrait, not good enough, not number 1, gaining much momentum, not well enough,
positive
takes hold
classic, not the best herzog,
negative
not very amusing, not one clever line, not a good movie, never comes together
0.4
4.9
-17
1.0
0.3
2.4
-0.4
freedom
feel
contradictory
things
freedom
feel
contradictory
things
Figure 4: Polarity scores for N N-grams (N 1 5) in the phrase "freedom to feel contradictory things" from bidrectional GRU cells. Left, forward cell; right, backward cell. Each bar represents N-gram. Red refers negative polarity scores while blue refers to positive ones.
A.2.2 3-CLASS SST DATASET
We also considered the dissenting scenarios for 3-class sentiment analysis. We extracted 160 pairs of labeled phrases starting with negation words and their sub-phrases with different labels from the 3-class SST dataset. We trained the model on the extracted pairs directly until all the instances were classified correctly. Table 9 showed that the sequence-level polarity scores of the longest N- grams in the phrases could generally capture the differences between the pairs, and dominate in the dimensions corresponding to the labels.
Table 9: Results the sampled 3-class SST dataset. "neu", "pos" and "neg" refer to that the sequence-level polarity scores have the largest value in the dimension corresponding to the label "neutral". "positive" and "negative".
LSTM
GRU
Тype
Num
Example
pos
neg
pos
neg
Neutral N-gram
30
2
37
0
37
not ultimate blame
Positive N-gram
23
5
6
18
4
28
never feels derivative
Negative N N-gram
2
92
6
89
95
not to see
13

---

Under review as a conference paper at ICLR 2021
-31
-14
0.2
-0.2
2.2
freedom
feel
contradictory
things
freeoo
50
feel
contradictory
things
Figure 5: Polarity scores for N --grams (N 1 5) in the phrase "freedom to feel contradictory things" from bidrectional LSTM cells. Each bar represents a N-gram. Red refers to negative polarity scores while blue refers to positive ones.
A.2.3 IMPACT OF N-GRAM LENGTHS
To understand the impact of sequences with different lengths, selected positive and negative N- grams (N=1-4) ending with the last token of the instances based on their association with positive and negative labels respectively. It appears that the token-level polarity scores for unigrams and sequence-level polarity scores for bigrams can reflect their association with labels better than that for trigrams and four-grand gur-ggams as shown in Figure O This demonstrates that although the sequence- features can be well captured and are important for sentiment analysis tasks, the shorter sequences in general may be playing more crucial roles than the longer ones.
0.36
0.14
Negsttve
Negative
0.34
0.32
0.30
0.08
0.08
0.06
0.04
0.02
0.02
-10
-5
10
-25
-20
-15
-30
-5
30
Feaoftyy Score
Polority Score
Postivve four-gram
0.150
Tram
Negative fnur-gram
0.12
Frag Denutty
0.075
0.08
0.000
0.02
0.000
0.00
Figure 6: Polarity score distribution for the N N-grams (N=1-4) that have strong association with specific label. Result from GRU cell on the SST dataset.
A.3 SYNTHETIC DATASET
Few real-world datasets provide sufficient exposures for tokens, phrases and their negation expres- sions. Therefore, synthesized instances with binary labels based adjectives and negation
14

---

Under review as a conference paper at ICLR 2021
Table 10: Polarity score distribution for tokens, negation phrases and double negation phrases. Re-
sults from 10 trials with random initializations.
GRU
LSTM
Type
Number
Polarity Score
Polarity Score
words
15
4.06 = 0.62
3.91 0.86
positive adjectives
negation
-5.85 4.81
-14.36 = 8.32
double negation
10
12.75 5.66
21.17 + 21.00
tokens
18
-4.17 0.75
-3.75= 0.90
negative adjectives
negation
37
6.59
15.05= 9.61
double negation
10
-11.27 + 8.08
-22.134 18.50
Table 11: Adjectives, negation and double negation examples for the synthetic dataset Adjectives
Negation Example
Double Negation Example
not charming. not good
good, nice, charming. awesome, fascinating, impressive, excellent, wonderful, attractive, interesting, inspiring. stunning. amazing, terrific, incredible
not incredible, not wonderful
not not attractive
not not inspiring not not awesome never never awesome never never wonderful never never scemed incredible
not nice, not amazing never charming, never amazing never incredible,
Positive
attractive
never look wonderful, never look charming
never seemed good
not horrible, not pointless not dire, not mediocre not terrible, not unfunny
awful, bad, uninspiring. dull, boring, ledious, horrible, terrible, pathetic, mediocre, shallow, pointless, unfunny, gruss, poor, dreadful, dire, useless
gross not not mediocre not not unfunny never never had never never seerned dire never never seemed shallow
Negative
bad, dull
never boring, never horrible never seem terrible, never seem pointless never seemed unfunny
words following simple grammatical rules. We created vocabulary of words including nouns, verbs, adjectives, adverbs, articles and negation words. Negation phrases and double negation phrases are also incorporated those instances. We let positive adjectives such as "inspiring" and the double negation expression "not not inspiring" appear in the instances labeled as positive, e.g., "her dramas were indeed inspiring", "her movies are not not inspiring" .ett the negation expression "not inspiring" appear in the instances labeled negative, e.g., "his movie is not inspiring". We did similarly for negative adjectives.
Table TO shows that the sequence-level representations could generally have polarity scores match- ing the roles of the negation and double negation N-grams. For the positive (negative) adjectives, their negation N-grams have negative (positive) sequence-level polarity scores while their double negation N N-grams have positive (negative) ones. The negation tokens are "nor" and "never". Models were trained until all the negation expressions have been classified correctly.
The key tokens, negation and double negation phrases are shown in Table
A.4 PERFORMANCE DURING TRAINING
We make comparisons between the performances of the standard GRU/LSTM cell and the N-gram representations during training. can be seen from Figure 7 and that the N-gram representa- tions perform similarly to the standard GRU/LSTM cell on the PTB, Wikitext-2 and Wikitext- datasets. Adam optimizers (Kingma & Ba 2014) were used.
A.5 MULTIPLE NEGATION ON A SIMPLE SET
To scrutinize the sequence-level features under controlled conditions, we created a training set con- sisting of the phrases: "good", "not good", "not not good". "not not not good", "not not not not good" and "not not not not not good" with alternating labels "positive" and "negative". We trained standard GRU or LSTM cell the training set until the loss converged (less than 10c-6) with the embedding size 256, hidden size 1024, then calculated the corresponding polarity scores for N-grams.
15

---

Under review as a conference paper at ICLR 2021
0.30
3.4
3.5
4.5
0.75
4.7%
Figure 7: Top, standard GRU Cell; bottom, approximate hidden state representation. From left to right: PTB, Wikitext-2, Wikitext- 103. Training and validation losses for language modeling tasks.
5.75
0.00
1pans
Space
10.0%
644
Figure 8: Top, standard LSTM Cell; bottom, approximate hidden state representation. From left to right: PTB, Wikitext-2, Wikitext-103 Training and validation losses for language modeling tasks.
Figure shows the sequence-level features derived from the pre-trained GRU LSTM cell were able to detect multiple negation, implying those features were likely to be significant for classi- fication decisions. For example, the four-gram "not not not good" generally has large negative sequence-level polarity score while the five-gram "not not not not good" generally has a large posi- tive one, and the six-gram "not not not not not good" has large negative again. These polarity scores could help the phrases reverse the polarity of their sub-phrases with opposite labels. Figure shows examples from a pre-trained GRU and LSTM cell with random initializations. We can see with each "nor", the sequence-level polarity score will reverse the polarity. Based on aforemen- tioned analysis, it may be essential for the models to have enough exposure relevant structures during the training phase.
16

---

Under review as a conference paper at ICLR 2021
50
0
-5
-100
8 -10-
115
g00d
Innot
2-hot
3 not
not
not
g00d
not
n
not
4
Sinot
Figure 9: Distributions of polarity scores for negation N-grams (N = polarity score distribution for either the token "good" or the times negation N-grams (shown as i-not, 2...5). Circles refer outliers. Results from 30 trials with random initializations. An LSTM cell is used.
6). Each box represents
12.6
10.9
19.2
18.0
30.1
36.5
-5.66
not
not
not
not
not
good
Figure 10: Polarity scores for N gramm (N = 6) "not not not not not good". Left, GRU; right, LSTM. Each vertical bar represents the polarity score (token-level or sequence-level) for the N N-gram that it covers. Red bars refer to negative polarity scores and blue bars refer to positive scores.
17


Reviews:
[{"review_id": "yo5gibcg96L", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to linearize GRU and LSTM cells (as error terms should be negligible when inputs are small in magnitude). Putting these linearized, or, really, affine, RNN cells together into a single-layer sequence processor, thanks to the affine-ness, we can decompose the score that is obtained by taking dot products with a query at each timestep into contribution by immediate unigram features and all subsequences leading to this unigram. The authors evaluate these scores, showing that they do and don't capture phenomena in a synthetic dataset and proceeding to show that when both training and evaluating with this simplified network on SST yields strong results.\n\n#### Strengths/what I loved:\n\n- Motivation and Related Work seemed nicely done, set this paper up nicely, and made me excited to read on!\n- I like the idea of testing double negation and omitting it during training and it was interesting to see the networks then fail to pick up on it (assuming the synthetic dataset is reasonable).\n- The visualizations of sequence-level scores (Figures 2 and 4) are very cleverly chosen and powerful, even if they take a bit of getting used to.\n- Figures 3 and 5 are striking: even on SST unigrams and perhaps bigrams seem to get you most of the way if you trust the approximate interpretation.\n\n#### Criticism/weaknesses:\n\n- It is unclear how the simplified/affine-ized architecture relates to vanilla RNNs---those, too, can be decomposed like that and one could just as well look at these features. To paraphrase: I don't see why any of the analysis and results in this paper is only true for gated cells and I wouldn't be surprised to see a vanilla RNN yield equally good approximate cells, since after all the task is very simplistic (Section 5.2 concludes as much, as even without sequence information the task is easy to solve, but the conclusion drawn from that result, namely that there is something about GRUs and LSTMs to read from this doesn't follow in my opinion).\n- The synthetic dataset is a mystery: yes, it contains sentences that contain the words shown in Table 6 (Appendix A.2), but... what are these sentences? Are they actual text from some dataset? Text sampled from some model or grammar? Random words without any sequential coherence? What is the vocabulary size? What does appearing \"mostly\" in positive or negative instances really mean? Just giving one or two examples would have made me a lot less worried and confused about what is going on here, but to base many if not most of the results on it, this dataset is woefully underdescribed.\n\n#### Questions:\n\n- Notation/definitions in section 5.1.1 are either very unclear or flat-out wrong: Sun & Lu (2020) do indeed define a notion of a \"token-level polarity score\" for each output class, but the notion of output classes is not mentioned at all here, in fact instead of the output embedding matrix W that Sun & Lu (2020) posit, this paper speaks only of a single vector w. I assume that that is the vector associated with the positive class and the prediction thus is strictly binary---very much unlike Sun & Lu (2020). In addition to that, the notion of a \"sequence-level polarity score\" that here complements the token-level score is not at all mentioned in Sun & Lu (2020) as far as I can see, so to say that their methodology is used is misleading in more than one way. Finally, the model described in Sun & Lu (2020) is one that uses attention, i.e., that does mean-pooling after the linear layer that is transforming the individual states. This paper mentions the linear layer, but not the pooling/attention, so it's unclear if that is a poor paraphrase of Sun & Lu or whether this paper here too diverges from the paper it claims to build on.\n- Are Figures 2 and 4 cherry-picked? Since nothing else is stated, I would assume so. In light of that, the longest subsequence on the right of Figure 2 being negative is rather strange. Do you have an explanation? Is \"those\" a negative word?\n- I think it would've been very interesting to see whether or not this simplification is legitimate, that is whether the assumptions made in the derivation are justified: train with the original cell formulation, but then evaluate *quantitatively* using the approximate cell to essentially create a table like Table 2 or perhaps even a scatter plot for individual logits to see how much things change or don't change. That would go a long way to convince me that the approximation is at least reasonable.\n- A.3: \"there's an apparent difference\" What is that difference? I don't see any.\n- A.4: What is the change here, can you highlight it or motivate it? The results certainly aren't particularly impressive, so I'm tempted to say this section hurts more than it helps...\n\n#### Typos and other small things:\n\n- 5.1: As you have multiple runs, which one was selected? The best according to some metric? The median or mean somehow? Or randomly chosen? Either way, I strongly feel that empirical results should always come with a sense of stability: what was the variance between runs? Between hyperparameters? How sensitive are results and what can we say about statistical significance?\n- The overlaid histograms definitely need to have some transparency or be shown another way---right now it is impossible to see what is \"happening\" in the blue bars as they are hidden by the orange bars.\n- The last sentence of Section 5 should link to A.4, I guess?\n\n\n---\n\nI read and appreciated the response, but my overall rating is still leaning negative.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vKVbAVrJZNI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "DISCLAIMER: this is not my field of research. With strong arguments I could be persuaded to change my score.\n\nThis paper introduces a method to unroll the Gated Recurrent Unit (GRU) and the Long Short-Term Memory (LSTM) unit using a taylor expansion. Essentially a linearization of the GRU and LSTM. Given the simplifications, the authors argue that their model captures N-gram information for the sequential information. The paper presents results suggesting that this approximation is able to capture much of the same sequential information as the LSTM and GRU on benchmarks such as SST-2, PDB, Wikitext-2, and Wikitext-103.\n\nWhat this paper excels at is a thorough theoretical formulation of the proposed approximation, and a comparison with an approximation without the sequential information. This shows that the approximate version is able to capture sequential information.\n\nHowever, the relevance of this paper is not clear to me, and the introduction and related works does little more to explain that relevance than stating: “understanding the essential features captured by GRU/LSTM”. The tasks that the method is tested on are synthetic data for sentiment and tasks/models that haven't been relevant since 2016. The motivation for these tasks, and the qualitative analysis is hard for me to understand.\n\nThe paper could use some reformulations and more emphasis on what exactly the purpose of the paper is, in particular I find the lack of consistency in present/past tense disrupts the reading experience.\n\nBelow I have made a few comments/questions:\n\nAbstract:\n“Sequential data in practice” - unclear what this means\n“sequence-level representations brought in by the gating mechanism” - I dont understand this sentence\n“essential components” - vaquely defined\n“Based on such a finding” - rephrase\n\nIntroduction:\n“gradient vanishing or explosion issues” - vanishing or exploding gradients\n“While such models ...” - this whole sentence is a little vague\n\nRelated work:\n“With the variants” - its variants\nGeneral comment (also for introduction): While you mention many interesting findings in recent years, it is difficult for me to assess how exactly your work differs. Please use the related works to emphasize what you are doing differently than previous work in your field.\n\nLSTM:\n“A LSTM cell” - An LSTM cell\n\nExperiments:\n“Figure 2” I don’t get what each bar represents\n“Subphrase labels” - are subphrase labels the node annotation?\n\nWhy is negation, and a synthetic variant, important to explain the relationship between N-grams and LSTMs?\nWhy do you choose the datasets you do? Why is SST-2 and benchmarking against older language models of interest? Why is an N-gram comparison interesting? Perhaps the authors should look into contemporary research on formal methods in sequential models for inspiration of tasks and where an interesting hypothesis might be: https://arxiv.org/abs/1906.03648\n\nUpdate:\nI have read the rebuttal and the updated paper. I don't see my issue of relevance addressed. My score remains the same.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cYn_EEOmaMv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a reliable interpretation of modern RNN models, through unrolling GRU and LSTM cells. The approximate state representations include a token-level term that only depends on the current input token and a sentence-level term that depends on all inputs until the current token. The deriving process is clear and illuminating. The experiment section shows that the approximation shares similar behavior and performance as the original model.\n\nOverall, the paper is well written and easy to follow. Although GRU and LSTM are no longer the default model for SOTA performance in the NLP community. I believe that this study still provides interesting insights for those who want to develop better recurrent or non-recurrent models in the future.\n\nMy major concern is that the language model experiment didn’t include a stronger baseline method, such as AWD-LSTM, which provides a significantly lower ppl compared to these in the paper.\n\nIt would be interesting to see a more detailed ablation study, that studies the importance of each term in A(x_t).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JGdZm3Mw_Ba", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to add a contribution on understanding how gated recurrent neural networks like GRUs and LSTMs can learn the representation of n-grams. The authors expand the sigmoid function and the hyperbolic tangent function using Taylor series to obtain approximated closed-form mathematical expression of hidden representation when using the GRU or the LSTM as the update rules. The approximated hidden representation of the GRU and the LSTM update rules can be separated into two terms, (1) the current token-level input feature and (2) the sequence-level feature, which is a weighted sum of all previous tokens. As the hidden representation consists of two feature terms, one can take each feature (either token-level or sequence-level) separately for a downstream task, e.g., evaluate how good when sequence-level feature is used for predicting polarity score in sentiment analysis.\n \nThe idea of improving theoretical understanding on how n-grams are modelled by gated recurrent activation functions is sound. However, I am not entirely satisfied with what has been investigated after obtaining the approximated closed-form expression of gated recurrent activation functions. The tasks that were used in the experiments are sentiment analysis and language modelling. In sentiment analysis, most of the plots were there to show how token-level features or sequence-level features align with the polarity score, and we can observe some sort of individual implication from each term. However, it is predictable that sequence-level feature should be meaningful. I don't see much of insights by showing that the polarity score from sequence-level features indeed align with this prediction. If we can to apply Taylor expansion to simple recurrent neural networks (RNNs), such that we can expand the hidden representation of a standard RNN into two terms: the current token-level input feature and sequence-level feature, how would the results look like and how can we relate them with what were reported in this paper? Is this paper particularly showing how gated RNNs are modelling n-grams or RNNs in general? A comparison would be nice to show how sequence features get improved in gated RNNs.\n\nIt is interesting to see that the approximated versions of GRUs and LSTMs can perform on a par with the original models on language modelling tasks, however, these results don't necessarily improve our understanding on how gated RNNs are capable of learning good representations of n-grams. They confirm that sequence features are indeed helpful though.\n\nIn Section 5.1, if there were multiple trials of experiments on the same task, why not report the average and the variance of the results instead of one set out of multiple results?\n\nIn Section 5.2, Adpative softmax (Joulin et al., 2017) was used for Wikitext-130. -> Adpative softmax (Joulin et al., 2017) was used for\nWikitext-103.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2y9f45jk_9v", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines n-gram level features encoded within the hidden states of recurrent neural models. The proposed method approximates the hidden states of LSTMs and GRUs with a first-order Taylor series, which the authors claim is an adequate approximation with small enough inputs. The authors apply their method to models trained on synthetic sentiment analysis and language modeling datasets. The paper is difficult to understand, and many assumptions are not properly justified. The experiments are also not convincing, as a large portion of the analysis focuses on small synthetic data, and some of them do not have clear takeaways or explanations as to why they were conducted in the first place. Overall, I cannot recommend the paper's acceptance in its current form.\n\ncomments/questions:\n- is the scenario of extremely low input magnitudes realistic? how generalizable are these findings to standard initializations used in NLP architectures\n-  similarly, on page 3 the authors assume that the higher order terms of h_{t-1} are \"insignificant\"; in practice, it is unclear how often this is true. i wish the paper would contain more justification behind these assumptions, as they are critical for judging how faithful the approximation is and thus how useful the proposed method is for diagnosing RNNs.\n- can't the authors actually show quantitatively how good the approximations are? if the higher-order terms indeed do not affect the quality of the approximation, that could be justified by some experiments. i'm not really convinced by Table 2: the approximations could be quite different from the original model but still yield good downstream accuracy.\n- what is a \"polarity score\" (bottom of page 5)? I didn't quite understand what this is supposed to represent, is it how predictive of a label a particular span is? \n- why are synthetic datasets used at all here? experiments on a small set of sentences with a tiny vocabulary and artificial \"double negatives\" are not compelling. Appendix A.2 does not fully specify this dataset (nor motivate why it was created); what is e.g., its average sentence length?\n- the results on a real sentiment dataset (SST2) are confusing (sec 5.1.3): what does figure 4 show me that I couldn't already learn by simply passing those two phrases into the model as separate inputs and looking at the model's prediction? \n- what is the point of training models with the approximations instead of the original GRU/LSTM cell equations?  i don't understand the significance of Table 3.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: the authors demonstrated that vanilla RNN, GRU and LSTM compute at each timestep a hidden state which is the sum of the current input and the weighted sum of the previous hidden states (weights can be either unit or complicated functions), when sigmoid and tanh functions are replaced by their second-order taylor series each. they refer to the first term as token-level and the second term as sequence-level, and claim that the latter can be thought of as summing n-gram features in the case of GRU & LSTM due to the complicated weight matrices used for the weighted sum, largely arising from the gating mechanisms. 

the reviewers are largely unsure about the significance of the findings in this paper due to a couple of reasons with which i agree. first, it is unclear whether the proposed approximation scheme is enough to capture much of what happens within either GRU or LSTM. if we consider a single step, it's likely fine to ignore the O(x^3) term arising from either sigmoid or tanh, but when unrolled over time, it's unclear whether these error terms will accumulate or cancel each other. without either empirically or theoretically verifying the sanity of this approximation, it's difficult to judge whether the authors' findings are specific to this approximation scheme or do indeed reflect what happens within GRU/LSTM. 

second, because the authors have used relative simple benchmarks to demonstrate their points, it is difficult, if not impossible, to tell whether the authors' findings are about the datasets themselves (which are all well known to be easily solvable or solvable very well with n-gram classification models and n-gram language models) or about GRU/LSTM, which is related to the first weakness shared by the reviewer. the observations that n-gram models and simplified GRU/LSTM models work as well as the original GRU/LSTM models on these datasets might simply imply that these datasets don't require any complicated interaction among the tokens beyond counting n-grams, which lead to the original GRU/LSTM trained to be simplified (n-gram detectors.) 

that said, i still believe this direction is important and is filled with many interesting observations to be made. i suggest the authors (1) verify the efficacy of their approximation scheme (probably empirical validation is enough, and (2) demonstrate their point with more sophisticated problems (carefully designed synthetic datasets are perfectly fine.)



Decision: Reject