Venue: NAACL 2019

# Serial Recall Effects in Neural Language Modeling

Hassan Hajipoor\({}^{1}\), Hadi Amiri\({}^{2}\), Maseud Rahgozar\({}^{1}\), Farhad Oroumchian\({}^{3}\)

\({}^{1}\)School of Electrical and Computer Engineering, University of Tehran, Iran

\({}^{2}\)Department of Biomedical Informatics, Harvard University, Boston, USA

\({}^{3}\)University of Wollongong in Dubai, Knowledge Village, Dubai, UAE

h.hajipoor@ut.ac.ir, hadi@hms.harvard.edu, rahgozar@ut.ac.ir,farhadoroumbian@uowdubai.ac.ae

###### Abstract

Serial recall experiments study the ability of humans to recall words in the order in which they occurred. The following serial recall effects are generally investigated in studies with humans: _word length_ and _frequency_, _primacy_ and _recency_, _semantic confusion_, _repetition_, and _transposition_ effects. In this research, we investigate LSTM language models in the context of these serial recall effects. Our work provides a framework to better understand and analyze neural language models and opens a new window to develop accurate language models.

## 1 Introduction

The goal of language modeling is to estimate the probability of a sequence of words in natural language, typically allowing one to make probabilistic predictions of the next word given preceding ones Bahl et al. (1983); Berger et al. (1996). For several years now, Long Short-Term Memory (LSTM) Graves (2013) language models have demonstrated state-of-the-art performance Melis et al. (2018); Merity et al. (2018a); Sundermeyer et al. (2012). Recent studies have begun to shed light on the information encoded by LSTM networks. These models can effectively use distant history (about \(200\) tokens of context) and are sensitive to word order, replacement, or removal Khandelwal et al. (2018), can learn function words much better than content words Ford et al. (2018), can remember sentence lengths, word identity, and word order Adi et al. (2017), can capture syntactic structures Kuncoro et al. (2018) such as subject-verb agreement Linzen et al. (2016). These characteristics are often attributed to LSTM's ability in overcoming the _curse of dimensionality_-by associating a distributed feature vector to each word Hinton et al. (1986); Neubig (2017)-and modeling long-range dependencies in faraway context Khandelwal et al. (2018).

The goal of our research is to complement the prior work to provide a richer understanding about how LSTM language models use prior linguistic context. Inspired by investigations in cognitive psychology about serial recall in humans Avons et al. (1994); Henson (1998); Polissenska et al. (2015)- where participants are asked to recall a sequence of items in order in which they were presented, we investigate how word length or frequency (_word-frequency_ effect), word position (_primary_, _recency_, and _transposition_ effects), word similarity (_semantic confusion_ effect), and word repetition (_repetition_ effect) influence learning in LSTM language models. Our investigation provides a framework to better understand and analyze language models at a considerably finer-grained level than previous studies, and opens a new window to develop more accurate language models.

We find that LSTM language models (a) can learn frequent/shorter words considerably better than infrequent/longer ones, (b) can learn recent words in sequences better than words in earlier positions,1 (c) have a tendency to predict words that are semantically similar to target words - indicating that these networks have a tendency to group semantically similar words while suggesting one specific word as target based on prior context, (d) predict as output the words that are observed in prior context, i.e. repeat words from prior context, and (e) may transpose (switch adjacent) words in output depending on word syntactic function.





## 2 Serial Recall Effects

Language models estimate the probability of a sequence as \(P(w_{1}^{n})=\prod_{i=1}^{n}P(w_{i}|w_{1}^{i-1})\), where \(w_{i}\) is the \(i\)th word and \(w_{i}^{j}\) indicates the sub-sequence from \(w_{i}\) to \(w_{j}\). These models minimize their prediction error against words in context, using e.g. the negative log likelihood loss function, during training: \(\mathcal{L}=-\frac{1}{n}\Sigma_{i=1}^{n}\log P(w_{i}|w_{1}^{i-1})\). In this paper, we show the loss of a model against sequence \(s\) and word \(w_{i}\in s\) by \(\mathcal{L}(s)\) and \(\mathcal{L}(s)[w_{i}]\) respectively.

We use the LSTM language model developed in Merity et al. (2018) for our experiments. Given a sequence of words \(w_{1}^{i-1}\) as context, this model predicts the next word in sequence. We refer to \(w_{i}\) and \(\hat{w}_{i}\) as _target_ and _predicted_ words respectively; given the global vocabulary \(\mathcal{V}\), \(\hat{w}_{i}=\arg\max_{w_{j}\in\mathcal{V}}\Pr(w_{j}|w_{1}^{i-1})\). We study this LSTM in the context of serial recall effects.

### Word-Frequency Effect

What is the effect of word frequency/length2 on the performance of LSTM language models? For this effect, we report the average loss for each word frequency as follows:

Footnote 2: Note that word length and frequency are directly correlated Bell et al. (2009).

\[\mathcal{L}_{WF}^{k}=1/|\mathcal{S}_{k}|\sum_{s\in\mathcal{S}_{ k},w_{i}\subseteq s}\mathcal{L}(s)[w_{i}], \tag{1}\]

where \(\mathcal{S}_{k}\) is the set of sequences that, at least, have one target word with term frequency of \(k\), and \(\mathcal{L}_{WF}^{k}\) is the overall loss for target words of frequency \(k\) which sheds light on the expected frequency of words for accurate language modeling.

### Primacy and Recency Effect

What is the effect of word position on the performance of LSTM language models? To analyze this effect, we compute the average loss of network with respect to the position of target words as follows:

\[\mathcal{L}_{PR}^{i}=1/Z\sum_{s}\mathcal{L}(s)[w_{i}], \tag{2}\]

where \(w_{i}\) is the target word in sequence \(s\), \(Z\) is the number of sequences as normalization factor, and \(\mathcal{L}_{PR}^{i}\) is the average of loss at position \(i\). This effect will shed light on network performance at specific positions in texts which can help rationalizing the need for new language modeling architectures, such as bidirectional LSTMs Schuster and Paliwal (1997), or the order by which input data should be processed Sutskever et al. (2014).

### Semantic Confusion Effect

Are predicted words semantically similar to the target ones in case of incorrect predictions? For this analysis, we report the average semantic similarity between target (\(w_{i}\)) and predicted (\(\hat{w}_{i}\)) words as follows:

\[\mathcal{SC}=1/Z\sum_{s,w_{i}\in s,w_{i}\neq\hat{w}_{i}}\texttt{ sim}(w_{i},\hat{w}_{i}), \tag{3}\]

where the function \(\texttt{sim}(.,.)\) computes word similarity either through WordNet Miller (1998) or cosine similarity of the corresponding embeddings of its arguments. This effect will shed light on how effective LSTMs are in disentangling semantically-similar concepts. This gives us a powerful metric to compare networks semantically, especially, in case of equal loss/perplexity.

### Repetition Effect

This effect refers to prediction of a word that already exists in context, i.e. in an earlier position in the sequence. Here, for each target \(w_{n}\), we compute the probability that, instead of \(w_{n}\), any of \(w_{1}^{n-1}\) words is predicted as output and report average across all samples:

\[\Pr(RE^{i})=\\ 1/Z\sum_{s}\Pr(w_{i}\in w_{1}^{n-1},w_{i}\neq w_{n}|w_{1}^{n-1}). \tag{4}\]

This effect will shed light on the extent to which network repeats/predicts observed words as possible responses. Given that words rarely repeat in sentences, the above metric can be used as a good regularizer for language modeling.

### Transposition Effect

This effect refers to word prediction in transposed positions, i.e. the case where the word pair \(w_{i}^{i+1}\) in an original sequence is more likely to be predicted by the network as \(w_{i+1}w_{i}\) in output. Here, for each pair \(w_{i}^{i+1}\) in target sequence, we count the number of times in which \(w_{i+1}\) is more probable to be predicted at position \(i\) (as compared to \(w_{i}\)) and \(w_{i}\) is more probable to be predicted at position \(i+1\) (as compared to \(w_{i+1}\)). We report the average number of transposition occurrences for all samples at each word position. This effect will shed light on how network learn nearby grammatical orders such as conjunction and adjective order.





## 3 Experiments

**Datasets**. We use two benchmark language modeling datasets: Penn Treebank (PTB) Marcus et al. (1993); Mikolov et al. (2010) and WikiText-2 (WT2) Merity et al. (2017). PTB and WT2 have vocabulary sizes of \(10\)K and \(33\)K respectively. We use the POS-tagged versions of these datasets provided by Khandelwal et al. (2018), and treat nouns (NN), verbs (VB), adjectives (JJ), and adverbs (RB) as content words, and others word classes as function words, see details in Figure 0(a).

**Settings**. We set LSTM's parameters as suggested in Merity et al. (2018) for PTB and follow its suggested parameter tuning procedure for WT2. For both datasets, we set context size to \(n=100\) obtained from \(\{5,20,50,100,200\}\) and validation data; note that the number of samples are equal for different sequence lengths.

### Results

We report LSTM performance in terms of prediction loss on development sets for all experiments.

**Word-Frequency Effect**: _More frequent target words are predicted (learned) more accurately than less frequent ones._ Figure 0(b) shows strong inverse correlation between word frequency and LSTM prediction loss. This is expected as neural models learn better with more data. In addition, although the overall loss of function words is considerably lower than that of content words (because of their overall higher frequency), Figure 0(b) shows that, for the same word frequency, content words are learned better than function words.

**Primacy and Recency Effects**: _Target words that appear later in sequences are predicted considerably better than those at earlier positions._ Figure 0(c) shows that prediction loss considerably decreases for target words that appear toward the end of the sequences. The results are consistent across both datasets. This effect can explain why bidirectional LSTMs which read input from opposite directions usually work better in NLP applications such as machine translation Firat et al. (2016); Domhan (2018). A remaining question that is worth investigating is whether bidirectional LSTMs learn first and last few words of sequences better than those in the middle, and if yes, how can we make these models more robust against word position.

**Semantic Confusion Effect**. _There is significant tendency to predict words that resemble (are semantically-similar to) target words._ Figure 2 shows the average WordNet and Embedding similarity between target and predicted words in PTB and WT2 across loss values. The results indicate high similarity between predicted and target words

Figure 1: (a) Dataset statistics, (b): Word-frequency effect, network learns frequent words better than infrequent ones, (c): Primacy and Recency effect, loss decreases for target words that appear at the end of sequences.

Figure 2: We report semantic similarity between predicted and target words, random and target words (lower bound), and nearest neighbor and target words (upper bound).



for smaller loss values. However, confusion consistently increases as prediction loss increases. The upper bound similarity (obtained by treating the nearest neighbor of each target word as predicted word) indicates there exists better candidates which LSTM fails to predict. Our further analyses show that LSTM has a tendency to group semantically similar words and then suggest one of them. We consider most similar words as a group of neighbors and examine how network assigns probabilities to them as compared to others. Figure 2(a) shows that the chance of neighbors of target (with size 150) is equal to chance of other words (with size 9849). As Figure 2(a) shows, the neighbors of predicted words (with size 600) carry equal chance as compared to other words (with size 9399). To find these thresholds, we gradually increase the number of neighbors and track the trend of approaching the probability of neighboring group to target. As shown in Figure 2(b), if the size of neighboring group is set to 150, these probabilities became equal. The similar way is repeated for finding the appropriate size for neighbors of predicted words.

**Repetition Effect**. _Repetition probability of function words is significantly higher than that of content words_. We report repetition effect, see Eq. (4), at POS tag level (where the predicted word should have the same POS tag as target word in prior context). As Figure 3(a) and 3(b) show, function words have higher repetition probability than content words. This is because function words are more frequent, and the average distance among them (i.e. number of intervening words) is considerably smaller than other POS tags (e.g. \(2.1\) words vs \(28.9\) and \(12.4\) words for RB and JJ respectively). We also find that repetition probability decreases as a function of word frequency in prior context, see Figure 3(c). This is because words (especially NNs and VBs) are often self-contained and their occurrence in prior context helps LSTM to avoid "repeating" them. In addition, we find that function words repeat more frequently than other types and repetition among NNs and VBs is higher than other POS tag pairs; Table 1 shows the confusion matrix for repetition across POS tag classes Perhaps, this could explain the recent language modeling improvement obtained in Ford et al. (2018) through developing separate yet sequentially connected network architectures with respect to POS tag class.

There are two factors determining the chance of

Figure 4: (a, b): Repetition effect across POS tag classes in PTB and WT2 and (c): Repetition probability decreases as a function of word occurrences in prior context.

Figure 3: Semantic confusion effect. (a) The chance of neighbors of target is equal to chance of other words. (b) When the neighbours size is set to 150, the probability of target neighbouring group became equal to the probability of target. Neighbours size of 600, leads to equal chance for predicted token.

 

repetition of a POS class: First, the average distance of consecutive tokens of that POS class; Table 2 reports the corresponding values from training set. Second, the accuracy of network in predicting POS classes which has been shown in Figure 5 and also reported in (Khandelwal et al., 2018). From these, the repetition probability of function words are expected to be higher than content words.

**Transposition Effect**: _Transpositions occur more frequently at the beginning of sequences and rarely at the end_. Figure 6 shows average number of transpositions at each word position across datasets. This result is meaningful because miss-predictions occur more frequently at earlier positions (see results for primacy and recency effect). In addition, transpositions are rare at higher positions because more context information can help LSTM to make accurate (transposition-free) prediction. In addition, Table 3 shows the percentage of transpositions across POS tag classes on PTB. The result show that LSTM mainly transposes 'RB NN' word pairs with 'NN RB.' In future, we will conduct further analyses to understand the reason.

The findings presented in this paper provide insight into how LSTMs model context. This information can be useful for improving language models. For instance, the discovery that some word types are repeated in predictions more than others can help regularizing neural language models by making them adaptive to the different word types.

## 4 Conclusion

We investigate LSTM language models in the context of serial recall indicators. We find that frequent target words and target words that appear later in sequences are predicted more accurately, predictions often resemble (are semantically similar to) target words, function words of prior context are more likely to be predicted as target words, and word pair transpositions occur more frequently at the beginning of sequences.

### Acknowledgments

We sincerely thank Nazanin Dehghani for her constructive collaboration during development of this paper and anonymous reviewers for their insightful comments and effective feedback.

## References

* Adi et al. (2017) Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. _International Conference on Learning Representations (ICLR)_.
* Avons et al. (1994) SE Avons, KL Wright, and Kristen Pammer. 1994. The word-length effect in probed and serial recall. _The

\begin{table}
\begin{tabular}{l c c c c c}  & NN & VB & JJ & RB & Func \\ \hline PTB & 3.6 & 7.1 & 12.4 & 28.9 & 2.1 \\ WT2 & 3.6 & 8.5 & 15.8 & 38.5 & 1.9 \\ \end{tabular}
\end{table}
Table 2: Average distance of tokens in POS classes.

Figure 5: Effective context to learn syntax.

\begin{table}
\begin{tabular}{c c c c c c}  & NN & VB & DJ & RB & Func \\ \cline{2-6} NN & 0.025\% & 0.006\% & 0.032\% & 0.015\% & 0.001\% \\ VB & 0.011\% & 0.001\% & 0.004\% & 0.002\% & 0.001\% \\ IJ & 0.021\% & **0.054**\% & **0.045**\% & **0.036**\% & 0.002\% \\ RB & **0.073**\% & 0.017\% & 0.017\% & 0.023\% & 0.012\% \\ Func & 0.002\% & 0.002\% & 0.002\% & 0.001\% & 0.006\% \\ \end{tabular}
\end{table}
Table 3: Confusion matrix for transpositions in PTB at POS tag level. LSTM transposes ‘RB NN’ and ’JJVB|JJ|RB,’ more than others pairs.

Figure 6: Transposition effect, transpositions occur more at the beginning of sequences.

 

Quarterly Journal of Experimental Psychology Section A_, 47(1):207-231.
* Bahl et al. [1983] Lalit R Bahl, Frederick Jelinek, and Robert L Mercer. 1983. A maximum likelihood approach to continuous speech recognition. _IEEE transactions on pattern analysis and machine intelligence_, (2):179-190.
* Bell et al. [2009] Alan Bell, Jason M Brenier, Michelle Gregory, Cynthia Girand, and Dan Jurafsky. 2009. Predictability effects on durations of content and function words in conversational english. _Journal of Memory and Language_, 60(1):92-111.
* Berger et al. [1996] Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. 1996. A maximum entropy approach to natural language processing. _Computational linguistics_, 22(1):39-71.
* Bolhuis and Van Kampen [1988] Johan J Bolhuis and Hendrik S Van Kampen. 1988. Serial position curves in spatial memory of rats: Primacy and recency effects. _The Quarterly Journal of Experimental Psychology_, 40(2):135-149.
* Domhan [2018] Tobias Domhan. 2018. How much attention do you need? a granular analysis of neural machine translation architectures. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1799-1808.
* Ebbinghaus [1913] H Ebbinghaus. 1913. Memory: A contribution to experimental psychology (ha ruger & ce bussenius, trans.). new york, my, us.
* Firat et al. [2016] Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In _Proceedings of NAACL-HLT_, pages 866-875.
* Ford et al. [2018] Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, and George Dahl. 2018. The importance of generation order in language modeling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2942-2946.
* Graves [2013] Alex Graves. 2013. Generating sequences with recurrent neural networks. _arXiv preprint arXiv:1308.0850_.
* Hanson [1998] Richard NA Hanson. 1998. Short-term memory for serial order: The start-end model. _Cognitive psychology_, 36(2):73-137.
* Hinton et al. [1986] Geoffrey E Hinton et al. 1986. Learning distributed representations of concepts. In _Proceedings of the eighth annual conference of the cognitive science society_, volume 1, page 12. Amherst, MA.
* Khandelwal et al. [2018] Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 284-294. Association for Computational Linguistics.
* Koncoro et al. [2018] Adhiguna Koncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1426-1436.
* Linzen et al. [2016] Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521-535.
* Marcus et al. [1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_, 19(2):313-330.
* Melis et al. [2018] Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. _International Conference on Learning Representations_.
* Merity et al. [2018a] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018a. An analysis of neural language modeling at multiple scales. _arXiv preprint arXiv:1803.08240_.
* Merity et al. [2018b] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018b. Regularizing and optimizing lstm language models. In _International Conference on Learning Representations_.
* Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In _International Conference on Learning Representations_.
* Mikolov et al. [2010] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _Eleventh Annual Conference of the International Speech Communication Association_.
* Miller [1998] George Miller. 1998. _WordNet: An electronic lexical database_. MIT press.
* Neubig [2017] Graham Neubig. 2017. Neural machine translation and sequence-to-sequence models: A tutorial. _arXiv preprint arXiv:1703.01619_.
* Polissenska et al. [2015] Kamila Polissenska, Shula Chiat, and Penny Roy. 2015. Sentence repetition: what does the task measure? _International Journal of Language and Communication Disorders_, 50(1):106-118.
* Schuster and Paliwal [1997] Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE Transactions on Signal Processing_, 45(11):2673-2681.
* Sundermeyer et al. [2012] Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In _Thirteenth annual conference of the international speech communication association_ 

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.