Multi-timescale Representation Learning in LSTM Language Models
Authors: Shivangi Mahto
Vy Ai Vo
Javier S. Turek
Alexander Huth
Venue: ICLR 2021 Poster (2021)
Keywords: Language Model
LSTM
timescales

Abstract: Language models must capture statistical dependencies between words at timescales ranging from very short to very long. Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to a power law. However, it is unclear how this knowledge can be used for analyzing or designing neural network language models. In this work, we derived a theory for how the memory gating mechanism in long short-term memory (LSTM) language models can capture power law decay. We found that unit timescales within an LSTM, which are determined by the forget gate bias, should follow an Inverse Gamma distribution. Experiments then showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution. Further, we found that explicitly imposing the theoretical distribution upon the model during training yielded better language model perplexity overall, with particular improvements for predicting low-frequency (rare) words. Moreover, the explicit multi-timescale model selectively routes information about different types of words through units with different timescales, potentially improving model interpretability. These results demonstrate the importance of careful, theoretically-motivated analysis of memory and timescale in language models.

Published as a conference paper at ICLR 2021
MULTI-TIMESCALE REPRESENTATION LEARNING IN LSTM LANGUAGE MODELS
Shivangi Mahto'
Vy A. Vo
Department of Computer Science
Brain Iasii-Isspired Computing Lab Intel Labs
The University of Texas at Austin
Austin, TX, USA
Hillsboro, OR, USA
edu shi Snivangi@uteene
vy. vo.viintell com
Javier S. Turek
Alexander G. Huth
Brain-Inspired Computing Lab
Depts. Computer Science & Neuroscience The University of Texas at Austin Austin, TX, USA
Intel Labs
Hillsboro, OR, USA
javier. www.:etuukeeinteete
huth@cs ut exas. edu
ABSTRACT
Language models capture statistical dependencies between words timescales ranging from very short to very long. Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to power law. However, it is unclear how this knowledge can be used for analyzing or designing neural network language models. this work, derived theory for how the memory gating mechanism in long short-term memory (LSTM) language models can capture power law decay. We found that unit timescales within an LSTM, which are determined by the forget gate bias, should follow an Inverse Gamma distribution. Experiments then showed that LSTM language models trained natural English text learn to approximate this theoretical distribution. Further, we found that explicitly imposing the theoretical distribution upon the model during training yielded better language model per- plexity overall, with particular improvements for predicting low-frequeet www.frrruunccy (rare) words. Moreover, the explicit multi-timescale model selectively routes information about different types of words through units with different timescales, potentially improving model interpretability. These results demonstrate the importance of careful, mottivatl analysis of memory and timescale in language models.
INTRODUCTION
Autoregressive language models are functions that estimate probability distribution over the next word in sequence from past words, p(w//ut-1..... w1). This requires capturing statistical de- pendencies between words over short timescales, where syntactic information likely dominates (Adi et al., 2017 Linzen et al., 2016). as well long timescales, where semantic and narrative information likely dominate (Zhu al. [2018] Conneau al.) 2018 Guulordva et al. 2018). Because this probability distribution grows exponentially with sequence length, some approaches simplify the problem by ignoring long-range dependencies. Classical n-gram models, for example, assume word wy is independent of all but the last n words, with typical
(Heafield 2201 Hidden Markov models (HMMs) assume that the influence previous words decays exponentially with distance from the current word (Lin Tegmark 2016).
In contrast, neural network language models such as recurrent (Hochreiter & Schmidhuberi 1997 Merity al. 2018 Melis et al. 2018)) and transformer networks (Melis al. 2219 Krause et al. 2019; Dai et al. 2019) include longer-range interactions, but simplify the problem by working lower-dimensional representational spaces. Attention-based networks combine position and
"Current affiliation: Apple Inc.

---

Published as a conference paper at ICLR 2021
content-based information in a small number of attention heads to flexibly capture different types of dependencies within a sequence (Vaswani et al. 2017 Cordonnier et al.; 2019). Gated recurrent neural networks (RNNs) compress information about past words into fixed-length state vector (Hochreiter & Schmidhuber 1997). The influence each word has on this state vector tends to decay exponentially over time. However, each element of the state vector have a different exponential time constant, or "timescale" (Tallec & Ollivier 2018), enabling gated RNNs like the long short-term memory (LSTM) network flexibly learn many different types of temporal relationships (Hochreiter & Schmidhuber 1997). Stacked LSTM networks reduce to single layer (Turek et al. (020), showing that network depth has an insignificant influence on how the LSTM captures temporal relationships. Yet in all these networks the shape of the temporal dependencies must be learned directly from the data. This seems particularly problematic for very long-range dependencies, which are only sparsely informative (Lin & Teggarkk 2016).
This raises two related questions: what should the temporal dependencies in a language model look like? And how can that information be incorporated into neural network language model?
To answer the first question, we look to empirical and theoretical work that has explored the de- pendency statistics of natural language. [in & Tegmark (2016) quantified temporal dependencies in English and French language corpora by measuring the mutual information between tokens as a function of the distance between them. They observed that mutual information decays as a power law, i.e. MT(wa, (k+++) x t-d for constant d. This behavior is common to hierarchically structured natural languages (Lin & Tegmark] 2016 (Sainburg al.) 2019) well as sequences generated from probabilistic context-free grammars (PCFGs) (Lin & Tegmark, 2016).
Now to the second question: if temporal dependencies in natural language follow a power law, how can this information be incorporated into neural network language models? To our knowledge, little work has explored how to control the temporal dependencies learned in attention However, many approaches have been proposed for controlling gated RNNs, including updating different groups of units at different intervals (El Hihi & Bengio 1996 Koutnik al., 2014 [iu et al.. 2015 Chung et al.) 2017), gating units across layers (Chung et al. 2015). and explicitly controlling the input and forget gates that determine how information is stored and removed from memory (Xul et al. 2016; Shen et al., 2018: Tallec & Ollivier 2018). Yet none of these proposals incorporate specific shape of temporal dependencies based on the known statistics of natural language.
models.
In this work, we build on the framework of Tallec & Ollivier (2118) to develop a theory for how the memory mechanism in LSTM language models can capture temporal dependencies that follow a power law. This relics on defining the timescale of an individual LSTM unit based on how the unit retains and forgets information. We show that this theory predicts the distribution of unit timescales for LSTM models trained on both natural English (Merity al.. 2018) and formal languages (Suzgun jet al. 2019). Further, we show that forcing models to follow this theoretical distribution improves language modeling performance. These results highlight the importance of combining theoretical modeling with an understanding of how language models capture temporal dependencies over multiple scales.
2 MULTI-TIMESCALE LANGUAGE MODELS
2.1 TIMESCALE OF INFORMATION
We are interested in understanding how LSTM language models capture dependencies across time. Tallec & Ollivier (018 elegantly argued that memory in individual LSTM units tends to decay exponentially with a time constant determined by weights within the network. We refer to the time constant of that exponential decay as the unit's 's representational timescale.
Timescale directly related to the LSTM memory mechanism (Hochreiter & Schmidhuber 1997). which involves the LSTM cell state c,, input gate in and forget gate ft.
f, o Te Winhh1 + bz) ft = a (W + W + by) G. tanh(W WeshWW + Wanhe- + be) C ft O G-11 + i Es
2

---

Published as a conference paper at ICLR 2021
where 24 is the imput time h4-1 is the hidden state, Wa, Wax, W sh, Wz,, Wek: War the different weights and b;, by. be the respective biases. (--) and tanh()) represent element- wise sigmoid and hyperbolic tangent functions. Input and forget gates control the flow of information in and out of memory. The forget gate St controls how much memory from the last time step 4-11 is carried forward to the current state ct. The input gate i, controls how much information from the input at and hidden state 1 at the current timestep is stored in memory for subsequent timesteps.
To find the representational timescale, we consider "free input" regime with zero input the LSTM after timestep lo. i.e., _t 0 for > to- Ignoring information leakage through the hidden state (i.e.. assuming Weh be 0, and W fh 0) the cell state update becomes et fo C Ct-1. For > to. it can be further simplified as
fo O co fo)(t-o)) Co.
where co = Ctp is the cell state at fo, and fo = o(bb) the value the forget gate, which depends only on the forget gate bias by. Equation shows that LSTM memory exhibits exponential decay with characteristic forgetting time
1
T log fo log( + e
(2)
That is, values in the cell state tend to shrink by a factor of e every T timesteps. We refer to the forgetting time in Equation as the representational timescale of an LSTM unit.
Beyond the "free input" regime, we can estimate the timescale for a LSTM unit by measuring the average forget gate value over set of test sequences,
Test
(3)
log f
KN Ei= Ei=f Si. in which fe is the forget gate value of the unit at (-th timestep for j-th test sequence, N is the number of test sequences, and K is the test sequence length.
where =
2.2 COMRINING EXPONENTIAL. TIMESCALES TO YIELD POWER LAW
From earlier work, we know that temporal dependencies in natural language tend to decay following a power law (Lin Tegmark 2016 Sainburg et al. 2019). Yet from Equation T we see that L.TTM memory tends to decay exponentially. These two decay regimes are fundamentally different-the ratio of a power law divided by an exponential always tends towards infinity. However, LSTM language models contain many units, each of which can have different timescale. Thus LSTM language models might approximate power law decay through a combination of exponential functions. Here we derive theory for how timescales should be distributed within an LSTM in order yield overall power law decay.
Let us assume that the timescale T for each LSTM unit is drawn from distribution P(T). We want to find P(T) such that the expected value over T of the function e e approximates a power law decay -d for some constant d,
( P/T)e-t/Tal.
r- x E, 1/T
(4)
Solving this problem reveals that P(T) is an Inverse Gamma distribution with shape parameter a = and scale parameter 3 = 1 (sec Section A. for derivation). The probability density function of the Inverse Gamma distribution is given as a(
[(a) (1/T) ((B/T))
g"
This theoretical result suggests that order to approximate the power law decay of information in natural language, unit timescales in LSTM language models should follow an Inverse Gamma distribution. We next perform experiments to test whether this prediction holds true for models trained on natural language and models trained samples from formal language with known temporal statistics. We then test whether enforcing an Inverse Gamma timescale distribution at training time improves model performance.
3

---

Published as a conference paper at ICLR 2021
2.3 CONTROLLING LSTM UNIT TIMESCALES
To enforce a specific distribution of timescales in an LSTM and thus create an explicit multi-timescale model, we drew again upon the methods developed in Tallec & Ollivier (2018). Following the analysis in Section 2.1 the desired timescale Tdesired for an LSTM unit can be controlled by setting the forget gate bias to the value
by = - log(e Fdextree 1).
(5)
The balance between forgetting information from the previous timestep and adding new information from the current timestep is controlled by the relationship between forget and input gates. To maintain this balance we set the input gate bias b, to the opposite value of the forget gate, i.e., by -by. This ensures that the relation i, = ft holds true. Importantly. these bias values remain fixed (i.e. are not learned) during training, in order to keep the desired timescale distribution across the network.
EVALUATION
3.1 EXPERIMENTAL SETUP
Here we describe experimental setup. We examined two regimes: natural language data, and synthetic data from formal language. All models were implemented in pytorch (Paszke al. 2019) and the code can be downloaded from https:// Hom/HuthLab/
mu multi i -imescale--SSS
3.1.1 NATURAL LANGUAGE
We experimentally evaluated LSTM language models trained on the Penn Treebank (PTB) (Marcus al. 1999 Mikolov et al. 2011) and WikiText-2 (WT2) (Merity et al. 2017) datasets. PTB contains a vocabulary of IOK unique words, with 930K tokens in the training. 200K in validation, and 82K in test data. WT2 a larger dataset with a vocabulary size of 33K unique words, almost 2M tokens in the training set, 220K in the validation set, and 240K in the test set. As a control, we also generated a Markovian version of the PTB dataset. Using the empirical bigram probabilities from PTB, we sampled tokens sequentially until a new corpus of the same size had been generated.
We compared two language models: standard stateful LSTM language model (Merity et al. 2018) as the baseline, and our multi-timescale language model. Both models comprise three LSTM layers with I 1500 units in the first two layers and 400 units in the third layer, with an embedding size of 400. Input and output embeddings were tied. All models were trained using SGD followed by mon-monotonically triggered ASGD for 1000 epochs. Training sequences were of length 70 with a probability of 0.95 and 35 with probability of 0.05. During inference, all test sequences were length 70. For training, all embedding weights were uniformly initialized in the interval 1-0.1.0.1, All weights and biases of the LSTT layers in the baseline language model were uniformly initialized -1 1 where H is the output size of the respective layer.
between H H
The multi-timescab language model has the same initialization, except for the forget and input gate bias values that are assigned following Equation and fixed during training. For layer 1, we assigned timescale T 3 to half the units and T to the other half. For layer 2, we assigned timescales to each unit by selecting values from an Inverse Gamma distribution. The shape parameter of the distribution was set to a = 0.56 after testing different values (see A.5). The timescales in this layer had 80% of the units below 20, and the rest ranging up to the thousands. For layer 3, the biases were initialized like the baseline language model and trained (not fixed).
3.1.2 FORMAL LANGUAGE: THE DYCK-2 2 GRAMMAR
We also tested the LSTM language models on a formal language with known temporal statistics. The Dyck-2 grammar (Suzgun et al. 2019) defines all possible valid sequences using two types of parentheses. Because Dyck-2 is a probabilistic context-free grammar (PCFG), its temporal dependencies will tend to decay following power law (Lin & Tegmark 2016). Further, timescales within particular sequences can be measured as the distance between matching opening and closing parenthesis pairs. We randomly sampled from the grammar with probabilities pı 0.25, P2 0.25,
4

---

Published as a conference paper at ICLR 2021
LSTM|PTB) timescales
(Inverse Gamma)
10a
LSTM[Markov-PTB timescales
Estinated Timesscale (Tai)
Markov-PT8 best fit (narrow Gaussian)
0.8
Statistccc
101
0.6
10"
0.2
10
500
1000
1500
2000
2500
Units (sorted in order decreasing Taxe
Inverse Gamma
Narrow Gaussian y
Figure 1: Empirical timescale distributions and best distribution fits for an LSTM language model trained on natural language (PTB; blue), and the same model trained on a Markovian corpus (Markov- PTB; red). Left: An Inverse Gamma distribution is the best fit for the PTB corpus (dotted blue), while a narrow Gaussian is the best fit for the Markovian corpus (dotted red). Right: We determined the best fits by varying distribution parameters (a for the Inverse Gamma, μ for the narrow Gaussian) and computing the Volmoogorv-Smmine and theoretical distributions (lower is better). The LSTM trained natural language (blue) is best fit by the Inverse Gamma distribution with a = 1.4, while the LSTM trained on a Markovian corpus (red) is best fit by the narrow Gaussian distribution with μ 0.5.
statistic to measure the difference between the empirical
and q 0.25 to generate training, validation, and test sets with 10K, 2K, and 5K sequences. Sequences contain maximum of 200 elements.
To solve this task, we followed Suzgun et al. (2019). Models were trained to predict which closing parentheses could appear next. Elements in the sequence were one-hot encoded and fed as input to the models. We trained a baseline model and a multi-timescale version. Each model consists of 256-unit LSTM layer followed by linear output layer. The output layer reduces the dimension to 2, the number of possible closing parentheses. These output values are each converted to a (0. 1) interval with sigmoid function. Parameters were initialized using uniform distribution. The multi- tilti-timesscle model has the forget and input gate bias values assigned using an Inverse Gamma distribution with a 1.50, and fixed during training. This parameter best matched the measured distribution of timescales. Training minimized the mean squared error (MSE) loss using the Adam optimizer (Kingma & Ba (2155) with learning rate 1c-4, 31 0.9. 32 0.999, and 1e-8 for 2000 epochs.
3.2
EXPERIMENTAL RESULTS
3.2. Do EMPIRICAL LSTM UNIT TIMESCALES APPROXIMATE LANGUAGE STATISTICS?
we predicted that the distribution of unit timescales within an LSTM language model should follow an Inverse Gamma distribution. To test this prediction, trained an LSTM language model on a natural language corpus (PTB), and then trained the same model on a Markovian dataset generated with the bigram statistics of PTB (Markov-PTB)) We estimated the timescale of each unit using Equation 3 While representing PTB should require Inverse Gamma distribution of timescales, representing Markov-PTB should only require a single timescale (i.e. a delta function, which can be approximated as narrow Gaussian with o=0.1). We tested this by generating family of Inverse Gamma distributions or narrow Gaussian functions, and measuring the difference between the empirical distributions and these theoretical distributions. This was quantified with the Golmogorov-Smingg test statistic (Massey Jr 1951).
In Section 2.2
The results in Figure suggest that theoretical prediction is correct. The Inverse Gamma distribution better fits the model trained on natural language, while the approximate delta function (narrow Gaussian distribution) better fits the model trained the Markovian corpus.
5

---

Published as a conference paper at ICLR 2021
Table Perplexity of the multi-timescale and baseline models for tokens across different frequency bins for the Penn TreeBank (PTB) and WikiText-2 (WT2) test datasets. We also report the mean difference in perplexity (baseline multi-timmssale) across 10,000 bootstrapped samples, along with the 95% confidence interval (CI).
Dataset: Penn TreeBank
Model
above 10K
IK-10K
100-1K
below 100
All tokens
Baseline
6.82
27.77
184.19
2252.50
61.40
Multi-timescale
6.84
27.14
176.11
2100.89
59.69
Mean diff.
-0.02
0.63
8.08
152.03
1.71
95% CI
[-0.0,, 0.02]
[0.38, 0.88]
[6.04, 10.2]
[119.1,186.0]
[1.41, 2.02]
Dataset: WikiText-2
Baseline
7.49
49.70
320.59
4631.08
69.88
Multi-timescall
7.46
48.52
308.43
4318.72
68.08
Mean diff.
0.03
1.17
12.20
312.13
1.81
95% CI
[0.01,0.06]
[0.83,1.49]
[9.96,14.4]
[267.9,356.3]
[1.61,2.01]
3.2.2 DOES IMPOSING AN INVERSE GAMMA DISTRIBUTION IMPROVE MODEL PERFORMANCE?
The previous section showed that the empirical distribution of timescales in LSTM LM trained on natural language is well-approximated by an Inverse Gamma distribution, as predicted. However, the model had to learn this distribution from small and noisy dataset. If the Inverse Gamma distribution indeed optimal for capturing power law dependencies, then it should be beneficial to simply enforce this distribution. To test this hypothesis, we constructed models with enforced Inverse Gamma timescales, which we call multi-timescale (MTS) language models (see Section 3.1.1 for details). Compared to the baseline LSTM model, the multi- multi-timescale model incorporates a timescale prior that matches the statistical temporal dependencies of natural language. The multi- calli--timescale model is particularly enriched in long timescale units (see A.5).
If this is an effective and useful prior, the multi-timescale model should perform better than the baseline when very long timescales are needed to accomplish a task. We first compared total language modeling performance between the baseline and multi-timescale models. The overall perplexities on the test datasets are shown in the far right column of Table I The multi-timescale language model outperforms the baseline model for both datasets by an average margin of 1.60 perplexity. Bootstrapping over the test set showed that this improvement is statistically significant. Test data were divided into 100-word sequences and resampled with replacement 10,000 times. For each sample, we computed the difference in model perplexity (baseline - multi-timescale) and reported the 95 % confidence intervals (CI) in Table [ Differences significant at p 0.05 if the CI does overlap with O.
Previous work has demonstrated that common, typically closed-class, words rely mostly on short timescale information, whereas rare, typically open-class, words require longer timescale informa- tion (Khandelwal et al., 2018 Griffiths al. 2005 Rosenteld] 1994 Iyer & Ostendori 1996). To test whether the improved performance of the muuli-timescale model could be attributed to the very long timescale units, we computed model perplexities across different word frequency bins. We hypothesized that the multi-timescale model should show the greatest improvements for infrequent words, while showing little to no improvement for common words.
We divided the words in the test dataset into 4 bins depending their frequencies in the training corpus: more than 10,000 occurrences; 1000-10,00 occurrences; 100-1000 occurrences; and fewer than 100 occurrences. Then we compared performance of the models for words in each bin in Table [] The multi-timescale model performed significantly better than baseline in both datasets for the less frequent bins (bootstrap test), with increasing difference for less frequent words. This suggests that the performance advantage of the multi timescale model is highest for infrequent words, which require very long timescale information.

---

Published as a conference paper at ICLR 2021
Baseline Muiti- timescale
Dyck2
10-1
0.8
Correct
10-2
0.6
96
0.4
10-3
0.2
10-4
0.0
25
50
150 Max. Timescale Sequence
75
100
125
175
200
10
Timescale (T)
Figure 2: The left plot shows the performance of the baseline (blue) and multi-timescaee (red) models on the Dyck-2 grammar as function of the maximum timescale a sequence (20 repetitions, standard deviation). A correctly predicted sequence is considered to have all the outputs correct. The graph shows that the multi-timescale model is predicting better up to 10% more sequences in the longer timescales regime (T > 75). The right plot describe the distribution of all timescales as computed from the training data. The timescale distribution decays as a power law. The sharp decay near T' 200 is due to the limitation of the sequence lengths imposed to generate the training data.
These results strongly support our claim that the Inverse Gamma timescale distribution is a good prior for models trained on natural language that is known to have power law temporal dependencies.
3.2.3 GENERALIZATION TO FORMAL LANGUAGES
Complex dependencies in natural language data may not necessarily follow any precise statistical law, so we also turned formal languages to evaluate our hypotheses. In this experiment we used the Dyck-2 grammar, which enables us to precisely measure the distribution of timescales within the data (see Figure 2 right). We first used this capability find the best a parameter value for the Inverse Gamma distribution fitting a power law function, and then to measure model performance as a function of timescale.
Following Suzgun et al. (2019), baseline and multi-timescale language models were trained to predict closing parenthesis types. A sequence considered correctly predicted when every output from the model is correct across the entire sequence. The baseline model was able to correctly predict the output of 9..66% of test sequences, while the multi-timescale model achieved 93.82% correct. Figure 2 (left) shows percent correct as function of the maximum timescale in a sequence over 20 training repetitions. Both models succeeded at predicting most sequences shorter than 75 steps. However, for longer timescales, the multi titleiittmecalee model correctly predicted 5-10% more sequences than the baseline model. This result supplements the idea that enforcing Inverse Gamma timescales improves the model's ability capture longer-range dependencies. Nevertheless, both models far from perfect on longer timescales, raising the question of how to further improve these longer timescales.
3.2.4 ROUTING OF INFORMATION THRROGH LSTM UNITS WITH DIFFERENT TIMESCALES
Our previous results demonstrated successful control the timescale distribution in our multi-timescale model, and that this improved model performance. Breaking out performance results by word frequency also showed that the multi-timescale model was better able to capture some words- specifically, low-frequency words-than the baseline LSTM. This suggests that the multi-timescall model, which is enriched in tong-timescale units relative to the baseline model, may be selectively routing information about infrequent words through specific long-tin tin-timms units. More broadly, the multi-timescale model could be selectively routing information to each unit based on its timescale. This could make the multi-timescale model more interpretable, since at least one property of each unit-its timescale- would be known ahead of time. To test this hypothesis, we again divided the test data into word frequency bins. If long timescale information is particularly important for low frequency words, then we would expect information about those words to be selectively routed through long timescale units. We tested the importance of each LSTM unit for words in each
7

---

Published as a conference paper at ICLR 2021
Ratio of pp) with and w/o unit ablation
1.08
>10K
1K-10K
1.06
100-1K
<100
1.04
1.02
1.00
0.98
Average timescale of ablated units (words)
Figure 3: Information routing across different units of the multi-timescale LSTM for PTB dataset. Each line shows results for words in a different frequency bin (i.c. rare words occur < 100 times, red line). The ordinate axis shows the ratio of model perplexity with and without ablation of group of 50 LSTM units, sorted and grouped by assigned timescale. Ratios above indicate decrease in performance following ablation, suggesting that the ablated units important for predicting words in the given frequency bin. Abscissa shows the average timescale of each group (left is longer timescale).
frequency bin by selectively ablating the units during inference, and then measuring the effect on prediction performance.
We divided the LSTM units from layer 2 of the multi-timescale model into 23 groups of consecutive units, sorted by assigned timescale. We ablated one group of units at time by explicitly setting their output to O, while keeping the rest the units active. then computed the model perplexity for different word frequency bins, and plotted the ratio of the perplexity with and without ablation. If performance gets worse for particular word frequency bin when ablating a particular group of units, it implies that the ablated units important for predicting those words.
Figure 3 shows this ratio across all frequency bins and groups for the PTB dataset (similar results for WikiText-2 shown in the supplement). Ablating units with long timescales (integrating over 20-300 timesteps) causes performance to degrade the most for low frequency words (below 100 and IK-10K occurences); ablating units with medium timescales (5-10 timesteps) worsens performance for medium frequency words (1K-10K occurrences); and ablating units with the shortest timescales (<1 timestep) resulted in worse performance on the highest frequency words. These results demonstrate that timescale-dependee information is routed through different units in this model, suggesting that the representations that are learned for different timescales are interpretable.
CONCLUSION
In this paper we developed a theory for how LSTM language models can capture power law temporal dependencies. We showed that this theory predicts the distribution of timescales in LSTM language models trained on both natural and formal languages. We also found that explicit multi-timescale models that are forced to follow this theoretical distribution give better performance, particularly over very long timescales. Finally, we show evidence that information dependent on different timescales routed through specific units, demonstrating that the unit timescales are highly interpretable. This enhanced interpretability makes it possible to use LSTM activations to predict brain data, as in (Jain & Huth| 2018), and estimate processing timescales for different brain regions (Jain et al.) 2020). These results highlight the importance of theoretical modeling and understanding of how language models capture dependencies over multiple timescales.
8

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank Shailee Jain for valuable feedback on the manuscript and useful discussions, and the anonymous reviewers for their insights and suggestions. Funding support for this work came from the Burroughs Wellcome Fund Career Award at the Scientific Interface (CASI), Intel Research Award, and Alfred P. Sloan Foundation Research Fellowship.
REFERENCES
Yossi Adi, Einat Kermany. Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learn- ing Representattinns, 2017. URL notteId-BJh62t.xl
Junyoung Chung, Çaglar Güilçehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks, In ICML, pp. 2067-2075, 2015. UR//epp://ppooeedigg... m.l:..mmm.... press/
v37/ htt//chung1.....
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In Proceedings of the 5th International Conference on Learning Representations, 2017. URL
httpps://on /tpprreveeeeeeviee net/ forum?dd-Slai
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 2126-2136 2018. URL https: prstps://wwwy wwww aww.eb. org. www.ooogggngoggogoooooooooooooo.
P18 1198//
Jean- Ban--Batsstee Cordonnier, Andreas Loukas, and Martin Jaggi. the relationship between self- attention and convolutional layers. In International Conference Learning Representations, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019. URL
https: www.aclweb.c org/ http///ahhoooyyyyeeoeoo
Salah El Hihi and Yoshua Bengio. In Advances in neural information processing
Hierarchical recurrent neural networks
for long-term dependencies.
URL http: http://paaaasssssm nips http://ppppppcccppcp..ppppppppee systems, pp. 493-499, 1996.
1102 www-hhierrccccaa..o www.rrrccica---eureeeetttt recurrent neeuurrent-nuuual -ttprrreesn-neeaaaanttto-------- tom tem. pdf
Thomas L Griffiths, Mark Steyvers, David M Blei, and Joshua B Tenenbaum. Integrating topics and syntax. In Advances in neural information processing systems, pp. 537-544, 2005.
Kristina Gulordava, Piotr Bojanowski, Édouard Grave. Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1195-1205, 2018. URL http:::/pw/// www.waaclwee.. aclweb
otg/anthology//e NN1hhooooyy/1111-1 1108/
Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Proceedings the sixth workshop on statistical machine translation, pp. 187-197, 2011.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8) 1735-1780, 1997. doi: 10.1162/neco. 1997.9.9..... 9..7..9..11. 1735. URL htttps:tt////:otoorrrt
nece 1997 9.
1735
Rukmini Iyer and Mari Ostendorf. Modeling long distance dependence in language: Topic mixtures vs. dynamic cache models. In Proceeding of Fourth International Conference on Spoken Language Processing. ICSL.P'96, volume pp. 236-239. IEFE, 1996.

---

Published as a conference paper at ICLR 2021
Shailee Jain and Alexander Huth. ing models for fmni. In Advances in neural information processing sys- tems, pp. 6628-6637, 2018. 1897-incorporatiin cot intointooo-- -nd
Incorporating context into language encod-
URL http://papers..ott wwt.::iisss..ppppoopoeooeoo ccppaper/ppppppappcppppccc...
wwwtext-
for- fmri
Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexander Huth. Interpretable multi-timescale models for predicting fmri responses to continuous natural speech. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 13738-13749. Curran As- sociates, Inc., 2020. URL 9e9a30b74c49ddi cot.
neurips ips cc
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 284-294, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P11//////1..// 1027. URL https: 7/ www. www.wwwwbbbb.. org/ http./////ttoooooga/-----000
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp
7/ 7/arxiv. otg/ abs /112 6980
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. clockwork mn. In Interna- tional Conference on Machine Learning, pp. 1863-1871, 2014. URL http::/pp/cceeeddd ngs
ml r. www.preessv222 2ttsss/v32/koutnii/ koutnikikk k144 html
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. arXiv preprint arXiv:1904.08378, 2019. URUhttps://arxiv.oorm
abs/ 8378
Henry W Lin and Max Tegmark. Critical behavior from deep dynamics: a hidden dimension in
natural language. arXiv preprint wr.ivv:/00..0733//7 2016. URL httt:://////// ctte:////ommmmmmm mi edu/ sites/ fau /t weallee/pppe wtsp//pblcaaatioss/sooaaaaca wwwp://66666...0. 06737 pdf
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg, Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535,
2016. URL htttts:://wwtt http:w/wclleeel. aclweb. org/ antholoogy/01611000 103777
Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuan-Jing Huang. Multi-timescale long
short-term memory neural network for modelling sentences and documents. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2326-2335, 2015.
URL https: accl web. wew.org.....................oo.. wwwpr///nnoooggy/.....
Mitchell Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. Treebank-3,
Linguistic Data Consortium, Philadelphia, 14, 1999. URL http:tttt///aaalcoca 1dc. upenn
edd/LDDC999
Frank J Massey Jr. The kolmogonoy-smirnoy test for goodness of fit. Journal of the American statistical Association, 46(253):68-78, 1951.
Gábor Melis, Tomáš Kočiský, and Phil Blunsom. Mogrifier Istm. International Conference on Learning Representations, 2019. URL
wttp://wwwweeeeoooee.eeee $.e55P on
Revue<20news
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language
models. In International Conference on Learning Representations, 2018. URL https:
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. In International Conference on Learning Representations, 2017. URL bttps:/ openreview wwt.:iewwwe//oommmmme// Ft
10

---

Published as a conference paper at ICLR 2021
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018. URL https
//openreview.net/ www.:nneeeooomidim/mmnemamm
Tomášs Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, and Jan Černocký, Empirical evaluation and combination of advanced language modeling techniques. In Twelfth Annual Conference of the International Speech Communication Association, 2011. URLhttps: httpp://ss//w 0w0..6055.....
isca-speech. org/ wwwhaveechieieeeeee http://tttteeeeeeeeeeeeee
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: /tttt://papers...o neuriss Seevv-sttllee styls-higl-eeeeeeeeeeeee perforheeoorrancceeeeeeeeeeeeeee
wwtprriisscccppppppe cc/paper/
9015 ayytorch-an-iperrree pdf
learning Library.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pupers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1202. URL
Ronald Rosenfeld. A hybrid approach adaptive statistical language modeling. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA SCHOOL OF COMPUTER SCIENCE, 1994.
Tim Sainburg, Brad Theilman, Marvin Thielk, and Timothy Q Gentner. Parallels in the sequential organization of birdsong and human speech. Nature communications, 10(1):1-11, 2019. URL
https://www wwwpw//wwwww...m/mm nature. hwtpm///rraaceeee111100111---11-
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Represen- tations, 2018. URL bttps://ppenreview..eee http://ww.eettrrooommomeelaallaa &source- Most_page---s
Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov 12, and Staart M Shieher. Lstm networks can perform dynamic counting. 2019, pp. 44, 2019.
Corentin Tallec and Yann Ollivier Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018. URL
Ad=SJckhk-Ab
Javier Turek, Shailee Jain, Vy Vo, Mihai Capotă, Alexander Huth, and Theodore Willke. Approxi- mating stacked and bidirectional recurrent architectures with the delayed recurrent neural network. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9648-96489 PMLR, 13-18 Jul 2020. URL httpattp:p//rocceeeieee /ptpe//e/ooeeeee..//./.. mmr http://eee.............eeee///es
html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jiacheng Xu, Danlu Chen, Xipeng Qiu, and Xuan-Jing Huang. Cached long short-term memory neural networks for document-leva sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1660-1669, 2016. URL https
7/www. a web. org/ / antho www./ggll...............gg/..b.. 6/////1161111///111
Xunjie Zhu, Tingfeng Li, and Gerard De Melo. Exploring semantic properties of sentence em- beddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 632-637, 2018. URL wwwwwwww.. ac aclweb.org/
anthology/PI8 2100/
11

---

Published as a conference paper at ICLR 2021
A SUPPLEMENTARY MATERIAL
A. DERIVATION OF INVERSE GAMMA TIMESCALE DISTRIBCTION
In Section 2.2 we examined how exponentials can be additively combined to yield a power law. Here we give detailed derivation of how choosing exponential timescales according to the Inverse Gamma distribution with shape parameter d and scale parameter 3 1 yields power law decay with exponent d.
Recall that our goal identify the probability distribution over exponential timescales P(T) such that the expected value over of the exponential decay function e & approximates a power law decay t- with some constant d,
000 //²0
x Erle Eyle-
P(T)e id.
Noting its similarity to the integral we attempting solve, we begin with the definition of the Gamma function I(a).
r(a)
(6)
Now we perform the substitution u(T) t/ 'T inside the integral, assuming that and T' >> 0. This substitution makes du d/d and also changes the limits of integration from (0, oo) to (u(0), u(oo)) (oo, 0), giving
r(a)
1
1 T
e
T2 aT)
(7)
e (ta-1) (t) (T-att) e +dr
ta fo T i
(9)
Now we set a = d and isolate the polynomial in I, giving
r(d)-1 / T --d-le dT.
(10)
This form closely resembles Equation suggesting that P(T) = -(d) However, note that we did not rigorously define the domain of t in the original formulation. In reality, we are only concerned with i 2 1, since 0 S for values of / near zero. We thus neither want nor need to approximate - for 0 S l < 1.
corresponds to distances of less than word, and 1-d explodes
We can adjust Equation which is defined for t. 2 0, to reflect this limitation by simply substituting t - s + 1, giving
(s (s+1)-d = I(d)-1 1 T---1e 42 dT 0
(11)
r(d)-1 / T- - 8 daT
(12)
( P(T)e- $ dT, where
(13)
T-d-1 - + InverseGammaTT:a d.3=1) 1).
P(T)
I(d) e
(14)
A.2 RELATIONSHIP BETWEEN FORGET GATE AND TIMESCALE
In Section 2.3 we showed that the forget gate bias controls the timescale the unit, and derived a distribution of assigned timescales for the nulti-timescale language model. After training this model,
12

---

Published as a conference paper at ICLR 2021
10
Estimated
106
Assigned
10
Timescale
10
103
102
10a
100
10-1
0
200
400
600
800
1000 1200
Units
Figure 4: Estimated timescale is highly correlated with assigned timescale, shown for all 1150 units in LSTM layer 2 of the multi-timescal language model.
we tested whether this control was successful by estimating the empirical timescale of each unit based shows that the assigned and estimated timescales in layer 2 are strongly correlated. This demonstrates that the timescale of an LSTM unit can be effectively controlled by the forget gate bias.
on their mean forget gate values using Equation [ Figure Ssowss
Figure I shows estimated timescales in the LSTM language model from Merity et al. (0118) trained on Penn Treebank (Marcus al. 1999 Mikolov et 2011). These timescales lie between 0 and 150 timesteps, with more than 90% of timescales being less than 10 timesteps, indicating that this network skews its forget gates to process shorter timescales during training. This resembles the findings by Khandelwal et al. (2018), which showed that the model's 's sensitivity is reduced for information farther than 20 timesteps in the past. Idcally, we would like to control the timescale of each unit to counter this training effect and select globally distribution that follows from natural language data.
A.3 FORGET GATE VISUALIZATION
To further examine representational timescales, next visualized forget gate values of units from all three layers of both the multi-timescas and baseline language models as described in Section A.3 The goal is to compare the distribution of these forget gate values across the two language models, and to assess how these values change over time for given input.
First, we sorted the LSTM units of each layer according to their mean forget gate values over test sequence. For visualization purposes, we then downsampled these values by calculating the average forget gate value of every 10 consecutive sorted units for each timestep. Heat maps of these sorted and down-sampled forget gate values are shown in Figure [5] The horizontal axis shows timesteps (words) across sample test sequence, and the vertical axis shows different units. Units with average forget gate values close to 1.0 (bottom) are retaining information across many timesteps, i.e. they are capturing long timescale information. Figure 5 shows that the baseline language model contains fewer long timescale units than the distributed across the layers than the multi-timescaa language model. Figure 5b also shows the (approximate) assigned timescales for units in the multi-timescall language model. As expected, layer contains short timescales and layer 2 contains a range of both short and long timescales. Layer 1 units with short (assigned) timescales have smaller forget gate values across different timesteps. In layer 2, we observe that units with large assigned timescale have higher mean forget gate values across different timesteps, for example the units with assigned timescale of 362 in 5b have forget gate values of almost 1.0 across all timesteps. Similar to the previous analysis, this demonstrates that our method is effective controlling the timescale of each unit, and assigns different distribution of timescales than the baseline model.
language model. They are also more evenly
A.4 WORD ABLATION
Another way to interpret timescale of information retained by the layers is to visualize the decay of information in the cell state over time. We estimated this decay with word ablation experiments as described in Section A.3.
13

---

Published as a conference paper at ICLR 2021
LSTM layer
LSTM layer 2
0.6
Hidden Units
forget
retain
LSTM layer 3
61
Timesteps
Timesteps
Timesteps
(a) Baseline Language Model
LSTM layer
LSTM layer 2
3.0
0.0
3.3
0.1
3.2
0.5
Assigner Times (mmeg ESC
3.3
1.0
forget<......................... seegtt
3.2
1.6
3.4
1.9
3.5
3.8
5.3
LSTM layer
3.7
10.9
4.0
23.9
3.9
54.2
4.0
362.2
~>>
6/
6
6>
Timesteps
Timesteps
Timesteps
(b) Multi-timescale Language Model
Figure 5: Forget gate values of LSTM units for test sentence from the PTB dataset. Units are sorted top to bottom by increasing mean forget gate value, and averaged in groups of 10 units enable visualization. Figure 5b) also shows average assigned timescale (rounded off) the units.
14

---

Published as a conference paper at ICLR 2021
Ac- normalized L2 norm
LSTM layer 1
LSTM layer 2
LSTM layer 3
10
10-1
10-2
Baseline Multi-timescale
10-3
100
10a
102
100
10t
102
100
10a
10
Ablated word pos
Ablated word pos
Ablated word pos
(a) PTB dataset
Åc- normalized L2 norm
LSTM layer
LSTM layer
LSTM layer 3
100
10-1
10
Baseline Multi-timescale
10-3 3
100
10
102
10
10a
102
10
10s
102
Ablated word pos Ablated word pos
Ablated word pos
(b) WikiText-2 dataset
Figure Change in cell state of all the three layers for both the baseline and Multi-timescale language models in word ablation experiment. A curve with a steep slope indicates that cell state difference decays quickly over time, suggesting that the LSTM layer retains information shorter timescales.
15

---

Published as a conference paper at ICLR 2021
In Figure 6 we show the normalized cell state difference averaged across test sentences for all three layers of both baseline (blue) and multi-timescale (orange) models. In the PTB dataset, information in layer of the baseline model decays more slowly than in layer 2. In this case, layer 2 of the baseline model retains shorter timescale information than layer 1. In the WikiText-2 dataset, the difference between layer and layer 2 of the baseline model is inverted, with layer 2 retaining longer timescale information. However, in the multi-timescale model the trend is nearly identical for both datasets, with information in layer decaying more slowly than layer This is expected for our multi-timescale model, which we designed to have short timescale dependencies in layer and longer timescale dependencies layer 2. Furthermore, the decay curves are very similar across datasets for the multi-timescale model, but not for the baseline model, demonstrating that controlling the timescales gives rise to predictable behavior across layers and datasets. Layer has similar cell state decay rate across both models. In both models, layer 3 is initialized randomly, and we expect its behavior to be largely driven by the language modeling task.
Next, explored the rate of cell state decay across different groups of units in layer of the multi-timescale language model. We first sorted the layer 2 units according to their assigned timescale and then partitioned these units into groups of 100 before estimating the cell state decay curve for each group. As can be seen in Figures 7 and g units with shorter average timescale have faster decay rates, whereas units with longer average timescale have slower information decay. While the previous section demonstrated that our manipulation could control the forget gate values, this result demonstrates that we can directly influence how information is retained the LSTM cell states.
Ac(2) for different groups
100
6.7K
Avg 50.6
Avg 9.8
10
6.0
Avg 4.0
Avg 2.7
10
Avg 1.4
Avg 0.6
10-3
Avg 0.3
10
10
102
Ablated word position
Figure 7: Change cell states of 100-unit groups having different average timescale of layer 2 in Multi-timescab model in word ablation experiment for PTB dataset. As the assigned timescale the group decreases the slope of the curve decreases indicating retained information of smaller timescale.
Ac(2) for different groups
100
Avg 50.6
Ave 9.8
10-1
6.0
4.0
Avg 2.7
10 2
Avg 1.4
Avg 0.6
10-3
0.3
All units
10
10
10a
Ablated word position
Figure 8: Change in cell states of 100-unit groups having different average timescale of layer 2 in Multi-timescale model in word ablation experiment for WikiText-2 dataset.
A.5 SHAPE PARAMETER FOR INVERSE GAMMA DISTRIBUTION
We compared the performance of multi- timescale language model for different shape parameters in inverse gamma distribution. Figure shows timescales assigned to LSTM units of layer
16

---

Published as a conference paper at ICLR 2021
corresponding to different shape parameters. These shape parameters wide range of possible timescale distribution to the units. Figure 10 shows that multi-timescale models performs best for 0.56.
10
a- 0.20
1013
a=0.45
Alloted Timescale
0.56
1010
a=0.80
a=1.40
107
10
10
200
400
600
800
1000
1200
Units
Figure 9: Assigned timescale to LSTM units of layer2 of multi-timescale language model correspond- ing different shape parameter a.
Model Performance
68
66
PTB
WikiText-2
64
62
60
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Shape parameters
Figure 10: Performance of multi-timescale models for different shape parameters a both PTB and WikiText-2 dataset.
A.6 SELECTING THE TIMESCALES FOR EACH LAYER
With the purpose select proper timescales to each layer in Section 3.1, we conducted experiments on designing LSTM language models with different combinations of timescales across the three layers. We found that layer (the closest layer to input) always prefers smaller timescales within the range from 1 to 5. This is consistent with what has been observed in literature: the first layer focuses more on syntactic information present in short timescales Peters et al. (2018); Jain & Huth (2018). We also observed that the layer i.e., the layer closest to the output, does not get affected by the assigned timescale. Since we have tied encoder-decoder settings while training, layer 3 seems to learn global word representation with specific timescale of information control the training task (language modeling). The middle LSTM layer (layer 2) was more flexible, which allowed us to select specific distributions of timescales. Therefore, in Section 3.1 by setting layer biases to small timescales, layer to satisfy the inverse gamma distribution and thus aim to achieve the power-law decay of the mutual information, and layer 3 with freedom to learn the timescales required for the current task.
achieve the Multi-timescale Language Model
A.7 INFORMATION ROUTING EXPERIMENTS ON WIKITEXT-2 DATASET
We also performed the information routing experiments for multi-timescale model trained WikiText-2 dataset. Figure [T shows uimescale-eppnddent routing in the model, same as what observed for PTB dataset Section 3.2.4.
17

---

Published as a conference paper at ICLR 2021
Ratio of ppl with and w/o unit ablation
1.08
>00K
1K-10K
1.06
100-1K
<100
1.04
1.02
1.00
0.98
Average timescale of ablated units (words)
Figure 11: Information routing across different units of the multi-timescale LSTM for WikiText-2 dataset.
Table Perplexity of the baseline and multi-timescale models over different training instances. Values are the mean and standard error over the training instances.
Dataset
Model
Performance
Baseline
61.64 0.28
PTB
Multittmmesclee
59.63 0.18
Baseline
70.23 + 0.24
WikiText-2
Multi-timescale
68.33 0.12
A.8 ROBUSTNESS OF MODEL PERFORMANCE
We quantified the variability in model performance due to stochastic differences in training with different random seeds. Table shows the mean perplexity and standard error across different training instances. The variance due to training is similar across the two models.
A.9 MODEL COMPARISON REPRODUCING A PREVIOUS REPORT
Previously, the LSTM baseline we used in this work was reported achieve perplexity 58.8 without additional fine-tuning (Merity et al. 2018). To more closely reproduce this value, we retrained the models using the legacy version of pytorch (0.4.1) used to train the LSTM in the original paper. We report both the overall perplexity and the performance in each word frequency bin (Table (3). While the overall perplexity still falls short the original value, we speculate this could be due the CUDA version, which was not reported. Still, we find that the multi-tinescale model significantly improves performance for the the most infrequent words, mirroring the results reported in the main text.
We also performed the information routing experiment on the multi timescale model trained with the older version of pytorch (Figure 12). Our results very similar those reported in the main text in Section 3.2.4, which demonstrate that timescale- deeendeet information is routed through different units.
18

---

Published as a conference paper at ICLR 2021
Table 3: Perplexity of the baseline and multi-timescale models trained with legacy version of pytorch. Performance is also reported separately for tokens across different frequency bins. Last row the mean difference in perplexity (baseline multi-timescale, across 10,000 bootstrapped samples, along with the 95% confidence interval (CI).
Dataset: Penn TreeBank
Model
above 10K
IK-IOK
100-1K
below 100
All tokens
Baseline
6.59
27.04
178.62
2069.81
58.98
Mulsi-timescale
6.79
26.68
167.97
1902.37
57.58
Mean diff.
-0.2
0.37
10.65
167.27
1.4
95% CI
[-0.27,-0.14]
[-0.02,0.73]
[7.77,13.78]
[120.22,215.7]]
[0.91,1.86]
Ratio of ppl with and w/o unit ablation
1.08
>10K
1K-10K
1.06
100-1K
<100
1.04
1.02
1.00
0.98
Average timescale of ablated units (timesteps)
Figure 12:
Information routing across different units of the multi-timescale LSTM trained with legacy version of pytorch the Penn Treebank dataset.
A. THE DYCK-2 GRAMMAR
The Dyck-2 grammar (Suzuun as follows:
2019 given by probabilistic context-free grammar (PCFG)
S
with probability P1
[S]
with probability p2
S
SS
with probability
with probability 1 - (p) + p2 + g)
where 0 < Pl, P2.9 and P1 p2 1.
19


Reviews:
[{"review_id": "jG0Z5eBi4L", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis work investigates representational power of LSTM to model natural language, in particular how well it models temporal dependencies within text. They define a notion of timescale of each LSTM unit and analitycally show that LSTM memory exhibits exponential decay, while natural language tends (based on prior work) to decay following the power law. Based on this, they figure that LSTM memory may decay following the power law *if the timescales approximate samples from the particular Inverse Gamma Distribution*. To achieve that they propose the multi-timescale LSTM unit, where the desired timescale is explicitly controlled via the forget gate bias.\n\nAuthors empirically validate their theoretical claims and show improvements in language modeling (PTB, Wikitext2) over the baseline LSTM using the proposed multi-timscale LSTM. Importantly, they show how multi-timescale LSTM gives improvement in modeling rare words, which are known to require longer temporal dependencies.\n\n## Strong points\n\n1. This work investigates the important (though not that popular) question of discrepancy between the temporal dependencies existing in natural language and the abilities of models we use to learn these dependencies in practice.\n2. The idea of including explicit control of the timescale (i.e. temporal horizon) of each LSTM unit is interesting and well-motivated.\n3. Experiments use a formal language too in addition to natural language modeling which allows to check if proposed approach generalizes in case of exactly computable timescale distribution.\n\n## Weak points\n\n1. There is **no code** available. I was interested in the way how test set bootstraping was performed in the experiments (see comments below for details) and found out there is no code, which is really sad. I hope authors will submit the reproducible code in the near future.\n2. Theoretical part gives some essential quantities while no derivations are given. Given that there is some free space left in the paper and the fact of unlimitied supplement material I don't see any reason to omit derivations (I struggle with some transitions between equations as you may found in the comments below).\n3. I am not sure how useful is the proposed approach on new tasks given hardly tuned hyperparameters for the Inverse Gamma Distribution proposal and LSTM model architecture for each task (more details in the comments below).\n\n## Recommendation\n\nI vote for accepting **upon fixing major weak points**: uploading reproducible code with experiments and adding all derivations necessary for essential theoretical claims in this work. Overall this is decent work which will be useful for future research in studying representational power of models we are using to learn complicated dependencies of natural language.\n\n## Questions\n\nAll the questions below are welcome to be used as **suggestions** to provide more details in the manuscript.\n\n### Theoretical part\n\n1. Eq.3: why can we simply average forget gates? Out of 'free input' regime $c_t$ from eq.1 would have more dependencies. Could you elaborate why can we estimate it like this. \n2. How to solve Eq.4 as getting Inverse Gamma Distribution? I struggle to find an obvious/trivial solution, please elaborate this in the manuscript.\n\n### Experiments\n3. From 3.1.1. '*Training sequences were of length 70 with a probability of 0.95 and 35 with a probability of 0.05. During inference, all test sequences were length 70.*' Why are you making such explicit scheduling? Given that each training sequence from WT2 is some excerpt from wikipedia (often longer than 70 words), how do you deal with tails of sequences? More detailed description of data loading will be helpful.\n4. Table 1: from my understanding columns with rare words attract most interest, and I wonder if you could add the varaiance among different training instances in Table 1 like you reported in the appendix (Table 2)? Or refactor Table 2 such that it has same freq based columns.\n5. Did you think of tokenization other than word-level? BPE for example: it gives more balanced token distribution for WT2 due to absence of UNK token there. I wonder if improvements in terms of rare words PPL will hold. How do you think (speculatively)?\n6. PTB results: >10K bin PPL got higher with your approach (also ratio drop below 1 in routing study). Why do you think this happens? Why do you think this does not happen with WT2 task? As I see with PTB the fixed forget bias underestimates true/gold high freq word probabilities on average, but why? \n\n### Other\n\n7. Is it possible to estimate/learn the IGD alpha parameter from the data itself of the task you work on? This grid search you provide makes me less convinced in how useful this approach is for the new task, where the $\\alpha$ is not known.\n8. How important is the model layers tuning you do? E.g. only specific layers have fixed forget bias, but others not, **why is that?** I am really interested in knowing if other ways of defining your model hurt the performance or keep it on the baseline level? I am sure this will be useful for all other readers too.\n9. Is it possible to apply this timescale control for other units e.g. GRU (no explicit forget gate)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "geg1PEZQPtL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out the relationship between words in natural language usually follow the power law. Gated recurrent neural networks such as LSTMs excel in modelling natural language, however, the forgetting mechanism of LSTMs  is ruled by the exponential decay. This work demonstrates a way to engineer the forgetting mechanism of LSTMs to mimic the power law relationship that is more presented in natural language. By applying their technique, the modified LSTM model can do better in modelling rare tokens, which usually span for longer timescales, hence the model can score lower perplexities on less frequent words. The key contribution of the paper is the derivation which shows that the forget gates of LSTMs are subject to exponential decay in zero-input regime after the first input token is given. And the expected value of exponential decay functions exp(-t/T) can approximate the power law when T is sampled from the Inverse Gamma distribution. \n\nThe experiments demonstrate that that drawing T from the Inverse Gamma distribution is a natural fit for natural language. Then, the authors propose a multiscale LSTM model that exploits this property. Each timescale T is drawn from the inverse Gamma distribution, which essentially becomes a forget bias term and is fixed during the training. Multiple Ts are drawn to mimic the power law. The multiscale LSTM captures the right inductive bias to perform better in modelling less frequent words which might be useful to keep in the memory for longer time. The paper is well written and both the motivation and explanation of the approach are clear. The experiments are appropriately designed, and the results support the main claim nicely.\n\nI do have some comments and questions on what's written in the paper though.\nIt is conventional to use notation h_{t-1} in the update rules for the input, forget and output gates since h_t is obtained from c_t, or at least clarify the update rule of the hidden state as h_{t+1} = o_t * tanh(c_t).\n\nInstead of not learning the forget bias at all, have you tried adding a regularisation loss that forces the forget bias term to stay closer to the prior (initially drawn T from the Inverse Gamma distribution)? \n\nWhat is the motivation of using smaller hidden size for the topmost layer for both the baseline and the multiscale LSTM?\n\nFor the multiscale LSTM, why were the timescales not sampled from the Inverse Gamma distribution and fixed during the training for the topmost layer? But instead, they were learned like the standard LSTM?\n\nHow were T=3 or T=4 chosen for the first layer of the multiscale LSTM? Were they found by evaluating performance on the validation set or were they chosen based on domain knowledge?", "extra_scores": {}, "extra_text": {}}, {"review_id": "a1e3B8PAKrA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel variant of LSTM by analyzing its behavior against\nscale-free distributions generally found in natural languages. Since the\nprediction of LSTM is essentially a convolution over each hidden unit, the\nauthors derived that the bias parameter should obey an inverse Gamma \ndistribution. This is a very neat and interesting result, which is also \nvalidated by a number of experiments in natural languages with scale-free\ndistributions and artificially generated corpus with non scale-free \ndistributions. \n\nMy only question is the setup of the proposed LSTM: in Section 3.1.1, the\nauthors say that the first layer of LSTM has a fixed timescale, and only the\nsecond layer has an inverse Gamma bias parameters. The third layer does not\nhave inverse-Gamma distribution and simply optimized.\nIs this architecture necessary for the result? If so, why the third layer\nshould not have the proposed inverse-Gamma time scales?\nFinally, in Figure 3, infrequent words actually use longer time scales, but\nthey also leverage short scales (i.e. red lines are U-shaped, not linear for\nlonger scales). I would like to know why this phenomenon happens.\n\nThat being said, this is a very interesting paper leveraging the structure of\nLSTM and scale-free property of natural languages. In addition to Dyck \nexperiments, some other languages, such as a generation from PCFG or some\nprogramming languages might be also interesting for experimentation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XbwJSltVS5V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "With strong arguments I could be persuaded to change my score.\n\nThe paper investigates how well LSTM language models learn the known statistical temporal dependency distributions of languages, which follows a power law. The authors deduct the expected distribution of LSTM unit timescales and present a method for manually controlling them by setting the forget gate bias using an inverse Gamma distribution. They show that while the timescales of a standard LSTM LM follow the expected distribution, manually controlling it instead of learning it still increases LM performance, especially for infrequent words. The findings are also validated on a formal language, where the time scales are explicitly known.\n\nWhat this paper excels at is a theoretical formulation of the proposed method and a motivation grounded in real contemporary problems in the field of modeling sequential data. However, I am not confident that the baselines and test suite employed by the authors is sufficient for me to accept their hypothesis.\n\nThe LSTM baselines in Table 1 are about 5 ppl points below the reportings of the paper they use as a baseline. Thus the authors \"improvement\" is actually significantly below the original paper from almost three years ago,which is a long time in this field, current models are now below half of the reported perplexity.\n\n\"Test data were divided into 100-word sequences and resampled with replacement 10,000 times. For each sample, we computed the difference in model perplexity (baseline − multi-timescale) and reported the 95% confidence intervals (CI) in Table 1. Differences are significant at p < 0.05 if the CI does not overlap with 0.\" - This test suite is unfamiliar to me. Please link a paper which uses the same test suite or follow the standard practices in the field of language modeling. Also if the authors are looking to argue for better handling longer time sequences, why cut samples off at 100 words? Shouldn't your model be better at longer sequences?\n\n\"This suggests that the performance advantage of the multi-timescale model is highest for infrequent words, which require very long timescale information\" - It is not intuitive to me that rare words have more long-term dependencies than non-rare words? Also, I am unsure if the improvement in performance on rare words might just be attributed to the fact that the baseline model is significantly underperforming the original (Merity et al., 18) model.\n\nDo you have confidence intervals for the DYCK language? Training models on DYCK-2 can be highly seed dependant. 5-10% improvement in performance seems like it could just be random noise.\n\nI don't get Figure 3.\n\nIn general the paper is well-written and easy to understand.\n\nUPDATE:\n\nI do not believe that Merity 18's results are not reproducible. Just look at the vast amount of work after the publication that builds on top of the results. I don't doubt that your way of evaluating the performance of language models can lead to hypothesis testing, but it is not what the field employs thus your results are not comparable to others, which is why I cannot accept this paper. If your method really does work as well as you argue it should be no issue obtaining an improvement over the actual baseline.\n\nUPDATE 2:\n\nThe new results with Merity's original benchmark leads me to increase my score from 4 to 7. I appreciate the effort in reproducing Merity's results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes matching the distribution of biases for an LSTM to estimates of long range mutual information from analyzing the statistics of languages. The authors shows empirical evidence that LSTMs seem indeed to be following such a distribution, using natural language and Dyck-2 grammar. They show that explicitly enforcing the distribution of biases in learning can actually help LSTM language models.
The reviewers had slight concern about some of the baseline numbers reported, but the authors took the time to address those concerns. Overall, it was an interesting and thought provoking paper that can provide a useful angle to consider when building recurrent models for a problem -- namely that of matching the properties / inductive bias of the model to that of the data.
Decision: Accept (Poster)