Before revision:  In this work, we construct explicitly multi-timescale language models by manipulating the input and URLet gate biases in a long short-term memory (LSTM) network. The distribution of timescales is selected to approximate power law statistics of natural language through a combination of exponentially decaying memory cells. We then empirically analyze the timescale of information routed through each part of the model using word ablation experiments and URLet gate visualizations. These experiments show that the multi-timescale model successfully learns representations at the desired timescales, and that the distribution includes longer timescales than a standard LSTM.
After revision:  In this work, we construct explicitly multi-timescale language models by manipulating the input and URLet gate biases in a long short-term memory (LSTM) language models can capture power law decay. We found that unit timescales within an LSTM, which are determined by the URLet gate bias, should follow an Inverse Gamma distribution. Experiments then showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution. Further, we found that explicitly imposing the theoretical distribution upon the model during training yielded better language model perplexity overall, with particular improvements for predicting low-frequency (rare) words. Moreover, the explicit multi-timescale model successfully learns representations at the desired timescales, and that the distribution includes longer timescales than a standard LSTM.