Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a novel variant of LSTM by analyzing its behavior against
scale-free distributions generally found in natural languages. Since the
prediction of LSTM is essentially a convolution over each hidden unit, the
authors derived that the bias parameter should obey an inverse Gamma 
distribution. This is a very neat and interesting result, which is also 
validated by a number of experiments in natural languages with scale-free
distributions and artificially generated corpus with non scale-free 
distributions. 

My only question is the setup of the proposed LSTM: in Section 3.1.1, the
authors say that the first layer of LSTM has a fixed timescale, and only the
second layer has an inverse Gamma bias parameters. The third layer does not
have inverse-Gamma distribution and simply optimized.
Is this architecture necessary for the result? If so, why the third layer
should not have the proposed inverse-Gamma time scales?
Finally, in Figure 3, infrequent words actually use longer time scales, but
they also leverage short scales (i.e. red lines are U-shaped, not linear for
longer scales). I would like to know why this phenomenon happens.

That being said, this is a very interesting paper leveraging the structure of
LSTM and scale-free property of natural languages. In addition to Dyck 
experiments, some other languages, such as a generation from PCFG or some
programming languages might be also interesting for experimentation.
review_title: Very interesting paper that combines scale-free property of natural languages with LSTM

final_score: rating: 7: Good paper, accept
review_content: This paper points out the relationship between words in natural language usually follow the power law. Gated recurrent neural networks such as LSTMs excel in modelling natural language, however, the forgetting mechanism of LSTMs  is ruled by the exponential decay. This work demonstrates a way to engineer the forgetting mechanism of LSTMs to mimic the power law relationship that is more presented in natural language. By applying their technique, the modified LSTM model can do better in modelling rare tokens, which usually span for longer timescales, hence the model can score lower perplexities on less frequent words. The key contribution of the paper is the derivation which shows that the forget gates of LSTMs are subject to exponential decay in zero-input regime after the first input token is given. And the expected value of exponential decay functions exp(-t/T) can approximate the power law when T is sampled from the Inverse Gamma distribution. 

The experiments demonstrate that that drawing T from the Inverse Gamma distribution is a natural fit for natural language. Then, the authors propose a multiscale LSTM model that exploits this property. Each timescale T is drawn from the inverse Gamma distribution, which essentially becomes a forget bias term and is fixed during the training. Multiple Ts are drawn to mimic the power law. The multiscale LSTM captures the right inductive bias to perform better in modelling less frequent words which might be useful to keep in the memory for longer time. The paper is well written and both the motivation and explanation of the approach are clear. The experiments are appropriately designed, and the results support the main claim nicely.

I do have some comments and questions on what's written in the paper though.
It is conventional to use notation h_{t-1} in the update rules for the input, forget and output gates since h_t is obtained from c_t, or at least clarify the update rule of the hidden state as h_{t+1} = o_t * tanh(c_t).

Instead of not learning the forget bias at all, have you tried adding a regularisation loss that forces the forget bias term to stay closer to the prior (initially drawn T from the Inverse Gamma distribution)? 

What is the motivation of using smaller hidden size for the topmost layer for both the baseline and the multiscale LSTM?

For the multiscale LSTM, why were the timescales not sampled from the Inverse Gamma distribution and fixed during the training for the topmost layer? But instead, they were learned like the standard LSTM?

How were T=3 or T=4 chosen for the first layer of the multiscale LSTM? Were they found by evaluating performance on the validation set or were they chosen based on domain knowledge?
review_title: An interesting work that shed light on how to design a neural architecture inspired by a theory.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

This work investigates representational power of LSTM to model natural language, in particular how well it models temporal dependencies within text. They define a notion of timescale of each LSTM unit and analitycally show that LSTM memory exhibits exponential decay, while natural language tends (based on prior work) to decay following the power law. Based on this, they figure that LSTM memory may decay following the power law *if the timescales approximate samples from the particular Inverse Gamma Distribution*. To achieve that they propose the multi-timescale LSTM unit, where the desired timescale is explicitly controlled via the forget gate bias.

Authors empirically validate their theoretical claims and show improvements in language modeling (PTB, Wikitext2) over the baseline LSTM using the proposed multi-timscale LSTM. Importantly, they show how multi-timescale LSTM gives improvement in modeling rare words, which are known to require longer temporal dependencies.

## Strong points

1. This work investigates the important (though not that popular) question of discrepancy between the temporal dependencies existing in natural language and the abilities of models we use to learn these dependencies in practice.
2. The idea of including explicit control of the timescale (i.e. temporal horizon) of each LSTM unit is interesting and well-motivated.
3. Experiments use a formal language too in addition to natural language modeling which allows to check if proposed approach generalizes in case of exactly computable timescale distribution.

## Weak points

1. There is **no code** available. I was interested in the way how test set bootstraping was performed in the experiments (see comments below for details) and found out there is no code, which is really sad. I hope authors will submit the reproducible code in the near future.
2. Theoretical part gives some essential quantities while no derivations are given. Given that there is some free space left in the paper and the fact of unlimitied supplement material I don't see any reason to omit derivations (I struggle with some transitions between equations as you may found in the comments below).
3. I am not sure how useful is the proposed approach on new tasks given hardly tuned hyperparameters for the Inverse Gamma Distribution proposal and LSTM model architecture for each task (more details in the comments below).

## Recommendation

I vote for accepting **upon fixing major weak points**: uploading reproducible code with experiments and adding all derivations necessary for essential theoretical claims in this work. Overall this is decent work which will be useful for future research in studying representational power of models we are using to learn complicated dependencies of natural language.

## Questions

All the questions below are welcome to be used as **suggestions** to provide more details in the manuscript.

### Theoretical part

1. Eq.3: why can we simply average forget gates? Out of 'free input' regime $c_t$ from eq.1 would have more dependencies. Could you elaborate why can we estimate it like this. 
2. How to solve Eq.4 as getting Inverse Gamma Distribution? I struggle to find an obvious/trivial solution, please elaborate this in the manuscript.

### Experiments
3. From 3.1.1. '*Training sequences were of length 70 with a probability of 0.95 and 35 with a probability of 0.05. During inference, all test sequences were length 70.*' Why are you making such explicit scheduling? Given that each training sequence from WT2 is some excerpt from wikipedia (often longer than 70 words), how do you deal with tails of sequences? More detailed description of data loading will be helpful.
4. Table 1: from my understanding columns with rare words attract most interest, and I wonder if you could add the varaiance among different training instances in Table 1 like you reported in the appendix (Table 2)? Or refactor Table 2 such that it has same freq based columns.
5. Did you think of tokenization other than word-level? BPE for example: it gives more balanced token distribution for WT2 due to absence of UNK token there. I wonder if improvements in terms of rare words PPL will hold. How do you think (speculatively)?
6. PTB results: >10K bin PPL got higher with your approach (also ratio drop below 1 in routing study). Why do you think this happens? Why do you think this does not happen with WT2 task? As I see with PTB the fixed forget bias underestimates true/gold high freq word probabilities on average, but why? 

### Other

7. Is it possible to estimate/learn the IGD alpha parameter from the data itself of the task you work on? This grid search you provide makes me less convinced in how useful this approach is for the new task, where the $\alpha$ is not known.
8. How important is the model layers tuning you do? E.g. only specific layers have fixed forget bias, but others not, **why is that?** I am really interested in knowing if other ways of defining your model hurt the performance or keep it on the baseline level? I am sure this will be useful for all other readers too.
9. Is it possible to apply this timescale control for other units e.g. GRU (no explicit forget gate)?
review_title: Interesting approach for explicit control of LSTM unit timescale in natural language modeling

final_score: rating: 7: Good paper, accept
review_content: With strong arguments I could be persuaded to change my score.

The paper investigates how well LSTM language models learn the known statistical temporal dependency distributions of languages, which follows a power law. The authors deduct the expected distribution of LSTM unit timescales and present a method for manually controlling them by setting the forget gate bias using an inverse Gamma distribution. They show that while the timescales of a standard LSTM LM follow the expected distribution, manually controlling it instead of learning it still increases LM performance, especially for infrequent words. The findings are also validated on a formal language, where the time scales are explicitly known.

What this paper excels at is a theoretical formulation of the proposed method and a motivation grounded in real contemporary problems in the field of modeling sequential data. However, I am not confident that the baselines and test suite employed by the authors is sufficient for me to accept their hypothesis.

The LSTM baselines in Table 1 are about 5 ppl points below the reportings of the paper they use as a baseline. Thus the authors "improvement" is actually significantly below the original paper from almost three years ago,which is a long time in this field, current models are now below half of the reported perplexity.

"Test data were divided into 100-word sequences and resampled with replacement 10,000 times. For each sample, we computed the difference in model perplexity (baseline − multi-timescale) and reported the 95% confidence intervals (CI) in Table 1. Differences are significant at p < 0.05 if the CI does not overlap with 0." - This test suite is unfamiliar to me. Please link a paper which uses the same test suite or follow the standard practices in the field of language modeling. Also if the authors are looking to argue for better handling longer time sequences, why cut samples off at 100 words? Shouldn't your model be better at longer sequences?

"This suggests that the performance advantage of the multi-timescale model is highest for infrequent words, which require very long timescale information" - It is not intuitive to me that rare words have more long-term dependencies than non-rare words? Also, I am unsure if the improvement in performance on rare words might just be attributed to the fact that the baseline model is significantly underperforming the original (Merity et al., 18) model.

Do you have confidence intervals for the DYCK language? Training models on DYCK-2 can be highly seed dependant. 5-10% improvement in performance seems like it could just be random noise.

I don't get Figure 3.

In general the paper is well-written and easy to understand.

UPDATE:

I do not believe that Merity 18's results are not reproducible. Just look at the vast amount of work after the publication that builds on top of the results. I don't doubt that your way of evaluating the performance of language models can lead to hypothesis testing, but it is not what the field employs thus your results are not comparable to others, which is why I cannot accept this paper. If your method really does work as well as you argue it should be no issue obtaining an improvement over the actual baseline.

UPDATE 2:

The new results with Merity's original benchmark leads me to increase my score from 4 to 7. I appreciate the effort in reproducing Merity's results.
review_title: Theoretically designed bias

Meta-review: The paper proposes matching the distribution of biases for an LSTM to estimates of long range mutual information from analyzing the statistics of languages. The authors shows empirical evidence that LSTMs seem indeed to be following such a distribution, using natural language and Dyck-2 grammar. They show that explicitly enforcing the distribution of biases in learning can actually help LSTM language models.
The reviewers had slight concern about some of the baseline numbers reported, but the authors took the time to address those concerns. Overall, it was an interesting and thought provoking paper that can provide a useful angle to consider when building recurrent models for a problem -- namely that of matching the properties / inductive bias of the model to that of the data.
Decision: Accept (Poster)