Before revision: Although neural language models are effective at capturing statistics of natural language , their representations are challenging to interpret. In particular , it is unclear how these modelsretain information over multiple timescales .
After revision: Language models must capture statistical dependencies between words at timescales ranging from very short to very long. Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to a power law. However , it is unclear how these modelsretain information over multiple timescales .