Before revision:  First, we show that initially training the model on short subsequences , before moving on to longer ones , both reduces overall training time and, surprisingly, gives a large improvement in perplexity. We then show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens ( when generating sequences that are larger than the maximal length that the transformer can handle at once ) . Existing methods require computationally expensive relative position embeddings;
After revision:  First, we show that initially training the model on short subsequences , before moving on to longer ones , both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens ( when generating sequences that are larger than the maximal length that the transformer can handle at once ) . Existing methods require computationally expensive relative position embeddings;