Before revision: We explore the benefits of decreasing the input length of transformers.
After revision: Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length.