Before revision:  We achieve a notable reduction in memory usage due to an improved differentiable top-k operator , making the model suitable to process long documents, as shown on an example of a summarization task .
After revision:  We achieve a notable reduction in memory usage due to an improved differentiable top-k operator . For example, our experiments on a challenging summarization task of long documents show that our method is much faster and up to 16 times more memory efficient while significantly outperforming both dense and state-of-the-art sparse transformer models. The method can be effortlessly applied to many models used in NLP and CV, simultaneously with other improvements since representation pooling addresses a different aspect of the attention's complexity problem .