Title: DeCAL Tokenwise Compression
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: transformer, representation, compression, encoder-decoder, context, summarization, question-answering, retrieval, nlp

Abstract: This paper introduces DeCAL, a new method for tokenwise compression.  DeCAL uses an encoder-decoder language model pretrained with denoising to learn to produce high-quality, general-purpose compressed representations from the encoder.  DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute.  We show that DeCAL at 2x compression can match uncompressed on several downstream tasks, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks.  DeCAL offers significant savings where pre-computed dense representations can be utilized, and we believe the approach can be further developed to be more broadly applicable.

TL;DR: DeCAL is a new method for tokenwise compression, focused on maximizing compression quality instead of saving compute