Lossless Data Compression with Transformer

1 INTRODUCTION . Lossless compression is a class of compression algorithms that allows for the perfect reconstruction of the original data . In the last decades , statistical methods for lossless compression have been dominated by PAQ-type approaches ( Mahoney , 2005 ) . The structure of these approaches is similar to the Prediction by Partial Matching ( PPM ) of Cleary & Witten ( 1984 ) and are composed of two separated parts : a predictor and an entropy encoding . Entropy coding scheme like arithmetic coding ( Rissanen & Langdon , 1979 ) are optimal and most of the compression gains are coming from improving the predictor . The predictor estimates the probability of a token given its past , which is a standard sequence modeling problem . In most PAQ-type approaches , this predictor relies heavily on hand-crafted features , but recent work has used the close connection between sequence modeling and machine learning to add neural networks based methods in the predictor . In particular , CMIX ( Knoll , 2014 ) has successfully added a Long-Short Term Memory ( LSTM , Hochreiter & Schmidhuber ( 1997 ) ) to the predictor of PAQ8 . However , these approaches are still heavily relying on hand-crafted features along with the neural networks , and purely neural network based models are still far from the state of the art ( Bellard , 2019 ) . In this work , we aim at building a purely neural network based model that compete the state of the art . We follow Knoll & de Freitas ( 2011 ) and formulate the token prediction problem solved by the predictor as the online learning of a sequence model . As opposed to previous work , we propose to use a Transformer ( Vaswani et al. , 2017a ) to replace entirely the predictor of PAQ8 and its handcrafted features . Transformer has emerged as the standard model for sequence modeling in natural language processing , achieving state-of-the-art performance in numerous applications , including , but not limited to , language modeling ( Dai et al. , 2019 ; Sukhbaatar et al. , 2019b ) , machine translation ( Vaswani et al. , 2017a ; Ott et al. , 2018 ) , or sentence representation ( Devlin et al. , 2018 ; Yang et al. , 2019 ) . As opposed to the recurrent neural networks ( RNNs , Elman ( 1990 ) ) used in lossless compression , they are able to capture very long-term dependencies by allowing direct connections between long-distance symbols . However , the training of a Transformer is unstable and slow , limiting its application to the online setting of lossless compression . We propose several improvements to its architecture and training to accelerate and stabilize its online training . We show that our neural network approach matches the state of the art on the competitive benchmark enwik8 used in the Hutter prize ( Mahoney , 2005 ) . We also provide a thorough study of all the important components of a Transformer and its training to achieve this performance . 2 RELATED WORK . In this section , we briefly introduce statistical based methods for lossless compression , and focus on PAQ based compression methods as well as purely neural network based methods . Statistical based methods . A popular statistical based method for lossless data compression is the Prediction by Partial Matching ( PPM ) of Cleary & Witten ( 1984 ) . It is based on a predictor and a entropy coding , that often takes the form of an arthimetic coding ( Rissanen & Langdon , 1979 ; Witten et al. , 1987b ) , but could also be a Huffman coding ( Huffman , 1952b ) . There are many variants of PPM such as PPM * C ( Cleary & Teahan , 1997 ) or cPPMI-64 Shkarin ( 2002 ) . Another statistical based method is the stochastic sequence memoizer of Gasthaus et al . ( 2010 ) that is based on a nonparametric Bayesian model . In particular , the stochastic memoizer of Wood et al . ( 2009 ) shares many similarities with PPM and reaches similar compression performance . However , most of these models are now surpassed by PAQ based methods like CMIX ( Knoll , 2014 ) . PAQ based compression methods . The series of PAQ algorithms , including PAQ8 ( Mahoney , 2005 ) , are statistical based methods designed to operate at bit-level , making it applicable to any type of data ( Salomon & Motta , 2010 ) . There are also different version of PAQ8 specialised in different data types for better compression performance . PAQ uses a context mixer that follows the same structure as PPM . However , unlike PPM , the predictor of the context mixers combines the probability of different prediction models ( Knoll & de Freitas , 2011 ) . More recently , a PAQ based model called CMIX ( Knoll , 2014 ) has been the state of the art for lossless text compression in terms of compression rate . As opposed to PAQ8 , CMIX uses a Long Short Term Memory network ( LSTM , Hochreiter & Schmidhuber ( 1997 ) ) in their probabilistic model . The context mixer used in CMIX is also based on a gated linear network ( Veness et al. , 2017 ) . Neural Network based compression method . More recently , there has been some attempts to use purely neural network based approaches for the probabilistic model . Notably Knoll released lstm-compress ( Knoll , 2015 ) , which uses only the LSTM introduced in CMIX and the preprocessing of CMIX . Bellard ( 2019 ) investigates the performance of pure neural approaches with modern and large neural networks such as LSTM and Transformer . As well as our approach , all of these approaches use arithmetic coding as entropy coding scheme . 3 LOSSLESS DATA COMPRESSION . In this section , we briefly introduce the core components of statistical methods for lossless compression . As opposed to lossy compression where it is allowed to lose information about the orginal data in the reconstruction , lossless compression methods restore the data to its original value . Lossless compression methods are composed of two main elements : a predictor and an entropy coding scheme . They are used successively to encode an input stream of tokens s0 , . . . , sT . More precisely , for each token , the predictor first evaluates its probability given its past , then the entropy coding scheme uses this probability to store the token into a lossless code . More precisely , if we assume that t tokens s0 , . . . , st−1 are already compressed into a lossless code , the predictor computes the probability of the next token to be equal to its value st given all the preceding tokens , i.e. , the predictor computes p ( st | st−1 , . . . , s0 ) . Then , the entropy coding scheme encodes the token st into a lossless code of bit-length log2 p ( st | st−1 , . . . , s0 ) . Once the token st has been encoded , these two steps are repeated on following tokens until the end of the sequence . The operations performed by the predictor are identical during compression and decompression phases , i.e. , in both cases it predicts the next token given all the preceding tokens . The only difference is that , during the compression , the preceding tokens are taken from the input file , while during the decompression , they are generated by the entropy coding scheme . Note that , as opposed to the predictor , the role of the entropy coding scheme differs during the two phases since , during decompression , it acts as a decoder , i.e. , given the compressed representation and the sequence of predicted probability distributions ( p ( st|st−1 , . . . , s0 ) ) 0≤t≤T it restores the original sequence of tokens . There are many efficient and optimal methods for the entropy coding scheme . The source coding theorem Shannon ( 1948 ) has introduced the fundamental idea that any probability distribution can be encoded into a lossless code . For a given data point , the bit-length of this code is equal to the negative log probability estimated by the model . Thus , efficient probabilistic models are crucial to design effective compression scheme since the smallest codes are obtained when the model estimates the true data distribution . Depending on the type of probabilistic model considered , several compression schemes have been developed to implement this idea in practice , namely Huffman coding Huffman ( 1952a ) , arithmetic coding Witten et al . ( 1987a ) and asymmetric coding Duda ( 2009 ) . The basic idea of all these entropy coding schemes is to assign shorter codes to the more likely tokens . In this work , we use the arithmetic coding scheme . 4 OUR APPROACH . In this section , we introduce our approach to lossless data compression using neural networks . First , we briefly review transformer networks , which are at the core of our method . We then discuss modifications to the standard model , to make it more suitable to long sequence modeling . Finally , we propose a strategy to make the online learning of large models more efficient . 4.1 TRANSFORMER NETWORKS . Transformer networks were introduced by Vaswani et al . ( 2017a ) in the context of machine translation . These models are made of a sequence of identical blocks , each comprising two sublayers . The first sublayer , called SelfAttention , is based on the attention mechanism from Bahdanau et al . ( 2014 ) . More specifically , given a sequence of T hidden states of dimension d , represented by the matrix X ∈ Rd×T , the output of this layer is given by Z = WvXSoftmax ( XTWTk WqX ) , where Wv , Wk and Wq are the parameters of the sublayer used to compute the values , keys and queries of the attention mechanism . Multiple self-attention mechanisms are usually applied in parallel , leading to the multi-head self-attention sublayer . The second sublayer , called FFN , is a fully connected feed forward network with RELU activation , which is applied independently at each time step . Each sublayer is followed by a skip connection and a layer norm operator , to finally obtain : Z = LayerNorm ( SelfAttention ( H ) +H ) , and Y = LayerNorm ( FFN ( Z ) + Z ) . Following Dai et al . ( 2019 ) , we use relative positional encodings at each layer , as well as a cache mechanism , to make the processing of long sequences more efficient . In the following , we will refer to this model as Transformer-XL . We refer the reader to Vaswani et al . ( 2017a ) and Dai et al . ( 2019 ) for a more detailed introduction to these models . 4.2 MODIFICATIONS TO THE TRANSFORMER-XL MODEL . We propose several modifications to the Transformer-XL model that we found to improve the compression rate empirically . First , we follow Devlin et al . ( 2018 ) and use the GELU activation of Hendrycks & Gimpel ( 2016 ) instead of ReLUs . Second , as opposed to Transformer-XL ( Dai et al. , 2019 ) , we do not use Adaptive Inputs ( Baevski & Auli , 2018 ) to tie the weights of embeddings and the classifier . Finally , we use different attention span size across all layers as was suggested by Sukhbaatar et al . ( 2019a ) . In particular , an observation from this work is that attention span can be short in the first half of the network while it has to be longer in the last layers . We exploit this observation by restricting the attention span of the first layers of the Transformer and is extended linearly with the depth . More specifically , we restrict the attention size of the first 6 layers to be equal to 6× k for the k-th layer . 4.3 ADDING n-GRAMS AS INPUT We also propose to enrich the input of the transformer with n-gram information . More specifically , at time step t , the input of the model is all the n-grams for n between 1 and N of the form xt−n+1 , ... , xt . Concretely , we have an embedding vector for each n-gram , and for a given time step , we average the vectors corresponding to the n-grams of different lengths . For large n , there are many rare n-grams , leading to an important increase of parameters and memory consumption . A potential solution would be to only keep the most frequent ones , but this would require computing a dictionary of n-grams and add it to the archive . Instead , we propose to hash the n-grams into a fixed number of bins , allowing to constrain the memory requirement , without having to increase the size of the archive .

Summary: The paper investigates using the transformer architecture for neural network-based lossless compression of text. The resulting model, obtained through a thorough investigation of the architecture hyper-parameters are on par with standard SOTA compression. The paper is well-written. In particular the authors have done a great job reviewing existing compression literature and positioning their method within the space of prior work.