HiLLoC: lossless image compression with hierarchical latent variable models
Authors: James Townsend
Thomas Bird
Julius Kunze
David Barber
Venue:  (2020)
Keywords: compression
variational inference
lossless compression
deep latent variable models

Abstract: We make the following striking observation: fully convolutional VAE models trained on 32x32 ImageNet can generalize well, not just to 64x64 but also to far larger photographs, with no changes to the model. We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale the VAE-based 'Bits-Back with ANS' algorithm for lossless compression to large color photographs, and achieving state of the art for compression of full size ImageNet images. We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression results.

Published as a conference paper at ICLR 2020
HILLOC: LOSSLESS IMAGE COMPRESSION WITH HI- ERARCHICAL LATENT VARIABLE MODELS
James Townsend; Thomas Bird;; Julius Kunze & David Barber
Department of Computer Science University College London <fi fEirstname... <surname> @cs ucl acc.uu
ABSTRACT
We make the following striking observation: fully convolutional VAE models trained on 32x32 ImageNet can generalize well, not just 64x64 but also far larger photographs, with no changes to the model. We use this property, applying fully convolutional models to lossless compression, demonstrating a method scale the VAE-based Bits-Back with ANS' algorithm for lossless compression (Townsend et al. 2019) to large color photographs, and achieving state the art for compression of full size ImageNet images. We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression results
INTRODUCTION
Bits back coding (Wallcce 1990 Hinton & van Camp 1993) is a method for performing lossless compression using latent variable model. an ideal implementation, the method can achieve an expected message length equal to the variational free energy, often referred to as the negative evidence lower bound (ELBO) of the model. Bits back was first introduced form a theoretical argument for using the ELBO as an objective function for machine learning (Hinton & van Camp 1993).
The first implementation of bits back coding (Frey) 1997| Frey Hinton 1996) made use of first-in- first-out (FIFO) arithmetic coding (AC) (Witten et al. 1987). However, the implementation did not achieve optimal compression, due to an incompatibility between FIFO coder and bits back coding, and its use only demonstrated on small dataset of 8x8 binary images.
Recently, zero-overhead bits back compression with a significantly simpler implementation has been developed by Townsend al. (2019). This implementation makes use of asymmetric numeral systems (ANS), a fistsin-first-uut (LIFO) entropy coding scheme (Duda) 2009). The method, known as 'Bits Back with Asymmetric Numeral Systems' (BB-ANS was demonstrated by compressing the MNIST test set using a variational auto-encoder (VAE) model (Kingma & Welling 2013: Rezende et al. 2014), achieving compression
within 1% of the model ELBO.
More recently, Hoogeboom et al. (2019) and Ho et al. (2019) have proposed flow-hased methods for lossless compression, and Kingma al. (2019) have presented 'Bit-Swap'. extending BB-ANS to hierarchical models. In this work we present an alternative method for extending to hierarchical VAEs. This entails the following novel techniques:
1. Direct coding of arbitrary sized images using fully convolutional model.
2. A vectorized ANS implementation supporting dynamic shape.
3. Dynamic discretization to avoid having to calibrate a static discretization.
4. Initializing the bits back chain using a different codec.
We discuss each of these contributions in detail in Section We call the combination of BB-ANS using a hierarchical latent variable model and the above techniques: 'Hierarchical Latent Lossless
Equul contribution.
Available httttp::/ / //gi thub. com haom/hhlloc--a submi ssior /hi 11cc

---

Published as a conference paper at ICLR 2020
Raw data
PNG
WebP
FLIF
Bit-Swap
HiLLoC
0.0
0.2
0.4
0.6
0.8
1.0
Compression ratio
Figure 1: A selection of images from the ImageNet dataset and the compression rates achieved on the dataset by PNG, WebP, FLIF, Bit-Swap and the HiLLoC codec (with ResNet VAE) presented in this work.
Compression' (HiLLoC). In our experiments (Section 4). we demonstrate that HiLLoC can be used to compress color images from the ImageNet test set at rates close the ELBO, outperforming all of the other codecs which we benchmark. We also demonstrate the speedup, of nearly three orders of magnitude, resulting from vectorization. We release open source implementation based on 'Craystack', a Python package which we have written for general prototyping of lossless compression with ANS.
2 BACKGROUND
In this section we briefly describe the BB-ANS algorithm first introduced by Townsend et al. (2019). begin by giving high-level description of ANS LIFO entropy coder (Dudaj 2009). along with notation for describing the basic ANS operations. Throughout the rest of the paper log to mean the base two logarithm, usually denoted logz. and we measure message lengths in bits.
2.1 ASYMMETRIC NUMERAL SYSTEMS
As an entropy coder, ANS was designed for compressing sequences of discretely distributed symbols. achieves a compressed message length equal to the negative log-probability (information content) of the sequence plus implementation dependent constant, which usually less than 32 bits. For long sequences, the constant overhead has negligible contribution to the overall compression rate. Thus, by Shannon's 's source coding theorem (Shannon 1948), ANS coding guaranteed be near-optimal for long sequences.
There are two basic operations defined by ANS, which we will refer as 'push' and 'pop'. Push encodes symbol by adding it to an existing message. It has the signature
push (message, symbol) + message'.
(1)
Pop is the inverse of push, and may be used to decode a symbol and recover message identical to that before pushing.
pop: message' (message, symbol)
(2)
When multiple symbols are pushed in sequence, they must be popped using the precise inverse procedure, which means popping the symbols in the opposite order. Hence why ANS is referred to as a last-in-first-out coder, or a stack.
The push and pop operations require access probabilistic model of symbols, summarized by probability mass function p over the alphabet possible symbols. The way that symbols are encoded depends on the model, and pushing symbol according p results in increase in message length of log p(e) Popping s results in an equal reduction in message length. For details on how the ANS operations implemented, see Duda 2009).
Note that any model/mass function can be used for the pop operation, i.c. there's no hard restriction to the distribution that was used to encode the message. In this way, rather than decoding the same
2

---

Published as a conference paper at ICLR 2020
data that encoded, pop actually be used to sample symbol from different distribution. The pop method itself is deterministic, so the source of randomness for the sample comes from the data contained within the message. This sampling operation, which can be inverted by pushing the sample back onto the stack, is essential for bits back coding.
For convenience, introduce the shorthand notation - p()) for encoding (pushing) a symbol according to p. and t- p(-) for decoding (popping).
2.2 BITS BACK WITH ANS
Suppose we have model for data which involves latent variable A sender and receiver wish to communicate a sample x. They have access to a prior on 2, denoted p(z), likelihood p(x 2) and (possibly approximate) posterior g(z 1)) but not the marginal distribution p(x). Without access to p(z), sender and receiver cannot directly code x using ANS. However, BB-ANS specifies an indirect way to push and pop æ. It does not require access to the marginal p(x)), but rather uses the prior, conditional, and posterior from the latent variable model.
Table D (a) shows. in order from the top, the three steps of the BB-ANS pushing procedure which the sender can perform to encode z. The 'Variables' column shows the variables known to the sender before each step. Tb) shows the inverse steps which the receiver can use to pop x, with the 'Variables" column showing what is known to the receiver after each step. After decoding the third step of popping, - 4(- æ), is necessary to ensure that BB-ANS pop is a precise inverse of push.
Table 1: Indirectly pushing and popping using BB-ANS. + and +- denote pushing and popping respectively. AL denotes the change in message length resulting from each operation. The three steps to push/pop ordered, starting the top of the table and descending.
(a) Pushing
(b) Popping
Variables
Operation
AL
Operation
Variables
AL
+ ((-1z) (z)
log q(= [x]
p(-)
log p(=)
2
p(-12)
+log p(a
+ (()1=)
2, 2
log p(
- p(-)
loog p(=)
- 4( xx)
+ log g(=
The change in message length from BB-ANS can easily be derived by adding up the quantities in the A J. column of Table For encoding we get
1
1
log 1 p(z)
A LLBB-AND ANS log
log
(3)
(=11x) xx)
plx
p(x,=)
log q(2 x))
(4)
Taking the expectation over gives the expected message length for datum x
p(z, 2)
((x) = -g((=12) log 7(= (z)
(5)
which the negative evidence lower bound (ELBO), also known as the free energy. This
commonly used training objective for latent variable models. The above equation implies that latent
variable models trained using the ELBO are implicitly being trained to minimize the expected message length of lossless compression using BB-ANS.
Note that, as Table shows, the first step of encoding a data point, x. using BB-ANS is to, counter- intuitively, decode (and thereby sample) latent < g( Iz). This requires that there is already buffer of random data pushed to the ANS coder, which can be popped. This data used to start the encoding process is recovered after the final stage of decoding, hence the name "bits back'.
we have multiple samples to compress, then we can use 'chaining', which is essentially repeated application of the procedure in Table (Townsend et al. 2019). In Section B.4 we describe how build up an initial buffer of compressed data by using a different codec to code the first images in a sequence.
3

---

Published as a conference paper at ICLR 2020
I + p(. 2)
2 - p(.)
Vectorized ANS coder
Figure Visualizing the process of pushing images and latents from VAE to the vec- torized ANS stack with Craystack. The ANS stack head is shaped such that images and latents can pushed and popped in parallel, without reshaping. Beneath the shaped top of the stack is the flat message stream output by ANS.
3 SCALING UP BITS BACK WITH ANS
now discuss the techniques we introduce to scale up BB-ANS.
3.1 FULLY CONVOLUTIONAL MODELS
When all of the layers in the generative and recognition networks of a VAE are either convolutional elementwise functions (i.e. the VAE has no densely connected layers), then it is possible to evaluate the recognition network on images of any height and width, and similarly pass latents of any height and width through the generative network to generate an image. Thus, such a VAE can be used as (probabilistic) model for images of any size.
We exploit this fact, and show empirically in Section 4 that, surprisingly, fully convolutional VAE trained on 32 32 images can perform well (in the sense of having a high ELBO) model for 64 x 64 images as well as far larger images. This in turn corresponds to good compression rate, and we implement lossless compression of arbitrary sized images by using VAE in this way.
3.2 VECTORIZED LOSSLESS COMPRESSION
The primary computational bottlenecks in the original BB-ANS implementation (Townsend et al.) 2019) were loops over data and latent variables occurring in the Python interpreter. We have been able vectorize these, achieving implementation which scale to large ImageNet images. The effect of vectorization on runtime is shown in Figure
vectorized implementation of ANS was described in Giesen (014) using SIMD instructions. This works by expanding the size of the ANS stack head, from a scalar to a vector, and interleaving the output/input bit stream. We implement this in our lossless compression library, Craystack, using Numpy. Please refer to the Craystack code and to Giesen (014) for more detail. We ensure that the compression rate overhead vectorization low by using the BitKnit technique described Giesen (2015), see Appendix D for more detail. Having vectorized, we found that most of the compute time our compression was spent in neural net inference, whether running on CPU or GPU, which we know already be reasonably well optimized.
In Craystack, we further generalize the ANS coder using Numpy's n-dimensional array view interface, allowing the stack head to be 'shaped' like an n-dimensional array, nested Python data-structure containing arrays. We can then use a shape which fits that of the data that we wish to encode or decode. When coding data according arrays, matching the shapes the observation & and the latent z. This allows for a straightforward implementation and clarifies the lack of data dependence between certain operations, such as the
VAE
ANS stack head shaped into pair of
4

---

Published as a conference paper at ICLR 2020
r pp.1(( and - p(-) during encoding, which theoretically be performed concurrently. This vectorized encoding process visualized in Figure
3.3 DISCRETIZATION
standard for state of the art latent variable models use continuous latent variables. Since ANS operates over discrete probability distributions, if we wish to use BB-ANS with such models it necessary to discretize the latent space so that latent samples can be communicated. [Townsend et al. (2019) described static discretization scheme for the latents in simple VAE with a single layer of continuous latent variables, and showed that this discretization has a negligible impact on compression rate. The addition of multiple layers of stochastic variables to VAE has been shown to improve performance (Kingma et al.) 2019) Kingma et al.) 2016 Maalde et al. 2019 Sonderby et al.) 2016). Motivated by this, propose a discretization scheme hierarchical VA with multiple layers of latent variables.
The discretization described in Townsend et al. intervals of equal mass under the prior p(z). For hierarchical model, the prior on each layer depends on the previous layers:
2019) is formed by dividing the latent space into
L-1 II p(z) 21+1.))
p(ILL) p(z))
(6)
isn't immediately possible to use the simple static scheme from Townsend al. (2019), J since the marginals p(z1). ....,....................... p(,z)))).. are not known. Kingma et al. (2019) estimate these marginals by sampling, and create static bins based on estimates. They demonstrate that this approach can work well. We propose alternative approach, allowing the discretization to vary with the context of the latents we are trying to code. We refer to our approach as dynamic discretization.
In dynamic discretization, instead of discretizing with respect to the marginals of the prior, discretize according to the conditionals in the prior, p(# 2++1.)) Specifically, for each latent layer I, partition each dimension into intervals which have equal probability mass under the conditional p(2] 2.+1:2). This directly generalizes the scheme used in BB-ANS (Towssend et al. 2019).
Dynamic discretization is more straightforward implement because it doesn't require callibrating the discretization to samples. However imposes a restriction on model structure, in particular requires that posterior inference is done lop-down. This precludes the of Bit-Swap. In Section 3.3.I we contrast the model restriction from dynamic discretization with the bottom-up, Markoy restriction imposed by Bit-Swap itself.
We give further details about the dynamic discretization implementation we use in Appendix A
3.3.
MODEL RESTRICTIONS
22
22
22
2
21
21
21
T
T
I
I
(a) HiLLoC generative
(b) HiLLoC inference
(c) Bit-Swap generative
(d) Bit-Swap inference
Figure 3: Graphical models representing the generative and inference models with HiLLoC and Sit-Swap. both using a 3 layer latent hierarchy. The dashed lines indicate dependence the fixed observation.
The first stage of BB-ANS encoding to pop from the posterior, 21.1. < q(-122) When using dynamic discretization, popping the layer z requires knowledge of the discretization used for z and
5

---

Published as a conference paper at ICLR 2020
thus of the conditional distribution p(2: #1+1:L). This requires the latents 24+1:L have already been popped. Because of this, latents in general must be popped (sampled) in 'top-down' order, i.e. 21, first, then 21- and so on down to 21.
The most general form of posterior for which top-down sampling is possible
L-1
( 12) = (( (x))
II [=1
This is illustrated, for a hierarchy of depth 3, in Figure 3b The Bit-Swap technique (Kingma et al. 2019) requires that inference be done bottom up, and that generative and inference models must both be Markov chain 21, 21..... 2L, and thus cannot skip connections. These constraints are illustrated in Figure 3c,d. Skip connections have been shown to improve model ELBO in very deep models (Sonderby et al.) [016 Maaløe et al. 2019). HiLLoC does not have this constraint, and we do utilize skip connections in our experiments.
3.4 STARTING THE BITS BACK CHAIN
As discussed in Section 3.3 our dynamic discretization method precludes the use of Bit-Swap for reducing the one-time cost of starting BB-ANS chain. We propose instead use a significantly simpler method to address the high cost of coding small number of samples with BB-ANS, namely we code the first samples using a different codee. The purpose of this is to build up a sufficiently large buffer of compressed data to permit the first stage of the BB-ANS algorithm to pop a latent sample from the posterior.
our experiments we use the 'Free Lossless Image Format' (FLIF) (Sneyers) & Wuille 2016) to build up the buffer. We chose this codec because it performed better than other widely used codecs, but in principal any lossless codec could be used.
The amount of previously compressed data required to pop posterior sample from the ANS stack (and therefore start the BB-ANS chain) roughly proportional the size of the image we wish to compress, since in a fully convolutional model the size of the latent space is determined by the image size.
We can exploit this to allow to obtain better compression rate than FLIF quickly possible. We do so by partitioning the first images we wish to compress with HiLLoC into smaller patches. These patches require smaller data buffer, and thus we can use the superior HiLLoC coding sooner than if we attempted to compress full images. We find experimentally that, generally, larger patches have a better coding rate than smaller patches. Therefore we increase the size of the image patches being compressed with HiL.LoC as more images are compressed and the size of the data buffer grows, until we finally compress full images once the buffer is sufficiently large. For our experiments on compressing full ImageNet images, we compress 32x32 patches, then 64 64x64, then 128 x 128 before switching to coding the full size images directly. Note that since our model can compress any shape image, we can compress the edge patches which will have different shape if the patch size does not divide the image dimensions exactly. Using this technique means that our coding rate improves gradually from the FLIF coding rate towards the coding rate achieved by HiLLoC on full images. We compress only 5 ImageNet images using FLIF before we start compressing 32 x22 patches using HiLLoC.
EXPERIMENTAL RESULTS
Using Craystack, we implement HiLLoC with ResNet VAE (RVAE) KKngmma al. 2016). This powerful hierarchical latent variable model achieves ELBOs comparable to state of the art autore- gressive m In all experiments we used an RVAE with 24 stochastic hidden layers. The RVAE utilizes skip connections, which are important to be able to effectively train models with such deep latent hierarchy. See Appendix E for more details.
We trained the RVAE on the ImageNet 32 training set, then evaluated the RVAE ELBO and HiLLoC compression rate on the ImageNet 32 test set. To test generalization, we also evaluated the ELBO
Unlike autoregressive models, for which decoding time scales with number of pixels, and is in practice extremely slow, both encnding and decoding with RVAFs fast.

---

Published as a conference paper at ICLR 2020
and compression rate on the tests sets of ImageNet64, CIFARIO and full size ImageNet. For full size ImageNet, we used the partitioning method described in m.4 The results are shown in T
For HiLLoC the compression rates are for the entire test set, except for full ImageNet, where 2000 random images from the test set.
Table 2:
Compression performance of HiLLoC with RVAE compared to other codecs. Rates measured in bits/dimension (raw data is 8 bits/dimension). For HiLLoC we display compression rate and theoretical performance (ELBO). All HiLLoC results are obtained from the same model, trained on ImageNet 32.
ImageNet 32
ImageNet 64
Cifar-10
ImageNet
Generic
PNG WebP
6.39
5.71
5.87
4.71
5,29
4.64
4.61
3.66
FLJI
4.52
4.19
4.19
3.37
Flow-based
IDF
4.18
3.90
3.34
IDF generalized
4.18
3,94
3.60
LBB
3.88
3.70
3.12
VAE-based
Bit-Swap
4.50
3.82
3.5
HiLLoC
4.20
3.90
3.56
3.15
HiLLoC (ELBO)
(4.18)
(3.89)
(3.55)
(3,14)
Table Showws that HiLLoC achieves competitive compression rates on all benchmarks, and state of the art on full size ImageNet images. The fact that HiLLoC can achieve state of the art compression on ImageNet relative to the baselines, even under a change of distribution, is striking. This provides strong evidence of its efficacy one might expect a degradation of performance relative to the original test set when changing the test distribution-ever more so when the resolution changes. However, in the settings we studied, the opposite was true, in that the average per-pixel ELBO (and thus the compressed message length) was lower on all other datasets compared to the ImageNet 32 validation set.
general method for lossless compression of natural images. Naively,
In the case of CIFAR, we conjecture that the reason for this is that its images are simpler and contain more redundancy than ImageNet. This theory is backed up by the performance of standard compression algorithms which, as shown in Table 2 also perform better on CIFAR images than they do on ImageNet 32. We find the compression rate improvement on larger images more surprising. We hypothesize that this is because pixels at the edge of an image are harder to model because they have less context to reduce uncertainty. The ratio of edge pixels to interior pixels lower for larger images, thus we might expect less uncertainty per pixel in larger image.
To demonstrate the effect of vectorization we timed ANS of single images at different, fixed, sizes, using a fully vectorized and fully serial implementation. The results are shown in Figure 4 which clearly shows a speedup of nearly three orders of magnitude for all image sizes. We find that the run times for encoding and decoding an average sized ImageNet image of 500 x 374 pixels (with vectorized ANS) is around 29s on a desktop computer with 6 CPU cores and a GTX 1060 GPU.
roughly linear in the number of pixels, and the time compress
DISCUSSION
Our experiments demonstrate HiLLoC bridge between large scale latent variable models and compression. To do this we use simple variants of pre-existing VAE models. Having shown that bits back coding is flexible enough compress well with large, complex models, we see plenty of work still to be done in searching model structures (i.c. architecture search), optimizing with trade-off between compression rate, encode/decode time and memory usage. Particularly pertinent for HiLLoC
"Itteger discrcte flows, retrieved from Hoogeboom et al. 2019).
'Integer discrete flows trained on ImageNet 32. ImageNet Net 64 images are split into four 32x32 patches.
Retrieved from Hoogeboom al. (019).
LLcaa hits back, retrieved from Ho et al. 2019).
For Bit-Swap. full size ImageNet images were cropped so that their side lengths were multiples of 32.
7

---

Published as a conference paper at ICLR 2020
10a
vectorized
seri
10a
time (6)
10a
10a
32x32
64x64
128x128
size
Figure 4: Runtime of vectorized vs. serial ANS implementations. Times were computed on desktop with 6 CPU cores and GTX 1060 GPU.
is latent dimensionality, since compute time and memory usage both scale with this. Since the model must be stored/transmitted to use HiLLoC, weight compression is also highly relevant. This is well-established research area in machine learning (Han al. 2016 Ullrich al. 2017).
Our experiments also demonstrated that one can achieve good performance on dataset of large images by training on smaller images. This result is promising, but future work should be done to discover what the best training datasets are for coding generic images. One question in particular is whether results could be improved by training larger images and/or images of varying size. We leave this to future work. Another related direction for improvement batch compression of images of different sizes using masking, analogous how samples of different length may be processed in batches by recurrent neural nets.
Whilst this work has focused latent variable models, there is also promise in applying state of the art fully observed auto-regressive models to lossless compression. We look forward to future work investigating the performance of models such WaveNet (van den Oord al.) 2016) for lossless audio compression as well as PixeICNN+ (Salimans et al. 2017) and the state of the art models Menick & Kalchbrenner (2019) for images. Sampling speed for these models, and thus decompression, scales with autoregressive sequence length, and can be very slow. This could be serious limitation, particularly in common applications where encoding is performed once but decoding is performed many times. This effect can be mitigated by using dynamic programming (Le) Paine et al. 2016 Raaachannrann al. 2017). and altering model architecture (Reed et al.) (017), but on parallel architectures sampling//dcomppessioon is still significantly slower than with VAE models.
On the other hand, fully observed models, as well as the flow based models of Hoogeboom et al. (2019) and Ho et al. (2019), do not require bits back coding, and therefore do not have pay the one-off cost of starting chain. Therefore they may be well suited to situations where one or a few be communicated. Similar to the way that
i..d. samples for our experiments, one could initially code images using a fully observed model then switch to a faster latent variable model once stack of bits has been built up.
FLIF to code the first images
CONCLUSION
We presented HiL,oo, an extension of BB-ANS to hierarchical latent variable models, and show that HiLLoC perform well with large models. We open-sourced our implementation, along with the Craystack package for prototyping lossless compression.
We have also explored generalization of large VAE models, and established that fully convolutional VAEs can generalize well to other datasets, including images very different size to those they were trained We have described how to compress images of arbitrary size with HiLLoC, achieving a compression rate superior to the best available codecs on ImageNet images. We look forward to future work reuniting machine learning and lossless compression.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We thank Paul Rubenstein for the substantial constructive feedback and advice which he gave us. We also thank the anonymous reviewers for their feedback. This work was supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1.

---

Published as a conference paper at ICLR 2020
REFERENCES
C. Bloom.
02- Understanding ans http://cblooomrattes
com/ 2000/201//2/ in ans- 9.mmm1 2014. Accessed: 2019-05-22.
J. Duda. Asymmetric numeral systems. ArXiv e-prints, 2009.
B. Frey. Bayesian networks for pattern classification, data compression, and channel coding. PhD thesis, University of Toronto, 1997.
B. Frey and G. Hinton, Free energy coding. In Proceedings of the Data Compression Conference, 1996.
F. Giesen. Interleaved entropy coders. ArXiv e-prints, 2014.
F. Giesen. rANS in practice. https://fgiese:
rans- in- practicectt/ 2015. Accessed: 2019-05-22.
S. Han, H. Mao, and W. Dally. Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
G. Hinton and D. Camp. Keeping neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory (COLT), 1993.
J. Ho, E. Lohn, and P. Abbeel. Compression with Flows via Local Bits-Back Coding. In Advances in Neural Information Processing Systems (NIPS), 2019.
E. Hoogeboom, J. W. T. Peters, R. van den Berg. and M. Welling. Integer Discrete Flows and Lossless Compression. In Advances in Neural Information Processing Systems (NIPS), 2019.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2013.
D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved varia- tional inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems (NIPS), 2016.
H. Kingma, Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICMI.,, 2019.
T. Le Paine, P. Khorrami, S. Chang, Y. Zhang, P. Ramachandran, M. A. Hasegawa-Johnson, and T. S. Huang. Fast Wavenet generation algorithm. ArXiv e-prints, 2016.
L. Maalee, M. Fraccaro, V. Liévin, and O. Winther. BIVA: very deep hierarchy of latent variables for generative modeling. ArXiv e-prints, 2019.
J. Menick and N. Kalchbrenner: Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
P. Ramachandran, T. Le Paine, P. Khorrami, M. Babaeizadeh, S. Chang, Y. Zhang, M. A. Hasegawa- Johnson, R. H. Campbell, and T. S. Huang. Fast generation for convolutional autoregressive models. ArXiv e-prints, 2017.
S. Reed, A. van den Oord, N. Kalchbrenner, S. Gómez Colmenarejo, Z. Wang, D. Belov, and N. de Freitas. Parallel multiscale autoregressive density estimation. In International Conference on Machine Learning (ICML), 2017.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference Machine Learning (ICML), 2014.
T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelenn++:
improving the pixelenn with discretized logistic mixture likelihood and other modifications. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
10

---

Published as a conference paper at ICLR 2020
C. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27, 1948.
J. Sneyers and P. Wuille. FLIF: Free lossless image format based maniac compression. In IEEE International Conference on Image Processing (ICIP), 2016.
C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther, Ladder variational autoencoders. In Advances Neural Information Processing Systems (NIPS), 2016.
Townsend, Bird, and D. Barber. Practical lossless compression with latent variables using bits back coding. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
K. Ullrich, E. Meeds, and M. Welling. Soft weight-sharing for neural network compression. In Proceedings of the International Conference Learning Reprexentations (ICLR). 2017.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. WaveNet: generative model for raw audio. ArXiv e-prints, 2016.
C. S. Wallace. Classification by minimum-message-unn tional Conference on Advances in Computing and Information (ICCI), 1990.
inference. In Proceedings of the Interna-
Witten, R. Neal, and J. Cleary. Arithmetic coding for data compression. Communications of the ACM, 30(6), 1987.

---

Published as a conference paper at ICLR 2020
A REPARAMETERIZING DISCRETIZED LATENTS
After discretizing the latent space, the latent variable at layer I can be treated simply an index in into one of the intervals created by the discretization. As such, we introduce the following notation for pushing and popping according discretized version of the posterior.
i (0(111+1:.:
(8)
Where Qi(- (4+1:L, 2) is the distribution over the intervals of the discretized latent space for 21, with interval masses equal to their probability under g(x Æ4) The discretization is created from splitting the latent space into equal mass intervals under p(z &4+1:)). The mass of given interval under some distribution is the CDF at the upper bound of the interval minus the CDF at the lower end of the interval. We have used 2 to indicate that these will be discrete ži values that are reconstructed from the indices it. In practise we take (i) to be the centre of the interval indexed by it. It is important to note that the Q: has an implicit dependence on the previous prior distributions p( [[2 11:)) for &
these prior distributions are required to calculate žt+1: and the discretization of the latent space.
Since we discretize each latent layer to be intervals of equal mass under the prior, the prior distribution over the indices i becomes a uniform distribution over the interval indices, ((r), which is not dependent Ê.s1. Note that this allows to push/pop the é according the prior in parallel. The full encoding and decoding procedures with a hierarchical latent model and the dynamic discretization we have described shown in Table 3 Note that the operations in the tables are ordered top to bottom.
Variables
Operation
Operation
Variables
x
iL
- Q2(.1z)
i1
+ U(.)
i1:2
e, it
iL-1 <- Qu-1 (1,,0)
x
+
p(. E1:2 (11.))
xr 11:1
in
Q1(-1 2)
x, d2:L
i2
Q2(- i3.
a, 13.L
a, 12:1
i1
a, in.t
x
p(- ((...)
+
(()
in.
Qr(-1x))
x
(a) Encoding
(b) Decoding
Table 3: The BB-ANS encoding and decoding operations, in order from the top, for a hierarchical latent model with / layers. The QI are posterior distributions over the indices i of the discretized latent space for the ith latent, 21. The discretization for the Ith latent is created such that the intervals have equal mass under the prior.
B CODEC FOR VARIABLE IMAGE SIZES
Here we describe a codec to compress a set of images of arbitrary size. The encoder now adds the dimensions of the image being coded to the stream of compressed data, such that the decoder knows what shape the image will before decoding described in Seection 3.2, we resize the top of the coder in between each coding/decoding step such that the size the top of the coder matches the sizes of the image and latents being coded. The codec is detailed in (able 4
Since we are using a vectorized ANS coder, as
To make the resizing procedure efficient, we resize via 'oolding' the top of the vectorized ANS coder such that This makes the cost of the resize logarithmic with the size difference between the vectorized coder and the targeted size.
roughly halving/doubling the number of individual ANS coders each time we fold.
12

---

Published as a conference paper at ICLR 2020
Table 4: Codec for image, z, with shape s. We code the image via the HiLLoC codec, and the dimensions of the image with the uniform codec, U. Since the coder has the same size, init init_size, before and after encoding/decoding, we can use this codec repeatedly code any number of arbitrary sized images.
Variables
Operation
Operation
Variables
2, 8
resize_coder(s)
8 + ++)(())
+ Hi
resize_ (rize___cder(s)
resize_coder(iit_ssize))
x +- HiLLLCC(()
3, 8
+ u-))
resize_ feazz__cder(initt_ege size)
I,
(a) Encoding
(b) Decoding
C COMPRESSION WITH PIXELVAE
To further demonstrate HiLLoC, we implement it with PixeIVAE model. We use a model with two latent layers, although the posterior is fully factorized. The implementation requires nesting an autoregressive codec inside the BB-ANS codec, since the observations and one of the latent layers in PixeIVAE have autoregressive generative distributions. Handling this complexity showcases Craystack, which was designed to support this kind of composition. It would also have been prohibitively slow to run on the datasets we compress without the vectorized ANS scheme discussed in(ectioo 3.2]
The achieved compression rate on the entire ImageNet validation set is displayed in Table
The autoregressive component of the PixeIVAE generative model leads an asymmetry between the times required for compression and decompression. Compression with the PixeIVAE model is readily parallelizable across pixels, since we already have access the pixel values we wish to compress and thus also the conditional distributions on each pixel. However, decompression (equivalently, sampling) is not parallelizable across pixels, since we must decompress pixel value in order to give us access to the conditional distribution on the next pixel. This means the time complexity of decompression is linear in the number of pixels, making prohibitively slow for most image sizes.
Table 5: The ELBO and compression rate HiLLoC with PixeIVAE, trained convergence on ImageNet 64, compared to other schemes. All schemes are evaluated on the ImageNet 64 validation set, and measured bits per pixel-channel.
PixelVAE
Raw data
PNG
WebP
FLIF
HiLLoC
ELBO
8
5.71
4.64
4.19
3.94
(3.67)
D VECTORIZATION WITHOUT OVERHEADS
To ensure that the compression rate overhead from using vectorization is low, we use a technique from the BitKnit codec (Giesen 2015). When we reach the end of encoding, we could simply concatenate the integers in the (vector) stack head to form the final output message. However, this is inefficient because the stack head not uniformly distributed. As discussed Giesen 2015). elements of the top of the stack have probability mass roughly
p(h) x 1/h.
Equivalently, the length is approximately uniformly distributed. More detailed discussion and an empirical demonstration of this is given by Bloom (2014) An efficient way to form the final output message at the end of decoding. is to fold the stack head vector by repeatedly encoding half of onto the other half, until only scalar remains, using the above distribution for the encoding. We implement this technique in Craystack and use it for our experiments. The number of (vectorized) encode steps required is logarithmic in the size (i.e. the number of elements) of the stack head.
13

---

Published as a conference paper at ICLR 2020
Some of the overhead from vectorization also comes the start of encoding, when, in existing implementations, the elements of the stack head vector are initialized to copies fixed constant. Information from these copies ends up in the message and introduces redundancy which scales with the size of the head. This overhead can he removed by initializing the stack head to a vector of length I and then growing the length of the stack head vector gradually as random data is added the stack, by decoding new stack head vector elements according to the distribution (9)
E RESNET VAE ARCHITECTURE
A full description of the RVAE architecture is given in Kingma et al. (2016), and a full implementation be found in our repository https: http://ttth.....hhlooottoccc.cae wwt.ccoo/chhhece submi wwwss/sssssiiieiiee hwwoonhhilloog but we give a short description below.
The RVAE is hicrarchical latent model, trained by maximization of the usual evidence lower bound (ELBO) on the log-likelihood;
log p(x, 2)
log logp(w) 2 Ey(s 2))
(10)
((= (
Take the latent hierarchy to be depth L, such that the latents are 21:L- There are skip connections in
both the generative model, p(x, 21.2), and the inference model, ( 1z). Due requirement
of using dynamic discretization, we use a top-down inference model This means that we can write
L-1
plz, (LL) = p(æ (1L))((2))
II 1=1
p(z) 2411L)
(11)
((212 (2)) 1(=1111 (x) II <(2 =-11,,)))
(12)
And the ELBO as
log logp(z) 2
E al (x) [logp(æ =I...) - (oc (x) pl)))))
(13)
E Be (x) (g(2)))))))))) / #4))))))) p(z) 2411)))
(14)
Where DKL is the KI. divergence. As in Kingma et al. (2016), the KI. terms are individually clamped as max DKL. A), where À is some constant. This is an optimization technique known as free bits, and aims prevent latent layers in the hierarchy collapsing such that the posterior is equal to the prior.
Each layer in the hierarchy consists of a ResNet block with two sets of activations. One set of activations are calculated bottom-up up (in the direction of x 22). and the other are calculated top-down. The bottom-u up activations are used only within ( æ)). whereas the top-down activations are used by both (#1.L .)) and p(r, 21 Every conditional distribution on a latent žι parameterized as diagonal Gaussian distribution, with mean and covariance function of the activations within the ResNet block, and the conditional distribution on is parameterized by a discretized logistic distribution. Given activations for previous ResNet blocks, the activations at the following ResNet block are a combination of stochastic and deterministic features of the previous latent layer, as well as from skip connections directly passing the previous activations. The features are calculated by convolutions.
Note also that all latent layers are the same shape. Since retained the default hyperparameters from the original implementation, each latent layer has 32 feature maps and spatial dimensions half those of the input (e.g. 2
wr for input of shape h w).
NNote that in Kingma al. 2016). this referred bidircctional inference.
14


Reviews:
[{"review_id": "BygJ9qoy9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for lossless image compression based on using\nfully convolutional VAE models. These models are shown to generalize well when\nthey are trained on small images (e.g. 32x32 and 64x64) and then applied to\nmuch larger images. The method is based on a fully vectorized implementation of\nbits back with asymetric numeral systems coding which is much faster than\nprevious non-vertorized implementations. An improvement with respect to similar\nmethods is to use a dynamic discretization of the latent variables which avoids\nhaving to callibrate a static discretization (as in previous methods).\nFinally, the authors initialize the bis back process with information about a\nfew initial images which are coded using a different codec.  The experiments\nperformed illustrate the gains of the method in terms of compression ratio and\nspeed.\n\nClarity:\n\nThe paper is extremelly well writen and it is very easy to read. The athors\nindicate that they will release open-source code to implement all their\nresults, which is very wellcome to improve reproducibility. However, I have to\nsay that the part describing the vectorized implementation of their method was\nrather confusing and the paper could benefit a lot from clarifying this part.\n\nQuality:\n\nThe experiments performed are sound and illustrate the gains produced by their\nmethod (although they do not achieve state of the art results). In particular,\nthe experiments show the speed up gain by the proposed vectorization and the gains\nproduced by the dynamic discretization. The experiments also show how the methods\ntrained on smaller images generalize well to larger images.\n\nNovelty:\n\nThe proposed approach is novel up to my knowledge. Although the methodological\ninnovations are not that advanced, the vectorization in the specific\napplication considered is novel, as well as the dynamic discretization.\n\nSignificance:\n\nThe proposed contributions are significant in my opinion. The vectorization\napproach can be very useful in practice and the dynamic discretization can also\nbe useful as shown by the experiments. One criticism could be that the authors\ndo not achieve state of the art results, but I consider this a minor thing.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygIsY5aKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper focuses on lossless source compression with bits back coding for hierarchical fully convolutional VAEs. The focus/contribution is three-fold: 1. Improve the compression rate performance by adapting the discretization of latent space required for the entropy coder ANS. The newly proposed discretization scheme allows for a dependency structure that is not restricted to a Markov chain structure in the encoder model q(z|x) and in the generative part of the model p(x,z). This is in contrast with bit-swap[1], which requires a markov chain structure. The dependency structure that is allowed in the proposed method is widely known to perform better than a markov chain structure, which can explain why it improves significantly over Bit-swap [1] (another hierarchical VAE compression algorithm that uses bits back coding.) 2. Increasing compression speed by implementing a vectorized version of ANS, and heaving an ANS head in the shape of a pair of arrays matching that of the latent variable and the observed variable. The latter allows for simultaneous encoding of the latent with the prior distribution and the image with the decoder distribution. 3. Showing that a model trained on a low-resolution imagenet 32 dataset can generalize its compression capabilities to higher resolution datasets with convincing results. \n\nDecision: Accept.\nThis paper is clearly written, makes clear claims and supports these claims with convincing experiments. The contributions are of practical use and I expect future work to benefit from this paper. \n\n\nSupporting arguments for decision:\nThe paper is well motivated; off the shelf compression algorithms such as PNG are also not trained on every dataset separately, and cross-dataset generalization is important if this model should be used in practice for many different images from different datasets and of different resolutions.  \n\nThe paper clearly supports the main claims. It improves upon the previous bits-back coding-based hierarchical VAE [1]. The only hypothesis that is not checked is the one that hypothesizes that the lower bpd for higher resolution images is due to the lower ratio of edge pixels versus non-edge pixels, but this is not a dealbreaker from my point of view. \n\nI would like the authors to revise their statement of state of the art compression performance on page 7 directly below table 2. “The fact that HiLLoC achieves state of the art compression rates relative to the baselines even under a change of distribution is striking, and provides strong evidence of its efficacy as a general method for lossless compression of natural images”. This is sentence should be made more nuanced as the proposed model only improves on Bit-Swap, but is still significantly outperformed by Local bits back coding (LBB [2]), and in the case of cifar-10 also by integer discrete flows (IDF [3]). On the other hand, it would be useful to still state that LBB is trained on every dataset separately, as well as IDF. Note also that in [3], a model that is trained on Imagenet32 and evaluated on the other datasets is also reported (see table 1 in [3]). It would be beneficial for the author to include the scores of this model, as the proposed method seems to perform slightly better at generalizing to new datasets.\n\nBecause of the buffer of initial bits required by bit-back coding, the compression/decompression of several data points has to be sequential if one wants to amortize this cost over several data points. Compression methods that don’t rely on bits-back coding, such as IDF [3], do not have this issue and can compress/decompress data points in parallel. Since this influences the practical usability of the model, it would be transparent to mention this. \n\nMy final main question is on the equivalence of evaluation methods of Bit-Swap and Hilloc on imagenet. The Bit-Swap paper states: “For MNIST, CIFAR-10 and Imagenet (32 × 32) we report the bitrates, shown in Table 5, as a result of compressing 100 datapoints in sequence (averaged over 100 experiments)...”. This means that Bit-Swap is not evaluated on the full test set of Imagenet 32 (as this contains 50000 images), as opposed to Hilloc. Do the authors think this is a problem? \nFurthermore, in the case of “full” Imagenet, Bit-swap uses a subset of 100 images for evaluation and crops them to a multiple of 32 pixels in height and width, so that bit-swap can compress patches and the result is the average of patches for on image. Hilloc appears to take 500 random images and does not state anything about cropping. Could the authors comment on this?\n\n\n\nAdditional feedback to improve paper (not part of decision assessment):\n- In the introduction, first paragraph: “ the method can achieve an expected message length equal to the variational free energy, often referred to as the evidence lower bound (ELBO) of the model. “ → “ the method can achieve an expected message length equal to the variational free energy, often referred to as the negative evidence lower bound (ELBO) of the model. “\n- Section 3.2, last paragraph: It is not clear if in practice the latent and image are actually encoded in parallel as the author states that this is “in theory” possible. \n- Page 4: “... we found that most of the compute time for our compression was spent in neural net inference, …” I assume you mean “inference” in any part of the encoder or decoder, and not specifically approximate inference of the encoder network. Perhaps clarify this to avoid confusion?\n- Section 4: When referring to the ResnetVAE by Kingma et al, it would be appropriate to also cite [4], as this is very similar to resnetVAE’s and was released earlier.\n\n\n\n[1] F. H. Kingma, P. Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICML), 2019.\n[2] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with Flows via Local Bits-Back Coding. arXiv e-prints, 2019.\n[3] Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer Discrete Flows and Lossless Compression. arXiv e-prints, 2019.\n[4] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems (NIPS), 2016.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeuOnp19S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for lossless image compression consisting of a VAE and using a bits-back version of ANS. The results are very impressive on a ImageNet (but maybe not so impressive on the other benchmarks). The authors also discuss how to speed up inference and present some frightening runtime numbers for the serial method, and some better numbers for the vectorized version, though they're nowhere close to being practical.\n\nI think this paper should be accepted. It has a better description of the BB ANS algorithm than I have read before, and it's a truly interesting direction for the field, despite the lack of immediate applicability.\n\nIf we are to accept this paper, I suggest the authors put a full description of the neural network used (it's barely mentioned). I think the authors also need to disclose how long it took to compress an average imagenet image (looking at the runtime numbers for 128x128 pixels is scary, but at least we'd get a better picture on the feasability).\n\nOverall, due to the fact that the authors pledge to open source the framework, I think some of the details will be found in the code, once released. I think this is an important step because there are so many details in this paper that one cannot reasonably reproduce the work by simply reading the text of this paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a lossless image compression consisting of a hierarchical VAE and using a bits-back version of ANS. Compared to previous work, the paper (i) improves the compression rate performance by adapting the discretization of latent space required for the entropy coder ANS (ii) increases compression speed by implementing a vectorized version of ANS (iii) shows that a model trained on a low-resolution imagenet 32 dataset can generalize its compression capabilities to higher resolution.

The authors addressed properly reviewers' concerns. Main critics which remain are (i) the method is not practical yet (long compression time) (ii) results are not state of the art - but the contribution is nevertheless solid.
Decision: Accept (Poster)