Context-adaptive Entropy Model for End-to-end Optimized Image Compression

1 INTRODUCTION . Recently , artificial neural networks ( ANNs ) have been applied in various areas and have achieved a number of breakthroughs resulting from their superior optimization and representation learning performance . In particular , for various problems that are sufficiently straightforward that they can be solved within a short period of time by hand , a number of ANN-based studies have been conducted and significant progress has been made . With regard to image compression , however , relatively slow progress has been made owing to its complicated target problems . A number of works , focusing on the quality enhancement of reconstructed images , were proposed . For instance , certain approaches ( Dong et al. , 2015 ; Svoboda et al. , 2016 ; Zhang et al. , 2017 ) have been proposed to reduce artifacts caused by image compression , relying on the superior image restoration capability of an ANN . Although it is indisputable that artifact reduction is one of the most promising areas exploiting the advantages of ANNs , such approaches can be viewed as a type of post-processing , rather than image compression itself . Regarding ANN-based image compression , the previous methods can be divided into two types . First , as a consequence of the recent success of generative models , some image compression approaches targeting the superior perceptual quality ( Agustsson et al. , 2018 ; Santurkar et al. , 2018 ; Rippel & Bourdev , 2017 ) have been proposed . The basic idea here is that learning the distribution of natural images enables a very high compression level without severe perceptual loss by allowing the generation of image components , such as textures , which do not highly affect the structure or the perceptual quality of the reconstructed images . Although the generated images are very realistic , the acceptability of the machine-created image components eventually becomes somewhat applicationdependent . Meanwhile , a few end-to-end optimized ANN-based approaches ( Toderici et al. , 2017 ; Johnston et al. , 2018 ; Ballé et al. , 2017 ; Theis et al. , 2017 ; Ballé et al. , 2018 ) , without generative models , have been proposed . In these approaches , unlike traditional codecs comprising separate tools , such as prediction , transform , and quantization , a comprehensive solution covering all functions has been sought after using end-to-end optimization . Toderici et al . ( 2017 ) ’ s approach exploits a small number of latent binary representations to contain the compressed information in every step , and each step increasingly stacks the additional latent representations to achieve a progressive improvement in quality of the reconstructed images . Johnston et al . ( 2018 ) improved the compression performance by enhancing operation methods of the networks developed by Toderici et al . ( 2017 ) . Although Toderici et al . ( 2017 ) ; Johnston et al . ( 2018 ) provided novel frameworks suitable to quality control using a single trained network , the increasing number of iteration steps to obtain higher image quality can be a burden to certain applications . In contrast to the approaches developed by Toderici et al . ( 2017 ) and Johnston et al . ( 2018 ) , which extract binary representations with as high an entropy as possible , Ballé et al . ( 2017 ) , Theis et al . ( 2017 ) , and Ballé et al . ( 2018 ) regard the image compression problem as being how to retrieve discrete latent representations having as low an entropy as possible . In other words , the target problem of the former methods can be viewed as how to include as much information as possible in a fixed number of representations , whereas the latter is simply how to reduce the expected bit-rate when a sufficient number of representations are given , assuming that the low entropy corresponds to small number of bits from the entropy coder . To solve the second target problem , Ballé et al . ( 2017 ) , Theis et al . ( 2017 ) , and Ballé et al . ( 2018 ) adopt their own entropy models to approximate the actual distributions of the discrete latent representations . More specifically , Ballé et al . ( 2017 ) and Theis et al . ( 2017 ) proposed novel frameworks that exploit the entropy models , and proved their performance capabilities by comparing the results with those of conventional codecs such as JPEG2000 . Whereas Ballé et al . ( 2017 ) and Theis et al . ( 2017 ) assume that each representation has a fixed distribution , Ballé et al . ( 2018 ) introduced an input-adaptive entropy model that estimates the scale of the distribution for each representation . This idea is based on the characteristics of natural images in which the scales of the representations vary together in adjacent areas . They provided test results that outperform all previous ANN-based approaches , and reach very close to those of BPG ( Bellard , 2014 ) , which is known as a subset of HEVC ( ISO/IEC 23008-2 , ITU-T H.265 ) , used for image compression . One of the principle elements in end-to-end optimized image compression is the trainable entropy model used for the latent representations . Because the actual distributions of latent representations are unknown , the entropy models provide the means to estimate the required bits for encoding the latent representations by approximating their distributions . When an input image x is transformed into a latent representation y and then uniformly quantized into ŷ , the simple entropy model can be represented by pŷ ( ŷ ) , as described by Ballé et al . ( 2018 ) . When the actual marginal distribution of ŷ is denoted as m ( ŷ ) , the rate estimation , calculated through cross entropy using the entropy model , pŷ ( ŷ ) , can be represented as shown in equation ( 1 ) , and can be decomposed into the actual entropy of ŷ and the additional bits owing to a mismatch between the actual distributions and their approximations . Therefore , decreasing the rate termR during the training process allows the entropy model pŷ ( ŷ ) to approximate m ( ŷ ) as closely as possible , and let the other parameters transform x into y properly such that the actual entropy of ŷ becomes small . R = Eŷ∼m [ − log2 pŷ ( ŷ ) ] = H ( m ) +DKL ( m||pŷ ) . ( 1 ) In terms of KL-divergence , R is minimized when pŷ ( ŷ ) becomes perfectly matched with the actual distributionm ( ŷ ) . This means that the compression performance of the methods essentially depends on the capacity of the entropy model . To enhance the capacity , we propose a new entropy model that exploits two types of contexts , bit-consuming and bit-free contexts , distinguished according to whether additional bit allocation is required . Utilizing these two contexts , we allow the model to more accurately estimate the distribution of each latent representation through the use of a more generalized form of the entropy models , and thus more effectively reduce the spatial dependencies among the adjacent latent representations . Figure 1 demonstrates a comparison of the compression results of our method to those of other previous approaches . The contributions of our work are as follows : • We propose a new context-adaptive entropy model framework that incorporates the two different types of contexts . • We provide the test results that outperform the widely used conventional image codec BPG in terms of the PSNR and MS-SSIM . • We discuss the directions of improvement in the proposed methods in terms of the model capacity and the level of the contexts . Note that we follow a number of notations given by Ballé et al . ( 2018 ) because our approach can be viewed as an extension of their work , in that we exploit the same rate-distortion ( R-D ) optimization framework . The rest of this paper is organized as follows . In Section 2 , we introduce the key approaches of end-to-end optimized image compression and propose the context-adaptive entropy model . Section 3 demonstrates the structure of the encoder and decoder models used , and the experimental setup and results are then given in section 4 . Finally , in Section 5 , we discuss the current state of our work and directions for improvement . 2 END-TO-END OPTIMIZATION BASED ON CONTEXT-ADAPTIVE ENTROPY MODELS . 2.1 PREVIOUS ENTROPY MODELS . Since they were first proposed by Ballé et al . ( 2017 ) and Theis et al . ( 2017 ) , entropy models , which approximate the distribution of discrete latent representations , have noticeably improved the image compression performance of ANN-based approaches . Ballé et al . ( 2017 ) assumes the entropy models of the latent representations as non-parametric models , whereas Theis et al . ( 2017 ) adopted a Gaussian scale mixture model composed of six weighted zero-mean Gaussian models per representation . Although they assume different forms of entropy models , they have a common feature in that both concentrate on learning the distributions of the representations without considering input adaptivity . In other words , once the entropy models are trained , the trained model parameters for the representations are fixed for any input during the test time . Ballé et al . ( 2018 ) , in contrast , introduced a novel entropy model that adaptively estimates the scales of the representations based on input . They assume that the scales of the latent representations from the natural images tend to move together within an adjacent area . To reduce this redundancy , they use a small amount of additional information by which the proper scale parameters ( standard deviations ) of the latent representations are estimated . In addition to the scale estimation , Ballé et al . ( 2018 ) have also shown that when the prior probability density function ( PDF ) for each representation in a continuous domain is convolved with a standard uniform density function , it approximates the prior probability mass function ( PMF ) of the discrete latent representation , which is uniformly quantized by rounding , much more closely . For training , a uniform noise is added to each latent representation so as to fit the distribution of these noisy representations into the mentioned PMF-approximating functions . Using these approaches , Ballé et al . ( 2018 ) achieved a state-of-the-art compression performance , close to that of BPG . 2.2 SPATIAL DEPENDENCIES OF THE LATENT VARIABLES . The latent representations , when transformed through a convolutional neural network , essentially contain spatial dependencies because the same convolutional filters are shared across the spatial regions , and natural images have various factors in common within adjacent regions . Ballé et al . ( 2018 ) successfully captured these spatial dependencies and enhanced the compression performance by input-adaptively estimating standard deviations of the latent representations . Taking a step forward , we generalize the form of the estimated distribution by allowing , in addition to the standard deviation , the mean estimation utilizing the contexts . For instance , assuming that certain representations tend to have similar values within a spatially adjacent area , when all neighborhood representations have a value of 10 , we can intuitively guess that , for the current representation , the chance of having a value equal or similar to 10 is relatively high . This simple estimation will consequently reduce the entropy . Likewise , our method utilizes the given contexts for estimating the mean , as well as the standard deviation , of each latent representation . Note that Toderici et al . ( 2017 ) , Johnston et al . ( 2018 ) , and Rippel & Bourdev ( 2017 ) also apply context-adaptive entropy coding by estimating the probability of each binary representation . However , these context-adaptive entropy-coding methods can be viewed as separate components , rather than one end-to-end optimization component because their probability estimation does not directly contribute to the rate term of the R-D optimization framework . Figure 2 visualizes the latent variables ŷ and their normalized versions of the two different approaches , one estimating only the standard deviation parameters and the other estimating both the mu and standard deviation parameters with the two types of mentioned contexts . The visualization shows that the spatial dependency can be removed more effectively when the mu is estimated along with the given contexts .

Summary: . The paper is an improvement over (Balle et al 2018) for end-to-end image compression using deep neural networks. It relies on a generalized entropy model and some modifications in the training algorithm. Experimentals results on the Kodak PhotoCD dataset show improvements over the BPG format in terms of the peak signal-to-noise ratio (PSNR). It is not said whether the code will be made available.