General generative model‐based image compression method using an optimisation encoder

IET Image ProcessingVolume 14, Issue 9 p. 1750-1758 Research ArticleFree Access General generative model-based image compression method using an optimisation encoder Mengtian Wu, The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorZaixing He, Corresponding Author zaixinghe@zju.edu.cn orcid.org/0000-0003-0577-8009 The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorXinyue Zhao, The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorShuyou Zhang, orcid.org/0000-0001-9023-5361 The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this author Mengtian Wu, The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorZaixing He, Corresponding Author zaixinghe@zju.edu.cn orcid.org/0000-0003-0577-8009 The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorXinyue Zhao, The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorShuyou Zhang, orcid.org/0000-0001-9023-5361 The State Key Lab of Fluid Power & Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this author First published: 26 May 2020 https://doi.org/10.1049/iet-ipr.2019.0715AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Image compression is an intensively studied subject in computer vision. The deep generative model, especially generative adversarial networks (GANs), is a popular new direction for this subject. In this study, the authors propose a new compression method based on a generative model and focus on its application by GANs. The decoder in the proposed method is modified from the GAN generator model, which can produce visually real-like synthetic images. It is one of the two models in GANs, which is trained through a two-players' contest game. The encoder is an optimisation algorithm called backpropagation-to-the-input, which derives from an image inpainting algorithm based on generative models. In the proposed method, the authors turn the encoding process into an optimisation task to search for optimal encoded representations. Compared with traditional methods, the proposed method can compress images from certain domains into extremely small and shape-fixed encoded space but still retain better visual representations. It is easy and convenient to apply without any retraining or additional modification to the generative models. 1 Introduction Image compression, mainly lossy image compression, is an important image processing technique whose purpose is to reduce bits of storage and transmission for images. Traditional formats such as JPEG and JPEG2000, along with their algorithms, still dominate this area [1, 2]. However, researchers are looking for other paths that might surpass the traditional methods. In recent years, deep learning-based methods, such as convolutional neural networks (CNNs) and autoencoders, have become a research focus [3-6]. Among these methods, the deep generative model [7] based method shows extremely good performance if applied to some image data sets with similar genres and content. Deep generative models are unsupervised learning frameworks whose purpose is to generate real-like fake images. In the image compression area, unlike the traditional methods, which favour the preservation of the pixelwise local structures, the generative model-based methods focus on holding the high-level semantic feature of the images. For example, in nature scenery images, the details of grass or trees are not sensitive to human recognition, replacing them with other similar grass or trees is unrecognisable by human eyes. In such situations, if we use a generative model to produce similar items, it would still be visually consistent. This idea is called concept compression [8]. Many studies have been performed on this subject. The variational autoencoder (VAE) [6] is based on an autoencoder [4]. Its idea is to generate new samples from a preassumed latent distribution while encoding the training data sets into this distribution. Another type of generative model is the generative adversarial network (GAN) [9]. GANs are widely applied in many image processing areas, e.g. image translation [10-12], superresolution [13-15] and image inpainting [16-18]. Compared with VAE, the GAN-generated images are less fuzzy and more real-like. This is mainly because VAEs use a simple loss function to estimate the pixelwise difference between the output and the real image, while GANs use an additional network and process to judge the output as 'real' or 'fake' with high-level semantic features. This difference also affects the decoded results in image compression. That is why we focus on GANs in this paper. The GAN-based methods use the generator, which is one of the two models trained through a two-player contest game, as the decoder for compression. Santurkar et al. [19] combined a VAE encoder and a pre-trained generator as a decoder to compress and decompress images. Agustsson et al. [20] used semantic labels to divide images into pieces and a similar GAN-based method to compress segmented parts separately. However, it relied highly on segmentation. In this paper, we propose a GAN-based method. Unlike traditional methods, this method aims to compress large data sets with images of the same type. As a machine learning model, its purpose is to learn the sharing features among the training data sets and represent the images with less encoded bits. Besides, the proposed method aims to not only compress those within the training data set, but also an unpredictable number of new images of the same type, which are not contained in the training data set. This is the so-called generalisation ability to new images or the robustness of the method. Overall, it aims to outperform traditional ways in the case for images from certain domain. Our method is based on the mechanism of GAN, where GAN's generator is a neural network model mapping low dimensional data to restricted high-dimensional data. Using the low dimensional data as a pointer to the high dimensional data serves as the decoding process mechanism of the proposed method and is the key for higher compression efficiency compared with traditional methods. Besides, the restricted high dimensional data space is trained by the GAN's mechanism which makes it only contain samples that are visually pleasing to human recognition. That is how our method's decoded images' qualities are also higher than traditional methods. In the construction of our method, before compression for any image, we train a GAN model with the images of the same type from a single data set. Then the pre-trained generator of the GAN is able to produce various synthetic images similar to the training images according to random vectors called latent variables. We use the latent variable as the encoded format for compressed image and such a generating process of the generator is considered the decoding process of the proposed method. The latent variable is a very short vector. That is how the proposed method costs much fewer bits compared with traditional methods. However, the latent variable is random, so the generated images are also unpredictable. In the process for encoding certain images, first we pick a random vector so its generated content is completely irrelevant with target images to be compressed. We use an algorithm called backpropagation-to-the-input (or searching for the closest encoding) to update the latent variable according to the difference between the generated image and the target image. After several iterations repeating the same step like this, the optimised latent variable's generated image would gradually become very close to the target image. This process for searching the optimised latent variable is considered the encoding process of our method. Also, the decoding process is just generating images from this optimised latent variable. Compared with other generative model-based methods, this encoding method has several advantages. In other methods, it is sometimes necessary to store or transmit the parameters of encoder and decoder networks. In addition, each encoder model is specific to the generator (decoder) model it works with. In other generative methods the generators might be constructed by different generative models or for generating different image domains, this requires building individual networks for each generator. This not only demands individual storage and training time but also requires manual set up by humans. Moreover, such encoders are often restricted to training data, resulting in poor encoding for new images. However, our encoding process is an adaptable algorithm that does not need a neural network structure. The algorithm it uses calling backpropagation-to-the-input is inspired by image inpainting applications using GANs [16-18]. In our encoding process, the encoded format is computed through a gradually optimising process. During each iteration, it updates the latent variable (the encoded format) step by step, making the decoded image from this variable closer to the origin. In other words, the encoded format is 'found' rather than encoded by our algorithm. One of the advantages is that unlike other encoder models, ours is universally adaptable for all kinds of generative models and image content since it is an independent algorithm rather than a neural network-based model, which means using a single encoder could work for any image content generator. In addition, it discards the heavy-weight model-based encoder by almost reducing half of the storage of networks. Moreover, because this method is a random process, unlike end-to-end model-based methods, which produce determined and exclusively decoded results from the input, ours can use multiple parallel encoding processes to obtain variable results and choose the best to enhance the final result. The contributions of this paper are two-fold. First, we propose a new image compression method using generative models. It can encode images with smaller bits but is still less distorted compared with traditional methods. Second, compared with other methods based on generative models, our method uses an encoder algorithm based on the optimisation process, thus having significant advantages over others. In addition, it has to be mentioned that the proposed method is based on deep learning. Since the deep learning neural network is a data-driven method which highly relies on the training data set. When the training data is not sufficiently good, such a method's generalisation ability will be affected. Since the proposed method is an unsupervised deep learning method, it also has the same limitation. When an image with some of its part which is very different from the training data set, the whole decoded image's visual quality would slightly be affected. 2 Proposed method 2.1 Background In lossy image compression, the compression process is noted as (: real image, z: encoded format, : decoded image). The decoder's pipeline, , is similar to the generator's function in generative models noted as . In generative models, the latent z is randomly chosen from a preassumed distribution called latent variables distribution . The generator maps z to a generated image x from the posteriori generative image space . To explain our method and its generalisation ability, we first make an assumption that is capable of producing any x we want, meaning that for any x, . For any real image , there must exist a lossy representation x and a latent vector representation , , which can achieve . This is achieved through the generative distribution approximating the real data distribution . In some sense, if we can find such a latent vector , its generated image x is a lossy representation for the real image , we can use it as the encoded format of the real image. Since the length of is much smaller than that of , the compression ratio is extremely high. Fig. 1 is the graph describing generative models' generating function and the relation between the latent variables space and the generative distribution. Fig. 1Open in figure viewerPowerPoint Generating process of generative models, which shows how a latent vector maps to a generated image: a low dimension manifold maps to a high-dimension manifold . The generative distribution approximates the real data to make the generated samples look like the real samples However, the intention of generative models is not to replicate some images given in the training process. Instead, it is trying to make the distribution of the generator's outputs approximating the real images' distribution . In other words, the generator's output is trying to present a high probability of belonging to the same distribution as the real images. This is different from the intention for image compression and gains considerable benefits for compression, which is introduced in the following sections. However, it also causes a problem in the encoding process. In generative models, the latent variable is completely random. As Fig. 1 describes, since it is extracted randomly from , the generated image's content is also unpredictable. There is no method for determining z's value, which generates a lossy representation for a certain image. In VAE, this is solved by its built-in encoder. However, the intention of VAE's encoder is to map the input data's encoded representation into the target latent space and produce images relative to the training data, so it is unstable while encoding the new images. For GANs, previous works solved this by building and training an extra encoder model. In this article, we use a different method for building the encoder. It can apply to any generative models and is especially easy and convenient for GANs. Therefore, in this paper, we focus on its application to GANs. In the next sections, we introduce GANs and the training technique we use for compression purposes. 2.2 GANs GANs are a kind of unsupervised learning model that incorporates two separate models – a discriminator and a generator. The training of GANs is a two models' contest game. During training, the generator produces a couple of fake images and mixes them with real images. It tries to trick the discriminator to believe that these images are also real. The discriminator, while unaware of the source, tries to label their correct probabilities denoting whether they are real. Each model updates and enhances itself within one iteration. This game ends with an equilibrium where the generator can produce real-like faked images are totally deceiving the discriminator. In the statistical view, we define x as the data from the image domain (both real and fake). is the distribution of the generative image domain and is the distribution of the training data set domain, in which and denote the probability of x belonging to each distribution. The input vector is generated from a uniform random distribution . The generator function maps z to the space of x, noted as . Correspondingly, the real data set image is denoted as . The discriminator function maps x to a scalar range from 0 to 1, denoting the probability of x belonging to . The target of discriminator D is to correctly label the samples, in which and . As the opponent, generator G attempts to generate samples to trick D, in which . and are trainable variants in G/D. The training alternately updates and . The following is the equation of this two-player game: (1)During the training, the two models' variables and are optimised alternately by the following gradients (2), (3) within a minibatch of m samples: (2) (3)As training proceeds, these two models will reach a balance in which we presume that the generator can produce any lossy representation for real images. 2.3 DCGAN framework and training strategy The deep convolutional GANs (DCGANs) [21] framework is one of the many branches of GANs. It adopts the structure of CNNs [22] in two models. With a well-learned hierarchy of intermediate representations for image data, it outputs better visual representation. In this paper, DCGAN is adopted to build GAN models. Fig. 2 and the following are the details of the DCGAN architecture: (i) The input of the generator is an ( in our experiment) length long vector . Each element in this vector is an independent uniform random variable ranging from −1 to 1 with eight bits to encode. A fully connected network projects the input layer into a high-level convolutional layer along with a series of deconvolutional layers (fractionally stride convolutional) connecting previous ones, and the final output is the image. Fig. 2Open in figure viewerPowerPoint Details of DCGANs architecture. The generator and discriminator are of an inverse structure (ii) The discriminator's input layer is the image receiving both from the generator and data set. A series of convolutional networks and a fully connected network project the input into an output scalar ranging from 0 to 1. (iii) For each convolutional and deconvolutional layer, the filter is of the shape . Following each convolutional layer, there is a batch normalisation (BN) layer [23], use rectified linear unit activation [24] for G, and a leaky rectified activation [25] for D. Fig. 2 shows the details of the networks. In Tables 1 and 2, we present the details of DCGAN's parameters, in which BN means batch normalisation, CONV and DCONV are short for convolutional networks and deconvolutional networks, respectively f is the length for filter size and s is the stride step. The estimation space of each network is 72 MB. Table 1. Generator networks structure Activation shape Activation size Parameters input (100,1) 100 0 DCONV1 (f = 5, s = 1) (4,4,1024) 16,384 1,638,401 BN1 (4,4,1024) 16,384 2 DCONV2 (f = 5, s = 1) (8,8,512) 32,768 13,107,712 BN2 (8,8,512) 32,768 2 DCONV3 (f = 5, s = 1) (16,16,256) 65,536 3,277,056 BN3 (16,16,256) 65,536 2 DCONV4 (f = 5, s = 1) (32,32,128) 131,072 819,328 output (64,64,3) 12,288 9603 Table 2. Discriminator networks structure Activation shape Activation size Parameters input (64,64,3) 12,288 0 BN1 (64,64,3) 12,288 2 CONV1 (f = 5, s = 1) (32,32,128) 131,072 9728 BN2 (32,32,128) 131,072 2 CONV2 (f = 5, s = 1) (16,16,256) 65,536 819,456 BN3 (16,16,256) 65,536 2 CONV3 (f = 5, s = 1) (8,8,512) 32,768 3,277,312 BN4 (8,8,512) 32,768 2 CONV4 (f = 5, s = 1) (4,4,1024) 16,384 13,108,224 output (1,1) 1 16,385 It is known that GANs are hard to train for the so-called model collapse where the generator can only produce limited varieties of samples. In the literature [9], the author describes the discriminator would end with (4)Meanwhile, the generator is attempting to obtain (5)The problem is that (4) is always learned too well before (5). Assume (4) has already been satisfied, placing (4) into (1) would obtain (6)which can also be expressed as (7)This is the form of Jensen–Shannon divergence (8)In this case, if but , . This could cause a vanishing gradient hindering the generator from further learning. It is a common problem in generating images. However, there is a trade-off between training and compression task where is not required exactly. We only need an approximation of the images. Therefore, the following strategies are taken in training, which makes the generator further from converging but to produce a broader range: (i) Update the discriminator once and the generator twice for every training minibatch. (ii) Restrict the training steps (10 epochs). The first method helps the generator to catch up with the training with discriminator. The second prevents the generator from falling in a limited domain. With these techniques, we prevent model collapse and make the generator more feasible for compression tasks. In conclusion, the DCGAN structure is different from the classical structure in the bits for encoding latent vector z, and the training of GAN is slightly different in the updating steps of G and D. 2.4 Encoding method In this section, we introduce the encoding of our method. In Section 2.1, an assumption is that for any x, , which means contains any of the approximation for a real image. The encoder's work is to find a latent variable z, which is . The algorithm we use is called backpropagation-to-the-inputs (or search for the closest encoding), which is a searching algorithm. It is inspired by an image inpainting model using GANs [10]. Image inpainting reconstructs the arbitrary missing regions in corrupted images. The recent semantic inpainting method using GANs outperforms traditional methods such as total low rank [26] and patch match [27]. In our framework, instead of reconstructing corrupted images, we modify the algorithm to reconstruct the compressed images. It is actually an optimisation process through a series of updating iterations. The following is its pipeline: (i) Generate a random vector from the latent variables distribution . The initial is the starting point. (ii) Generates an image from , . The first generated image must be irrelevant to the real image . (iii) Compute the distance between and : (9)where is the contextual distance between the real and decode images and is the perceptual penalty. (iv) Update the latent vector by . is the gradient of corresponding to (9) attempting to minimise the cost function in (9), thus reducing the distance between and . (10) (v) Repeat steps of (2)–(4) for n iterations until obtaining the final products of . This process can be denoted as in which the final product attempts to approximate: (11)and the generated product attempts to approximate , an approximation of . The technique is that, as its name backpropagation-to-the-inputs, the trainable variable switches to the input. In the encoding process, only the generator networks is used, but remains static from beginning to end. Only the input layer z is changeable. In other words, z determines the content it generates with an intention from the cost function in (9). Fig. 3 is a graph describing this optimisation process and is an example of how a totally irrelevant generated image gradually converges to a target real image. Fig. 3Open in figure viewerPowerPoint Encoding process (a) Optimisation process, (b) Example of how a random initialised sample gradually become similar to the target image we want to compress In this process, each iteration of (2)–(4) is an optimisation process gradually making the decoded sample closer to . In lossy compressions, we only need to obtain a close result. Here, 'close' is defined as the difference between and . In (11), we noted it as calling the contextual loss. It is a cost function indicating the pixel-wise distance between the real and the decoded image. Using SGD methods, the cost function decreases step by step. In our experiments, is calculated as the L2 norm (12) is the perceptual loss with the weight of . This loss is independent of , an evaluation merely to the decoded samples. In our experiments, it is noted as follows: (13)in which we use the discriminator to evaluate the decoded image. This means minimising it leading closer to 1 making the decoded image becomes more 'real'. There are many other evaluation functions for decoded images in generative compression methods. For example, some use the inner layers in the pretrained models such as AlexNet [28] or VGG [29] to calculate high-level feature distance. Since there is no unified standard for these cost functions, we will not explore this topic further. Instead, we want to emphasise the advantages of our encoder method. In other similar algorithms, the encoder is usually a symmetric network as the generator. For example, in paper [19], the encoder was a convolution network with symmetric complexity as the generator. It was also trained by the training data sets and had two obvious shortcomings. First, the network has to be manually set up specific to the generator by humans. Second, it is also specific to its training data sets. This means there must be several independent encoders for each different image domain. This is time- and storage-consuming. In our method, we overcome those shortcomings. First, our encoder can apply to any generative model without a specific network. In the encoding process, it only needs to know the latent vector's dimension and preassumed distribution since it is an adaptive algorithm rather than a model-based network. Second, it can be directly applied to any image without the preknowledge of the pretrained generator or training process. In addition, other compression methods must store both the networks of the decoder (generator) and the encoder. Our method saves approximately half of the store without a network-based encoder. 2.5 Multiple parallel encoding method In addition, our encoder's method has a special technique for improving its performance. That is the multiple optimisation encoding processes to compress the same image. In Fig. 4, we provide such an example. It is based on the feature that our encoding algorithm is a random process, so the decoded result is sensitive to its initial . In the example of Fig. 4, with different start points, the endpoints are all close to the target image but in different actual positions. We could conduct many processes and choose the best one (with the least loss evaluation) to improve the final result. In the example of Fig. 4, we conduct 128 encoding processes and evaluate their lossy evaluations (structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and L2 norm). This is not time consuming since our algorithm is parallel computable. Fig. 4b shows that the decoded examples' loss evaluation (SSIM) almost follows a normal distribution. A single decoded result is the one with an average score, a corrupted result on the left of the distribution, or a superb result on the right. However, with more processes, it is more likely to obtain a better result by sampling from more decoded samples. Fig. 4c shows that the SSIM and PSNR are highly correlated, but the contextual loss is less linearly relevant to the other 2 evaluations. We choose the best sample according to their SSIM. In the examples in the experiment, we provide decoded samples extracted from the fitting curve of the normal distribution as in Fig. 4b, in which the examples' SSIMs are most similar to the value of −2σ, −σ, μ, σ, 2σ of the fitting normal distribution. Fig. 4Open in figure viewerPowerPoint Multiple parallel encoding processes. Each process gets different results according to its initial position (a) Some of the decoded examples, (b) Histogram of SSIM evaluated from 128 samples and its fitting normal distribution curve, (c) The relation of 128 samples' SSIM, PSNR and Contextual loss (L2 norm) This encoding technique gains a great advantage over other encoder models. Since the network-based encoder is obtained by the training process, the fixed encoder can only produce one particular output to the encoded content, not well in solving randomness or overfitting. The multiple parallel processes can improve the output since it is not restricted by the networks. 2.6 Generalisation In this paper, we present a new method targeting compressing images from a certain data set. For real applications, the bits of the networks are necessary as the preknowledge for compressing images. In our method, we only need the decoder (generator) networks, which is an advantage over other neural network-based methods, which also require extra bits for the encoder. In Section 2.3, we present the network for our DCGAN models and estimate that the bits for the decoder (generator) are ∼72 MB. Compared with large data sets such as CelebA (our experiments only include the training data), which is ∼2.3 GB, it is relatively small and still efficient if transmitting or storing the whole data set. However, this was not our initial intention. Ours was to compress infinite images belonging to similar content and genre. For example, in social media applications, there is a need to process only the images of user portraits. In this situation, transmitting or storing images is mainly achieved by the application already knowing the preknowledg