Supervised‐learning‐based algorithm for color image compression

ETRI JournalVolume 42, Issue 2 p. 258-271 ORIGINAL ARTICLEFree Access Supervised-learning-based algorithm for color image compression Xue-Dong Liu, Corresponding Author Xue-Dong Liu zxndp@126.com orcid.org/0000-0002-6598-953X Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, China Correspondence Xue-Dong Liu, Key Lab. of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, China. Email: zxndp@126.comSearch for more papers by this authorMeng-Yue Wang, Meng-Yue Wang Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, ChinaSearch for more papers by this authorJi-Ming Sa, Ji-Ming Sa Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, ChinaSearch for more papers by this author Xue-Dong Liu, Corresponding Author Xue-Dong Liu zxndp@126.com orcid.org/0000-0002-6598-953X Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, China Correspondence Xue-Dong Liu, Key Lab. of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, China. Email: zxndp@126.comSearch for more papers by this authorMeng-Yue Wang, Meng-Yue Wang Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, ChinaSearch for more papers by this authorJi-Ming Sa, Ji-Ming Sa Key Laboratory of Broadband Wireless Communications and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan, Hubei, ChinaSearch for more papers by this author First published: 22 October 2019 https://doi.org/10.4218/etrij.2018-0557Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract A correlation exists between luminance samples and chrominance samples of a color image. It is beneficial to exploit such interchannel redundancy for color image compression. We propose an algorithm that predicts chrominance components Cb and Cr from the luminance component Y. The prediction model is trained by supervised learning with Laplacian-regularized least squares to minimize the total prediction error. Kernel principal component analysis mapping, which reduces computational complexity, is implemented on the same point set at both the encoder and decoder to ensure that predictions are identical at both the ends without signaling extra location information. In addition, chrominance subsampling and entropy coding for model parameters are adopted to further reduce the bit rate. Finally, luminance information and model parameters are stored for image reconstruction. Experimental results show the performance superiority of the proposed algorithm over its predecessor and JPEG, and even over JPEG-XR. The compensation version with the chrominance difference of the proposed algorithm performs close to and even better than JPEG2000 in some cases. 1 INTRODUCTION Advances in information technology have led to an increased use of multimedia such as images and videos to share information over the Internet. Furthermore, users are increasingly demanding better quality multimedia as well as faster transmission rates. To meet these demands, researchers are continuously pursuing improvements in image compression techniques. It is well-known that redundancy exists in the adjacent pixels of an image. By exploiting this redundancy, an image can be compressed to reduce the necessary storage space and transmission bandwidth. The existing compression techniques such as JPEG 1, JPEG2000 2, and JPEG-XR 3 transform images into the frequency domain, and then store the coefficients of the frequency domain, which are encoded elaborately. At the decoder, the coefficients are transformed inversely into the spatial domain to reconstruct the original image either precisely or approximately. Traditionally, a color image coder processes three color channels independently. In fact, an interchannel correlation exists between the three color components of an image. This correlation has been exploited to improve the coding efficiency of the state-of-the-art video coding standard, that is, High Efficiency Video Coding (HEVC) 4-6. Models that exploited interchannel correlation using the reconstructed luma to linearly predict the chroma with parameters derived from neighboring reconstructed luma and chroma pixels for HEVC were proposed in 5 and 6. In colorization-based coding, the encoder selects a few representative pixels (RPs) whose chrominance values and positions are stored or transmitted together with the luminance component; in the decoding stage, the chrominance values for all the pixels are colorized by the RPs and the decompressed luminance image. The main issue in colorization-based coding is how to extract the RP so that the rate-distortion (RD) performance is good. Lee and others 7 formulated the colorization-based coding problem into an L1 minimization problem. For a fixed reconstruction error value and a given colorization matrix, the selected set of RPs is guaranteed to be the smallest set. To improve the image coding performance, Uruma and others 8 proposed the RP compression algorithm using the graph Fourier transform, and accordingly introduced the colorization technique using a representative graph spectrum based on the graph Fourier transform. Recently, remarkable progress has been made in the application of machine learning technologies to image compression. Reference 9 proposed the use of color information from a few RPs to learn a model that predicts the colors of the rest of the pixels and recovers the original image by storing the RPs and grayscale edition of the image. Based on the statistical theories of optimal experimental design 10, He and others 11 proposed an active learning algorithm, called graph-regularized experimental design (GRED), and showed that it was more effective than the method used in 9. Noting that all the real colors are known at the encoding stage, 12 proposed an active learning algorithm that selects a subset of pixels as seeds, that is RPs, to achieve total prediction error minimization (TEM) of chrominance. The seeds were stored together with the luminance component for the decoder, which were used to train a prediction model with a semi-supervised-learning algorithm. The model was then applied to predict the colors of all the pixels. It has been reported that TEM outperforms the methods proposed in 9 and 11. The three aforementioned algorithms share two key procedures: selecting representative color seed pixels by means of standard active learning and predicting color values with a model trained by a semi-supervised-learning algorithm. Seed point selection procedures by active learning are time-consuming. In fact, it is not necessary to use complex active learning to ensure total error minimization. Because all the color values are available at the encoder, we can use supervised learning to construct a Laplacian-regularized least square (LapRLS) objective function to minimize the total prediction error naturally over all the training points. Consequently, time-consuming active learning is omitted. In view of the massive number of training points and size of matrices in the expression of the optimal solution of the prediction model as well as the resultant computational burden, we use kernel principal component analysis mapping (KPCAM) 13. KPCAM generates a space with reduced dimension m instead of the original number of training points n. Aside from KPCAM, effective strategies such as chrominance subsampling and entropy coding are applied to further reduce the computational cost and bit rate. In experiments, we explored the laws governing the effects of various factors on the RD performance of the proposed algorithm. The rest of this paper is organized as follows. A brief review of the previous similar works from which the proposed algorithm evolved is given in Section 2. Our supervised-learning-based algorithm (SLBA) for color image compression is described in Section 3. Experimental results are discussed in Section 4, and conclusions are drawn in Section 5. 2 PREVIOUS SIMILAR WORKS Exploiting the correlation between different channels of a color image to further reduce data size is a recent concept. In previous similar works 9, 11, 12, semi-supervised learning was utilized to minimize the loss of labeled examples for training and then generalized to all examples for prediction. Graph-based semi-supervised-learning methods construct a problem graph G to denote the neighborhood relation among examples (both labeled and unlabeled). A function of the G acts as a regularizer to implement the manifold assumption or local consistency assumption 13. Then, another regularizer is added to ensure the numerical stability of the solution. Integration of the two regularizers with prediction error forms the LapRLS objective function, whose optimal solution leads to a prediction or regression model. The optimal solution is of a closed form that deals with the inverse of an n × n matrix, where n is the number of total examples, including both labeled and unlabeled ones. To select the most representative seed points, the previous methods used various active learning schemes. Reference 9 used a heuristic strategy to select the RPs from regions where a high prediction error occurs. Reference 11 selected seeds such that the determinant of the covariance matrix of the regression coefficients is minimized to ensure that the optimal solution is as stable as possible. Making full use of the fact that all true colors are available at the encoder, Reference 12 selected the most informative pixels to ensure that the total prediction error is minimized and the algorithm outperforms the two previous ones. In the aforementioned machine learning-based color image compression methods, luminance component Y was stored whereas the two chrominance components, Cb and Cr, were predicted from a model f trained by semi-supervised learning. f predicts the color values of Cb and Cr corresponding to the input feature vector (R, C, I), where (R, C) is the location of a pixel, and I is the luminance value. The predicted chrominance values are called labels, denoted by y. In semi-supervised learning, a set of labeled pairs {(x1, y1), …, (xl, yl)} is needed to train the prediction model while another set of unlabeled pixels {xl+1, …, xn} helps improve the learning performance of model f, where xi is the ith feature point and yi is the corresponding label. References 9, 11, 12 present three algorithms of the same type. The proposed algorithm SLBA evolves from TEM 12, which is considered the best one of its kind so far. As a variant of TEM, SLBA outperforms it. For a better understanding of SLBA and the reasons why it is superior to TEM, we introduce TEM briefly as follows. In TEM, the regression model is trained by all the training points although the objective function attempts to minimize the prediction error only on the smaller number of labeled points. This is shown in where X is the set of labeled points and U is the set of all training points; YX is a vector containing color labels corresponding to set X; FX and FU are the vectors containing color values predicted by f corresponding to X and U, respectively; L is the so-called graph Laplacian matrix; λ1 and λ2 are two constants; ||∙|| is the norm operator; T denotes the transpose operation. The first term is the prediction error over the labeled points. The two latter terms are the regularizers that help improve the regression model. The points in set X are called color seeds. This is the semi-supervised-learning paradigm. Seed selection plays a very important role because different seeds may result in different prediction performances. Reference 12 adopted active learning to select points as seeds so that f can minimize the total prediction error. There are two schemes to select the optimal seed pixel points: one starts with an empty set X and then adds the selected color seeds iteratively; the other starts with all the pixels, that is, X = U and then removes pixels iteratively. The latter is adopted in TEM because it performs better. The selection process also needs to compute the total prediction error each time a point is removed. This is implemented exhaustively for every candidate point. Those points are removed if the resultant X can produce the total prediction error minimum. The iteration continues until the predefined number of points in set X, that is l, is reached. The selection of seed points is a time-consuming process. In summary, TEM uses an active learning algorithm to select a subset of pixels as seeds. The stored luminance and color seeds are used to train a regression model by a semi-supervised-learning algorithm. The model is then applied to all the pixels to predict their chrominance values. The selection of seeds is time-consuming, and the storage requirement of the seeds is higher. In view of these disadvantages of TEM, we developed SLBA. 3 SUPERVISED-LEARNING-BASED ALGORITHM FOR COLOR IMAGE COMPRESSION 3.1 Supervised-learning by LapRLS Our work focuses on color image compression by exploiting interchannel correlation. The proposed SLBA evolves from the semi-supervised-learning approach, LapRLS, mentioned earlier 12. The previous objective function of LapRLS shared a term denoting the prediction error over only l labeled pixels. In addition, there are other regularizers constructed by all (both labeled and unlabeled) n training pixel points in the objective function. The resultant optimal solution that minimizes the objective function involves the inverse of a large-sized matrix. In TEM, there are n training points, but the learnt f minimizes the prediction error over only l (l « n) labeled points. To achieve total error minimization, TEM launches an iterative active learning procedure, which accounts for the majority of time consumed. In fact, we can achieve total error minimization by modifying the objective function of LapRLS, thereby avoiding the time-consuming active learning process. Compared with previous methods based on semi-supervised and active learning, it is necessary to highlight the improvements and innovations proposed to the method in this paper. SLBA can minimize the total prediction error without the time-consuming active learning procedure. KPCAM is adopted to reduce computational cost. The required m representative points are sampled from the same fixed grid of the reconstructed luminance component, which makes it possible for both the encoder and decoder to use the same parameters to predict chrominance values of pixels without signaling location information. Unlike previous methods 9, 11, 12, which need a set of color seeds, including location, luminance, and chrominance values, SLBA only uses 2m parameters to convey Cb and Cr information. SLBA further refines our previous method 14. The fractional parameters are rounded off and then entropy coded, which improves the RD performance of SLBA because the needed bit overhead is lower. Finally, utilizing the insensitivity of the human eye to chrominance, SLBA improves our previous method 14. Chrominance subsampling is adopted, which results in a lower bit rate and remarkable RD performance. In TEM, the term in the LapRLS objective function denoting prediction error over l labeled points is , where xi is the ith seed point in X and yi is the corresponding label in YX. Because all pixel colors are available at the encoder, we modified the term to , where ui is the ith training point and n is the total number of training points. This is the supervised-learning paradigm that attempts to minimize the prediction error over all the n training pixels: (1)where YU = [y1, …, yn]T and FU = [ f (u1), …, f(un)]T. To enhance the nonlinear prediction ability, we will learn a function f in the reproducing kernel Hilbert space (RKHS) 15. According to the representer theorem, there exists a set of coefficients αi so that f can be expressed as a linear combination , where is a kernel function that satisfies the reproducing kernel property <f, k(u,·)> = f(u) in the RKHS, and <·,·> denotes the inner product operation. To ensure the numerical stability of the solution of the objective function, we attempt to minimize which forms a regularizer in the objective function, where ||·|| denotes the norm operation. Using the reproducing kernel property, we have (2)where α = [α1 ··· αn]T, with the superscript T denoting transpose operation and K is the symmetrical kernel Gram matrix with elements Kij = k(ui, uj). In addition, we make the local consistency assumption or manifold assumption 13 that if two pixels are close and have similar luminance values, then their chrominances should be similar. This means that if ||ui − uj|| is small, then |f(ui) − f(uj)| should also be small. This assumption forms another regularizer that helps improve the learning performance. The regularizer can be formulated by spectrum graph theory 16. According to the theory, the similarity information between points is represented by an adjacency matrix W, in which elements Wij = 1 if ui and uj are close enough; otherwise, Wij = 0. We adopted the k-nearest neighbors (KNN) graph to construct matrix W, that is, Wij = 1 if ui is within the k-nearest neighbors of uj, or uj is within the k-nearest neighbors of ui. Then, we obtain a diagonal degree matrix D with elements . Matrix L = D−W is referred to as the graph Laplacian matrix. According to the manifold assumption, should be small and can be formulated concisely as G (f) = FUTLFU, where FU is defined as before. Integrating (1) with the two aforementioned regularizers, we obtain the LapRLS objective function: (3)where λ1 and λ2 are constants similar to the Lagrange multiplier. Considering the representer theorem, we have (4) Substituting (2) and (4) into (3), we obtain (5)which becomes a minimization problem with respect to the coefficient vector α. Let We obtain the optimal solution (6)where I is an n × n identity matrix. 3.2 Improve computational efficiency It can be seen from (6) that an n × n matrix must be inverted to calculate optimal α*. The calculation complexity is O(n3). When the image is large, the calculation imposes a huge computational burden. Moreover, the prediction using (4) needs all the n data points even if only one output f(u) is to be computed. In this section, we introduce a nonlinear mapping method to transform the data points into an m-dimensional space so that linear LapRLS can be conducted. As a result, the complexity depends on the space dimension m instead of the number of data points n. KPCAM was adopted to achieve the nonlinear map φz: X → ℝm, where in which Z = {z1, …, zm} is a selected subset of the training point set U and KZZ is the kernel Gram matrix whose (i, j)th element is a certain nonlinear kernel function value k(zi, zj). We use notation φi to denote φz(ui) for simplicity. The linear LapRLS can be implemented using linear kernel function k(φi, φj) = φiTφj in the RKHS so that the regression model is. (7)where It is clear that f becomes a linear function with respect to coefficient vector . Let and In addition, regularizer exhibits a new form: (8) Substituting FU in (3) with FΦ and (8) into (3), we get the new form of the objective function Let . The optimal solution is determined as (9)where Im is an m × m identity matrix. Compared with (6), (9) needs to calculate the inverse of a matrix of m × m instead of n × n. The complexity of computing w* is O(m3). When m is significantly smaller than n, the computational efficiency can be improved significantly. 3.3 Complete codec framework The proposed complete codec algorithm is summarized as follows. After an image is input to the encoder, color space conversion is first performed to get luminance component Y and chrominance components Cb and Cr. Because the human eye is insensitive to chrominance, we implement chrominance subsampling; the subsampling pattern is 4:2:0, as shown in Figure 1. Y is coded by the standard technique. The code stream of Y is transmitted through the channel or stored in the memory. The code stream is decompressed at both the encoder and decoder into a reconstructed luminance Figure 1Open in figure viewerPowerPoint Chrominance subsampling of 4:2:0 In theory, n in (1) is the number of pixels in an image. In the supervised-learning paradigm, n is also the number of training points. It is computationally very demanding that all the pixels act as training points because LapRLS needs to construct a large adjacency matrix W. Therefore, it is necessary to reduce the computational cost and memory requirements. Selecting only the representative points for training is a natural solution. Reference 9 segmented the image using a normalized cut 17 into regions. From each region, a pixel is randomly selected for subsequent learning. However, the segmentation itself imposes a high computational cost. To achieve a trade-off between the computational cost and representativeness of selected training points, we uniformly subsample n pixel points in as training points for supervised learning, which is similar to but simpler than the schemes in 9, 11, 12. The training points are sampled from the same grids of at the coder and decoder, which ensures that the predictions generated at the two ends are equal. The experimental results show that this sampling scheme is effective despite its simplicity. To implement KPCAM, a set that includes m points, that is, Z = {z1, …, zm} ⸦ U, must be determined. In general, points representative of U are chosen as ziʹs. In 12, k-means clustering was used to ensure the representativeness of ziʹs. Here, we sought a more feasible and effective scheme. The experimental results show that simple uniform sampling from the training points can determine the ziʹs by which satisfactory predictions can be made. Hence, we resampled the n training points to obtain zi (i = 1, …, m) for KPCAM, as shown in Figure 2. The two coefficient vectors w*Cb and w*Cr determined by (9) are used to predict Cb and Cr, respectively. They are entropy coded and denoted by w*s hereinafter for simplicity. Figure 2Open in figure viewerPowerPoint Training points and representative points zi for KPCAM At the decoder, is sampled in the same way as that at the encoder to obtain the same training points and representative points {z1, …, zm} for KPCAM, without the need for location information. By means of KPCAM, the mapping φi of every ui can be computed, and then the two chrominance components are predicted by (7). In the end, a color image is recovered by color space conversion into the RGB space. The codec framework of SLBA is shown in Figure 3. Figure 3Open in figure viewerPowerPoint Codec framework of SLBA Similar to TEM, the proposed SLBA also has an improved version, called SLBA-Compensation (SLBA-C), a counterpart of TEM-Compensation (TEM-C) in 12, which uses prediction difference images to compensate the regression results. In SLBA-C, Cb and Cr are predicted by (7) at the encoder. Chrominance difference images are generated by subtracting the predicted Cb and Cr from the original ones, and then coded by JPEG2000, as in TEM-C. At the decoder, the decoded difference images are added to the predicted Cb and Cr to refine the regression results. The codec framework of SLBA-C is shown in Figure 4. Figure 4Open in figure viewerPowerPoint Codec framework of SLBA-C In summary, SLBA is superior to TEM in three aspects. The first and the most important is that SLBA needs no time-consuming seed pixel point selection. Total error minimization is achieved by supervised learning instead of selecting optimal seed points. Second, chrominance information is conveyed by w*s that are entropy coded and occupy significantly less bits than are allocated to seeds in TEM. Third, utilizing the insensitivity of the human eye to chrominance, chrominance subsampling is adopted to further reduce computational cost and speed up implementation. 4 EXPERIMENTS In this section, we first present the implementation details. Then, we present the experimental results for various parameter configurations and the comparisons between SLBA/SLBA-C and other algorithms. Test images used in our experiments are shown in Figure 5, all of which are 512 × 512 in size. The first three images in the top row of Figure 5 are called Lena, baboon, and peppers, respectively. Figure 5Open in figure viewerPowerPoint Test images used in our experiments. Each image is 512 × 512 in size We measured the prediction accuracy with structural similarity (SSIM) 18 and peak signal-to-noise ratio (PSNR). The PSNR is formulated as PSNR = 10lg(2552/MSE), where is the mean squared error between the original image I and reconstructed image I'; M and N are the row and column sizes of the images, respectively; and ||·||F is the Frobenius norm 19. 4.1 Implementation details Here, n = 8192 points were sampled uniformly as training points from all the pixel points of the test images used in our experiments, as shown in Figure 2. We also subsampled luminance component Y, which is identical to the chrominance subsampling, to obtain training points for the prediction model. To compare the performance with TEM/TEM-C, which is reportedly the best among the homogeneous learning-based color image compression methods, we set the primary operation parameters to be the same as those in TEM. Specifically, λ1 and λ2 in (9) are 0.001 and 0.002, respectively; besides the linear kernel function in (7), we also adopted the Gaussian kernel function as the normal nonlinear kernel function, that is, ; parameter σ in the expression is also formulated as where is the mean of ui, with i = 1, …, n. We used the KNN graph to construct the adjacency matrix W mentioned in Section 3.1. Generally, the training pixel point (R, C, I) is the closest to the eight pixel points around it. A KNN graph with K = 8 (8-neighbor) should be superior to the case of K = 4 (4-neighbor) used in TEM. Tests were performed to verify this conjecture, which were conducted under the conditions where n = 8192, m = 2048, with the luminance component Y coded losslessly by JPEG2000, and the chrominance components Cb and Cr predicted with w*s rounded off. The average PSNRs of the reconstructed Cb and Cr by SLBA with the KNN graph of different K values are compared in Table 1. We can see that K = 8 results in a slightly improved performance compared with that of K = 4. Therefore, we set K = 8 in all our experiments. Table 1. Average PSNRs (dB) of predicted chrominance in SLBA with K = 4 and K = 8 in KNN graph for three images Image name K = 4 K = 8 Lena 39.9103 39.9122 Baboon 29.5927 29.5976 Peppers 33.0487 33.0536 The w*s determined by (9) are fractional. To reduce the bit overhead, all the entries of w*s are rounded off and then Huffman coded to further exploit the statistical redundancy. 4.2 Impact of m on performance of SLBA/SLBA-C The number of representative points in set Z used by KPCAM, denoted by m, impacts both the prediction accuracy and time consumed by SLBA/SLBA-C. PSNRs were calculated for the predicted Cr and Cb, and then the average PSNR was presented. To explore the impact of different m values, we compared the PSNRs of the reconstructed chrominance and the SSIMs of the reconstructed images Lena, baboon, and peppers. The metrics were evaluated under the condition that luminance and chrominance differences are coded losslessly by JPEG2000. The average PSNR and SSIM values are shown in Figure 6A and 6B, respectively. From the two figures we can see that the chrominance prediction quality measured by PSNR and SSIM increases with m in SLBA. This is because the regression model in (7) is of more generality when w*s are obtained by the larger set Z. However, when m exceeds 1024, the PSNR and SSIM curves become flat. This is because an excessive m leads to information redundancy. Figure 6 also shows that although the chrominance predicted by SLBA is inaccurate when the value of m is small, the chrominance quality by SLBA-C remains nearly constant. This is because prediction errors can be compensated even when they are large owing to the small value of m. The quality of the reconstructed chrominance by SLBA-C depends mainly on the compression ratio specified for the luminance component and the chrominance prediction error in the JPEG2000 coder. Figure 6Open in figure viewerPowerPoint (A) Average chrominance PSNR and (B) average SSIM at various values of m We also tested the time consumed by SLBA-C at various values of m, for example, 128, 256, 512, 1024, and 2048. Figure 7 depicts the time consumed by SLBA-C vs m. We can see that the tim