Compressed domain image resizing based on saliency histogram equalisation and seam carving

The Journal of EngineeringVolume 2020, Issue 10 p. 870-876 Research ArticleOpen Access Compressed domain image resizing based on saliency histogram equalisation and seam carving Hsi-Chin Hsin, Corresponding Author Hsi-Chin Hsin hsin@nuu.edu.tw Department of Computer Science and Information Engineering, National United University, Miaoli, TaiwanSearch for more papers by this author Hsi-Chin Hsin, Corresponding Author Hsi-Chin Hsin hsin@nuu.edu.tw Department of Computer Science and Information Engineering, National United University, Miaoli, TaiwanSearch for more papers by this author First published: 02 October 2020 https://doi.org/10.1049/joe.2020.0116Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract With the increasing popularity of mobile devices and internet services, resizing of compressed images is in high demand. In this study, a compressed domain content-aware image resizing (CAIR) system is proposed. It takes a pair of meshes obtained by using saliency histogram equalisation and seam carving to convert the discrete cosine transform (DCT) blocks of the original image into the DCT blocks of the resized one without the burden of decompression and compression computations. Experimental results show that the proposed system is preferable to other existing CAIR algorithms, especially in terms of running time due mainly to the simplicity of histogram manipulation and dynamic programming. 1 Introduction In the world of multiple screens with different sizes, resolutions and aspect ratios and various devices like TV, PC, tablet and smartphone, to name, but a few, are abundantly present in our daily lives; thus, there is an increasing demand for effective displaying of images and videos. Conventional techniques using simple cropping and uniform scaling are usually at the risk of losing and distorting crucial information. To keep the regions of interest (ROI) intact as much as possible, the idea of content-aware image resizing (CAIR) has drawn a lot of attention in recent years [1]. State-of-the-art CAIR algorithms can be classified into two categories: the discrete approach and the continuous approach; the former shrinks/enlarges images by gradually removing/inserting unimportant pixels in a structured manner [2]. Avidan and Shamir [3] defined a seam as a path of connected pixels from one side of an image to the opposite side and proposed a novel algorithm known as seam carving (SC) for CAIR. To improve the performance of SC, Shamir and Sorkine [4] took account of more features related to the ROI. Hua et al. [5] combined SC with uniform scaling based on the similarity measure using scale-invariant feature transform. To speed up SC, Patel and Raman [6] proposed an adaptive method for multiple pixel wide SC in the spatial domain. In the work of Hsin et al. [7], seams of wavelet trees were constructed, based on which, a scale-recursive SC algorithm was proposed in the wavelet domain. In the continuous approach, the original image is partitioned into cells, and each cell is mapped onto the corresponding one in the resized image by mesh warping. Gal et al. [8] preserved the shapes of prominent objects while stretching and squeezing homogeneous regions via non-homogeneous mapping. In the work of Chen et al. [9], resizing of images was formulated as a convex quadratic problem, which can be solved by quadratic programming. Wang et al. [10] rescaled salient objects uniformly and concealed non-uniform distortions in the regions of non-interest (RON). Hsin [11] utilised the saliency histogram (SH) obtained by normalising the saliency map (SM) of an image to assess the presence of significant regions, and proposed the SH equalisation (SHE) algorithm [11] and the SHE-SC algorithm [12] to construct a pair of non-uniform meshes for image warping in the SH domain. Various CAIR algorithms have extended to video retargeting by imposing the temporal coherence of salient and moving objects [13-17]. In many cases, especially for the internet applications and mobile communications, images are typically compressed to effectively use limited resources, e.g. memory space and transmission bandwidth [18]. Fang et al. [19] proposed a discrete cosine transform (DCT)-based saliency detection model for image resizing in the joint photographic experts group (JPEG) compressed domain, and extended the above work to video saliency detection by incorporating the motion vectors obtained from a video bitstream in the final spatial–temporal SM [20]. In the work of Zhang et al. [21], a practical system was proposed for transcoding the original video bitstream to the retargeted one in the H.264/advance video coding compressed domain. This paper presents an efficient system based on the previous work [11, 12], to obtain the DCT blocks of the resized image directly from the DCT blocks of the original image in an attempt to avoid the unnecessary computations of inverse and forward DCTs. The rest of the paper is organised as follows. Related work is briefly reviewed in Section 2. The proposed compressed domain image resizing (CDIR) system is detailed in Section 3. Experimental results are shown in Section 4. Section 5 summarises the advantage of resizing images in the compressed domain and the significance of the contribution in concluding the paper. 2 Related work The SH obtained by normalising the SM of an image provides useful information on the ROI, which is beneficial to CAIR. This paper takes the SH approach to the problem of CAIR in the compressed domain. Two of the related algorithms, SHE [11] and SHE-SC [12] are reviewed in this section. The advantage of SHE and SHE-SC is due largely to computational simplicity. 2.1 SHE algorithm In the framework of image warping, a pair of meshes are needed to map the original image onto the resized one. For effective CAIR, the ROI with high-SH values should be well preserved at the cost of ignoring the RON with low SH values. As contrast enhancement can be achieved by image histogram equalisation, SHE enhances the ROI by equalising the SH of an image likewise. Specifically, let denote the SH of an image I, the marginal SH (MSH) (1) is used to partition I into L vertical strips with the equal accumulated amount of SH values as follows: (2) where is the coordinate of the lower right vertex of the n th vertical strip in the y -direction, , with the boundary condition . Similarly, the partition of K horizontal strips with the equal accumulated amount of SH values can be obtained by equalising the MSH in the x -direction. Take the separable 2D mesh constructed above as an initial mesh the refinement of vertices can be obtained by using the conditional expectation (3) (4) where is the refinement of vertex based on its neighbourhood denoted by , and are the local MSH given by (5) (6) The refining process proceeds iteratively until the difference of SH values between mesh quads is negligible. There are other constraints on the vertices of a proper mesh: avoidance of fold-over, prevention of drastic bending and boundary condition [11]. 2.2 SHE-SC algorithm SHE provides an efficient way to represent an image with a non-uniform mesh; the higher the SH value, the finer the mesh cell. To represent the resized image with an adaptive mesh, the SHE-SC algorithm [12] modifies the SC algorithm [3] by taking mesh cells as super-pixels. Let be a non-uniform mesh composed of quads, V, E and Q denote sets of vertices, edges and quads. Each quad, Q; , , can be considered as a super-pixel; thus, a seam of quads is equivalent to a seam of super-pixels. Specifically, a vertical seam of eight-connected quads in the x -direction can be defined as (7) where is a mapping from to , and the corresponding vertical seam of super-pixels is as follows: (8) where is the coordinates of the upper left vertex of , which is regarded as a super-pixel in the spatial domain, and therefore is equivalent to . Similarly, a horizontal seam of eight-connected quads in the y -direction can be defined as (9) where is a mapping from to , and the equivalent horizontal seam of super-pixels, , in the spatial domain is as follows: (10) To evaluate the significance of a quad (or equivalently, super-pixel), the following average SH can be used (11) where denotes the SH value at coordinates and is the size of , which is the number of pixels in . As M is obtained by SHE, each quad has the same accumulated amount of SH values given by (12) where N t is the total number of quads in M. Thus, is inversely proportional to , and the least significant seam (LSS) of quads is the one with the maximum sum of quad sizes. In other words, the LSS of quads in the vertical direction is as follows: (13) which can be efficiently obtained via dynamic programming [12]. By adjusting the width of each quad in appropriately, the width of M can be resized accordingly. To gradually adjust the width of a mesh, a sequence of increasingly significant vertical seams of quads is first identified, and each seam is resized by 1 pixel width. The height of a mesh can also be gradually resized in a similar manner. Based on the pair of meshes, one is obtained by SHE [11] to represent the original image, and the other with the desired size is obtained by SHE-SC [12], the resized image can be obtained by simple interpolation. 3 Image resising in the compressed domain This section presents an efficient system for CAIR in the compressed domain such that the burden of decompression and compression computations can be reduced. Specifically, it generates the DCT blocks of the resized image based on the DCT blocks of the compressed image without performing inverse and forward DCTs. 3.1 System overview Fig. 1 depicts the proposed system. It performs resizing operations on DCT blocks with three core modules: (i) construction of SM, (ii) creation of constrained meshes and (iii) conversion of DCT blocks. Fig. 1Open in figure viewerPowerPoint Diagram of the CDIR system First, SM is constructed based on the features extracted directly from the DCT blocks defined in JPEG. If the input is a JPEG code stream, there is a need for partial decoding to reconstruct DCT block coefficients; nevertheless, inverse quantisation and inverse DCT can still be saved. Next, mesh warping based on SM is adopted to convert the DCT blocks of the original image into the DCT blocks of the resized one. Details are given in the following sections. 3.2 Construction of SM Recent studies have shown that luminance, colour and texture features are suitable for saliency detection [19-22]. This paper takes the DC coefficients of DCT blocks as a down-sampled image [21] and applies the discriminative regional feature integration (DRFI) algorithm [23] for construction of luminance-colour SM (LCSM). In addition, the AC coefficients of DCT blocks are important features for images with textures as there is a great portion of energy in middle-to-high frequency sub-bands. The AC-based likelihood of texture [21] has been used for construction of texture SM (TSM). To combine the LCSM and TSM into the final SM, many fusion methods are available in the literature. The three commonly used fusion methods, namely product of maps, sum of normalised maps and maximum of normalised maps are as follows: (14) (15) (16) where , and denote the final SM, LCSM and TSM of , respectively, and is the normalisation operator. It is worth noting that many of the high frequency AC coefficients are equal to or close to zero. Consequently, the computational complexity of TSM is of order , where N denotes the number of the first several AC coefficients scanned from each DCT block in the zig-zag order to construct texture feature vectors. Specifically, would be sufficient in most cases [20]. Furthermore, only simple adders and booth multipliers are needed for the implementation of TSM [21]. On the other hand, the hardware implementation of DRFI is likely to be challenging, as the dimension involved is highly demanding. It raises another practical issue on the hardware design of LCSM, and there are various types of saliency detection algorithms in the literature [24], which is beyond the scope of the paper. 3.3 Creation of constrained meshes The continuous approach to CAIR is typically formulated as non-linear warping in spatial domain. It takes a pair of meshes, namely the input mesh representing the original image and the output mesh representing the resized image such that warping of salient objects can be well-preserved while deforming of non-salient regions and backgrounds is tolerable. As the SM obtained in the first module is dependent on DCT blocks, the SM approach is suitable for mesh warping in the compressed domain. Existing SM-based mesh creation algorithms usually take a uniform mesh as the input mesh, and deform it into a non-uniform mesh used as the output mesh. However, as the output of the proposed CDIR system is the DCT blocks of the resized image and the size of DCT blocks is basically fixed, the output mesh should be uniform, and moreover, the size of cells needs to be the same as that of DCT blocks. Instead of creating the uniform output mesh directly, the proposed method is a two-step algorithm: initial construction of a pair of non-uniform meshes followed by a simple geometric transformation under the aforesaid uniform constraint on the output mesh. Due to the advantages of histogram manipulation and dynamic programming, the SHE algorithm [11] and the SHE-SC algorithm [12] have been used to construct the initial input and output meshes denoted by and , respectively. Let be the desired constrained output mesh, the corresponding input mesh, , can be obtained by applying a simple bilinear transformation to , and as follows (17) where is a vertex in , is the corresponding vertex in , and are the parameters; can be determined based on the four pairs of vertices: ; and is the set of 4 neighbouring vertices of in . Fig. 2 depicts the schematic relationship between pairs of meshes and via (17). Although bi-cubic transformation can be used to enhance smoothness at the cost of computational efficiency, experimental results show that there is little gain, more specifically almost indistinguishable from bilinear transformation especially for image shrinking. Fig. 2Open in figure viewerPowerPoint Schematic relationship between pairs of meshes and It is worth mentioning that other SM-based algorithms can be used to construct the initial pair of meshes, ; and in case of non-quadrilateral cells, a polynomial transformation [25] is still available for the constrained pair of meshes . 3.4 Conversion of DCT blocks The objective of this module is to convert the DCT blocks of the compressed image into the DCT blocks of the resized image. For the sake of computational simplicity, take the initial pair of meshes obtained by using one-dimensional (1D) SHE and 1D SHE-SC into consideration. Thus, the final pair of constrained meshes, , is composed of rectangular cells, and therefore the arbitrary downsizing (AD) algorithm [26] is applicable to the problem of converting DCT blocks from into . Specifically, let a and b denote a pair of corresponding cells in and , respectively, as shown in Fig. 3. As each of the cells in corresponds to a DCT block in the resized image, the DCT of b can be obtained from the set of DCT blocks related to a by using the AD algorithm as follows: (18) where , , corresponds to the DCT blocks overlapped fully or partially with a, and are the pre-matrix and post-matrix defined as (19) is an identity matrix of size , which is actually the size of DCT blocks and cell b; cell a is of size , and are the DCT representations of the shift matrices defined in [26], whose sizes are and , respectively. It is noted that the size of and is , which are independent of the input DCT blocks, , and can be computed and stored in advance. As a result, through the use of look-up table, the proposed CDIR system is time efficient. In addition, as the memory space required to store the pre-matrix and post-matrix is solely determined by the size of DCT blocks, which is normally set to in practice, the additional hardware cost is marginal. Fig. 3Open in figure viewerPowerPoint Conversion of DCT block, b, in the resized image from the DCT blocks, , in the original image It is difficult to convert DCT blocks from a non-uniform mesh to a uniform mesh, as one can expect, which is likely to be extremely time-consuming and resource-intensive by extending the AD algorithm, and thus is not practical for the development of fast CDIR system. 4 Experimental results The CDIR system has been implemented in Matlab ® on a PC equipped with Intel® Core™ i7 2.2 GHz CPU. The size of DCT blocks is . As suggested in [24], the DRFI algorithm [23] outperforms other existing algorithms and therefore has been applied to the DC coefficients of DCT blocks to construct LCSM. The AC-based texture likelihood [21] has been used as TSM. The final SM is obtained by combining LCSM and TSM through the product of maps. The benchmark image data set [27] was used for objective evaluation. Fig. 4 shows the performance of CDIR compared with the spatial domain SHE-SC [12]. The SMs of the test images shown in Fig. 4 a are obtained by using DRFI [23] in the spatial domain and module 1 of CDIR in the compressed domain. As can be seen from Figs. 4 b and c, the block-based DCT approach tends to reduce resolution because SM is constructed at the block level. Nevertheless, as shown in Figs. 4 d and e, which are the resized images obtained by using SHE-SC with SMs, Figs. 4 b and c, respectively, the SH used as a probability density function is suitable for moderate variations in the construction of SM. Moreover, by comparing Fig. 4 e with Fig. 4 f, CDIR is visually comparable to SHE-SC, yet CDIR is preferable to SHE-SC with regard to saving of inverse and forward DCTs. Fig. 4Open in figure viewerPowerPoint Visual comparison (a) Test images, (b), (c) SMs obtained by using DRFI [23] in spatial domain and the module 1 of CDIR, respectively, (d)– (f) Resized images obtained by using SHE-SC [12] with SMs, (b) and (c) and CDIR with SM (c) In addition to SHE-SC, other existing spatial domain image resizing (SDIR) algorithms: SC [3] and gradient vector flow paths (GVFP) [27] have been used for comparison with CDIR. The width/height of images shown in the first column of Fig. 5 is reduced by 30%. It is observed that there are partially distorted regions using SC; and CDIR is comparable to both SHE-SC and GVFP. However, only CDIR resizes images in the compressed domain. Fig. 5Open in figure viewerPowerPoint Results of resizing images (first column) at 70% of width/height using CDIR (second column), and the SDIR algorithm: SHE-SC (third column), SC (fourth column) and GVFP (fifth column) To evaluate CDIR and SDIR objectively, the average cost defined in [27], which measures the difference between the original image and its resized one in terms of salient objects has been adopted. The benchmark data set [27] is composed of 1000 images with their respective ground-truth salient regions. The height and width of the images range from around 200 to 400 pixels. Fig. 6 shows the average costs of resizing images by factors ranging from 0.7 to 0.95 using SC, GVFP, SHC-SC in the spatial domain and CDIR in the compressed domain. As noted, CDIR is comparable to SHE-SC, preferable to GVFP and outperforms SC. Fig. 6Open in figure viewerPowerPoint Objective evaluation based on the average cost of resizing 1000 benchmark images [27] by factors ranging from 0.7 to 0.95 using SC (dashed line), GVFP (dash-dotted line), SHC-SC (dotted line) in the spatial domain and CDIR (solid) in the compressed domain Excluding decompression/compression times, Table 1 gives the average running time for resizing images at 70% with respect to the maximum dimension of the original image using the above-mentioned algorithms. As the goal of CDIR is to resize images in the compressed domain, the computation time of IDCT, which is needed for decompression of resized images, has not been taken into account. It reveals that CDIR and SHE-SC are superior to SC and GVFP. Moreover, the computation time of both SC and GVFP is dependent on the size of images, as they are performed in the pixel domain. On the other hand, SHE-SC and CDIR are principally dependent on the size of meshes and therefore preferred in terms of running time. Table 1. Average running time for resizing images at 70% with respective to the maximum dimension of the original image using SC, GVFP, SHE-SC, and CDIR Average running time, s Algorithm 21.59 SC 25.14 GVFP 0.18 SHE-SC 0.16 CDIR 5 Conclusion Visual saliency plays a key role in many applications. As images are generally compressed in order to make effective use of transmission bandwidth and storage space, it is preferred that the SH of an image can be obtained in the compressed domain. The CDIR system, which converts DCT blocks from the original image to the resized one, is time-efficient by taking account of the computational simplicity of SHE and SC. Compared to other existing CAIR algorithms, the CDIR system is visually comparable (or marginally superior), and especially preferable in terms of running time. Moreover, the advantage of using CDIR is that there is no need to perform inverse and forward DCTs. 6 Acknowledgments This study was supported by the Ministry of Science and Technology and Taiwan supported this work under the Grant MOST 108-2221-E-239-017. 7 References 1Hsu C.C., Lin C.W., Fang Y. et al.: ‘Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss’, IEEE J. Sel. Top. Signal Process., 2014, 8, (3), pp. 377– 389 2Kiess J., Kopf S., Guthier B. et al.: ‘A survey on content-aware image and video retargeting’, ACM Trans. Multimedia Comput. Commun. Appl. (TOMM), 2018, 14, (3), p. pp. 1– 28, article no. 76 3Avidan S., Shamir A.: ‘Seam carving for content-aware image resizing’, ACM Trans. Grap. Siggraph, 2007, 26, pp. 1– 10 4Shamir A., Sorkine O.: ‘Visual media retargeting’. Proc. ACM Siggraph Asia, Yokohama Japan, 2009, pp. 1– 13 5Hua S., Chen G., Wei H. et al.: ‘Similarity measure for image resizing using SIFT feature’, EURASIP J. Image Video Process., 2012, 2012, p. 11 6Patel D., Raman S.: ‘Accelerated seam carving for image retargeting’, IET Image Process.., 2019, 13, (6), pp. 885– 895 7Hsin H.-C., Chan Y.-C., Su C.-K.: ‘Wavelet saliency map and its application to image resizing’. APCEAS 2015, Osaka, Japan, 2015, pp. 146– 152 8Gal R., Sorkine O., Cohen-Or D.: ‘Feature-aware texturing’. Proc. Eurographics Symp. on Rendering, Nicosia, Cyprus, 2006, pp. 297– 303 9Chen R., Freedman D., Karni Z. et al.: ‘Content-aware image resizing by quadratic programming’. Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA., 2010, pp. 13– 18 10Wang Y.S., Tai C.L., Sorkine O. et al.: ‘Optimized scale and stretch for image resizing’, ACM Trans. Graph., 2008, 27, pp. 118– 125 11Hsin H.-C.: ‘Saliency histogram equalization and its application to image resizing’, IET Image Process.., 2016, 10, (10), pp. 787– 798 12Hsin H.-C.: ‘Combination of saliency histogram equalization and seam carving for image resizing’, IET J. Eng., 2017, 2017, pp. 285– 291 13Rubinstein M., Shamir A., Avidan S.: ‘Improved seam carving for video retargeting’, ACM Trans. Graph., 2008, 27, (3), pp. 1– 9 14Krähenbühl P., Lang M., Hornumg A. et al.: ‘A system for retargeting of streaming video’, ACM Trans. Graph., 2009, 28, (5), p. pp. 1– 10 15Wang Y.-S., Lin H.-C., Sorkine O. et al.: ‘Motion-based video retargeting with optimized crop-and-warp’, ACM Trans. Graph., 2010, 29, (4), p. pp. 1– 9 16Li B., Duan L.-Y., Wang J. et al.: ‘Spatiotemporal grid flow for video retargeting’, IEEE Trans. Image Process., 2014, 23, (4), pp. 1615– 1628 17Hsin H.-C.: ‘Video retargeting based on SH equalization and seam carving’, IET Image Process.., 2019, 13, (8), pp. 1333– 1340 18Zhou Y., Zhang L., Zhang C. et al.: ‘Perceptually aware image retargeting for mobile devices’, IEEE Trans. Image Process., 2018, 27, (5), pp. 2301– 2313 19Fang Y., Chen Z., Lin W. et al.: ‘Saliency detection in the compressed domain for adaptive image retargeting’, IEEE Trans. Image Process., 2012, 21, (9), pp. 3888– 3901 20Fang Y., Lin W., Chen Z. et al.: ‘A video saliency detection model in compressed domain’, IEEE Trans. Circuits Syst. Video Technol., 2014, 24, (1), pp. 27– 38 21Zhang J., Li S., Kuo C.-C.J.: ‘Compressed-domain video retargeting’, IEEE Trans. Image Process., 2014, 23, (2), pp. 797– 809 22Theoharatos C., Pothos V.K., Laskaris N.A. et al.: ‘Multivariate image similarity in the compressed domain using statistical graph matching’, Pattern Recognit.., 2006, 39, (10), pp. 1892– 1904 23Jiang H., Wang J., Yuan Z. et al.: ‘Salient object detection: a discriminative regional feature integration approach’. IEEE Int. Conf. Computer Vision and Pattern Recognition, Portland, Oregon, USA., 2013, pp. 2083– 2090 24Borji A., Cheng M.-M., Jiang H. et al.: ‘Salient object detection: a benchmark’, IEEE Trans. Image Process., 2015, 24, pp. 5706– 5722 25Gonzalez R.C., Woods R.E.: ‘ Digital image processing, Pearson’, 2017 26Shu H., Chau L.-P.: ‘An efficient arbitrary downsizing algorithm for video transcoding’, IEEE Trans. Circuits Syst. Video Technol., 2004, 14, (6), pp. 887– 891 27Battiato S., Farinella G.M., Puglisi G. et al.: ‘Saliency-based selection of gradient vector flow paths for content aware image resizing’, IEEE Trans. Image Process., 2014, 23, (5), pp. 2081– 2095 Citing Literature Volume2020, Issue10October 2020Pages 870-876 FiguresReferencesRelatedInformation