Accelerated seam carving for image retargeting

IET Image ProcessingVolume 13, Issue 6 p. 885-895 Research ArticleFree Access Accelerated seam carving for image retargeting Diptiben Patel, Corresponding Author Diptiben Patel diptiben.patel@iitgn.ac.in orcid.org/0000-0001-8848-9901 Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat, 382355 IndiaSearch for more papers by this authorShanmuganathan Raman, Shanmuganathan Raman Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat, 382355 IndiaSearch for more papers by this author Diptiben Patel, Corresponding Author Diptiben Patel diptiben.patel@iitgn.ac.in orcid.org/0000-0001-8848-9901 Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat, 382355 IndiaSearch for more papers by this authorShanmuganathan Raman, Shanmuganathan Raman Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat, 382355 IndiaSearch for more papers by this author First published: 15 April 2019 https://doi.org/10.1049/iet-ipr.2018.5283Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Display of images on different display devices having varied size and aspect ratio requires one to resize them. Many attempts have been made to perform content-aware image retargeting while generating an image compatible with a target display size. Seam carving is one of the image retargeting operators which alters the size of an image by removing least energy pixels. However, it requires high computational time in order to perform retargeting. In this study, the authors accelerate the naive seam carving process by removal or insertion of multiple pixel wide batch seam in a single iteration rather than a single pixel wide seam. Along with the energy of pixels to be removed, inserted energy after the removal of a batch seam is also minimised in order to prevent the inclusion of false edges. The width of a batch seam is a critical factor which is made adaptive during the retargeting process to preserve the energy of an image. They have shown a significant decrease in computational time with the increase in the width of a batch seam. They have compared the proposed technique with other state-of-the-art image retargeting operators using different quality assessment metrics and visual results. 1 Introduction In the era of internet, digital media (such as image or video) and its access through diverse display devices have drawn a lot of attention. It has led to the requirement of resizing a media according to the size, aspect ratio, and screen resolution of a target display device. Traditional ways of resizing an image such as scaling (SCL) and cropping (CR) are unable to preserve the salient content and/or its aspect ratio, though they are able to preserve the global structure of an image. The SCL of an image alters the aspect ratio of salient contents by squeezing or stretching it in one direction. While CR preserves the aspect ratio of salient contents, it may remove the important information at the image border. Alternatively, content-aware image retargeting techniques use knowledge about the image to preserve the salient contents and their aspect ratios altogether. The knowledge about the image is mapped to energy map signifying relative importance of pixels. Among the many techniques proposed for content-aware image retargeting, seam carving is an efficient technique proposed by Avidan et al. [1]. One pixel wide path, called seam, of lowest energy, is found from top to bottom (vertical) or left to right (horizontal) of an image. The lowest energy seam is inserted or removed in an iterative manner in order to fit the image onto the target display device. The removal or insertion of the lowest energy seam incurs a very small change in the overall energy of an image. Seam carving technique assures the smallest change in the overall energy of an image but faces a problem of computational complexity. In order to change the size of an image by just one column/row, it requires energy map calculation and finding a seam using dynamic programming or graph cut formulation. The motive of this study is to accelerate the seam carving process by finding multiple connected seams (we call them batch seam) and evaluate the performance of removal/insertion of a batch seam as compared to a single seam removal/insertion process. In a smooth region or a low energy region of an image, the neighbouring pixels tend to have similar features (the same is not true at the edges). There is very less probability of creating an edge of higher magnitude while removing or inserting a seam of single pixel width passing through the low-energy region. However, spatially distant pixels in a row or a column become new neighbours after removing or inserting a consecutive array of pixels (during batch seam carving). The new neighbours may create higher magnitude edges which incur undesirable information not available in an original image. We minimise the magnitude of these undesired new edges using the notion of forward energy. The forward energy takes care of the minimum energy to be inserted along with the minimum energy removed. The major contributions of the proposed retargeting technique are listed below. The proposed image retargeting technique exploits the formulation of finding a batch seam of more than one pixel width using dynamic programming. The removal or insertion of a batch seam is more prone to energy insertion in terms of undesired edges. These edges are prevented using forward energy minimisation during the seam carving process. The proposed technique accelerates the retargeting process with an increase in the width of the batch seams while achieving image retargeting performance comparable to that of single pixel width seam carving and that of other state-of-the-art image retargeting operators. The width of a batch seam is varied according to the image content in order to preserve the overall energy of an image during the seam carving process. The rest of the paper is organised as follows. Section 2 gives a brief overview of some of the retargeting works. Section 3 explains about energy map calculated for the retargeting process. Having an energy map, the batch seam carving process for accelerating seam carving is formulated. The energy of an image is preserved by varying the width of batch seam during the retargeting process. Detailed analysis of results and its variations are explored in Section 4. We conclude the paper in Section 5 with the future scope of work. 2 Related work Content-aware applications in computer vision tend to define the energy map signifying relative importance of each pixel in an image/video. Many techniques are proposed to define energy map of an image using attention model, saliency, edge detection, object detection, to name a few. Content-aware image retargeting techniques use this energy map to resize an image preserving the total energy of an original image. Traditional techniques of SCL and CR are improvised by making them adaptive with the knowledge of energy map of an image. Besides traditional techniques, content-aware image retargeting techniques are mainly categorised as continuous image retargeting techniques and discrete image retargeting techniques. Traditional image retargeting techniques: Content-aware image SCL techniques define local scaling factor as per the energy map and the target display size. These techniques optimally define local scaling factor over non-overlapping rectangular grid or triangular mesh based on the energy map in terms of saliency, object location, higher-order image structures such as line and texture [2-6]. Content-aware image CR techniques aim to find the best window of the target display size, which preserves the context to the best maximum level. The window is found either by using image attributes or learning. Attributes used for finding the best cropping window are saliency, eye fixation, and aesthetic features in [7-9]. Learning from cropped images by expert photographers is used to enhance the overall appearance and to remove unwanted content by Yan et al. [10]. Zhang et al. have used image semantics and preserved relative distance between them using manifold embedding and Bayesian network theory [11]. Continuous image retargeting techniques: Continuous image retargeting techniques find warping (WARP) function to map each pixel on the source image grid to the target image grid with the help of an energy map. Image grid is formulated as a mesh optimisation problem while preserving saliency and minimising distortion in [12, 13]. While minimising the distortion, various works focus on to preserve structural components present in the image. The shape of the salient object and line structure are preserved using a quad-tree structure [14]; parallelism, collinearity, and orientation [15]; similarity transformation [16]; as-similar-as possible and adaptive Laplacian regularisation energy function [17]. As the human visual system is highly perceivable to irregularity and deformation in the textural region, textural uniformity is preserved using non-linear least-square optimisation [18]; summarisation of repetitive translational textural patterns [19]. The non-textural region is retargeted using the WARP method. Lau et al. have represented an orientation-preserving bijective warping map by its Beltrami representation to preserve the geometry of salient content [20]. Image retargeting for browsing application is guided by saliency, object completeness map and cyclic seam carving in [21]. Discrete image retargeting techniques: Discrete image retargeting techniques find least energy pixels which can be removed or inserted without making much change in the total energy of an image. Firstly, Avidan et al. have defined an image operator called ‘seam carving’ which when applied on an image finds an optimal 8-connected pixel path of least energy in either vertical or horizontal direction using dynamic programming [1]. Later, Rubinstein et al. have improvised seam carving operator by minimising energy inserted into the image along with energy to be removed using graph cut formulation [22]. Han et al. have found multiple 3D surfaces of least energy by optimising voxel-based graph theory [23]. Pritch et al. [24], Ding et al. [25] and Cho et al. [26] have derived shift map defining shifting of each pixel in an output image with reference to the original image using graph labelling formulation, importance filtering, self-supervised learning, respectively. Discrete depth information and object ordering in terms of depth layers are used to overlap the objects having a different depth and reduce object distortion for image retargeting purpose in [27-29]. Composition, position, and rearrangement of objects as a whole are carried out using image aesthetics [30], human gaze behaviour [31], and similar object features [32]. Kim et al. have defined a texture aware energy map using higher-order statistics to guide image retargeting process [33]. Pixel fusion using seam searching and inter coherence filtering is utilised for image retargeting application in [34]. Image retargeting for folded and application specific target display device is performed by modifying seam carving problem in [35, 36]. Different seam carving techniques are studied and compared in [37]. Combining continuous and discrete image retargeting: Performance of the image retargeting techniques is improved by combining different operators such as SCL, CR, WARP and/or seam carving. These techniques optimise the sequence and the number of times each operator is applied in order to get a retargeted image of target display size [38-44]. The different image retargeting techniques are summarised and compared in [45]. There have been some works speeding up the seam carving process by removing or inserting multiple seams in a single iteration. Han et al. have found multiple seams in a single iteration using a global optimisation process, smoothness, and shape priors [46]. The distance between multiple seams makes them uniformly distributed all over the image which is contradicting the tendency of consecutive seams to pass through the same region of the lowest energy. Domingues et al. have changed the aspect ratio of an image by carving stream of multiple pixel width using image inpainting [47]. After finding least energy seam of a single pixel width, they inserted a stream of multiple pixel widths around the least energy seam using the inpainting process. Inserting pixels using the inpainting process may not preserve the energy of an image. Mishiba et al. have resized images by multiple pixel widths considering connected blocks as a seam [48]. Creation of new edges and fixed block size are the limitations of this work. In this study, we introduce a technique for accelerating the naive seam carving process by searching for a seam of width more than a single pixel. The assumption behind considering seam of multiple pixel width is that the consecutive seams tend to pass from the same region until there is no possibility to expand further. The considered assumption is provided with visual results of seam locations overlaid on the original images later in the paper. A variation of the proposed algorithm is defined to make the width of the batch seam variable based on the constraint of energy preservation. 3 Accelerated seam carving The input to the proposed approach is an image I of size , desired size of the retargeted image , the width of the batch seam K(see Fig. 1: Algorithm 1) or threshold value (see Fig. 2: Algorithm 2). The output of the proposed approach is a retargetedimage of size . The primary goal of the proposed approach isto accelerate the seam carving process such that energy of the retargeted image ispreserved and quality is as perfect as that of the naive seam carving approach. Fig. 1Open in figure viewerPowerPoint Algorithm 1: Batch seam retargeting using constant width K forvertical seams Fig. 2Open in figure viewerPowerPoint Algorithm 2: Batch seam retargeting using adaptive width K forvertical seams For the retargeting application, the motivation of considering multiple pixel wide seams is the tendency of seams to pass through the low-energy region of the same area irrespective of the width of batch seam up to some extent. The same is observed from Fig. 3. Figs. 3a–f show original image, seam locations imparted on original image for batch seam width of K = 2, 3, 4, 5 and 6, respectively. Note that for different values of K, the removal or insertion of a total number of pixels shown in red lead to the retargeted image of the same size. Observing the seam pixel locations, it is evident that seams are passing through the same region irrespective of the width of batch seam K. Fig. 3Open in figure viewerPowerPoint Seam locations on original image for different batch seam width (a) Original image, seams for batch seam width, (b) , (c) , (d) , (e) , (f) 3.1 Energy map The energy map for the proposed approach consists of the saliency map and the absolute first-order gradient values in each direction. Saliency detection algorithm proposed by Margolin et al. finds a salient region of an image by finding distinct patches in terms of colour and texture [49]. Patches from a salient region show distant statistics from the patch collectively represented by the average of all the patches. A patch is distinct if it is distant from the average of all patches in a principal component direction. Distinctness of every patch is a factor integration of distinctness in colour space and pattern space. Let S be the saliency map of an image I of size . The energy map E of an image is defined by combining the saliency map with the absolute gradient values as (1)Here, and . The absolute gradient value is added to estimate local variations even in the regions where no saliency is detected and to avoid zero value conflict in the seam detection process. This study concentrates more on the process of finding the least energy batch seam. The energy map calculation is kept simpler. However, one can go for energy map calculation focusing on face detection, depth ordering, object semantics, 3D saliency or other higher level cues as proposed in earlier works on image retargeting [1, 27, 33, 50-53]. 3.2 Batch seam detection and image retargeting In order to alter the size of an image I by K columns or rows, the batch seam of K-pixel width is detected and removed or inserted in a single iteration. The process is repeated until the target display size is achieved. Here, K is the number of lowest energy consecutive pixels (also called array pixels) in a single row (column) for the vertical (horizontal) batch seam detection process. Vertical batch seam of K-pixel width for the image I is defined as (2) Here, is a Cartesian product generating a product set of ordered pairs from multiple sets. is a row index i of a pixel. indicates the column index of a pixel in a batch seam having starting column location j. is a vertical batch seam of 8-connected pixels containing K consecutive pixels in each row of the image. Similarly, horizontal batch seam of K-pixel width for the image I is defined as (3) Here, indicates the row index of a pixel in a batch seam having starting row location i. is a column index j of a pixel. is a horizontal batch seam of 8-connected pixels containing K consecutive pixels in each column of the image. Considering 8-connected pixel nature between seam pixels of consecutive rows, all possible batch seams for pixel are shown in Fig. 4. After removal of array pixels during the batch seam carving process, pixels at a spatial distance of K-pixel become new neighbours as shown in Fig. 4. These pixels may pertain to distant features creating new ‘edges’ of significant magnitude. The new ‘edges’ created after vertical batch seam removal are shown with pink lines. These newly constructed edges insert energy, named as forward energy, to an image. This forward energy needs to be minimised in order to preserve the total energy of an image. The cost of the forward energy is measured by taking an absolute forward difference between the pixels, which are going to become new neighbours after the batch seam removal or insertion. The absolute forward difference for all the three cases is as shown in (4) (4)where and . , , and are three possible forward energy for vertical batch seam with reference to the pixel location . The cumulative energy for the vertical batch seam is calculated by traversing the image starting from the first row to the last row for all possible minimal energy batch seams of K-pixel width. We add a minimum of the sum of cumulative energy of batch seam from the row and the corresponding forward energy to the energy of a pixel to find the cumulative energy at pixel location . The cumulative energy for each pixel location is defined as (5) Fig. 4Open in figure viewerPowerPoint Illustration of three possible costs of forward energy for the vertical batch seam. After removal of the batch seam, new edges created are shown with pink colour Here, , and . (For image border pixels at and , only two out of the three batch seams are possible.) The minimum value of at the last row indicates the minimal 8-connected vertical batch seam. Backtracking in the upper direction from this minimum value of gives the vertical batch seam of minimum energy as (6)We find the horizontal batch seam in a similar manner to that of the vertical batch seam but traversing from left to right of the image. Forward energy for three possibilities of the horizontal batch seam in terms of absolute forward difference is given as (7)where and . , , and are three possible forward energy for horizontal batch seam with reference to the pixel location . The cumulative energy for the horizontal batch seam is calculated by traversing the image starting from the first column to the last column for all possible minimal energy batch seams of K-pixel width. We add a minimum of the summation of cumulative energy of batch seam from the column and the corresponding forward energy to the energy of a pixel to get the cumulative energy at pixel location . The cumulative energy for each pixel location is defined as (8)Here, , and . Backtracking in the left direction from this minimum value of gives the horizontal batch seam of minimum energy as (9)After finding the batch seam of minimum energy, removal or insertion of it changes the size of an image by K pixels with a minimal change in the total energy of an image. A vertical batch seam retargeting algorithm using constant width K is defined in Algorithm 1. After every loop in Algorithm 1, the original image is retargeted (increased or decreased in size along horizontal dimension) by K pixels. In every iteration, we check the absolute difference between the available retargeted image size and required retargeted image size . If the absolute difference is less than K, the batch seam width K is updated to the absolute difference to get the retargeted image of desired size . To alter the height of an image, horizontal batch seam retargeting is performed using Algorithm 1 by replacing (5) and (6) by (8) and (9), respectively. 3.3 Adaptability of batch seam width The critical factor for the batch seam carving process is the batch seam width K. K indicates the length of array pixels to be removed in a single iteration. Also, indicates the seam carving process similar to defined in [1] with forward energy taken into consideration. Fig. 5 shows the curves of average preserved energy values across the number of images for different values of batch seam width K. Note that the preserved energy of an image is the average value of the energy map of an image. Since every original image has different initial energy and we are interested in observing the rate of increase in it with batch seam removal, the preserved energy value of the original image is subtracted from the preserved energy value of a retargeted image after each batch seam removal. Two facts are verified from Fig. 5. The average value of the preserved energy increases with the removal of least energy batch seams. Fig. 5Open in figure viewerPowerPoint Average preserved energy curves versus number of vertical seam removal for width K = 1, 2, 3, 4, 5, 6, 7, 8 and variable K with different threshold values The preservation of energy effectively goes on decreasing with the increase in the value of K. During the process of batch seam carving, inserted energy due to ‘new edges’ and removed energy due to seam removal is sometimes more than the average energy per pixel to be removed. It results in a decrease of preserved energy, in contrast, to increase in a preserved energy of retargeted image. To prevent the decrease in a preserved energy, the value of K is decreased adaptively based on the change in preserved energy after every iteration. Algorithm 2 defines the batch seam retargeting using adaptive width K of the batch seam. In every loop of Algorithm 2, we check the change in the preserved energy of an image from that of the previous iteration. If the decrease in the preserved energy is less than the threshold , batch seam width K is reduced by 1. Furthermore, image retargeting is carried out with the updated value of K. Also, we check the absolute difference between the available retargeted image size and required retargeted image size . If the absolute difference is less than the updated value of K, the batch seam width K is updated to the absolute difference to get the retargeted image of desired size . The maximum value of K is chosen by observing the change in preserved energy very close to that of single pixel wide seam carving. 4 Results and discussion We experimented image retargeting with batch seam detection on two datasets: PASCAL-S dataset [54] and RetargetMe dataset [45]. The PASCAL-S dataset consists of total 850 images of larger as well as smaller objects. The dataset provides both fixations and salient object annotations. The RetargetMe dataset consists of total 80 images collected from various image retargeting papers. These images contain attributes challenging to the image retargeting operators such as people, faces, lines, clear edges, evident foreground objects, texture elements, specific geometric structures, and symmetric objects. We have empirically chosen value to be equal to 0.7 signifying the contribution of saliency and absolute gradient value for the calculation of energy map E. The image retargeting process incurs geometric distortion and content loss in a retargeted image, many quality assessment metrics (QAM) using different handcrafted and deep learned features are proposed to evaluate the retargeting process [55, 56]. Quantitative analysis of the proposed batch seam retargeting process is performed using three different QAM proposed in [57-59]. In [57], Zhang et al. have defined geometric change between the original image and the retargeted image using the backward registration process and Markov random field. Estimated geometric change is used to evaluate local changes between the original image and the retargeted image. Under the guidance of geometric change estimation, quality of the retargeted image is defined based on local changes using aspect ratio similarity metric . In [58], Zhang et al. have included mid-level feature: edge group similarity , and high-level feature: face block similarity . The mid-level edge group similarity feature captures shape or structural deformation. The high-level face block similarity feature captures the face deformation if available in the image. The objective quality prediction metric used to evaluate the proposed algorithm captures the five key factors consistent with the human visual system: preservation of salient regions, preservation of global structure, preservation of image aesthetics, preservation of symmetry and capture of newly induced artefacts [59]. The finds the distance between the original image and the retargeted image using a weighted sum of five factors mentioned above. Note that . , , and metrics are similarity measures indicating larger values for better retargeted images. metric is a distance measure indicating smaller values for better retargeted images. To match all other measures, we compute and report to analyse the proposed batch seam retargeting process in this study. 4.1 Analysis using PASCAL-S dataset We have randomly selected 100 images from the PASCAL-S dataset to evaluate the performance of the proposed technique [54]. Fig. 6 shows the average values across the number of images against the number of vertical seams removed. One can observe from Fig. 6 that the quality of the retargeted image decreases with an increase in the values of K. The width of images from the PASCAL-S dataset is generally 500 pixels. Considering the change in aspect ratio of the retargeted image as 0.75, 125 vertical seams need to be removed. The maximum value of K for an adaptive algorithm of batch seam retargeting is chosen for which the preserved energy deviation after 125 seams removal is within 20% interval (shown with red dashed lines in Fig. 5) of the preserved energy corresponding to . Observing Fig. 5, the slope of the curves for batch seam width up to is decreasing in that interval concluding for adaptive batch seam retargeting (observe zoomed in rectangle of Fig. 5). Also, the average value after 125 vertical seams removal degrades only by 3% while switching from to maintaining the quantitative metric quality of images (refer to Fig. 6). Hence, the maximum initial value of K is chosen to be 6 in our experiments. The average preserved energy for adaptive values of K with different thresholds , and are shown in Fig. 5 which are observed to be improved than that of . By changing the batch seam size K adaptively under the constraint of increase in the preserved energy improves the quality of the retargeted image as shown by dashed curves in Fig. 6. One can observe from Figs. 5 and 6 that starting from and reducing the width of a batch seam in an adaptive manner during the retargeting process is able to preserve the energy and improve the quality of the retargeted image. Fig. 6Open in figure viewerPowerPoint Average aspect ratio similarity curves versus number of vertical seam removal for width K = 1, 2, 3, 4, 5, 6 and variable with different threshold values The average time (in seconds) required for the execution of the proposed batch seam retargeting using different values of K is shown in Fig. 7. It is observed from Fig. 7 that the time required for the removal of vertical seams to match the target display size shows drastic reduction with an increase in the value of K. The reduction in time compromises very little on quality as is evident from Fig. 6. As compared to , the time required for retargeting an image has reduced by 40% using the width of batch seam maintaining the quality nearly similar to that of . Also, as shown by dashed curves in Fig. 7, the time required to retarget an image using adaptive width K is