Video retargeting through spatio‐temporal seam carving using Kalman filter

IET Image ProcessingVolume 13, Issue 11 p. 1862-1871 Research ArticleFree Access Video retargeting through spatio-temporal seam carving using Kalman filter Harpreet Kaur, Harpreet Kaur Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, IndiaSearch for more papers by this authorSwarnjeet Kour, Swarnjeet Kour Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, IndiaSearch for more papers by this authorDebashis Sen, Corresponding Author Debashis Sen dsen@ece.iitkgp.ac.in Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology Kharagpur, Kharagpur, IndiaSearch for more papers by this author Harpreet Kaur, Harpreet Kaur Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, IndiaSearch for more papers by this authorSwarnjeet Kour, Swarnjeet Kour Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, IndiaSearch for more papers by this authorDebashis Sen, Corresponding Author Debashis Sen dsen@ece.iitkgp.ac.in Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology Kharagpur, Kharagpur, IndiaSearch for more papers by this author First published: 01 August 2019 https://doi.org/10.1049/iet-ipr.2019.0236Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract A Kalman filter-based spatio-temporal seam determination scheme is proposed in this study for video retargeting through seam carving. In probably a first, the authors' retargeting approach is designed to be hardware friendly, and to achieve spatial and temporal coherence through an optimal solution with explicit mechanisms to reduce jitter and structural distortion. In a video frame, while spatially coherent seam is determined by the popular approach of seam energy minimisation, temporally coherent seam is determined using Kalman prediction and updation processes. Further, these spatial and temporal seams are combined judiciously to obtain the spatio-temporal seam to be removed/repeated for decreasing/increasing the frame size. The authors show that the proposed Kalman filter-based approach has less theoretical complexity compared to the existing. Extensive experimental results show that the proposed approach consistently outperforms the state-of-the-art, both qualitatively and quantitatively in performance, and in computational time. 1 Introduction Video display units are key components in any modern system targeted toward video consumption. They can be of a wide variety, like monitors and hand-held devices, which have different resolutions and sizes [1]. Hence, digital videos have to change in size and aspect ratio for adapting to different screens [2, 3]. This gives rise to an important video processing operation called video retargeting. Video retargeting is resizing the video without distortion while preserving its contents to fit it on a screen [4, 5]. Various simple video resizing methods have been considered in the literature. One of those is scaling, a linear alteration that enlarges or shrinks videos by some factor in all directions [4]. Cropping is another, which removes pixels from the periphery of the video frames. Both scaling and cropping does not consider the knowledge of video content locations during resizing [4]. Methods that employ warping create nonlinear distortions to obtain the resized videos [6-8]. These methods are content-aware, where only unimportant areas in videos are altered (cropped, scaled, or surface modified) for video resizing. Most warping based approaches for videos are derived from their image counterparts by introducing constraints to maintain smoothness across the adjacent frames but seldom perform satisfactorily, especially in regions with significant motion. Seam carving is a content-aware method for resizing images [4]. It removes the 8-connected path of pixels running from left to right for horizontal seams and from top to bottom for the vertical seam. It considers an energy function to form low energy seams that are removed in the image using dynamic programming. However, image seam carving method, if directly employed on videos, can create motion-related distortion [9]. Both spatial and temporal coherences of seams need to be considered in video seam carving while treating frames in space-time as 3D cubes [10]. Memory requirements and computation speed also need to keep in mind due to the huge amount of data involved. A few video retargeting approaches are briefed in Section 2. Although existing seam carving based approaches use innovative solutions to maintain seam coherency through video frames, including heuristics [11], none of them is a theoretically (closed-form) optimal solution. Moreover, sufficient thought has not been given on whether the proposals are readily implementable in hardware (like [12]). Kalman filter uses a series of values of a quantity that changes over time/space to predict the value of the quantity at the next instance [13]. It provides an optimal linear solution, where the prediction allows for statistical noise and errors in the series of values [13]. Hence, the Kalman filter has been successfully used in many applications, especially for estimation [14] and tracking [15]. As the Kalman filter optimally computes the value at the current instance from previous instances, the computed value is inherently coherent with the previous. We propose to exploit this property of the Kalman filter and use it to compute a temporally coherent seam in a frame from the seams in the previous frames. Kalman filter not only provides a theoretically optimal solution, but also can be readily implemented in hardware [16, 17]. In this paper, we present a video seam carving algorithm that estimates a seam in a frame based on spatial seams in the current frame and seams computed in temporally earlier frames, such that the spatio-temporal seam estimated is both spatially and temporally coherent. Kalman filter performs temporal estimation of a seam, and then updates it considering a spatial seam having the lowest energy in its local neighbourhood. In a frame, this seam is then adaptively combined with spatial (spatially coherent) seam computed for that frame as in [4]. Video retargeting is performed using the proposed spatio-temporal seam computation, and the performance is compared to the mentioned state-of-the-art video retargeting algorithms. Apart from qualitative analysis, the effectiveness of the proposed algorithm is demonstrated in terms of temporal coherence, saliency preservation, and time complexity. We also highlight the explicit mechanisms at work that ensures a reduction in jitter and structural distortion. The paper is organised as follows. While Section 2 contains related work on video retargeting, Section 3 presents components of video retargeting via seam carving. Section 4 describes the proposed spatio-temporal seam computation using a Kalman filter, and performance (including theoretical time complexity) comparisons of our method with the state-of-art are given in Section 5. Section 6 concludes the paper. 2 Related work Various kinds of video retargeting approaches have been proposed in the last decade. We discuss a few of them here in chronological order. Liu and Gleicher [18] proposed a warping based image retargeting approach and justified its suitability for videos as well. The approach finds important regions and performs retargeting by preserving not only those regions, but also their surrounding context. Liu and Gleicher [19] also proposed a method which uses the combination of cropping and scaling for video retargeting. Dynamic video shots are represented by virtual pans and cuts. To minimise loss of content and composition, this method optimises cropping and scaling by considering important information in frames, and face and motion detection. The approach treats shots independently, which could introduce distortions in frames at shot boundaries while retargeting videos with multiple shots. Wolf et al. [20] presented a warping based technique, which works in two steps. It first finds a saliency map of video frames and then removes pixels from non-salient portions of frames. Saliency is computed using face detectors and motion detectors as well. However, experiments show that abrupt removal through warping even from the non-salient homogeneous regions sometimes introduces distortion. Tao et al. [21] proposed a method where salient areas of the frame are extracted by identifying foreground objects that are in motion, which are considered important video contents. An active window is taken to cover these areas and prevent them from distorting objects. However, experiments show distortion being introduced sometimes in the background. Zhang et al. [22] use a shrinkability map for retargeting the video. The shrinkability map for each pixel in frames of video is defined by a random walk. Scaling is then used considering the shrinkability map to retarget the video. Rubinstein et al. [9] proposed a video retargeting method based on seam carving, which uses a graph cut approach to find the seam. The method computes seam by finding minimal monotonic and connected cut in a graph. Also, this method also considers the energy inserted into an image after the removal of the seam. This method is computationally intensive as large amounts of information about video frames are stored and processed. Sometimes, while maintaining temporal continuity of the seams, distortion is introduced in the contents of the retargeted video [23]. Hence Wang et al. [23] came up with a solution to make the temporal continuity aware of the motion of video entities. Rubinstein et al. [24] proposed a method that combines multiple operators like cropping, scaling, warping, and seam carving. This method does the combination in an optimised way to maintain spatial and temporal coherences while retargeting the video. Kim et al. [25] consider performing a constrained optimisation in the Fourier domain to minimise distortion maintaining the required size of the output frame. The retargeting is achieved by dividing video frames into strips with similar textures and scaling each strip adaptively based on its importance. Wang et al. [26] perform cropping to remove temporally-recurring contents and motion preserving warping in the homogenous spatial areas through a variational optimisation to get the retargeted video. In this process, the distortion is properly distributed in both spatial and temporal dimensions resulting in better preservation of salient object shapes. Lu et al. [27] propose a cropping-and-scaling retargeting approach by considering the non-linear fusion of spatial and temporal saliencies, which are computed considering human perceptual characteristics. Hu and Rajan [28] propose a new measure called the spatial–temporal naturality for video retargeting in a hierarchical framework. Use of this measure avoids propagation of motion analysis errors during retargeting making their approach suitable for under complex camera and object motion. Grundmann et al. [10] proposed a seam carving based method which processes a video, frame by frame, producing discontinuous seam as opposed to geometrically smooth and spatially continuous seam. It takes into account the difference of appearance in the spatially retargeted and most temporally coherent frame to find the seam. The technique is found to be computationally expensive in Section 5. It is also observed there that sometimes removal of pixels of discontinuous seams creates unwanted distortion, especially when objects are moving very fast. Chao et al. [29] proposed a coarse to fine temporal optimisation for video retargeting, which is based on seam carving and uses dynamic programming for retargeting. This method uses energy functions, motion weight prediction, and pixel-based optimisations to extract temporal information. The motion weight prediction reduces the search area of dynamic programming by predicting the coarse location of the seam in the current location. The pixel-based optimisation then maintain temporal coherency by exploiting the concept of pixel-based optical flow in the reduced search range. Wang et al. [30] use a compressibility assessment scheme for structures in a video frame by combining the entropies of gradient magnitude and orientation distributions. This compressibility assessment score is used as content saliency to perform the video retargeting. In [31], Yen et al. circumvent the difficulties in video retargeting due to camera and object motions by considering panoramic mosaic generation from video shots. Scaling maps for video shots are derived from the panoramic mosaic which inherently reduces distortion while maintaining temporal coherency. Lin et al. [32] present a content-aware warping approach for video retargeting which performs an object-based significance estimation to reduce distortion employing space-time salient object determination. The approach is found to preserve both spatial shapes and temporal motions consistently. Yan et al. [11] propose a matching area-based method for video retargeting using seam carving. It is a frame-wise seam carving method. It uses the concept of modifying energy map used for carving on the basis of the matching area. The basic idea is to divide the pixels in the current frame into reward/punish regions according to their similarities to the pixels on the previous seam. Energy map will be modified on the basis of the reward and punish regions. Then the usual seam carving algorithm is applied to find the seam in the current frame. We find this approach consistently outperforms most others in the literature. For video retargeting, Li et al. [33] segment videos in spatio-temporal grids which are temporally coherent in nature and then perform content-preserving retargeting of the grids. In addition, temporal redundancies in a grid are exploited to avoid those with inconsistent content from coherent deforming. The authors claim to achieve the processing through small-scale quadratic programming with low computation and memory costs. In Wang et al. [34], saliency curves representing the shapes of salient regions are extracted and preserved, along with temporal consistency, to perform video retargeting. Measures of deformation of saliency curves are devised and incorporated into the energy map computation for seam carving. Such sophisticated modification of energy map, is generic in nature, and can be used in various approaches using seam carving. Yan et al. [35] minimise a function for video retargeting that includes both spatial and temporal constraints. The temporal coherency in the retargeted video is validated through frame differencing. In the last few years, the focus of research on retargeting has shifted towards very specific applications, such as for semantically-rich photos [36], panoramic videos [37], and stereoscopic videos [38, 39]. Considerations for implementation on operating systems of mobile devices have also been explored [40]. Novel concepts such as seam warping, a fusion of warping and seam carving, are being attempted [41]. The approaches above including the latest ones that perform well do not focus on hardware friendliness. Hence, the latest video retargeting approaches readily implementable in hardware are only the baseline ones [12]. FPGA implementations of seam carving [42] and seam energy computation [12] have been performed. So, a sophisticated video retargeting approach could be readily hardware implementable, if its seam computation in the temporal direction is made hardware friendly. Further, most existing seam carving based video retargeting approaches do not explicitly deal with the second core issue of jitter, with the first being structural distortion. Therefore, a hardware-friendly approach that explicitly contains mechanisms to reduce both jitter and structural distortion may immensely benefit the field, which we provide in this contribution (see Section 4.4). 3 Video retargeting via video seam carving Our proposal in this paper is regarding an improved video seam carving approach for retargeting. The block diagram of our video retargeting system is shown in Fig. 1, which is similar to most spatio-temporal seam carving based systems in the literature, like that of [10, 11, 34]. The video and target size are fed as inputs to the system. Fig. 1Open in figure viewerPowerPoint Block diagram of a seam carving based video retargeting system. The blocks marked * signify our contributions are in computing temporal and spatio-temporal seams employing Kalman filter The first step in the system is to compute energy maps of video frames, which quantify the importance of each pixel in a frame. Measures such as gradient magnitude [10, 11], saliency [43], salient region shapes [34] or local entropy [44] can be used the energy or importance map computation. As our main focus is on computing spatio-temporal seams given any kind of energy maps, irrespective of how they are computed, we consider the use the gradient magnitude as it is the most popular one. Thus, the energy maps of video frames are computed as (1)where I is the frame, (i, j) is the pixel location, and x and y are the directions. State-of-the-art approaches of [10, 11] use a gradient, while [34] uses region shapes based on saliency. The next step in the system is the computation of spatio-temporal seam. It is an appropriate combination of spatial and temporal seams, where the spatial seam is computed as in [1] using dynamic programming, solving: (2)where e(i, j) is the energy at a frame's pixel (i, j) calculated from (1). The frame is traversed from top to bottom (for the vertical seam) to calculate cumulative minimum energy M considering all possible connected seams for each pixel in the top row. At the bottom row, the pixel associated with minimum M value is considered and the optimal vertical seam is obtained by backtracking the connected minimum energy locations upwards from this location. This optimal path acts as a spatial seam given by (3)where denotes the location of the top row pixel whose vertical seam S computed using (2) has the minimum cost among vertical seams corresponding to all top row pixels. Note that, a seam is a set of pixels. We have described vertical seam computation only. Horizontal seams can be computed similarly through left to right traversals in frames. Note that this spatial seam computation is also employed in [10, 11, 34]. To get spatio-temporal seams, the next step is the computation of temporal seam, followed by a judicious combination of spatial and temporal seams. This is where our approach differs from the existing. Temporal seam computed by our proposed algorithm is described in the next section (Section 4). The proposed combination of both seams to find the spatio-temporal seam having a proper balance between spatial and temporal coherences is also described in the next section. The temporal and spatio-temporal seam computations in [10, 11, 34] have already been briefed in Section 2. The final step in the system is to remove or insert (repeat) the spatio-temporal seams in frames to get the retargeted video. Seam carving does have issues such as distortion due to its discrete nature and there have been proposals in the literature to overcome these issues [45]. In our proposal, we consider seam carving and focus on sophisticated spatio-temporal seam computation. A preliminary version of the proposed approach has been presented at a conference [46]. 4 Kalman filter for spatio-temporal seam estimation Kalman filter uses a series of measurements observed over time and produces estimates of unknown variables [47]. Kalman filter has been extensively used in restoration and denoising of videos [48] and video tracking [49]. In our approach, temporal estimation of the seam (which we refer as a temporal seam) is performed using Kalman filter. It consists of two phases: prediction and updation. The prediction phase uses a previous frame seam to estimate a current frame seam. In the updation phase, the predicted seam of the current frame is combined with an observation in the current frame to refine the seam estimate. The observation is a spatial seam having the lowest energy in a pre-defined neighbourhood. A brief overview of our proposed method is given in Algorithm 1 (see Fig. 2). Note that, for the sake of brevity, we present only the vertical seam computation here. Fig. 2Open in figure viewerPowerPoint Algorithm 1: Proposed method for selecting a vertical seam 4.1 Seam prediction For estimation of a current frame seam Xt, a two-component state having position and velocity is considered as (4)where is the ith pixel among a set of pixels/positions representing a seam of the frame at time t. gives the velocity associated with the ith pixel. R is the number of rows (as the vertical seam is considered). As spatio-temporal seam is changing its position from frame to frame, the previous frame's spatio-temporal seam position can be used to find the current frame's temporal seam position by using the equations of motion for position and velocity which are given as (5) (6)The prime symbol indicates parameters related to spatio-temporal seam and acc stands for acceleration. The symbol τ stands for the time interval between t and t–1. These equations of motion can be rewritten as (7)yielding the expression (8)where gives the position and velocity of the corresponding spatio-temporal seam in the previous frame. , are various control inputs given by (9)Let be the frames of a video and be the sets of pixels in a vertical spatio-temporal seam in all the frames, respectively. Let a set of pixels Pj be represented as , where R is the number of rows. Change in position of seam pixels from consecutive frames gives velocity. Let be the sets of velocity values of a spatio-temporal seam in the frames , respectively, where . We get by (10)Similarly, change in seam velocity from consecutive frames gives acceleration. Let be the sets of acceleration values of a spatio-temporal seam in the frames , respectively, where . We get by (11), and in (7) are obtained from spatio-temporal seams, with the velocity and acceleration quantities calculated using (10) and (11), respectively. Note that, as evident from (10) and (11), velocity is available when and acceleration is available when . Therefore, spatio-temporal seams, for which (7) must be used, are obtained starting from the frame , with both velocity and acceleration as zero at the frame and with acceleration as zero at the frame . Only spatial seam computed through (3) is employed at the frame . Now, the initial prediction of the seam can be obtained from the prediction performed in (8). We represent this seam prediction by , which is obtained by (12)where . The expression in (12), as evident from (4), will yield . In Kalman filtering, prediction such as in (8) is updated considering an observation. Therefore, to update the prediction in (8), and also that in (12), an observation regarding the seam position is required. 4.2 Seam updation It is obvious that the seam predicted in (12) must correspond to a spatially feasible in the current frame. To consider this aspect, we take the observation for updating the prediction in (12) as a spatially computed seam in the local vicinity of the predicted . The spatial seam considered as the observation is the seam which is optimal in accordance to (3) locally within a region of 50 pixels on either side (along rows) of the average column location of the vertical seam We represent this locally optimal spatial seam as . For the updation, we need to compute Kalman gain, which considers the variances involved in prediction as well as the variance involved in the observation. The following covariance matrix gives the variances involved in the prediction of (8): (13)Initially, is taken as (14)In (13), refers to the noise in prediction, which is given in terms of variances in position and velocity as (15)In Kalman filtering, the noise is often obtained as the covariance of the random quantity from (8), assuming it to be zero-mean normally distributed. Assuming the random quantity to be standardised, we get as (16)where the quantity τ is the time between frames, which in terms of frame numbers is essentially 1. The expression in (16) indicates that while quantifies the effect of time elapsed after the previous prediction on seam position prediction, τ quantifies the same for seam velocity prediction. Now, the Kalman gain is given by (17)where refers to observation 'measurement' noise. We consider it as the variance in considering all i values (18)Note the value of is unavailable as we do not actually perform any measurement of observation. The expression in (18) is obtained from the reasonable assumption that higher the variation in the spatial seam , higher might be the noise in the measurement of as larger spatial ranges would be involved in measurement. Now, the prediction in (8) is updated using the Kalman gain in (17) as follows: (19)If the predicted seam is almost the same as the observed one, then the prediction in (8) is accurate and need not be changed. But if the predicted seam is substantially different from the observed one, then the discrepancy in the prediction of (8) is probably substantial and is corrected employing the Kalman gain. The covariance matrix of (13) is also updated for use in the next frame as follows: (20)where is an identity matrix. From (19) we get the updated prediction of the seam as (21), which is a set of pixels, is the temporal estimation of the current frame seam Xt. The double prime symbol indicates parameters related to temporal seam. The temporal seam computed using the Kalman filter allows maintenance of temporal coherency between seams in consecutive frames. 4.3 Combining spatial and temporal seams To get the spatio-temporal estimate of the seam , we shall now consider both the temporal seams obtained from (21) and the optimal spatial seam in the entire current frame obtained through (3). The region between the optimal spatial seam and the temporal seam is considered. Then, all possible connected seams computed through (2) for each pixel in the top row within the region are considered as candidates for the spatio-temporal estimate. An example of the region between the optimal spatial seam and the temporal seam is illustrated in Fig. 3. Fig. 3Open in figure viewerPowerPoint Optimal spatial (white), temporal (yellow) and some spatio-temporal candidate (red) seams Let be set of candidate spatio-temporal seams in frames in the frames , respectively, where and considering as the number of candidate seams in the jth frame and as a pixel in the lth row of the jth frame that lies in the kth candidate seam. Two parameters are considered for getting the spatio-temporal seam. One is the distance of a candidate seam from the temporal seam, referred as distanceDiffjk for the jth frame and the kth candidate seam. The other is the energy difference of candidate seam with the optimal spatial seam, referred as energyDiffjk for the jth frame and the kth candidate seam. So we have (22) (23)where e[p] denotes the energy at pixel p. These parameters are then combined as (24)The overbars in (23) denote normalisation of the parameters concerning their maximum values. The parameter distanceDiffjk will have the maximum value when the candidate seam is the optimal spatial seam and the maximum energyDiffjk will be R times the maximum energy value in the frame. The spatio-temporal seam (in the jth frame or at time t) is chosen as one of the candidate seams having the lowest H value as (25)The spatio-temporal seam estimate of Xt obtained in (25) is used as the in (5) for the seam prediction and updation in the next frame. and for the same purpose are also obtained using as in (10) and (11). Our approach of estimating a spatio-temporal seam from the temporal and spatial seams is an optimal one in simultaneously maintaining temporal coherency and reducing the energy of the spatio-temporal seam. We get a balance by optimally compromising on the maximisation of temporal coherency (as done by the Kalman filter) and the minimisation of seam energy (as done in (3)). 4.4 Advantages of the proposed approach 4.4.1 Low jitter and structural distortion A retargeted video can contain jitter and structural distortion. Jitter occurs when seams to be removed/replicated in consecutive frames are spatially far apart. Structural distortion occurs when a seam in a frame runs through region/object boundaries. For a frame, the seam computed in (12) after the prediction process of Kalman filter is entirely based on projects from the seam positions in the previous frames, and hence, seams in consecutive frames will not be positioned spatially far apart (temporal coherency) resulting in jitter-free vide