# Seam-Carved Image Tampering Detection Based on the Cooccurrence of Adjacent LBPs

Authors: Dengyong Zhang; Xiao Chen; Feng Li; Arun Kumar Sangaiah; Xiangling Ding
Journal: Security and Communication Networks
 (2020)

---

## Abstract

Seam carving has been widely used in image resizing due to its superior performance in avoiding image distortion and deformation, which can maliciously be used on purpose, such as tampering contents of an image. As a result, seam-carving detection is becoming crucially important to recognize the image authenticity. However, existing methods do not perform well in the accuracy of seam-carving detection especially when the scaling ratio is low. In this paper, we propose an image forensic approach based on the cooccurrence of adjacent local binary patterns (LBPs), which employs LBP to better display texture information. Specifically, a total of 24 energy-based, seam-based, half-seam-based, and noise-based features in the LBP domain are applied to the seam-carving detection. Moreover, the cooccurrence features of adjacent LBPs are combined to highlight the local relationship between LBPs. Besides, SVM after training is adopted for feature classification to determine whether an image is seam-carved or not. Experimental results demonstrate the effectiveness in improving the detection accuracy with respect to different scaling ratios, especially under low scaling ratios.


---

## Body

## 1. Introduction

As image processing technologies, such as scene graphs prediction [1]and image tampering technology, have steadily developed, the content-aware image retargeting techniques have emerged and have attracted increasing attention. One of these, seam carving [2], is able to avoid image distortion and deformation when applied to an image, creating no obvious difference in visual effect. However, this method can also be used for malicious tampering purposes, for instance, removing specific objects in the image or modifying with the semantic content conveyed by the original image, which creates great obstacles for image forensics tasks. Therefore, seam-carving detection has become an important issue, and designing a method for detecting images that may have been subjected to seam carving is of vital importance.Over the past years, some approaches have been proposed to aim at the detection of seam carving. Sarkar et al. [3] proposed a forensic method based on the Markov feature, which exploits the 324-dimensional Markov feature for classification in the block-based Discrete Cosine Transform domain. However, it achieves a less-than-ideal detection accuracy with a small scaling ratio. As described in [4], Lu and Wu proposed a detection method based on forensic hash to research the problems of seam-carving estimation and tampering localization. However, a forensic hash must be built ahead of time, so it is active; furthermore, it can be detected so easily that counterfeiters can remove it in some ways. Wei et al. [5] proposed dividing the image into small squares of size 2×2 and looking for the patch that could possibly recover the small squares from seam carving. This method eliminates the patch transition probability among the three small connected squares. Yin et al. [6] first used local binary pattern (LBP) to preprocess the image data and then defined six new features based on the half-seam to reveal the energy change in half of an image; subsequently, these were combined with the existing 18 energy features, after which those features were finally classified using support vector machine (SVM). However, this method is still not ideal for image detection accuracy with low scaling ratios. Wattanachote and Shih [7] proposed a forensic approach based on Blocking Artifact Characteristics Matrix (BACM). In an original JPEG image, regular symmetrical data are presented in a BACM block matrix. Following seam carving, these symmetrical data are reconstructed so that their symmetry is destroyed. From this, 22 features are proposed and a high recognition rate is obtained. However, this method can be easily affected by the quality factor (QF). Ke et al. [8] put forward a forensic method based on the additional seam-carving operation. First, an additional seam-carving operation is performed on the testing image. The approach then extracts 11-dimensional features by calculating the similarity, energy relative error, and seam distance difference between an image and the version of the image undergoing seam carving; this facilitates identifying whether the image has been tampered using seam carving or not. It can be observed that this method has wide applicability. Ye and Shi [9] proposed a method that incorporates a local derivative pattern, Markov transition probability, and the subtractive pixel adjacency model. They also utilized recursive feature elimination based on the linear support vector machine to reduce the feature dimensionality, which greatly improves detection accuracy. Liu et al. [10] combined calibrated adjacent joint density with a rich model-based method originally used for steganalysis and exploited the feature selection algorithm to reduce the feature dimensionality of the combined feature set by using a smaller and more optimized feature set, enabling further improvements on the detection accuracy for the forensic task. Subsequently, as discussed in [11], Liu developed a hybrid large feature mining-based method. As there are many types of large features, ensemble learning is utilized to process high-dimensional features, effectively solving the problem of differentiating between seam-carved and untouched JPEG images. However, a lot of work is involved. Han et al. [12] further proposed a blind detection method based on the block artifact grid (BAG) mispairing characteristic, which firstly extracts BAG from a JPEG image and then constructs 10-dimensional features of the BAG chart. Clustering technology is thus used to extract these features and obtain the classification results. This method is not only able to detect whether the image has been seam-carved or not but also can go a step further and locate the object removed from an image by seam carving. Cieslak et al. [13] proposed a forensic approach based on the combination of convolutional neural networks (CNNs) and local binary pattern (LBP). This method first transforms images to the LBP domain and then inputs them to the CNN in order to determine whether or not these images have undergone a seam-carving operation. The experimental results reach a detection accuracy of more than 81%. Similarly, Ye et al. [14] proposed a CNN-based deep learning architecture. This approach makes use of the joint optimization of feature extraction and pattern classification for employing more effective features, thereby improving the classification speed and accuracy. Zhang et al. [15] proposed a detection approach based on uniform local binary patterns (ULBP). In this approach, ULBP is exploited to decrease the species of binary patterns without losing any information, thereby reducing the dimensionality of features and mitigating the effects of high-frequency noise. Lu and Niu [16] combined the histogram features of the local neighborhood magnitude occurrence pattern (LNMOP) with a histogram of oriented gradient (HOG) and selected the final features for the classifier from the extracted LNMOP features. However, this approach does not consider the postprocessing of tampered images.A large number of approaches exist for the detection of seam-carved images, which has strongly promoted the development of the seam-carving forensics task. However, even if some images (such as higher-resolution biopsy slice images [17] with many details) are rarely tampered under low scaling ratio conditions, the content expressed may be changed. Thus, some scope for further improving the detection accuracy still exists. As we know, in an image that undergoes seam carving, the minimum cumulative energy seams defined by the energy function are removed, resulting in the change of not only the energy but also the local texture. Most existing methods employing this characteristic directly use the local binary pattern (LBP) features of the image for forensic detection; however, these methods cannot completely represent the change of local texture information.Accordingly, we introduce the cooccurrence of adjacent LBPs into the forensic task. Cooccurrence is defined as the simultaneous occurrence of all forms of adjacent LBPs, which are generally used to extract information that is bound up with the global structure in various characteristics based on local region; as described in [18], we obtain this information through the use of autocorrelation matrices calculated from two adjacent LBPs. The advantage of this feature is that it comprises both the original LBPs and the cooccurrence of adjacent LBPs; therefore, the location relationship of adjacent LBP values can be more explicitly determined. In this paper, the cooccurrence feature is combined with energy bias and noise-based features, as well as half-seam-based features, for feature extraction purposes. Subsequently, a trained support vector machine (SVM) classifier is used to determine whether seam carving has been applied to the image or not. Our experimental results prove that this forensic method achieves superior detection performance relative to existing methods. The contributions of the proposed approach are threefold: (1) we analyse the distortion that affects image resizing by seam carving and conclude that the relationship between adjacent pixels is an important clue for the forensics of seam-carving operation; inspired by [18], we extract the cooccurrence feature of adjacent LBPs; (2) we combined the cooccurrence feature of adjacent LBPs with the existent energy features to form a new feature set; (3) we apply PCA to reduce the dimensionality of the features to get an identified feature set; a series of experiments have verified that the proposed method has higher detection accuracy in identifying whether the image has undergone seam carving or not. Moreover, the experimental results demonstrate that the proposed approach has better robustness for TIFF and JPEG images.The remainder of this paper is organized as follows. Section2 briefly introduces the theory of the seam-carving algorithm. Section 3 introduces some related backgrounds and discusses the seam-carving detection method based on the cooccurrence of adjacent LBPs. The experimental results are reported in Section 4. Finally, Section 5 summarizes the paper.


## 2. Seam Carving

The seam-carving algorithm preserves important areas of the image and performs operations on unimportant areas. Because of the smoothness of unimportant areas in an image, it is not easy to cause visual perception if these areas are modified. As discussed in [2], the implementation of the seam-carving algorithm assigns different energy values to each pixel of the image on the basis of the importance of the pixels in said image and then calculates the parts that can be removed according to these energy values. The type of algorithm can bypass the limitations of traditional image processing technology, such as excessive deleting deformation [19] or distortion caused by image scaling, which is the reason why it can be so widely utilized.A seam is an eight-connected path, consisting of a set of connected pixels that pass through the image from top to bottom or from left to right. The least important parts, namely, the lowest energy pixel paths (called “optimal seams” in the image), are deleted as possible by using a seam-carving algorithm to reduce the image scale and achieve the desired width and height, while retaining the important content of the image in vision. In the calculation of the seam, the energy function of each path is defined as follows:(1)eI=∂∂xI+∂∂yI,where I is an image of n×m. There are two types of seams, namely, vertical seam and horizontal seam. Taking the vertical seam as an example, its definition is as provided in equation (2), while the definition of the horizontal seam is similar:(2)S=sii=1n=i,pii=1n,s.t.∀i,pi−pi−1≤1,where i denotes the row coordinates and pi denotes the corresponding column coordinates of pixels. On the basis of the above formula, the pixel value set of a seam S can be obtained as follows:(3)Is=Isii=1n=Ii,pii=1n.It should be noted that, for a vertical seam, there is only one pixel in each row of the seam. In light of the energy functioneI provided above, the optimal seam that minimizes this seam cost is defined as follows:(4)s∗=minsEs=min∑i=1neIsi.Dynamic programming is exploited to select the optimal seam, after which the optimal seam is continuously eliminated in order to reduce the image size or remove the target object. Taking the vertical seam as an example, the step traverses the image from the second row to the last row and computes the possible minimum energy matrixMi,j for each pixel i,j.(5)Mi,j=ei,j+minMi−1,j−1,Mi−1,j,Mi−1,j+1.Once the cumulative minimum energyM has been constructed for all possible seams, the optimal seams are found continuously by backtracking from the minimum value of the last row of M. The image content that has undergone seam carving is visually stable. Any possible visual artifacts appear only near the removed seams, while the remainder of the image remains unchanged. This is because, when an optimal seam is removed, the pixel paths at the right of the seam will move to the left to compensate for the missing pixel path. Generally speaking, it is difficult to visually detect such weak changes.An example of seam carving is presented in Figure1. Here, we perform seam carving on an image; the vertical seams to be eliminated are displayed and labeled in red. By removing these vertical seams, the width of the image is reduced by 15%. As can be seen from Figure 1(c), there is no obvious deformation or distortion of the refrigerator, and the most important content remains stable. This is because the smooth area in the image is mainly removed when seam carving is performed, which makes the change of image unable to be detected visually. Therefore, we study texture information of the image and then introduce the cooccurrence of adjacent LBPs and combine it with the energy features to detect this visually imperceptible tampering behavior.Figure 1
An image resized by seam carving. (a) An original image. (b) The image with vertical seams. (c) The seam-carved image.
(a)(b)(c)


## 3. Proposed Method

The energy feature can be used to describe the inherent characteristics of the image. In the process of seam carving, the lowest energy seams as defined by the energy function in the image are removed, bringing about changes in energy information of the image. At the same time, local texture change is important information for the operation of seam carving, which reflects the characteristics of the image surface; more specifically, it depicts the repeated local patterns and their arrangement rules in the image and additionally exhibits rotation invariance as well as excellent noise resistance performance. This combination of energy features and local texture features enables image information to be more comprehensively embodied, meaning that the proposed method can achieve higher detection accuracy. Therefore, in this paper, we briefly extract the energy features and local texture features related to the global structure in the LBP domain, where LBP is used to report the change of local texture.There are several steps involved in our proposed approach. First, the LBP values of the suspicious image are calculated with the goal of transforming the image to the LBP domain. Second, we extract the above-mentioned features in the LBP domain involving energy-based features, seam-based features, noise-based features, half-seam-based features, and the cooccurrence feature of adjacent LBPs. Among those features, as Ryu and Lee’s paper has introduced, energy-based features include average energy, average row energy, average column energy, and average energy difference, for a total of four features. The seam-based features refer to the minimum value, maximum value, mean value, standard deviation, and the difference between the maximum value and the minimum value in both directions (i.e., for both column and row directions) following the construction of the cumulative minimum energy matrixM for all possible seams. There are 10 features in total. In order to remove the noise from the candidate image I, the image is filtered using a Wiener filter F with windows; moreover, the noise is computed by n=I−FI, from which the average value, standard deviation, skewness, and kurtosis based on the noise can be calculated. As described in Yin’s paper, half-seam-based features are defined with reference to the half seams rather than the whole seams of the image; in other words, these are calculated according to the accumulated minimum energy matrix M of half of the image. Subsequently, three statistics (called min, max, and mean) are calculated for the specified column and row, for a total of six features. Based on the 24 features obtained at the end of this process, we propose to apply the cooccurrence feature of adjacent LBPs to seam-carving detection. As a result, we obtain a grand total of 25 LBP-based features totally. Following feature extraction, the SVM classifier is adopted for classification task; because of its supervised nature, it needs to be trained before the testing stage. In the final classification phase, we input the features extracted from the image under study into the trained SVM classifier. Finally, we obtain the detection results under different scaling ratios and compare them with the data derived through other methods. Figure 2 presents the general framework of our proposed method.Figure 2
The framework of the proposed method.


### 3.1. Local Binary Pattern (LBP)

As described in [20], LBP is a kind of operator applied to describe the local texture characteristics of an image. It is defined as having a window size of 3×3, with the center pixel value of the window used as the threshold value. Compared with the adjacent 8 pixel values, if the adjacent pixel value is bigger than the center pixel value, the position of this adjacent pixel is represented by 1; if not, it is represented by 0. The formula can be expressed as follows:(6)LBPxc,yc=∑p=18sIp−Ic∗2p,sx=1,x≥0,0,otherwise.Here,Ic is the center pixel value at location xc,yc of the image, Ip denotes the value of its adjacent pixel, and p indicates the total number of adjacent pixels.In this way, we can obtain an 8-bit binary number, which is generally converted into a decimal number. This value then is retained as the LBP value of the window center pixel in order to express the texture information of the3×3 region. By recursively calculating the LBP value of each pixel, the input image can be transformed into LBP domain. Figure 3 presents an example of a basic LBP operator. It can be seen from the figure that the LBP value of the center pixel is 83.Figure 3
An example of a basic LBP operator.


### 3.2. The Co-occurrence of Adjacent LBPs

In this paper, the cooccurrence characteristic of adjacent LBPs is applied to the detection of image seam-carving operation. Figure4 illustrates the difference between the LBP histogram and the histogram of special cooccurrence of adjacent LBPs.Figure 4
The comparison of LBP histogram and histogram of the cooccurrence of adjacent LBPs. (a) Two images where the center pixel LBP values areX, Y. (b) LBP histograms of the two images. (c) Histograms of the cooccurrence of adjacent LBPs of the two images.
(a)(b)(c)Figure4(a) displays two example images composed of two LBP patterns (X and Y); here, the numbers of Xs and Ys in each image are the same. The gray and white surrounding squares represent the values of adjacent pixels that are larger and smaller than the center pixel value, respectively. As can be seen in Figure 4(b), when the numbers of different LBP values in two disparate images are identical, the same results are obtained utilizing the original LBP histograms. However, the histograms of the cooccurrence of adjacent LBPs extracted from the two images are different, as shown in Figure 4(c). In this case, it is clear that the cooccurrence of adjacent LBPs can more efficiently give expression to the local texture characteristics of an image.In terms of the procedure for extracting the cooccurrence feature of adjacent LBPs, the encoding method proposed in [18] is exploited. When calculating the LBP value of an image (as shown in Figure 5), there are two sparser configurations of LBPs, including four pixels in the diagonal direction and four pixels in the cross direction of eight adjacent pixels; one of these is selected to reduce the computational complexity, enabling all possible LBP values to be obtained with a total number of N (N ≤ 16).Figure 5
Two sparser configurations of LBPs. (a) LBP (+). (b) LBP (×).There are four location cases from each reference LBP to its neighbor LBP, as shown in Figure6. Thereby, the configurations patterns of adjacent LBPs are combined in these four ways to facilitate continuously calculating the autocorrelation matrices of adjacent LBPs.Figure 6
The configuration patterns of the cooccurrence of adjacent LBPs.
(a)(b)(c)(d)The autocorrelation matrixCk is calculated continuously according to these four patterns, which can be expressed as(7)Ci,jk=∑a∈Ifiafja+kT,fia=1,if LBPa0,otherwise,where a is the position of every pixel in the image. k is the distance from the reference LBP to its adjacent LBP. i and j are two LBP values. After calculating all the autocorrelation matrices, eventually, these matrices are vectorized and combined to create 4N2 dimensional features.In order to achieve better detection performance, we utilize the Principal Component Analysis (PCA) [21] method, the purpose of which is to extract important information from a data table and represent this information as a new set of orthogonal variables, called the principal component. This approach is exploited to reduce the dimensionality of the high-dimensional data, allowing the introduced feature to be better combined with other features (Algorithm 1).Algorithm 1:Extraction of cooccurrence feature of adjacent LBPs.
Input: an image, IOutput: co-occurrence feature of adjacent LBPs of the image, CoALBP(1)
Transform I into LBP domain based on Figure5(2)
Define four configurations of adjacent LBPs shown in Figure6(3)
Calculate the auto-correlation matrices based on formula (7)(4)
Vectorize those matrices and obtain the CoALBP feature(5)
Reduce the dimensionality of the feature using PCA


### 3.3. SVM Classifier

After extracting the twenty-five features, a classifier is required to determine whether the image has undergone seam carving or not. As we know, support vector machine (SVM) can be utilized not only in medical scenarios such as prediction of the extubation failure [22] but also in forensic tasks such as image security.In this paper, first, we adopt the downloaded LibSVM [23] in MATLAB and svm-scale function to normalize the experimental data (such as features and labels) to be trained. The RBF kernel function is then selected. Grid search and 5-fold cross validation is exploited for parameter optimization selection in order to choose the best parameters c (the penalty factor) and g (parameter of gamma function), which are selected to train the entire training sets using SVM-train function to obtain a SVM model. Finally, according to the SVM model, the svm-predict function is used to detect image under investigation, while the confusion matrix is used to determine the classification result and the classifier accuracy.


## 3.1. Local Binary Pattern (LBP)

As described in [20], LBP is a kind of operator applied to describe the local texture characteristics of an image. It is defined as having a window size of 3×3, with the center pixel value of the window used as the threshold value. Compared with the adjacent 8 pixel values, if the adjacent pixel value is bigger than the center pixel value, the position of this adjacent pixel is represented by 1; if not, it is represented by 0. The formula can be expressed as follows:(6)LBPxc,yc=∑p=18sIp−Ic∗2p,sx=1,x≥0,0,otherwise.Here,Ic is the center pixel value at location xc,yc of the image, Ip denotes the value of its adjacent pixel, and p indicates the total number of adjacent pixels.In this way, we can obtain an 8-bit binary number, which is generally converted into a decimal number. This value then is retained as the LBP value of the window center pixel in order to express the texture information of the3×3 region. By recursively calculating the LBP value of each pixel, the input image can be transformed into LBP domain. Figure 3 presents an example of a basic LBP operator. It can be seen from the figure that the LBP value of the center pixel is 83.Figure 3
An example of a basic LBP operator.


## 3.2. The Co-occurrence of Adjacent LBPs

In this paper, the cooccurrence characteristic of adjacent LBPs is applied to the detection of image seam-carving operation. Figure4 illustrates the difference between the LBP histogram and the histogram of special cooccurrence of adjacent LBPs.Figure 4
The comparison of LBP histogram and histogram of the cooccurrence of adjacent LBPs. (a) Two images where the center pixel LBP values areX, Y. (b) LBP histograms of the two images. (c) Histograms of the cooccurrence of adjacent LBPs of the two images.
(a)(b)(c)Figure4(a) displays two example images composed of two LBP patterns (X and Y); here, the numbers of Xs and Ys in each image are the same. The gray and white surrounding squares represent the values of adjacent pixels that are larger and smaller than the center pixel value, respectively. As can be seen in Figure 4(b), when the numbers of different LBP values in two disparate images are identical, the same results are obtained utilizing the original LBP histograms. However, the histograms of the cooccurrence of adjacent LBPs extracted from the two images are different, as shown in Figure 4(c). In this case, it is clear that the cooccurrence of adjacent LBPs can more efficiently give expression to the local texture characteristics of an image.In terms of the procedure for extracting the cooccurrence feature of adjacent LBPs, the encoding method proposed in [18] is exploited. When calculating the LBP value of an image (as shown in Figure 5), there are two sparser configurations of LBPs, including four pixels in the diagonal direction and four pixels in the cross direction of eight adjacent pixels; one of these is selected to reduce the computational complexity, enabling all possible LBP values to be obtained with a total number of N (N ≤ 16).Figure 5
Two sparser configurations of LBPs. (a) LBP (+). (b) LBP (×).There are four location cases from each reference LBP to its neighbor LBP, as shown in Figure6. Thereby, the configurations patterns of adjacent LBPs are combined in these four ways to facilitate continuously calculating the autocorrelation matrices of adjacent LBPs.Figure 6
The configuration patterns of the cooccurrence of adjacent LBPs.
(a)(b)(c)(d)The autocorrelation matrixCk is calculated continuously according to these four patterns, which can be expressed as(7)Ci,jk=∑a∈Ifiafja+kT,fia=1,if LBPa0,otherwise,where a is the position of every pixel in the image. k is the distance from the reference LBP to its adjacent LBP. i and j are two LBP values. After calculating all the autocorrelation matrices, eventually, these matrices are vectorized and combined to create 4N2 dimensional features.In order to achieve better detection performance, we utilize the Principal Component Analysis (PCA) [21] method, the purpose of which is to extract important information from a data table and represent this information as a new set of orthogonal variables, called the principal component. This approach is exploited to reduce the dimensionality of the high-dimensional data, allowing the introduced feature to be better combined with other features (Algorithm 1).Algorithm 1:Extraction of cooccurrence feature of adjacent LBPs.
Input: an image, IOutput: co-occurrence feature of adjacent LBPs of the image, CoALBP(1)
Transform I into LBP domain based on Figure5(2)
Define four configurations of adjacent LBPs shown in Figure6(3)
Calculate the auto-correlation matrices based on formula (7)(4)
Vectorize those matrices and obtain the CoALBP feature(5)
Reduce the dimensionality of the feature using PCA


## 3.3. SVM Classifier

After extracting the twenty-five features, a classifier is required to determine whether the image has undergone seam carving or not. As we know, support vector machine (SVM) can be utilized not only in medical scenarios such as prediction of the extubation failure [22] but also in forensic tasks such as image security.In this paper, first, we adopt the downloaded LibSVM [23] in MATLAB and svm-scale function to normalize the experimental data (such as features and labels) to be trained. The RBF kernel function is then selected. Grid search and 5-fold cross validation is exploited for parameter optimization selection in order to choose the best parameters c (the penalty factor) and g (parameter of gamma function), which are selected to train the entire training sets using SVM-train function to obtain a SVM model. Finally, according to the SVM model, the svm-predict function is used to detect image under investigation, while the confusion matrix is used to determine the classification result and the classifier accuracy.


## 4. Experimental Results and Discussion

The hardware configuration used for experiments is a personal computer (Intel(R) Core(TM) i5-7400 CPU @ 3.00 GHz 8.00 GB RAM). We employ MATLAB 2016b as the experimental tool. Moreover, LibSVM [23] is used for the classification task. Python and Gnuplot are used for the parameter optimization.In order to prove the robustness of our proposed method in different format images, experiments are carried out on TIFF and JPEG images. In addition, our method focuses on the detection of low scaling ratio images and strives to improve their detection accuracy.As there is currently no complete image database for seam-carving forensics available, the Uncompressed Color Image Database (UCID) is utilized here due to its abundant content (e.g., people, plants, goods, etc.). We obtain 1338 original images without compression from the UCID image set and then implement seam-carving technology to resize these images in the following two cases: (1) after compressing images, under the condition that the quality factor (QF) is 10, 20, 50, 75, and 100; (2) exploiting uncompressed TIFF format images. In the former cases, the scaling ratios used for seam carving are 1%, 2%, 5%, 10%，20%, 30%, and 50%. For example, consider an image with a size of512×384 where the scaling ratio is 10% in the vertical direction; this means that the width is reduced by 10%, resulting in the image size changing to be 461×384. Thus, 1338 compressed images with QF of 10, 20, 50, 75, and 100 at scaling ratios of 1%, 2%, 5%, 10%, 20%, 30%, and 50% can be obtained, for a total of 46,830 images; there are also 1338 uncompressed images at scaling ratios of 1%, 2%, 5%, 10%, 20%, 30%, and 50%, respectively, for a total of 9366 images.Therefore, based on the adding of the original images, we divide the entire image set into identical sized training and testing sets. For compressed image sets at every QF and for uncompressed image sets, we have several training sets, the scaling ratios of which are 1%, 2%, 5%, 10%, 20%, 30%, and 50%. In addition, we have several testing sets at scaling ratios of 1%, 2%, 5%, 10%, 20%, 30%, and 50%. Furthermore, we also extract images from the uncompressed image sets with scaling ratios of 10%, 20%, 30%, and 50% to create a mixed set. The number of images in different scaling ratio is equal, and then we divide these sets equally into training and testing sets. Since the UCID image set is divided equally, the number of images in each subset is the same. To summarize, in the training and the testing sets, each subset contains 669 images with a specific scaling ratio. During the experiment, we firstly extract the cooccurrence feature of adjacent LBPs and twenty-four energy features of the images under investigation. We then use the SVM classifier after training to complete the classification tasks of these features, inferring how many images have been tampered (while the others have not been tampered with) under each subset of different scaling ratios, and then use these experimental data to evaluate the detection performance of our proposed approach in different situations and to compare it with other existing methods.Generally speaking, the quality factor (QF) has a certain influence on the detection operation. However, it can also be seen from Table1 that, at the condition of each specific QF, the method proposed in this paper can achieve extremely high accuracies for images with different scaling ratios; specifically, all images with large scaling ratios can be correctly detected. Furthermore, this also demonstrates the wide applicability of this method under different QFs (i.e., the method is stable and less affected by quality factors). When the value of QF is 0, meaning that the images are not compressed, it can be observed that this approach is also suitable and can achieve considerable accuracy.Table 1
Detection accuracy of different scaling ratios under different QF.
Scaling ratio (%)Quality factor (QF)0 (%)10 (%)20 (%)50 (%)75 (%)100 (%)191.2692.5294.9693.2696.1492.26292.1093.5594.2595.9597.9593.80594.3395.3197.3197.0197.0197.011096.1196.8596.8599.7097.8099.702098.8398.9599.1098.6598.6598.803099.9410010099.8599.8510050100100100100100100In order to demonstrate more clearly that the combination of the cooccurrence features of adjacent LBPs and 24 energy features yields the best experimental results, we also carried out a comparison experiment in which the cooccurrence features were added separately for the detection without the 24 energy features. EFCOFAL is used to describe the combination features of the cooccurrence feature of adjacent LBPs and 24 energy features, and COFAL is adopted to describe the cooccurrence features of adjacent LBPs.As can be seen from Table2, the effect of using the cooccurrence features of adjacent LBPs alone is not ideal compared with the effect when it they are combined with the 24 energy features, especially when scaling ratios are low.Table 2
Comparison of the accuracy of EFCOFAL and COFAL.
Scaling ratios (%)Accuracy (%)EFCOFALCOFAL191.2681.54292.1085.63594.3391.551096.1194.142098.8398.303099.9499.245010099.98Table3 summarizes the comparison results of the six forensic methods. As the image scaling ratio increases, the detection accuracy of the six approaches evidently also increases as across the board. In general, the proposed method achieves the best accuracy, which is higher on average by 35.74%, 30.75%, 13.27%, and 9.87% than the other four methods; particularly under small scaling ratios of 1%, 2%, and 5%, this method achieves more outstanding performance. Compared with the method proposed by Ye et al., it can be seen that the average accuracy of the proposed method is 0.25% higher when the scaling ratios are among 5% and 50%.Table 3
Comparison of the accuracy of seam-carving detection with different scaling ratios.
Scaling ratiosAccuracy (%)Wei et al. [5]Ryu and Lee [24]Yin et al. [6]Wattanachote et al. [7]Ye et al. [14]Ours1%50.0752.3951.1287.59—91.262%50.9752.7752.4787.86—92.105%50.2058.3063.8388.5493.9994.3310%57.9165.2280.0089.5096.7196.1120%74.1875.3794.4889.7098.5598.8330%91.3485.5298.6690.1399.0899.9450%94.9396.2799.8594.9099.60100Average63.8868.8786.3589.7597.5996.08The method proposed by Wei et al. does not consider the alternation of the internal nature (such as energy change) in an image caused by seam carving. Ryu and Lee [24] proposed a method based on energy bias and noise features in the LBP domain. Compared with the former method, this latter method takes advantage of the inherent change in the characteristics of the image following seam carving. On this basis, Yin et al. added six half-seam-based features that more comprehensively reflect the change of energy and local texture. However, the energy distribution changes when seams are inserted in a seam-carved image, which can offset the alternation of energy distribution caused by seam carving and make the forensic task more difficult. The change of local texture is also unable to reflect the location relationship of the adjacent LBP values in the local region, leading to the detection accuracy not being ideal when the scaling ratio is low. The method proposed by Wattanachote et al. involves displaying regular symmetrical data in the original JPEG image in the Blocking Artifact Characteristics Matrix (BACM), while the symmetrical data in the block reconstructed by the seam carving are destroyed. Accordingly, 22 features are proposed and used for the feature classification, and considerable accuracy is obtained as a result. However, due to the influence of the quality factor (QF), the accuracy exhibits a large wave motion, which is also (in brief) the reason why the performance of this method is not inadequate enough. Ye et al. proposed a deep-learning-based method; essentially, the CNN is exploited, after which more effective features are used to substantially boost the classification rates and obtain high detection accuracy.In essence, the method proposed in this paper, which is based on the local texture features and energy features of an image in the LBP domain, introduces the cooccurrence feature of adjacent LBPs, which can reflect the location relationship information of the LBP values corresponding to adjacent pixels in an image in order to improve the detection performance and thereby reduce the detection difficulty caused by the variety of changes to energy features. The experimental results reveal that, in fact, our proposed approach also achieves better detection accuracy.The receiver operating characteristic (ROC) curves for the four approaches are plotted in Figure7. The subfigures, that is, Figures 7(a)–7(f), represent the ROC performance under different scaling ratios from 1% to 50%, respectively. It can be observed that the corresponding area under ROC curves (AUC) obtained by the proposed method is significantly larger than the other three methods, whether the scaling ratio is large or small, demonstrating that our method can achieve higher accuracy and confirming its robustness.Figure 7
Comparison of ROC curves among the proposed method and three existing approaches. (a) ROC curve for 1% carved image. (b) ROC curve for 5% carved image. (c) ROC curve for 10% carved image. (d) ROC curve for 20% carved image. (e) ROC curve for 30% carved image. (f) ROC curve for 50% carved image.
(a)(b)(c)(d)(e)(f)In addition, we also exploit the cross experiment to test the effect of our proposed approach alongside the other four methods on different training and testing sets. The cross experiment image sets are conducted with five different scaling ratios, specifically 10%, 20%, 30%, 50%, and a mixed scaling ratio, where the images with scaling ratios of 10%, 20%, 30%, and 50% are uniformly distributed in the mixed image set.The detection accuracies under various experimental situations are listed in Table4, which displays the accuracy of cross experiment obtained using the UCID database. Table 4(a) presents the results of Wei et al.; the method achieves higher accuracy in the detection of mixed sets than the other three methods but not our proposed method. Table 4(b) demonstrates the results obtained by Ryu and Lee; it can be seen that when the mixed set is used as test set and training set, the experimental accuracy is generally not high. Table 4(c) lists the results of Yin et al., and Table 4(d) presents the results obtained by Wattanachote et al.; neither of the two methods has good robustness. Finally, Table 4(e) shows the results of our proposed approach. These results indicate that, overall, the detection accuracy of our proposed method is higher than that of the other four methods. Furthermore, our proposed method is more robust than the other four methods in the cross experiments.Table 4
The detection accuracies under cross experimental situations of different methods.
TestTrain10%20%30%50%Mixed(a) Wei et al. [5]10%—64.8663.7561.5872.4720%65.47—82.4862.3780.6730%61.6592.03—71.9291.9250%72.1494.4694.83—94.35Mixed66.2184.9983.1572.5784.85(b) Ryu and Lee [24]10%—62.1862.9362.4859.4920%72.27—69.5161.0667.6430%79.1579.75—72.9477.7350%85.2188.4276.01—90.21Mixed62.0361.8860.6259.4962.33(c) Yin et al. [6]10%—60.2456.0551.6463.8320%83.56—75.0456.2087.5230%92.6092.23—84.6886.4750%99.0399.4899.48—98.51Mixed69.2867.9466.5259.0476.01(d) Wattanachote et al. [7]10%—55.1954.6749.6354.1920%58.52—64.0556.9563.4530%53.7070.96—66.1872.3550%38.1576.9482.81—75.97Mixed52.6967.0068.8365.8869.74(e) Ours10%—85.2866.8266.3785.4320%99.78—80.3489.3987.5930%99.8598.03—98.3296.7150%100100100—99.48Mixed72.8070.9368.5462.3385.13Besides, according to Section3.2, we know that even though the dimensionality of the cooccurrence feature has been reduced in order to reduce the computational complexity in the process of generating the cooccurrence feature of adjacent LBPs, the feature dimensionality is still too large compared with other energy features. As shown in Table 5, when we use the feature directly without further dimension reduction, the detection accuracy is very low when testing the image with low scaling ratios.Table 5
Detection accuracy of different scaling ratios using features without dimension reduction.
Scaling ratios (%)12510203050Accuracy (%)29.4577.7389.5493.4295.6795.07100In this paper, we propose an image forensic approach based on the cooccurrence of adjacent LBPs, which can effectively report the location relationship of adjacent LBP values. From the above experimental results, it is evident that our method achieves good detection accuracy under different scaling ratios. When the scaling ratio is high, the detection accuracy of our method is almost 100%. Meanwhile, when the scaling ratio is low, it also achieves higher accuracy than other methods; this is of great significance for those images that have been tampered in a nonobvious way. The detection results of TIFF and JPEG images also show that our proposed approach is robust.


## 5. Conclusions and Future Work

Seam carving is widely utilized due to its ability to protect the important areas of an image from a visual perspective, meaning that the pivotal contents of the image are not distorted or deformed. Moreover, this technology may also be utilized maliciously, which can result in change to the semantic contents of the image; however, this situation may not be perceivable by the naked eye, meaning that it is more likely that people will be misled or that some harmful behaviors will occur, which endanger society. Therefore, despite the challenges, it is necessary to develop and improve seam-carving detection research. In this paper, a forensic method designed for the seam-carving detection task and based on the cooccurrence of adjacent LBPs is proposed. Experimental results demonstrate that our method has better detection performance and good robustness under different QF values and scaling ratios. This is of great significance for forensic work in the field of image security. However, the proposed approach only detects whether or not the image has been seam-carved and cannot determine the specific place at which seam carving has occurred within the image. In the future, we will continue to research location detection [25] of the seam-carved image. Moreover, many video/image processing methods [26–31] will be adopted to extract the identified features. We will also try to apply deep learning [32–37] methods to identify whether the image is seam-carved or not.



---

*Source: 8830310-2020-12-21.xml*