Siamese convolutional neural network‐based approach towards universal image forensics

IET Image ProcessingVolume 14, Issue 13 p. 3105-3116 Research ArticleFree Access Siamese convolutional neural network-based approach towards universal image forensics Aniruddha Mazumdar, Corresponding Author Aniruddha Mazumdar m.aniruddha@iitg.ac.in orcid.org/0000-0001-6909-0428 Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Assam, 781039 IndiaSearch for more papers by this authorPrabin Kumar Bora, Prabin Kumar Bora Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Assam, 781039 IndiaSearch for more papers by this author Aniruddha Mazumdar, Corresponding Author Aniruddha Mazumdar m.aniruddha@iitg.ac.in orcid.org/0000-0001-6909-0428 Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Assam, 781039 IndiaSearch for more papers by this authorPrabin Kumar Bora, Prabin Kumar Bora Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Assam, 781039 IndiaSearch for more papers by this author First published: 02 October 2020 https://doi.org/10.1049/iet-ipr.2019.1114Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This study proposes a novel deep learning-based method which can detect different types of image editing operations carried out on images. Unlike most of the existing methods, which can only detect the editing operations considered in the training stage, the proposed method can generalise to manipulations not seen in the training stage. The method is based on the classification of image pairs as either similarly or differently processed using a deep siamese neural network. Once the network learns features that can discriminate different editing operations, it can check whether an image is processed with an editing operation, not present in the training stage, using the one-shot classification strategy. An image forgery detection and localisation technique is also proposed using the trained siamese network. The experimental results show the efficacy of the proposed method in detecting different editing operations and also show the ability in detecting and localising image forgeries. 1 Introduction With the availability of various image editing software, it has become possible to create visually plausible image forgeries with minimal effort. Because of this, a large number of forged images are now available on the web. These images, when used on different platforms such as electronic and social media, may create tensions in society. These concerns necessitate the development of image forensics techniques for checking the authenticity of images before using them as critical information. The detection of different types of image editing operations carried out on an image is an important problem in image forensics. This is because manipulated images are generally post-processed by different types of image editing operations to remove the trace of manipulations left by the forgery process. For example, median filtering can be applied to remove the trace of JPEG blocking artefacts present in an image [1]. However, every editing operation leaves behind a unique signature, which is utilised by researchers to detect the type of operation carried out on the image. There are many methods available in the literature, which focus on the detection of image editing operations applied to images. For example, in [2–4] the authors extracted features for detecting the traces left by resizing and resampling operations, in [5–7] features related to median filtering traces are extracted, in [8, 9] features are extracted to detect contrast-enhancement operation, and in [10] JPEG artefact related features are extracted for forgery detection. The processing history of an image can also expose different types of forgeries present in images. This is because while creating forgeries, the forged parts are often processed through various image editing operations, such as resizing, blurring, contrast enhancement, compression, to make them look visually plausible. On the other hand, the authentic parts may not go through the same type of editing operations as the forged parts. Therefore, the presence of different types of image editing operations at different parts of an image can give a clue about its authenticity. Although the methods [2–10] are good at detecting specific types of manipulations, methods from each of the categories work only under their own assumptions about the traces of manipulations left by the forgery process. For example, the median filtering detection methods cannot detect traces left by the resampling operation. To handle this limitation, researchers have focused on developing universal forensics methods, which can detect multiple manipulations in a single framework. The first universal forensics method was proposed by Qiu et al. [11], where different steganalysis features were used to detect different types of image processing operations. The method is based on the observation that different image editing operations destroy the natural statistics of the image pixels present in an authentic image in the same way steganography methods do while manipulating the pixels for embedding a message. Fan et al. [12] proposed another general-purpose forensics method for detecting different types of image editing operations. The authors proposed to create a Gaussian mixture model (GMM) of image patches corresponding to each editing operation. Then, the average log-likelihood of patches under the different GMMs corresponding to different classes are compared to decide the class of the patches. Inspired by the success in other computer vision areas, the forensics community has recently focused on applying deep learning-based methods for image manipulation detection. Chen et al. [13] proposed the first deep learning-based median filtering detection method. In this method, the first layer computes the median filtering residual and the subsequent layers extract and classify the features useful for median filtering detection. Bayar and Stamm [14] proposed a deep learning-based universal forensics method for detecting different types of image manipulating operations. The image editing features are automatically learned from the training data by employing a convolutional neural network (CNN) [15]. The authors proposed a new convolutional layer, which suppresses the image content and enhances features important for detecting different editing operations. Although these universal manipulation detection methods perform well, all the manipulation operations have to be known before training the network. However, a large number of image editing operations are included in the image editing software, like Adobe Photoshop and GIMP. Also, new image editing operations are being developed and incorporated in these editing software. In addition to that, there may be multiple editing operations performed subsequently to make the forged parts look similar to the authentic parts. Therefore, it is not practical to incorporate all the editing operations in the training process as required by the existing universal manipulation detection methods, i.e. [11, 12, 14]. This necessitates the development of universal forensics method which can detect not only the image editing operations present in the training stage, but also can generalise to editing operations unknown in the training stage. This paper proposes a novel deep learning-based forensics method which can not only detect known manipulations but also can check the presence of manipulations not present in the training stage. The proposed method takes two images as inputs, and checks whether they have undergone the same or different manipulation operations [The initial version of the paper is uploaded in arXiv https://arxiv.org/abs/1808.06323]. The method is inspired by the works of Chen et al. [13] and Bayar and Stamm [14], where they showed that CNNs are capable of learning image manipulation features automatically from the training data. However, instead of learning features to classify images to different manipulation classes, the proposed method learns the features which can discriminate various image editing operations. To achieve this, we employ a siamese network-based metric learning technique [16], which is capable of learning generic image features that can generalise to unknown classes. The siamese network has twin CNNs accepting two images as the input and classifies the image pair as either similarly processed (SP) or differently processed (DP). Once the network learns to differentiate between SP and DP image pairs, we use it to detect image editing operations present in a gallery of manipulations in a one-shot classification fashion. The gallery contains the image processing operations commonly used in the anti-forensics methods. The gallery may contain manipulations that are not present in the training stage. Therefore it can be augmented by adding more and more image processing operations, which may be required by a forensics analyst at the time of analysis. We assume that we have at least one image from each manipulation class considered in the gallery. Given a test image, we first check whether the image is unaltered or manipulated by comparing it with a reference unaltered image using the trained siamese network. If the image is classified as manipulated, we check whether it is manipulated using any of the editing operations present in the test gallery. In case, the test image is manipulated using an unknown image editing operation not considered in the test gallery, the method will not be able to identify the exact class of the manipulation. However, it will still be able to tell that the image has undergone a manipulation with an unknown editing operation. We also propose a forgery detection and localisation technique using this trained siamese network. The method first divides the image under investigation into a number of patches and then compare them in a pairwise manner using the siamese network for localising the forgeries. The main contributions of this work are as follows: 1. A novel siamese network-based manipulation detection technique is proposed, which can (i) detect image editing operations considered in the training stage, (ii) detect known editing operations not considered in the training stage, and (iii) check whether a test image has undergone any unknown image editing operation. 2. Based on the above manipulation detection method, an image forensics technique is proposed to localise and detect different image forgeries effectively. Recently, Mayer and Stamm [17] have, independently of our work, proposed their siamese network-based Forensics Similarity method for checking whether a pair of images has the same forensic trace or not. More specifically, they have trained a siamese network to check whether a pair of images are (i) captured by the same or different camera models, and (ii) modified by the same or different image editing operations. They have also shown the ability of the trained siamese network in detecting and localising forgeries, such as splicing. Although both the proposed and Forensics Similarity methods use siamese networks for manipulation detection, there are a number of differences in the network architecture. More notably, the proposed method uses a constrained convolution layer operating on the green channel of the input images. On the other hand, Forensics Similarity method processes all the three RGB channels of the input images, and relaxes the constraint imposed on the first convolutional layer. Also, the proposed method computes a distance metric between the twin CNN features, whereas Forensics Similarity method concatenates the twin CNN features along with their element-wise multiplication. In addition to that, we propose to employ the one-shot classification technique to check the presence of unknown manipulations in an image under investigation. The rest of the paper is organised as follows: Section 2 describes the background of deep learning-based image forensics techniques and also outlines the motivation for the proposed work. Section 3 presents the proposed manipulation detection method. Section 4 presents a novel forgery localisation technique. Section 5 discusses the results of the experiments carried out to show the efficacy of the proposed method. Section 6 concludes the paper and highlights future work. 2 Background and motivation In this section, we briefly explain the existing deep learning-based forensics methods, and then outline the motivation for the proposed method. The forensics methods proposed for detecting image editing operations are based on the assumption that each editing operation leaves behind the trace of the particular manipulation. These traces can be detected by examining the relationship between the neighbouring pixels, as any image manipulation destroys the natural statistics of pixels and modifies them in a unique way [11]. Chen et al. [13] and Bayar and Stamm [14] proposed to employ CNNs to automatically learn the features useful for the detection of different manipulation operations. CNNs have proved to be very good in different computer vision tasks, e.g. object detection and recognition [18]. However, they did not perform well when applied directly to image manipulation detection [13]. This is because the conventional CNNs capture the image content rather than important forensics features. Therefore, to suppress the image content and enhance the relationship between neighbouring pixels, most of the forensics methods employ prediction error filters in the first layer of the CNNs. These prediction error filters perform high-pass filtering on the input image and produce noise residuals which are then fed to the subsequent layers of the CNN. For instance, Chen et al. [13] proposed to first extract the median filtering residuals for detecting median filtering operation. Bayar and Stamm [14] proposed a new convolutional layer, named as the constrained convolutional layer, as the first layer in their proposed CNN to adaptively suppress the image content for detecting different image editing operations. The constrained convolutional layer learns a set of prediction error filters by constraining the weights in each of the K filters as (1)where denotes the weight at position (l,m) of the kth filter and denotes the weight at the centre of the corresponding filter kernel. This procedure is repeated for all the pixels by moving the kernels throughout the image. This prediction error layer extracts the local dependency of pixels with its neighbours, which is the important information from the forensics point of view [11]. The concept of using the prediction error filters in the first layer is motivated by different image forensics and steganalysis methods. Steganalysis methods like the spatial-domain rich models (SRM) [19] and the subtractive pixel adjacency matrix (SPAM) [20] utilise this concept of using different prediction filters for computing the prediction errors, which are later used as features to detect hidden messages present in the stage images. In forensics, Rao et al. [21] proposed to initialise the first layer of a CNN with the SRM filters. Recently, Zhou et al. [22] proposed a forensics method, where they used 3 SRM filters as the first layer of a CNN. They also showed that increasing the number of filters does not necessarily boost performance. The universal forensics method proposed by Bayar and Stamm [14] shows that CNNs can automatically learn features important for detecting different image editing operations. However, there are some limitations of the method. For training the CNN, all the image editing operations should be known a priori. If an image is subjected to an operation other than the ones used for training, the method will not be able to detect it. To overcome this limitation, we propose a novel forensics method which can not only detect the image manipulations present in the training stage, but also can check whether an image is being processed by manipulations unknown during the training stage. 3 Proposed manipulation detection method The proposed method is based on the classification of image patch pairs as either SP or DP through a metric learning-based technique using a siamese network [23, 24]. More specifically, the siamese network learns a distance metric to check whether two image patches have gone through the same type of editing operations or not. The reasons for the siamese network-based classification of patch pairs are as follows: 1. Unlike simple CNN-based methods, e.g. [13, 14], the siamese network can learn more generic image manipulation features through a distance metric learning-based approach. This is an important advantage of the proposed method, as it allows to discriminate/detect manipulations not present in the training stage. 2. Since, the methods in [11, 12, 14] learn class-specific features to classify the images into one of the different but fixed types of manipulations, they are more vulnerable to anti-forensics. Anti-forensics methods can be developed to hide the traces left by each of the operations considered in these methods [25]. On the other hand, the proposed method learns a distance metric to check whether two image patches have undergone the same type of manipulation or not. Hence, developing anti-forensics techniques to counter the proposed method will be more difficult. 3. In a forged image, the tempered regions are generally post-processed with different image editing operations to make them look visually plausible. As a result, there are differences between the editing operations present in the manipulated and the authentic regions in the forged image. Therefore, checking the differences in the editing operations might expose the forgery. The proposed manipulation detection method using siamese network-based patch pair classification technique is described below. 3.1 Siamese convolutional neural network Siamese network was first proposed by Broomley et al. [23] for solving the signature verification problem. A siamese network consists of two identical sub-neural networks accepting one input each. The outputs of the two sub-networks are passed to a distance layer, which computes a distance metric between them. The distance metric produces a high value if the two images in a pair come from two different classes and a low value if they come from the same class. The proposed method employs a deep siamese CNN, which is capable of learning features that can discriminate different image editing operations. Fig. 1 shows the block diagram of the proposed framework. It has twin neural networks CNN1 and CNN2 sharing the same set of weights. It accepts two input image patches, which are parallelly processed by CNN1 and CNN2 producing two feature vectors and . The distance layer [16] computes a distance metric between and , then applies the sigmoid non-linearity to map the output in the range and produces the logistic prediction p. Because of the sharing of weights, CNN1 and CNN2 map two similar input images to very close points in the feature space. The proposed siamese CNN automatically learns the features that can check whether a pair of image patches is SP or DP. Fig. 1Open in figure viewerPowerPoint Framework of the siamese network which takes a pair of input image patches and produce prediction p indicating whether the pair is SP or DP 3.2 Network architecture 3.2.1 CNN Each of CNN1 and CNN2 has the architecture shown in Fig. 2. The input to each CNN is a gray-scale image of size . It contains 1 constrained convolutional layer, 3 convolutional layers, 2 average-pooling layers and 3 fully-connected layers. The hyper-parameters used in the proposed CNN are based on empirical results, which are presented in the experimental section. The block diagram of the CNN is shown in Fig. 2. The constrained convolutional layer contains 3 constrained convolutional filters [14] whose weights follow the constraints given by (1). This layer is followed by a convolutional layer with 64 filters of size with stride 1. The ReLU non-linearity is applied element-wise to the output of this layer, followed by an average-pooling layer with a kernel size and stride 2. The output of this layer is fed to another convolutional layer with 128 filters of size and stride 1. The ReLU non-linearity is applied element-wise to the output of this layer. It is followed by an average-pooling layer with a kernel size and stride 2. The reason for using convolution filters are: (i) it has fewer parameter than bigger filters, therefore reduces the overfitting and (ii) it combines the features present at the same location across different feature maps. The output of this layer is fed to another convolutional layer with 128 filters of size and stride 1. The ReLU non-linearity is applied element-wise to the output of this layer. It is followed by an average-pooling layer with a kernel size and stride 2. This layer is followed by two fully-connected layers, each with 1024 neurons. The sigmoid non-linearity is used in each of these layers. The neurons in the fully-connected layers are dropped out [26] with a probability of at each iteration of the training process. The output of the final fully-connected layer represents the features learned by the CNN. Fig. 2Open in figure viewerPowerPoint CNN architecture used in the proposed siamese network 3.2.2 Distance layer Given a pair of images and as inputs, CNN1 and CNN2 compute the feature vectors and , respectively. A distance layer computes a distance metric between them, which is then fed to a single sigmoidal output neuron. This neuron computes the prediction of the input image pair as , where is the sigmoid non-linearity function and is a learnable parameter representing the importance of each component of the feature vectors in the classification of the patch pair. 3.3 Manipulation detection using one-shot classification Once the siamese network learns to discriminate between the SP and the DP image patch pairs, we use it to detect different image editing operations in a pairwise manner. More specifically, given a test image patch and a set of image patches coming from each of the editing operations present in a gallery of manipulations, we first compute the pairwise prediction of and . The prediction represents the similarity between the types of editing operations applied on and . Thus, the type of operation applied on is given by the class for which prediction is maximum, i.e. (2) The main advantage of using a siamese network is that once it learns to discriminate/detect different classes present in the training stage, it can generalise to unseen classes with very few images per class. This is because the siamese network learns more generic discriminative features than simple CNNs by learning a distance metric from the training data [16]. Once the network learns the metric, it can be used to compare images coming from classes not present in the training stage [24]. In the extreme case, given only one example per unseen class, the network can check whether a test image patch has undergone any of these manipulations. This is the one-shot classification method. Therefore, once we train the proposed siamese CNN to discriminate different editing operations present in the training stage, it learns features that are capable of detecting/discriminating unseen editing operations. Now, given a test image patch, we first compare it with a reference unaltered image using the trained siamese network. If the pairwise prediction , the test patch is classified as unaltered. Otherwise, the test patch is considered to be manipulated, and the type of manipulation is detected using one-shot classification technique. We assume to have at least one reference image patch from each manipulation class, included in the gallery of manipulations which we want to check whether applied on the test image patch or not. To achieve this, we compare the test image patch in a pairwise manner with one reference image patch from each class. The class of the test image patch is given by the class of the reference image corresponding to the maximum prediction score p, as given by (2). However, we need to ensure that the maximum prediction score is sufficiently high. In this work, we assume that it is bigger than . This is because in case the test image patch is manipulated using an editing operation not present in our gallery of manipulations, the maximum prediction score will not correspond to the true manipulation class. Also, in this case, the maximum prediction score will not be very high. Hence, if the maximum prediction score is less than , we do not assign any class to the image patch and consider this as a manipulated image. The proposed manipulation detection method is presented in Algorithm 1 (see Fig. 3). Fig. 3Open in figure viewerPowerPoint Algorithm 1: % Algorithm for detecting the class of manipulation applied on a test image patch% 4 Forgery localisation and detection We now describe the method to localise and detect different forgeries present in images using the proposed siamese network. The method is based on the following assumptions: (i) all the parts of an authentic image go through the same types of editing operations, (ii) while creating a forged image, there are often different image editing operations carried out on the forged parts to make them look visually plausible, and (iii) the area of the forged parts is less than the area of the authentic parts. For example, in image splicing forgery, the spliced objects may come from a different noisy environment or maybe manipulated using various image editing operations. In the case of copy-move forgery, one region may be copied and pasted in a different location after performing some post-processing operations like up/downsampling. Therefore, detecting the presence of more than one type of image editing operations in an image can give a clue about the authenticity of an image. To check the presence forgery in a test image, it is first divided into a number of overlapping patches, where N is the number of patches. We consider overlap between two adjacent patches. One of these patches is randomly selected as a reference patch, , and others are compared with in a pairwise fashion using a trained siamese network. The siamese network is trained to classify a pair of patches as either SP or DP, as explained in the previous section. Since it gives the probability p of classifying a pair as SP, ideally, all the pairs of patches in an authentic image will have higher values of p. In the case of a forged image, the reference patch may come either from the authentic part or from the forged part. If the reference patch comes from the forged part, there will be high pairwise probabilities for forged patches and low pairwise probabilities for authentic patches. On the other hand, if the reference patch comes from the authentic part, the forged patches will have low pairwise probabilities and the authentic patches will have high pairwise probabilities. Therefore, we first decide the class of the reference patch, i.e. whether authentic or forged and then detect the forged patches accordingly. Suppose, represents the percentage of patches having p values lower than a predefined threshold th, i.e. (3)where 1(.) is an indicator function which gives 1 whenever its argument is true otherwise it gives 0, and is the p value for the patch . If is less than , i.e. (4)the reference patch, , is considered as authentic. Otherwise, is considered as forged. Once the class of the reference patch is decided, the forged patches are detected as per the following two cases: • Case-1: If is authentic, the patches corresponding to the pairs having p values less than th is considered as the forged patches. This is based on the assumption that the area of the forged region is less than the area of the authentic region. • Case-2: If is forged, the patches corresponding to the pairs with p values greater than th is considered as forged patches. Once the forged patches are detected, a binary mask image is created by assigning the pixels in the forged patches a value of 0 and pixels in the authentic regions a value of 1. The binary mask helps in the qualitative and quantitative analysis of the effectiveness of the forgery localisation technique. We have experimented with different image patch sizes, i.e. , and , and found that patches of size with a stride of 32 pixels is the best option among them. Although, the manipulation detection accuracy increases as the patch size increases, the localisation ability of the forgery detection method reduces with increasing patch size. On the other hand, if the patch size is too small, the manipulation detection will not be reliable and hence it will produce many false-alarms in the forgery detection process. The steps of the forgery localisation technique are outlined in the following Algorithm 2 (Fig. 4). Fig. 4Open in figure viewerPowerPoint Algorithm 2: % Algorithm for forgery localisation using the siamese network trained to discriminate different i