Image forgery detection. Using the power of CNN's to detect... | by Vishal Singh | Towards Data Science

With the advent of social networking services such as Facebook and Instagram, there has been a huge increase in the volume of image data generated in the last decade. Use of image (and video) processing software like GNU Gimp, Adobe Photoshop to create doctored images and videos is a major concern for internet companies like Facebook. These images are prime sources of fake news and are often used in malevolent ways such as for mob incitement. Before action can be taken on basis of a questionable image, we must verify its authenticity. The IEEE Information Forensics and Security Technical Committee (IFS-TC) launched a detection and localization forensics challenge, the First Image Forensics Challenge in 2013 to solve this problem. They provided an open dataset of digital images comprising of images taken under different lighting conditions and forged images created using algorithms such as:
Content-Aware Fill and PatchMatch (for copy/pasting)
Content-Aware Healing (for copy/pasting and splicing)
Clone-Stamp (for copy/pasting)
Seam carving (image retargeting)
Inpainting (image reconstruction of damaged parts — a special case of copy/pasting)
Alpha Matting (for splicing)
The challenge consisted of 2 phases.
Phase 1 required participating teams to classify images as forged or pristine (never manipulated)Phase 2 required them to detect/localize areas of forgery in forged images
Phase 1 required participating teams to classify images as forged or pristine (never manipulated)
Phase 2 required them to detect/localize areas of forgery in forged images
This post will be about a deep learning approach to solve the first phase of the challenge. Everything from data cleaning, pre-processing, CNN architecture to training and evaluation will be elaborated.
During the pre deep learning era of artificial intelligence i.e. before the Image Net challenge of 2012, researchers in image processing used to design hand made features for solving problems of image processing in general and image classification in particular. One such example is the Sobel kernel used for edge detection. The set of image forensic tools used earlier can be grouped into 5 categories namely
Pixel-based techniques that detect statistical anomalies introduced at the pixel levelFormat-based techniques that leverage the statistical correlations introduced by a specific lossy compression schemeCamera-based techniques that exploit artifacts introduced by the camera lens, sensor, or on-chip postprocessingPhysics-based techniques that explicitly model and detect anomalies in the three-dimensional interaction between physical objects, light, and the cameraGeometry-based techniques that make measurements of objects in the world and their positions relative to the camera
Pixel-based techniques that detect statistical anomalies introduced at the pixel level
Format-based techniques that leverage the statistical correlations introduced by a specific lossy compression scheme
Camera-based techniques that exploit artifacts introduced by the camera lens, sensor, or on-chip postprocessing
Physics-based techniques that explicitly model and detect anomalies in the three-dimensional interaction between physical objects, light, and the camera
Geometry-based techniques that make measurements of objects in the world and their positions relative to the camera
Courtesy: https://ieeexplore.ieee.org/abstract/document/4806202
Almost all these techniques exploit the content-based features of an image i.e. the visual information present in the image. CNN's are inspired by visual cortex. Technically, these networks are designed to extract features meaningful for classification i.e. the ones which minimize the loss function. The network parameters — kernel weights are learned by Gradient Descent so as to generate the most discriminating features from images fed to the network. These features are then fed to a fully connected layer that performs the final task of classification.
After looking at a few forged images, it was evident that locating the forged areas by human visual cortex is possible. CNN is thus the perfect deep learning model for this job. If a human visual cortex can detect it, certainly there is more power in a network which would be specifically designed for that task.
Before going into the dataset overview, the terminology used will be made clear
Fake image: An image that has been manipulated/doctored using the two most common manipulation operations namely: copy/pasting and image splicing.
Pristine image: An image that has not been manipulated except for the resizing needed to bring all images to a standard size as per competition rules.
Image splicing: The splicing operations can combine images of people, adding doors to buildings, adding trees and cars to parking lots etc. The spliced images can also contain resulting parts from copy/pasting operations. The image receiving a spliced part is called a “host” image. The parts being spliced together with the host image are referred to as “aliens”.
The entire dataset for both the first and second phase can be found here. For this project, we will be using only the train set. It contains 2 directories — one containing fake images and their corresponding masks and the other containing pristine images. Mask of a fake image is a black and white (not grayscale) image describing the spliced area of the fake image. The black pixels in the mask represent the area where manipulation was performed in the source image to get the forged image, specifically it represents the spliced region.
The dataset consists of 1050 pristine and 450 fake images. Color images are usually 3 channel images one channel for each red, green and blue colors, however sometimes the fourth channel for yellow may be present. Images in our dataset are a mix of 1, 3 and 4 channel images. After looking at a couple of 1 channel images i.e. grayscale images, it was evident that these images
were very few in numberwere streams of black or blue color
were very few in number
were streams of black or blue color
The challenge setters added these images on purpose as they wanted solutions robust to such noise. Although some of the blue images can be images of a clear sky. Hence some of them were included while others discarded as noise. Coming to four channel images — they too didn’t have any useful information. They were simply grids of pixels filled with 0 values. Thus our pristine dataset after cleaning contained about 1025 RGB images.
Fake images are a mix of 3 and 4 channel images, however, none of them are noisy. Corresponding masks are a mix of 1, 3 and 4 channel images. The feature extraction we will be using requires information from only one channel of the masks. Thus our fake image corpus has 450 fakes. Next up we did a train-test split to keep 20% of 1475 images for final testing.
The dataset in its present state is not apt for training a model. It must be transformed into a state which is well suited for the task at hand i.e. detection of anomalies at the pixel level introduced due to forging operations. Taking ideas from here, we designed the following methodology to create relevant images from the given data.
For every fake image, we have a corresponding mask. We use that mask to sample the fake image along the boundary of the spliced region in such a way so as to ensure at least a 25% contribution from both forged part and unforged part of the image. These samples will have the distinguishing boundaries that would be present only in fake images. These boundaries are to be learned by the CNN we design. Since all 3 channels of the mask contain the same information (fake part of an image in different pixels), we need only 1 channel to extract samples.
To make boundaries even more distinct, the grayscale images were converted to binary using Otsu’s thresholding (implemented in OpenCV) after denoising using a Gaussian filter. After this operation, sampling was merely a matter of moving a 64×64 window (with a stride of 8) through the fake image and counting 0 valued (black) pixels in the corresponding mask and sampling in case the value lies in a certain interval.
# Convert grayscale images to binary
binaries=[]

for grayscale in x_train_masks:
    blur = cv2.GaussianBlur(grayscale,(5,5),0)
    ret,th = cv2.threshold(blur,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
    binaries.append(th)

~binaries[28]

array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=uint8)
After sampling, we got 175,119 64×64 patches from fake images. In order to generate 0 labeled (pristine) patches, we sampled roughly the same number from authentic images. Finally, we had 350,728 patches which were split into train and cross-validation sets.
Now we have a large dataset of good quality input images. It is time to experiment with various CNN architectures.
The first architecture we tried was inspired by the architecture given in the original research paper. They had input images of size 128×128×3 and hence a large network. Since we have half the spatial size, our network was also smaller. This is the first tried architecture.
Here green layers are convolutional and blue ones are Max pool. This network was trained on 150,000 train samples (for testing purpose) and 25,000 validation samples. The network had 8,536 parameters which were relatively less compared to the train samples, hence avoiding the need for a more aggressive dropout. A dropout rate of 0.2 was applied to the flattened output of 20 units. We used Adam optimizer with a default value of learning rate (0.001) and beta_1, beta_2. After about ___ epochs the results were as follows
Train accuracy : 77.13%, Train loss : 0.4678
Validation accuracy : 75.68%, Validation loss : 0.5121
These numbers are not very impressive given the fact that in 2012 a CNN beat year long researched features developed by experts, by a huge margin. However, these numbers are not very bad either given the fact that we used absolutely no knowledge of image forensics (pixel statistics and related concepts) to get a ___ accuracy on unseen data.
Since it was a CNN which beat all classical machine learning algorithms at the ImageNet classification task, why not leverage the working of one of these powerful machines to solve the problem at hand? This is the idea behind Transfer learning. In a nutshell, we use the weights of a pre-trained model, which probably was trained on a much larger dataset and gave much better results in solving its problem, to solve our problem. In other words, we ‘transfer’ the learning of one model to build ours. In our case, we use the VGG16 network, trained on ImageNet dataset to vectorize images in our dataset.
Taking ideas from here, we tried 2 approaches
Use bottleneck features output by VGG16 and build a shallow network on top of thatFine tune the last convolutional layer of the VGG16+Shallow model in (1) above
Use bottleneck features output by VGG16 and build a shallow network on top of that
Fine tune the last convolutional layer of the VGG16+Shallow model in (1) above
As is intuitive, 2 gave much better results than 1. We tried multiple architectures for the shallow network before finally arriving on this
The input to the flattening layer is bottleneck features output by VGG16. These are tensors of shape (2×2×512) since we have used 64×64 input images.
Above architecture gave the following results
Train accuracy : 83.18%, Train loss : 0.3230
Validation accuracy : 84.26%, Validation loss : 0.3833
It was trained using Adam optimizer with a custom learning rate that reduced after every 10 epochs (besides regular updates by Adam after every batch).
The second approach requires the fine-tuning of the last layers. The important thing to note here is that we must use a pre-trained top layer model for fine tuning purpose. The objective is to change already learned weights slightly so as to fit the data better. If we use some randomly initialized weights, slight variations won’t do them any good and large variations would wreck the learned weights of the convolutional layer. We also need a very small learning rate to fine tune our model (for the same reason as mentioned above). In this post, it is suggested to use SGD optimizer for fine-tuning. However, we observed Adam outperforming SGD at this task.
Fine-tuned model gave the following results
Train accuracy : 99.16%, Train loss : 0.018
Validation accuracy : 94.77%, Validation loss : 0.30
Slightly overfit model which can be remedied by using an even smaller learning rate (we used 1e-6).
Apart from VGG16 we also tried bottleneck features from ResNet50 and VGG19 models pre-trained on Image-Net dataset. Features from ResNet50 outperform VGG16. VGG19 didn’t give a very satisfactory performance. We fine-tuned the ResNet50 architecture (last convolution layer) in a way similar to that of VGG16 using the same learning rate update strategy and it gave more promising results with lesser overfit problem.
Train accuracy : 98.65%, Train loss : 0.048
Validation accuracy : 95.22%, Validation loss : 0.18
To sample images from the test set created earlier, we employed a similar strategy as used to create the train and cross-validation sets i.e. sampling fake images at boundaries using their masks and sampling an equal number of pristine images having same dimensions. The fine-tuned VGG16 model was used to predict labels for these patches and gave the following results
Test accuracy : 94.65%, Test loss : 0.31
ResNet50, on the other hand, gave the following results on test data
Test accuracy : 95.09%, Test loss : 0.19
As we can see, our models are performing decently. We still have much room for improvement. If more data can be generated through data augmentation (shear, resize, rotate, and other operations) perhaps we can fine tune more layers of SOTA networks.
In this post, we talked about detecting a fake image. However, once a fake image has been detected, we must determine the forged area in that image. Localization of spliced area in a fake image will be the topic of next post. The whole code for this part can be found here.
That’s it for this post. Do let me know other good techniques to detect fake images in the comments section. Until next time...Farewell.