Image forgery detection
By: Vishal Singh
Tags: Machine Learning, Deep Learning, Cnn

After sampling, we got 175,119 64×64 patches from fake images. In order to generate 0 labeled (pristine) patches, we sampled roughly the same number from authentic images. Finally, we had 350,728 patches which were split into train and cross-validation sets.

Now we have a large dataset of good quality input images. It is time to experiment with various CNN architectures.

Custom CNN Architecture

The first architecture we tried was inspired by the architecture given in the original research paper. They had input images of size 128×128×3 and hence a large network. Since we have half the spatial size, our network was also smaller. This is the first tried architecture.

First architecture

Here green layers are convolutional and blue ones are Max pool. This network was trained on 150,000 train samples (for testing purpose) and 25,000 validation samples. The network had 8,536 parameters which were relatively less compared to the train samples, hence avoiding the need for a more aggressive dropout. A dropout rate of 0.2 was applied to the flattened output of 20 units. We used Adam optimizer with a default value of learning rate (0.001) and beta_1, beta_2. After about ___ epochs the results were as follows

Train accuracy : 77.13%, Train loss : 0.4678

Validation accuracy : 75.68%, Validation loss : 0.5121

These numbers are not very impressive given the fact that in 2012 a CNN beat year long researched features developed by experts, by a huge margin. However, these numbers are not very bad either given the fact that we used absolutely no knowledge of image forensics (pixel statistics and related concepts) to get a ___ accuracy on unseen data.

Transfer learning

Since it was a CNN which beat all classical machine learning algorithms at the ImageNet classification task, why not leverage the working of one of these powerful machines to solve the problem at hand? This is the idea behind Transfer learning. In a nutshell, we use the weights of a pre-trained model, which probably was trained on a much larger dataset and gave much better results in solving its problem, to solve our problem. In other words, we ‘transfer’ the learning of one model to build ours. In our case, we use the VGG16 network, trained on ImageNet dataset to vectorize images in our dataset.

Taking ideas from here, we tried 2 approaches

Use bottleneck features output by VGG16 and build a shallow network on top of that Fine tune the last convolutional layer of the VGG16+Shallow model in (1) above

As is intuitive, 2 gave much better results than 1. We tried multiple architectures for the shallow network before finally arriving on this

Top layer architecture

The input to the flattening layer is bottleneck features output by VGG16. These are tensors of shape (2×2×512) since we have used 64×64 input images.

Above architecture gave the following results

Train accuracy : 83.18%, Train loss : 0.3230

Validation accuracy : 84.26%, Validation loss : 0.3833

It was trained using Adam optimizer with a custom learning rate that reduced after every 10 epochs (besides regular updates by Adam after every batch).

Custom update rule

The second approach requires the fine-tuning of the last layers. The important thing to note here is that we must use a pre-trained top layer model for fine tuning purpose. The objective is to change already learned weights slightly so as to fit the data better. If we use some randomly initialized weights, slight variations won’t do them any good and large variations would wreck the learned weights of the convolutional layer. We also need a very small learning rate to fine tune our model (for the same reason as mentioned above). In this post, it is suggested to use SGD optimizer for fine-tuning. However, we observed Adam outperforming SGD at this task.

Fine-tuned model gave the following results

Train accuracy : 99.16%, Train loss : 0.018

Validation accuracy : 94.77%, Validation loss : 0.30

Slightly overfit model which can be remedied by using an even smaller learning rate (we used 1e-6).

Apart from VGG16 we also tried bottleneck features from ResNet50 and VGG19 models pre-trained on Image-Net dataset. Features from ResNet50 outperform VGG16. VGG19 didn’t give a very satisfactory performance. We fine-tuned the ResNet50 architecture (last convolution layer) in a way similar to that of VGG16 using the same learning rate update strategy and it gave more promising results with lesser overfit problem.

Train accuracy : 98.65%, Train loss : 0.048

Validation accuracy : 95.22%, Validation loss : 0.18

Final model prediction on test data

To sample images from the test set created earlier, we employed a similar strategy as used to create the train and cross-validation sets i.e. sampling fake images at boundaries using their masks and sampling an equal number of pristine images having same dimensions. The fine-tuned VGG16 model was used to predict labels for these patches and gave the following results

Test accuracy : 94.65%, Test loss : 0.31

ResNet50, on the other hand, gave the following results on test data

Test accuracy : 95.09%, Test loss : 0.19

As we can see, our models are performing decently. We still have much room for improvement. If more data can be generated through data augmentation (shear, resize, rotate, and other operations) perhaps we can fine tune more layers of SOTA networks.

In this post, we talked about detecting a fake image. However, once a fake image has been detected, we must determine the forged area in that image. Localization of spliced area in a fake image will be the topic of next post. The whole code for this part can be found here.

That’s it for this post. Do let me know other good techniques to detect fake images in the comments section. Until next time…Farewell.

Sources