SigNet (Detecting Signature Similarity Using Machine Learning/Deep Learning): Is This the End of Human Forensic Analysis?
By: Aadit Kapoor
Tags: Machine Learning, Artificial Intelligence, Python, TensorFlow, Data Science

SigNet (Detecting Signature Similarity Using Machine Learning/Deep Learning): Is This the End of Human Forensic Analysis?

SigNet (Detecting Signature Similarity using Machine Learning/Deep Learning): Is this the end of Human Forensic Analysis?

My grandfather was an expert in handwriting analysis. He spent all his life analyzing documents for the CBI (Central Bureau Of Investigation) and other organizations. His unique way of analyzing documents using a magnifying glass and different tools required huge amounts of time and patience to analyze a single document. This is back when computers were not fast enough. I remember vividly that he photocopied the same document multiple times and arranged it on the table to gain a closer look at the handwriting style.

Handwriting analysis involves a comprehensive comparative analysis between a questioned document and the known handwriting of a suspected writer. Specific habits, characteristics, and individualities of both the questioned document and the known specimen are examined for similarities and differences.

As this problem consists of detecting and analyzing patterns, Machine Learning is a great fit to solve this problem.

A handwritten document captures a lot of detail. (https://unsplash.com/photos/AbQNy5Vvpjc)

Why and How?

Why: My grandfather’s unique way of analyzing documents using a magnifying glass and different tools required huge amounts of time and patience to analyze a single document. This is back when computers were not fast enough. I remember vividly that he photocopied the same document multiple times and arranged it on the table to gain a closer look at the handwriting style. While I agree that we cannot replace that job with an A.I with a 100% accuracy, we can certainly build a system capable of aiding human beings.

How: To build our Signature Similarity network, we will use utilize the wonders of Deep Learning. We will go through three approaches to extract the similarity between our handwritten signatures. For our initial data, we will use the HandWritten Signatures dataset from Kaggle.

Requirements

For this project we will require:

Python 3.8: The Programming Language

TensorFlow 2: The Deep Learning Library

Numpy: Linear Algebra

Matplotlib: Plotting images

Scikit-Learn: General Machine Learning Library

The Dataset

The dataset contains real and forged signatures of 30 people. Each person has 5 genuine and 5 forged signatures.

The Directory Structure of our data.

For loading the data, I have created a simple load_data() that iterates through all the datasets and extracts real and forged signatures with a label of 1 and 0 respectively.

In addition to this, I have also created a dictionary of tuples consisting of images and labels. (To be used later in the project).

def load_data(DATA_DIR=DATA_DIR, test_size=0.2, verbose=True, load_grayscale=True):

"""

Loads the data into a dataframe.



Arguments:

DATA_DIR: str

test_size: float

Returns:

(x_train, y_train,x_test, y_test, x_val, y_val, df)

"""

features = []

features_forged = []

features_real = []

features_dict = {}

labels = [] # forged: 0 and real: 1

mode = "rgb"

if load_grayscale:

mode = "grayscale"



for folder in os.listdir(DATA_DIR):

# forged images

if folder == '.DS_Store' or folder == '.ipynb_checkpoints':

continue

print ("Searching folder {}".format(folder))

for sub in os.listdir(DATA_DIR+"/"+folder+"/forge"):

f = DATA_DIR+"/"+folder+"/forge/" + sub

img = load_img(f,color_mode=mode, target_size=(150,150))

features.append(img_to_array(img))

features_dict[sub] = (img, 0)

features_forged.append(img)

if verbose:

print ("Adding {} with label 0".format(f))

labels.append(0) # forged

# real images

for sub in os.listdir(DATA_DIR+"/"+folder+"/real"):

f = DATA_DIR+"/"+folder+"/real/" + sub

img = load_img(f,color_mode=mode, target_size=(150,150))

features.append(img_to_array(img))

features_dict[sub] = (img, 1)

features_real.append(img)

if verbose:

print ("Adding {} with label 1".format(f))

labels.append(1) # real



features = np.array(features)

labels = np.array(labels)



x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=test_size, random_state=42)

x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.25, random_state=42)



print ("Generated data.")

return features, labels,features_forged, features_real,features_dict,x_train, x_test, y_train, y_test, x_val, y_val def convert_label_to_text(label=0):

"""

Convert label into text



Arguments:

label: int

Returns:

str: The mapping

"""

return "Forged" if label == 0 else "Real" features, labels,features_forged, features_real, features_dict,x_train, x_test, y_train, y_test, x_val, y_val = load_data(verbose=False, load_grayscale=False)

Visualization of the data

The images are loaded with a target_size of (150,150,3).

A snapshot of the data loaded followed by the label. (1 represents real and 0 represents forged)

Approach #1: Similarity in images (signatures) using MSE and SSIM.

For this approach, we will compute the similarity between images by using MSE (Mean Squared Error) or SSIM(Structural similarity). As you can see the formulas are pretty straightforward and fortunately Scikit-Learn provides an implementation for SSIM.

def mse(A, B):

"""

Computes Mean Squared Error between two images. (A and B)



Arguments:

A: numpy array

B: numpy array

Returns:

err: float

"""



# sigma(1, n-1)(a-b)^2)

err = np.sum((A - B) ** 2)



# mean of the sum (r,c) => total elements: r*c

err /= float(A.shape[0] * B.shape[1])



return err def ssim(A, B):

"""

Computes SSIM between two images.



Arguments:

A: numpy array

B: numpy array



Returns:

score: float

"""



return structural_similarity(A, B)

Now let us take two images from the same person, one of them is real and the other is a fake.

First Image

Second Image

Results for MSE and SSIM

As you can see, MSE Error does not have a fixed bound whereas SSIM has a fixed bound between -1 and 1. Lower MSE represents Similar images whereas lower SSIM represents Similar images.

Approach #2: Building a classifier using CNNs that can detect forged or real signatures.

With this approach, we will try to come up with a classifier (using CNNs) to detect forged or real signatures.

As CNN's are known to detect intricate features among images, we will experiment with this classifier.

We are bound to encounter with overfitting as we do not have enough data.

We will probably use Image Augmentation to generate more training data.

Our Model Architecture

On training our model, we are bound to encounter overfitting and after applying techniques to overcome the problem, the model did not improve.

Model Training

Our Model’s loss

Approach #2.1: Transfer Learning using Inception

To improve our model we will use transfer learning and fine-tune the model for this particular problem.

The InceptionV3 Model

For this approach, we will load pre-trained weights and add a classification head at the top to cater to this problem.

# loading Inception

model2 = tf.keras.applications.InceptionV3(include_top=False, input_shape=(150,150,3)) # freezing layers

for layer in model2.layers:

layer.trainable=False # getting mixed7 layer

l = model2.get_layer("mixed7") x = tf.keras.layers.Flatten()(l.output)

x = tf.keras.layers.Dense(1024, activation='relu')(x)

x = tf.keras.layers.Dropout(.5)(x)

x = tf.keras.layers.Dense(1, activation='sigmoid')(x)

net = tf.keras.Model(model2.input, x) net.compile(optimizer='adam', loss=tf.keras.losses.binary_crossentropy, metrics=['acc']) h2 = net.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=5)

Our Model Training

Model’s loss

Model’s accuracy

These two approaches show that if we use transfer learning, we get much better results than using a plain CNN model.

Keep in mind, these approaches do not learn the similarity function but these focus on the classifying whether the image is forged or real.

There are still many ways we can improve our model, one is by augmenting data.

Approach #3: Siamese networks for image similarity

With our third approach, we will try to learn the similarity function. We will use something called Siamese networks (due to the nature of our data i.e fewer training examples).

In this approach, we will use Siamese networks to learn the similarity function. Siamese means ‘twins’ and the biggest difference between normal NNs is that these networks try to learn the similarity function instead of trying to classify (fitting the function).

We first create a common feature vector for our images. We will pass two images (positive and negative) and use a contrastive loss function (Distance metric (L1 distance)) and in the end, we squash the output between 1 and 0 (sigmoid) to get the final result.

Siamese network (Image from [1])

Our Feature Vector Model

# creating the siamese network

im_a = tf.keras.layers.Input(shape=(150,150,3))

im_b = tf.keras.layers.Input(shape=(150,150,3)) encoded_a = feature_vector(im_a)

encoded_b = feature_vector(im_b) combined = tf.keras.layers.concatenate([encoded_a, encoded_b])

combine = tf.keras.layers.BatchNormalization()(combined)

combined = tf.keras.layers.Dense(4, activation = 'linear')(combined)

combined = tf.keras.layers.BatchNormalization()(combined)

combined = tf.keras.layers.Activation('relu')(combined)

combined = tf.keras.layers.Dense(1, activation = 'sigmoid')(combined) sm = tf.keras.Model(inputs=[im_a, im_b], outputs=[combined])

sm.summary()

Our complete siamese network

Dataset Generation

To generate the required dataset, we will try two approaches. First, we will generate data on the basis of labels. If two images have the same label (1 or 0), then they are similar. We will generate data in pairs in the form (im_a, im_b, label). Second, we will generate data on the basis of a person's number. According to the dataset, 02104021.png represents the signature produced by person 21 (i.e real).

Data generation Approach #1:

Here we are assuming similarity on the basis of labels. If two images have the same label (i.e 1 or 0) then they are similar.

def generate_data_first_approach(features, labels, test_size=0.25):

"""

Generate data in pairs according to labels.

Arguments:

features: numpy

labels: numpy

"""

im_a = [] # images a

im_b = [] # images b

pair_labels = []

for i in range(0, len(features)-1):

j = i + 1

if labels[i] == labels[j]:

im_a.append(features[i])

im_b.append(features[j])

pair_labels.append(1) # similar

else:

im_a.append(features[i])

im_b.append(features[j])

pair_labels.append(0) # not similar



pairs = np.stack([im_a, im_b], axis=1)

pair_labels = np.array(pair_labels)

x_train, x_test, y_train, y_test = train_test_split(pairs, pair_labels, test_size=test_size, random_state=42)

x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.25, random_state=42)

return x_train, y_train, x_test, y_test, x_val, y_val, pairs, pair_labels x_train, y_train, x_test, y_test, x_val, y_val, pairs, pair_labels = generate_data_first_approach(features, labels) # show data

plt.imshow(pairs[:,0][0]/255.)

plt.show()

plt.imshow(pairs[:,1][0]/255.)

plt.show()

print("Label: ",pair_labels[0])

Preview of our dataset

Training the dataset with Dataset Generation #1

Now we will train the network. Due to computational limitations, we only train the model on a single epoch.

# x_train[:,0] => axis=1 (all 150,150,3) x_train[:,1] => axis=1 (second column)

sm.fit([x_train[:,0], x_train[:,1]], y_train, validation_data=([x_val[:,0],x_val[:,1]], y_val),epochs=1)

Siamese Network’s result

The metric is calculating the L1-Distance (MAE) between the y_hat and y.

between the y_hat and y. Due to computation limitations, we only train it for one epoch

This represents a very simple siamese network capable of learning the similarity function.

Data Generation Approach #2

In this approach, we try to set up a dataset where we cross multiply each signature with other number signature.

The inputs and the outputs must be the same size.

def generate_data(person_number="001"):

x = list(features_dict.keys())

im_r = []

im_f = []

labels = [] # represents 1 if signature is real else 0

for i in x:

if i.startswith(person_number):

if i.endswith("{}.png".format(person_number)):

im_r.append(i)

labels.append(1)

else:

im_f.append(i)

labels.append(0)

return im_r, im_f, labels def generate_dataset_approach_two(size=100, test_size=0.25):

"""

Generate data using the second approach.

Remember input and output must be the same size!



Arguments:

features: numpy array

labels: numpy array

size: the target size (length of the array)

Returns:

x_train, y_train

"""

im_r = []

im_f = []

ls = [] ids = ["001","002","003",'004','005','006','007','008','009','010','011','012','013','014','015','016','017','018','019','020','021','022',

'023','024','025','026','027','028','029','030']



for i in ids:

imr, imf, labels = generate_data(i)



# similar batch

for i in imr:

for j in imr:

im_r.append(img_to_array(features_dict[i][0]))

im_f.append(img_to_array(features_dict[j][0]))

ls.append(1) # they are similar



# not similar batch

for k in imf:

for l in imf:

im_r.append(img_to_array(features_dict[k][0]))

im_f.append(img_to_array(features_dict[l][0]))

ls.append(0) # they are not similar



print(len(im_r), len(im_f))

pairs = np.stack([im_r, im_f], axis=1)

ls = np.array(ls)



x_train, x_test, y_train, y_test = train_test_split(pairs, ls, test_size=test_size, random_state=42)

x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.25, random_state=42)

return x_train, y_train, x_test, y_test, x_val, y_val, pairs, ls x_train, y_train, x_test, y_test, x_val, y_val, pairs, ls = generate_dataset_approach_two() # show data

plt.imshow(x_train[:,0][0]/255.)

plt.show()

plt.imshow(x_train[:,0][1]/255.)

print("Label: ",y_train[0])

Represents a forged signature. (0)

Training the Network with Dataset Generation #2

Training the network (Due to computational limitations, we train the model for a single epoch)

The biggest difference between dataset generation #1 and #2 is the way inputs are arranged. In dataset #1 we select random signatures according to their labels but in #2 we select signatures from the same person throughout.

Conclusion

To conclude, we present a plausible method to detect forged signatures using Siamese Networks and most importantly we show how we can easily train a Siamese network only a few training examples. We see how we can easily achieve great results using transfer learning.

References

[1] https://arxiv.org/pdf/1709.08761.pdf

Github: https://github.com/aaditkapoor/SigNet