DeepFuseOSV: online signature verification using hybrid feature fusion and depthwise separable convolution neural network architecture

IET BiometricsVolume 9, Issue 6 p. 259-268 Research ArticleFree Access DeepFuseOSV: online signature verification using hybrid feature fusion and depthwise separable convolution neural network architecture Chandra Sekhar Vorugunti, Corresponding Author Chandra Sekhar Vorugunti chandrasekhar.v@iiits.in Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorViswanath Pulabaigari, Viswanath Pulabaigari Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorPrerana Mukherjee, Prerana Mukherjee Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorAbhishek Sharma, Abhishek Sharma Electronics and Communication, Indian Institute of Information Technology-Naya Raipur, Chattisgarh, IndiaSearch for more papers by this author Chandra Sekhar Vorugunti, Corresponding Author Chandra Sekhar Vorugunti chandrasekhar.v@iiits.in Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorViswanath Pulabaigari, Viswanath Pulabaigari Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorPrerana Mukherjee, Prerana Mukherjee Computer Science and Engineering, Indian Institute of Information Technology-SriCity, Andhra Pradesh, IndiaSearch for more papers by this authorAbhishek Sharma, Abhishek Sharma Electronics and Communication, Indian Institute of Information Technology-Naya Raipur, Chattisgarh, IndiaSearch for more papers by this author First published: 16 September 2020 https://doi.org/10.1049/iet-bmt.2020.0032Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Online signature verification (OSV) is a widely utilised technique in the medical, e-commerce and m-commerce applications to lawfully bind the user. These high-speed systems demand faster writer verification with a limited amount of information along with restrictions on training and storage cost. This study makes two major contributions: (i) A competent feature fusion technique in which traditional statistical-based features are fused with deep representations from a convolutional auto-encoder; and (ii) a hybrid architecture combining depth-wise separable convolution neural network (DWSCNN) and long short term memory (LSTM) network delivering state-of-the-art performance for OSV is proposed. DWSCNN is utilised for extracting deep feature representations and LSTM is competent in learning long term dependencies of stroke points of a signature. This hybrid combination accomplishes better classification accuracy (lower error rates) even with one-shot learning, i.e. achieving higher classification accuracies with only one training signature sample per user. The authors have extensively evaluated their model using three widely used datasets MCYT-100, SVC and SUSIG. These exhaustive experimental studies confirm that the DeepFuseOSV framework results in the state-of-the-art outcome by achieving an equal error rate (EER) of 13.26, 2.58, 0.07% in Skilled 1, Skilled 10 and Random 10 categories of MCYT-100, respectively, 7.71% in Skilled 1 category of SVC, 1.70% in Random 1 category of SUSIG. 1 Introduction The traditional writer verification techniques like passwords, patterns, personal identification numbers etc. are prone to risks like stealing or forgetting them. Hence, online signature verification (OSV) is widely used in financial, e-commerce and m-commerce applications to legally bind an individual by detecting the forgery of online signatures [1-3]. As presented in Fig. 1, online signature encompasses a combination of static features (x, y co-ordinates) and dynamic features (pressure, azimuthal angle etc.) which makes them more efficient in representing writer characteristics compared to offline signatures [1, 4, 5]. Fig. 1Open in figure viewerPowerPoint Example online signature OSV is a promising and stimulating area of research in the field of handwritten text recognition using artificial intelligence. In the literature, several studies on OSV model/frameworks exist which can be approximately characterised into (i) local feature-based; (ii) global feature-based; (iii) function-based; and (iv) deep learning-based [6]. In local feature-based technique, the features are captured at each stroke point i.e. x, y co-ordinates, pressure, azimuthal angle, the tilt angle of the device etc. are used to classify a test signature. In global features-based OSV, the features are extracted by considering the whole signature, e.g.; number of strokes/lognormal, the standard deviation of x -coordinate etc. The feature-based (local/ global) OSV frameworks learn intra-class variations of online signatures. In function-based OSV, the local features at each stroke point of a reference and test signature are compared. Based on the predefined threshold, the test signature is delegated as genuine or counterfeit. The other function-based OSV systems incorporate interval-valued classifier [7, 8], hidden Markov model (HMM) [4], dynamic time warping [1, 9-12], edit distance [13], neuro fuzzy [14], Gaussian mixture models [1], support vector machine [15] and neural machine [16], long short term memory (LSTM) [12, 13, 17-20] etc. The paper is organised in this fashion. In Section 2, we discuss the related work. Section 3 discusses about the contribution of our proposed work. Preliminaries are discussed in Section 4. Proposed 'DeepFuseOSV' framework and its basic constituents are discussed in Section 5. In Section 6, particulars of training, testing, experimental investigation of our framework with most recent and best in class systems are examined. Conclusions are specified in Section 7. 2 Related work Very few OSV systems are proposed utilising deep learning techniques. Recently Ruben et al. [21] structured a novel semi-supervised system for assessing the signature complexity utilising recurrent neural networks (RNNs). The proposed model is utilised to assign a complexity label to each signature. In our previous work [22], we defined a deep convolutional Siamese network-based OSV framework. In the case of one-shot learning, the model resulted in an EER of 21.84%. As Siamese network is a twin network, the main pitfall in Siamese networks compared to CNN-based OSV frameworks is: (i) requiring a large number of parameters to train the framework. (ii) For effective learning about the differences between genuine and forgery pairs, for each probe signature, Siamese networks demand a huge number of signature pairs, i.e. (genuine, genuine) and (genuine, forgery). This results in two main drawbacks, the higher training time of the network and the output from these networks cannot be fed to conventional classifiers. To overcome these pitfalls, in this work, we have adapted the combination of depth-wise separable convolution-based CNN (DWSCNN) plus LSTM framework. DWSCNN results in a reduced number of parameters and LSTM networks learn the inter and intra writer variations of signature sequences effectively. 2.1 Essential of one/few shot learning in OSV In the context of online signatures, it is impossible and difficult to obtain an adequate number of signature samples from the users. This can be credited to the sensitivity of applications with which the signatures are associated, e.g. m-banking and m-payment [12, 23, 24] etc. As depicted in Table 1, the maximum genuine and forgery signature samples available per user is 25 which is in the case of MCYT dataset. Therefore, the OSV system must fulfil the vital prerequisite: The system must learn intra and inter personal variants with few signature samples at least as one. Table 1. Complete particulars of datasets used in the experimental evaluation of proposed framework DataSet→ SVC SUSIG MCYT total number of users 40 94 100 features per signature 47 47 100 genuine samples per user 20 20 25 forgery samples per user 20 10 25 total real signatures 800 1880 2500 total of fake signatures 800 940 2500 Relatively very few works have been proposed with regards to OSV systems with one/few-shot learning, i.e. learning the writer dependent characteristics with one/few signature patterns. Galbally et al. [1] formulated a HMM-based OSV technique, in which plausible signature samples are produced from one real signature. On similar lines, Diaz et al. [25-27] designed an OSV model, in which, sigma-lognormal parameters are figured from a single real signature sample, and plausible samples are created by mimicking the writer hand movements. Very recently, Rousseeuw and Croux [28] proposed an OSV by formulating a system which simulates the writer's skeletal arm. A new set of features is computed positioned on the movement of the wrist, elbow and shoulder combination. The above few-shot learning-based OSV frameworks [1, 25-28] experiences the accompanying disadvantages: (i) These models are not widely investigated on multiple data sets. (ii) These models are not assessed on every single category, i.e. Skilled_01(S_01), Skilled_05 (S_05), Skilled_10 (S_10), …, S_(N −5), Random_01 (R_01), Random_05 (R_05), Random_10 (R_10), …, R_(N −5), where 'N' = number of genuine signature samples per user. In general, the above models are assessed on only S_01 or S_05 categories. In this context, in our exploratory work [29], we have given a first of its attempt to address the above pitfalls. Very limited and insufficient contribution in one/few-shot learning-based OSV motivated us to contribute to this work. To address the above difficulties and disadvantages of real-time OSV systems, we propose DeepFuseOSV that can realise a few-shot learning, including one-shot learning. 3 Major contributions of our work The novel contribution of this work is the use of hybrid feature fusion and hybrid LSTM, CNN architecture to achieve one/few-shot learning. A feature-level fusion is implemented, where in a representation feature vector is formed using handcrafted features and deep features returned by a convolution autoencoder. The resultant feature vector utilises the full advantage of both handcrafted and deep-learned features and resulted in the enhanced discriminating power of DeepFuseOSV to classify a test signature. We have effectively utilised a novel hybrid convolutional neural network which is a stacking of DWSCNN and LSTM layers. The hybrid set of features from step 1 is given as an input to the deep DWSCNN. The depth-wise separable layers learn a complete short-term spatial relationship and LSTM layers learn long term temporal dependencies for effective test signature classification from a lesser number of samples as minimum as one. To make the empirical justification of our findings, we have led a broad evaluation of our proposed OSV system by implementing experiments on the three most widely used datasets, i.e. MCYT-100, SVC and SUSIG. We have appraised the superiority of the proposed framework by comparing the proposed OSV framework with the modern and state of the art frameworks. It is found to exceed the recent state of the artworks, which will be discussed in subsequent sections. 4 Preliminaries 4.1 Long short-term memory To acquire the long-term dependencies between stroke successions of an online signature, the framework must be able to associate critical patterns such as peaks and valleys to comprehend the present stroke patterns. LSTM is a proficient model to control the information flow, by fixing the amount of time to store, delete the previous stroke patterns and correlate the stored information to the new signature patterns to classify the test signature, which is an advantage compared to other sequence models like RNNs, HMM etc. [3, 27]. Fig. 2a, portrays the variations of pressure profiles of the first three users of MCYT-100 dataset. It is clear that the genuine signature profiles have low inter writer variations and high intra writer variations. Fig. 3b illustrates the genuine and skilled forgery signatures of user-19 of MCYT-100 dataset. The last subfigure of Fig. 3b illustrates the plotting of pressure and azimuthal angle signals of genuine and skilled forgery signature. Both the genuine profiles are tightly overlapped with corresponding skilled forgery profiles, which reflects the real-time scenario. To classify a test signature, the framework must learn the dependencies/variations/structure of patterns of each profile end to end. Learning a subset of patterns is inadequate to classify a test signature. Hence, in this work, we have used LSTM layers to learn the long-term dependencies of signature strokes. LSTM units contain three gates and a cell. The functional equations of LSTM are given as follows [19, 20]: (1) (2) (3) (4) (5) (6) where sig is the Sigmoid function, 't' is the current time. represents the input at the current time , represents the output from the previous and current block, respectively. , , represent the input, output and forget gates, respectively. , represent the memory from previous and current blocks, respectively. , , , represent the weights and are biases of LSTM to be learned during training at the input, forget and output gates, respectively. tanh is the hyperbolic tangent function, represent element-wise multiplication and summation, respectively. Fig. 2Open in figure viewerPowerPoint The intra, inter writer variations and pressure profiles in online signatures (a) The intra and inter writer variations (dependencies) of pressure signals of different users, (b) The pressure (p) and azimuthal (A) angle signals of a genuine and skilled forgery signature of user-19 Fig. 3Open in figure viewerPowerPoint Overview of hybrid feature set which forms as input to our DeepFuseOSV framework 4.2 Intuition behind fusion of handcrafted and deep representative features The handcrafted features are computed based on certain statistical properties or pre-defined algorithm, which reflects the domain-specific characteristics of an input signature, e.g. number of pen ups, average velocity of a pen in the X -direction etc. In deep CNNs, the initial convolution layers extract low-level latent representations from the input signature. The lower level features extract local spatial relationships, which are quite comprehensive and independent of any definitive classification task. Based on the low-level latent representations, the middle layers of CNN compute intermediate features, representing the underlying contextual structure of the output [30]. The final layer extracts the higher-level features from intermediate features through complex non-linear transformation. The high level deep representational features are more sophisticated, through which it learns intra-class variability of an input signature. Hence, CNN learns the feature representations incrementally and low-level features form an essential part of learning for CNN. The representation ability of CNN increases with the increase of the depth of the layers [31]. As depicted in Fig. 3, the feature vector which is computed based on the statistical properties detailed in Table 2 represents the global characteristics of a signature. The global feature vector forms an input to both the clustering algorithm and convolutional auto-encoder (CAE). The clustering algorithm, outcomes the best domain-specific set of features as cluster heads. The CAE outcomes deep representative features which represent the spatial, local and temporal dependencies among the global features. The fusion of output features from the clustering algorithm (domain-specific) and the CAE (context-aware) compliment each other and increase the discriminating capability compared to individual sets of features. Table 2. Set of global features computed for each signature total duration of signature (X 1st−pendown −x min)/Δx local maxima in y number of pen ups (Y 1st−pendown −Y min)/Δy local maxima in x sign changes of dx /dt, dy /dt (X last−pendown −x max)/Δx (X max −X min)/X accquisitionrange average jerk (Y last−pendown −Y max)/Δy (max distance between two points)/(A min) standard deviation of vy /Δy (Acceleration rms a)/a max θ (initial direction). standard deviation of acceleration in 'x' and 'y' directions StdDev of x / Δx standard deviation of velocity in 'x' and 'y' directions. histogram based features StdDev of y /Δy θ (before last pen up). Complete set of features is available in [32]. 5 Proposed deepFuseOSV architecture 5.1 Writer dependent feature clustering Let represents 'p' signature samples of writer 'k', where 'k' ranges from 1 to T. 'T' means the total number of writers in the dataset. A signature is categorised as a set of 'm' global features = . Let represents 'm' feature vectors of writer 'k'. is a column vector representing the j th feature of signature samples of writer 'k'. 'p' represents the total number of signature samples per user. We have applied K -means clustering on a set of feature vectors to aggregate similar feature vectors. For each cluster, a cluster representative which best represents the entire set of features in a cluster is selected, where 'K' is set to 60% of the total number of features. Subject to MCYT, the feature vectors are grouped into '60' clusters and in case of SVC and SUSIG into '30' clusters. To pick the best percentage of clusters from the total set of features, we have led a careful experimental analysis by varying the percentage of the number of clusters from 20 to 80% of the total number of features and EER resulted with respect to Skilled 1 and Random 1 categories for each dataset is recorded. The percentage of clusters which brought about least EER is considered as final clustering value for experimentation process, i.e. 60%. 5.2 Computing the cluster representatives The statistical measures like median, mode and mean absolute difference etc. suffer from major pitfalls like less representative capabilities in the context of time series data like online signatures, impact of fluctuations of sampling, effective for symmetric distributions etc. Subsequently, to choose one feature as a cluster representative, we have chosen median of medians (MoM) which was proposed by Rousseeuw and Croux [28] (7) For each 'p', compute the median of , which outputs 'm' values. The median of these set of 'm' values provides the final value of MoM, (the factor c is a consistency factor, and its default value is 1.1926.). MoM indicates the typical distance between the features of a cluster. The computational complexity of MoM is . The user is advised to read [28] for further analysis of MoM. To choose the element which best qualifies as a representative of the corresponding cluster, for each feature, we have calculated MoM based on (7). The feature which has minimum MoM value is picked as a head of the cluster. 5.3 Input signature format As shown in Fig. 3, the input online signature is represented as a row vector of dimension 1 × 100 subject to MCYT and 1 × 47 subject to SVC and SUSIG dataset. The dimensions 100, 47 means the total number of global features for each signature. Subject to MCYT, the feature vectors are grouped into '60' clusters and in context of SVC and SUSIG into '30' clusters. A statistical dispersion metric MoM is utilised to compute the cluster representative for each cluster. Top 60% of features, i.e. (60, 30) are chosen depending on the above criteria. The same set of global features is fed into a CAE, which outputs the latent representations of size 80 for MCYT and 40 for {SUSIG, SVC} datasets. Out of final representations, top 50% of the features (40,17) are selected and combined with the set of cluster representatives which results a hybrid set of features of length (100,47) and these joint representations form an input to our proposed DWSCNN framework for classification. 5.4 Separable convolution layer As depicted in Fig. 4, our proposed DWSCNN comprise a group of six layers. The primary layers are depth wise separable convolutional and batch normalisation layers. As illustrated in Fig. 3, an online signature of dimension is given to the first DWS convolution layer. Each DWS convolution layer includes 64 filters of dimension which results in three feature maps of size 1 × 100. A 1 × 1 pointwise convolution operation is implemented on these feature maps, which yields a feature vector of dimension 100 × 64. Fig. 4Open in figure viewerPowerPoint Diagrammatic representation of separable 1D convolution based proposed OSV framework architecture To standardise the input to each layer with zero mean/unit variance, to balance out the framework learning process and for faster convergence, batch normalisation [23] activity is performed on the output feature vector of size 100 64 from the DWS convolution layer. A 1 × 1 pointwise convolution activity is actualised on the output feature vector, which yields a feature vector of dimension 100 × 64. Same set of operations is executed on the second DWS layer, which consists of 64 filters, each of dimension 1 3 and yields a feature vector of measurement 100 64. The deep features from the DWS layers better capture the short-term dependencies of the signature stroke points, these deep features representing short term feature dependencies are passed as input to the LSTM layer. The LSTM layer consists of 64 units and yields a feature vector of dimension 100 64. The deep representation features are passed to the fully connected layers for classification. As presented in Table 3, the depth wise separable 1D convolution requires 2.03% reduced parameters contrasted to a standard 1D convolution operation. The reduced parameter count leads to elevated representational efficiency and faster framework learning. As a best practice [33], a dropout of 40%, 50% is practised at both the DWS convolutional layers. Table 3. Assessing the amount of operations and weights for standard and separable convolution operations Convolution layer type Standard Separable total operations W × K × C in × C out W × K × C in + W × C in × C out total weights K × C in × C out K × C in + C in × C out reduction factor (2.03%) 1/ C in + 1/(N 2) 1/ C in + /(N 2) 248,066 24,3121 C in, C out = Number of input and output channels, respectively, W = width of kernel 'K'. N = Number of kernels. 5.5 Fully connected network The deep features of dimension 100 64 captured by the DWSCNN and LSTM layer are reshaped into 1 6400 by the flatten layer. The deep feature representations pass on as a contribution to the multilayer perceptron (MLP), which acts as a classifier to classify the test signature. The MLP includes an input layer, a hidden layer and SoftMax layer for final classification. The first fully connected layer consists of 64 neurons and yields a vector of dimension 1 64, which is moved as an input to the hidden layer, which contains 128 neurons. The hidden layer yields a feature vector of dimension 1 128. Batch normalisation and dropout are applied on the output from the hidden layer which brings about a high-level feature vector of measurement 1 128, which is given to the sigmoid layer of size 2, which finally classifies the signature as real or fake. A dropout of 50%, 40% is executed at each fully connected layer, respectively. To reproduce the results, we have shown all the parameters and their values in Table 4. We have set 'ReLU' as an activation function in convolution and hidden layers and sigmoid function in classification layer. The above final parameters of the framework were chosen dependent on the training and validation accuracies of K -fold cross validation. Table 4. Overview of the constituting building blocks of proposed framework Layer Size Parameters separable convolution 1D + Batch normalisation + DropOut 64 × 1 × 3 {kernel_initialiser, bias_initialiser, depthwise_initialiser} = 'random_uniform', pad = 'same', activation = 'relu', 0.4 separableConvolution + BatchNormalisation + DropOut 64 × 1 × 3 {kernel_initialiser, bias_initialiser, depthwise_initialiser} = 'random_uniform', pad = 'same', activation = 'relu', 0.5 LSTM layer 64 units recurrent_dropout = 0.2, return_sequences = True fully connected + BatchNormalisation + DropOut 64 {kernel_initialiser, bias_initialiser} = 'random_uniform', pad = 'same', activation = 'relu', 0.5 fully connected + BatchNormalisation + DropOut 64 {kernel_initialiser, bias_initialiser} = 'random_uniform', pad = 'same', activation = 'relu', 0.4 6 Experimentation and results 6.1 Parameter settings Attaining one-shot learning is a challenging requirement for OSV framework. Accordingly, to select the best combination of parameters for the proposed OSV framework, we apply 10-fold cross-validation to appraise the proposed OSV framework. To set the parameters, we have assessed the proposed framework on the Skilled_01 category of MCYT-100 dataset, in which the dataset is split into a training set and test set. The train set consists of samples of 90% of users and test set with 10% of users. We sample the training set 10 times by random sampling with replacement. The intuition behind selecting MCYT-100 dataset is that it consists of a greater number of signature samples per user compared to SVC and SUSIG datasets. The parameters to set in our framework are: the number of separable convolution layers (SL) = {2, 3, 4}, number of kernels in each separable convolution layer (K) = {16, 32, 64}, kernel size (KS) = {3, 5}, number of dense layers (DL) = {2, 3}, number of nodes in each dense layer (DN) = {16, 32, 64, 128}, activation function (A) = {tanh, Relu}, optimiser (O) = {Adam, SGD}, the initialisation values for kernel and bias = {random_uniform, Glorot_uniform}. Recent research by the authors of [19, 34] summarises that, A = 'Relu', optimiser = 'Adam', kernel initialiser = 'random uniform' is the optimal values. A K -fold cross-validation (K = 10) is performed and the classification accuracy outcome of the proposed framework in ten cases is computed and the average is noted. The same technique is followed for all possible combination of parameters. The combination of parameters resulted in the least EER is considered as final. The outcome of parameter analysis is depicted in Fig. 5. In our case, as depicted in Fig. 5c), the combination of parameters {SL = 2, K = 64, KS = 3, DL = 2, DN = 64, LSTM nodes L = 64, LR = 0.004, optimiser = Adam} resulted in least EER value. Hence, we have performed a set of experiments using this set of parameters. Fig. 5Open in figure viewerPowerPoint The EER outcome of the proposed framework by fixing the number of separable convolution layers and varying other parameters of the framework 6.2 Experimental analysis On selecting the optimal set of parameters, to appraise the proposed OSV framework, we have performed a wide set of experiments which covers every single possible category of experimentation using three broadly utilised publicly accessible datasets, i.e. MCYT-100 [1, 2, 4, 5, 35], SVC – Task 2 [24, 36, 37], SUSIG [3, 26, 27]. Recently Ruben et al. [20] developed a novel largest online signature dataset named 'DeepSignDB', which is a collection of signature samples from widely used datasets like MCYT-330, BiosecurID, Biosecure DS-2, e-BioSignDS1 and e-BioSignDS2. 'DeepSignDB' consists of 70K signatures of 1526 users. Having the largest number of signature samples, 'DeepSignDB' address the challenge of data scarcity in developing novel deep learning frameworks for OSV. Even though it is the largest online signature database, very few OSV frameworks are experimented with 'DeepSignDB'. Hence, we have chosen the most widely used publicly available datasets, i.e. MCYT-100, SVC and SUSIG. We have executed our experiments on Nvidia, GForce RTX 2080 4×12 390 GB GPU. We have adopted standard procedures as described in [16, 23, 38] to split the train and test signature samples for various categories and datasets. A concise depiction of each dataset, related to a total count of users, the total number of forgeries and genuine signature samples available for each user, number of global features per signature etc. are listed in Table 1. Similar to the latest LSTM-based OSV frameworks [12, 13, 17-20, 39-41], to evaluate our framework thoroughly, we have considered five categories of experimentation, i.e. S_01, S_05, S_10, S_15 and S_20 in skilled forgery category and R_01, R_05, R_10, R_15 and R_20 in the random forgery category. If the dataset consists of a total 'T' number of users, each user with 'G' genuine and 'F' number of forgery signature samples, in case of skilled forgery, i.e. S_N, where N = 01, 05, 10, …, (G −5), the framework is trained with 'N' genuine, 'N' forgery signature samples and tested with '(G −N)' genuine, '(F −N)' forgery samples. In case of random forgery, i.e. R_N, where N = 01, 05, 10, …, (G −5), the framework is trained with 'N' genuine and a set of 'T −1' randomly selected one genuine signature sample per user as forgery samples a