# Neural Network-based System for Automatic Passport Stamp Classification

Wala Zaaboub, Lotfi Tlig, Mounir Sayadi, Basel Solaiman

► To cite this version:

Wala Zaaboub, Lotfi Tlig, Mounir Sayadi, Basel Solaiman. Neural Network-based System for Automatic Passport Stamp Classification. Information Technology And Control, 2020, 49 (4), pp.583-607. 10.5755/j01.itc.49.4.25919 . hal-03101199

HAL Id: hal-03101199

https://imt-atlantique.hal.science/hal-03101199

Submitted on 27 Jan 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
HOW TO CITE: Zaaboub, W., Tlig, L., Sayadi, M., Solaiman, B. (2020). Neural Network-based System for Automatic Passport Stamp Classification. *Information Technology and Control*, 49(4), 583-607. https://doi.org/10.5755/j01.itc.49.4.25919

# Neural Network-based System for Automatic Passport Stamp Classification

**Wala Zaaboub, Lotfi Tlig and Mounir Sayadi**

Laboratory SIME, ENSIT, University of Tunis, 1008, Tunis, Tunisia; e-mail: wala.zaaboub@enis.u-sfax.tn

**Basel Solaiman**

ITI, IMT-Atlantique, Technopôle Brest, Plouzané, France; e-mail: basel.solaiman@imt-atlantique.fr

Corresponding author: wala.zaaboub@enis.u-sfax.tn

The international tourism growth forces governments to make a big effort to improve the security of national borders. Protecting the borders from illegal immigrants and simplifying border checkpoints for law-abiding citizens and visitors is a delicate compromise. In the era of speed, it is indispensable to analyze passport pages by an intelligent application that recognize and classify stamps of travel documents in order to ensure faster, safer and more efficient stamp controlling. This paper proposes a model of such a system based on artificial neural network. The major contribution of this paper is about its topic related to passport stamps not yet addressed in the literature of object detection and recognition. As main aim, we proposed a framework that performs detection and classification to assist the border control. To the best of our knowledge, this is the first classification method for passport stamps. The originality of the proposed system based on low-cost neural network concerns several axes; the robustness in unconventional contexts, the high speed compared to other techniques such as the convolutional neural network, the low computational complexity with the help of using a classic classifier, the simplicity using intelligent tools guaranteeing the efficiency explained by promising accuracies with maximum accuracy of 0.945, and the high reliability explained by other classification metrics such as precision, recall and F1-score.

**KEYWORDS:** Multi-layer perceptron, Texture analysis, Pattern recognition, image classification, Artificial intelligence.
# 1. Introduction

International tourism is always on the rise; 1.5 billion tourist arrivals were recorded in 2019, globally. A rise of 6% over the previous year, according to the World Tourism Organization (UN TWO). This trend is positive when it is about tourists, business-persons, and regular immigrants. However, unwanted visitors can also use these same routes used for simple and practical travel: terrorists, refugees without visas and illegal migrants. This situation places governments around the world in the delicate task of simplifying border checkpoints for law-abiding citizens and visitors, while protecting their borders from illegal immigrants.

In order to effectively guarantee the safekeeping of the entry point, the European Union for example has introduced compulsory stamping of travel documents (passports) of third-country nationals when crossing the borders. Before 2003, there were different practices in member states of the Schengen zone that did not allow effective control through unsystematic stamping.

In fact, the border police ensure that the passenger passport is stamped when he enters and leaves the country. The officer also checks the dates of entry and exit to know if the passenger is obeying the low by respecting the period of stay.

Without an entry stamp, the passport holder may be suspected of having exceeded the length of his stay when he tries to leave the country. The absence of an entry stamp in the travel documents constitutes a presumption of irregularity of stay on the country territory. Without an exit stamp, the entry may be refused the next time the passenger wants to enter the country since he can be expected to have exceeded the allowed period of his previous visit.

In this paper, we will introduce a system that helps the border control officers by doing these stamps checking operations automatically. Thus, our task is to detect and extract stamps from each passport page and then classify them into three classes: Entry local stamp, exit local stamp and stamp of other countries.

In addition, crossing the border without stamping the passport can cause real problems when leaving the country. In this case, the passenger risks being considered as a clandestine immigrant, which may be

worthwhile for him to be detained, to pay a large fine and to be expelled. Even if the passenger leaves definitively the territory, he risks of having troubles. At worst, he will risk being detained for non-compliance with immigration rules.

The implementation of the state of emergency in France in November 2015 following the attacks and the reintroduction of systematic controls on all flights from or to countries in the Schengen area have considerably slowed the operations of the border police at airports. The situation has been further aggravated by the reinforcement by the European Union of the controls at the external borders of the Schengen zone. This is what it leads to a proliferation of queues, longer waiting times and delays in flights.

With a valid bio-metric passport (a passport equipped with a microchip, which contains the bio-metric information used to authenticate the identity of the passport holder) we have a time saving to pass the controls even faster. Nevertheless, notice that the chip of the e-passport does not have the information about the entry/exit stamps.

The objective of controlling the stamps must find a concrete and rapid application in the facts. That is the purpose of this paper. Our motivation is the need of automatic stamp control that is able to analyze the passport paper, detect and classify different stamps in it. Therefore, the proposed approach helps to ensure a flexible system that supports faster, safer and more efficient border control processes. An assistant of border control officers at land, air and sea borders.

The literature survey shows that there are few pieces of work on stamp detection and recognition in document images [4]. Existing methods employ different features of digital images, they explore stamp properties; shape [41, 49], color [11, 28], and textual information [31, 43]. A recent study shows that few stamp detection and classification methods have been proposed. They either work on color or shape stamps. There is no general approach for the diversity of stamps.

Reading passport using a dedicated passport reading system, MRZ (Machine Readable Zone), allows only the character recognition [22, 24, 36]. In [24] the author proposed an effective recognition algorithm of
the passport MRZ information using a combined neu-
ral network recognizer of Convolutional Neural Net-
work and Artificial Neural Network. However, these
works are not able to recognize the passport stamp.

Extracting key information from a photo entered into
the system and returning them to a specialist of the
domain is quite a difficult task, because the system of
information extraction should know what to look for,
how to search, as well as remove unwanted elements
in the image. For that purpose, the system requires an
artificial intelligence, like the use of neural networks
and intelligent things [37, 38].

The aim of this paper is to have an intelligent system
able to extract the interesting information about
the different stamps existing in each passport page
and return them to the border police to facilitate the
check operation. It is a solution for detecting and
recognizing passport stamps of different colors and
shapes (Figure 1). In fact, we do not rely on color
properties only like in [28] and we are not limited to
specific shapes like in [41]. In this paper, we will de-
velop a texture analysis method for stamp detection
and extraction and a lower-cost neural solution for
stamp classification. The rest of paper is organized
as follows: Section 2 presents the related works.
Section 3 describes the image processing of passport
pages. The proposed stamp extraction approach is
given in Section 4. In Section 5, the proposed stamp
classification approach is described in detail. Sec-
tion 6 explains how the experiments are performed

and presents the results. Section 7 concludes the pa-
per and gives perspectives for future work.

## 2. Related Works

Building literature for the subject area is highly re-
quired to compare with the proposed system, even
this paper tackle different issue. This section high-
lights how our study aims to stand out within the
studies referred and thereby establish the original-
ity. We referred to prior research works published
in our field that have resemblance or are related to
our study. In a wider context, stamp extraction and
classification is considered as an object detection
and recognition problem. There are many proposals
for such problems in literature covering different
sectors.

2.1. Object Detection

In literature, there are few research works on seal
and stamp detection in scanned document. Seal
properties such as; color, shape, and textual informa-
tion, have been explored for detection methods. The
seal detection techniques can be categorized into
three categories; color- information-based methods,
Shape/graphical- based methods and textual-infor-
mation-based methods.

Color information can help in distinguishing graphi-
cal objects from other components of documents. As

<figure><img src="image_2.png" /><figcaption>**Figure 1**

Examples of entry/exit stamps of different countries</figcaption></figure>
color is generally used in seal or stamp imprint, color segmentation methods are used for seal detection.

In [11] and [28], a segmentation based on scalar color feature is used for graphical region detection and then, candidate graphical regions are obtained and classified based on several features. Logos and stamps are detected depending on color-related features and geometrical features from each processing unit such as pixel density, dimension and position of bounding boxes.

A prior knowledge about seal boundary shape (circle, rectangle or ellipse) can help in localizing seals in documents, like in [49]. Using basic-shape detectors, authors detect elliptic or circular stamps from degraded documents. In this category, methods are limited to specific shape, like in [41].

Textual information can help in seal detection on document image. A strategy based on frequency selective filtering is proposed for logo and stamp detection in [31]. Textual contents in the document are main source of high spatial frequency. Accordingly, the high frequency filtering helps to retain logo and stamp regions, and suppress the visual perception of the textual symbols. Then, the combination of the median filtering and Gaussian blurring operation helps to suppress noise because of textual residuals, in order to localize graphical regions. Singh et al. [43] use textual information for seal detection purpose. Here, the detection is based on the concept of Generalized Hough Transform.

A similar problem is the logo detection in scanned documents. The logo detection techniques can be categorized into three categories; connected-component-based methods, local descriptor-based methods and block-based methods.

In the connected-component-based methods, a list of connected-components (CC) is extracted from the document, and then each CC is characterized by computing different features like height, width, aspect ratio, area and density. These features are used in a classification step to discriminate between logo and non-logo components. Naganjaneyulu et al. [30] use 12 gray level co-occurrence matrix features extracted from CC and a pre-trained Support Vector Machine classifier (SVM) is used to detect logos. Veershetty et al. [44] eliminate noise and symbols using a morphological

operation, and extract each logo from the document using the CC rule. Dixit et al. [13] find CCs and perform area-based thresholding to get pool of possible logo candidates. This method is based on the assumption that logo region has a higher spatial density than non-logo region, which is not always true. Noting that the connected component-based techniques have obvious shortcomings. They are generally sensitive to degradation, noise, skew and scale variation. They merge logo and non-logo regions together when the gap between them is small. Moreover, regarding its higher complexity, these techniques still very consuming time.

In local-descriptors-based methods, local detectors are calculated at the pixel level in order to extract a set of key-points. Then, these key-points are exploited by local descriptors to detect logo regions. Chatbri et al. [9] introduce a shape descriptor that extracts key-points and automatically detects the salient ones by filtering out redundant and sensitive key-points. Noting that the local-descriptors-based techniques are able to treat well defined shapes and transformations due to brightness, scale and rotation invariance of descriptors. For color-scanned documents, methods that follow this approach can be well adapted to the problem, but most of all are invariant to image transformations. Moreover, these pixel-based methods tend to be sensitive to the noise and they are time consuming. In logo retrieval task, Wiggers et al. [46] avoid the manual feature extraction by learning the image representation using a Convolutional Neural Network (CNN). Authors use a pre-trained CNN model to cope with the lack of training data, which is fine-tuned to achieve a discriminant representation of the logo.

In region-based methods, documents zones (or blocks) are obtained by a page segmentation step on the document, and then a classification step is applied to classify those blocks into logo and non-logo regions. In [2] and [3] authors use a decision tree with specific probability features computed for patches. Noting that the region-based techniques rely heavily on the accuracy of the segmentation step, which can be difficult in less structured documents with complex layout. In [8] Chanda et al. use the advanced algorithm YOLOv3 (or You Only Look Once) to detect
logo and seal in historical documents. Here, there is no need for any preprocessing step like binarization or layout- analysis.

In a wider context, passport stamp detection is considered as an object detection. Here, some objects are aligned to concepts, such as animals, people or text. Other objects are related to texture and material properties, such as dot arrays and specular reflections. Recently, object detection can be performed using deep convolutional networks. A set of algorithms like R-CNN in [17], Fast-R-CNN in [18], Faster-R-CNN in [40], their advanced counterpart Yolo and its most sophisticated successor YOLOv3 in [39], are able to identify the different objects in the image and their boundaries. YOLO is an object detection algorithm different from the region-based algorithms R-CNN, Fast-R-CNN, and Faster- R-CNN. Those three algorithms look only into parts of the image having high probabilities of containing objects.

## 2.2. Object Classification

In the state of the art, there are few research works on classification and recognition of seals and stamps in scanned document. In [45] authors explore the deep learning technology to automatically recognize Chinese seal, which is an important part of the Traditional Chinese painting and calligraphy. Authors propose new Siamese Network with Multi- task Learning to recognize the different seals and report the relevant information in real time. Forczmański et al. [14] detect and classify stamps in scanned documents. Authors apply color segmentation in order to find potential stamps. They employ methods aimed at detecting objects of specific shapes; circle, lines and other shapes using heuristics. Isolated objects are extracted and classified using a set of shape descriptors. In [15] the processed stamp is represented by a shape descriptor using Point Distance Histogram and classified using hand-drawn templates and ideal shapes. Petej et al. [35] explore two machine-learning algorithms for comparison purpose; the SVM classifier using OpenSURF features and the Random ferns classifier.

A similar problem is the logo classification in scanned documents. Gallego et al. [16] present a method for logo multi-label classification and retrieval. Authors train CNN to classify logos from the European

Union TradeMark (EUTM) dataset according to their shapes, colors, sectors and figurative designs. Guru et al. [20] present a classification method based on K-Means clustering algorithm using three global features; texture, color and shape. In [34], authors design an efficient CNN for classifying TV logo with the help of MSER algorithm.

For object recognition covering different sectors, Block et al. [6] propose an automatic detection and classification of mild and severe imprint defects in stamped metal parts using RetinaNet. It is an assistant to the automotive industry to improve the production line performance.

Žemgulys [47] proposes a technique for the recognition of hand gestures in general and sports referee signals in particular. Authors explore local binary pattern (LBP) features with SVM classifier, to recognize signals of basketball referee from recorded game videos.

Lam et al. [26] propose a deep-learning classifier for holograms of 3-D deformable handwritten numerals, based on a pair of CNNs.

From this literature review, we can deduce that seal detection/recognition has been explored, however such published methods do not focus on the stamp detection/ recognition problem in travel documents. Hence, this paper tries to address that unexplored area. To conclude this section, a comparison of the most recent state of the art methods is presented in Table 1.

Existing works do not pay attention to develop detection and classification system for passport stamps. The main originality of our paper is its ability to solve a trending issue. The control of travel documents by automatic passport stamp recognition is a challenging problem in the real world. This current problem is still new and research has not been carried out on the subject matter. So far, most studies on document object detection/recognition have focused on historical documents. In this study the focus is shifted to the contemporary issue. The originality of this work concerns also several axes such as the robustness in unconventional contexts, the low temporal and computational complexities, the efficiency and the simplicity of the proposed approach using intelligent tools.
<table><caption>**Table 1**
Comparison of the most recent related works</caption><thead><tr><td>Reference</td><td>Application</td><td>Method</td><td>Dataset</td><td>Accuracy (%)</td><td>Precision (%)</td></tr></thead><tbody><tr><td>[11]</td><td>Stamp segmentation</td><td>Color-based segmentation</td><td>StampVer dataset</td><td>-</td><td>84.59</td></tr><tr><td>[28]</td><td>Stamp segmentation</td><td>Color-based segmentation</td><td>Tobacco-800 dataset</td><td>-</td><td>83.8</td></tr><tr><td>[49]</td><td>Stamp detection</td><td>Shape-based detection</td><td>University of Maryland<br>Arabic document<br>databases</td><td>-</td><td>35.23</td></tr><tr><td>[31]</td><td>Stamp detection</td><td>frequency selective filtering</td><td>StampVer database</td><td>-</td><td>94.8</td></tr><tr><td>[43]</td><td>Seal detection</td><td>Generalized Hough<br>Transform</td><td>Seal dataset (80 seals)</td><td>-</td><td>-</td></tr><tr><td>[30]</td><td>Logo detection</td><td>CC-based detection</td><td>Tobacco-800 dataset</td><td>91.47</td><td>98.10</td></tr><tr><td>[44]</td><td>Logo detection</td><td>CC-based detection</td><td>Tobacco-800 dataset</td><td>-</td><td>87.80</td></tr><tr><td>[13]</td><td>Logo detection</td><td>CC-based detection</td><td>Tobacco-800 dataset</td><td>89.52</td><td>-</td></tr><tr><td>[46]</td><td>Logo detection</td><td>local-descriptors-<br>based detection</td><td>Tobacco-800 dataset</td><td>-</td><td>72</td></tr><tr><td>[3]</td><td>Logo detection</td><td>Block-based detection</td><td>Tobacco-800 dataset</td><td>91.50</td><td>75.25</td></tr><tr><td>[8]</td><td>Logo and seal detection</td><td>YOLOv3</td><td>Tobacco-800 dataset</td><td>-</td><td>89</td></tr><tr><td>[17]</td><td>Object detection</td><td>R-CNN</td><td>ILSVRC2013 detection<br>dataset</td><td>47.9</td><td>53.7</td></tr><tr><td>[18],</td><td>Object detection</td><td>Fast-R-CNN</td><td>PASCAL VOC 2012</td><td>-</td><td>65.7</td></tr><tr><td>[40]</td><td>Object detection</td><td>Faster-R-CNN</td><td>VOC 2007 + VOC 2012</td><td>-</td><td>70.4</td></tr><tr><td>[45]</td><td>Chinese seal<br>recognition</td><td>Siamese Network</td><td>Chinese seal database</td><td>95.46</td><td>96.04</td></tr><tr><td>[15]</td><td>Stamp recognition</td><td>Shape-based recognition</td><td>Collection of images<br>from the internet</td><td>-</td><td>81</td></tr><tr><td>[35]</td><td>Stamp classification</td><td>SVM classifier</td><td>Collection of stamps<br>from real documents</td><td>-</td><td>98</td></tr><tr><td>[16]</td><td>Multi-label logo<br>classification</td><td>CNN classifier</td><td>European Union<br>TradeMark (EUTM)<br>dataset</td><td>-</td><td>97.56</td></tr><tr><td>[20]</td><td>Logo classification</td><td>K-Means classifier</td><td>UoMLogo database</td><td>71.73</td><td>66.89</td></tr><tr><td>[34]</td><td>TV Logo classification</td><td>CNN classifier</td><td>Collection of tv logo<br>images</td><td>85</td><td>-</td></tr><tr><td>[6]</td><td>Classification of defects<br>in stamped metal<br>surfaces</td><td>RetinaNet</td><td>Collection of imprint<br>defect images</td><td>-</td><td>90.55</td></tr><tr><td>[47]</td><td>Recognition of sports<br>referee signals</td><td>SVM classifier</td><td>Collection of images<br>from basketball game<br>video</td><td>95.6</td><td>-</td></tr></tbody></table>
## 3. Image Processing for Passport Papers

To develop our project, we exploit the image processing techniques and the intelligent classifying systems. Our pre-processing phase for the passport paper contains three steps; conversion, resizing and filtering. Here, there is no need to skew since a passport page contains several stamps of different orientations.

The scanned passport paper is converted from RGB to gray-level image. Color can be useful information in the stamp recognition system, but our method does not depend on a specific color. For resizing, passport paper is scaled to a standard size. This operation ensures independence from the capturing device resolution; furthermore, followed operations will work with the assumption of a fixed image size. This resolution is fixed in such a way as to keep the details of the image and to have an acceptable calculation time. For filtering, we use the median filter that reduces noise while maintaining the image contours. This filtering eliminates outliers without being limited to averaging, which in most cases contaminates the neighboring values with this aberrant value and blurs the image. It respects the contrast, brightness and contours of the image, and it eliminates extreme values.

The median filter value $f(x,y)$ can be written as below (Equation (1)):

$$f(x,y)=\text{median}_{a,b \in S(x,y)}\{g(a,b)\}, \quad (1)$$

where $S(x,y)$ is the neighborhood space of image pixels according to their coordinates, and $g(a,b)$ represents the existing neighborhood pixels.

By reducing the filter size, the image smoothness can be lost, and by increasing the filter size, some image features can be lost and replaced with background values. Therefore, for noise reduction, it is better to use models with small radius of the neighborhood. The filter size parameter is set to 3×3, which is considered as the best value to have the fastest filter and the best segmentation result. As illustrated in Figure 2, a 5×5 filter gives a bad segmentation result counter to the 3×3 filter.

<caption>Figure 2</caption>

Choice influence of filter size parameter: (a) original image, (b) segmentation result on stamp/non-stamp with 5×5 filter, and (c) segmentation result on stamp/non-stamp with 3×3 filter

<figure><img src="image_3.png" /></figure>

# 4. Stamp Extraction from Passport Page

## 4.1. Segmentation Pre-processing: Contour Detection

For the detection of contours, we will use the Canny method [12] which allows a good object detection of weak contours with low error rate, and ensures a good object localization by the minimization of distances between predicted and real contours. The conservation of weak edges ensures the preservation of the textual information of the object of interest, which will be used as a key information in the next step of segmentation. This point justifies the use of the Canny method.
The first step of this method is the application of the Gaussian filter $G_σ$ (Equations (2) and (3)) that is used to smooth the image I in order to remove noise.

$$g(x,y)=G_{\sigma}(x,y) \cdot I(x,y) \qquad (2)$$

$$G_{\sigma}=\frac{1}{\sqrt{2\pi\sigma^{2}}} \exp \left(-\frac{x^{2}+y^{2}}{2\sigma^{2}}\right) \qquad (3)$$

where $σ$ is the standard deviation of the Gaussian distribution.

The gradient is calculated in horizontal and vertical directions by convolving the image with gradient mask. The gradients can be determined by using a Sobel filter as shown in Equations (4) and (5).

$$G_x = \begin{bmatrix} -1 & 0 & +1 \\ -1 & 0 & +1 \\ -1 & 0 & +1 \end{bmatrix} g \qquad (4)$$

$$G_y = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ -1 & +2 & +1 \end{bmatrix} g \qquad (5)$$

Then, magnitude $M(x,y)$ and direction $θ(x,y)$ that defines strength and orientation of each pixel (x,y) are calculated by the following Equations (6) and (7).

$$M(x,y)=\sqrt{G_{x}^{2}(x,y)+G_{y}^{2}(x,y)} \qquad (6)$$

$$\theta(x,y)=\arctan\left[G_{y}(x,y) / G_{x}(x,y)\right] \qquad (7)$$

The image magnitude $M$ produces results in blurred edges. Non Maximum Suppression is applied to convert thick edges into thin edges and is done by suppressing minima in the gradient image.

To enhance the result and determine potential edges, double thresholding is needed. Potential edges are calculated based on the gradient magnitude. Edges with edge strength below the lower threshold are disregarded, and edges with edge strength above the higher threshold are preserved.

To finalize the edge detection, weak edges that are not connected to strong edges should be suppressed, this is done by edge tracking algorithm.

We notice the need for contour detection as a pre-processing phase for stamp / non-stamp segmentation. Segmenting the detected contours gives a better

result by detecting the different stamps on a passport paper. After several tests, the higher threshold parameter is set to 0.3 and the lower threshold is 0.1, in order to have the best segmentation result. As presented in Figure 3, a higher threshold of 0.4 gives a bad segmentation result counter to 0.3 threshold.

<caption>Figure 3</caption>

Choice influence of contour threshold: (a) the original image, (b) contour detection result with threshold = 0.4, (c) its segmentation result, (d) contour detection result with threshold = 0.3 and (e) its segmentation result

<figure><img src="image_4.png" /><figcaption>(a)</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b)</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c)</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d)</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(e)</figcaption></figure>
Figure 4 illustrates the need for contour detection for stamp segmentation. Segmenting the detected contours gives a better result by detecting the different stamps on a passport paper (Figure 4 (c)).

Figure 4

Need for contour detection: (a) the original image,
(b) segmentation result of the original image and (c)
segmentation result of detected contours

<figure><img src="image_9.png" /></figure>

## 4.2. Passport Page Segmentation Based on Texture

We will exploit the texture in the document to detect and extract the different stamps from a stamped passport paper. Stamp zone is generally a compact area, counter to the travel document background. Thus, each document area has a specific texture measure. Here, the texture analysis boost to the image zoning level.

The problem of stamp detection on passport paper will be treated as a stamp / non-stamp segmentation problem.

Among the difficulties of this phase is that the background of this type of document is a bit special. Counter to the classic administrative document having homogeneous background (usually white), we can say that the background of a passport paper is textured, as shown in Figure 5.

Figure 5

Example of empty passport papers indicating the difficulty of stamp / background separation

<figure><img src="image_10.png" /></figure>

Another difficulty of the extraction phase is that stamps have different orientations in the document. Overlapping stamps can also make this step worse. In addition, the poor quality of the document that is due to the lack of stamp ink can make this step more difficult. Moreover, having other items on the passport paper is another difficulty. Figure 6 illustrates all these situations.

Figure 6

The difficulties of stamp extraction: stamps having different orientations in the top row, low quality stamps in the middle row, and other elements in a passport paper in the bottom row

<figure><img src="image_11.png" /></figure>

In this work, the discrimination ability of the texture-based segmentation is evaluated by a discrimination criterion, which is the Fisher score $F_s$ [19]. An evaluation of this criterion for many texture features allows us to argue our choice of the relevant feature
characterizing the different textures in a passport page. The characterization degree $F_S$ is given by the Equation (8):

$$F_S=\text{trace}\{S_{\text{intra}}^{-1}, S_{\text{inter}}\} \quad (8)$$

where $S_{\text{inter}}$ (Equation (9)) and $S_{\text{intra}}$ (Equation (10)) are the mean of the between-class dispersion matrices and the mean of the within-class dispersion matrices, respectively.

$$S_{\text{inter}} = \frac{1}{C} \sum_{k=1}^{C} \left( \underline{\mu}_{k} - \underline{\mu} \right) \left( \underline{\mu}_{k} - \underline{\mu} \right)^{\text{T}}, \quad (9)$$

$$S_{\text{intra}} = \frac{1}{C \times N_S} \sum_{k=1}^{C} \sum_{i=1}^{N_S} \left( \underline{x}_{k,i} - \underline{\mu}_{k} \right) \left( \underline{x}_{k,i} - \underline{\mu}_{k} \right). \quad (10)$$

With $C$ is the number of texture classes, $N_S$ is the number of samples randomly extracted from each texture class, $\underline{x}_{k,i}$ defined the $i$-th feature vector of the $k$-th texture class ($1 \le k \le C$ and $1 \le i \le N_S$), $\underline{\mu}_{k}$ is the mean of feature vectors of the $k$-th texture class, which is defined by the Equation (11).

$$\underline{\mu}_{k} = \frac{1}{N_S} \sum_{i=1}^{N_S} \underline{x}_{k,i} \quad (11)$$

Here $\underline{\mu}$ is the overall mean of the features vectors, calculated with the Equation (12).

$$\underline{\mu} = \frac{1}{C} \sum_{k=1}^{C} \underline{\mu}_{k} \quad (12)$$

The couple of texture features having the largest Fisher score are: « Mean » and « Hölder exponent », they are selected as the most discriminative for our case.

The multifractal texture feature (Hölder exponent) $\alpha$ is defined by the following Equation (13).

$$\alpha(\vec{x}) = \lim_{r \to 0} \frac{\log T_{\psi}^{r} \mu(\vec{x})}{\log r} \quad (13)$$

where $\vec{x}$ is a point of the image, $r$ is an expansion factor, $\psi$ is the wavelet, $\mu$ is the multi-fractal measure, and $T_{\psi}^{r} \mu$ is the wavelet projection.

Among wavelet functions adapted to the singularity analysis, the authors select the Lorentz wavelet (Equation (14)) to estimate the singularity exponents.

$$\psi(\vec{x}) = (1 + |\vec{x}|^2)^{-\gamma} \quad (14)$$

The feature « Mean », denoted $M$, is calculated on a sliding window ($W \times W$) of central pixel $I(x,y)$ from the original image $I$ with a size of $N_R \times N_C$, and is given by the Equation (15).

$$M = \frac{1}{N_R \times N_C} \sum_{i=-\frac{W-1}{2}}^{\frac{W-1}{2}} \sum_{j=-\frac{W-1}{2}}^{\frac{W-1}{2}} I(x+i, y+j) \quad (15)$$

Likewise, the « Mean » is calculated over a sliding window of the Hölder image, and assigned to the central pixel, to create the “image of texture feature” that will be used in the classification step.

We consider that the system of the fuzzy classification [32] based on a combination of luminous inten-

<figure><img src="image_12.png" /><figcaption>**Figure 7**

Texture-based segmentation of passport paper</figcaption></figure>
sity feature (Mean) with a multifractal texture feature (Hölder exponent) is a qualified process for stamp detection.

The proposed texture-based segmentation of passport page on stamp / non-stamp is presented in Figure 7. Instead of having the original image as input, we process the image of contours created in the previous step (Section 4.1).

It is important to notice that main advantages of this texture-based method are the insensitivity to shape, color, orientation and scale-variation of stamps.

## 4.3. Stamp Extraction

The detection of stamps is performed in two steps: (i) the detection of the outer contours of the segmented regions (stamp + non-stamp), then (ii) the elimination of each initial candidate stamp which is a non-stamp item in reality.

This elimination is based on the assumption that all the stamps have almost the same size in a passport paper. A contextual rule defining possible sizes of passport stamps ensures the detection in a rigorous way. Then, filling all the predicted stamp zones allows us to have a correct stamp / non-stamp mask. Finally, the extraction is done with the help of boxes enclosing the predicted stamp zones.

# 5. Passport Stamp Classification

In this investigation, a novel method based on a Multi-layer Perceptron artificial network was proposed to classify passport stamps into three categories: entry local stamp, exit local stamp, and foreign stamp.

## 5.1. Feed-Forward Neural Network Architecture

The artificial neural network, ANN imitates body neuron that is able to estimate most of the nonlinear functions with high precision. It learns and exhibits the capability for generalization beyond the training data. ANNs are being used as classifier tools in various fields such as medical diagnosis [5, 7], financial forecasting [29, 42] and signal and image classification [33, 48].

In the literature, there are many types of neural network architecture; we focus on the Multi-layer perceptron (MLP) [1, 27], that its basic model is shown in Figure 8. MLP architecture is a feed-forward artificial neural network model with multiple hidden layers. Expect the input layer, each neuron uses nonlinear activation function.

MLP network works on the basis of the following Equation (16).

$$y = \Gamma_n [W_n \Gamma_{n-1} [W_{n-1} \dots \Gamma_l [W_l x]] \dots] \quad (16)$$

where x is the input vector, y is the output vector, Wᵢ (i=1,2,..., n) denotes the weight matrix of the ith layer, and the diagonal operator Γᵢ[...] (i=1,2,..., n) is an activation function matrix of the ith layer.

<caption>Figure 8</caption>

Model of feedforward neural network with one hidden layer and one output layer

<figure><img src="image_13.png" /></figure>

## 5.2. MLP Neural Network Training

MLP neural network uses a back-propagation algorithm to train models. We used the Levenberg-Marquardt the standard back-propagation algorithm that is the fastest and most accurate for multi-layer neural networks [10, 21].

The learning is the estimation of the parameters (weight and bias) by minimizing a performance function. The training process works on the basis of the following Equation (17):

$$\Delta w = - \eta \frac{\partial E}{\partial w} \quad (17)$$
where $\Delta w$ is the amount of weight adjustment and $\eta$ is a learning rate that is fixed to 1e-3 after several tests. E define the performance function that is in this work the Mean Squared Error (MSE) defined by the Equation (18)

$$E = \frac{1}{k} \sum_{i=1}^{k} (t_i - y_i)^2 \qquad (18)$$

where k defined the number of output neurons, $t_i$ is the i-th targeted output and $y_i$ is the i-th predicted output.

The Levenberg-Marquardt optimization algorithm is based on gradient evaluation by retro-propagation. It improves the convergence speed of the learning algorithm or avoids it being stuck to an unfavorable local solution. It is designed to approach second-order training speed without the necessity to compute the Hessian matrix H that can be approximated as described in the Equation (19) and the gradient g can be calculated as shown in Equation (20).

$$H = J^T J, \qquad (19)$$

$$g = J^T e. \qquad (20)$$

where e is a vector of network errors, and J is the Jacobian matrix containing first derivatives of the network errors with respect to the weights and biases. Backpropagation is used to determine the Jacobian matrix J of performance with respect to the weight and bias variables X. Variables are adjusted according to Levenberg-Marquardt, as shown in Equation (21).

$$dX = -(J J^T mu)/J e \qquad (21)$$

where I is the identity matrix, and mu is the momentum parameter that is used to calculate the weight update to avoid the problem of local minimum. It is the case of a network stuck to local minimum and convergence does not occur. The value of mu is increased by an increase factor until the change above results in a reduced performance value. Then, the change is made to the network and mu is decreased by a decrease factor.

After several tests, we set 1e+1 and 1e-2 for, respectively, increase factor and decrease factor of mu, and an initial value of 1e-4, as presented in Table 2.

Training with the Levenberg-Marquardt algorithm supports training with validation and test vectors. It

<caption>**Table 2**</caption>

Neural network training parameters

<table><tr><td>Maximum number of epochs</td><td>1e+3</td></tr><tr><td>Maximum training time</td><td>infinite</td></tr><tr><td>Maximum value of momentum</td><td>1e+10</td></tr><tr><td>Optimal performance (error)</td><td>0</td></tr><tr><td>Minimum gradient value</td><td>1e-7</td></tr><tr><td>Maximum number of validation failures</td><td>6</td></tr><tr><td>Initial value of momentum</td><td>1e-4</td></tr><tr><td>Decrease factor of momentum</td><td>1e-2</td></tr><tr><td>Increase factor of momentum</td><td>1e+1</td></tr><tr><td>Learning rate</td><td>1e-3</td></tr></table>

can use a validation set to stop training if the network performance on this set remains the same for a specific number of iterations.

Usually, training stops when the maximum number of iterations is reached, the performance is minimized to the goal, the performance gradient falls below the minimum gradient value, the maximum training time is exceeded, or the momentum parameter exceeds the maximum value.

To have an algorithm independent from all these stop conditions, we set the parameters of the first row in the Table 2 to the highest value; 1e+3 for the maximum number of training iterations (epochs), infinite for the maximum training time and 1e+10 for the maximum momentum. In addition, we set the parameters of the second row in the Table 2 to the less value; 0 for the optimal performance (error), and 1e-7 for the minimum gradient value. Therefore, the algorithm depends only on the parameter Maximum number of validation failures per training step. The training stops when the validation performance is increased more than 10 times. This parameter is set to six after many tests as presented in Table 2. We noticed that if the network performance does not improve after six iterations, it will be the same in 100 iterations that argue the value of the parameter.

### 5.3. Texture Features Extraction

To minimize the training time, instead of having the whole image as the network input, we will only introduce the texture features extracted from images. Features are extracted from the method of Sum and Difference Histograms (SDH). The most appropriate features for classifying the stamps are; homogeneity (Equation (22)), energy (Equation (23)), correlation
(Equation (24)), and contrast (Equation (25)):

$$ \text{Homogeneity} = \sum_{j} \left( \frac{1}{1 + j^2} \hat{P}_{d}(j) \right), \qquad (22) $$

$$ \text{Energy} = \sum_{i} (\hat{P}_{S}(i))^2 \sum_{j} (\hat{P}_{d}(j))^2, \qquad (23) $$

$$ \text{Correlation} = \frac{1}{2} \left( \sum_{i} (i-2\mu)^2 \hat{P}_{S}(i) - \sum_{j} (j^2 \hat{P}_{d}(j)) \right), \qquad (24) $$

$$ \text{Contrast} = \sum_{j} (j^2 \hat{P}_{d}(j)), \qquad (25) $$

where $\hat{P}_S(i)$ is the element at index $i$ of the normalized sum histogram and $\hat{P}_d(j)$ is the element at index $j$ of the normalized difference histogram. The features “contrast” and “homogeneity” (local homogeneity) quantify the texture roughness, “correlation” quantifies the texture uniformity and the feature “energy” (global homogeneity) quantifies the texture homogeneity. Thus, our choice is argued for the stamp separation by the fact that the combination of these features successfully describes the different passport stamps. The purpose of this task is for training time reduction and best network training.

## 5.4. MLP Neural Network Architecture for Passport Stamps Classification

We optimize the number of hidden layers in order to reduce neural network's training time and increase the classification accuracy. The number of hidden neurons was chosen after many trials in order to successfully distinguish the different stamp classes. The proposed neural network includes four input neurons in the input layer which are extracted from the stamp images as features, seven fully connected hidden layers (5, 7, 9, 9, 11, 13 and 13 neurons) and three output neurons at the output layer. We chose the Hyperbolic tangent sigmoid as transfer function for the seven hidden layers. First, passport stamps are manually classified to define the target outputs. These outputs are presented as [1 0 0] for entry local stamp, [0 1 0] for exit local stamp and [0 0 1] for foreign stamp. Then, MLP output will be computed in the training process according to these target values. After the MLP neural network is trained, the values of test set are used to evaluate the network performance. The basic structures of the MLP neural network proposed in this paper is depicted in Figure 9. For more explanation around the structure of the proposed framework, the UML diagram for the proposed approach is illustrated in Figure 10.

<figure><img src="image_14.png" /><figcaption>**Figure 9**
Artificial neural network model designed for stamp classification</figcaption></figure>
<caption>Figure 10</caption>

An UML activity diagram showing the proposed framework. Activities and Actions within a dotted expansion region are executed once (are not executed for each incoming object)

<figure><img src="image_15.png" /></figure>

## 6. Experimental Study

In this section, we aim to evaluate our approach for stamp classification. We first used different training sets to assess the effectiveness of the proposed method. Second, we compare our approach with other classifiers.

Because of the low variance in the stamp shape of Schengen areas, we will focus only on one European county. Our experiment will be made for France case where we will have three classes: entry France stamp (Figure 11(a)), exit France stamp (Figure 11(b)), and foreign stamp (Figure 11(c)).

The machine learning of neural network requires a large amount of data. Here, the size of the dataset plays an important role in the success of learning. For experiment purposes, we collected stamps extracted from many real passport papers.

To design a more practical system, our experiments are not limited to scanned passport papers with high quality; they include captures from Webcam or smartphone. Image acquisition is designed to be performed with a scanner or any simple camera. As passport papers acquired from any camera, stamps have different resolutions. In order to have a consistent dataset, each extracted passport stamp should be rotated and placed in a fixed image- frame of 340×270 (pixels).

Our aim at this level is to increase the number of stamps to have a neural network able to successfully learn the different stamp types. To do that, first we added noise to each stamp image. Here, we used Gaussian noise, Poisson noise and Salt and pepper noise. Second, we created other images by adjusting the intensity values.
<caption>**Figure 11**</caption>

Sample stamps used for the performance evaluation: (a) entry France stamps (b) exit France stamps and (c) foreign stamps

<figure><img src="image_16.png" /></figure>

Finally, we rotated stamp images by different angles. After these steps, we managed to have a dataset of 9120 stamps, 3040 stamps per class.

The validation of the proposed approach is through the quantitative and the qualitative evaluation.

Figure 12 illustrates the input/output results of each step of the proposed stamp extraction approach. Referring to the achieved visual results, presented in figure 12, we can say that the proposed method produced effective and robust segmentation and classification results. Indeed, we were able to identify and to decide between the various zones: stamp and non-stamp. In addition, we could identify the boundaries of the objects. The quality of segmentation is still justified by better stamp distinction from blurred passport papers. In order to evaluate the performance and the effectiveness of the proposed approach, we conduct several experiments with different training subsets TS. We randomly selected TS=10%, 20%, 30%, 40%, 50%, 60%, 70%, 80% and 90% of our dataset as training sets from each class to investigate the impact of varying the number of training samples to be selected on the effectiveness of each method. The remaining stamps are used for testing.

As a first and baseline experiment, networks are trained on our dataset. We train multiple versions of the network with the different dataset partitions. Because of the random initialization, the experiments

<caption>**Figure 12**</caption>

Example of the proposed stamp extraction approach from textured passport page: (a) Original image, (b) result of edge detection, (c) result of passport page segmentation, (d) detected stamps (stamp/non-stamp mask), and (e) extracted stamps

<figure><img src="image_17.png" /></figure>
<caption>**Figure 13**</caption>

Boxplots of training and testing accuracies obtained by the proposed method using different training sets: (a) training accuracies, and (b) testing accuracies

<figure><img src="image_18.png" /></figure>

were executed fifteen times with different random training sets.

The accuracy value over training images and test images is measured as the performance criteria, in order to quantitatively evaluate the obtained results and to compare the effectiveness of the proposed model.

In order to identify outliers and evaluating extremes, full statistical analysis of the training and testing accuracies obtained by the proposed approach using different training sets (15 experiments for each training set size) are depicted by two boxplots in Figure 13. These boxplots provide basic information about the different accuracy values. It graphically depicts the group of numerical data according to their quartiles.

Because the trained MLP network produces different results according to the initial training condition, we have averaged 15-trial results of the average accuracy for each training set size. Average training accuracies and testing accuracies obtained by the MLP classifier using different training sets, are presented in Table 3.

It can be observed that regardless the training subset size, the performance of the proposed method is promising with values ranging between 81.60% and 94.47%. Having 70% of the dataset as a training set achieves the highest average testing accuracy of 94.47%.

<caption>**Table 3**</caption>

Average training accuracy and testing accuracy obtained by MLP classifier using different training set

<table><thead><tr><td>Training subsets</td><td>Average training accuracy (%)</td><td>Average testing accuracy (%)</td></tr></thead><tbody><tr><td>10%</td><td>88.80</td><td>81.60</td></tr><tr><td>20%</td><td>92.67</td><td>87.92</td></tr><tr><td>30%</td><td>95.99</td><td>91.31</td></tr><tr><td>40%</td><td>95.33</td><td>91.76</td></tr><tr><td>50%</td><td>96.14</td><td>93.76</td></tr><tr><td>60%</td><td>96.54</td><td>94.01</td></tr><tr><td>70%</td><td>96.8</td><td>94.47</td></tr><tr><td>80%</td><td>95.44</td><td>93.12</td></tr><tr><td>90%</td><td>95.44</td><td>93.12</td></tr></tbody></table>

To further evaluate the proposed method, other classification metrics are computed from the confusion matrix (Table 4) for TS=70% achieving a testing accuracy of 96.02% with 6384 training samples and 2128 training samples per class. These metrics are precision, recall and F1-score. They are evaluated for each
<caption>**Table 4**</caption>

Confusion matrix for TS=70%, values between parentheses are the correspondent percentages (class1: ENLS, class2: EXLS and class3: FS)

<table><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Actual/Real</th></tr><tr><th>ENLS</th><th>EXLS</th><th>FS</th></tr></thead><tbody><tr><th rowspan="3">Predicted</th><th>ENLS</th><td><strong>874</strong><br>(95.83%)</td><td>28<br>(03.07%)</td><td>15<br>(01.64%)</td></tr><tr><th>EXLS</th><td>26<br>(02.85%)</td><td><strong>876</strong><br>(96.05%)</td><td>20<br>(02.19%)</td></tr><tr><th>FS</th><td>12<br>(01.31%)</td><td>08<br>(00.88%)</td><td><strong>877</strong><br>(96.16%)</td></tr></tbody></table>

class:Entry Local Stamp (denoted ENLS), EXit Local Stamp (denoted EXLS) and Foreign Stamp (denoted FS), as presented in Table 5. It can be seen that the percentages are ranging between 95.01% and 97.77% indicating the capacity of the proposed method to identify each class.

<caption>**Table 5**</caption>

Classification metrics obtained by the proposed method

<table><thead><tr><th rowspan="2">classes</th><th colspan="2">Samples</th><th rowspan="2">Precision (%)</th><th rowspan="2">Recall (%)</th><th rowspan="2">F1-Score</th></tr><tr><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>ENLS</td><td>2128</td><td>912</td><td>95.31</td><td>95.83</td><td>95.57</td></tr><tr><td>EXLS</td><td>2128</td><td>912</td><td>95.01</td><td>96.05</td><td>95.53</td></tr><tr><td>FS</td><td>2128</td><td>912</td><td>97.77</td><td>96.16</td><td>96.96</td></tr></tbody></table>

The proposed approach is compared with Extreme Learning Machines (ELM) [23] and Convolutional Neural Network (CNN) classifiers [25].

The CNN classifier seeks to learn features by itself, and these learned features are used in stamp classification. After many runs, we fixed the parameters of the CNN model, i.e., the learning rate, the number of convolution layers, the number of nodes in the hidden layers, the kernel size. Table 6 reports the architecture of CNN. The learning rate is fixed to 0.01 for the linear regression with softmax function and the number of training epochs is set to two.

For ELM classifier, the number of hidden layers is twenty and the neurons use sigmoid as activation function.

<caption>**Table 6**</caption>

Parameters setting of the architecture of the CNN model

<table><thead><tr><th>Hidden layer</th><th>Convolution</th><th>Rectified Linear Unit</th><th>Pooling</th></tr></thead><tbody><tr><td>1</td><td>3 × 3 × 1 × 8</td><td>Yes</td><td>2 × 2</td></tr><tr><td>2</td><td>3 × 3 × 8 × 16</td><td>Yes</td><td>2 × 2</td></tr><tr><td>3</td><td>3 × 3 × 16 × 32</td><td>Yes</td><td>2 × 2</td></tr></tbody></table>

The performance of the three methods are depicted in Figure 14. It is worthwhile to notice that, regardless of the number of training samples per class, the obtained average accuracies showing that our approach based on a standard neural network outperforms the CNN and ELM models.

<caption>**Figure 14**</caption>

Average accuracy values achieved by the MLP classifier in comparison to other methods

<figure><img src="image_19.png" /></figure>

Compared with ELM, the achieved accuracies illustrated in Figure 15 show that our approach has the power to perfectly detect stamps regardless of the number of runs. As can be seen in Figure 15, using 70% of the dataset as training set achieves the best performance in terms of accuracy. Using 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80% or 90% of the dataset as a training set from each class, does not affect the performance of the MLP classifier over the ELM classifier. Our method outperforms the ELM even with 304 training samples per class, which is the case of a training set containing only 10% of the whole dataset (TS=10%).
<figure><img src="image_20.png" /><figcaption>Figure 15</figcaption></figure>

Comparison between MLP and ELM using different
training set size: (a) TS=10%, (b) 20%, (c) 30%, (d) 40%, (e)
50%, (f) 60%, (g) 70%, (h) 80% and (i) 90%

<figure><img src="image_21.png" /></figure>
<figure><img src="image_22.png" /></figure>

Compared with ELM and CNN classifiers, the different accuracies presented in Figure 16 can further prove the effectiveness of the proposed approach. It presents six boxplots evaluating extremities, and illustrating full statistical analysis of the classification results of the three methods. In fact, the obtained training and testing accuracies of ELM and CNN methods with different training sets do not exceed the results of our classifier.

<caption>Figure 16</caption>

Comparison of the proposed method with ELM and CNN classifiers using boxplots of 15-trial accuracies for each training set size on 9120 stamps: (a) boxplot of training accuracies obtained by the proposed classifier, (b) boxplot of testing accuracies obtained by the proposed classifier, (c) boxplot of training accuracies obtained by the ELM classifier, (d) boxplot of testing accuracies obtained by the ELM classifier, (e) boxplot of training accuracies obtained by the CNN classifier, and (f) boxplot of testing accuracies obtained by the CNN classifier

<figure><img src="image_23.png" /></figure>
<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

Using 50% of the dataset as training set, with 1520 training samples per class, the test accuracy can be increased from 66.87% (TS=10%) to 96.23% which is the maximum accuracy of the MLP classifier, as presented in Table 7. It is an error reduction of 30%. Conventional neural network classifier outperforms Extreme Learning Machine method with a testing average accuracy of 76.56%, as shown in Table 7. However, better results are obtained by the proposed approach based on the multi-layer perceptron classifier, with a maximum average accuracy of 96.23% (Table 7). Here, the failure of CNN is explained by the needs a huge dataset to train the network. As, this work is original and not yet treated in the literature related to passport stamp classification, there is no image base available. Therefore, the conventional neural network cannot be a solution for our task.

<table><caption>**Table 7**
Maximum Accuracy of different classifiers</caption><thead><tr><td>Methods</td><td>Training Accuracy</td><td>Testing Accuracy</td></tr></thead><tbody><tr><td>Conventional Neural Network (CNN)</td><td>80.03%</td><td>76.56%</td></tr><tr><td>Extreme Learning Machine (ELM)</td><td>63.82%</td><td>62.55%</td></tr><tr><td>Multi-Layer Perceptron (MLP)</td><td>98.40%</td><td>96.23%</td></tr></tbody></table>
It is indispensable to pay attention to the problems of modern classification like CNN, where the need for a large training set and the huge time expenditure needed to train such a classifier.

As our goal is to achieve a reasonable run time and high performance, the CNN model does not offer an accurate classifier solution for our task due to the long training time, as presented in Table 8. Using AMD Radeon (TM) R5 M330 as GPU and Intel(R) Core(TM) i5-7200U CPU @ 2.50 GHz (4 CPUs), 2.70 GHz as CPU, the proposed approach based on MLP is more than 15 times faster than the CNN network model. It presents a huge reduction of the runtime needed for training and testing.

<caption>**Table 8**</caption>

Time needed to train and test the classifiers using 70% of the dataset as training set

<table><thead><tr><td>Methods</td><td>Training</td><td>Testing</td></tr></thead><tbody><tr><td>CNN (2 epochs)</td><td>16 min, 24 sec</td><td>1 min, 15 sec</td></tr><tr><td>CNN (4 epochs)</td><td>47 min, 3 sec</td><td>1 min, 23 sec</td></tr><tr><td>MLP</td><td>1 min, 30 sec</td><td>0.0733 sec</td></tr><tr><td>ELM</td><td>0.4219 sec</td><td>0.0156 sec</td></tr></tbody></table>

We note that ELM achieves better results in terms of execution time, it is trained and tested in real-time. However, with a maximum accuracy of 62.55%, it cannot be an accurate classifier. As the main idea of this work is to provide a fast and accurate classifier, MLP is the preferred network for our case. In order to have a compact, quickly assimilated summary of the execution time data, boxplots of training time for the three classifiers are presented in Figure 17. It provides a visual summary of the time data showing median values, dispersion of the data set, and signs of skewness. In terms of speed, the proposed classifier proved to be faster than CNN. Although, ELM has the minimum execution time.

The results for seven-layer Levenberg-Marquardt feed-forward neural network show that stamps were classified correctly. The network model performed in this paper has yielded an average accuracy of 94.47% (for TS=70%), which is achieved in almost one min-

<caption>**Figure 17**</caption>

Training time of the proposed method, ELM and CNN classifiers using many training set sizes: (a) training time boxplot of the proposed classifier, (b) training time boxplot of the ELM classifier, and (c) training time boxplot of CNN classifier

<figure><img src="image_28.png" /></figure>
ute for training and test. The degree of accuracy of a
system usable in a real environment is promising.

Another strong point of our system is to distinguish
between entry and exit stamps, since for the Schengen
space case the entry stamp is almost identical to the
exit one, as shown in the Figure 11 (a) and (b).

Experiments show that our system has the power to
perfectly detect stamps with high degree of degra-
dation. It can robustly handle the poor image qual-
ity and it is insensitive to stamp shape, orientation
and scale-variation. It leads to improved robustness
against overlapping stamps and image degradation
that otherwise would corrupt stamp detection. It can
easily separate stamps from other elements in pass-
port paper.

7. Conclusion

The compulsory stamping of passports is a way to
improve border control and combat illegal immigra-
tion. However, it leads to a proliferation of queues and
longer waiting time. Therefore, automatic reading of
passport information is a primary security issue in
border control. Our proposed system helps the bor-
der officers performing these stamps checking op-
erations automatically. A system supporting faster
and safer border control processes. This solution is
possible using intelligent tools to extract key infor-
mation and quickly process them. At first, texture fea-

ture extraction is performed in order to find potential
stamps. Next, image segmentation aimed at detect-
ing objects of specific textures are employed. Then,
isolated objects are extracted and classified with the
help of multi-layer perceptron artificial network into
three categories: entry local stamp, exit local stamp,
and foreign stamp. Trained system can in a fairly
short time return the information about the category
of each stamp in a passport page. The proposed stamp
classification by a low-cost neural network model is
performed after an image pre-processing and stamp
extraction using a texture analysis method and a fuzzy
classifier. One of the strong points of this approach is
that do not rely on stamp color and is not limited to
specific stamp shape. We have demonstrated the ef-
fectiveness of our technique by evaluating the average
accuracy (94.47%) on real travel documents having
different types of stamps. Nonetheless, our results
have been achieved using a rather small dataset of
images and with only three classes of stamps. Recog-
nizing the origin of each stamp would be a future work.
Moreover, using a system with graphical processing
unit (GPU) chips, and an extended dataset it is pos-
sible to get promising classification results with the
help of CNN model. We will work on the creation of a
huge dataset of passport stamps needed for convolu-
tional neural network in order to promote the applica-
tion of CNN techniques. Also, we have analyzed only
passport pages containing isolated stamps, which is a
limitation of the proposed system. In future, we plan
to extend it for overlapped stamps.

References

1. Abedini, F., Menhaj, M. B., Keyvanpour, M. R. An MLP-based Representation of Neural Tensor Networks for the RDF Data Models. Neural Computing and Applications, 2019, 31, 1135-1144. https://doi.org/10.1007/s00521-017-3053-1

2. Alaei, A., Delalandre, M. A Complete Logo Detection/Recognition System for Document Images. In 2014 11th IAPR International Workshop on Document Analysis Systems (DAS), IEEE, 2014, 324-328. https://doi.org/10.1109/DAS.2014.79

3. Alaei, A., Delalandre, M., Girard, N. Logo Detection Using Painting Based Representation and Probability Fea-

tures. In 12th International Conference on Document
Analysis and Recognition (ICDAR), IEEE, 2013, 1235-
1239. https://doi.org/10.1109/ICDAR.2013.250

4. Alaei, A., Roy, P. P., Pal, U. Logo and Seal Based Administrative Document Image Retrieval: A Survey. Computer Science Review, 2016, 22, 47-63. https://doi.org/10.1016/j.cosrev.2016.09.002

5. Alharbi, A., Alghahtani, M. Using Genetic Algorithm and Elm Neural Networks for Feature Extraction and Classification of Type 2-Diabetes Mellitus. Applied Artificial Intelligence, 2019, 33, 311-328. https://doi.org/10.1080/08839514.2018.1560545
6. Block, S. B., da Silva, R. D. D., Dorini, L., Minetto, R. Inspection of Imprint Defects in Stamped Metal Surfaces using Deep Learning and Tracking. IEEE Transactions on Industrial Electronics, 2020. https://doi.org/10.1109/TIE.2020.2984453

7. Cai, Y., Li, Y., Qiu, C., Ma, J., Gao, X. Medical Image Retrieval Based on Convolutional Neural Network and Supervised Hashing. IEEE Access, 2019, 7, 51877-51885. https://doi.org/10.1109/ACCESS.2019.2911630

8. Chanda, S., Prasad, P. K., Hast, A., Brun, A., Martensson, L., Pal, U. Finding Logo and Seal in Historical Document Images-An Object Detection Based Approach. In Asian Conference on Pattern Recognition, Springer, Cham, 2019, 821-834. https://doi.org/10.1007/978-3-030-41404-7_58

9. Chatbri, H., Kameyama, K., Kwan, P., Little, S., O'Connor, N. E. A Novel Shape Descriptor Based on Salient Keypoints Detection for Binary Image Matching and Retrieval. Multimedia Tools and Applications, 2018, 1-24. https://doi.org/10.1007/s11042-018-6054-x

10. Demuth, H. B., Beale, M. H., De Jess, O., Hagan, M. T. Neural Network Design. Martin Hagan, 2014.

11. Dey, S., Mukherjee, J., Sural, S. Stamp and Logo Detection from Document Images by Finding Outliers. In Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG), IEEE, 2015, 1-4. https://doi.org/10.1109/NCVPRIPG.2015.7489947

12. Ding, L., Goshtasby, A. On the Canny Edge Detector. Pattern Recognition, 2001, 34, 721-725. https://doi.org/10.1016/S0031-3203(00)00023-6

13. Dixit, U. D., Shirdhonkar, M. Automatic Logo Detection and Extraction Using Singular Value Decomposition. In International Conference on Communication and Signal Processing (ICCSP), IEEE, 2016, 0787-0790. https://doi.org/10.1109/ICCSP.2016.7754252

14. Forczmański, P., Markiewicz, A. Low-LEVEL Image Features for Stamps Detection and Classification. In Proceedings of the 8th International Conference on Computer Recognition Systems CORES, Springer, Heidelberg, 2013, 383-392. https://doi.org/10.1007/978-3-319-00969-8_37

15. Forczmański, P., Frejlichowski, D. Classification of Elementary Stamp Shapes by Means of Reduced Point Distance Histogram Representation. In International Workshop on Machine Learning and Data Mining in Pattern Recognition, Springer, Berlin, Heidelberg,

2012, 603-616. https://doi.org/10.1007/978-3-642-31537-4_47

16. Gallego, A. J., Pertusa, A., Bernabeu, M. Multi-label Logo Classification Using Convolutional Neural Networks. In Iberian Conference on Pattern Recognition and Image Analysis, Springer, Cham, 2019, 485-497. https://doi.org/10.1007/978-3-030-31332-6_42

17. Girshick, R., Donahue, J., Darrell, T., Malik, J. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and pattern Recognition, 2014, 580-587. https://doi.org/10.1109/CVPR.2014.81

18. Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, 2015, 1440-1448. https://doi.org/10.1109/ICCV.2015.169

19. Gu, Q., Li, Z., Han, J. Generalized Fisher Score for Feature Selection. arXiv preprint arXiv:1202.3725, 2012.

20. Guru, D. S., Kumar, N. V. Symbolic Representation and Classification of Logos. In Proceedings of International Conference on Computer Vision and Image Processing, Springer, Singapore, 2017, 555-569. https://doi.org/10.1007/978-981-10-2104-6_50

21. Hagan, M. T., Menhaj, M. B. Training Feedforward Networks with the Marquardt Algorithm. IEEE Transactions on Neural Networks, 1994, 5, 989-993. https://doi.org/10.1109/72.329697

22. Hartl, A., Arth, C., Schmalstieg, D. Real-time Detection and Recognition of Machine-Readable Zones with Mobile Devices. In Proceedings. International Conference on Computer Vision Theory and Applications, 2015, 79-87. https://doi.org/10.5220/0005294700790087

23. Huang, G. B., Zhu, Q. Y., Siew, C. K. Extreme Learning Machine: Theory and Applications. Neurocomputing, 2006, 70, 489-501. https://doi.org/10.1016/j.neucom.2005.12.126

24. Kim, J. Recognition of Passport MRZ Information Using Combined Neural Networks. Journal of the Korea Society of Digital Industry and Information Management, 2019, 15(4), 149-157.

25. Krizhevsky, A., Sutskever, I., Hinton, G. E. Imagenet Classification with Deep Convolutional Neural Networks. In Advances in neural Information Processing Systems, 2012, 1097-1105.

26. Lam, H. H., Tsang, P. W. M., Poon, T. C. Ensemble Convolutional Neural Network for Classifying Ho-
lograms of Deformable Objects. Optics Express,
2019, 27(23), 34050- 34055. https://doi.org/10.1364/
OE.27.034050

27. Mamarabadi, M., Rohani, A. Clustering of Fungal Hexosaminidase Enzymes Based on Free Alignment Method Using MLP Neural Network. Neural Computing and Applications, 2018, 30, 2819-2829. https://doi.org/10.1007/s00521-017-2876-0

28. Micenkov, B., van Beusekom, J. Stamp Detection in Col-
or Document Images. In 2011 International Conference
on Document Analysis and Recognition, IEEE, 2011,
1125-1129. https://doi.org/10.1109/ICDAR.2011.227

29. Mo, H., Wang, J. Return Scaling Cross- Correlation Forecasting by Stochastic Time Strength Neural Network in Financial Market Dynamics. Soft Computing, 2018, 22, 3097-3109. https://doi.org/10.1007/s00500-017-2564-0

30. Naganjaneyulu, G., Krishna, C. S., Narasimhadhan, A. A Novel Method for Logo Detection Based on Curvelet Transform Using GLCM Features. In Proceedings of 2nd International Conference on Computer Vision and Image Processing, Springer, 2018, 1-12. https://doi.org/10.1007/978-981-10-7898-9_1

31. Nandedkar, A. V., Mukherjee, J., Sural, S. A Spectral Filtering Based Deep Learning for Detection of Logo and Stamp. In Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG), IEEE, 2015, 1-4. https://doi.org/10.1109/NCVPRIPG.2015.7490053

32. Nedeljkovic, I. Image Classification Based on Fuzzy Logic. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2004, 34, 3-7.

33. Olivas-Padilla, B. E., Chacon-Murguia, M. I. Classification of Multiple Motor Imagery Using Deep Convolutional Neural Networks and Spatial filters. Applied Soft Computing, 2019, 75, 461-472. https://doi.org/10.1016/j.asoc.2018.11.031

34. Pan, D., Shi, P., Qiu, Z., Sha, Y., Zhongdi, X., Zhoushao, J. TV Logo Classification Based on Convolutional Neural Network. In 2016 IEEE International Conference on Information and Automation (ICIA), IEEE, 2016, 1793-1796. https://doi.org/10.1109/ICInfA.2016.7832108

35. Petej, P., Gotovac, S. Comparison of Stamp Classification Using SVM and Random Ferns. In IEEE Symposium on Computers and Communications (ISCC), IEEE, 2013, 850-854. https://doi.org/10.1109/ISCC.2013.6755055

36. Petrova, O., Bulatov, K. Methods of Machine-Readable Zone Recognition Results Post-Processing. In Eleventh International Conference on Machine Vision, 2019, 11041. https://doi.org/10.1117/12.2522792

37. Połap, D. Analysis of Skin Marks Through the Use of Intelligent Things. IEEE Access, 2019, 7, 149355-149363. https://doi.org/10.1109/ACCESS.2019.2947354

38. Polap, D., Wlodarczyk-Sielicka, M. Classification of Non-Conventional Ships Using a Neural Bag-of-Words Mechanism. Sensors, 2020, 20(6), 1608. https://doi.org/10.3390/s20061608

39. Redmon, J., Farhadi, A. Yolov3: An Incremental Improvement. CoRR abs/1804.02767, 2018.

40. Ren, S., He, K., Girshick, R., Sun, J. Faster R-CNN: Towards Real-Time Object Detection with region Proposal Networks. In Advances in neural information processing systems, 2015, 91-99.

41. Roy, P. P., Pal, U., Llados, J. Document Seal Detection Using GHT and Character Proximity Graphs. Pattern Recognition, 2011, 44, 1282-1295. https://doi.org/10.1016/j.patcog.2010.12.004

42. Singh, P. Rainfall and Financial Forecasting Using Fuzzy Time Series and Neural Networks Based Model. International Journal of Machine Learning and Cybernetics, 2018, 9, 491-506. https://doi.org/10.1007/s13042-016-0548-5

43. Singh, P., Raman, B., Roy, P. P. Detection of Seal and Signature Entities with Hierarchical Recovery Based on Watermark Self Embedding in Tampered Digital Documents. Displays, 2018, 54, 47-59. https://doi.org/10.1016/j.displa.2018.09.004

44. Veershetty, C., Hangarge, M. Logo Retrieval and Document Classification Based on LBP Features. In Data Analytics and Learning, Springer, 2019, 131-141. https://doi.org/10.1007/978-981-13-2514-4_12

45. Wang, Z., Lian, J., Song, C., Zheng, W., Yue, S., Ji, S. CSRS: A Chinese Seal Recognition System With Multi-Task Learning and Automatic Background Generation. IEEE Access, 7, 2019, 96628-96638. https://doi.org/10.1109/ACCESS.2019.2927396

46. Wiggers, K. L., Britto, A. S., Heutte, L., Koerich, A. L., Oliveira, L.E.S., Document Image Retrieval Using Deep Feature. In International Joint Conference on Neural Networks (IJCNN), IEEE, 2018, 1-8. https://doi.org/10.1109/IJCNN.2018.8489722

47. Žemgulys, J., Raudonis, V., Maskeliūnas, R., Damaševičius, R. Recognition of Basketball Referee Signals from
Real-Time Videos. Journal of Ambient Intelligence and Humanized Computing, 2020, 11(3), 979-991. https://doi.org/10.1007/s12652-019-01209-1

48. Zeng, W., Li, M., Yuan, C., Wang, Q., Liu, F., Wang, Y. Classification of Focal and Non Focal EEG Signals Using Empirical Mode Decomposition (EMD), Phase Space Reconstruction (PSR) and Neural Networks. Artificial Intelligence Review, 2019, 52, 625-647. https://doi.org/10.1007/s10462-019-09698-4

49. Zhu, G., Jaeger, S., Doermann, D. A Robust Stamp Detection Framework on Degraded Documents. In Document Recognition and Retrieval XIII, International Society for Optics and Photonics, 2006, 60670B. https://doi.org/10.1117/12.643537

<figure><img src="image_29.png" /></figure>

This article is an Open Access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 (CC BY 4.0) License (http://creativecommons.org/licenses/by/4.0/).