Banknote Portrait Detection Using Convolutional Neural Network
Ryutaro Kitagawa∗ , Yoshihiko Mochizuki∗ , Satoshi Iizuka∗ , Edgar Simo-Serra∗ ,Hiroshi Matsuki† , Naotake Natori† , Hiroshi Ishikawa∗∗Waseda University, Department of Computer Science and Engineering,†Toshiba Corporation, Energy Systems & Solutions Company, Japan
Abstract
Banknotes generally have different designs according totheir denominations. Thus, if characteristics of each design
can be recognized, they can be used for sorting banknotes
according to denominations. Portrait in banknotes is one
such characteristic that can be used for classification. A
sorting system for banknotes can be designed that recognizes portraits in each banknote and sort it accordingly. In
this paper, our aim is to automate the configuration of sucha sorting system by automatically detect portraits in sample banknotes, so that it can be quickly deployed in a newtarget country. We use Convolutional Neural Networks todetect portraits in completely new set of banknotes robust
to variation in the ways they are shown, such as the size andthe orientation of the face. .

1Introduction

Quick and accurate banknote sorting is essential in operations that handle large amount of cash such as banks,
cash transportation companies, and casinos. Banknote sorting is the task of separating banknotes according to theirdenominations. In large operations, such a task is largely
automated by cash-sorting machines.
Banknote sorting involves tasks such as• denomination recognition,• counterfeit detection,• damaged note detection, and• counting,and cash-sorting machines generally perform these tasks
automatically and with high throughput.
Cash-sorting machines exist in large numbers for majorcurrencies such as the US Dollar and the Euro. However,there are many more currencies in the world for which suchmechanization is less prevalent. The reason is the cost offine-tuning such machines for banknotes in a given currency. The fine-tuning involves finding and specifying characteristics of each banknote denomination in the given currency so that a pattern-recognition system can detect thecharacteristics to sort each note according to its denomination. Because each currency has its own design, manual tuning of cash-sorting machine takes time and costs toomuch in the case of minor currencies.
To alleviate this problem, here we focus on portraits
that appear on many denominations of various currencies
as characteristics to recognize the banknotes. By automatically finding portraits on a given set of sample notes, ourautomatic configuration system can facilitate the quicker
tuning of cash-sorting machines to a new currency.
Although it is relatively easy for humans to distinguishbanknotes by viewing the patterns and letters that appear

on them, it is inefficient for a machine vision system totry to use the whole face of a banknote to classify it. It ispreferable to specify a number of distinguishing characteristic part on each kind of banknote in a given currency forthe machine to identify it. For that purpose, pre-identifying
such characteristic parts and their positions on the banknote
is essential.
Many banknotes feature portraits of various historicalfigures and prominent individuals. They are portrayed invarious ways: sometimes only the head is shown, sometimes from the bust up; they can be a frontal view, in profile,or in between. To detect the portraits on previously unseen
set of sample banknotes, a method that is robust to such variety is needed. We use a Convolutional Neural Network(CNN) for that purpose. It is now very well known that
CNN shows very high performance in certain recognition
tasks, starting with the triumph of the method utilizing it[1] in the Imagenet Large Scale Visual Recognition Challenge (ILSVRC), a contest in general object recognition.

2 Related Work
There are a number of methods known to detect faces using neural networks. For instance, in 1994 Vaillant et al.
[2] searches for regions that may contain a face in the input
image using a neural network, and uses another neural network to determine if it truly contain a face. Also, in 1998
Roley et al. [3] proposed to detect faces using a neural network learned using the Bootstrap method, achieving a highperformance in detecting frontal faces. This method waslater expanded so that it is robust to the rotation of the face[4]. In 2002 Garcia et al. [5] presented a neural networkthat can detect faces in input images of different sizes, withvarious lighting conditions, and varying orientation of thefaces. Further, in 2005 Osadchy et al. [6] succeeded in realtime face detection and pose estimation using a Convolutional Neural Network.
Not only has neural networks shown their high performance in face detection, but they have demonstrated usefulness in detecting objects other than faces. And in recentyears, training deep neural networks has become possible,
leading to detection of multiple simultaneous objects. As
a newer method to detect multiple classes of objects using neural networks, the R-CNN method [7] by Girshick etal. is known. It uses Selective Search [9] to find candidate regions and then classify each region by using a Convolutional Neural Network that has been trained using theILSVRC 2013 dataset. However, it is not necessarily optimal to use such a method for detecting general objects
for our purpose, which is more specialized to finding portraits. Li et al. [8] finds that detection of small faces andcomplicated-looking faces does not always work when anetwork trained for general object recognition such as usedin R-CNN is used. Thus, it is preferable to use a dedicateddataset to train a portrait detection network.

Input
Step1
Execute formultiple times

The probability thatthe region contains a portrait

Step2

Fine-tuned
CNN

：0.01

Probability Map

Non-Maximum Suppression

Output

Step3
Figure 1. The process of the proposed method. The method determines the rectangles that contain portraits in the input
banknote image. Step 1: Candidate region is specified as a moving window. Step 2: Using a CNN, the probability
that the candidate region is a portrait is obtained. The probability is recorded as the value of a probability map at thecorresponding position. Step 3: A Non-Maximum Suppression process restricts the candidate regions.

𝑰𝒘

The probability that the region
contains a portrait
𝑰𝒉𝒘

𝒘𝒉…

…
…
…
…
…

𝒔𝑰𝒉𝒉

𝑰𝒉

…
…
…
…
…

𝒘𝒘…

Probability Map 𝑰𝒉

Input 𝑰
Figure 2. Generating the probability map. The probability that a portrait is in each candidate region specified by amoving window is given by the CNN and assigned to the corresponding position in the probability map. The widthof the probability map is the number of the candidate regions when the window is moved in the horizontal direction,
whereas its height is the number of the candidate regions when the window is slided in the vertical direction.

3 Proposed Method
Our method takes an image of a banknote and produces
the position of rectangles containing portraits in the image,if such exist. If more than one portraits are in the input
image, each is detected individually as a separate rectangle.
Fig. 1 shows the process of the method. It consists of thefollowing three steps, where Step 3 is done only once, after
Steps 1 and 2 are executed for multiple times.
Steps:
1. Candidate region is specified. Here, a moving window
generates candidate regions so that all possible candidates are systematically enumerated.
2. Using a CNN, the probability that the candidate region
is a portrait is obtained. We construct a map of probabilities by giving the center of the region the proba-bility that it contains a portrait. In Fig. 1, the part withhigher probability is shown in red.
3. A Non-Maximum Suppression process restricts thecandidate regions.

3.1 Training the CNN
In Step 2 above, a CNN is given a candidate region asthe input, and computes the probability that the region contains a portrait. Here, we fine-tuned the Alexnet[1], which
is a pre-trained network, using a dataset of portraits we prepared. In fine-tuning it, we re-trained only the two layers
on the output side, excluding the three layers after the input layer. This is because not fine-tuning the excluded partdoes not have much effect on the accuracy as the input-sideof a CNN processes relatively local features such as edges.
This speed up the training, since there are fewer layers toactually train.

Non-Maximum
Suppression

𝑰𝒉𝒘
𝒄𝒊−𝟏 𝒄𝒋−𝟏 𝒊𝒋−𝟏𝑰𝒉𝒉

𝒄𝒊+𝟏

𝒄𝒊−𝟏 𝒄𝒋−𝟏 𝒊𝒋−𝟏𝒋−𝟏

𝒄𝒊−𝟏𝒋 𝒄𝒊𝒋 𝒄𝒊+𝟏𝒋

𝑖𝑓 𝑐𝑖𝑗𝑖𝑠 𝑛𝑜𝑡 𝑚𝑎𝑥

𝒄𝒊−𝟏 𝒄𝒊𝒋+𝟏 𝒄𝒊−𝟏
𝒋+𝟏

𝒄𝒊+𝟏

𝒋−𝟏

𝒄𝒊−𝟏𝒋 0 𝒄𝒊+𝟏𝒋
𝒄𝒊−𝟏 𝒄𝒊𝒋+𝟏 𝒄𝒊−𝟏

𝒋+𝟏

𝒋+𝟏

𝒋+𝟏

𝟑×𝟑
Probability Map 𝑰𝒉Region 𝑫Figure 3. The NMS process. NMS replaces values that are not locally maximum by 0. If the original probability cij isnot maximum among its neighbours, the new value c′ij at the position is set to 0.

3.2 Probability MapIn Step 3, a Non-Maximum Suppression process is used
to find the peaks of the probability.. In Fig. 3, the probability map is depicted where higher probability is shown in
red regions in the map. In Fig. 2, the process of generating the probability map is depicted. This corresponds to theSteps 1 and 2, where the input image I is turned into theprobability map I h . As shown in this figure, the probability
that each candidate region contains a portrait is given as thevalue of the probability map I h at the corresponding pointin the map, i.e., the center of the region. Thus the size ofthe probability map I h is the same as the expanse of thecentral points of the possible candidate regions. Thus, thehand the height Ihh of the probability map I h canwidth Iw
be computed as:⌊⌋
Iw − wwhIw=+ 1,
(1)
s
⌊
⌋
Ih − w hIhh =+ 1.
(2)sHere, s is the stride width.

3.3

In our method, we additionally smooth the probability map
with a Gaussian filter, so that noise is reduced and the peak
becomes clearer. Then the remaining non-zero values arepeak values, however small. We need to cull small values
lest positions with small but peak probability is found tohave a portrait. Therefore, we preset a threshold θ and takeonly the peaks with larger probability than θ as the output
points.

4 Dataset
The CNN we use in our method computes the probability that the given input image is a portrait. To train it,
we prepared a moderately large number of portrait images
appearing in banknotes as positive samples and images ofparts of banknotes that do not contain portraits as negativesamples. First, we procured 3330 banknote images, eachof which shows the whole of a banknote, counting the front
and the back side of a banknote as two separate images. Outof these, we randomly sampled 2330 images as the training
dataset and used the other 1000 images as test images. Then
we manually cut out 889 portraits and 3673 non-portrait regions from the 2330 training images.

Non-Maximum Suppression

Even when there is only a single portrait in the input image, multiple candidate regions that partially overlaps theportrait has non-zero probability of containing the portrait.
However, we want to pinpoint the single region that containthe whole of the portrait. Thus, we use a Non-Maximum
Suppression (NMS) method to single out the center of sucha region for each of (possibly more than one) portraits. InNMS, we look at each 3 × 3 square in the probability map
and if the number at the center is not the maximum among
the nine numbers, we replace the number at the center with
0. (The number referred as the nine numbers should be theoriginal one before possibly being replaced by 0 in theirturn.) After this process, only the numbers that are higher
than their surrounding retain their value. These points arecalled the peak positions. Thus, applying NMS on the probability map obtained in the previous subsection results inonly the points with the highest probability of being thecenter of a region containing a portrait among their neighbours remaining non-zero. Fig. 3 depicts the NMS process
on the probability map. In Fig. 3, cij denotes the original
probability value of I h at coordinate (i, j) and D the 3 × 3
region around (i, j). The new value c′ij after NMS can bewritten:{0(cij = min(l,m)∈D clm )′cij =(3)cij (cij ̸= min(l,m)∈D clm )5 Experimental Results
5.1 Experiments
We performed portrait detection on the 1000 test images.
These images are not used for training. Each shows thewhole of a banknote. The moving window has the size
ww × wh , which is adapted according with the width Iwof the input image as
wwwh= ⌊0.1497Iw ⌋ ,= ⌊0.4180Iw ⌋ .

(4)
(5)

Here, the numbers 0.1497 and 0.4180 are the average sizeof the portraits when the width of the banknote image isset to 1. In the experiments, we varied the stride width sof the moving window as s = 3, 5, 10, 15. We also tried
two different kernel filter size k and variance σ: (k, σ) =(5, 1.1), (11, 2). Furthermore, the value of θ was set to 0.8.
Since the input of the CNN is fixed to the input size 227 ×
227 of Alexnet, we resized the candidate region to this size
before feeding it to the CNN.

Table 1. Precision and Recall with varying stride s and Gaussian kernel width kStride s335510101515511511511511
Gaussian kernel width kPrecision
0.860 0.508 0.600 0.550 0.560 0.597 0.657 0.470
0.993 0.919 0.946 0.896 0.877 0.561 0.813 0.158
Recall5.2

Evaluation

We calculated how many of the output rectangles correctly detected portraits as the Precision and Recall numbers. We determined if an output rectangle correctly detects
a portrait by computing the overlap ratio L of the output
rectangle and the ground-truth rectangle. The overlap ratio is defined by the following formula, where A is the areaof the ground-truth rectangle, B is the area of the output
rectangle, and C is the area of the intersection of the tworectangles:
L=C.
A+B−C

7 Acknowledgments(6)This work was partially supported by JST CREST.

We determined that the portrait is correctly detected when
this number L exceeds a threshold ϕ, and designated thecase as the True Positive (TP). Conversely, when L does
not exceed ϕ, we determined that the portrait is not correctly detected, and designated the output rectangle as False
Positive (FP) and the ground-truth rectangle as Flase Negative (FN). In the experiments, we used the value ϕ = 0.3,
which corresponds to the state when about the half of therectangles overlap, if the two rectangles are of the same
size. Then we counted the number of TP, FP, and FN foreach test image. Then we computed the Precision and Recall from the sum of the number of TP, FP, and FN for the1000 test images, according to the formula:
P recision =Recall=TP,FP + TPTP.
FN + TP

moving windows to generate candidate regions and compute by a CNN the probability that each region contains aportrait. Then we use Non-Maximum Suppression to output the rectangles with the highest probability of containing
portraits. The method achieves high performance in detecting portraits. On the other hand, there are many instanceswhere a portrait is found where there is none. To improvethis, we consider the most important avenue is to improve
the training dataset, which is not trivial given the limited
number of portraits in banknotes.

(7)
(8)

5.3 Results
Table 1 shows the Precision and Recall with varying
stride s and Gaussian kernel width k. It indicates that thesmaller the stride s, the better. This is because there are
more candidate regions when the stride is smaller, and thus
more accurate match between the candidate region and theportrait is possible. Also, it is indicated the smaller of thetwo Gaussian kernel width produces better results.

6 Conclusion
In this paper, we proposed a method that, given an imageof a banknote as the input, find portraits in the input image and output rectangles containing the portraits. We use

References[1] A. Krizhevsky et al. “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems, pp.1097-1105, 2012.
[2] Vaillant et al.: “Original approach for the localisation of objects in images,” IEE Proceedings-Vision, Image and Signal
Processing, vol.141, no.4, pp.245–250, 1994.
[3] Rowley et al.: “Neural network-based face detection,” IEEE
Transactions on Pattern Analysis and Machine Intelligence,
vol.20, no.1, pp.23–38, 1998.
[4] Rowley et al.: “Rotation invariant neural network-based facedetection,” Proceedings of 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.38–44, 1998.
[5] Garcia et al.: “A neural architecture for fast and robust facedetection,” Proceedings of 16th International Conference onPattern Recognition, vol.2, pp.44–47, 2002.
[6] Garcia et al.: “Synergistic face detection and pose estimation
with energy-based models,” The Journal of Machine Learning Research, vol.8, pp.1197–1215, 2007.
[7] Girshick et al.: “Rich feature hierarchies for accurate object
detection and semantic segmentation,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp.580–587, 2014.
[8] Li et al.: “A Convolutional Neural Network Cascade for Face
Detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5325–5334, 2015.
[9] Uijlings et al.: “Selective search for object recognition,” IEEE
Transactions on Pattern Analysis and Machine Intelligence,
vol.104, no.2, pp.154–171, 2013.