<figure><img src="image_1.png" /></figure>

HAL
open science

# Few Shot Classification of Smartphone Photos using Hidden Markov Model

Zulkarnaen Hatala, Muhammad Khudzaly

► To cite this version:

Zulkarnaen Hatala, Muhammad Khudzaly. Few Shot Classification of Smartphone Photos using Hidden Markov Model. 2024. hal-04691064

HAL Id: hal-04691064

https://hal.science/hal-04691064v1

Preprint submitted on 7 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Few Shot Classification of Smartphone Photos using Hidden Markov Model

Zulkarnaen Hatala<fnref n="1" />, Muhammad Khudzaly<fnref n="2" />

**Article Info**

**Article history:**

Received month dd, yyyy

Revised month dd, yyyy

Accepted month dd, yyyy

**Keywords:**

Photo Classification

Hidden Markov Model

Deep Learning

Discrete Cosine Transform

**ABSTRACT**

This writing investigates the classification of images into their classes or categories. We observed photographs that had been taken using a smartphone. These photos are then classified using two different approaches. The first method uses Hidden Markov Model (HMM) and the second technique employs Deep Learning with Convolution Neural Network (CNN) architecture. The same data are used to train both models. For HMM we use Discrete Cosine Transform to extract salient features of images. The number of training examples are smaller compared to the test set. Here we carried out a few shot classification. For recognition, Viterbi Algorithm is applied. Performances of both procedures were measured. HMM is slightly better with 98% accuracy, while CNN achieves 90%.

This is an open access article under the [CC BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/).

<figure><img src="image_3.png" /></figure>

**Corresponding Author:**

Zulkarnaen Hatala

Department of Information Technology

Politeknik Negeri Ambon

23240 Wailela Rumah Tiga, Ambon, Indonesia

Email: dzulqarnaenhatala@gmail.com

## 1. INTRODUCTION

Nowadays, photographic events are easily and freely captured using a mobile phone. A camera inside a smartphone is capable of shooting photos better and easier anywhere and anytime at no cost. Taking pictures has become a habit of people around the world nowadays. A single person can have an average of 640 photographs (photos) taken by his (or her) smartphone [1]. Besides its function as a camera, a smartphone integrates its capability for further processing and organizing of document images. A smartphone user with the profession of an educator such as a professor, lecturer, or teacher can keep student assignments using the phone. This advantage is achieved without the need to employ a dedicated flatbed paper scanner which is not mobile causing inflexibility. With an easy-to-use camera-based phone, resulting in so many images taken for each person, surely there is a need to efficiently manage and organize these images for many extended purposes. A professor needs to retrieve all students' papers and distinguish them from other photos such as panoramas, his or her family events, ceremonials, etc. Since the images in smartphones are massive, even if the classification of similar images into their category is quite easy, the repetition process could be exhaustive to be performed by humans. Smartphones like computers on the other side can be programmed to perform classification and they are faster and more accurate. Furthermore, the person is free to move his (or her) images into a better processing device like a laptop to achieve lesser execution time and higher accuracy.

The classification task of images falls into methods in the field of machine learning. These images that are taken by phone are rather challenging than images acquired using a dedicated flatbed scanner. This is because smartphone photos lack regularity and are distorted by the movement of human hands. The images also suffer from various lighting conditions.

<footnote n="1">¹Program Studi Teknik Informatika, Politeknik Negeri Ambon, Indonesia</footnote>

<footnote n="2">²Fakultas Teknik Industri, Universitas Diponegoro Semarang, Indonesia</footnote>
<figure><img src="image_4.png" /></figure>

Hidden Markov Model (HMM) [1], [2] is a statistical method that has been used in many real-world applications. Historically HMM has gained success in the field of speech-related topics, including recognition[2], and verification[3]. Lately, HMM has been applied in many image classification researches. HMM applied on face recognition to achieve reasonable computation was elaborated in [4]. Still in image recognition, HMM in conjunction with Neural Network (NN), Principle Component Analysis (PCA), and Gabor Filter (GF) is explored in [5]. HMM is also used to categorize images of clothing [6]. The classification of small low-resolution thumbnail images is done in [7]. On the other side, NN alone is also used extensively in image classification. Deep learning with a twin network is used to classify handwritten alphabets [8].

In this paper, the images taken using smartphones are categorized using two different approaches, Hidden Markov Model and Neural Network. When using HMM we use DCT as a feature extraction technique. The small number of DCT coefficients should capture a simple salient feature that mimics the human recognizing process or even further patterns missed by human eyes. The local similarity of neighboring rows and columns shows dependencies of inter-neighboring blocks inside an image. These facts motivate the classification of photographs using HMM. Another approach we used is a Neural network with Deep Learning [8] which is also applied in this research as a comparison to HMM.

## 2. METHOD FOR PHOTO CLASSIFICATION

This section describes a method to classify existing images using HMM. The stages are summarized below:

a. Image size normalization, photos are resized into constant width. Original images taken by smartphone are much bigger. The scaling of image size is intended to achieve faster processing time.

b. Feature extraction. An image is divided into blocks with constant width and height across the image. These blocks of images are nonoverlap. The upper left pixel of the block has X,Y coordinates. The blocks with the same Y coordinates are in the same row (blocks). Blocks in a column share the same X coordinates. The Discrete Cosine Transform (DCT) coefficients are calculated for each block [9], [10]. A single block is associated with single feature. So in a horizontal direction, since the image is normalized of constant width then the number of blocks along a row is constant. For a vertical direction, the numbers of rows vary depending on the image length. Every end of a horizontal row a special feature is added called a marker [11].

c. HMM Model training. We use a fixed class of HMMs to model along a row (horizontal direction of image). Each with 4 excitation states. These HMMS are to model all vector features along the row. Firstly a single HMM is initialized using Viterbi training algorithm [12], [13], [14] employed over all training data. Then this HMM is copied into six HMMS for each row. Later the Baum Welch algorithm are used to train individual HMM based on image data. Finally, all HMMs are joined (embedded) into a single large HMM and trained using all training set available.

d. Classification, Viterbi algorithm which is constrained to a recognition network for each category is performed against test images. This variant of Viterbi algorithm are called token passing [15], [16]. In case there are multiple recognition networks, Viterbi recognition algorithm will determine the probability of each test image against all available networks. Each network is associated with a specific category. The test image is decided to be of a category if it has the best viterbi score to an associated network of that category.

## 3. RESULTS AND DISCUSSION

Data taken using a smartphone camera are employed in this research. All images are taken using Whatsapp software, a social media application. The owner of the phone is a lecturer by profession. We limited the kind of images to only two categories, fingerprint images, and student submission papers. As an employee, the professor must log his finger into biometric machine every workday at the university. Since the university has not yet published the log confirmation, the professor also shoots photo for her/his own proof. In the second case, he/she regularly runs paper-based tests for a class exam session. Whenever he completes marking the paper he shoots a photo of the submission and secures it in her/his smartphone. The examples of these images are shown in Figure 2.
<figure><img src="image_5.png" /><figcaption>Figure 1. Two categories of images (a) student submission paper (b) fingerprint machine</figcaption></figure>

## HMM Methods:

All of these images are resized into 280 pixels. The original images are mostly 2160x3540 pixels in dimension, so the cropped images are 13% of the original size. With the block size of 4x4, we have 70 blocks for a single **row** along the horizontal direction. Whereas along the vertical direction of the image, the total number of rows varies, it depends on the original size of the image. For a single row, these 70 blocks will result 70 features just along that horizontal direction. A block is associated with a single feature. The single row has 70 **column** (block) vectors. The DCT coefficients are calculated for each block. Only six of the coefficients are used as described in [10]. Aside these six values, are also calculated 10 values of first and second derivatives between two consecutive observations. The results are 16 values which constitute a single feature vector. This vector is assumed to be emitted by one of the four states of an HMM.

All HMMs are consisted of 6 states with 4 excitation states. Other two states are state 1 for entry state and state 6 for the exiting (absorbing) state. At the propagation time, the HMM topology permits a state to stay in the same state and will jump only into its right neighbor state. Initially, an HMM has the probabilities of staying and move to its right set equally to 0.5 for states 2, 3, 4, and 5. All means and variances are initially set to 0.0 and 1.0. Another special ‘marker’ HMM [11] to represent end of the row of image is constructed with all variances are zero and its means is set to the biggest number. This marker pixels are placed at the rightest after every row of the image. The **marker** HMM has its parameters that constant and never altered during the training session.

Firstly, a single master HMM are initialized using Viterbi algorithm using all training set. The resulting HMM are copied into another HMMs. Next, another training algorithm called Baum Welch is applied on individual HMM using only its (isolated) labeled data of the segmented image. Last, in the training process is the embedded training process using Baum Welch where a single big HMM is constructed for each image. In this process the algorithm work simultaneously for every HMM for a single image.

Two HMM recognition networks are constructed, one is associated with the fingerprint board category and the other corresponds to student examination paper images. The classification is performed using Viterbi recognition algorithm [15]. For each test image, the best Viterbi path are found and the scores are calculated. The scores from different networks are compared. The category of an image is the one which is associated with the network that has the best score. Most of the process here is accomplished using the set of libraries and tools of Hidden Markov Tookit (HTK) [17].

## Neural Network method:

We used Siamese network [8], [18] to categorize photos. The twin network is an NN that applies deep learning [19] technique. The use of multiple hidden layers approach which was applied here is mainly described in [8]. The method in [8] requires fixed dimensions of width and height of the photos. After resizing proportionally of the original images, the images are optionally cropped or padded on the left and right sides. After padding with white pixels we get 280 of their widths. In this way, the dimension of images forwarded into input layers are kept constant at 498x280 pixels. The color of image is also passed as a 3rd dimension of the tensor [20]. This color information contains three channels for red, green and blue (RGB) respectively. These images are passed to the input layer, and then subsequently processed through types of layers. The network contains convolutional layers that perform automated feature extraction. [21]. This type of ANN also popular by its name Convolutional Network Network (CNN). We utilized tensorflow and keras libraries [22] to implement the whole model tasks which were training and classification. To train Siamese with only two images taken from each of two categories, we will have batch size of only six. After repeating
<figure><img src="image_6.png" /></figure>

this epoch about 7000 cycles, the resulting parameters of the network are then used to test the other 109 samples.

## Test Results:

We used 113 photos belonging into two categories, student answer sheets, and fingerprint board. 55 of the photos belong to the student answer sheet and the rest is of the board. From each of the two classes we only take two examples for training purposes. The results of the two-shot classification for both methods are presented in the table below. HMM use separated feature extraction outside its training process, while CNN feature extraction is embedded in the training process. From this table, HMM is slightly better than CNN.

<table><caption>Table 1. Classification results</caption><thead><tr><td>No</td><td>Method</td><td>Preprocessing</td><td>Feature Extraction</td><td>Number of class</td><td>Number of train photos</td><td>Number of test photos</td><td>Accuracy</td></tr></thead><tbody><tr><td>1</td><td>Hidden Markov Model</td><td>Convert images to black and white</td><td>DCT</td><td>2</td><td>4</td><td>109</td><td>98%</td></tr><tr><td>2</td><td>Siamese Network (CNN)</td><td>Left right Padding</td><td>Automatic</td><td>2</td><td>4</td><td>109</td><td>90%</td></tr></tbody></table>

## 4. CONCLUSION

The classification of personal images inside smartphones are useful task that can be done effectively using machine learning techniques. This research has put side by side two image classification methods, using HMM and Deep Learning Neural Network. Firstly using HMM, we use DCT as a feature extraction technique. While the feature extraction stage is necessary in HMM method, CNN contrary does not need this step. This extra process makes CNN outperform HMM in complexity of the procedures. In another aspect, CNN require higher processing speed and larger memory of computing devices, while HMM is generally faster. HMM also can tolerate slightly different pixel weights and widths of images, while CNN requires fixed dimensions for all training and test data. We note also that when we use HMM, training data must have a "label" manually to mark portions of images that are significantly different. Segmentation and labeling is not needed when using CNN. From the classification results, we note that HMM is slightly better than Siamese CNN.

## REFERENCES

[1] B. Mor, S. Garhwal, and A. Kumar, "A Systematic Review of Hidden Markov Models and Their Applications," *Arch. Comput. Methods Eng.*, vol. 28, no. 3, pp. 1429-1448, May 2021, doi: 10.1007/s11831-020-09422-4.

[2] L. R. Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition," *Proc. IEEE*, vol. 77, no. 2, 1989.

[3] I. B. Kadhim, A. N. Nasret, and Z. S. Mahmood, "Enhancement and modification of automatic speaker verification by utilizing hidden Markov model," *Indones. J. Electr. Eng. Comput. Sci.*, vol. 27, no. 3, p. 1397, Sep. 2022, doi: 10.11591/ijeecs.v27.i3.pp1397-1403.

[4] D. Ali, I. Touqir, A. M. Siddiqui, J. Malik, and M. Imran, "Face Recognition System Based on Four State Hidden Markov Model," *IEEE Access*, vol. 10, pp. 74436-74448, 2022, doi: 10.1109/ACCESS.2022.3188717.

[5] A. Aggarwal, M. Alshehri, M. Kumar, P. Sharma, O. Alfarraj, and V. Deep, "Principal component analysis, hidden Markov model, and artificial neural network inspired techniques to recognize faces," *Concurr. Comput. Pract. Exp.*, vol. 33, no. 9, p. e6157, 2021.

[6] S. Tena, R. Hartanto, and I. Ardiyanto, "Content-based image retrieval for fabric images: A survey," *Indones. J. Electr. Eng. Comput. Sci.*, vol. 23, no. 3, p. 1861, Sep. 2021, doi: 10.11591/ijeecs.v23.i3.pp1861-1872.

[7] M. Mouret, C. Solnon, and C. Wolf, "Classification of Images Based on Hidden Markov Models," in *2009 Seventh International Workshop on Content-Based Multimedia Indexing*, Chania, Crete: IEEE, Jun. 2009, pp. 169-174. doi: 10.1109/CBMI.2009.22.

[8] G. Koch, R. Zemel, R. Salakhutdinov, and others, "Siamese neural networks for one-shot image recognition," in *ICML deep learning workshop*, Lille, 2015.

[9] V. V. Kohir and U. B. Desai, "Face recognition using a DCT-HMM approach," in *Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201)*, Princeton, NJ, USA: IEEE Comput. Soc, 1998, pp. 226-231. doi: 10.1109/ACV.1998.732884.

[10] J. Li and R. M. Gray, *Image segmentation and compression using hidden Markov models*, vol. 571. Springer Science & Business Media, 2000.

[11] F. S. Samaria, "Face recognition using hidden Markov models," PhD Thesis, University of Cambridge Cambridge, UK, 1994.

[12] L. R. Rabiner, J. G. Wilpon, and B.-H. Juang, "A segmental k-means training procedure for connected word recognition," *ATT Tech. J.*, vol. 65, no. 3, pp. 21-31, 1986.

[13] G. D. Forney, "The viterbi algorithm," *Proc. IEEE*, vol. 61, no. 3, pp. 268-278, 1973.
<figure><img src="image_7.png" /></figure>

[14] A. Allahverdyan and A. Galstyan, "Comparative analysis of viterbi training and maximum likelihood estimation for higgs," *Adv. Neural Inf. Process. Syst.*, vol. 24, 2011.

[15] S. J. Young, N. H. Russell, and J. H. S. Thornton, "Token Passing: a Simple Conceptual Model for Connected Speech Recognition Systems".

[16] R. Solera-Ureña, J. Padrell-Sendra, D. Martín-Iglesias, A. Gallardo-Antolin, C. Peláez-Moreno, and F. Díaz-de-María, "SVMs for Automatic Speech Recognition: A Survey," in *Progress in Nonlinear Speech Processing*, vol. 4391, Y. Stylianou, M. Faundez-Zanuy, and A. Esposito, Eds., in Lecture Notes in Computer Science, vol. 4391, Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 190–216. doi: 10.1007/978-3-540-71505-4_11.

[17] S. Young et al., "The HTK book," *Camb. Univ. Eng. Dep.*, vol. 3, no. 175, p. 12, 2002.

[18] H. Shao, D. Zhong, X. Du, S. Du, and R. N. Veldhuis, "Few-shot learning for palmprint recognition via meta-siamese network," *IEEE Trans. Instrum. Meas.*, vol. 70, pp. 1–12, 2021.

[19] Y. Lai, "A comparison of traditional machine learning and deep learning in image recognition," in *Journal of Physics: Conference Series*, IOP Publishing, 2019, p. 012148.

[20] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexis, and C. Faloutsos, "Tensor decomposition for signal processing and machine learning," *IEEE Trans. Signal Process.*, vol. 65, no. 13, pp. 3551–3582, 2017.

[21] T. Wiatowski and H. Bölcskei, "A mathematical theory of deep convolutional neural networks for feature extraction," *IEEE Trans. Inf. Theory*, vol. 64, no. 3, pp. 1845–1866, 2017.

[22] O. Dürr, B. Sick, and E. Murina, *Probabilistic deep learning: With python, keras and tensorflow probability*. Manning Publications, 2020.