# Segmentation Methods for Image Classification Using a Convolutional Neural Network on AR-Sandbox

Andres Ovidio Restrepo Rodriguez, Daniel Esteban Casas Mateus, Paulo Alonso Gaona-García, Adriana Gomez Acosta, Carlos Enrique Montenegro Marin

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Andres Ovidio Restrepo Rodriguez, Daniel Esteban Casas Mateus, Paulo Alonso Gaona-García, Adriana Gomez Acosta, Carlos Enrique Montenegro Marin. Segmentation Methods for Image Classification Using a Convolutional Neural Network on AR-Sandbox. 15th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), May 2019, Hersonissos, Greece. pp.391-398, 10.1007/978-3-030-19823-7_33. hal-02331321

HAL Id: hal-02331321

https://inria.hal.science/hal-02331321

Submitted on 24 Oct 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Segmentation methods for image classification using a Convolutional Neural Network on AR-Sandbox

Andres Ovidio Restrepo Rodriguez¹, Daniel Esteban Casas Mateus¹, Paulo Alonso Gaona Garcia¹, Adriana Gomez Acosta² and Carlos Enrique Montenegro Marin¹

¹ Universidad Distrital Francisco Jose de Caldas, Bogota, Colombia {aorestrepor, decasasm}@ correo.udistrital.edu.co, {pagaonag, cemontenegrom}@udistrital.edu.co

² Fundación Universitaria San Mateo, Bogota, Colombia acgomez@funsanmateo.edu.co

**Abstract:** Fields such as early education and motor rehabilitation provide a space for their integration with augmented reality devices as AR-Sandbox, in order to provide support to these fields, generating a feedback of tasks carried out through the recognition of images based on convolutional neural networks. However, the nature of the AR-Sandbox generates a high noise level of the acquired images, for this reason the present study has as purpose the implementation and comparison of three segmentation methods (Canny Edge Detector, Color-space and Threshold) for the training and prediction phase of a convolutional neural network model previously established. When carrying out this study, it was obtained that the combined model with color-space segmentation presents an average percentage of 99% performance for the classification of vowels, described by the AUC of the ROC curve, this being the model with the best performance.

**Keywords:** Canny Edge Detector, Color-space, Convolutional Neural Networks, Image processing, Segmentation, Threshold, Vowels.

## 1 Introduction

The AR-Sandbox is an augmented reality device, which, through the projection of digital topographic maps, allows to close the gap between 2D and 3D visualization, improving the thought and spatial modeling skills. This device has a large field of action, providing support and revolutionizing the traditional paradigm of education in early childhood and in medicine such as rehabilitation through motor therapy [1], for this, it is important that the AR-Sandbox provide feedback of the activity carried out.

The image recognition on the AR-Sandbox presents a high degree of difficulty, since when capturing the image from this device a high level of noise is presented, due to the multiple pigmentations according to the level of the sand, exposed by the nature of this augmented reality tool. The AR-Sandbox makes use of a Kinect and a projector to make colorful representations in 3D on a sandbox [2].

This article presents preliminary results for model implementation results of a convolution neural network (CNN), as an image recognition technique on the AR-Sandbox, evaluating the application of three segmentation methods to the training dataset and the test dataset,
for the training and prediction phase, in order to determine which segmentation method best
fits the characteristics of the AR-Sandbox.

## 2 Background

Previous image processing besides image recognition through convolutional neural
networks have had a large number of applications, which its goal is to optimize pattern
recognition [3]. From different computer vision techniques, convolutional neural networks
have been in a continuous growth since their creation [4].

Image segmentation methods are used to optimize the analysis and interpretation of images
through a process depending on the segments of each image, there are a variety of
algorithms and methods with advantages and disadvantages depending on the features of
the image, these methods are classified depending on pixel intensity value, color, texture,
etc. [5]. Authors like [6] use image segmentation in order to get information for underwater
images, which are difficult to get and analyze due to environmental conditions, for these
reasons, canny edge detector algorithm is used because it gave localization and response
with extreme accuracy and it has a superior capability under noise conditions.

Finally, in previous studies, [7] where the purpose of this research was to establish a model
of a CNN for the classification of geometric figures by optimizing hyperparameters using
random search, evaluating the impact of the implementation of a previous phase of color-
space segmentation to a set of tests captured from the AR-Sandbox, authors find that using
the proposed method, an average decrease of 39.45% to a function of loss and an increase
of 14.83% on average in the percentage of correct answers is presented.

Based on this panorama, the present study intends to carry out the recognition of images by
the means of deep learning techniques, such as convolutional neural networks and image
processing by three segmentation methods, with the purpose of determining the
performance variation of a convolutional neural network, applying these segmentation
methods the test and training datasets. In the next section, the selected study scenario is
presented.

## 3 General approach

Figure 1, presents the general approach of this study, which has as its starting point a base
model of a convolutional neural network. The study starts from previous results of [7] that
had the purpose of creating contributions towards image recognition based on immersive
techniques, through deep learning using CNN, hyperparameters optimization and image
processing. From this base model, CNN is trained, using a training dataset composed of a
series of vowels (A, E, I, O, U), which will be previously put through image processing,
through Python and OpenCV, using three segmentation methods, Canny Edge Detector [6], color-space segmentation [7] and Threshold [8]. When obtaining the trained model, we proceed to make the model evaluation, making use of a test dataset, which was acquired in its entirety through the AR-Sandbox. The same segmentation method applied in the training phase is applied previously to the test dataset, for the evaluation phase of the model the following evaluation items are defined: Confusion Matrix, and the ROC Curves.

<figure><img src="image_3.png" /><figcaption>Figure 1. General approach</figcaption></figure>

## 4 CNN base model

In order to carry out the recognition and classification of vowels from the AR-Sandbox, a model of a convolutional neural network was selected [7], where the definition of a base model of a CNN was carried out, in order to perform the hyperparameters optimization by random search, defining a dictionary of hyperparameters. The model selected was for the recognition of geometric figures acquired from the AR-Sandbox. The selected model is presented in figure 2.

<figure><img src="image_4.png" /><figcaption>Figure 2. CNN model</figcaption></figure>
# 5 Image Processing

As a first step, the acquisition of images was done through the AR-Sandbox, which aims to support early education and rehabilitation in motor therapy, in order to change the traditional paradigm of exercises. Next, the segmentation methods and their process are presented.

## 5.1 Canny Edge Detector

As a first step, we proceed to select the original image, later the image has to be resized, in order to apply Gaussian filter and Sobel procedure, finally, Non maximum suppression and hysteresis are applied to image [9]. Figure 3 presents the process.

<figure><img src="image_5.png" /><figcaption>Figure 3. Canny Edge Detector segmentation</figcaption></figure>

## 5.2 Color-space

As a first instance, the target color and a range of close colors are selected, in order to separate the colors in the image, therefore, the values within these parameters are painted of white color and the rest of a black color, this is called mask, which provides a contrast in the image. The white part of the mask will take the green color while the rest of the image is still black. Figure 4 presents the process.

<figure><img src="image_6.png" /><figcaption>Figure 4. Color-space segmentation</figcaption></figure>

## 5.3 Threshold
As a first step, the original selected image must be passed to grayscale, later you have to apply the threshold function using Otsu binarization [10]. Figure 5 presents the process.

<figure><img src="image_7.png" /><figcaption>Figure 5. Threshold segmentation</figcaption></figure>

## 5.4 Similarity Indexes

In order to obtain the probability of similarity between the original image and the segmented image using the methods mentioned above, we proceed to calculate the coefficients of Jaccard and Sørensen-Dice, making use of Matlab, where the individual coefficient of each image is obtained in order to obtain the average value of each of the coefficients for each segmentation method. The obtained coefficients are presented in Table 1.

<table><caption>Table 1. Similarity indexes</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Index</th></tr><tr><th>Jaccard</th><th>Sørensen-Dice</th></tr></thead><tbody><tr><td>Canny</td><td>0.3239</td><td>0.4356</td></tr><tr><td>Color-space</td><td>0.7143</td><td>0.7801</td></tr><tr><td>Threshold</td><td>0.4869</td><td>0.6244</td></tr></tbody></table>

# 6 Results

In this section the results are shown, in the first part the results show the ROC curves of the used test dataset and finally the models are compared by success rate obtained from the confusion matrices of each segmentation method.

To acquire the images and make the test dataset of images which are made in the AR-Sandbox, it was necessary to convene a group of ten people who made a total of 15 images each, which are 3 for each class.

## 6.3 ROC curves

Figure 6A shows the ROC curves for the dataset with color-space, in Figure 6B the ROC curve for the dataset with Canny Edge Detector is presented, in Figure 6C the curve for the
dataset with Threshold and Figure 6D the curve without a segmentation method. Each of these figures has seven curves which are areas under the curve of the micro and macro level of the ROC curve, in addition the curves corresponding to each vowel represented by a number from 1 to 5 in alphabetical order.

<figure><img src="image_8.png" /><figcaption>Figure 6. ROC curves.</figcaption></figure>

## 6.2 Confusion matrix

With the help of the Python sk-learn library an algorithm is implemented for the creation of confusion matrices that will show the success rate of each method with respect to the test dataset. Table 2, shows the percentage of success rate of each segmentation method.

<table><caption>Table 2. Success Rate segmentation methods</caption><thead><tr><th rowspan="2"></th><th colspan="4">Method</th></tr><tr><th>Without Segmentation</th><th>Canny</th><th>Color-space</th><th>Threshold</th></tr></thead><tbody><tr><th>Success Rate</th><td>57.33%</td><td>87.33%</td><td>92.67%</td><td>58.00%</td></tr></tbody></table>

## 7 Analysis of results

Regarding the confusion matrix, the highlighted segmentation method is color-space with 92.67% having a difference of 5.34% with Canny Edge Detector, 34.67% with Threshold and 57.33% without segmentation method.
As evidenced in figure 6, the model that stands out with respect to the AUC, making the average between micro average and macro average, is the Color-space model with a 0.99 performance, followed by Canny Edge Detector with a 0, 97, Threshold with 0.835 and finally the model without segmentation method with 0.73. In addition, it can be seen that the model with color-space segmentation increases the precision value by 0.26. Table 3 presents a detailed analysis by vowel.

<table><caption>Table 3. Specific analysis: ROC curve</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2"></th><th colspan="5">Vowel</th></tr><tr><th>A</th><th>E</th><th>I</th><th>O</th><th>U</th></tr></thead><tbody><tr><td>Canny</td><td rowspan="3">AUC</td><td>0,95</td><td>0,99</td><td>0,96</td><td>0,98</td><td>0,99</td></tr><tr><td>Color-Space</td><td>0,98</td><td>1,00</td><td>0,98</td><td>0,99</td><td>0,99</td></tr><tr><td>Threshold</td><td>0,89</td><td>0,99</td><td>0,92</td><td>0,59</td><td>0,84</td></tr><tr><td>Without segmentation</td><td></td><td>0,87</td><td>0,71</td><td>0,92</td><td>0,72</td><td>0,82</td></tr></tbody></table>

## 8 Conclusions

From the analysis presented and taking as criteria the ROC curve and confusion matrix, it was determined that the segmentation method with the highest performance for the recognition of vowels on CNN is Color-space, given the multiple pigmentation characteristics of the AR-Sandbox, the choice of a target color represents a greater percentage of correct answers in its prediction phase. By applying the same segmentation method in the training and prediction phase, positive contribution is made in the identification and extraction of characteristics for the classification of images, as mentioned by the authors in [7].

On the other hand, it is determined that the similarity indexes are not decisive for the choice of a segmentation method for the recognition of images by CNN in an AR-Sandbox, since as can be seen the Canny Edge Detector method presents the lowest coefficients of similarity contradicting the results of confusion matrices and ROC curves that show a high performance of the CNN trained with this segmentation method.

From a model with a high level of precision for objects identification from AR-Sandbox, one can begin to expand the spectrum of functionality and use this augmented reality device in early education and rehabilitation.

According to the above, the combination of multimedia, deep learning together with image processing provides an opportunity to perform a multidisciplinary work in parallel with traditional paradigms in the fields of education and medicine.