# Object-Based Image Retrieval Using the U-Net-Based Neural Network

Authors: Sandeep Kumar; Arpit Jain; Ambuj Kumar Agarwal; Shilpa Rani; Anshu Ghimire
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Day by day, all the research communities have been focusing on digital image retrieval due to more internet and social media uses. In this paper, a U-Net-based neural network is proposed for the segmentation process and Haar DWT and lifting wavelet schemes are used for feature extraction in content-based image retrieval (CBIR). Haar wavelet is preferred as it is easy to understand, very simple to compute, and the fastest. The U-Net-based neural network (CNN) gives more accurate results than the existing methodology because deep learning techniques extract low-level and high-level features from the input image. For the evaluation process, two benchmark datasets are used, and the accuracy of the proposed method is 93.01% and 88.39% on Corel 1K and Corel 5K. U-Net is used for the segmentation purpose, and it reduces the dimension of the feature vector and feature extraction time by 5 seconds compared to the existing methods. According to the performance analysis, the proposed work has proven that U-Net improves image retrieval performance in terms of accuracy, precision, and recall on both the benchmark datasets.


---

## Body

## 1. Introduction

Nowadays, digital image techniques lead to the tremendous usage of the image retrieval process on the internet. The image retrieval system retrieves different images over the internet with different captions and labels under each image stored in the database. An image retrieval system that uses content as a search key for browsing is known as content-based image retrieval (CBIR) [1]. The main goal of the CBIR methodology is to extract meaningful information from images such as color shape and texture for effective retrieval. The research community contributed to CBIR in the direction of image properties, relevance feedback, fuzzy color, and texture histogram [2]. The proposed algorithms, color histogram, based on relevant image retrieval (CHRIR) [3, 4], work with the image’s low-level features, such as objects’ physical features for image retrieval. However, these visual features might not reveal the proper semantics of the image. These algorithms may not suit and may generate erroneous results when considering content images in a broad database. Therefore, to improve the CBIR system’s accuracy, region-based image retrieval methods using image U-Net-based segmentation were introduced [5]:(i)
Haar discrete wavelet transform (H-DWT) is a popular transformation technique that transforms any image from the spatial domain to frequency domain. The wavelet transformation method represents a function as a family of essential functions termed wavelets [2, 6, 7]. Wavelet transform extracts signals at different scales while input passes through the low-pass and high-pass filters. Wavelets are increasingly becoming popular because of their multiresolution capability and suitable energy compaction property. Haar wavelet is used to represent an image by computing the wavelet transform. It involves low-pass filtering as well as high-pass filtering operations simultaneously [8]. At each scale, the image is decomposed into four frequency sub-bands, namely LowLow, LowHigh, HighLow, and HighHigh, where Low stands for low frequency and High stands for high frequency. Haar wavelet’s function X(t) can be described as(1)Xt=1,0≤t≤0.5,−1,0.5≤t<1,0,otherwise.Its scaling functionχ(t) can be defined as χ(t) = 1,0≤t<1,0,otherwise.(ii)
Lifting scheme: it is a well-known approach used for the second generation wavelets [5]. It has much potential in CBIR because of its simple structure, low complexity in computation, convenient construction, etc. It has proved its potential in performing iterative primal lifting and dual lifting [9–11] with multiresolution analysis. Using a lifting scheme, we can build wavelets having more vanishing moments and smoothness, thus enabling them to be more adaptable and nonlinear. The lifting scheme is used for designing wavelets and performing wavelet transformation techniques such as discrete wavelet transform (DWT).Most of the traditional techniques used machine learning techniques, and these techniques work on the whole image, making it a more time-consuming process. Therefore, this paper proposed a U-Net-based neural network for segmentation purposes and Haar DWT and lifting wavelet schemes were used for feature extraction in content-based image retrieval (CBIR). Haar wavelet is preferred as it is easy to understand, very simple to compute, and the fastest. U-Net-based neural network (CNN) gives more accurate results than the existing methodology because deep learning techniques can extract low-level and high-level features from the input image, which is the novelty of this research. In Section2, we presented a literature survey. In Section 3, we explained our proposed architecture and methodology. Section 4 discusses the results of 2 benchmark datasets, and Section 5 represents the conclusion section.


## 2. Literature Work

Digital image retrieval and its applications are vast to study. There are many traditional techniques in image retrieval, but the key issue is that various techniques may have different types of variations, i.e., accuracy, error, and detection rate. Many research communities [1–16] proved that object detection and image retrieval error rate are less, as shown in Table 1.Table 1
Comparison of the existing image retrieval techniques.
S. no.AuthorYearMethod/methodologyComments1Liying and Lirong [13]2017Scale-invariant feature transform (SIFT)Performs retrieval on more than a single query to increase the retrieval accuracy.Convolutional neural network (CNN)2Dhotre et al. [4]2017The color feature extracted through CHRIR and wavelet transform performed using multilevel Haar wavelet transform (MHWT)Proved to be a faster retrieval method on an image database with one of the physical features. Works more accurately with increased retrieval speed and minimized time.3Jayanthi and Karthikeyan [2]2015FCTH, CEDD, HWT, and DWT using fuzzy linking and Gabor filtersDatabase with 1000 color images results in better recall and average precision of retrieval.4Thepade and Shinde [3]2015Haar wavelet transforms with Canny edge detection based on shape features using gradient techniques such as Prewitt, Laplace, and Sobel, and the slope magnitude technique with the Manhattan similarity functionDatabase with 350 color imagesFrei-Chen and Sobel give better performance than the other algorithms that used Canny implementation.5Jayanthi and Karthikeyan [2]2015HWT and DWT using Gabor filters and fuzzy linkingGives good results in average precision and recall value.6Gupta and Kushwah [8]2015Haar discrete wavelet transform (H-DWT), gray level co-occurrence matrix (GLCM)Improved results in comparison with previous methods.Support vector machine (SVM)7Agarwal et al. [5]2014Used color edge detection and DWTDatabase used was Wang’s image database. Gave high precision and recall values.8Agarwal et al. [5]2014Color edge detection, DWTHigh precision and recall indicate an exemplary retrieval systemCanny edge detection9Ying Chen and Wu [6]2011Uses optical flow to extract the information from the video; extraction process with Haar waveletLocates the feature almost near the query point using index values.10Chatzichristofis et al. [7]2010The method based on color and edge directivity descriptor (CEDD)Demonstrated successful retrieval on benchmark datasets.Utilizes the binary Haar wavelet transform for extraction11Quellec et al. [12]2010The multidimensional wavelet filter bankCan be used in a different dimension of signal and different lattices.12Verma et al. [1]2009Texture analysis-based schemeOwn dataset with 100 color images of size 256 × 256 pixels each. Accuracy up to 73%.I level Haar wavelet used for image decomposition F-norm theory to decrease the dimension of the extracted feature; fuzzy logic similarity measure used13Huang et al. [10]2005Lifting scheme F-norm theoryGood at retrieval.14Wong et al. [9]2005AdaBoost-based face defection method and the lifting wavelet transform (LFWT) techniqueEfficient with small memory to detect the face. Suits best for multimedia applications15Munjal and Bhatia [17]2019UCID dataset used, CEDD and Gabor wavelet transform (GWT) for feature vectorAccuracy = 91.9%.16Varish et al. [18]2020Fusion of histograms of gradients and invariant moments, Corel 1K and GHIM-10K dataset used for validationPrecision = 89% and 90%.Recall = 17.80% and 3.60.F-score = 29.49% and 6.92%.17Wadhera and Agarwal [19]20203D center symmetric LBP + Gaussian filter + gray level co-occurrence matrix (GLCM), STex texture, and ESSEX face databasePrecision = 61% and 97.4%.Retrieval rate = 97.4%.18Ajam et al. [20]2019LBP + HSV + entropy, Corel 10K and Corel 5K databases usedRetrieval rate = 59.51% and 49.13%.19Xiaobo et al. [21]2021Adaptive threshold + directional LBP, and Corel 1K databasePrecision = 67.63%.


## 3. Proposed Methodology

In Figure1, the flowchart describes the proposed methodology in which the image retrieval has to be done. The following steps explain the proposed methodology.Figure 1
Flow chart of the proposed system.


### 3.1. Image Acquisition

An image is taken as input which has to be converted into a grayscale image. The converted image is then sent to the preprocessing step for further process. In the acquisition process, the image with Real-World Data is converted into an array of numerical data. The image must be captured with the appropriate camera and converted into a computerized pattern [22–24].


### 3.2. Preprocessing

Preprocessing is performed to remove distortions and other unwanted features while processing the image and extract the proper portion of the image corresponding to the analysis of image retrieval using different algorithms [25–27] such as boundary detection. Preprocessing involves removing unwanted features, resizing the image, boundary detection, and normalization. The image is processed through different phases in preprocessing, such as resize, boundary detection, and normalization.


### 3.3. Segmentation

There are various traditional methods to normalize the image for segmentation, but the U-Net-based neural network detects the object more efficiently.The proposed methodology used 3-layer U-Net architecture, and it is one of the fully convolutional neural networks which works with very few training models yet yields compelling segmentation results. U-Net consists of a 3-layer convolution neural network, ReLU functions, and pooling functions, and in each layer, the pooling operations are replaced by the upsampling operators such that the network’s output gives an image with increased resolution. U-Net performs the classification on every pixel and generates the output with the same size as the input. The U-Net architecture is symmetric and usually has a U shape. The left side of the network is a contracting network, and the right is expanding network. The architecture is shown in Figure2. Downsampling will be done on the left side of the U-Net architecture, and on the right side, upsampling will be done. Each block in the architecture takes an input and passes through 2 convolution layers, 3 × 3 with a stride of 2 and 2 × 2 max-pooling with the corresponding cropped feature map. Table 2 shows the full description of the input image with 3 phases of downsampling and upsampling. Once the image is segmented accurately and features can be extracted, the encoding path, i.e., downsampling, is passed through 3 × 3 × 3 convolutions. It is followed by ReLU (rectified linear unit) operation with 16 channels and 2 × 2 × 2 max-pooling with stride 2. It consists of 3 phases/layers of convolution. At each layer, the feature channels get doubled. In total, 11 convolution layers were taken.Figure 2
U-Net-based architecture diagram.Table 2
Complete description of U-Net-based architecture.
PhasesInput imageConvolution layersChannelsReLU and max-poolingOutput imageSampling typeStride164 × 64 × 403 × 3 × 3162 × 2 × 232 × 32 × 20Downsampling2232 × 32 × 203 × 3 × 3322 × 2 × 216 × 16 × 10Downsampling2316 × 16 × 103 × 3 × 3642 × 2 × 216 × 16 × 10Downsampling2116 × 16 × 103 × 3 × 3322 × 2 × 232 × 32 × 20Upsampling2232 × 32 × 203 × 3 × 3642 × 2 × 264 × 64 × 40Upsampling2364 × 64 × 401 × 1 × 1642 × 2 × 264 × 64 × 1Desired sample2


### 3.4. Feature Extraction

In this process, the image is reduced using classification to more manageable parts stored as a dataset for further image processing [2, 28–31]. The process is as shown in Figure 3. These large data sets contain many variables that must be processed and need many computing resources to process.Figure 3
Flow chart of feature extraction technique.The method of feature extraction used may alter depending upon the traditional and nontraditional methods [32–35]. After segmentation, the YUV component of the input image is extracted as shown in Figure 4. Once the YUV component was extracted, the Sobel and Canny edge detection and wavelet transformation were applied for in-depth feature extraction [36–40]. The entire sequence of feature extraction is shown in Figure 3.Figure 4
YUV component of the input image.


### 3.5. Classification

The extracted data will be in binary format, stored in the database during the enrollment process, or verified with the existing data during the matching process [41–47]. If the similarity index of the image is more, then a similar kind of image will be retrieved. The similarity distance is estimated by Manhattan distance, Euclidean distance and Chebyshev rule. The mathematical formulations are shown as follows:(2)DM=x1−x2+y1−y2,DE=x1−x22+y1−y22,DCheb=maxx1−x2,y1−y2.


## 3.1. Image Acquisition

An image is taken as input which has to be converted into a grayscale image. The converted image is then sent to the preprocessing step for further process. In the acquisition process, the image with Real-World Data is converted into an array of numerical data. The image must be captured with the appropriate camera and converted into a computerized pattern [22–24].


## 3.2. Preprocessing

Preprocessing is performed to remove distortions and other unwanted features while processing the image and extract the proper portion of the image corresponding to the analysis of image retrieval using different algorithms [25–27] such as boundary detection. Preprocessing involves removing unwanted features, resizing the image, boundary detection, and normalization. The image is processed through different phases in preprocessing, such as resize, boundary detection, and normalization.


## 3.3. Segmentation

There are various traditional methods to normalize the image for segmentation, but the U-Net-based neural network detects the object more efficiently.The proposed methodology used 3-layer U-Net architecture, and it is one of the fully convolutional neural networks which works with very few training models yet yields compelling segmentation results. U-Net consists of a 3-layer convolution neural network, ReLU functions, and pooling functions, and in each layer, the pooling operations are replaced by the upsampling operators such that the network’s output gives an image with increased resolution. U-Net performs the classification on every pixel and generates the output with the same size as the input. The U-Net architecture is symmetric and usually has a U shape. The left side of the network is a contracting network, and the right is expanding network. The architecture is shown in Figure2. Downsampling will be done on the left side of the U-Net architecture, and on the right side, upsampling will be done. Each block in the architecture takes an input and passes through 2 convolution layers, 3 × 3 with a stride of 2 and 2 × 2 max-pooling with the corresponding cropped feature map. Table 2 shows the full description of the input image with 3 phases of downsampling and upsampling. Once the image is segmented accurately and features can be extracted, the encoding path, i.e., downsampling, is passed through 3 × 3 × 3 convolutions. It is followed by ReLU (rectified linear unit) operation with 16 channels and 2 × 2 × 2 max-pooling with stride 2. It consists of 3 phases/layers of convolution. At each layer, the feature channels get doubled. In total, 11 convolution layers were taken.Figure 2
U-Net-based architecture diagram.Table 2
Complete description of U-Net-based architecture.
PhasesInput imageConvolution layersChannelsReLU and max-poolingOutput imageSampling typeStride164 × 64 × 403 × 3 × 3162 × 2 × 232 × 32 × 20Downsampling2232 × 32 × 203 × 3 × 3322 × 2 × 216 × 16 × 10Downsampling2316 × 16 × 103 × 3 × 3642 × 2 × 216 × 16 × 10Downsampling2116 × 16 × 103 × 3 × 3322 × 2 × 232 × 32 × 20Upsampling2232 × 32 × 203 × 3 × 3642 × 2 × 264 × 64 × 40Upsampling2364 × 64 × 401 × 1 × 1642 × 2 × 264 × 64 × 1Desired sample2


## 3.4. Feature Extraction

In this process, the image is reduced using classification to more manageable parts stored as a dataset for further image processing [2, 28–31]. The process is as shown in Figure 3. These large data sets contain many variables that must be processed and need many computing resources to process.Figure 3
Flow chart of feature extraction technique.The method of feature extraction used may alter depending upon the traditional and nontraditional methods [32–35]. After segmentation, the YUV component of the input image is extracted as shown in Figure 4. Once the YUV component was extracted, the Sobel and Canny edge detection and wavelet transformation were applied for in-depth feature extraction [36–40]. The entire sequence of feature extraction is shown in Figure 3.Figure 4
YUV component of the input image.


## 3.5. Classification

The extracted data will be in binary format, stored in the database during the enrollment process, or verified with the existing data during the matching process [41–47]. If the similarity index of the image is more, then a similar kind of image will be retrieved. The similarity distance is estimated by Manhattan distance, Euclidean distance and Chebyshev rule. The mathematical formulations are shown as follows:(2)DM=x1−x2+y1−y2,DE=x1−x22+y1−y22,DCheb=maxx1−x2,y1−y2.


## 4. Results

Overall, GUI is prepared for the proposed work using the MATLAB 2014a tool by taking input of the image as a query image as shown in Figure3. The entire work was performed on a laptop with the configuration of Intel I3, of NVIDIA graphics card with 4 GB RAM. Various hyperparameters are used in the architecture. A total of 50 epochs is used while training the model. The validation split is considered as 0.1. It considers 90% of the images for the training purpose and 10% for the testing purpose. The dropout value is 0.2; it means that of five inputs, one is excluded from each cycle. Sixteen filters are used for the convolution purpose, and the learning rate lies between 0 and 1. For evaluation of the proposed work, the Corel 1K database and Corel 5K database cover many semantic categories, as shown in Figures 5 and 6. These datasets are widely used for content-based image retrieval techniques. Totally 10800 images are available in the Corel 1K dataset, and they are divided into 80 different groups according to the various categories. The database includes butterflies, horses, bushes, flowers, etc., and each category contains more than 100 images. The users determine the partitioning of the database into meaningful featured categories because of image similarity.Figure 5
Sample format of the Corel 1K/Wang database (people, beach, buildings, buses, dinosaurs, elephants, flowers, horses, and mountains).Figure 6
Sample format of the Corel 5K database (butterfly, bear, sunset, card, symbol, tiger, sippy, people, vegetable, cloud, beers, trees, doors, cars, bridges, waterfalls, bullock cart, revolver, pattern, flag, and art).Figure7 shows the overall GUI of the proposed work. The proposed work gives high accuracy, precision, and recall up to 93.1%, 99.77%, and 87.23%, and 88.39%, 84.75%, and 81.01%, respectively, for Corel 1K and Corel 5K datasets, as shown in Table 3 as well as Figures 8–10. Based on the novel proposed work, feature extraction time reached 4.187 sec, as shown in Table 4 and Figure 11. In Corel 1K benchmark datasets, nine samples were considered for evaluation and similarity matrices of these datasets are shown in Table 5 and Figure 12. In Corel 5K benchmark datasets, 21 various samples were considered for evaluation and similarity matrices of these datasets are shown in Table 6 and Figure 13. The overall precision values of the proposed work are high compared with the existing methodology, and the results are shown in Table 7 and Figure 14. The mathematical formulation of the parameters is shown as follows:(3)accuracy=100∗TP+TNN,recall=TPTP+FP,precision=TPTP+FN,where TP is true positive, TN is true negative, FN is false negative, FP is false positive, and N is the size of the dataset.Figure 7
GUI of the proposed work.Table 3
Average accuracy, precision, and recall of the retrieval image.
DatasetMTSD accuracy (%)Proposed accuracy (%)MTSD precision (%)Proposed precision (%)MTSD recall (%)Proposed recall (%)Corel 1K68.4993.0156.3491.77—87.23Corel 5K53.3988.3954.4884.75—81.01Figure 8
Accuracy comparison of the proposed work with the existing methodology.Figure 9
Precision comparison of the proposed work with the existing methodology.Figure 10
Recall comparison of the proposed work with the existing methodology.Table 4
Dimension (D) of the feature vector and feature extraction time.
MethodsDimensionFeature extraction time (sec)MTSD1799.1615Proposed2204.1815Figure 11
Dimension and time comparison of the proposed work with the existing methodology.Table 5
Similarity metrics on the Corel 1K dataset.
GroupL1EuclideanChebyshevPeople91.0579.8191.76Beach87.8789.2789.60Buildings83.8987.9988.82Buses79.7079.9290.36Dinosaurs89.1890.5990.01Elephants89.6690.7788.16Flowers90.7180.3989.12Horses89.7391.7287.36Mountain87.5790.8989.81Figure 12
Similarity metrics on the Corel 1K dataset.Table 6
Similarity metrics on the Corel 5K dataset.
GroupL1EuclideanChebyshevButterfly81.0579.2482.29Bear82.7076.9879.86Sunset80.9075.1570.18Card77.6479.1173.56Symbol83.5781.3679.09Tiger81.1679.1182.36Sippy83.9180.0883.01People83.6381.1081.73Vegetable79.5179.0680.17Cloud84.1080.2582.97Beers82.1981.3683.97Trees80.9078.1573.00Doors78.1871.0274.08Cars79.2674.0874.83Bridges76.8271.5976.59Waterfalls80.4576.7881.08Bullock cart83.6779.7181.11Revolver84.0180.1883.31Pattern79.1177.0177.08Flag78.0876.873.12Art74.1979.7070.14Figure 13
Similarity metrics on the Corel 5K dataset.Table 7
Average precision comparison of the proposed work with the existing methodology.
DatasetPrecision of [37] (%)Precision of [38] (%)Precision of [39] (%)Precision of [40] (%)Precision of [41] (%)Proposed precision (%)Corel 1K75.0182.0979.969.4076.6091.77Corel 5K——62.9753.9059.1084.75Figure 14
Precision comparison of the proposed work with the existing methodology.


## 5. Conclusion

The U-Net-based architecture makes the proposed work different from the existing methods, giving a high detection rate. The U-Net-based neural network detects the object more efficiently. It is a fully convolutional neural network (CNN) that works with very few training models yet yields compelling segmentation results. It is a three-layered segmentation architecture that improves the overall accuracy of our content-based image retrieval system by around 93%. For evaluation of the proposed work, the Corel 1K database and Corel 5K database are used. The results show that accuracy and precision are very high compared to the existing methodology. The feature extraction time of our proposed methodology is also significantly less compared to the MTSD method. Hence, we can conclude that our proposed methodology is very fast, accurate, efficient, and precise compared to the MTSD method.



---

*Source: 4395646-2021-11-10.xml*