# Refined Color Texture Classification Using CNN and Local Binary Pattern

Authors: Khalid M. Hosny; Taher Magdy; Nabil A. Lashin; Kyriakos Apostolidis; George A. Papakostas
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Representation and classification of color texture generate considerable interest within the field of computer vision. Texture classification is a difficult task that assigns unlabeled images or textures to the correct labeled class. Some key factors such as scaling and viewpoint variations and illumination changes make this task challenging. In this paper, we present a new feature extraction technique for color texture classification and recognition. The presented approach aggregates the features extracted from local binary patterns (LBP) and convolution neural network (CNN) to provide discriminatory information, leading to better texture classification results. Almost all of the CNN model cases classify images based on global features that describe the image as a whole to generalize the entire object. LBP classifies images based on local features that describe the image’s key points (image patches). Our analysis shows that using LBP improves the classification task when compared to using CNN only. We test the proposed approach experimentally over three challenging color image datasets (ALOT, CBT, and Outex). The results demonstrated that our approach improved up to 25% in the classification accuracy over the traditional CNN models. We identify optimal combinations for each dataset and obtain high classification rates. The proposed approach is robust, stable, and discriminatory among the three datasets and has shown enhancement in classification and recognition compared to the state-of-the-art method.


---

## Body

## 1. Introduction

The motivation of the usage of CNN models as features descriptors is the ability of the deep neural network to capture the high-level features that can be a key point for the classification of the texture images. Texture plays a significant role in distinguishing objects in color images. Texture classification involves a two-phase process. The first phase is extracting the features, which provides a feature-based description for each texture type; this phase tends to select features unaffected by image transformation, such as scaling, translation, and rotation. The second phase tends to recognize the texture from the extracted features. Texture recognition is increasingly set as a critical issue in computer vision that has many applications such as biomedical image processing [1–5], object detection [6–8], and remote sensing [9–11] application fields.The scale-invariant feature transform (SIFT), proposed by Lowe [12], is a common sparse descriptor. Mikolajczyk and Schmid changed the SIFT descriptor by altering the gradient location orientation grid and the quantization parameters of the histograms [13]. Chen et al. [14] proposed a robust discriminative descriptor called Weber local descriptor (WLD) based on human perception. Gabor wavelet descriptor is considered one of the most widely used dense descriptors [15]. The Gabor wavelet has extensively used image recognition applications such as face recognition, scene analysis, motion tracking, and face recognition [16]. Ershad and Fekri [17] proposed a new approach to analyze the texture based on its grain components and classify it from grain components histogram and statistical features. Local binary pattern (LBP) is one of the texture descriptor methods used to describe and represent texture. Ojala et al. [18] presented it as an adequate gray-level image features descriptor. It was then applied to the color image by Mäenpää and Pietikäinen [19] and used in many color representation and recognition tasks such as facial expression recognition [20, 21], gender classification [22, 23], scene classification [24, 25], medical image analysis [26–28], and 3D face recognition [29]. Ershad and Tajeripour [30] proposed a noise-resistant and multiresolution version of LBP called HCLBP and proposed an approach that adapts to output images acquired using every kind of digital camera. The convolution neural network (CNN) is a deep learning architecture that attracts widespread interest in real-world applications because it automatically extracts and recognizes features automatically, eliminating the need for manual feature extraction and selection techniques.Lately, convolution neural networks (CNNs) are undergoing a revolution in color texture recognition applications [14, 31–36]. Dabeer et al. [37] introduced a novel automatic technique for breast cancer detection based on CNN. Hosny et al. [38] proposed an automatic skin lesion classification approach based on transfer learning theory and a pretrained deep neural network. Genovese et al. [39] introduced a new technique for fusion palm print and inner finger texture using a single hand acquisition based on CNN. Umer et al. [40] presented a CNN technique that can identify COVID-19 patients from regular people from chest X-rays. Wu and Lee [41] used the CNN model for enhancing sound texture in acoustic scenes. Orii et al. [42] suggested a new recognition approach for tactile texture using CNN. Weskley et al. [43] described a new approach based on CNN for classifying the degree of bread browning during baking. Hosny et al. [44] provided a precise improvement method on their approach for skin lesions classification centered on deep neural network AlexNet. They used transfer learning which can accurately classify seven various types of lesions. Kassem et al. [45] proposed a novel technique using transfer learning and pretrained deep neural network, GoogLeNet, to accurately classify eight various lesion types. Hosny et al. [46] proposed a new approach for color texture recognition; the proposed approach has a significant benefit. It uses LBP and multichannel moments to obtain local and global color texture features, leading to a high classification result. This success motivates us to propose a new technique for color texture classification based on the convolution neural network and local binary pattern to extract global and local features and combine them to obtain high color texture classification results.The rest of the paper is arranged as follows. In Section2, convolution neural network models and its architecture are presented. In Section 3, we introduce the local binary patterns descriptor. Section 4 presents the proposed method and simulation results are shown in Section 5.


## 2. Convolution Neural Network

The primary concept of using CNN features is its capability to capture the high-level features that are considered the key point to separate the texture image. Convolution neural network (CNN) captures low detail features from the first convolution layers and then generates high detail features in the last layers. Selecting the proper right set of features has always been a challenging task for the machine learning algorithm. CNN has numerous extensive applications in different fields. Whereas image classification and recognition are challenging tasks for machine learning algorithms, CNN is highly promising for accuracy [47]. CNN generally consists of convolution, pooling, and fully connected layers, as shown in Figure 1.Figure 1
CNN model general architecture.


### 2.1. Convolution Layer

It is used to apply filters to the input feature map and produce convolved features. The following two factors define the convolution layer.(1)
The size of the block extracted from the input feature map.(2)
The depth of convolved features.


### 2.2. Pooling Layer

Generally, applied after the convolution layer, it is a downsampling (dimensional reduction) operation such as average pooling, min pooling, and max pooling.


### 2.3. Fully Connected Layer

It is typically found towards the end of the CNN architecture. It works on such a flattened input, in which each input is linked to all nodes.Each model of CNN has its characteristics. Table1 summarizes the description of different CNN architecture models, image input size, accuracy, and model parameters number.Table 1
Architecture of different CNN models.
CNN ModelParameters number (M)Model descriptionImage input sizeAlexNet [48]605 conv + 3 fc layers227 × 227GoogLeNet [49]721 conv + 1 fc layers224 × 224Vgg16Net [50]13813 conv + 3 fc layers224 × 224Vgg19Net [51]14416 conv + 3 fc layers224 × 224SqueezeNet [52]118 layers224 × 224ResNet-18 [53]1117 conv + 1 fc layers224 × 224ResNet-50 [53]2649 conv + 1 fc layers224 × 224ResNet-10144100 conv + 1 fc layers224 × 224MobileNet [54]427 conv + 1 fc224 × 224Xception [55]2336 layers229 × 229


## 2.1. Convolution Layer

It is used to apply filters to the input feature map and produce convolved features. The following two factors define the convolution layer.(1)
The size of the block extracted from the input feature map.(2)
The depth of convolved features.


## 2.2. Pooling Layer

Generally, applied after the convolution layer, it is a downsampling (dimensional reduction) operation such as average pooling, min pooling, and max pooling.


## 2.3. Fully Connected Layer

It is typically found towards the end of the CNN architecture. It works on such a flattened input, in which each input is linked to all nodes.Each model of CNN has its characteristics. Table1 summarizes the description of different CNN architecture models, image input size, accuracy, and model parameters number.Table 1
Architecture of different CNN models.
CNN ModelParameters number (M)Model descriptionImage input sizeAlexNet [48]605 conv + 3 fc layers227 × 227GoogLeNet [49]721 conv + 1 fc layers224 × 224Vgg16Net [50]13813 conv + 3 fc layers224 × 224Vgg19Net [51]14416 conv + 3 fc layers224 × 224SqueezeNet [52]118 layers224 × 224ResNet-18 [53]1117 conv + 1 fc layers224 × 224ResNet-50 [53]2649 conv + 1 fc layers224 × 224ResNet-10144100 conv + 1 fc layers224 × 224MobileNet [54]427 conv + 1 fc224 × 224Xception [55]2336 layers229 × 229


## 3. Local Binary Patterns

The local binary patterns (LBP) are considered one of the most effective texture descriptors. It was introduced by Ojala et al. [18] for the first time, and it is used to represent the local features of an image which means the key points of an image. The classic LBP operator is described as a window of 3 × 3 pixels. The center pixel of this window is considered a threshold; if the neighboring pixel's value is less than the threshold value, the pixel value is labeled 0, else it is 1. This method will generate an 8 bit binary number converted to a decimal number, as shown in Figure 2.Figure 2
An example of LBP code calculation.The classic LBP descriptor can be expressed in the form below:(1)LBPI,JPc=∑j=0J−1GPj−Pc2j,where (2)Gm=0,m<0,1,otherwise.In (1), Pc=Pm,n is the central pixel at (m, n) location, and Pj = P(mj, nj) is a neighboring pixel of the central pixel Pc, where(3)mj=m+Rcos2πjJ,nj=n−Rsin2πjJ.where J represents the neighbor pixels number and R is the distance from Pc.Ershad and Tajeripour [56] proposed a new approach for color texture classification based on LBP and Kullback–Leibler divergence, which used different neighborhood sizes to achieve effective classification performance.


## 4. The Proposed Method

We proposed a novel feature extraction approach that aggregates the features extracted from local binary patterns (LBP) and convolution neural network (CNN) to obtain more accurate and consistent features for representation, leading to higher texture recognition accuracy results. The performance of our descriptors is assessed both individually and in combination with local binary patterns (LBP). Firstly, we have fixed the image size at the preprocessing phase to be equal to the proposed image size of the pretrained deep learning models (224 × 224 and 227 × 227). We have made equalization of each image histogram to normalize under different illumination conditions and then smooth it to reduce the noise introduced by the equalization. The CNN and LBP have done the feature extraction. The local binary patterns are used to capture an image's local features representing a particular image portion. The radius of LBP neighborhoods was 1, and the number of neighbors was 8. The dimensions of the feature vector extracted from CNN are [1,57]. Convolution neural network is used to capture global features that represent the image en bloc. The dimensions of the feature vector extracted from CNN are [1, 1000]. We aggregated the features of the two feature extraction approaches (CNN, LBP) in one vector. FC = [w, x, y, z]; FL = [l, m, n, o]; Agg features = [FC, FL]. The dimensions of the resultant feature vector are [1, 1059]. We find that the aggregation of features enhances efficiency and reliability over using CNN models individually. The flowchart of the proposed approach is shown in Figure 3.Figure 3
The proposed method flowchart.


## 5. Experimentation and Result Discussion

### 5.1. Datasets

Numerical experiments were carried out to validate and analyze the performance of the proposed descriptor. The experiments were executed on a Core i7 machine with 16 GB RAM. Three benchmark color textures datasets (ALOT, CBT, and Outex) were used to perform the numerical experiments.


#### 5.1.1. ALOT Dataset

ALOT dataset [57] is a series of color image textures. The rotation angle and illumination conditions are varied. More than 27.500 images are included in it. The original size of each image is 768 × 512. We focused on a fixed collection of 1500 texture images that varies from 15 classes of color textures. This collection has been proposed by [58]. Color texture samples of the ALOT database are shown in Figure 4.Figure 4
Texture samples of ALOT dataset.


#### 5.1.2. Colored Brodatz Dataset

The second dataset, colored Brodatz texture (CBT) dataset [58], is a grayscale-to-color extension of the original Brodatz dataset. This dataset has the advantage of keeping the original dataset's rich textural content while also providing a wide range of color content. It contains 112 color texture images that have various background intensities. The original size of each image is 512 × 512. We used a subset of 1500 color images from 15 classes. This subset has been proposed by Bianconi [59]. Texture samples of the CBT database are shown in Figure 5.Figure 5
Texture samples of colored Brodatz dataset (CBT).


#### 5.1.3. Outex Dataset

The third dataset, the Outex dataset [60], is an extensive range of color textures; the range of textures is displayed in varied types of lighting, resolution, and rotated angles. It contains 320 surface color textures. The original size of each image is 746 × 538. A subgroup has been identified, containing 1500 images from 15 classes. This subgroup was proposed by [58]. Color texture samples of the Outex database are shown in Figure 6.Figure 6
Texture samples of Outex dataset.


### 5.2. CNN Models Descriptors

To begin, we used CNN models to extract features for the grayscale and color image datasets. The number of features extracted from each image was 1000. The three-color image datasets were split into training and testing sets at a ratio of 70 : 30, respectively, in these simulations. The training set is used in classifier practice, while the testing set is used for testing, analysis, and evaluation. Using the linear support vector machine classifier because of its efficiency in solving multiclass classification problems, we checked the performance of CNN models over the Brodatz grayscale dataset. The results revealed that the performance of CNN has some limitations in terms of accuracy and needs some enhancement. Table2 details the results of the CNN descriptor over the grayscale Brodatz dataset.Table 2
The accuracy of results obtained from CNN models and SVM classifier over grayscale Brodatz dataset.
Dataset texture descriptorBrodatz dataset (%)AlexNet76.05GoogLeNet79.25Vgg16Net80.55Vgg19Net81.40SqueezeNet79.95ResNet-1890.12ResNet-5092.50ResNet-10177.20MobileNet96.90Xception80.00Then we checked the performance of CNN models over the three challenging datasets (ALOT, Outex, and CBT). We have computed an accuracy and confusion matrix for performing classification in all the given datasets. CNN models are trained once over these datasets. Our tests showed that their performance is excellent and fair but needs to enhance the accuracy results. The highest accuracy result of CNN models in the ALOT dataset was 90.89%, using Xception as a descriptor. In the Outex dataset, it was 91.78% by using ResNet-101 as a descriptor. In CBT, the dataset was 86.45% by using AlexNet as a descriptor. Table3 details the results (the best performed CNN in each dataset is indicated in bold) of CNN models using the SVM classifier for the three datasets.Table 3
The accuracy of results obtained from CNN models and SVM classifier.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet84.4586.4587.70GoogLeNet89.3484.2386.89Vgg16Net88.6782.6789.12Vgg19Net88.2385.1287.34SqueezeNet78.8976.6789.78ResNet-1881.1278.8980.00ResNet-5090.4581.3480.45ResNet-10189.5684.0091.78MobileNet86.8985.5690.89Xception90.8986.2390.23The highest accuracies are indicated in bold.We have computed an accuracy and confusion matrix for performing classification in all the given datasets. Figure7 represents ROC curves of CNN models as descriptors using support vector machine classifier for ALOT dataset.Figure 7
ROC curves of CNN models using SVM classifier for the ALOT dataset.Figures8 and 9 represent ROC curves of CNN models as descriptors using support vector machine classifier for the CBT and Outex dataset, respectively. The ROC analysis revealed that the CNN model descriptors had a fair AUC score (∼0.8), which needs improvement to achieve the highest accuracy result. This observation led the authors to look forward to a local features extraction descriptor with the CNN model.Figure 8
ROC curves of CNN models using SVM classifier for the CBT dataset.Figure 9
ROC curves of CNN models using SVM classifier for the Outex dataset.


### 5.3. LBP Models Descriptors

Secondly, we used a traditional LBP descriptor to extract features for the color image datasets. The number of features extracted from each image was 59; also using the SVM, we checked the consistency of CNN models over the Brodatz dataset. The result showed that the performance needs some improvements. Table4 shows the accuracy results of LBP using the SVM classifier for the grayscale Brodatz dataset.Table 4
Results of LBP using SVM classifier for the grayscale Brodatz dataset.
Dataset texture descriptorBrodatz (%)LBP descriptor78.20Then we checked the consistency of CNN models over the three challenging datasets (ALOT, Outex, and CBT). Our analysis and tests showed that their performance is excellent and fair but needs to enhance the accuracy results. The accuracy result of LBP models in the ALOT dataset was 70.3%, in the CBT dataset was 73.45%, and in the Outex dataset was 66.2%. Table5 details the accuracy results of LBP using the SVM classifier for the three datasets.Table 5
Results of LBP using SVM classifier for the three datasets.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)LBP descriptor70.373.4566.2


### 5.4. CNN + LBP Descriptors

The first set of analyses confirmed that the accuracy results obtained using only CNN models as grayscale and color image feature descriptors have some limitations in classification accuracy and need some improvement. Therefore, we used an aggregated feature extraction descriptor (CNN + LBP) for the grayscale and color textures in the second experiment. Convolution neural networks are used to capture global features that describe the image overall. In contrast, Local binary patterns capture the local features of an image that describe a particular image part. We checked the performance of our proposed descriptor over the Brodatz grayscale dataset, and our result was remarkable and proved the adequate performance of the descriptor over a grayscale dataset. Table6 details the proposed descriptors' results using the SVM classifier for the grayscale Brodatz dataset.Table 6
Summary results of the proposed descriptors using SVM classifier for grayscale Brodatz dataset.
Dataset texture descriptorBrodatz dataset (%)AlexNet + LBP95.80GoogLeNet + LBP95.25Vgg16Net + LBP97.00Vgg19Net + LBP100.00SqueezeNet + LBP98.20ResNet-18 + LBP94.32ResNet-50 + LBP99.20ResNet-101 + LBP94.30MobileNet + LBP100.00Xception + LBP97.00This performance encourages the authors to check descriptor performance over color datasets (ALOT, Outex, and CBT). Firstly, the number of features extracted from each image was 2559 (2500 global features from the CNN model and 59 local features from LBP). We found that the increased number of features in the combinations would significantly slow the machine learning stage. So we have to reduce the number of features extracted from each image to 1059 (1000 global features from the CNN model and 59 local features from LBP). We have fixed the original image size at the preprocessing phase to equal the proposed image size of the pretrained deep learning models (224 × 224 and 227 × 227) to be used in the input layer. The three datasets are split between training and testing datasets at a ratio of 70 : 30, respectively. Using a routine test for detecting overfit of our descriptor by monitoring the loss and accuracy on the training and validation sets, our result demonstrated that our model has no overfit. In the three challenging datasets (ALOT, Outex, and CBT), these tests confirm that the proposed descriptors show a clear advantage over the recognition accuracy, are effective and discriminant, and outperform the existing state-of-the-art methods. Using SVM classifier, the highest accuracy result of our descriptors in the ALOT dataset was 100% by using Xception + LBP descriptor, in Outex dataset was 96.67% by using Xception + LBP, and in CBT dataset was 100% by using ResNet-101 + LBP descriptor. The lowest accuracy result of our descriptors in the ALOT dataset was 93.12% by using ResNet-18 + LBP, in Outex dataset was 88.89% by using SqueezeNet + LBP, and in CBT dataset was 94% by using ResNet-18 + LBP.Table7 details the proposed descriptors' results using the SVM classifier for the A LOT dataset.Table 7
Summary results of the proposed descriptors using SVM classifier for ALOT dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9530.0040.9530.9510.9530.9490.9550.95295.30GoogLeNet + LBP0.9950.0010.9950.9900.9950.9910.9900.99299.50Vgg16Net + LBP0.9750.0030.9970.9940.9970.9931.0000.99797.50Vgg19Net + LBP0.9820.0020.9800.9810.9840.9820.9840.98098.20SqueezeNet + LBP0.9480.0050.9490.9450.9460.9480.9480.94694.89ResNet-18 + LBP0.9310.0060.9320.9300.9310.9320.9290.92893.12ResNet-50 + LBP0.9980.0020.9960.9960.9980.99709910.99399.78ResNet-101 + LBP0.9890.0010.9850.9830.9890.9860.9870.98098.89MobileNet + LBP0.9690.0020.9680.9680.9680.9601.0000.96996.89Xception + LBP1.0000.0000.9890.9910.9990.9970.9990.997100.00Figure10 represents ROC curves of the proposed descriptors using SVM classifier for ALOT dataset.Figure 10
ROC curves of the proposed descriptors using SVM classifier for the ALOT dataset.Table8 details precisely the results of the proposed descriptors using the SVM classifier for the CBT dataset.Table 8
Summary results of the proposed descriptors using SVM classifier for CBT dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9110.0070.9030.9160.9140.9100.9110.91491.12GoogLeNet + LBP0.9220.0040.9200.9190.9200.9220.9220.92392.23Vgg16Net + LBP0.9260.0030.9230.9200.9250.9260.9490.94692.67Vgg19Net + LBP0.9330.0040.9450.9320.9330.9310.9960.94593.34SqueezeNet + LBP0.8880.0090.8820.8820.8820.8730.9480.87688.89ResNet-18 + LBP0.9060.0060.8890.9040.9010.8950.9680.91790.67ResNet-50 + LBP0.9980.0020.9960.9960.9970.99709910.99392.45ResNet-101 + LBP0.9450.0040.9420.9450.9430.9390.9970.94594.45MobileNet + LBP0.9600.0020.9610.9610.9600.9600.9980.96096.00Xception + LBP0.9660.0010.9650.9600.9620.9620.9610.96696.67Figure11 represents ROC curves of the proposed descriptors using SVM classifier for CBT dataset.Figure 11
ROC curves of the proposed descriptors using SVM classifier for the CBT dataset.Table9 details precisely the results of the proposed descriptors using the SVM classifier for the Outex dataset.Table 9
Summary results of the proposed descriptors using SVM classifier for Outex dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9770.0010.9750.9770.9750.972.09790.97397.76GoogLeNet + LBP0.9880.00030.9850.9880.9850.9810.9830.95698.80Vgg16Net + LBP0.9930.00010.9900.9930.9900.9890.9900.98599.34Vgg19Net + LBP0.9970.0010.9980.9970.9970.9971.0001.00099.78SqueezeNet + LBP0.9730.0020.9720.9730.9740.9680.9680.96697.34ResNet-18 + LBP0.9400.0050.9320.9300.9330.9350.9300.92894.00ResNet-50 + LBP0.9770.0010.9760.9780.9760.9740.9750.97397.78ResNet-101 + LBP1.0000.0000.9980.9930.9990.9961.000.998100.00MobileNet + LBP0.9930.0020.9920.9890.9870.9900.9930.98899.34Xception + LBP0.9970.0010.9950.9910.9990.9970.9990.99799.78Figure12 represents ROC curves of the proposed descriptors using support vector machine classifier for Outex dataset.Figure 12
ROC curves of the proposed descriptors using SVM classifier for the Outex dataset.Table10 reveals the high accuracy result obtained by the proposed method for the three datasets.Table 10
The accuracy of results obtained from the proposed descriptors and SVM classifier.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP95.3491.1297.70GoogLeNet + LBP99.5692.2398.80Vgg16Net + LBP97.5692.6799.30Vgg19Net + LBP98.2093.3499.70SqueezeNet + LBP94.8988.8997.34ResNet-18 + LBP93.1290.6794.00ResNet-50 + LBP99.7892.4597.78ResNet-101 + LBP98.8994.45100.00MobileNet + LBP96.8996.0099.34Xception + LBP100.0096.6799.78From Table9 and Table 10, it is clear that the pretrained model Xception + LBP, ResNet + LBP, and Xception + LBP stand out in the performance when tested with ALOT, Outex, and CBT datasets, respectively. The result demonstrated that ResNet-18 + LBP has the lowest accuracy rate in two datasets (A LOT and Outex) and also SqueezeNet + LBP in CBT dataset.Figure13 provided a comparison between our descriptor’s promising accuracy result and the accuracy result obtained by CNN models for the ALOT dataset. Figure 14 provided a comparison between the promising accuracy result obtained by our descriptor and the accuracy result obtained by CNN models for the CBT dataset. Figure 15 provided a comparison between our descriptor's promising accuracy result and the accuracy result obtained by CNN models for the Outex dataset.Figure 13
CNN vs. CNN + LBP in terms of accuracy for the ALOT dataset.Figure 14
CNN vs. CNN + LBP in terms of accuracy for the CBT dataset.Figure 15
CNN vs. CNN + LBP in terms of accuracy for the Outex dataset.From previous results, we can observe that the accuracy increased when we used LBP + CNN as a descriptor for the three datasets, and to evaluate its performance, we presented a comparison between it and other state-of-the-art methods introduced by [61] and by [58] in Table 11.Table 11
Comparison between results of LBP + CNN and other descriptors for Outex and ALOT dataset.
Texture descriptorDatasetsOutex (%)ALOT (%)Legendre-c.f GLCM83.266.2Discr. Cheb. GLCM84.964.4LBP-RI74.670.6MLBP93.671.4GWN78.488AlexNet + LBP95.3497.70GoogLeNet + LBP99.5698.80Vgg16Net + LBP97.5699.30Vgg19Net + LBP98.2099.70SqueezeNet + LBP94.8997.34ResNet-18 + LBP93.1294.00ResNet-50 + LBP99.7897.78ResNet-101 + LBP98.89100.00MobileNet + LBP96.8999.34Xception + LBP100.0099.78Then we evaluated our proposed descriptors' performance when provided with a limited training set of textures to detect if it is a robust approach or not. Table12 reveals the accuracy result obtained by the proposed method for the three datasets by providing only 20% for the training set.Table 12
The accuracy of results obtained from proposed descriptors and SVM classifier by providing 20% for a training set.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP65.1261.3570.70GoogLeNet + LBP79.5672.2778.08Vgg16Net + LBP77.5672.6771.30Vgg19Net + LBP78.0060.0266.70SqueezeNet + LBP54.9568.8258.14ResNet-18 + LBP63.2065.2574.50ResNet-50 + LBP80.0872.2578.08ResNet-101 + LBP78.2074.0079.25MobileNet + LBP66.4272.1073.48Xception + LBP79.9876.7059.18Table13 reveals the accuracy result obtained by the proposed method for the three datasets by providing only 50% for the training set.Table 13
The accuracy of results obtained from proposed descriptors and SVM classifier by providing 20% for the training set.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP74.3481.1280.20GoogLeNet + LBP75.2071.4381.42Vgg16Net + LBP77.6574.7279.50Vgg19Net + LBP80.2072.2283.60SqueezeNet + LBP74.9069.4877.42ResNet-18 + LBP73.0271.7082.00ResNet-50 + LBP81.8262.2271.82ResNet-101 + LBP69.2076.5084.21MobileNet + LBP70.9984.2471.25Xception + LBP80.4085.4080.82


## 5.1. Datasets

Numerical experiments were carried out to validate and analyze the performance of the proposed descriptor. The experiments were executed on a Core i7 machine with 16 GB RAM. Three benchmark color textures datasets (ALOT, CBT, and Outex) were used to perform the numerical experiments.


### 5.1.1. ALOT Dataset

ALOT dataset [57] is a series of color image textures. The rotation angle and illumination conditions are varied. More than 27.500 images are included in it. The original size of each image is 768 × 512. We focused on a fixed collection of 1500 texture images that varies from 15 classes of color textures. This collection has been proposed by [58]. Color texture samples of the ALOT database are shown in Figure 4.Figure 4
Texture samples of ALOT dataset.


### 5.1.2. Colored Brodatz Dataset

The second dataset, colored Brodatz texture (CBT) dataset [58], is a grayscale-to-color extension of the original Brodatz dataset. This dataset has the advantage of keeping the original dataset's rich textural content while also providing a wide range of color content. It contains 112 color texture images that have various background intensities. The original size of each image is 512 × 512. We used a subset of 1500 color images from 15 classes. This subset has been proposed by Bianconi [59]. Texture samples of the CBT database are shown in Figure 5.Figure 5
Texture samples of colored Brodatz dataset (CBT).


### 5.1.3. Outex Dataset

The third dataset, the Outex dataset [60], is an extensive range of color textures; the range of textures is displayed in varied types of lighting, resolution, and rotated angles. It contains 320 surface color textures. The original size of each image is 746 × 538. A subgroup has been identified, containing 1500 images from 15 classes. This subgroup was proposed by [58]. Color texture samples of the Outex database are shown in Figure 6.Figure 6
Texture samples of Outex dataset.


## 5.1.1. ALOT Dataset

ALOT dataset [57] is a series of color image textures. The rotation angle and illumination conditions are varied. More than 27.500 images are included in it. The original size of each image is 768 × 512. We focused on a fixed collection of 1500 texture images that varies from 15 classes of color textures. This collection has been proposed by [58]. Color texture samples of the ALOT database are shown in Figure 4.Figure 4
Texture samples of ALOT dataset.


## 5.1.2. Colored Brodatz Dataset

The second dataset, colored Brodatz texture (CBT) dataset [58], is a grayscale-to-color extension of the original Brodatz dataset. This dataset has the advantage of keeping the original dataset's rich textural content while also providing a wide range of color content. It contains 112 color texture images that have various background intensities. The original size of each image is 512 × 512. We used a subset of 1500 color images from 15 classes. This subset has been proposed by Bianconi [59]. Texture samples of the CBT database are shown in Figure 5.Figure 5
Texture samples of colored Brodatz dataset (CBT).


## 5.1.3. Outex Dataset

The third dataset, the Outex dataset [60], is an extensive range of color textures; the range of textures is displayed in varied types of lighting, resolution, and rotated angles. It contains 320 surface color textures. The original size of each image is 746 × 538. A subgroup has been identified, containing 1500 images from 15 classes. This subgroup was proposed by [58]. Color texture samples of the Outex database are shown in Figure 6.Figure 6
Texture samples of Outex dataset.


## 5.2. CNN Models Descriptors

To begin, we used CNN models to extract features for the grayscale and color image datasets. The number of features extracted from each image was 1000. The three-color image datasets were split into training and testing sets at a ratio of 70 : 30, respectively, in these simulations. The training set is used in classifier practice, while the testing set is used for testing, analysis, and evaluation. Using the linear support vector machine classifier because of its efficiency in solving multiclass classification problems, we checked the performance of CNN models over the Brodatz grayscale dataset. The results revealed that the performance of CNN has some limitations in terms of accuracy and needs some enhancement. Table2 details the results of the CNN descriptor over the grayscale Brodatz dataset.Table 2
The accuracy of results obtained from CNN models and SVM classifier over grayscale Brodatz dataset.
Dataset texture descriptorBrodatz dataset (%)AlexNet76.05GoogLeNet79.25Vgg16Net80.55Vgg19Net81.40SqueezeNet79.95ResNet-1890.12ResNet-5092.50ResNet-10177.20MobileNet96.90Xception80.00Then we checked the performance of CNN models over the three challenging datasets (ALOT, Outex, and CBT). We have computed an accuracy and confusion matrix for performing classification in all the given datasets. CNN models are trained once over these datasets. Our tests showed that their performance is excellent and fair but needs to enhance the accuracy results. The highest accuracy result of CNN models in the ALOT dataset was 90.89%, using Xception as a descriptor. In the Outex dataset, it was 91.78% by using ResNet-101 as a descriptor. In CBT, the dataset was 86.45% by using AlexNet as a descriptor. Table3 details the results (the best performed CNN in each dataset is indicated in bold) of CNN models using the SVM classifier for the three datasets.Table 3
The accuracy of results obtained from CNN models and SVM classifier.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet84.4586.4587.70GoogLeNet89.3484.2386.89Vgg16Net88.6782.6789.12Vgg19Net88.2385.1287.34SqueezeNet78.8976.6789.78ResNet-1881.1278.8980.00ResNet-5090.4581.3480.45ResNet-10189.5684.0091.78MobileNet86.8985.5690.89Xception90.8986.2390.23The highest accuracies are indicated in bold.We have computed an accuracy and confusion matrix for performing classification in all the given datasets. Figure7 represents ROC curves of CNN models as descriptors using support vector machine classifier for ALOT dataset.Figure 7
ROC curves of CNN models using SVM classifier for the ALOT dataset.Figures8 and 9 represent ROC curves of CNN models as descriptors using support vector machine classifier for the CBT and Outex dataset, respectively. The ROC analysis revealed that the CNN model descriptors had a fair AUC score (∼0.8), which needs improvement to achieve the highest accuracy result. This observation led the authors to look forward to a local features extraction descriptor with the CNN model.Figure 8
ROC curves of CNN models using SVM classifier for the CBT dataset.Figure 9
ROC curves of CNN models using SVM classifier for the Outex dataset.


## 5.3. LBP Models Descriptors

Secondly, we used a traditional LBP descriptor to extract features for the color image datasets. The number of features extracted from each image was 59; also using the SVM, we checked the consistency of CNN models over the Brodatz dataset. The result showed that the performance needs some improvements. Table4 shows the accuracy results of LBP using the SVM classifier for the grayscale Brodatz dataset.Table 4
Results of LBP using SVM classifier for the grayscale Brodatz dataset.
Dataset texture descriptorBrodatz (%)LBP descriptor78.20Then we checked the consistency of CNN models over the three challenging datasets (ALOT, Outex, and CBT). Our analysis and tests showed that their performance is excellent and fair but needs to enhance the accuracy results. The accuracy result of LBP models in the ALOT dataset was 70.3%, in the CBT dataset was 73.45%, and in the Outex dataset was 66.2%. Table5 details the accuracy results of LBP using the SVM classifier for the three datasets.Table 5
Results of LBP using SVM classifier for the three datasets.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)LBP descriptor70.373.4566.2


## 5.4. CNN + LBP Descriptors

The first set of analyses confirmed that the accuracy results obtained using only CNN models as grayscale and color image feature descriptors have some limitations in classification accuracy and need some improvement. Therefore, we used an aggregated feature extraction descriptor (CNN + LBP) for the grayscale and color textures in the second experiment. Convolution neural networks are used to capture global features that describe the image overall. In contrast, Local binary patterns capture the local features of an image that describe a particular image part. We checked the performance of our proposed descriptor over the Brodatz grayscale dataset, and our result was remarkable and proved the adequate performance of the descriptor over a grayscale dataset. Table6 details the proposed descriptors' results using the SVM classifier for the grayscale Brodatz dataset.Table 6
Summary results of the proposed descriptors using SVM classifier for grayscale Brodatz dataset.
Dataset texture descriptorBrodatz dataset (%)AlexNet + LBP95.80GoogLeNet + LBP95.25Vgg16Net + LBP97.00Vgg19Net + LBP100.00SqueezeNet + LBP98.20ResNet-18 + LBP94.32ResNet-50 + LBP99.20ResNet-101 + LBP94.30MobileNet + LBP100.00Xception + LBP97.00This performance encourages the authors to check descriptor performance over color datasets (ALOT, Outex, and CBT). Firstly, the number of features extracted from each image was 2559 (2500 global features from the CNN model and 59 local features from LBP). We found that the increased number of features in the combinations would significantly slow the machine learning stage. So we have to reduce the number of features extracted from each image to 1059 (1000 global features from the CNN model and 59 local features from LBP). We have fixed the original image size at the preprocessing phase to equal the proposed image size of the pretrained deep learning models (224 × 224 and 227 × 227) to be used in the input layer. The three datasets are split between training and testing datasets at a ratio of 70 : 30, respectively. Using a routine test for detecting overfit of our descriptor by monitoring the loss and accuracy on the training and validation sets, our result demonstrated that our model has no overfit. In the three challenging datasets (ALOT, Outex, and CBT), these tests confirm that the proposed descriptors show a clear advantage over the recognition accuracy, are effective and discriminant, and outperform the existing state-of-the-art methods. Using SVM classifier, the highest accuracy result of our descriptors in the ALOT dataset was 100% by using Xception + LBP descriptor, in Outex dataset was 96.67% by using Xception + LBP, and in CBT dataset was 100% by using ResNet-101 + LBP descriptor. The lowest accuracy result of our descriptors in the ALOT dataset was 93.12% by using ResNet-18 + LBP, in Outex dataset was 88.89% by using SqueezeNet + LBP, and in CBT dataset was 94% by using ResNet-18 + LBP.Table7 details the proposed descriptors' results using the SVM classifier for the A LOT dataset.Table 7
Summary results of the proposed descriptors using SVM classifier for ALOT dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9530.0040.9530.9510.9530.9490.9550.95295.30GoogLeNet + LBP0.9950.0010.9950.9900.9950.9910.9900.99299.50Vgg16Net + LBP0.9750.0030.9970.9940.9970.9931.0000.99797.50Vgg19Net + LBP0.9820.0020.9800.9810.9840.9820.9840.98098.20SqueezeNet + LBP0.9480.0050.9490.9450.9460.9480.9480.94694.89ResNet-18 + LBP0.9310.0060.9320.9300.9310.9320.9290.92893.12ResNet-50 + LBP0.9980.0020.9960.9960.9980.99709910.99399.78ResNet-101 + LBP0.9890.0010.9850.9830.9890.9860.9870.98098.89MobileNet + LBP0.9690.0020.9680.9680.9680.9601.0000.96996.89Xception + LBP1.0000.0000.9890.9910.9990.9970.9990.997100.00Figure10 represents ROC curves of the proposed descriptors using SVM classifier for ALOT dataset.Figure 10
ROC curves of the proposed descriptors using SVM classifier for the ALOT dataset.Table8 details precisely the results of the proposed descriptors using the SVM classifier for the CBT dataset.Table 8
Summary results of the proposed descriptors using SVM classifier for CBT dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9110.0070.9030.9160.9140.9100.9110.91491.12GoogLeNet + LBP0.9220.0040.9200.9190.9200.9220.9220.92392.23Vgg16Net + LBP0.9260.0030.9230.9200.9250.9260.9490.94692.67Vgg19Net + LBP0.9330.0040.9450.9320.9330.9310.9960.94593.34SqueezeNet + LBP0.8880.0090.8820.8820.8820.8730.9480.87688.89ResNet-18 + LBP0.9060.0060.8890.9040.9010.8950.9680.91790.67ResNet-50 + LBP0.9980.0020.9960.9960.9970.99709910.99392.45ResNet-101 + LBP0.9450.0040.9420.9450.9430.9390.9970.94594.45MobileNet + LBP0.9600.0020.9610.9610.9600.9600.9980.96096.00Xception + LBP0.9660.0010.9650.9600.9620.9620.9610.96696.67Figure11 represents ROC curves of the proposed descriptors using SVM classifier for CBT dataset.Figure 11
ROC curves of the proposed descriptors using SVM classifier for the CBT dataset.Table9 details precisely the results of the proposed descriptors using the SVM classifier for the Outex dataset.Table 9
Summary results of the proposed descriptors using SVM classifier for Outex dataset.
Features descriptorRP rateWP ratePrecisionRecallF-measureMCCROC areaPRC areaAccuracy (%)AlexNet + LBP0.9770.0010.9750.9770.9750.972.09790.97397.76GoogLeNet + LBP0.9880.00030.9850.9880.9850.9810.9830.95698.80Vgg16Net + LBP0.9930.00010.9900.9930.9900.9890.9900.98599.34Vgg19Net + LBP0.9970.0010.9980.9970.9970.9971.0001.00099.78SqueezeNet + LBP0.9730.0020.9720.9730.9740.9680.9680.96697.34ResNet-18 + LBP0.9400.0050.9320.9300.9330.9350.9300.92894.00ResNet-50 + LBP0.9770.0010.9760.9780.9760.9740.9750.97397.78ResNet-101 + LBP1.0000.0000.9980.9930.9990.9961.000.998100.00MobileNet + LBP0.9930.0020.9920.9890.9870.9900.9930.98899.34Xception + LBP0.9970.0010.9950.9910.9990.9970.9990.99799.78Figure12 represents ROC curves of the proposed descriptors using support vector machine classifier for Outex dataset.Figure 12
ROC curves of the proposed descriptors using SVM classifier for the Outex dataset.Table10 reveals the high accuracy result obtained by the proposed method for the three datasets.Table 10
The accuracy of results obtained from the proposed descriptors and SVM classifier.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP95.3491.1297.70GoogLeNet + LBP99.5692.2398.80Vgg16Net + LBP97.5692.6799.30Vgg19Net + LBP98.2093.3499.70SqueezeNet + LBP94.8988.8997.34ResNet-18 + LBP93.1290.6794.00ResNet-50 + LBP99.7892.4597.78ResNet-101 + LBP98.8994.45100.00MobileNet + LBP96.8996.0099.34Xception + LBP100.0096.6799.78From Table9 and Table 10, it is clear that the pretrained model Xception + LBP, ResNet + LBP, and Xception + LBP stand out in the performance when tested with ALOT, Outex, and CBT datasets, respectively. The result demonstrated that ResNet-18 + LBP has the lowest accuracy rate in two datasets (A LOT and Outex) and also SqueezeNet + LBP in CBT dataset.Figure13 provided a comparison between our descriptor’s promising accuracy result and the accuracy result obtained by CNN models for the ALOT dataset. Figure 14 provided a comparison between the promising accuracy result obtained by our descriptor and the accuracy result obtained by CNN models for the CBT dataset. Figure 15 provided a comparison between our descriptor's promising accuracy result and the accuracy result obtained by CNN models for the Outex dataset.Figure 13
CNN vs. CNN + LBP in terms of accuracy for the ALOT dataset.Figure 14
CNN vs. CNN + LBP in terms of accuracy for the CBT dataset.Figure 15
CNN vs. CNN + LBP in terms of accuracy for the Outex dataset.From previous results, we can observe that the accuracy increased when we used LBP + CNN as a descriptor for the three datasets, and to evaluate its performance, we presented a comparison between it and other state-of-the-art methods introduced by [61] and by [58] in Table 11.Table 11
Comparison between results of LBP + CNN and other descriptors for Outex and ALOT dataset.
Texture descriptorDatasetsOutex (%)ALOT (%)Legendre-c.f GLCM83.266.2Discr. Cheb. GLCM84.964.4LBP-RI74.670.6MLBP93.671.4GWN78.488AlexNet + LBP95.3497.70GoogLeNet + LBP99.5698.80Vgg16Net + LBP97.5699.30Vgg19Net + LBP98.2099.70SqueezeNet + LBP94.8997.34ResNet-18 + LBP93.1294.00ResNet-50 + LBP99.7897.78ResNet-101 + LBP98.89100.00MobileNet + LBP96.8999.34Xception + LBP100.0099.78Then we evaluated our proposed descriptors' performance when provided with a limited training set of textures to detect if it is a robust approach or not. Table12 reveals the accuracy result obtained by the proposed method for the three datasets by providing only 20% for the training set.Table 12
The accuracy of results obtained from proposed descriptors and SVM classifier by providing 20% for a training set.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP65.1261.3570.70GoogLeNet + LBP79.5672.2778.08Vgg16Net + LBP77.5672.6771.30Vgg19Net + LBP78.0060.0266.70SqueezeNet + LBP54.9568.8258.14ResNet-18 + LBP63.2065.2574.50ResNet-50 + LBP80.0872.2578.08ResNet-101 + LBP78.2074.0079.25MobileNet + LBP66.4272.1073.48Xception + LBP79.9876.7059.18Table13 reveals the accuracy result obtained by the proposed method for the three datasets by providing only 50% for the training set.Table 13
The accuracy of results obtained from proposed descriptors and SVM classifier by providing 20% for the training set.
Dataset texture descriptorALOT (%)CBT (%)Outex (%)AlexNet + LBP74.3481.1280.20GoogLeNet + LBP75.2071.4381.42Vgg16Net + LBP77.6574.7279.50Vgg19Net + LBP80.2072.2283.60SqueezeNet + LBP74.9069.4877.42ResNet-18 + LBP73.0271.7082.00ResNet-50 + LBP81.8262.2271.82ResNet-101 + LBP69.2076.5084.21MobileNet + LBP70.9984.2471.25Xception + LBP80.4085.4080.82


## 6. Conclusion

Overall, this paper presents a new feature extraction approach for color texture recognition techniques that aggregate features extracted from local binary pattern and convolution neural network. Features combinations provide more reliable, consistent performance. Our studies demonstrated that suggested descriptors have good features such as stability and robustness against different orientations and remarkably accurate classification results in recognition tasks. The experiments were performed on three benchmark color datasets (ALOT, CBT, and Outex) using support vector machine as a classifier. We have obtained results demonstrating the proposed classification technique's efficiency, robustness, and high accuracy against image transformation such as scaling, rotation, and translation. By analyzing classification and recognition accuracy, the findings are promising and encouraging.



---

*Source: 5567489-2021-11-16.xml*