# Recognition ofZiziphus lotus through Aerial Imaging and Deep Transfer Learning Approach

Authors: Ahsan Bin Tufail; Inam Ullah; Rahim Khan; Luqman Ali; Adnan Yousaf; Ateeq Ur Rehman; Wajdi Alhakami; Habib Hamam; Omar Cheikhrouhou; Yong-Kui Ma
Journal: Mobile Information Systems
 (2021)

---

## Abstract

There is a growing demand for the detection of endangered plant species through machine learning approaches.Ziziphus lotus is an endangered deciduous plant species in the buckthorn family (Rhamnaceae) native to Southern Europe. Traditional methods such as object-based image analysis have achieved good recognition rates. However, they are slow and require high human intervention. Transfer learning-based methods have several applications for data analysis in a variety of Internet of Things systems. In this work, we have analyzed the potential of convolutional neural networks to recognize and detect the Ziziphus lotus plant in remote sensing images. We fine-tuned Inception version 3, Xception, and Inception ResNet version 2 architectures for binary classification into plant species class and bare soil and vegetation class. The achieved results are promising and effectively demonstrate the better performance of deep learning algorithms over their counterparts.


---

## Body

## 1. Introduction

Rates for the destruction of habitat and plant species are on the rise worldwide due to several alterations in the land cover and land use which can significantly impact the environment and society. Hence, there is a pressing need for a management system that incorporates solid scientific principles for environmental management of species especially plant species [1]. Among many factors that influence the natural ecosystem, global land cover is perhaps the most important of them. Combined with remote sensing technology, this burgeoning domain has important applications on the Internet of Things (IoT)-based technologies [2]. Remote sensing datasets if effectively and efficiently processed can discover important patterns that are helpful in sustainable environmental protection efforts [3].Anthropogenic land use and cover change (LUCC) is one of the primary factors behind alterations in the environment on a global scale. Understanding the underlying factors behind LUCC holds the key to sustainable efforts to reduce deforestation and forest degradation [4]. Shrub and bush, agricultural land, turf, and grass are grassland-type land cover classification systems where research activities on climate change modelling, environmental protection, and regional/national land-use planning are currently underway [5].Ziziphus lotus is regarded as keystone scrubs, whose seeds are consumed and dispersed by foxes and other vertebrates, that grow on low-land wad terraces and are threatened by the spreading of greenhouse gases, agricultural practices, and land-use patterns [6–9]. Traditionally, species distribution models (SDMs) are used in the identification and location of the population of rare and threatened species. These models are heavily used in the conservation of plant species.However, due to its enhanced monitoring range, rapid speed, and potential to acquire vast amounts of information, satellite remote sensing represents one of the most practical approaches for land-use patterns. Remote sensing sensors whose spectral bands range from visible to infrared regions of the electromagnetic spectrum are used in the mapping of land cover among other applications [10]. On the downside, there are certain limitations of monitoring the conservation status of habitats with remote sensing as it cannot be directly applied to assess small-scale characteristics [11]. Due to the availability of a wide range of sensors, it is practically possible to accurately estimate the use of different technologies for the protection of wildlife [12]. However, monitoring experts are still in doubt if different remote sensing technologies can meet their demands in financial terms. These experts found it difficult to use their knowledge with these technologies [13]. Furthermore, due to the way spatial information is utilized, it is difficult to assess its impact on the urban expansion especially concerning natural habitats [14].Despite these limitations, satellite technologies are widely deployed to study land-use patterns [15], fragmentation of forest formations [16], implications of land-use patterns related to population and development [17], and selective species and habitat protection [18] as well as estimate natural and artificial changes in landscapes [19].Classification methods are widely deployed to study airborne visible/infrared imaging spectroradiometer hyperspectral imagery [20], Landsat images [21], and panchromatic high-resolution data from urban areas [22]. A very high-resolution spatial remote sensing provides detailed information about vegetation [23], man-made, water, green vegetation, and bare soil [24], as well as mapping wildlife habitat [25].Deep learning techniques such as convolutional neural networks (CNNs) have application in healthcare [26] and other domains [27–30] and are getting popularity for the classification of land cover using light detection and ranging (LIDAR) and Landsat imagery across different time points [31], scene classification using very high-resolution cameras for remote sensing applications [32], and hyperspectral imagery [33–36]. Traditionally, methods based on segmentation approaches, such as object-based image analysis (OBIA) [37–39], are used for the classification of land cover mapping, disaster management, environmental monitoring, and civil and military intelligence. However, there are numerous problems in the successful application of such approaches such as the type of segmentation challenges and conceptual foundations. In essence, OBIA is far from being an operationally established paradigm for specific research or commercial activities.Emergent artificial intelligence (AI)-based methods such as IoT-based methods have found various applications in wireless sensor networks [40, 41] and are finding their way into connection of millions of objects to help in getting meaningful results from unprocessed data [42]. It is committed to provide best possible solutions to deal with data and information [43, 44]. A big challenge in IoT is to uniquely identify each object with representation and storage of information that is getting exchanged among the objects. Applications of IoT are still in nascent stages with such diversity as natural calamities prediction, water shortage detection, smart homes, healthcare, smart farming, smart transport, smart cities, and smart security.Because of the inherent limitations of the existing object recognition methods for recognizing objects in high-resolution images obtained from remote sensing apparatus, there is a need for the evaluation of new learning paradigms in this domain. In this research, we present transfer learning as the method of choice for the classification ofZiziphus lotus shrub from the bare soil and vegetation in remote sensing images. Distinguishing Ziziphus lotus from neighbouring plants is a difficult task for non-experts and the existing computational methods because the surrounding plants and the background soil differ strongly in the close-by regions [45]. We use Xception [46], Inception version 3 [47], and Inception ResNet version 2 [48] deep transfer learning architectures for fine-tuning ImageNet dataset-based features on our problem. We compared our approach with the existing works in the literature and found that our approach outperforms OBIA.Rest of the contents of this paper are presented as follows. A review of prior art is provided in Section2. A description of the proposed methodology is given in Section 3 followed by the experiments, discussion, and conclusion in Sections 4, 5, and 6, respectively.


## 2. Prior Art

A number of attempts have been made in the literature for the conservation of biodiversity especially plant species despite the complexity of the task. The authors [49] deployed a computational approach consisting of image binarization for classification into background and leaf, using steps such as denoising to extract 12 shape-based features to achieve an accuracy of 90%. Jin et al. [50] proposed an approach combining binarization, contour, and corner detection as well as segmentation achieving 76% classification rate for multiclass (8 classes) classification problem. Similarly, Seeland et al. [51] proved the efficacy of speeded up robust features (SURF) in combination with scale-invariant feature transform (SIFT)-based features for plant species identification over traditional approaches. Greg et al. [52] used a dataset comprising 213 endangered plant species using point pattern analyses such as cross pair correlation function to identify how different types of ecosystems are dependent. Studies have also targeted D. pectinatum in tropical regions across three national natural reserves in China [53, 54] using artificial neural networks and support vector machine (SVM)-based classifiers. Du et al. [55] used the gap analysis technique for the conservation of 31 threatened plant species in Sanjiang Plain, China. Hamabata et al. [56] used the RNA sequencing technique to get exhaustive sequences of plant species belonging to marine life finding that these species accumulated variations which lead to their extinction.Similarly, smart agriculture practices have been applied for the monitoring of apple growth cycle [57] using pattern recognition techniques such as active shape and Gaussian distribution models. In addition, the virtual reality technology provides an environment enabling agricultural students to enhance their talents [58]. Another key technology is remote sensing used to study urban landscape-based systems in the process of urban development [59]. Similarly, fuzzy logic has also been applied to build a system for the management of environmental damage [60] as well as conservation of parrot species using transfer learning [61].Optical remote sensing images provide rich information content to recognize objects in a fundamental and challenging way, and the task of recognition from aerial images is attaining significant attention [62]. OBIA is an important type of these methods and is used for accurate and timely recognition of weeds [63], vegetation mapping [64], extracting cropland parcels for precision agriculture and other fields [65], mapping small-scale agriculture [66], and mapping of marine life [67].CNNs are gradient-based optimization networks that are used for the recognition of remote sensing data inclusive of vegetation class [68], land use [69–71], scene classification [32], and multiclass problems [72]. They are known to be universal approximators capable of efficiently representing arbitrarily complex functions given sufficient capacity, and they have excellent generalization power as well as high representational capacity.When combined with fine-tuning [73], dropout, and data augmentation [74], CNN-based deep architectures can boost the performance significantly on a given problem. However, mathematical understanding and implications of these systems are still in an early stage.


## 3. Proposed Methodology

### 3.1. Study Areas

The training and validation zone for training and validating the CNN based model is located in Cabo de Gata-Níjar Natural Park, 36°49′43″ North, 2°16′22″ West, which is located in the province of Almería, Spain. The vegetation is found to be scarce and patchy, dominated byZiziphus lotus plants that are surrounded by a mix of bare soil and small scrubs [45]. There are two test zones. The first test zone is located one and a half kilometer away from the training zone, 36°49′28″ North, 2°17′28″ West. The second test zone is located in Rizoelia National Forest Park in Cyprus, 34°56′09″ North, 33°34′26″ East [45].


### 3.2. Remote Sensing Dataset

There are two classes in our dataset:Ziziphus lotus plant class and bare soil and vegetation class. There are 180 images in the training set, 20 images in the validation set, and 11 images in the test set. There are 90 images each of the two classes in training set while there are 10 images each of the two classes in the validation set. There are six images of Ziziphus lotus class and five images of bare soil and vegetation class in the test set. Sample training and test samples are shown in Figures 1 and 2.Figure 1
A sample of bare soil and vegetation class.Figure 2
A sample ofZiziphus lotus class.


### 3.3. Classification Using 10-Fold Cross-Validation Approach

Cross-validation [75] is a common approach to assess a model in terms of its performance to an independent dataset which in our case is the validation set of 20 examples of both classes. This method helps in selecting optimal hyperparameters to overcome overfitting. There are a number of ways to implement such a strategy such as k-fold, stratified k-fold, and so on. In this work, we considered k-fold cross-validation approach where k is 10 with balanced classes in the dataset.


## 3.1. Study Areas

The training and validation zone for training and validating the CNN based model is located in Cabo de Gata-Níjar Natural Park, 36°49′43″ North, 2°16′22″ West, which is located in the province of Almería, Spain. The vegetation is found to be scarce and patchy, dominated byZiziphus lotus plants that are surrounded by a mix of bare soil and small scrubs [45]. There are two test zones. The first test zone is located one and a half kilometer away from the training zone, 36°49′28″ North, 2°17′28″ West. The second test zone is located in Rizoelia National Forest Park in Cyprus, 34°56′09″ North, 33°34′26″ East [45].


## 3.2. Remote Sensing Dataset

There are two classes in our dataset:Ziziphus lotus plant class and bare soil and vegetation class. There are 180 images in the training set, 20 images in the validation set, and 11 images in the test set. There are 90 images each of the two classes in training set while there are 10 images each of the two classes in the validation set. There are six images of Ziziphus lotus class and five images of bare soil and vegetation class in the test set. Sample training and test samples are shown in Figures 1 and 2.Figure 1
A sample of bare soil and vegetation class.Figure 2
A sample ofZiziphus lotus class.


## 3.3. Classification Using 10-Fold Cross-Validation Approach

Cross-validation [75] is a common approach to assess a model in terms of its performance to an independent dataset which in our case is the validation set of 20 examples of both classes. This method helps in selecting optimal hyperparameters to overcome overfitting. There are a number of ways to implement such a strategy such as k-fold, stratified k-fold, and so on. In this work, we considered k-fold cross-validation approach where k is 10 with balanced classes in the dataset.


## 4. Experiments

A classifier is a mapping that maps an unlabelled instance to a label. Letz be the set of unlabelled instances and G be the space of possible labels. Let a = z × γ be the set of labelled instances and X = {r1, r2, … rn} be a set consisting of n labelled instances, where ri = <fi ϵ z, gi ϵ G>. A classifier β maps an unlabelled instance f ϵ z to a label g ϵ G. The correct recognition rate of a classifier β is dependent on the probability of correctly mapping f ϵ z to g ϵ G.We use the classification architecture shown in Figure3. The input image to all the models is a tensor of shape 299 × 299 × 3 according to the requirements of Inception version 3, Xception, and Inception ResNet version 2 architectures. The image is then passed through the transfer learning model trained on ImageNet dataset features whose output is then fed to the flatten layer. Flatten layer does not affect the batch size and converts a batch of size batch × channels × height × width to a simple vector output of shape batch × (channels × height × width).Figure 3
Architecture for Xception, Inception version 3, and Inception ResNet version 2-based models.Fully connected (FC) layer 1 is made up of 256 neurons activated by rectified linear unit (ReLU) activation function. After that, there is a dropout layer with 50% probability to improve overfitting on the model by dropping neurons. Finally, there is another FC layer with a single neuron activated by sigmoid function whose purpose is to classify the input image in one of the two categories. For the training dataset, we use horizontal and vertical flipping as data augmentation techniques and normalize the input through division by 255. For the validation and test sets, we simply normalize the input through division by 255 and do not apply any other form of data augmentation. As a loss function which needs to be minimized, we used binary cross-entropy. The optimizer that we chose to minimize binary cross-entropy is the well-known stochastic gradient descent (SGD). The initial learning rate is set to 0.00001. We decay the learning rate with the passage of epochs using step decay strategy. We also used gradient clipping to make the network robust against overfitting and exploding gradient problems. We trained the model for 30 epochs by setting the batch size to 5.


### 4.1. Classification Using Xception Architecture

We used Xception architecture pretrained on the ImageNet dataset and tested the model on an independent test set. This architecture employs the idea of depthwise separable convolution performed over the channels of an input which is then passed through a pointwise 1 × 1 convolution projecting a channel’s output onto a new space. An advantage of such an approach is an efficient use of the model parameters.


### 4.2. Classification Using Inception Version 3 Architecture

Just like Xception architecture, we used Inception version 3 architecture pretrained on the ImageNet dataset and tested the model on an independent test set. In Figure4, a canonical Inception version 3 module is presented. In this architecture, cross-channel and spatial correlations are learned by the filters inside convolutional layers. It also uses the idea of auxiliary classifiers, acting as regularizes, to combat the vanishing gradient problem. Large-sized convolutional blocks are replaced with small-sized ones through the process of factorization to save the computations.Figure 4
A canonical Inception version 3 module.


### 4.3. Classification Using Inception ResNet Version 2 Architecture

Just like Xception and Inception version 3 architectures, we used Inception ResNet version 2 architecture pretrained on the ImageNet dataset and tested the model on an independent test set. This architecture is a hybrid version of both Inception and ResNet architectures. It uses residual connections to improve the training speed. In this architecture, each Inception block is followed by a 1 × 1 convolution without activation used for scaling up the dimensionality of the filter bank in order to match the depth of the input which allows for compensation in dimensionality reduction induced by the Inception block.


## 4.1. Classification Using Xception Architecture

We used Xception architecture pretrained on the ImageNet dataset and tested the model on an independent test set. This architecture employs the idea of depthwise separable convolution performed over the channels of an input which is then passed through a pointwise 1 × 1 convolution projecting a channel’s output onto a new space. An advantage of such an approach is an efficient use of the model parameters.


## 4.2. Classification Using Inception Version 3 Architecture

Just like Xception architecture, we used Inception version 3 architecture pretrained on the ImageNet dataset and tested the model on an independent test set. In Figure4, a canonical Inception version 3 module is presented. In this architecture, cross-channel and spatial correlations are learned by the filters inside convolutional layers. It also uses the idea of auxiliary classifiers, acting as regularizes, to combat the vanishing gradient problem. Large-sized convolutional blocks are replaced with small-sized ones through the process of factorization to save the computations.Figure 4
A canonical Inception version 3 module.


## 4.3. Classification Using Inception ResNet Version 2 Architecture

Just like Xception and Inception version 3 architectures, we used Inception ResNet version 2 architecture pretrained on the ImageNet dataset and tested the model on an independent test set. This architecture is a hybrid version of both Inception and ResNet architectures. It uses residual connections to improve the training speed. In this architecture, each Inception block is followed by a 1 × 1 convolution without activation used for scaling up the dimensionality of the filter bank in order to match the depth of the input which allows for compensation in dimensionality reduction induced by the Inception block.


## 5. Discussion

The training/validation accuracy and loss plots for Inception version 3 architecture are shown in Figures5 and 6.Figure 5
Training versus validation accuracy curve achieved at fold-9 for Inception version 3 architecture.Figure 6
Training versus validation loss curve achieved at fold-9 for Inception version 3.As shown in the plots, the training starts with a relatively high error as the classifier is still learning the mapping between the two classes and their corresponding labels. As it gets better and better with the passing of epochs, the classifier returns a better mapping. The fluctuations represent the learning behavior of the gradients due to the stochasticity/randomness in the samples.The training and validation accuracy and loss plots for the Xception architecture are shown in Figures7 and 8 while the training and validation accuracy and loss plots for Inception ResNet version 2 architecture are shown in Figures 9 and 10.Figure 7
Training versus validation accuracy curve achieved at fold-3 for Xception architecture.Figure 8
Training versus validation loss curve achieved at fold-3 for Xception architecture.Figure 9
Training versus validation accuracy curve achieved at fold-6 for Inception ResNet version 2 architecture.Figure 10
Training versus validation loss curve achieved at fold-6 for Inception ResNet version 2.A general trend can be seen in these plots/figures in that the Inception ResNet version 2 model performs a bit worse than the other two models. The trend depicts overfitting by the Inception ResNet version 2 architecture which could be due to a large number of parameters of this model as it naturally needs a large number of samples for training. The number of parameters that are required to train the Inception ResNet version 2 model is more than twice that of Inception version 3 and Xception models. As the number of parameters increases, so does the requirement for the number of training samples. The role of regularization such as gradient clipping is to mitigate overfitting in the regime which helps in achieving the effective capacity of the machine learning models.An interesting dichotomous phenomenon can be observed in Figures9 and 10. In Figure 9, a rapid decrease in accuracy can be observed near epoch 6 which could be explained by the fact that saturated neurons kill the gradients as ReLU activation function may get saturated in the negative region during the training process. Dead ReLU has problems getting activated or updated. Another possible reason for this phenomenon could be the loss function. The loss function has a high condition number, and, in this case, it is stuck in a local minimum with zero gradients. Learning rate is a hyperparameter of SGD, and it determines the size of the steps to reach a local minimum. The direction of slope of the surface determines the way gradients are reaching the valley since one update at a time is performed with high variance causing the objective function to fluctuate heavily and getting stuck in a local minimum due to highly non-convex nature of error function since SGD is prone to noise.We now define accuracy, precision, recall, andF1-score in the context of our experiments:(1)accuracy=TP+TNTP+TN+FP+FN,precision=TPTP+FP,recall=TPTP+FN,F1−score=2×precision×recallprecision+recall.Here true positive (TP) represents theZiziphus lotus class samples that are correctly classified as belonging to Ziziphus lotus class, false positive (FP) represents the bare soil and vegetation class samples that are incorrectly classified as belonging to Ziziphus lotus class, and true negative (TN) represents the bare soil and vegetation class samples that are correctly identified as belonging to bare soil and vegetation class, whereas false negative (FN) represents the Ziziphus lotus class samples that are incorrectly classified as belonging to bare soil and vegetation class.We achieved an accuracy of 100% on the test dataset for all the three architectures. All the 11 samples belonging to the 2 classes in the test dataset are correctly categorized in their respective classes. Hence, we achieved the perfect score of 1 on precision, recall, andF1-score for all the three architectures. Sample test dataset results on the Inception ResNet version 2 architecture for the two classes are shown in Figures 11 and 12. The class score for the Ziziphus lotus class test sample is 1, and for the bare soil and vegetation class, it is 1.5358785 × 10−10.Figure 11
Test setZiziphus lotus class sample correctly classified by Inception ResNet version 2 architecture.Figure 12
Test set bare soil and vegetation class sample correctly classified by Inception ResNet version 2 architecture.In comparison, Guirado et al. [45] reported a precision rate of 100%, recall rate of 95%, and an F1-score of 96.5% for the ResNet-based deep transfer learning classifier while they reported a precision rate of 91.78%, recall rate of 97.29%, and an F1-score of 92.90% for the non-deep learning-based OBIA classifier. The results hence prove the effectiveness of our approach through application of Inception version 3, Xception, and Inception ResNet version 2 transfer learning-based architectures for this problem.


## 6. Conclusion

In this work, we explored, analyzed, and compared the performance of deep learning architectures for the classification ofZiziphus lotus which is an endangered species in the European habitat ecosystem. We fine-tuned Inception version 3, Xception, and Inception ResNet version 2 architectures which were pretrained on ImageNet dataset-based features. We achieved promising results which established the usefulness of deep learning approaches in this domain of problems. We found that these three architectures need minimum human supervision and their inference time is minimal in comparison to the other methods which are not based on deep learning such as OBIA. In the future, we are planning to extend this study through the inclusion of more endangered shrub species in different parts of forests in dryland biomes across the globe in accordance with the global initiatives taken by the Food and Agricultural Organization of the United Nations. We are also planning to deploy novel deep learning architectures such as graph convolutional networks and capsule networks to recognize images of plant species of conservation concern.



---

*Source: 4310321-2021-12-15.xml*