# Land Cover Classification Using SegNet with Slope, Aspect, and Multidirectional Shaded Relief Images Derived from Digital Surface Model

Authors: Dae Geon Lee; Young Ha Shin; Dong-Cheon Lee
Journal: Journal of Sensors
 (2020)

---

## Abstract

Most object detection, recognition, and classification are performed using optical imagery. Images are unable to fully represent the real-world due to the limited range of the visible light spectrum reflected light from the surfaces of the objects. In this regard, physical and geometrical information from other data sources would compensate for the limitation of the optical imagery and bring a synergistic effect for training deep learning (DL) models. In this paper, we propose to classify terrain features using convolutional neural network (CNN) based SegNet model by utilizing 3D geospatial data including infrared (IR) orthoimages, digital surface model (DSM), and derived information. The slope, aspect, and shaded relief images (SRIs) were derived from the DSM and were used as training data for the DL model. The experiments were carried out using the Vaihingen and Potsdam dataset provided by the German Society for Photogrammetry, Remote Sensing and Geoinformation (DGPF) through the International Society for Photogrammetry and Remote Sensing (ISPRS). The dataset includes IR orthoimages, DSM, airborne LiDAR data, and label data. The motivation of utilizing 3D data and derived information for training the DL model is that real-world objects are 3D features. The experimental results demonstrate that the proposed approach of utilizing and integrating various informative feature data could improve the performance of the DL for semantic segmentation. In particular, the accuracy of building classification is higher compared with other natural objects because derived information could provide geometric characteristics. Intersection-of-union (IoU) of the buildings for the test data and the new unseen data with combining all derived data were 84.90% and 52.45%, respectively.


---

## Body

## 1. Introduction

The field of DL has grown significantly over the past decade coupled with rapid improvements in computer performance. Since McCulloch and Pitts introduced artificial neuron that is a computational model of the neural networks to mimic the human brain in 1943 [1], DL as a branch of machine learning has evolved steadily to this day. In recent years, advances in image processing computer vision, information and communication technology, and geoinformatics have accelerated the development and use of the DL. Many DL tasks involve visual information processing such as object recognition and identification from imagery [2–4]. It is true that the optical images provide rich, diverse, and explicit information. “Seeing is believing,” i.e., human beings have been relying on the visual information to understand the real world than through any other media. Images obtained from optical sensors are formed by recording reflected light, in the visible spectral range, from the surfaces of the terrain objects. In this aspect, it is not sufficient to reveal real-world features by utilizing image alone.Performance of the DL depends on the training data that sometimes leads to the issue of overfitting. Various schemes have been suggested to avoid overfitting such as drop-out, early-stopping, regularization, cross-validation, and hyper-parameter tuning. However, such efforts might not be the fundamental solutions to prevent overfitting. Extracting intrinsic and characteristic information from the original data to utilize training data would make DL more robust. Recent deep learning researches with a similar concept are found. Maltezos et al. proposed building detection method by training the CNN model with multidimensional features that include entropy, height variation, intensity, normalized height, planarity, and standard deviation extracted from light detection and ranging (LiDAR) data. Each feature reflects the unique physical property of the objects [5]. Audebert et al. introduced the SegNet DL model with multimodal and multiscale data for semantic labeling. Optical IR images, DSM created from LiDAR data, normalized DSM (nDSM), and normalized difference vegetation index (NDVI) derived from multispectral data are used to train the DL model [6]. Zhou et al. proposed a CNN-based AlexNet DL model with fusion of point cloud data and high-resolution images for land cover classification. Specifically, a stratified segmentation scheme with grey-level cooccurrence matrix (GLCM) [7, 8] was introduced to improve segmentation efficiency for increasing classification accuracy. Mean, variance, homogeneity, dissimilarity, and entropy from GLCM features were DL model training process [9]. Alidoost and Arefi applied LeNet-5 [10] for automatic recognition of various roof types by training features extracted from both LiDAR data and orthoimages [11]. Pibre et al. presented multiple source data fusion strategies to detection trees [12]. Two different modes were applied to integrate heterogeneous data for training DL models that is a similar idea to [6]. In the study, early fusion and late fusion of the IR aerial images and NDVI and DSM created from LiDAR data were performed. Better results were obtained with early fusion when both NDVI and DSM were used [12, 13].The recent researches of DL have focused on efforts to improve training performance by utilizing multisource data and/or creating information from the raw data. In the field of geoinformatics and remote sensing, the major data sources are optical and laser sensors. Point cloud data that consists of 3D coordinates can be obtained directly from laser sensors (i.e., LiDAR data) or indirectly by stereo image matching. In addition, derived information from the original raw data are utilized. Examples of the derived information are NDVI and DSM created from multispectral imagery and LiDAR data, respectively. Combining multisource data (e.g., optical and multispectral imagery, point cloud data, and DSM) with derived information from original raw data (e.g., NDVI, cooccurrence features, and surface orientation) provide more reliable results.As for the DL models, CNN is one of the most extensively used models and has been successfully applied to high-level image processing (or late-vision in computer vision) tasks such as image recognition, scene analysis and description, classification, and medical image computing. Well-known CNN models are LeNet, AlexNet, GoogLeNet, VGGNet, ResNet, Mask R-CNN, SegNet, etc. Most of them are winners of the Image Net Large Scale Visual Recognition Challenge (ILSVCR) [14]. One of the major applications of DL is object classification from images [15, 16]. The goal of this paper is to perform semantic segmentation for land cover classification using CNN based SegNet model. The training datasets are composed of IR images and DSM with derived information from the DSM. The derived information includes surface orientation (i.e., slope and aspect of each DSM grid cell) and multidirectional SRIs. DSM is a 3D representation of the terrain surface features, including natural and man-made objects, that are formed with dense point clouds of the 3D coordinates (X, Y, Z). In the field of geoinformatics, the point clouds with 3D coordinates can be obtained directly from laser sensors (i.e., LiDAR data) or indirectly by stereo image matching. Since images are 2D data and provide reflected light from the terrain surfaces, images lack 3D information about the real-world objects. On the other hand, DSM could provide richer 3D geometric information of the objects than the imagery [17].Feature is a primitive characteristic or attribute of the data. Thus, it is important to extract or derive unique features from the various data, then use those features for DL model training for semantic segmentation. Segmentation entails division or separation of the data into regions of similar characteristics or attributes. Therefore, the purpose of the segmentation is to form meaningful regions by grouping features that have common properties distinct from their neighboring regions [18]. In general, segmentation does not involve classifying each segment but rather subdivides regions without attempting to recognize the individual regions. On the other hand, semantic segmentation or classification involves identification and labeling of the regions or individual objects according to the category [15]. There is no theory or standard method available for the segmentation yet. Most of the existing methods are based on the hoc approach; therefore, DL is expected to be a promising innovative method to solve such a challenging task that requires human intelligence.The experiments for this paper were carried out using the ISPRS benchmark dataset for terrain feature classification. The land cover of the study site is categorized into six classes; building, tree, low vegetation (grass), impervious surface (road), car, water, and clutter/background in the label data. The main intent of this paper is to classify terrain objects by training the DL model using multisource data; optical IR images, DSM, and DSM-derived data including slope, aspect, and multidirectional SRIs. The experiments were carried out as follows: (1) training with each type of data independently, and (2) training with combining all data. Each data collected from a specific sensor could not convey sufficient information about the real-world. In this aspect, multisource data would be complementary in the training process. The results were analyzed based on evaluation metrics and visual inspection. In conclusion, training by combining multisource data could provide a synergistic effect and multidirectional SRI plays an important role.


## 2. Materials and Proposed Methods

### 2.1. Description of Datasets

The ISPRS benchmark dataset [19, 20] was used for training, evaluation, and test of the SegNet model. Table 1 shows the description of the datasets, and Figure 1 presents the configuration of Vaihingen datasets. The datasets consist of IR true orthoimages, DSM, airborne LiDAR data, and label data. True orthoimages are geometrically corrected, occlusion-free, and georeferenced images. 60%, 20%, and 20% of the datasets were used as training, evaluation, and test data, respectively. Figure 2 is another dataset from the Potsdam area that was used as “new unseen” data to apply to the trained DL model that was trained with the Vaihingen datasets. Four datasets that are 3_12, 4_10, 4_11, and 7_08 area were selected as new unseen data for the experiments. It is notable to mention that LiDAR data was not used in the experiment; instead, DSM was used to derive slope and aspect and to create multidirectional SRIs. DSM could be generated using LiDAR data. However, some significant preprocessing is required such as noise removing and interpolation because of the irregularly distributed point clouds with low point density of the LiDAR data. The high-resolution DSM in the ISPRS dataset was created using INPHO MATCH-DSM software with sequential multiimage matching and finite element interpolation [21]. In this regard, we decided that utilizing DSM is more appropriate than LiDAR data to perform the proposed approach.Table 1
Description of ISPRS benchmark dataset for the Vaihingen area.
Data typeGSD (m)Data collectionCommentColor IR (IR-R-G) true orthoimage0.09Intergraph DMC (altitude: 900 m)33 patches with georeferenced TIFF formatDSM0.09Point clouds created by image matching16 patches with georeferenced TIFF formatLiDAR data0.50 (4 pts/m2)Leica ALS50 (altitude: 500 m)10 strips with LAS formatLabel data0.09Manually digitized ground truthLand cover classification of 6 classes with TIFF formatFigure 1
Vaihingen dataset is composed with IR images, DSM, and label data for training, validation, and test.Figure 2
New unseen dataset from the Potsdam area.


### 2.2. Slope and Aspect

The slope and aspect (i.e., surface orientation) of the surface elements (i.e., DSM grid cells) can be computed from the coordinates of the3×3 grid window of the DSM (see Figure 3).Figure 3
3×3 grid window of DSM for computing slope and aspect.Slope and aspect of each surface element is computed by Equations (1)–(10).
(1)dX1=ZX+1,Y+1−ZX−1,Y+12∙∆X,(2)dX2=ZX+1,Y−ZX−1,Y2∙∆X,(3)dX3=ZX+1,Y−1−ZX−1,Y−12∙∆X.The average slope inX-direction is
(4)SX=dX1+dX2+dX33.Slopes inY-direction are computed by
(5)dY1=ZX−1,Y+1−ZX−1,Y−12∙∆Y,(6)dY2=ZX,Y+1−ZX,Y−12∙∆Y,(7)dY3=ZX+1,Y+1−ZX+1,Y−12∙∆Y.The average slope inY-direction is
(8)SY=dY1+dY2+dY33,where ∆X and ∆Y are ground sampling distance (GSD) of DSM in X- and Y-direction, respectively. Finally, slope (S) and aspect (A) at the center of the 3×3 DSM grid window are computed as follows [22]:
(9)S=tan−1SX2+SY2,(10)A=tan−1SYSX.Figure4 illustrates examples of DSM, slope, and aspect using Vaihingen test data.Figure 4
DSM and derived data. (a) DSM. (b) Slope. (c) Aspect.
(a)(b)(c)


### 2.3. Multidirectional Shaded Relief Images

Shading is an important visual cue (i.e., shape-from-shading) to recognize the shape of the objects because each surface patch of the objects appears in different brightness depending on the orientation of the surfaces [23–25]. The SRIs can be created using surface orientation (i.e., slope and aspect) of the DSM and virtual light source. The amount of the reflected light from each surface element is recorded in the corresponding pixel of the SRIs and computed by Equation (11).
(11)R=cos90°–βcosS+sin90°–βsinScosα–A,where R is the magnitude of the reflected light, α and β are azimuth and elevation angle of the light source, respectively, and S and A are slope and aspect of the surface element, respectively. Each value of R is converted to an 8-bit image with range from 0 to 255. The multidirectional SRIs are generated by changing the location of the light source [26]. Four SRIs were generated with light sources of NW, NE, SE, and SW direction (see Figure 5).Figure 5
Multidirectional shaded relied images created from different locations of the light source.


### 2.4. Preparation of Dataset for Training, Evaluation, and Label Data

Each region of the dataset, including training, evaluation, and label data, is partitioned with a 50% overlap along bothX and Y directions, and the partitioned tile size is 150×250 as shown in Figure 6. Data partitioning, especially with overlapping, could provide significant benefits: (1) increasing amount of the training data because DL requires a large number of data, and (2) improving DL performance by training the same object (or area) in various situations.Figure 6
Preparation of training, evaluation, and label dataset by partitioning with 50% overlap.


### 2.5. SegNet Model

DL using the CNN has been proved to be successful and effective on semantic segmentation (or called semantic labeling) [15]. The SegNet model is a deep fully CNN for multiclass pixel-wise semantic segmentation. SegNet, by Computer Vision and Robotics Group at the University of Cambridge, was designed to be efficient in memory and computational time. It is also significantly smaller in the number of the trainable parameters than other network models. The SegNet is composed of encoder and decoder with symmetrical architecture, and the encoder is based on the convolutional layers from VGG16 that has 13 convolutional and 3 fully-connected layers. VGG16 has been developed by Visual Geometry Group of the University of Oxford and 1st runner-up of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) with an error rate of under 10% in 2014 [27].On the other hand, the fully-connected layers in VGG16 are replaced by convolutional layers in SegNet. The SegNet uses a bottleneck architecture in which the feature maps are upsampled to match the original input resolution, and therefore it is possible to perform pixel-wise segmentation with one-to-one resolution. This is beneficial to label data with the same resolution of the original input data [6]. As with most of the CNN models, the encoder network of the SegNet performs convolution with a filter bank to produce feature maps, batch normalization to increase the stability of the network, and accelerate training [28]. Then, rectified linear unit (ReLU) is applied as an activation function, and max-pooling with a 2×2 window and stride 2 is performed.The number of convolutional kernels (i.e., filters) is from 64 to 512 with a size of3×3. The decoder network upsamples the feature maps using max-pooling indices from the corresponding encoder feature maps. The feature maps are convolved with a trainable decoder filter bank to produce dense feature maps. Then, batch normalization is applied to each feature map and then fed to the softmax classifier at the output layer of the decoder (see Figure 7). Softmax is used to map the nonnormalized output of the network to a probability distribution over predicted classes [29].Figure 7
SegNet model architecture. Series of dots represent convolutional layer groups.


### 2.6. SegNet Model Training

Partitioned patches (or tiles) of the multisource dataset (i.e., IR orthoimage, DSM, slope, aspect, and multidirectional SRIs) generated by partitioning are fed into the SegNet model for training and evaluation. The numbers of the input patches of each data are 4870, 1730, and 1390 for training, evaluation, and test, respectively. Most of the current DL models require a trial-and-error method to achieve optimal solutions. This approach is not trivial and takes ample training time because diverse parameters are involved in the DL model. The combination of different parameter setting leads to tremendous numbers of cases. Therefore, transfer learning was adopted to determine the hyper-parameters including learning rate, mini-batch size, and number of the epochs. Another consideration is normalized DSM (nDSM). The nDSM is obtained by subtracting bare-ground height (i.e., digital terrain model (DTM)) from the DSM as Equation (12):
(12)nDSM=DSM–DTM.Many DL models utilize nDSM instead of DSM [6, 9, 30]. The main reason to use nDSM is that the nDSM only reflects relative heights of the objects regardless of the terrain elevation. In other words, differences among objects in elevation are taken into account in nDSM [31]. DTM has to be available to create nDSM (e.g., contours from topographic maps) or complicated filtering process to separate ground and nonground features (e.g., morphological filtering, progressive densification, surface-based method, and segment-based method) is required [32]. However, the robust DL has to require less preprocessing such as generating nDSM. There might be some controversial issues about using nDSM for object recognition or classification. The ultimate goal of DL is to resemble human intelligence. Human beings are able to recognize objects regardless of their location in 3D space. Namely, it would be invariant with respect to geometric transformation including shift, rotation, and scale.nDSM might be a controversial issue in DL. The concept of nDSM is that all objects have to be vertically relocated onto the reference datum before training DL models that utilize DSM. However, nDSM might distort the shape of the objects in some cases. For example, a building with flat roof on a sloped surface is to be a building with a sloped roof if nDSM is applied as shown in Figure8.Figure 8
nDSM changes the original shape of object. (a) DSM. (b) nDSM.ZT and ZS denote terrain and object height from datum, respectively.Another issue of the nDSM is for DSM obtained from terrestrial LiDAR data or images of the street scene. If nDSM should be used, then objects have to be moved horizontally to the reference plane. In this case, the horizontal reference plane must be defined. Since DL attempts to resemble human recognition ability, it is much better and robust models that nDSM is not required because human beings recognize objects regardless of objects’ location. nDSM should not affect the computation of slope and aspect and creating SRI. In this matter, we did not use nDSM as training but the original DSM. DSM and DSM-derived data (i.e., slope, aspect, and SRI) could provide 3D spatial geometric characteristics that are important information to identify and distinguish different types of objects while images provide only 2D visual information. In this regard, we propose utilizing various data for training the DL model to obtain reliable results. Figure9 shows the workflow of our proposed method.Figure 9
Workflow of proposed method for land cover classification.


### 2.7. Evaluation Metrics for Performance Measures

Evaluating performance is one of the fundamental tasks in DL. Classification accuracy, in general, can be described as the number of the correct predictions from the total predictions performed. Classification accuracy alone is not sufficient to evaluate the performance of the DL model. Since the ultimate goal of DL is to expand the trained model to other datasets (i.e., new unseen data) that is not involved with training, it is an important issue to evaluate the robustness of a DL model for the new unseen data. The test data has similar characteristics to the training data since the test data, in most cases, belongs to the same region of the training data (i.e., Vaihingen dataset), while the new unseen data has somewhat different characteristics because the new unseen data is selected from a different place (i.e., Potsdam dataset).Different evaluation criteria have been proposed to assess the quality of the semantic segmentation. Commonly used evaluation metrics for classification are accuracy, precision, recall, F1 score, and intersection-of-union (IoU) [33]. Usually, variations on pixel accuracy and IoU have been used frequently [34]. We applied overall accuracy (Equation (13)) and IoU (Equation (14)) to evaluate semantic segmentation results:
(13)Accuracy=TP+TNTP+TN+FP+FN,(14)IoU=AreaofoverlapAreaofunion=Predictedarea∩ActualareaPredictedarea∪Actualarea=TPTP+FP+FN,where TP is true positive, TN is true negative, FP is false positive, and FN is false negative.Accuracy might lead to misinterpretation when the class representation is small within the image, as the measure is biased in mainly reporting on how well in identifying negative cases. On the other hand, IoU is calculated for each class separately, then averaged over all classes to provide mean IoU score of semantic segmentation prediction. The criterion to be correct or not to be is 0.5. If IoU is larger than 0.5 (i.e., 50%), it is normally considered a good prediction. IoU has been used in numerous papers and popular object detection challenges such as ImageNet ILSVRC, Microsoft COCO, and PASCAL VOC.


## 2.1. Description of Datasets

The ISPRS benchmark dataset [19, 20] was used for training, evaluation, and test of the SegNet model. Table 1 shows the description of the datasets, and Figure 1 presents the configuration of Vaihingen datasets. The datasets consist of IR true orthoimages, DSM, airborne LiDAR data, and label data. True orthoimages are geometrically corrected, occlusion-free, and georeferenced images. 60%, 20%, and 20% of the datasets were used as training, evaluation, and test data, respectively. Figure 2 is another dataset from the Potsdam area that was used as “new unseen” data to apply to the trained DL model that was trained with the Vaihingen datasets. Four datasets that are 3_12, 4_10, 4_11, and 7_08 area were selected as new unseen data for the experiments. It is notable to mention that LiDAR data was not used in the experiment; instead, DSM was used to derive slope and aspect and to create multidirectional SRIs. DSM could be generated using LiDAR data. However, some significant preprocessing is required such as noise removing and interpolation because of the irregularly distributed point clouds with low point density of the LiDAR data. The high-resolution DSM in the ISPRS dataset was created using INPHO MATCH-DSM software with sequential multiimage matching and finite element interpolation [21]. In this regard, we decided that utilizing DSM is more appropriate than LiDAR data to perform the proposed approach.Table 1
Description of ISPRS benchmark dataset for the Vaihingen area.
Data typeGSD (m)Data collectionCommentColor IR (IR-R-G) true orthoimage0.09Intergraph DMC (altitude: 900 m)33 patches with georeferenced TIFF formatDSM0.09Point clouds created by image matching16 patches with georeferenced TIFF formatLiDAR data0.50 (4 pts/m2)Leica ALS50 (altitude: 500 m)10 strips with LAS formatLabel data0.09Manually digitized ground truthLand cover classification of 6 classes with TIFF formatFigure 1
Vaihingen dataset is composed with IR images, DSM, and label data for training, validation, and test.Figure 2
New unseen dataset from the Potsdam area.


## 2.2. Slope and Aspect

The slope and aspect (i.e., surface orientation) of the surface elements (i.e., DSM grid cells) can be computed from the coordinates of the3×3 grid window of the DSM (see Figure 3).Figure 3
3×3 grid window of DSM for computing slope and aspect.Slope and aspect of each surface element is computed by Equations (1)–(10).
(1)dX1=ZX+1,Y+1−ZX−1,Y+12∙∆X,(2)dX2=ZX+1,Y−ZX−1,Y2∙∆X,(3)dX3=ZX+1,Y−1−ZX−1,Y−12∙∆X.The average slope inX-direction is
(4)SX=dX1+dX2+dX33.Slopes inY-direction are computed by
(5)dY1=ZX−1,Y+1−ZX−1,Y−12∙∆Y,(6)dY2=ZX,Y+1−ZX,Y−12∙∆Y,(7)dY3=ZX+1,Y+1−ZX+1,Y−12∙∆Y.The average slope inY-direction is
(8)SY=dY1+dY2+dY33,where ∆X and ∆Y are ground sampling distance (GSD) of DSM in X- and Y-direction, respectively. Finally, slope (S) and aspect (A) at the center of the 3×3 DSM grid window are computed as follows [22]:
(9)S=tan−1SX2+SY2,(10)A=tan−1SYSX.Figure4 illustrates examples of DSM, slope, and aspect using Vaihingen test data.Figure 4
DSM and derived data. (a) DSM. (b) Slope. (c) Aspect.
(a)(b)(c)


## 2.3. Multidirectional Shaded Relief Images

Shading is an important visual cue (i.e., shape-from-shading) to recognize the shape of the objects because each surface patch of the objects appears in different brightness depending on the orientation of the surfaces [23–25]. The SRIs can be created using surface orientation (i.e., slope and aspect) of the DSM and virtual light source. The amount of the reflected light from each surface element is recorded in the corresponding pixel of the SRIs and computed by Equation (11).
(11)R=cos90°–βcosS+sin90°–βsinScosα–A,where R is the magnitude of the reflected light, α and β are azimuth and elevation angle of the light source, respectively, and S and A are slope and aspect of the surface element, respectively. Each value of R is converted to an 8-bit image with range from 0 to 255. The multidirectional SRIs are generated by changing the location of the light source [26]. Four SRIs were generated with light sources of NW, NE, SE, and SW direction (see Figure 5).Figure 5
Multidirectional shaded relied images created from different locations of the light source.


## 2.4. Preparation of Dataset for Training, Evaluation, and Label Data

Each region of the dataset, including training, evaluation, and label data, is partitioned with a 50% overlap along bothX and Y directions, and the partitioned tile size is 150×250 as shown in Figure 6. Data partitioning, especially with overlapping, could provide significant benefits: (1) increasing amount of the training data because DL requires a large number of data, and (2) improving DL performance by training the same object (or area) in various situations.Figure 6
Preparation of training, evaluation, and label dataset by partitioning with 50% overlap.


## 2.5. SegNet Model

DL using the CNN has been proved to be successful and effective on semantic segmentation (or called semantic labeling) [15]. The SegNet model is a deep fully CNN for multiclass pixel-wise semantic segmentation. SegNet, by Computer Vision and Robotics Group at the University of Cambridge, was designed to be efficient in memory and computational time. It is also significantly smaller in the number of the trainable parameters than other network models. The SegNet is composed of encoder and decoder with symmetrical architecture, and the encoder is based on the convolutional layers from VGG16 that has 13 convolutional and 3 fully-connected layers. VGG16 has been developed by Visual Geometry Group of the University of Oxford and 1st runner-up of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) with an error rate of under 10% in 2014 [27].On the other hand, the fully-connected layers in VGG16 are replaced by convolutional layers in SegNet. The SegNet uses a bottleneck architecture in which the feature maps are upsampled to match the original input resolution, and therefore it is possible to perform pixel-wise segmentation with one-to-one resolution. This is beneficial to label data with the same resolution of the original input data [6]. As with most of the CNN models, the encoder network of the SegNet performs convolution with a filter bank to produce feature maps, batch normalization to increase the stability of the network, and accelerate training [28]. Then, rectified linear unit (ReLU) is applied as an activation function, and max-pooling with a 2×2 window and stride 2 is performed.The number of convolutional kernels (i.e., filters) is from 64 to 512 with a size of3×3. The decoder network upsamples the feature maps using max-pooling indices from the corresponding encoder feature maps. The feature maps are convolved with a trainable decoder filter bank to produce dense feature maps. Then, batch normalization is applied to each feature map and then fed to the softmax classifier at the output layer of the decoder (see Figure 7). Softmax is used to map the nonnormalized output of the network to a probability distribution over predicted classes [29].Figure 7
SegNet model architecture. Series of dots represent convolutional layer groups.


## 2.6. SegNet Model Training

Partitioned patches (or tiles) of the multisource dataset (i.e., IR orthoimage, DSM, slope, aspect, and multidirectional SRIs) generated by partitioning are fed into the SegNet model for training and evaluation. The numbers of the input patches of each data are 4870, 1730, and 1390 for training, evaluation, and test, respectively. Most of the current DL models require a trial-and-error method to achieve optimal solutions. This approach is not trivial and takes ample training time because diverse parameters are involved in the DL model. The combination of different parameter setting leads to tremendous numbers of cases. Therefore, transfer learning was adopted to determine the hyper-parameters including learning rate, mini-batch size, and number of the epochs. Another consideration is normalized DSM (nDSM). The nDSM is obtained by subtracting bare-ground height (i.e., digital terrain model (DTM)) from the DSM as Equation (12):
(12)nDSM=DSM–DTM.Many DL models utilize nDSM instead of DSM [6, 9, 30]. The main reason to use nDSM is that the nDSM only reflects relative heights of the objects regardless of the terrain elevation. In other words, differences among objects in elevation are taken into account in nDSM [31]. DTM has to be available to create nDSM (e.g., contours from topographic maps) or complicated filtering process to separate ground and nonground features (e.g., morphological filtering, progressive densification, surface-based method, and segment-based method) is required [32]. However, the robust DL has to require less preprocessing such as generating nDSM. There might be some controversial issues about using nDSM for object recognition or classification. The ultimate goal of DL is to resemble human intelligence. Human beings are able to recognize objects regardless of their location in 3D space. Namely, it would be invariant with respect to geometric transformation including shift, rotation, and scale.nDSM might be a controversial issue in DL. The concept of nDSM is that all objects have to be vertically relocated onto the reference datum before training DL models that utilize DSM. However, nDSM might distort the shape of the objects in some cases. For example, a building with flat roof on a sloped surface is to be a building with a sloped roof if nDSM is applied as shown in Figure8.Figure 8
nDSM changes the original shape of object. (a) DSM. (b) nDSM.ZT and ZS denote terrain and object height from datum, respectively.Another issue of the nDSM is for DSM obtained from terrestrial LiDAR data or images of the street scene. If nDSM should be used, then objects have to be moved horizontally to the reference plane. In this case, the horizontal reference plane must be defined. Since DL attempts to resemble human recognition ability, it is much better and robust models that nDSM is not required because human beings recognize objects regardless of objects’ location. nDSM should not affect the computation of slope and aspect and creating SRI. In this matter, we did not use nDSM as training but the original DSM. DSM and DSM-derived data (i.e., slope, aspect, and SRI) could provide 3D spatial geometric characteristics that are important information to identify and distinguish different types of objects while images provide only 2D visual information. In this regard, we propose utilizing various data for training the DL model to obtain reliable results. Figure9 shows the workflow of our proposed method.Figure 9
Workflow of proposed method for land cover classification.


## 2.7. Evaluation Metrics for Performance Measures

Evaluating performance is one of the fundamental tasks in DL. Classification accuracy, in general, can be described as the number of the correct predictions from the total predictions performed. Classification accuracy alone is not sufficient to evaluate the performance of the DL model. Since the ultimate goal of DL is to expand the trained model to other datasets (i.e., new unseen data) that is not involved with training, it is an important issue to evaluate the robustness of a DL model for the new unseen data. The test data has similar characteristics to the training data since the test data, in most cases, belongs to the same region of the training data (i.e., Vaihingen dataset), while the new unseen data has somewhat different characteristics because the new unseen data is selected from a different place (i.e., Potsdam dataset).Different evaluation criteria have been proposed to assess the quality of the semantic segmentation. Commonly used evaluation metrics for classification are accuracy, precision, recall, F1 score, and intersection-of-union (IoU) [33]. Usually, variations on pixel accuracy and IoU have been used frequently [34]. We applied overall accuracy (Equation (13)) and IoU (Equation (14)) to evaluate semantic segmentation results:
(13)Accuracy=TP+TNTP+TN+FP+FN,(14)IoU=AreaofoverlapAreaofunion=Predictedarea∩ActualareaPredictedarea∪Actualarea=TPTP+FP+FN,where TP is true positive, TN is true negative, FP is false positive, and FN is false negative.Accuracy might lead to misinterpretation when the class representation is small within the image, as the measure is biased in mainly reporting on how well in identifying negative cases. On the other hand, IoU is calculated for each class separately, then averaged over all classes to provide mean IoU score of semantic segmentation prediction. The criterion to be correct or not to be is 0.5. If IoU is larger than 0.5 (i.e., 50%), it is normally considered a good prediction. IoU has been used in numerous papers and popular object detection challenges such as ImageNet ILSVRC, Microsoft COCO, and PASCAL VOC.


## 3. Experimental Results and Discussion

Semantic segmentation results of three test datasets from Vaihingen data (i.e., areas I, K, and O in Figure1) and four new unseen datasets from Potsdam data (i.e., areas 3_12, 4_10, 4_11, and 7_08 in Figure 2) are presented. The major hyper-parameters for experiments were set as shown in Table 2.Table 2
Hyper-parameter values for experiments.
ParameterValueLearning rate0.001Training cycle (iteration)334,675Maximum iteration608,500Iteration/epoch1217/275Validation frequency1217Mini-batch size4ShuffleEvery epochFigures10 and 11 are IR image and label data of Vaihingen and Potsdam datasets, respectively. Figures 12 and 13 show the classification results of Vaihingen and Potsdam datasets, respectively. Results of the Potsdam data were obtained by using trained model with the Vaihingen dataset. It is obvious that the accuracy of the test data is higher than that of the new unseen data.Figure 10
Vaihingen test area. (a) IR images. (b) Label data (ground truth).
(a)(b)Figure 11
Potsdam new unseen data. (a) IR images. (b) Label data (ground truth).
(a)(b)Figure 12
Classification results of Vaihingen test data. (a) IR images. (b) DSM. (c) Slope. (d) Aspect. (e) Multidirectional SRI. (f) Combining all data.
(a)(b)(c)(d)(e)(f)Figure 13
Classification results of Potsdam new unseen data. (a) IR images. (b) DSM. (c) Slope. (d) Aspect. (e) Multidirectional SRI. (f) Combining all data.
(a)(b)(c)(d)(e)(f)Classification accuracies of the training, evaluation, test, and new unseen data are listed in Tables3 and 4 (Note: evaluation metrics are expressed in percent (%)). The result from combining all data (IoU=84.90%) for the test data was improved compared with the results from individually trained data. However, multidirectional SRI provided better results (IoU=66.27%) than combining all data (IoU=52.45%) for new unseen data. We expect that multidirectional SRI has a relatively larger contribution to the model. Even though IoU from combining all data is lower than that from multidirectional SRI, in general, IoU with higher than 0.5 (i.e., 50%) is considered a successful classification. Since man-made features such as buildings have distinctive geometric characteristics compared to natural features, training DL model using DSM-derived data is particularly effective to buildings. Therefore, the proposed approach might be feasible to identify and extract buildings for the further application of 3D building modeling.Table 3
Classification evaluation of Vaihingen dataset for training, evaluation, and test.
DatasetTraining accuracy (%)Evaluation accuracy (%)Test data (%)OverallBuildingAccuracyIoUAccuracyIoUIR image97.7491.4983.7157.9889.3484.08DSM91.9785.2569.9041.3283.8877.66Slope94.4288.4167.9340.3178.5971.28Aspect95.8789.0764.5535.9079.0170.58SRI93.5089.3770.0042.5983.7475.90All data91.3690.6183.5655.7589.4584.90Table 4
Classification evaluation of Potsdam dataset for new unseen data.
DatasetNew unseen data (%)OverallBuildingAccuracyIoUAccuracyIoUIR image45.5623.3494.2435.95DSM43.8617.6682.7144.89Slope41.0918.1177.6837.68Aspect32.9711.6861.8034.24SRI52.5328.2877.9266.27All data58.8630.7589.3752.45The evaluation metrics could not provide information on how well ground truth and predicted classes for individual objects matched. In this matter, differences between each result and label data were depicted for visual evaluation. White and black indicate correctly classified and misclassified pixels, respectively (see Figures14 and 15). Interpolation of the DSM might cause delocalization and zagging effect, especially on the objects’ boundaries. Thus, most of the misclassified pixels are found along the boundaries of the objects.Figure 14
Differences between classification results and label data of Vaihingen test data. (a) IR image. (b) DSM. (c) Slope. (d) Aspect. (e) Multidirectional SRI. (f) Combining all data.
(a)(b)(c)(d)(e)(f)Figure 15
Differences between classification results and label data of Potsdam new unseen data. (a) IR image. (b) DSM. (c) Slope. (d) Aspect. (e) Multidirectional SRI. (f) Combining all data.
(a)(b)(c)(d)(e)(f)For the Vaihingen test data, some of the low vegetation areas were classified as impervious surfaces (i.e., roads) with DSM and DSM-derived data (i.e., slope, aspect, and SRI). However, training the DL model using all data could improve classification results (see Figure14(f)). As for the Potsdam new unseen data, more misclassified regions were found except for the case of buildings. Similar to the Vaihingen test data, a training model using a combination of all data provided better results (see Figure 15(f)). Test data might be insufficient to evaluate the performance of the trained DL models particularly to investigate the possibility of universal use. For this reason, we applied not only test data but also new unseen data to the trained model; as a result, some meaningful results were confirmed. Utilizing multisource data and derived information that are appropriate to represent characteristics of the objects could improve semantic segmentation. DSM and DSM-derived information could play an important role in recognizing buildings.Classification results from training all data (i.e., IR image, DSM, slope, aspect, and SRI) are better than results from training individual data. The experiments show that DSM and its derived information could be appropriate to recognize man-made objects including buildings because DSM and DSM-derived information have explicit geometric characteristics. In consequence, the classification accuracy of the building is much higher than that of other objects. Figures16–18 show buildings of the label data and corresponding classification results from using all data of Vaihingen and Potsdam areas, respectively. It is noticeable that buildings in the test data were well identified, while buildings in the new unseen data were less accurately identified. A considerable number of cars of the Potsdam data were identified as buildings; however, most of the major buildings are correctly classified.Figure 16
Building class of Vaihingen test data. (a) Label data. (b) Results from combining all data.
(a)(b)Figure 17
Building class of Potsdam new unseen data. (a) Label data. (b) Results from combining all data.
(a)(b)Figure 18
Comparison of (a) classification result, (b) morphological filtering, and (c) label data of Potsdam new unseen data.
(a)(b)(c)Some conventional image processing technique (e.g., morphological filtering with erosion and dilation [35]) might improve the results. Figure 18 illustrates an example of applying morphological filtering to the building class from Potsdam data. Therefore, it is recommendable to consider postprocessing if necessary.


## 4. Conclusions

Human beings have the ability to integrate various visual cues (i.e., shape, size, color, tone, texture, depth, stereopsis, and motion) based on knowledge, experience, and innate cognition for visual perception. Semantic segmentation or classification involves object detection, recognition, and identification by utilizing various information. In consequence, it is not sufficient to successfully carry out the semantic segmentation of real-world objects by one type of sensory data. Every object has its own unique physical and geometrical characteristics. Such intrinsic characteristics could be obtained partially from imagery, 3D data, and derived information from various sources. Therefore, one of the key tasks is to utilize multisource data for complementary effects to extract characteristics of the objects from the data.In this paper, we proposed a DL model that utilizes multisource data including optical IR image, DSM, and DSM-derived slope, aspect, and multidirectional SRIs for semantic segmentation to classify land cover features. The dataset used for training the DL model are based on the DSM that has distinct 3D geometric characteristics. Therefore, specific objects (i.e., buildings) are identified well dominantly. Nevertheless, overall results from the new unseen data were not quite satisfied; buildings were adequately identified compared to the other objects due to the DSM with derived information. Each object has its own unique characteristics with various aspects. The key task of DL is to reveal representative features from the objects during training. Therefore, training by utilizing various types of data that are suitable to specific objects could improve the performance of DL. In particular, DSM-derived data would be helpful to identify buildings.In general, the scalability of the trained DL models is evaluated using test data. The problem of using the test data is that, in most cases, the test data belongs to the same area with similar properties as the training data. Therefore, we evaluated the model using a dataset from different areas, i.e., test data from the Vaihingen area, and new unseen data from the Potsdam area. We found some meaningful results from the test data and new unseen data. Combining all data yielded the highest IoU of the building for the test data (i.e., 84.90%). On the other hand, multidirectional SRI provided the highest IoU of the building (i.e., 66.27%) while combining all data yielded IoU of 52.45%, but it was higher than individually trained data. We expect that multidirectional SRI plays an important role for training DL model.The ultimate goal of DL is to expand trained models for universal use. However, the major problem of DL is data dependency. Geospatial data has a variety of regional properties; hence, it is a valuable task to provide pretrained models suitable for the specific regions (e.g., urban, residential, mountainous, and agricultural areas). In addition, flexible transfer learning that could avoid training DL models from scratch would make DL a more powerful tool in various fields. In addition, the integration strategy of various combinations of the multisource dataset might maximize the synergistic effect, for example, applying different priorities or weights for each dataset.



---

*Source: 8825509-2020-09-12.xml*