<table><caption>Table 3.1 Summary of ground truth sample points collected for study.</caption><thead><tr><th rowspan="2">Land Cover</th><th colspan="2">Roorkee</th></tr><tr><th>Training and validation</th><th>Testing</th></tr></thead><tbody><tr><td>BS</td><td>200</td><td>40</td></tr><tr><td>TV</td><td>200</td><td>40</td></tr><tr><td>SV</td><td>200</td><td>40</td></tr><tr><td>Built-up</td><td>200</td><td>40</td></tr><tr><td>Water</td><td>200</td><td>40</td></tr><tr><td>Total</td><td>1000</td><td>200</td></tr></tbody></table>

<figure><img src="image_24.png" /><figcaption>Figure 3.3 Sample images of different land covers collected during ground truth survey. (a) Tall vegetation, (b) bare soil, (c) agriculture cropland, and (d) sugarcane field.</figcaption></figure>

### 3.3.4. Preprocessing

The PALSAR-2 data considered in the study is a single look complex (SLC) data. Preprocessing is required before using the data for the study. The steps involved in preprocessing are similar to as discussed in [168]. A Gamma MAP filter of window size 3×3 for the purpose of speckle removal is used as this filter retains polarimetric signature characteristics of SAR image [169], and [170]. This filter also removes speckle noise from gamma distributed scenes such as forested areas and agricultural lands whereas keeping the original pixel values for non-gamma distributed scenes such as water. The window size is finalized on the basis of spatial resolution of input image and application. Geocoding is done using SRTM 3 arc-second global DEM model. Other tasks like radiometric calibration and terrain correction are performed similar to [168]. The
output of this step is a pre-processed backscattering or S-matrix. Four features namely HH, HV, VH, and VV are obtained. Since, HV and VH are similar due to the reciprocity theorem, only HH, HV, and VV are used further in the current study.

Figure 3.1(a) show a false colour composite image of the pre-processed Roorkee study area. It is observed from the false colour composite image that some parts of built-up areas are showing volume scattering (represented by green colour) behaviour similar to dense vegetated areas such as tree plantations. For example, in Figure 3.1(a), built-up area (marked as A) is showing backscatter behaviour similar to backscatter behaviour of tree plantations (marked as B). It can be further observed from enlarged patches shown in Figure 3.1(c). It is because of the randomly oriented, densely packed and non-similar structures constitute the built-up area and hence is showing volume scattering as dominant scattering. The main objective of the current study is to develop features and classification techniques that can overcome this challenge and achieve higher classification performances.

**3.4. Sub Task 1: Polarization Signatures: Critical Analysis as Potential PolSAR Features for LC Classification**

This section discusses the PSs based feature computation process and its utilization for land cover (LC) classification. Section 3.4 covers the sub task 1 of the study. Flowchart shown in Figure 3.4 summarizes the process of, computation of PSs, and its critical analysis as potential PolSAR features.

**3.4.1. Polarization signatures computation: Observed and standard**

Due to complex geometrical structure, natural targets (land covers) show complex scattering response [171], [172]. Scattering responses from natural targets are represented as a combination of simple point targets scattering responses which are well defined [173]. In this study, PSs from standard targets namely *Flat Plate (FP)*, *horizontal dipole (HD)*, *vertical dipole (VD)*, and *dihedral (Di)* are considered as standard targets. Geometrical structures, scattering matrices, and PSs of these standard targets are shown in Figure 3.5. The PSs of these standard targets are further utilized in representing PSs of natural targets (observed or image-pixel PSs). Further details about standard targets are provided in [174]. PSs, both observed (image pixel) and standard target are obtained using equation (3.1). The PSs are obtained for ellipticity angle $\chi$ ranging from -45° to +45° and orientation angle $\psi$ ranging from -90° to +90° with a step size of 5° in each direction. Each obtained PS is a 18×37 two dimensional matrix.
<figure><img src="image_25.png" /><figcaption>Figure 3.4 Flowchart showing polarization signature correlation features (PSCFs) computation using fully polarimetric SAR data.</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>Figure 3.5 Geometrical structures, scattering matrices, and co-polarization signatures of different standard targets.</figcaption></figure>
### 3.4.2. Polarization signature correlation features or PSCFs

A land cover (e.g. bare soil) behaves as a complex mixture of different standard scatterers or standard targets and therefore its PS is difficult to interpret. Therefore, at first, there is a need to find correlations between polarimetric/ PSs of land covers and PSs of standard targets. These correlations indicate the degree of similarity between the two scatterers.

Correlation coefficient is a value, which varies from +1 and -1 and it is calculated to represent the linear inter-dependence of two variables or sets, of data. For two variables, say x and y, the correlation coefficient is computed using equation (3.3).

$$CC = \frac{S_{xy}}{S_x S_y} \qquad (3.3)$$

In this section, correlation is computed between each image pixel-standard target PS pair. Here, x is the observed (image pixel) PS, y is the standard target PS, $S_x$ is standard deviation in x, $S_y$ is standard deviation in y, $S_{xy}$ is co-variance between x and y, and CC is correlation coefficient between x and y.

The following pairs are considered between which correlation is computed.

* Co-polarized (pixel-dihedral) or Corr_co_Di

The term *pixel* represents observed co-polarization signature from a pixel in image, and term *dihedral* corresponds to co-polarization signature of dihedral standard target. *Co-polarized (pixel-dihedral)* means correlation coefficient between observed co-polarization signature and dihedral co-polarization signature. Similarly, other pairs are represented as

* Co-polarized (pixel-flat plate) or Corr_co_FP

* Co-polarized (pixel-horizontal dipole) or Corr_co_HD

* Co-polarized (pixel-vertical dipole) or Corr_co_VD

* Cross-polarized (pixel-dihedral) or Corr cross_Di

* Cross-polarized (pixel-flat plate) or Corr cross_FP

* Cross-polarized (pixel-horizontal dipole) or Corr cross_HD

* Cross-polarized (pixel-vertical dipole) or Corr cross_VD

In total, 8 such pairs are obtained in the process. These features are termed as polarization signature correlation features or PSCFs. These PSCFs are potential features for LC classification. PSCF computation process followed in section 3.4.2 is summarized in Figure 3.6 for the sake of clarity
<figure><img src="image_27.png" /><figcaption>Figure 3.6 Figure illustrates the computation of polarization signature correlation features (PSCFs) between observed (image pixel) and standard targets PSs.</figcaption></figure>

A critical analysis is essential at this stage in order to identify the best feature for segregating a particular land cover class. For the same, a separability index based feature selection process is carried out in section 3.4.3.

### 3.4.3. Assessment and critical analysis of PSCFs

In this section, each PSCF obtained in section 3.4.2 are analysed and discussed. Class separation for each PSCF is evaluated by computation of class separability index (SI) proposed in [175]. Nomenclature used in section 3.4.2 is used in this section also.

The separability measure is computed using equation 3.4. For a given feature, SI<sub>i,j</sub> is the measure of separability between class i and j. μ<sub>i</sub> and μ<sub>j</sub> are mean values for class i and j respectively, and σ<sub>i</sub>, σ<sub>j</sub> are standard deviation for class i and j respectively.

$$SI_{i,j} = \frac{|\mu_i - \mu_j|}{|\sigma_i + \sigma_j|} \qquad (3.4)$$
In order to measure SI between different land cover/class pairs, correlation values are collected for all the ROI points (for Roorkee study area) mentioned in section 3.3.3. The SI values obtained for all the PSCFs as features are provided in Table 3.2.

Separability assessment is also conducted for decomposition features obtained using Cloude decomposition [139], modified Yamaguchi four component decomposition or m-Y4D [176], generalized Freeman Durden decomposition or GFD [177], and H-A-Alpha quad pol decomposition or H-A-Alpha [178] methods. SI values for class pairs are shown in Table 3.3.

A comparison between the PSCF features and the considered decomposition features on the basis of SI value for selected class pairs is done. The selected class pairs are, TV to built-up, TV to SV, and BS to Water. These three class pairs are difficult to segregate in a mixed class scenario and therefore are considered for comparison. From Table 3.2 and Table 3.3, it is evident that for TV to built-up class pair, corr_co_Di is the most significant feature (0.7125), followed by C¹(0.439), H³ (0.408), F³ (0.372), and Y¹(0.250). For TV to SV class pair, Y² is the most significant feature (0.502) followed by corr_co_FP (0.4657), C²(0.269), H¹ (0.364), and F² (0.337). For BS to Water class pair, F² is the most significant feature (1.943) followed by C³ (1.499), Y³ (1.392), corr_co_HD (0.8207), and H¹ (0.456). From the comparisons, it can be inferred that there is no clear indication of a set of features (from the same method) that are most significant for all class pairs. In short, there is no clear winner. Also, it is evident that the proposed PSCFs can be potential features to segregate land cover classes in mixed class scenarios.

Land cover classifications are performed using the PSCFs and other considered decomposition features in order to provide a comparative assessment based on LC classification performance. The classifier considered to be used with these features is support vector machines (SVM) with radial basis function (RBF) kernel because of its inherent optimum classification capability. All the classified images are shown in Figure 3.7. Overall classification performance indicators are provided in Table 3.4. It is evident from Table 3.4 that LC classification with PSCFs provides higher classification accuracy and kappa coefficient than the decomposition based features.

It is also evident from Table 3.4 that even if the proposed PSCF features performed better than the compared features, the classification accuracy can be further improved. For the proposed PSCF-SVM classification strategy, built-up (marker A) and TV (marker B) have low user accuracies (69.8% and 68.4% respectively) compared to other classes.

The classification performance can be further improved using more optimal class boundaries. Decision tree (DT) classifier provides the freedom to set optimal class boundary values during classification. Therefore, an optimal class boundary based DT classifier is considered and developed in this study. Details on the development of the optimal class boundary
estimation algorithm are discussed in section 3.5 of this chapter. Also, to reduce the
computational complexity of the proposed classifier, feature selection is carried out with PSCFs.

It is clear from Table 3.2 that built-up can be segregated from all other classes (except tall veg) using corr_co_Di as a feature. Corr_co_Di is showing highest SI value amongst all the PSCFs. Further, it can be observed that (bare soil + water) are clearly separable from other classes (except short vegetation) using corr_co_FP as feature. With SI value of 0.83, bare soil and water can be separated using corr_co_HD. Water is also easily separable from other classes using corr_co_VD. Since both corr_co_HD and corr_co_VD can separate water, either or both of them can be used. It is evident from Table 3.2 that the cross-polarization PSCFs do not contain any complementary information for segregation process and hence are not considered for classification. A water canal (Ganga canal) is visible in Figure 3.1(b). Water in the canal is not identifiable in corr_co_HD or corr_co_VD (refer to Figure 3.8(a) and Figure 3.8(b)). A complementary feature is required which can address the issue. In Figure 3.1(a), the water canal is appearing blue (VV polarimetric band). Hence, VV can be used for classifying water in the canal. It is also evident from Figure 3.1(a), that only VV is not a sufficient choice for the classification of water as “bare soil” is not separable from water using only VV as feature. Hence, corr_co_HD or corr_co_VD, and VV polarimetric band are used. In summary, PSCFs selected for segregating a particular class based on significance are listed in Table 3.5.

Table 3.5 indicates that no PSCFs have been selected for segregating vegetation (both SV and TV) because of low SI values for the class pair over all the PSCFs. Observing Table 3.2, it is clear that the best choice among all the PSCFs for segregating tall and short vegetation is corr_co_FP. In Table 3.2, corr_co_FP is having the highest SI value among all PSCFs for both short and tall vegetation. Table 3.6 provides the updated and final list of selected PSCFs to be used during classification.
<table><caption>Table 3.2 Separability index (SI) values for various class pairs for all features.</caption><thead><tr><th rowspan="2">Class pair</th><th colspan="9">SI value</th></tr><tr><th>Corr_co_Di</th><th>Corr cross_Di</th><th>Corr_co_FP</th><th>Corr cross_FP</th><th>Corr_co_HD</th><th>Corr cross_HD</th><th>Corr_co_VD</th><th>Corr cross_VD</th></tr></thead><tbody><tr><td>TV to BS</td><td>0.9427</td><td>0.0598</td><td>0.9940</td><td>0.3287</td><td>0.7855</td><td>0.4879</td><td>0.2188</td><td>0.4879</td></tr><tr><td>TV to Water</td><td>0.7928</td><td>0.0535</td><td>0.8634</td><td>0.3010</td><td>0.9089</td><td>0.4089</td><td>0.9546</td><td>0.4089</td></tr><tr><td>TV to Built-up</td><td>0.7125</td><td>0.1530</td><td>0.4930</td><td>0.2818</td><td>0.2953</td><td>0.5044</td><td>0.1452</td><td>0.5044</td></tr><tr><td>TV to SV</td><td>0.4333</td><td>0.0560</td><td>0.4657</td><td>0.0743</td><td>0.4249</td><td>0.1780</td><td>0.1244</td><td>0.1780</td></tr><tr><td>BS to Water</td><td>0.0596</td><td>0.0062</td><td>0.0280</td><td>0.0010</td><td>0.8207</td><td>0.0046</td><td>0.7306</td><td>0.3046</td></tr><tr><td>BS to Built-up</td><td>1.2884</td><td>0.2356</td><td>1.3986</td><td>0.529</td><td>0.5803</td><td>0.9359</td><td>0.3855</td><td>0.9359</td></tr><tr><td>BS to SV</td><td>0.4682</td><td>0.0113</td><td>0.4986</td><td>0.2269</td><td>0.3446</td><td>0.3765</td><td>0.0904</td><td>0.3765</td></tr><tr><td>Water to Built-up</td><td>1.1135</td><td>0.2264</td><td>1.2345</td><td>0.5015</td><td>0.9442</td><td>0.8352</td><td>0.9656</td><td>0.8352</td></tr><tr><td>Water to SV</td><td>0.3690</td><td>0.015</td><td>0.4230</td><td>0.2083</td><td>0.7516</td><td>0.2992</td><td>0.6806</td><td>0.2992</td></tr><tr><td>Built-up to SV</td><td>0.7968</td><td>0.2032</td><td>0.9014</td><td>0.3285</td><td>0.1806</td><td>0.6944</td><td>0.2787</td><td>0.6944</td></tr></tbody></table>

<table><caption>Table 3.3 Separability index (SI) values for various class pairs for popular features.</caption><thead><tr><th rowspan="2">Class pair</th><th colspan="15">SI value</th></tr><tr><th>C<sup>1</sup></th><th>C<sup>2</sup></th><th>C<sup>3</sup></th><th>Y<sup>1</sup></th><th>Y<sup>2</sup></th><th>Y<sup>3</sup></th><th>Y<sup>4</sup></th><th>F<sup>1</sup></th><th>F<sup>2</sup></th><th>F<sup>3</sup></th><th>H<sup>1</sup></th><th>H<sup>2</sup></th><th>H<sup>3</sup></th></tr></thead><tbody><tr><td>TV to BS</td><td>0.744</td><td>0.780</td><td>0.500</td><td>0.833</td><td>1.635</td><td>0.313</td><td>0.851</td><td>0.040</td><td>1.733</td><td>0.357</td><td>0.763</td><td>0.460</td><td>0.804</td></tr><tr><td>TV to Water</td><td>1.080</td><td>1.139</td><td>1.828</td><td>2.260</td><td>1.763</td><td>0.706</td><td>0.833</td><td>0.847</td><td>2.625</td><td>1.670</td><td>0.232</td><td>0.173</td><td>0.328</td></tr><tr><td>TV to Built-up</td><td>0.439</td><td>0.001</td><td>0.428</td><td>0.250</td><td>0.009</td><td>0.216</td><td>0.230</td><td>0.209</td><td>0.118</td><td>0.372</td><td>0.241</td><td>0.146</td><td>0.408</td></tr><tr><td>TV to SV</td><td>0.008</td><td>0.269</td><td>0.276</td><td>0.050</td><td>0.502</td><td>0.011</td><td>0.171</td><td>0.013</td><td>0.337</td><td>0.067</td><td>0.364</td><td>0.010</td><td>0.022</td></tr><tr><td>BS to Water</td><td>0.368</td><td>0.480</td><td>1.499</td><td>1.364</td><td>0.543</td><td>1.392</td><td>0.148</td><td>1.350</td><td>1.943</td><td>1.466</td><td>0.456</td><td>0.349</td><td>0.333</td></tr><tr><td>BS to Built-up</td><td>1.174</td><td>0.670</td><td>0.079</td><td>0.905</td><td>1.481</td><td>0.577</td><td>1.002</td><td>0.306</td><td>1.744</td><td>0.649</td><td>0.469</td><td>0.477</td><td>1.056</td></tr><tr><td>BS to SV</td><td>0.688</td><td>0.533</td><td>0.090</td><td>0.831</td><td>1.244</td><td>0.319</td><td>0.704</td><td>0.069</td><td>1.481</td><td>0.364</td><td>0.396</td><td>0.347</td><td>0.716</td></tr><tr><td>Water to Built-up</td><td>1.496</td><td>0.993</td><td>1.105</td><td>1.925</td><td>2.351</td><td>0.422</td><td>0.970</td><td>0.563</td><td>2.562</td><td>1.647</td><td>0.005</td><td>0.181</td><td>0.603</td></tr><tr><td>Water to SV</td><td>0.939</td><td>0.882</td><td>1.202</td><td>1.928</td><td>0.801</td><td>0.731</td><td>0.663</td><td>0.953</td><td>2.408</td><td>1.445</td><td>0.108</td><td>0.018</td><td>0.273</td></tr><tr><td>Built-up to SV</td><td>0.393</td><td>0.235</td><td>0.117</td><td>0.173</td><td>0.453</td><td>0.227</td><td>0.377</td><td>0.214</td><td>0.538</td><td>0.245</td><td>0.105</td><td>0.152</td><td>0.361</td></tr></tbody></table>

C¹: Cloude-Pd (double bounce), C²: Cloude-Pv (volume), C³: Cloude-Ps (surface), (Cloude Decomposition)

Y¹: M-Y4D-Pd (double bounce), Y²: M-Y4D-Pv (volume), Y³: M-Y4D-Ps (surface), Y⁴: M-Y4D-Pc (helix), (Modified Yamaguchi Decomposition)

F¹: GFD-Pd (double bounce), F²: GFD-Pv (volume), F³: GFD-Ps (surface), (Generalized Freeman Durden Decomposition)

H¹: H-A-Alpha-Entropy, H²: H-A-Alpha-Anisotropy, H³: H-A-Alpha-Alpha, (H-A-Alpha Decomposition)
<figure><img src="image_28.png" /><figcaption>Figure 3.7: Classified Image; (a) M-Y4D-SVM, (b) GFD-SVM, (c) Cloude-SVM, (d) H-A-Alpha-SVM, (e) PSCF-SVM (proposed), and (f) Google Earth image for reference purpose.
Colour scheme; red = built-up, light green = agriculture or SV, green = TV., blue = water, and yellow = bare soil. Marker A indicates built-up area and marker B indicates tall vegetation area.</figcaption></figure>

<table><caption>Table 3.4 Classification performance of PS based and decomposition based PolSAR features.
Classes: built-up (BU), water (W), bare soil (BS), short vegetation (SV), and tall vegetation (TV).</caption><thead><tr><th>Method</th><th>Overall Accuracy (%)</th><th>Kappa</th><th>Class</th><th>User Accuracy (%)</th><th>Producer Accuracy (%)</th></tr></thead><tbody><tr><td rowspan="5">H-A-Alpha-SVM</td><td rowspan="5">53</td><td rowspan="5">0.40</td><td>BS</td><td>63.2</td><td>82.9</td></tr><tr><td>TV</td><td>44.6</td><td>88.9</td></tr><tr><td>SV</td><td>58.8</td><td>53.3</td></tr><tr><td>BU</td><td>14.3</td><td>1.3</td></tr><tr><td>Water</td><td>47.7</td><td>12.8</td></tr><tr><td rowspan="5">M-Y4D-SVM</td><td rowspan="5">72.4</td><td rowspan="5">0.65</td><td>BS</td><td>66.1</td><td>82.3</td></tr><tr><td>TV</td><td>70.9</td><td>88.0</td></tr><tr><td>SV</td><td>76.9</td><td>77.5</td></tr><tr><td>BU</td><td>61.5</td><td>50.3</td></tr><tr><td>Water</td><td>88.9</td><td>54.8</td></tr><tr><td rowspan="5">GFD-SVM</td><td rowspan="5">71.9</td><td rowspan="5">0.63</td><td>BS</td><td>63.4</td><td>79.9</td></tr><tr><td>TV</td><td>67.5</td><td>76.5</td></tr><tr><td>SV</td><td>76.5</td><td>74.5</td></tr><tr><td>BU</td><td>65.0</td><td>72.3</td></tr><tr><td>Water</td><td>87.4</td><td>60.7</td></tr><tr><td rowspan="5">Cloude-SVM</td><td rowspan="5">63.7</td><td rowspan="5">0.53</td><td>BS</td><td>54.1</td><td>77.3</td></tr><tr><td>TV</td><td>68.2</td><td>66.2</td></tr><tr><td>SV</td><td>67.9</td><td>59.5</td></tr><tr><td>BU</td><td>52.5</td><td>58.5</td></tr><tr><td>Water</td><td>85.9</td><td>57.0</td></tr><tr><td rowspan="5">PSCF-SVM</td><td rowspan="5">74</td><td rowspan="5">0.69</td><td>BS</td><td>69.5</td><td>79.9</td></tr><tr><td>TV</td><td>69.8</td><td>89.1</td></tr><tr><td>SV</td><td>79</td><td>65.0</td></tr><tr><td>BU</td><td>68.4</td><td>58.5</td></tr><tr><td>Water</td><td>83.6</td><td>70.1</td></tr></tbody></table>
<figure><img src="image_29.png" /><figcaption>Figure 3.8 Correlation images (a) corr_co_HD and (b) corr_co_VD.</figcaption></figure>

<table><caption>Table 3.5 List of selected feature corresponding to land cover classes.</caption><thead><tr><td>Class</td><td>Feature selected for segregation</td></tr></thead><tbody><tr><td>Built-up</td><td>Corr_co_Di</td></tr><tr><td>Water</td><td>Corr_co_HD, corr_co_VD</td></tr><tr><td>Canal</td><td>VV polarimetric band</td></tr><tr><td>Bare soil</td><td>Corr_co_FP</td></tr></tbody></table>

An alternate analysis of all the PSCFs as potential features for decision tree based land cover classification is presented in [179]. Scatter plot and OTSU multi-thresholding methods are used for separability assessment and similar results are reported. The class boundary values obtained using methods in [179] are rigid and do not change with the change in the PSCF statistics which in turn depends on the input PolSAR image. The changes captured in a PolSAR image can be via changes in landscape seasons or man-made. These changes needs to be considered while calculating class boundary thresholds. Therefore, in this study, a input image statistics based adaptive class boundary estimation approach is proposed to improve the results from [179]. Development of proposed approach is discussed in section 3.5.

## 3.5. Sub Task 2: Optimal Class Boundary Estimator based Decision Tree Approach for LC Classification

The approach is developed and presented in a sequential manner. Section 3.5 is broken down to four sub-sections and each sub-section is discussed in detail. Details of dataset are provided in section 3.3.2. Flowchart in Figure 3.9 summarizes the proposed algorithm.

### 3.5.1. PSCF statistics

Making the algorithm adaptive means the algorithm estimates class boundaries based on PSCF statistics (mean, standard deviation, etc.). Statistics for selected PSCFs i.e. mean (M) and standard deviation (S) are computed and are shown in Table 3.7. These statistics are used for the computation of class boundaries.
<table><caption>Table 3.6 Updated list of selected feature corresponding to land cover classes.</caption><thead><tr><td>Class</td><td>Feature to be used for segregation</td></tr></thead><tbody><tr><td>Built-up</td><td>Corr_co_Di</td></tr><tr><td>Water</td><td>Corr_co_HD, Corr_co_VD</td></tr><tr><td>Water Canal</td><td>VV polarimetric band</td></tr><tr><td>Bare soil</td><td>Corr_co_FP</td></tr><tr><td>Tall vegetation</td><td>Corr_co_FP</td></tr><tr><td>Short vegetation</td><td>Corr_co_FP</td></tr></tbody></table>

<table><caption>Table 3.7 Polarization signature correlation feature statistics.</caption><thead><tr><td>Feature</td><td>Mean (M)</td><td>Std. dev. (S)</td></tr></thead><tbody><tr><td>Corr_co_FP</td><td>0.5269</td><td>0.1266</td></tr><tr><td>Corr_co_Di</td><td>0.357</td><td>0.0716</td></tr><tr><td>Corr_co_HD</td><td>0.0580</td><td>0.1850</td></tr><tr><td>Corr_co_VD</td><td>-0.0580</td><td>0.1850</td></tr><tr><td>VV</td><td>-11.243</td><td>1.564</td></tr></tbody></table>

### 3.5.2. Relationship between class boundaries and corresponding PSCF statistics

A relationship is formulated between class boundaries and corresponding PSCF statistics (the one selected for segregating that particular class (refer Table 3.6)). The relationship is formulated and explained with an example. Let's take the case of “built-up” class. Built-up is strongly segregated by “corr_co_Di” feature, $M_{corr_co_Di}$ and $S_{corr_co_Di}$ are mean and standard deviation of corr_co_Di. Therefore, built-up class spread can be expressed as;

$$M_{corr\_co\_Di} + n_1 * S_{corr\_co\_Di} \le corr\_co\_Di \le M_{corr\_co\_Di} + n_2 * S_{corr\_co\_Di} \quad (3.5)$$

Where, $n_1$ and $n_2$ are real and scalar. $M_{corr_co_Di} + n_1 * S_{corr_co_Di}$ is the class lower boundary value and $M_{corr_co_Di} + n_2 * S_{corr_co_Di}$ is class upper boundary value. This expression means, if corr_co_Di pixel value lies in between these boundary values then the pixel belongs to built-up class. Similar expressions are developed for other classes (refer equation (3.6) through (3.9)).

* For Bare Soil

$$M_{corr\_co\_FP} + n_3 * S_{corr\_co\_FP} \le corr\_co\_FP \le M_{corr\_co\_FP} + n_4 * S_{corr\_co\_FP} \quad (3.6)$$

* For Water

$$M_{corr\_co\_HD} + n_5 * S_{corr\_co\_HD} \le corr\_co\_HD \le M_{corr\_co\_HD} + n_6 * S_{corr\_co\_HD} \quad (3.7)$$

* For Short vegetation

$$M_{corr\_co\_FP} + n_7 * S_{corr\_co\_FP} \le corr\_co\_FP \le M_{corr\_co\_FP} + n_8 * S_{corr\_co\_FP} \quad (3.8)$$
* For Tall vegetation

$$M_{corr\_co\_FP} + n_9 * S_{corr\_co\_FP} \le corr\_co\_FP \le M_{corr\_co\_FP} + n_{10} * S_{corr\_co\_FP} \quad (3.9)$$

Experiments suggest that class boundary thresholds for canal are not affected by VV band statistics and hence are not included for further study.

The goal of representing the class boundaries as in equation (3.5) through (3.9) is to obtain class boundaries that achieve best classification results under a given situation. Overall accuracy (OA) is an indicator of classification performance. Higher the OA, better the classification result. It is straightforward to understand that more accurate class boundaries result in higher OA. An empirical relationship is intuitive and can be developed between OA and class boundaries. The empirical relationship can then be treated as an optimization problem which can be addressed by an optimization technique. Since, M and S are constant for a given PSCF, therefore class boundary varies with only with $n_i$ (i = 1 to 10). An empirical relationship between OA and $n_i$ (i = 1 to 10) is relevant. Section 3.5.3 discusses the development of an empirical relationship between OA and $n_i$ (i = 1 to 10).

### 3.5.3. Relating classification accuracy and class boundaries: Empirical modelling

In order to achieve the best overall accuracy (OA) with given feature space, the values of $n_i$ (i = 1 to 10) should be optimized. To get optimal $n_i$ (i = 1 to 10) values, an empirical function needs to be developed relating OA to $n_i$ (i = 1 to 10) which can be optimized. OA is defined in equation (3.10).

$$OA = \frac{\text{total no of correctly classified pixels}}{\text{Total no of pixels tested}} \quad (3.10)$$

PSCFs are used based on Table 3.6 for segregating corresponding classes. Hence, for each class, individual overall accuracy (IOA) is computed and OA is rewritten as equation (3.11).

$$OA = \frac{\text{IOA}_{\text{built-up}} + \text{IOA}_{\text{bare soil}} + \text{IOA}_{\text{water}} + \text{IOA}_{\text{SV}} + \text{IOA}_{\text{TV}}}{5} \quad (3.11)$$

Where, for example $IOA_{tall\_veg}$ is individual overall accuracy (IOA) in TV vs other class classification scenario using $corr\_co\_FP$ as feature. Further, these IOAs can be empirically related to $n_i$ (i = 1 to 10). Empirical relations developed between IOAs and $n_i$ (i = 1 to 10) are
provided in Table 3.8. These relations are developed using surface fitting methods. For example, $IOA_{tall_veg}$ is related to $n_9$ and $n_{10}$ by a second degree polynomial surface.

$$ IOA_{TV} = -573.7 - 998.9 * n_9 + 1742 * n_{10} - 1491 * n_9^2 - 27.5 * n_9 * n_{10} - 1568 * n_{10}^2 \quad (3.12) $$

Similar procedure is applied for all other classes and corresponding empirical relations are obtained. A coefficient of determination (R²) value is also provided in Table 3.8. An optimization method can now be used to provide optimal $n_i$ (i = 1 to 10) values that can maximize OA under constraints.

### 3.5.4. Optimizing the empirical model

Genetic algorithm (GA) [180] is used as an optimization tool for finding optimal $n_i$ (i = 1 to 10) values. GA is a metaheuristic inspired by the process of natural selection that belongs to the larger class of evolutionary algorithms [181]. Genetic algorithms are commonly used to generate high-quality solutions to optimization and search problems by relying on biologically inspired operators such as mutation, crossover and selection. They imitate the biological processes of reproduction and natural selection to solve for the 'fittest' solutions. Like evolution, many of a genetic algorithm's processes are random, however this optimization technique allows one to set the level of randomization and the level of control. These algorithms are far more powerful and efficient than random search and exhaustive search algorithms yet require no extra information about the given problem.

<table><caption>Table 3.8 Empirical relations between individual overall accuracies (IOA) and $n_i$ values.</caption><thead><tr><th>Performance parameters</th><th>Derived empirical relation</th><th>R<sup>2</sup> (coefficient of determination)</th></tr></thead><tbody><tr><td>IOA<sub>built-up</sub></td><td>80.41 * sin(5.3 * n<sub>1</sub> - 2.9) + 5.1 * sin(33.5 * n<sub>1</sub> + 6.4)</td><td>0.99</td></tr><tr><td>IOA<sub>bare_soil</sub></td><td>79.44 * sin(5.5 * n<sub>3</sub> - 2.9) + 1.1 * sin(69.2 * n<sub>3</sub> - 21.6)</td><td>0.96</td></tr><tr><td>IOA<sub>water</sub></td><td>2000 * sin(4.7 * n<sub>5</sub> + 1.9) + 1993 * sin(4.7 * n<sub>5</sub> - 1.3) + 1.23 * sin(51.2 * n<sub>5</sub> + 2.2)</td><td>0.92</td></tr><tr><td>IOA<sub>SV</sub></td><td>-513.4 - 902.6 * n<sub>7</sub> + 2210 * n<sub>8</sub> + 93.5 * n<sub>7</sub><sup>2</sup> + 1250 * n<sub>7</sub> * n<sub>8</sub> - 1984 * n<sub>8</sub><sup>2</sup></td><td>0.91</td></tr><tr><td>IOA<sub>TV</sub></td><td>-573.7 - 998.9 * n<sub>9</sub> + 1742 * n<sub>10</sub> - 1491 * n<sub>9</sub><sup>2</sup> - 27.5 * n<sub>9</sub> * n<sub>10</sub> - 1568 * n<sub>10</sub><sup>2</sup></td><td>0.99</td></tr></tbody></table>
The choice of GA as optimization method is because of the probability of more than one maxima/minima. Conventional methods like *first derivative* could be faster but can suffer from local maxima/minima problem.

The objective function of the optimization problem at hand is (refer equation (3.13);

$$ \text{maximize} \quad (f(x)) = \text{maximize}(OA) \qquad (3.13) $$

Subjected to

$$ OA_{\text{initial}} \geq 50\% \text{ (based on experiment)} \qquad (3.13a) $$

$$ -2.0 \leq n_1 \leq 2.0, n_2 = 2.0 \qquad (3.13b) $$

$$ -0.5 \leq n_3 \leq 1.0, n_4 = 1.0 \qquad (3.13c) $$

$$ n_5 = -2.0, -2.0 \leq n_6 \leq 1.0 \qquad (3.13d) $$

$$ -1.5 \leq n_7 \leq -1.0, -0.5 \leq n_8 \leq 0 \qquad (3.13e) $$

$$ -2.0 \leq n_9 \leq -0.5, -1.0 \leq n_{10} \leq -1.5 \qquad (3.13f) $$

The upper and lower bounds of $n_i$ ($i = 1$ to 10) values are obtained via experimental study during developing the empirical relation formulation as discussed in section 3.5.3 and are provided in equation 3.13(a-f). The internal parameters to GA are finalized using *trial and error* method.

After optimizing equation 3.13, the optimal $n_i$ ($i = 1$ to 10) values obtained are shown in Table 3.9. These optimal $n_i$ ($i = 1$ to 10) values are used to obtain class boundaries which in turn will be used in decision tree classifier in section 3.5.5.

### 3.5.5. Optimal class boundaries based decision tree classifier

The optimal $n_i$ ($i = 1$ to 10) values obtained in section 3.5.4 are used to compute optimal class boundary values (using equation (3.5) through (3.9)) and henceforth, these class boundary values are used to develop a decision tree classification scheme. The decision tree based on the optimal class boundary values, in turn based on optimal $n_i$ ($i = 1$ to 10) values is illustrated in Figure 3.10. The priority of a class to be segregated first is evaluated on how high is its SI value.

<table><caption>Table 3.9 Optimal $n_i$ values obtained using the genetic algorithm.</caption><thead><tr><th>n<sub>1</sub></th><th>n<sub>2</sub></th><th>n<sub>3</sub></th><th>n<sub>4</sub></th><th>n<sub>5</sub></th><th>n<sub>6</sub></th><th>n<sub>7</sub></th><th>n<sub>8</sub></th><th>n<sub>9</sub></th><th>n<sub>10</sub></th></tr></thead><tbody><tr><td>0.315</td><td>2.0</td><td>0.446</td><td>1.0</td><td>-2.0</td><td>-1.667</td><td>-1.4</td><td>-0.2</td><td>-0.022</td><td>0.998</td></tr></tbody></table>
<figure><img src="image_30.png" /><figcaption>Figure 3.9 Flowchart illustrates the development process of the optimal class boundary estimator based decision tree classifier.</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 3.10 Decision tree based on optimal class boundaries values.</figcaption></figure>
### 3.5.6. Results and discussion

The optimal class boundary estimator based decision tree classifier obtained in section 3.5.5 (refer Figure 3.10) is utilized to perform LC classification on Roorkee dataset. The classified image is shown in Figure 3.11(a). The proposed optimal class boundary estimator based DT classifier is compared with the RBF-SVM classifiers. The same PSCFs are used as features with the RBF-SVM. Classified image the RBF-SVM is shown in Figure 3.11(b) respectively. Overall classification performance indicators for the proposed method and RBF-SVM are provided in Table 3.10. From Table 3.10, it is evident that the proposed method achieves best classification accuracy (75.3%) with kappa coefficient (0.69). Also, the user accuracy for built-up and TV have improved to 73.8% and 72.4% respectively (Marker A, and B respectively). Markers A and B in Figure 3.1 are clearly separated using the proposed optimal class boundary approach (refer figure 3.11(a)).

However, computing PSCFs from PSs for further utilization with classifiers seems an extra step. It is motivating to know that two dimensional CNNs (2D-CNNs) are powerful image classifiers which can extract complex features directly from data. Since, the PSs are two dimensional representation of radar backscatter response of a land cover (refer section 3.2.1), they can be used as input images for pixel wise LC classification. The idea of utilizing CNNs with PSs for LC classification is proposed in section 3.6.

<table><caption>Table 3.10: Classification performance indicators for the proposed method and RBF-SVM classifier with PSCFs as features.</caption><thead><tr><th>Method</th><th>OA (%)</th><th>Kappa</th><th>Class</th><th>User Accuracy (%)</th><th>Producer Accuracy (%)</th></tr></thead><tbody><tr><td rowspan="5">Proposed optimal class boundary estimator based DT classifier</td><td rowspan="5">75.3</td><td rowspan="5">0.69</td><td>BS</td><td>69.5</td><td>79.9</td></tr><tr><td>TV</td><td>73.8</td><td>89.1</td></tr><tr><td>SV</td><td>78.9</td><td>65.0</td></tr><tr><td>BU</td><td>72.4</td><td>71.0</td></tr><tr><td>Water</td><td>83.6</td><td>70.0</td></tr><tr><td rowspan="5">RBF-SVM</td><td rowspan="5">74</td><td rowspan="5">0.69</td><td>BS</td><td>69.5</td><td>79.9</td></tr><tr><td>TV</td><td>69.8</td><td>89.1</td></tr><tr><td>SV</td><td>79</td><td>65.0</td></tr><tr><td>BU</td><td>68.4</td><td>58.5</td></tr><tr><td>Water</td><td>83.6</td><td>70.1</td></tr></tbody></table>
<figure><img src="image_32.png" /><figcaption>Figure 3.11: Classified image: (a) Optimal class boundary estimator based DT classifier, (b) RBF-SVM, and (c) Google Earth image for reference purpose. Colour scheme: red = built-up, light green = SV, green = TV, blue = water, and yellow = BS. Marker A indicates built-up area and marker B indicates tall vegetation area.</figcaption></figure>

**3.6. Sub Task 3: Direct Utilization of PolSAR Polarization Signatures via Convolutional Neural Networks for LC Classification**

The current section presents utilization and analysis of advanced pattern classification methods for PolSAR land cover classification. Convolutional neural networks (CNNs) are recently the most popular, efficient, and powerful techniques in image feature extraction and classification applications. Their parameter sharing and local connectivity properties makes them best tools for extracting information from images [35]. CNNs in combination with deep learning technology are being widely used in PolSAR data applications [58]. In this study, CNNs are utilized to extract features from PSs and perform land cover classification. A CNN based PSs pattern classification technique is proposed here which considers the influence of the neighbourhood of a pixel and thereby utilizing the PSs directly.

**3.6.1. Two dimensional convolution over polarization signatures: The concept**

Figure 3.12 demonstrates the utilization of two dimensional convolutions on PolSAR PSs. Figure 3.12(a) and Figure 3.12(b) depicts isometric view and top view respectively of the PS of a PolSAR image pixel. A 2D convolution filter as shown in Figure 3.12(d) convolves over the PS in a sliding window manner and provides convolutional features. These features can be used in classification, regression or recognition.
<figure><img src="image_33.png" /><figcaption>Figure 3.12: Two dimensional convolution over polarization signature. (a) Polarization Signature of a PolSAR image pixel. (b) Top view. (c) Polarization signature subset. (d) Convolutional filter.</figcaption></figure>

### 3.6.2. PSs-CNN model architecture design and training

A CNN model is constructed and configured with the help of convolutional and fully connected layers to perform land cover classification with PSs. The proposed CNN model is termed as PSs-CNN model. The proposed PSs-CNN model configuration is provided in Table 3.11. The input to the PSs-CNN model is a pixel PSs. Pixel (observed) PSs used here are obtained as discussed in section 3.4.1. Each PS corresponds to a 19×37 image. The 19×37 input image has two channels i.e. *copol* and *crosspol*. Therefore, overall the size of input to the PSs-CNN is 19×37×2. At first, the PSs-CNN used four convolutional layers. Number of convolution filters and filter sizes used in each layer are listed Table 3.11. ReLU activation function and dropout fraction of 10% is used in each convolutional layer. The dropout regularization strategy is employed to achieve less overfitting and better generalization. Further, two fully connected or FC layers, first with 50 nodes and second with 10 nodes are considered. Both FC layers used the ReLU activation function and a dropout fraction of 10%. Finally, a softmax classification layer with 5 labels (based on 5 output land cover classes) is used. A batch normalization process is carried out before the convolutions to speed up the training process and to reduce the network sensitivity. A seven layered PSs-CNN classification model is realized as shown in Figure 3.13.

The proposed PSs-CNN model is trained on labelled samples provided in Table 3.1. The loss function considered during training is the categorical cross entropy function. The learning rate is set to 0.01 and the number of training epochs is set to 200. Adam optimizer is used for network weight optimization.
<table><caption>Table 3.11 Summary of PSs-CNN model configuration, hyperparameters and training</caption><thead><tr><th>Layer</th><th>Input</th><th>Convolutional layers; C1, C2, C3, C4,C5</th><th>Fully connected layers, FC1, FC2</th><th>Output layer</th></tr></thead><tbody><tr><td>PSs- CNN<br>Architecture</td><td>Input image size =<br>19×37×2<br>Normalization =<br>Batch Normalization</td><td>Filters = 10, 10, 10, and 20<br>Kernel size = 3×5, 3×5, 3×5, and 9×21<br>Activation = ReLU,<br>ReLU, ReLU, and ReLU<br>Dropout fraction = 0.1, 0.1, and 0.1<br>Pooling size = 0, 0, 0, 0, 0,</td><td>Nodes =50,10<br>Activation =<br>ReLU, ReLU<br>Dropout fraction = 0.1, 0.1</td><td>Labels = 5<br>Activation =<br>Softmax</td></tr><tr><td>PSs- CNN<br>Training<br>parameters</td><td>Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam</td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_34.png" /><figcaption>Figure 3.13 Architecture of the proposed PSs-CNN model.</figcaption></figure>

### 3.6.3. Results and discussion

Figure 3.14(a) is the classified image for using the PSs-CNN model as classifier. To assess the classification performance of proposed PSs-CNN model, the PSs-CNN model performance is compared with two popular PSs based land cover classifiers i.e. the normalized signature correlation mapper (NSCM) based minimum distance classifier and the normalized Euclidean distance (NED) based minimum distance classifier. Figure 3.14(b) is classified image obtained using the NSCM classifier whereas Figure 3.14(c) is the classified image obtained using the NED classifier. It is evident from Figure 3.14 that PSs-CNN outperforms the NSCM and NED based classifiers. Table 3.12 summarizes the quantitative assessment of classification performance of PSs-CNN, NED, and NSCM classifiers. The PSs-CNN classifier performs land cover classification with good overall accuracies, i.e. 72.8% with kappa coefficient of 0.69. In contrast,
the NED performed the classification with 62.3 % and the NSCM performed the classification with 64.1%. Both NED and NSCM showed low kappa values (refer Table 3.12). Because of the point-wise operability, the NED and NSCM classifiers are not able to capture and exploit the neighbourhood relationship of a pixel and thereby are not able to efficiently address the classical classification challenges with PolSAR data such as, Built-up vs Tall veg, where volume scattering is dominant [182]. The proposed PSs-CNN classifier resolves the issue significantly by extracting and utilizing the subtle differences in the PSs of built-up and forest. Markers A and B (refer Figure 3.1(a)) are clearly separated using the proposed approach (refer Figure 3.14(a)).

<table><caption>Table 3.12 Classification performance indicators for the PSs-CNN, NSCM, and NED classifiers with PSs as features. BS = bare soil, TV = Tall vegetation, SV = Short vegetation, BU = Built-up.</caption><thead><tr><th>Method</th><th>OA (%)</th><th>Kappa</th><th>Class</th><th>User Accuracy (%)</th><th>Producer Accuracy (%)</th></tr></thead><tbody><tr><td rowspan="5">PSs-CNN</td><td rowspan="5">72.8</td><td rowspan="5">0.69</td><td>BS</td><td>88.5</td><td>79.9</td></tr><tr><td>TV</td><td>73.8</td><td>89.1</td></tr><tr><td>SV</td><td>62.9</td><td>65.0</td></tr><tr><td>BU</td><td>70</td><td>58.5</td></tr><tr><td>Water</td><td>79.6</td><td>70.1</td></tr><tr><td rowspan="5">NSCM</td><td rowspan="5">64.1</td><td rowspan="5">0.54</td><td>BS</td><td>100.0</td><td>61.5</td></tr><tr><td>TV</td><td>100.0</td><td>44.5</td></tr><tr><td>SV</td><td>65.6</td><td>82.2</td></tr><tr><td>BU</td><td>38.5</td><td>57.6</td></tr><tr><td>Water</td><td>56.2</td><td>73.2</td></tr><tr><td rowspan="5">NED</td><td rowspan="5">62.3</td><td rowspan="5">0.51</td><td>BS</td><td>88.4</td><td>61.1</td></tr><tr><td>TV</td><td>100.0</td><td>44.0</td></tr><tr><td>SV</td><td>65.5</td><td>82.2</td></tr><tr><td>BU</td><td>33.8</td><td>47.0</td></tr><tr><td>Water</td><td>56.2</td><td>73.2</td></tr></tbody></table>

<figure><img src="image_35.png" /><figcaption>Figure 3.14 Classified Image: (a) PSs-CNN, (b) NSCM, (c) NED, and (d) Google earth image. Colour scheme; red = built-up, light green = SV, green = TV, blue= water, and yellow = BS. Marker A indicates built-up area and marker B indicates tall vegetation area.</figcaption></figure>
## 3.7. Overall Results and Discussion

Comparison between the two novel approaches developed in this study i.e. the optimal class boundary estimator based DT classifier and the PSs-CNN is made. Classification results from the two novel classifiers and the RBF-SVM are summarized in Table 3.13. The comparison indicated that the optimal class boundary estimator based DT classifier is showing an approx. 3% improved classification accuracy than PSs-CNN classifier. However, overall comparison suggest that PSs-CNN is easier to implement, does not require an extra step (computing PSCFs), and is scalable.

<table><caption>Table 3.13 Classification performance indicators for the optimal class boundary estimator based DT, PSs-CNN, and RBF-SVM classifier.</caption><thead><tr><td>Method</td><td>OA (%)</td><td>Kappa</td></tr></thead><tbody><tr><td>PSs-CNN with polarization signatures</td><td>72.8</td><td>0.69</td></tr><tr><td>Optimal class boundary estimator-DT with PSCFs</td><td>75.3</td><td>0.71</td></tr><tr><td>RBF-SVM with PSCFs</td><td>74</td><td>0.70</td></tr></tbody></table>

## 3.8. Conclusions

Overall a study where an attempt is made to extract information from one of the most popular satellite image modality i.e. PolSAR images is presented in this chapter. The study is composed of three sub-tasks.

Under the first sub task i.e. *sub task 1*, the investigation of polarization signatures (PSs) as potential PolSAR features for land cover (LC) classification in mixed class scenarios is done. Novel PSs based features i.e. PSCFs are computed based on the degree of similarity between observed and standard PSs. Further, PSCFs are analysed, and are compared with decomposition based features for the purpose of LC classification. Comparisons indicate potential of PSCFs as significant features for LC classification in mixed land cover class scenarios.

Second sub task i.e. *sub task 2*, covers successful development and implementation of an optimal class boundary estimator based DT classifier which exploits the PSCFs to achieve improved LC classification. During the development of the optimal class boundary estimator based DT classifier, image and individual class-wise PSCFs statistics are utilized to formulate empirical relationship between classification performance indicator (overall accuracy) and class boundary values. Optimization of this empirical relationship with genetic algorithm resulted in optimal class boundaries which are utilized in a DT to perform LC classification. Classification results using this approach are compared with classification results from RBF-SVM, while keeping the features identical i.e. PSCFs. Classification performance of the proposed approach
especially user and producer accuracies for built-up class (72.4% and 71.0% respectively) and tall veg class (73.8%, and 89.1% respectively) indicate that the proposed approach, i.e. the optimal class boundary estimator based DT classifier, achieved better classification results in mixed LC class scenarios.

In the third sub task i.e. *sub task 3*, development of a two dimensional CNN (2D-CNN) based novel model for LC classification is presented. The 2D-CNNs can directly exploit PSs to perform LC classification, and therefore, the approach reduces the efforts to compute PSCFs. Also, since 2D-CNNs are powerful image extractors, they are able to extract the subtle contextual patterns present in PSs of different LCs. In summary, a novel PSs-CNN LC classification model is realized. Classification results from this PSs-CNN model are compared with NED and NSCM based minimum distance classifiers which are popularly used with PSs for land cover classification. Comparison indicated a superior performance of the novel PSs-CNN model for land cover classification.

Comparison between the two novel approaches proposed in this study i.e. the optimal class boundary estimator based DT classifier and the PSs-CNN model based classifier is made. The comparison indicated that the optimal class boundary estimator based DT classifier is showing an approx. 3% improved classification accuracy than PSs-CNN classifier. However, the PSs-CNN model is easier to implement, does not require an extra step (computing PSCFs), and is scalable.

Therefore, the study presented in this chapter concludes that; i) PSs have shown significant potential as PolSAR features for land cover applications, ii) segregation of built-up class and tall vegetation class in PolSAR data can be improved by using optimal class boundary estimators with decision trees for classification, and iii) 2D-CNNs have shown potential as classifiers with PSs for LC classification in terms of ease of implementation, and scalability. However, the potential of PSs-CNN model needs to be further investigated for other important PolSAR data based earth observation applications.
# NOVEL APPROACH FOR DETECTION OF HUMAN SETTLEMENTS AFFECTED DURING FLOODS USING POLARIZATION SIGNATURES AND CNNS

During floods, an updated and accurate information on flood affected human settlements helps mitigate the disaster efficiently. This chapter presents an efficient approach to map human settlements, especially villages in the rural parts of India that are affected during flood using PolSAR imagery, convolutional neural networks (CNNs), and image morphology. This chapter is an extension of chapter 3 in a sense to utilize the PSs-CNN model (proposed in chapter 3) for a crucial, and practical application.

The approach involves three stages: -first, built-up area extraction from a PolSAR image, -second, detection of villages in a built-up area image, and -third, identification and mapping villages that are affected by flood. In the first stage, a CNN based novel classification model is proposed to extract built-up areas from PolSAR image. This model utilizes PolSAR polarization signatures (PSs) as features. The second stage proposes a novel village detection methodology. This methodology detects whether a particular object in the built-up area image (obtained in the first stage) is a village or not. In the third stage, flood affected villages are identified and mapped and non-flooded villages are masked out using another image morphology based methodology.

## 4.1. Introduction

Floods are the most frequent, widespread, and impactful disasters in India. A major population of India lives in towns and villages and during the floods, rescue operations to these affected areas are both difficult and challenging. Rapid and reliable management and mitigation is key to minimize the loss of human lives and property during a flood disaster. Accurate and updated information of flood affected area can lead to an efficient disaster mitigation plan. In the context, satellite images based flood disaster monitoring is one of the most helpful and reliable technology to counter these flood disasters. Especially, synthetic aperture radar (SAR) imaging has proved to be a very successful and promising technology for both, monitoring floods or, planning rescue operations in flood affected areas [16], [183]–[186]. Polarimetric synthetic aperture radar (PolSAR) imaging sensors are developed to capture the changes in polarization states introduced by the interaction of EM waves with earth surface features (land covers). Among many other
applications, PolSAR imaging has also proved its significance and uniqueness in flood
monitoring and mitigation [183], [187]–[189]. PolSAR data is usually captured in four *linear*
polarization states namely horizontal transmit – horizontal receive (HH), horizontal transmit –
vertical receive (HV), vertical transmit – horizontal receive (VH), and vertical transmit – vertical
receive (VV). In fact, the HH, HV, VH, and VV polarization states indicate to the double bounce,
multiple bounce, and single bounce scattering behaviours of a target respectively. The scattering
responses in different polarization states help the identification and classification of different
land covers which is helpful during flooded area assessment. For example, built-up structures
usually show double bounce scattering response. This means that these built-up structures appear
bright (high scattering response) in HH PolSAR image. On the other hand, tall vegetation areas
(e.g. forests) show multiple bounce scattering (volume scattering) response, or smooth surfaces
such as water show single bounce scattering response [190]. However, the PolSAR data has been
transformed, represented, and utilized in different other forms and bases in various flood
monitoring studies [191], [192]. Each PolSAR data representation has its advantages and
disadvantages and its selection depends on application and available processing methodologies.

Monitoring human settlements in the rural parts of India during floods is difficult even
with PolSAR data. The built-up structures (e.g. houses) in these human settlements, especially
villages of rural India are closely packed and randomly oriented. Therefore, these villages show
volume scattering response along with double bounce scattering response in a PolSAR image.
The mixed scattering response from these villages hinders their segregation from tall vegetation
in a PolSAR image. This challenge is also discussed in chapter 3 (refer section 3.1). Therefore,
the detection and mapping of these villages during a flood using PolSAR image is challenging
as there is high risk of false alarms, or missing out a flood affected village during rescue
operations. The current study considers PolSAR polarization signatures (PSs) as features to
improve the detection and mapping of villages that are affected by flood. Polarization signature
(PS) is a two dimensional representation relating backscatter response from a target at a particular
polarization state. Polarization signatures or PSs are introduced in [155] and later on represented
graphically by many researchers in different ways [154]. PSs have been proved to be useful for
characterization of land covers such as built-up, forest, or water [149]. A comparison of PSs
based features and decomposition based features for land cover characterization is also presented
in section 3.4. The study demonstrates the significance of the utilization of PSs for land cover
characterization and classification in mixed land cover scenarios which are similar to the scenario
at hand in the current study. Therefore, the utilization of PSs as PolSAR features is considered
in the current study. A detailed description on PSs is provided in section 3.2.1.
Moreover, it is also evident from most PSs based land cover studies that these studies utilized distance/correlation measures between standard target PSs and observed PS for detection, identification, and classification of land covers [149], [153], [193] Normalized signature correlation mapper (NSCM) [194], and normalized Euclidean distance (NED) [193] are the popular distance/correlation based classifiers used with PSs for various applications. There is a lack in PSs based studies because of the fact that PSs are two dimensional features whereas the common processing techniques used with these PSs such as the NSCM or the NED, perform vector operations, and therefore are unable to consider the neighbourhood influence during target (land cover) identification or classification. Hence, the PSs are under-utilized and provide sub optimal results.

The current study also addresses this challenge of under-utilization of PSs and proposes a novel approach based on convolutional deep neural networks to fully exploit the PSs as land cover features. Convolutional deep neural networks (CNNs) are recently one of the most popular, efficient, and powerful techniques in image feature extraction and classification applications. Their parameter sharing and local connectivity properties makes them best tools to extract complex and localized features from images [35]. A brief introduction to CNNs is provided in chapter 3. In the recent years, CNNs in combination with deep learning technology are being widely used in PolSAR data applications [58]. The utilization of CNNs with PSs for land cover classification has been introduced in chapter 3. The study in chapter 3 indicates the significance of CNNs as PSs feature extractors. The study also reports improvement in classification performance with utilization of CNNs with PSs for land cover classification. Therefore, the study presented here in chapter 4 explores and extends the utilization of CNNs with PSs in order to extract built-up areas from a PolSAR image.

The study also utilizes *apriori* knowledge and image morphology to build village detection filters and to mask out misclassifications. Image morphology has been extensively used in studies to detect built-up areas from satellite images [195].

To summarize, the study in this chapter presents a novel approach to detect and map villages affected by a flood with the help of PSs, CNNs, *apriori* knowledge and image morphology. The approach presented here involves three stages. These are;

- *first*, design and development of PSs based CNN model for extraction of built-up areas from PolSAR image which contains the flooded region,

- *second*, development of novel methodology for design of village detection filters based on *apriori* knowledge and image morphology, and

- *third*, development of novel methodology for identification, and mapping of flood affected villages using image morphology.
This chapter is divided into following sections. Section 4.2 provides detailed information on the experiment setup, study areas and datasets considered and, ancillary information collected for the study. Section 4.3 discusses the methodologies developed in the three stages of the study. Qualitative and quantitative evaluation of the results from each methodology is also done and discussed in this section. At last, section 4.4 concludes the study.

## 4.2. Experiment Setup

### 4.2.1. Study area

Two areas are considered for the analysis, development and testing of the proposed methodology. The first study area is common from chapter 3. This study area is termed as 'Roorkee' study area in this chapter. Details of this study area are already provided in section 3.3.1. The second study area includes Haridwar city of the northern India and neighbouring region. The area extends from 29°58'23"N, 77°49'52"E to 29°49'23"N, 78°10'5"E The area covers about 270 km² of landscape. This second study area is termed as 'Haridwar' study area in this chapter. Both study areas have mixed class type landscape, which is beneficial for the evaluation of classifiers performance in mixed class scenarios. Figure 4.1(a) and Figure 4.2(a) shows a Google Earth image of the Roorkee and Haridwar study areas respectively.

### 4.2.2. Satellite data

PolSAR data used in chapter 3 (refer section 3.3.2) is considered here as well for experiment purpose. Preprocessing is required with raw PolSAR data as it suffers from artefacts such as speckle noise. Preprocessing of the PALSAR-2 PolSAR data is similar to as explained in chapter 3 (refer section 3.3.4). Four features namely HH, HV, VH, and VV are obtained. Since, HV and VH are similar due to the reciprocity theorem, only HH, HV, and VV are used further. Figure 4.1(b) shows a false colour image of the Roorkee dataset and Figure 4.2(b) shows a false colour image of the Haridwar dataset

### 4.2.3. Ground truth

Ground truth is collected on two counts. First, ground truth sample points are collected for classifier training and validation. Five land cover classes are considered namely bare soil (BS), tall vegetation (TV), short vegetation (SV), built-up, and water. Ground truth class sample points are measured directly on the terrain based on visual inspection. Approximately 1200 data points are collected for both the study areas. Details of the ground truth collected for both study areas are shown in Table 4.1.
Second, the total number of villages (a common term opted here to represent all types of human settlements in the study region) in each study area are counted for validation of the proposed approach. Table 4.1 also lists the actual number of villages in the Roorkee and Haridwar study areas. Further, Figure 4.3(a, and b respectively) highlights (in white colour) the villages in the Roorkee and Haridwar study regions captured from Google Earth imagery via a digitization process. Any village in the rural parts of India consists of a minimum of 20 houses and is spread over a minimum area of 7600-meter square. These physical dimensions help in detection of villages in a PolSAR image.

It is important to note here that in this chapter; different terminologies have been used at different stages to represent different versions of built-up areas. Basic terminologies have been defined here.

* Built-up structure: Man-made structures.

* Built-up area: A collection of built-up structures.

* Village or town: Built-up area larger than a predefined size.

* Object: A connected group of image pixels.

* Detected villages: Villages that have been successfully detected.

* Flooded villages: Villages that are affected by flood.

These terminologies help to better understand the different stages of the study.

Also, in order to evaluate the performance of proposed approach, artificial flood scenarios are considered and used during the study. Details of the process of creating artificial flood scenarios are provided in section 4.3.3.2.

<figure><img src="image_36.png" /><figcaption>Figure 4.1 (a) Google Earth imagery of the Roorkee with markers indicating major land cover classes. Markers highlighted in dots: red = built-up, green = TV, blue = water, light green = SV, and yellow = BS (Courtesy: Google Earth). (b) False colour representation of preprocessed SAR data.</figcaption></figure>
<figure><img src="image_37.png" /><figcaption>Figure 4.2 (a) Google Earth imagery of the Haridwar with markers indicating major land cover classes. Markers highlighted in dots: red = built-up, green = TV, blue = water, light green = SV and yellow = BS (Courtesy: Google Earth). (b) False colour representation of preprocessed SAR data</figcaption></figure>

<table><caption>Table 4.1. Summary of ground truth samples collected for the study. *BS = bare soil, TV = dense tall vegetation, SV = short vegetation. All points are acquired directly on the field by visual inspection at selected GPS coordinates. Also, the total number of villages in both the study regions are listed here.</caption><thead><tr><th rowspan="2">Land Cover</th><th colspan="2">Roorkee</th><th colspan="2">Haridwar</th></tr><tr><th>Training and validation</th><th>Testing</th><th>Training and validation</th><th>Testing</th></tr></thead><tbody><tr><td>BS</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>TV</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>SV</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Built-up</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Water</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Total</td><td>1000</td><td>200</td><td>1000</td><td>200</td></tr></tbody><tfoot><tr><td colspan="2">Total number of villages in Roorkee region</td><td colspan="3">93</td></tr><tr><td colspan="2">Total number of villages in Haridwar region</td><td colspan="3">60</td></tr></tfoot></table>

<figure><img src="image_38.png" /><figcaption>Figure 4.3 Areas highlighted in white are villages in the (a) Roorkee, and (b) Haridwar study areas. Village areas are extracted from Google Earth imagery of the corresponding regions. (Courtesy: Google Earth).</figcaption></figure>
## 4.3. Proposed Approach, Implementation, and Discussion

This section discusses the development of a novel approach to detect, identify, and map flood affected villages in a PolSAR image. The overall approach is divided into three stages.

First, design and development of a novel PSs based CNN model for extraction of built-up areas in PolSAR image. The design and development of the novel PSs based CNN model is discussed in sections 4.3.1 and 4.3.2. Model performance evaluation and comparisons are discussed in section 4.3.2.

Second, development of a novel methodology for design of village detection filters using *apriori* knowledge and image morphology. These filters help in detection of villages in a built-up area image and also help suppress false alarms or misclassifications. Section 4.3.3.1 covers the development of this methodology. Qualitative and quantitative evaluation of the detection performance of this methodology is also covered in this section.

Third, development of methodology for identification and mapping of villages that are affected by flood and masking out villages that are not affected by floods. A novel methodology, which exploits the phenomena of enclosing of a village by water if it's flooded, is developed. Section 4.4.3.3.2 discusses the development of this methodology. Evaluation of this novel methodology is discussed in section 4.4.3.3.2. The three methodologies proposed in the three stages sum up to provide a novel flood affected village detection and mapping approach. Flowchart shown in Figure 4.4 depicts the overall proposed approach.

### 4.3.1. Land cover polarization signatures

According to the discussion in section 3.2.1, both, co-polarized and cross-polarized PSs are computed from pre-processed PolSAR data using equation (3.1). Co-polarized PS plots for considered land cover class (averaged over all class samples, refer to Table 4.1) are shown in Figure 4.5. Each PS plot in figure 4.5 shows backscatter power response for ellipticity angle $\chi$ ranging from -45° to +45° and orientation angle $\psi$ ranging from -90° to +90° with a step size of 5° in each direction resulting in a 18×37 two dimensional matrix. It is evident from Figure 4.5 that PSs of considered land covers are different from each other.

These PSs are used as features for land cover characterization and classification in many studies. Classification schemes used in [149], [179], and [193] for PolSAR PSs based land cover classification are distance/correlation based which do not consider the neighbourhood relationship of a pixel during classification and therefore PSs remain under-utilized. There is a need to explore and develop advanced classifiers that can capture local spatial relationship of a PS pixel during classification which might improve the classification. A CNN based PSs
classification technique is introduced in chapter 3 which considers the influence of the
neighbourhood of a PS pixel and thereby utilizes the PSs efficiently. The current study extends
the utilization of PSs with CNNs to extract built-up areas from PolSAR image.

<figure><img src="image_39.png" /><figcaption>Figure 4.4 Flowchart of the proposed approach</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>Figure 4.5 Sample co-polarized PSs of land cover classes mentioned in Table 4.1. (a) bare soil, (b) TV, (c) SV, (d) built-up, and (e) water.</figcaption></figure>
4.3.2. Stage 1: CNN and polarization signatures based classification model for built-up area
extraction

In this stage, CNNs are utilized to extract features from PSs for built-up area extraction. A CNN based PSs classification technique is first introduced in chapter 3. The technique was able to capture the influence of the neighbourhood of a PS pixel and thereby utilizing the PSs more efficiently. A novel CNN model similar to the model discussed in section 3.6 is developed here with the help of convolutional and fully connected, layers. The proposed CNN model is termed as PSs-CNN model from here onwards in this chapter. The proposed PSs-CNN model configuration is provided in Table 4.2. Unlike the CNN model in chapter 3 (refer section 3.6) where per pixel PS (polarization signature of one pixel) is used for classification, a window of PSs is used as input here. The window based approach helps suppresses the random volume scattering responses from closely packed built-up structures in built-up areas and thereby improves the segregation of these built-up areas.

A 3×3 window of neighbouring PSs is used here as input. This corresponds to a 54×111 (refer section 4.3.1) image as input to the PSs-CNN model. This 54×111 input image has two channels i.e. copol and crosspol. Therefore, the overall size of input to the PSs-CNN is 54×111×2. The PSs-CNN used five convolutional layers. The number of convolution filters and filter sizes used in each layer are listed in Table 4.2. ReLU activation function and dropout fraction of 10% is used in each convolutional layer. The dropout regularization strategy is employed to reduce overfitting and achieve better generalization. Further, two Fully Connected or FC layers, first with 50 nodes and second with 10 nodes are used. Both FC layers used the ReLU activation function and a dropout fraction of 10%. Finally, a Softmax classification layer with 3 output labels is used. The first output class is the built-up class which needs to be extracted, the second is 'rest' class (flood), and the third output class is TV class because TV and built-up are the only classes that are not completely submerged in water during a flood. A batch normalization process is carried out before the convolutions to speed up the training process and to reduce the network sensitivity. Overall, an eight layered PSs-CNN classification model is realized as shown in Figure 4.6.
<table><caption>Table 4.2 Summary of PSs-CNN model configuration, hyperparameters and training</caption><thead><tr><th>Layer</th><th>Input</th><th>Convolutional layers C1, C2, C3, C4, C5</th><th>Fully connected layers FC1,FC2</th><th>Output layer</th></tr></thead><tbody><tr><td>PSs- CNN<br>Architecture</td><td>Input image size =<br>57*111*2<br>Normalization =<br>Batch<br>Normalization</td><td>Filters = 10, 10, 10,<br>15, 15,<br>Kernel size =9x13,<br>9x13, 9x13, 15x33,<br>15x33<br>Activation = ReLU,<br>ReLU, ReLU, ReLU,<br>ReLU<br>Dropout fraction =<br>0.1, 0.1, 0.1, 0.1, 0.1<br>Pooling size = 0, 0, 0,<br>0, 0,</td><td>Nodes =50,10<br>Activation = ReLU,<br>ReLU<br>Dropout fraction =<br>0.1, 0.1</td><td>Labels = 3<br>Activation =<br>Softmax</td></tr><tr><td>PSs- CNN<br>Training<br>parameters</td><td colspan="4">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam</td></tr></tbody></table>

<figure><img src="image_41.png" /><figcaption>Figure 4.6 Architecture of the proposed PSs-CNN.</figcaption></figure>

The developed PSs-CNN model is trained on labelled samples collected via ground survey (refer Table 4.1). During the training of PSs-CNN, the bare soil, SV, and water class samples are combined and trained as one output class i.e. "rest" in order to artificially create a flood like scenario. The loss function considered during training is the categorical cross entropy function. The learning rate is set to 0.01 and the number of training epochs is set to 200. Adam optimizer is used for network weight optimization. Model architecture and hyperparameters are set similar for Roorkee and Haridwar datasets for consistency purposes and are listed in Table 4.2.

Figure 4.7(a) shows the classified image for Roorkee and Figure 4.7(c) shows the classified image for Haridwar using the PSs-CNN model as classifier. To assess the classification performance of novel PSs-CNN model, the model performance is compared with two classifiers that are mostly used with PSs in land cover classification i.e. the normalized signature correlation
mapper (NSCM) based minimum distance classifier and the normalized Euclidean distance (NED) based minimum distance classifier.

Signature correlation mapper or SCM is a cross correlogram constructed by calculating the cross-correlation between two polarimetric signatures. The SCM obtains from the Pearson correlation coefficient and varies from -1 to 1. The SCM between two signatures p₁ and p₂ is given by equation (4.1).

$$ SCM(p_1, p_2) = \left\{ \frac{\sum_{1}^{m} p_1 p_2 - \sum_{1}^{m} p_1 \sum_{1}^{m} p_2}{\sqrt{\sum_{1}^{m} p_1^2 - \left(\sum_{1}^{m} p_1\right)^2}} \right\} \quad (4.1) $$

where m is the number of polarization states [149], [194]. NSCM is obtained by normalizing the range of SCM between 0 and 1 using equation (4.2).

$$ NSCM = 0.5 \times (SCM + 1) \quad (4.2) $$

Euclidean distance or ED is the distance metric which provides a quantitative measure of the distance between vectors representing the two polarimetric signatures, and it reflects the dissimilarity between the polarimetric signatures [149], [196]. The ED between two signatures p₁ and p₂ is given by equation (4.3).

$$ ED(p_1, p_2) = \sqrt{\sum_{1}^{m} (p_1 - p_2)^2} \quad (4.3) $$

where m is the number of polarization states. Further, NED is the normalized version of ED. Both NSCM and NED have been utilized with PSs for land cover characterization in [149] and for land cover classification in [193].

Figure 4.7(e) and Figure 4.7(g) shows classified images using the NSCM classifier (minimum distance is considered as classifier) for Roorkee and Haridwar respectively whereas Figure 4.7(i) and Figure 4.7(k) shows the classified images using the NED classifier (minimum distance is considered as classifier) for Roorkee and Haridwar respectively. It is evident from Figure 4.1, Figure 4.2, and Figure 4.7(a, c, e, g, i, and k) that the PSs-CNN outperforms the NSCM and the NED on both, the Roorkee, and the Haridwar datasets. Because of the point-wise operability, the NED and NSCM classifiers are not able to capture and exploit the neighbourhood relationship of a PS pixel and thereby are not able to efficiently segregate the built-up areas as discussed in section 4.1. The developed PSs-CNN classifier resolves this issue significantly by extracting and utilizing the subtle differences in the PSs of built-up and TV and successfully
separates the two. It is evident from Figure 4.7(a) & Figure 4.7(c) that PSs-CNN classifier is successful in segregating the built-up (represented by red colour) and the TV (represented by green colour) classes. Built-up class is extracted from the PSs-CNN, NSCM, and NED classified images are shown in Figure 4.7(b, f, and j) and Figure 4.7(d, h, and l) for Roorkee and Haridwar respectively. It is evident that the PSs-CNN is able to extract the built-up areas (highlighted in white) very efficiently.

Table 4.3 summarizes the quantitative assessment of classification performance of the PSs-CNN, NSCM, and NED classifiers for Roorkee and Haridwar datasets. The PSs-CNN classifier performs land cover classification with good overall accuracy (OA), i.e. 87% for Roorkee and 89% for Haridwar dataset. In contrast, the NED performed the classification with 75% OA for Roorkee and 73% OA for Haridwar, and the NSCM performed the classification with 74% OA for Roorkee and 76% OA for Haridwar. Both NED and NSCM show low kappa values (refer Table 4.3) whereas the kappa for PSs-CNN is relatively high. Furthermore, the PSs-CNN classifier successfully segregates the built-up class. The built-up class is segregated with 81.8% user and 90.0% producer accuracy for Roorkee and 82.5% user and 92.5% producer accuracy for Haridwar using the PSs-CNN classifier which supports the visual interpretations.

<figure><img src="image_42.png" /><figcaption>Figure 4.7 Roorkee classified image obtained using: (a) the PSs-CNN classifier, (e) the NED classifier, and (i) the NSCM classifier. Areas classified as built-up in Roorkee study area using: (b) the PSs-CNN classifier, (f) the NED classifier, and (j) the NSCM classifier. Haridwar classified image obtained using: (c) the PSs-CNN classifier, (g) the NED classifier, and (k) the NSCM classifier. Areas classified as built-up in Haridwar study area using: (d) the PSs-CNN classifier, (h) the NED classifier, and (l) the NSCM classifier. Colour scheme: red = built-up, green = TV, and blue = rest (Water + BS + SV).</figcaption></figure>
<table><caption>Table 4.3 Summary of the classification performance of the considered classifiers considered for Roorkee and Haridwar datasets. OA = overall accuracy. Rest = BS + Water + SV.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Method</th><th rowspan="2">OA (%)</th><th rowspan="2">Quadratic Kappa</th><th colspan="3">User accuracy (%)</th><th colspan="3">Producer accuracy (%)</th></tr><tr><th>Rest</th><th>TV</th><th>Built-up</th><th>Rest</th><th>TV</th><th>Built-up</th></tr></thead><tbody><tr><td rowspan="3">Roorkee</td><td>PSs-CNN</td><td>87.0</td><td>0.77</td><td>96.2</td><td>71.4</td><td>81.8</td><td>85.8</td><td>87.5</td><td>90.0</td></tr><tr><td>NSCM</td><td>74.7</td><td>0.56</td><td>86.7</td><td>57.6</td><td>60.6</td><td>83</td><td>75</td><td>50</td></tr><tr><td>NED</td><td>75.0</td><td>0.55</td><td>86.2</td><td>56.8</td><td>63.6</td><td>83.3</td><td>72.5</td><td>52.5</td></tr><tr><td rowspan="3">Haridwar</td><td>PSs-CNN</td><td>89.0</td><td>0.80</td><td>96.5</td><td>75.8</td><td>82.5</td><td>93.3</td><td>72.5</td><td>92.5</td></tr><tr><td>NSCM</td><td>76.0</td><td>0.58</td><td>91.8</td><td>52.7</td><td>58.4</td><td>85</td><td>47.5</td><td>77.5</td></tr><tr><td>NED</td><td>73.0</td><td>0.51</td><td>83.7</td><td>51.2</td><td>62.1</td><td>83.3</td><td>47.5</td><td>67.5</td></tr></tbody></table>

### 4.3.3. Detection and mapping of flood affected villages using apriori knowledge and image morphology

This section presents post classification methodologies based on *apriori* knowledge and image morphology to map the villages affected by flood from built-up area image obtained in stage 1. This section covers stage 2 and stage 3 of the proposed approach. The methodology presented in stage 2 detects villages in the built-up area image obtained in stage 1 (refer section 4.3.2). The methodology utilizes *apriori* knowledge and image morphology to build village detection filters. These filters detect villages in the built-up area image and significantly remove false alarms or misclassifications. Development of this methodology is discussed under stage 2 in section 4.3.3.1. Stage 3 presents a novel methodology to identify and map villages that are affected by flood and mask out villages that are not affected by flood. This methodology also utilizes image morphology to map flood affected villages. Development of this methodology is discussed in section 4.3.3.2.

#### 4.3.3.1. Stage 2: Village detection

The focus of this stage is to detect villages in the built-up area image obtained in stage 1 in section 4.3.2. In the methodology, at first, the built up area image — a binary image— is cleaned before detection because the image might contain objects (built-up areas) that are close to each other and could be part of same village or, there might be objects with holes in them which is undesirable. These artefacts could lead to inaccurate detection and mapping of villages. Therefore, a series of morphological operations are utilized to clean the binary image. The objects or components which are close to each other, are merged together at first using a morphological image *closing* filter provided in equation (4.4) [197].

$$I_{\text{closed}} = (I \oplus S) \ominus S \qquad (4.4)$$
Where, $I_{\text{closed}}$ is the closed image, I is the image on which the morphological closing operation is to be performed, S is the structuring element with which the closing operation is performed, $\oplus$ and $\ominus$ represents the morphological dilation and erosion operations respectively. The vicinity, within which two objects can be merged together is set via the size of the structuring element parameter. The vicinity is set based on *apriori* knowledge of the approximate minimum distance between two villages and image pixel resolution.

The approximate minimum distance between two villages is considered here to 200 meters (based on ground surveys and experiment) and the ALOS PALSAR-2 PolSAR data considered in this study has 6-meter spatial resolution. Therefore, S is set as a disc of 15 pixels' diameter (100 meters) and two objects within this distance are merged as one. The effect of the closing operation is highlighted in Figure 4.8. A segment of built-up area image shown in Figure 4.7 is shown in Figure 4.8(a). Figure 4.8(b) depicts the effects of closing operation on the segment. It can be observed from Figure 4.8(b) that some objects are merged together.

Next, the holes in the objects in the $I_{\text{closed}}$ image are to be filled. The holes are filled using equation (4.5) [197].

$$I_{\text{filled}} = I_{\text{closed}} \cup X_k \qquad (4.5)$$

Where, $I_{\text{filled}}$ is the image with holes filled, $I_{\text{closed}}$ is the image with holes obtained from equation (4.4), $\cup$ is the *union* operator, and $X_k$ is obtained through iterative process governed by equation (4.6) [197],

$$X_k = (X_{k-1} \oplus S) \cap I_{\text{complement}} \qquad (4.6)$$

Where again, $X_0$ is the $I_{\text{closed}}$ image, S is the connectivity, and $I_{\text{complement}}$ is the negative of $I_{\text{closed}}$. Figure 4.8(c) depicts the result of the hole filling operation.

Finally, an area limit is set to filter out objects that are not villages. The idea is that any object considered to be a village is to have at least 5000 square meters of ground area. The area limit is established based on *apriori* knowledge of the study region. An area limit (AL) of 5000 square meters (140 pixels approx. in case of PALSAR-2 image considered here) is set. Equation (4.7) demonstrates the process of filtering objects based on its area.

$$(O_k)_{\text{filtered}} = \begin{cases} 1 & \text{if } (O_k)_{\text{filled}} > AL \\ 0 & \text{if } (O_k)_{\text{filtered}} < AL \end{cases} \text{ for } k = 1, 2, \dots, N \qquad (4.7)$$

Where, $(O_k)_{\text{filtered}}$ is the $k^{th}$ object in the filtered image and $(O_k)_{\text{filled}}$ is the $k^{th}$ object in the $I_{\text{filled}}$ image, AL is the area limit, and N is the total number of objects in $I_{\text{filled}}$ image.
Figure 4.8(d) depicts output of the area based filtering. It is observed from Figure 4.8(d) that many unwanted objects have been removed successfully. The objects remained in the filtered image are successfully detected as villages.

The closing, hole filling, and the area filter operations in sequence act as a 'detection filter' which detects villages in a built-up area image and mask out false alarms or misclassifications. This village detection filter is applied to the built-up area images of the PSs-CNN, NSCM, and NED.

Output images after applying the village detection filter are shown in Figure 4.9. These output image(s) which contains only the successfully detected villages is termed as "DV" image(s) hereafter in this chapter.

The DV images corresponding to the PSs-CNN, NSCM, and NED classifiers are assessed using reference data. The reference images for both Roorkee and Haridwar are provided in section 4.2.3 (refer Figure 4.3). The reference images contain all the villages in the corresponding study areas. Figure 4.9 shows false colour composites (FCC) of the DV images of the PSs-CNN, NSCM, and NED, and reference images for Roorkee and Haridwar areas. Figure 4.9a) shows the FCC namely reference-DV<sub>PSs-CNN</sub>-DV<sub>NED</sub> for Roorkee. In Figure 4.9(a), red colour objects are reference objects (actual villages), green colour objects are villages as per the DV<sub>PSs-CNN</sub>, and blue colour objects are villages as per the DV<sub>NED</sub>. Further, white colour objects are objects which are present in all three images i.e. the reference image, the DV<sub>PSs-CNN</sub> image, and the DV<sub>NED</sub> image.

It means that the object is actually a village and is correctly extracted by both, the PSs-CNN and the NED. Yellow colour objects are villages as per the reference and the PSs-CNN but not as per the NED. Cyan colour objects are villages as per the reference and the NED but not as per the PSs-CNN. The abundance of yellow colour in Figure 4.9(a) indicates to the efficient extraction of built-up areas by the PSs-CNN classifier. Similarly, the reference-DV<sub>PSs-CNN</sub>-DV<sub>NSCM</sub> false colour composite for Roorkee shown in Figure 4.9(c) also evidences the better performance of the proposed PSs-CNN classifier.

However, the reference-DV<sub>PSs-CNN</sub>-DV<sub>NED</sub> and the reference-DV<sub>PSs-CNN</sub>-DV<sub>NSCM</sub> false colour composites for Haridwar in Figure 4.9(b) and Figure 4.9(d) show numerous green colour objects. These objects are falsely detected as villages as per PSs-CNN and indicate false alarms. A quantitative analysis is therefore required to assess the performance of the classifiers in more detail.
<figure><img src="image_43.png" /><figcaption>Figure 4.8 Results from different morphological operations considered during village detection: (a) Segment of image shown in Figure 4.7(a). Corresponding segments of: (b) Image after the *closing* operation, (c) Image after the *holes filling* operation, and (d) image after the *filtering* operation.</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>Figure 4.9 (a) False colour composites for Roorkee using reference-DV<sub>PSs-CNN</sub>-DV<sub>NED</sub>, (b) False colour composites for Haridwar using reference-DV<sub>PSs-CNN</sub>-DV<sub>NED</sub>, (c) False colour composites for Roorkee using reference-DV<sub>PSs-CNN</sub>-DV<sub>NSCM</sub>, and (d) False colour composites for Haridwar using reference-DV<sub>PSs-CNN</sub>-DV<sub>NSCM</sub>. Colour scheme: red = reference, green = DV<sub>PSs-CNN</sub>, blue = DV<sub>NED</sub> or DV<sub>NSCM</sub></figcaption></figure>

For quantitative evaluation of the DV images corresponding to the PSs-CNN, NSCM, and NED classifiers, at first, a bounding box to mark the extent of each object in both the
reference and the test image is computed. Then, a bounding box overlap ratio (BBOR) is computed using equation (4.8) for each object pair of the reference and the test image.

$$BBOR = \frac{\text{Overlap Area}_{(\text{test,reference})}}{\min(\text{Bounding Box Area}_{\text{reference}}, \text{Bounding Box Area}_{\text{test}})} \quad (4.8)$$

If the BBOR of a particular object pair is above a predefined threshold, then the corresponding object in the test image is counted a successful detection, otherwise not. The threshold is set to 0.1 based on experiment and critical analysis, and is set identical for both Roorkee and Haridwar study areas for consistency purpose.

True positives (villages successfully detected), true negatives (villages missed), and false alarms or false positives (objects detected as villages in the test image but are actually not) are computed using equation (4.9, 4.11, and 4.12).

$$\text{True Positive (TP)} = \sum_{i=1}^{m} \text{logical}(\text{BBOR} > \text{threshold}) \quad (4.9)$$

Where, m is the total number of objects (villages) in the reference image, and

$$\text{logical}(x) = \begin{cases} 1 & \text{if } x > \text{threshold} \\ 0 & \text{if } x < \text{threshold} \end{cases} \quad (4.10)$$

$$\text{True Negative (TN)} = \text{Total number of villages in reference image} - \text{TP} \quad (4.11)$$

$$\text{False Positive (FP)} = \text{Total number of objects in test image} - \text{TP} \quad (4.12)$$

True positives (TP), true negatives (TN), and false positives (FP) are computed for Roorkee and Haridwar study areas for PSs-CNN, NED, and NSCM. Table 4.4 summarizes the quantities assessment of the DV images of the PSs-CNN, NED, and NSCM classifiers. The PSs-CNN successfully extracts 75 (out of 93) villages in the Roorkee study area and 47 (out of 60) villages in the Haridwar study area. Whereas, NED successfully extracts 26 villages in Roorkee and 20 villages in Haridwar, and NSCM successfully extracts 22 villages in Roorkee and 27 villages in Haridwar. The low detection rate in case of the NED and the NSCM resonates with their low classification accuracies for built-up class (please refer Table 4.2).

The false alarms are also high for the NED and the NSCM. The low values of kappa for NED and NSCM (please refer Table 4.2) indicate to the high false alarms. The PSs-CNN offers higher detection rate and low false alarms for both Roorkee and Haridwar study areas.
<table><caption>Table 4.4 summary of quantitative assessment of performance of PSS-CNN, NED, and NSCM</caption><thead><tr><th>Dataset</th><th>Measures</th><th>NED</th><th>NSCM</th><th>PSs-CNN</th></tr></thead><tbody><tr><td rowspan="4">Roorkee</td><td>Total number of villages in the study area</td><td>93</td><td>93</td><td>93</td></tr><tr><td>TP (Villages successfully mapped)</td><td>26</td><td>22</td><td>75</td></tr><tr><td>TN (Villages missed)</td><td>67</td><td>71</td><td>18</td></tr><tr><td>FP (or False Alarms)</td><td>31</td><td>41</td><td>1</td></tr><tr><td rowspan="4">Haridwar</td><td>Total number of villages in the study area</td><td>60</td><td>60</td><td>60</td></tr><tr><td>TP (Villages successfully mapped)</td><td>20</td><td>27</td><td>47</td></tr><tr><td>TN (Villages missed)</td><td>40</td><td>33</td><td>13</td></tr><tr><td>FP (or False Alarms)</td><td>33</td><td>39</td><td>5</td></tr></tbody></table>

### 4.3.3.2. Stage 3: Mapping flood affected villages

Since, a PolSAR image acquired to monitor a flood might include both, flooded and not flooded regions. Therefore, it is necessary for any approach which maps only flooded villages to mask out the non-flood affected villages present in the PolSAR image. The village detection methodology proposed in section 4.3.3.1 does not separate out flooded and non-flooded villages. Therefore, a new methodology is proposed to achieve the task. This methodology presented in this section encompasses stage 3 of the overall approach.

Intuitively, the villages that are flooded must be completely enclosed by water, otherwise not. This means that any village affected by flood should appear as a 'hole' in the water class (rest class in this case) image and any village that is not affected by flood should not. Based on this observation, a criterion is set i.e. if an object in the DV image obtained in section 4.3.3.1 is a subset of an object in the rest class image, then, that object in the DV image is a flooded village otherwise not. This criterion will separate the flooded village from non-flooded villages.

To realize the criterion, a bounding box overlap ratio (BBOR2) is computed over each pair of DV image object and the "holes filled" version of rest class image object. The holes filled rest class image is obtained using equation similar to (4.8) If the BBOR2 is 1, then the corresponding object in the DV image is a flooded village and if the BBOR2 is less than 1, the corresponding object in the DV image is not a flooded village. This methodology is depicted in Figure 4.10.

The methodology is evaluated on three different scenarios on the DV image of Roorkee corresponding to the PSs-CNN. The three different scenarios are shown in Figure 4.11. In the first scenario i.e. scenario 1, flood has affected the whole study region (already discussed in section 4.3.2) whereas, in the second and third scenarios i.e. scenario 2 and 3, flood has affected parts of the study region (partial flooding). When the DV image of Roorkee corresponding to the PSs-CNN is subjected to these three scenarios, the resulting flood situations are shown in Figure 4.12(a, b, and c). It is clear from Figure 4.12(a) that all the villages are affected by flood whereas in Figure 4.12(b) and Figure 4.12(c), only few villages are affected. Employing the proposed methodology to the three flood situations results in identification and mapping of flood affected
villages. The flood affected villages identified in the three flood situations are highlighted in yellow in Figure 4.12(a, b, and c). It is evident from Figure 4.12(a, b, and c) that the methodology proposed in this section successfully identifies and maps villages that are affected by flood and masks out villages that are not affected by flood.

In summary, two methodologies are proposed in section 4.3.3. The first methodology developed in section 4.3.3.1 detects villages and removes false alarms and misclassifications in a built-up area image obtained from the PSs-CNN classifier. The second methodology developed in section 4.3.3.2 detects maps flooded villages and masks out non-flooded villages. These two methodologies in combination with the PSs-CNN based built-up extraction model completes the flood affected village mapping approach.

<figure><img src="image_45.png" /><figcaption>Figure 4.10 Methodology flowchart for flood affected village mapping.</figcaption></figure>
<figure><img src="image_46.png" /><figcaption>Figure 4.11 Different scenarios: (a) Scenario 1, (b) Scenario 2, and (c) Scenario 3. Colour scheme: blue = flooded region.</figcaption></figure>

<figure><img src="image_47.png" /><figcaption>Figure 4.12 Flood maps: (a) Affected villages under flood condition 1. (b) Affected villages under flood condition 2. (c) Affected villages under flood condition 3. Colour scheme: red = villages not affected by flood, yellow = villages affected by flood, and blue = flooded region.</figcaption></figure>
## 4.4. Conclusion

This chapter presented an efficient approach to map flood affected villages using PolSAR image. The approach included three stages i.e. *first*, built-up area extraction, *second*, village detection, and *third*, flood affected village mapping.

In the first stage, the approach utilized PolSAR polarization signatures (PSs) and CNNs to extract built-up areas from a PolSAR image. A novel PSs-CNN classifier is developed. The PSs-CNN model provided improved built-up area extraction performance in contrast to the NED and the NSCM classifiers. In the second stage, the approach introduced and implemented a novel village detection methodology. The methodology uses a detection filter to detect whether a particular object in the built-up area image (obtained in stage 1) is a village or not. The detection filter uses *apriori* knowledge and image morphology to detect villages in the built-up area image. The detection methodology also significantly removes false alarms and misclassifications. In the third and final stage, flood affected villages are identified and non-flooded villages are masked out using another image morphology based methodology. This methodology computed overlap ratios between objects (successfully detected as villages) and flooded area and utilized it for identification of flood affected villages.

Quantitative and qualitative assessment of the performance of the novel approach over two different study areas is carried out. Different artificial flood scenarios are created to evaluate the performance of developed approach. The developed approach is evaluated under these flood scenarios and flood affected villages are identified and mapped. Number of villages successfully detected and false alarms are computed with the help of ground truth information (actual number of villages or towns in the study regions are known). Results indicate to high detection accuracy (above 90%). The approach may be scalable and potentially satisfactory for real time usage.
PERMUTED SPECTRAL AND PERMUTED SPECTRAL-SPATIAL
CNN MODELS FOR POLSAR-MULTISPECTRAL DATA BASED
LAND COVER CLASSIFICATION

In contrast to chapter 3 and chapter 4 where novel DNN models are proposed and developed for unimodal satellite image, this chapter explores the performance of conventional DNN models with multisensor/multimodal satellite images. Also, based on the performance of these conventional models, novel DNN models are proposed here.

In context to satellite based multisensor earth observation data, today, both synthetic aperture radar (SAR) and multispectral (MS) image data are easily available with good spatial and temporal resolutions. The two data modalities with different spectral characteristics complement each other in many earth observation data applications such as land cover classification, crop monitoring, and change detection. However, it is still a challenge to develop techniques, methods, or approaches which can process the two data modalities together efficiently and without losing information from either. This chapter presents a study which attempts to introduce deep learning and DNN based novel models for multisensor (, PolSAR-MS) satellite image data processing that utilizes localized convolutions in both spatial and spectral domains to perform land cover (LC) classification. The study also attempts to establish the significance of performing ensembled localized spectral convolutions in comparison to non-localized or localized spectral convolutions. Two novel models namely; permuted local spectral convolutional neural network (Perm-LS-CNN), and permuted local spectral-spatial convolutional neural network (Perm-LSS-CNN) are proposed in order to achieve better classification results in terms of both accuracy and generalization.

"Generalization is the measure of the ability of machine learning techniques on how
well these techniques perform on 'out of sample' data."
## 5.1. Introduction

The synthetic aperture radar (SAR) and multispectral (MS) image data modalities are crucial for earth studies and complement each other in many earth observation applications such as land cover classification, crop monitoring, or change detection. In general, multispectral image provides spectral reflectance information and PolSAR image provides scattering and polarizing behaviour information about the earth surface objects. These earth surface objects are more commonly termed as land covers. The basic idea behind utilizing the two image data modalities together is to acquire better spatial, spectral, and temporal information on the land covers. Although, higher spectral information can be achieved through hyperspectral images but at a cost of spatial and temporal information. Also, since it is an earth observation strategy based on optical imaging, therefore it suffers from the classical challenges (clouds, night-time). Similarly, use of polarimetric SAR (PolSAR) in standalone fashion for land cover classification resulted in a maximum of 75% classification accuracy in mixed land cover class scenarios (refer section 3.7). For applications such as land use land cover classification or land cover monitoring, all three aspects are crucial and a multisensor data based approach is more beneficial. Using PolSAR and MS image data modalities together is shown to be a convenient and successful strategy. Methodologies listed in many articles such as in [52], or more recently in [44], [55], [198] have been used by the earth observation data processing community in processing multispectral and PolSAR data together. Authors in [44] provide a meta-analysis on the methodologies in successful applications of PolSAR and MS image data fusion. The methodologies include traditional, machine learning, and knowledge based decision trees. In [44], authors also remarked that the studies in multisensor data fusion based land cover classification mostly use MS and PolSAR data with machine learning methods. Most popular machine learning based multisensor data fusion techniques include artificial neural networks (ANNs or NNs), and support vector machines (SVMs). In fact, studies in [44]–[46] and [55] have also recognized the significance of machine learning in various multisensor image data based applications. From the aforementioned review, two points are evident; -first, utilization of PolSAR and MS data together is a successful strategy, and -second, machine learning techniques are successful in utilizing the two i.e. PolSAR and MS, image data modalities together.

However, it is challenging for (NNs) with shallow depth (number of layers ≤ 2) to extract high level abstract/complex features from high dimensional earth observation image data without overfitting or losing generalization [199]. Generalization refers to the ability of how well an algorithm performs on ‘out of the seen samples’ data points. If an algorithm is performing well
outside training data, then the algorithm is showing good generalization performance and vice versa. Researchers in many studies[5], [58], [100] are exploring new advanced machine learning approaches that are able to extract complex features from high dimensional earth observation image data. In the direction, deep neural networks with deep learning are the potential and powerful tools for extracting high level information from earth observation image data[126], [200] Convolutional neural networks (CNNs) are recently the most widely used deep network models in earth observation image data based land cover classification [58]. CNNs have the ability to extract complex features directly from data [35], [201], [202]. Another advantage of developing CNNs for PolSAR-MS multisensor image data processing is that CNNs can directly utilize multisensor high dimensional earth observation image data with efficient performance. Unlike the conventional data fusion methods, this reduces the need of fusing the PolSAR-MS images together before using it further.

In [5], authors present a comparison of CNN based models with few popular earth observation image data classifiers in processing hyperspectral images. The CNN model in [5], is designed to effectively capture global spectral features along with local spatial features. Similar studies are reported in [96],[101] and [203]. The CNN models in these studies employed one, two, and three dimensional convolutions to extract spectral and spatial information from unimodal earth observation image data. Moreover, CNNs have also been used with multisensor image data. For example, CNNs have been utilized with PolSAR-MS data in earth observation image data applications. In [72], authors used one and two dimensional CNNs on Landsat8-Sentinel1 multisensor image data for land cover and crop type classification and further compared the results with random forest (RF) classifier. Their model reports better classification performance by capturing local contextual (spectral and/or spatial) features whereas the RF classifier captured only the global dominant features. The study in [72], indicates potential of CNNs with PolSAR-MS multisensor image data for earth observation applications and motivates for similar studies.

However, one important aspect which comes to consideration while processing this PolSAR-MS data using CNNs is the efficient exploitation of the spectral information present in the high dimensional multisensor (PolSAR-MS) data. The exploitation of spectral information is crucial for CNN based multisensor data applications. Exploitation of spectral information present in hyperspectral image has been done in past, either exclusively with one dimensional CNNs or, along with the spatial information using two, or three dimensional CNNs. Many studies [100], [101], [204]–[206] and more have used one dimensional CNNs as spectral feature extractors for hyperspectral image based classification. In contrast, extracting spectral information along with spatial information from hyperspectral image using CNNs is a recently
popular approach. The approach has however improved the results in land cover monitoring and classification applications. Because hyperspectral images have large spectral dimensions, extracting spectral information is computationally expensive and compounding with extraction of spatial information makes it worse. Therefore, some studies [5], [102], [105], [207] used principal component analysis (PCA) to reduce the high dimensionality of hyperspectral image and then used two dimensional CNNs for classification applications. While, alternate to this, some studies [108], [207], [208] used one and two dimensional CNNs separately to extract the spectral and spatial information and combined the separately extracted features later. Further, some studies used three dimensional CNNs to extract local spectral and spatial information simultaneously. In [96], [98], authors used three dimensional CNNs as spectral-spatial feature extractors and classifiers for hyperspectral satellite image data.

In summary, enough evidence suggests that; one, two, and three dimensional CNNs have been successfully used to extract spectral and/or spatial information from hyperspectral image data. However, the significance of spectral and/or spatial CNNs as land cover classifiers with high dimensional multisensor image data is still interesting to investigate. Therefore, the study presented in this chapter focusses on two aspects.

* To critically analyse the one dimensional (1D), two dimensional (2D), and three dimensional (3D) CNN capabilities to perform land cover classification using multi-sensor (PolSAR-MS) data.

* Development of novel classification frameworks based on 1D (spectral) or 3D (spectral-spatial) CNNs; in order to improve the generalization ability of the classifier during land cover classification with PolSAR-MS satellite data.

1D, 2D, and 3D CNN frameworks for PolSAR-MS data based land cover classification are discussed in section 5.4. The idea of extracting the spectral information efficiently from PolSAR-MS data using CNNs is achieved by performing a permuted localized spectral convolution alone, and along with localized spatial convolution on PolSAR-MS data. Spectral permutation before localized spectral convolution is proposed to increase the generalization ability of the CNN classifier. Based on this idea, two models are proposed, namely permuted local spectral CNN (Perm-LS-CNN) and permuted local spectral-spatial CNN (Perm-LSS-CNN). Both models are discussed in section 5.5.

The remaining of the chapter is organized as follows; Section 5.2 provides a brief description on popular CNN models for earth observation image data processing. Section 5.3 provides information on the experimental setup i.e. study areas, datasets, ground truth, and other resources considered and used during the study. Section 5.4 discusses the development of conventional 1D, 2D, and the 3D CNN models for processing PolSAR-MS image data and presents internal configurations of these models. Section 5.5 presents two novel CNN based PolSAR-MS data processing frameworks for land cover classification. Results from the
developed methods and compared classifiers are discussed in section 5.6. Finally, section 5.7
concludes the chapter.

**5.2. Theoretical Background**

**5.2.1. Convolutional neural networks**

The fundamental concepts on CNNs are discussed in section 3.2.2 with an example of 2D CNNs.
Similar networks can be realized for 1D or 3D CNNs accordingly. However, the architectures of
1D, 2D, and 3D CNNs for remote satellite image data processing are as follows.

* For 1D CNNs, the satellite image data input is a vector (usually a vector of spectral features). **W**<sub>m</sub> is a matrix and **b**<sub>m</sub> is vector. In general, 1D CNNs are used as feature extractors in satellite image data, especially hyperspectral data processing.

* For 2D CNNs, the convolution is performed on the two spatial dimensions (e.g. images). The input may contain extra dimensions (such as spectral information), but these are not used in the convolution. **W**<sub>m</sub> is a 3D matrix and **b**<sub>m</sub> is vector. It is important to note that in 2D CNNs, local convolutions are only performed in the spatial dimensions and not in the spectral dimension. 2D CNNs are used as spatial feature extractors and classifiers.

* In 3D CNNs, convolutions are performed in the spatial dimensions and the temporal dimension simultaneously. However, in the absence of temporal information and presence of high spectral information, spectral dimension replaces the temporal dimension and local spectral convolutions are possible. **W**<sub>m</sub> is a 4D matrix and **b**<sub>m</sub> is vector. Therefore, 3D CNNs are used as spatial temporal or spatial spectral feature extractors and classifiers in satellite image data processing.

The present study utilizes and investigates the ability of 1D, 2D, and 3D CNNs as land
cover classifiers on PolSAR-MS data. Also, the present study proposes novel LC classification
models based on these CNNs.

**5.3. Experiment Setup**

**5.3.1. Study area**

Two areas are considered for the analysis, development and testing of the proposed
methodologies. The first study area is common from chapter 3 and chapter 4 (Roorkee study
area). Details of this study area are presented section 3.3.1. This study area is termed as ‘Roorkee
study area’ in this chapter. The second study area is also common from chapter 4 (Haridwar study
area). This second study area is termed as ‘Haridwar study area’ in this chapter. Details of this
study area are presented in section 4.2.1. Both study areas have mixed class type landscape,
which is beneficial for the evaluation of classifiers performance in mixed class scenarios. Figure
4.1(a) and Figure 4.2(a) shows a Google Earth images of the Roorkee and Haridwar study areas respectively.

### 5.3.2. Dataset

Datasets used in the study are multisensor i.e. multispectral (MS) and polarimetric synthetic aperture radar (PolSAR) data acquired over the study areas i.e. Roorkee and Haridwar. Multispectral data used is the Landsat-8 multispectral data acquired on March 17, 2015 (band specifications are provided in Table 1.2. The PolSAR data used is the ALOS PALSAR-2 L band fully polarimetric single look complex PolSAR data acquired on March 13, 2015 (common from chapter 3 and chapter 4).

Preprocessing is required with both MS and PolSAR raw data. Multispectral data mostly suffers from atmospheric perturbations whereas PolSAR data suffers from speckle noise. Preprocessing is done separately on MS and PolSAR data. Preprocessing of PolSAR PALSAR-2 data is similar to as explained in chapter 3 (refer section 3.3.4). Derived PolSAR features are also used in this study. Polarization signatures (PSs) and further, polarization signatures correlation features (PSCFs) are computed from the single look complex PolSAR data with procedure similar to as in chapter 3 (refer section 3.4). Based on the analysis of chapter 3 (refer section 3.4), four PSCFs namely “corr_co_Di”, “corr_co_FP”, “corr_co_HD”, and “corr_co_VD” are used as derived PolSAR features in the current study. Overall, three basic (HH, HV, and VV) and four derived (corr_co_Di, corr_co_FP, corr_co_HD, and corr_co_VD) PolSAR feature are used in the current study. Figure 5.1(a) and Figure 5.2(a) depicts a false colour composite from the pre-processed PolSAR image data for Roorkee and Haridwar study areas respectively.

Landsat-8 operational land imager (OLI) multispectral data is atmospherically corrected first. Out of the 11 bands shown in Table 1.2, band 2 (blue) through 7 (short wave infrared) are selected for the study. These selected bands are pan sharpened as explained in [209] and then resampled to the PALSAR-2 data spatial resolution of 6 meters using the “bicubic spline” interpolation method [210]. Figure 5.1(b) and Figure 5.2(b) depicts a false colour composite from the pre-processed MS image data for Roorkee and Haridwar study areas respectively. The re-sampled, pre-processed MS bands are co-registered with the pre-processed PolSAR channels and the stacked 13 bands (six from multispectral and seven from PolSAR) are used as input data for further analysis and study. It is clear from Figure 5.1 or Figure 5.2 that PolSAR and MS satellite image modalities complement each other. For example, built-up areas and forests in the Roorkee study area exhibit similar backscattering response (light green colour in PolSAR image, Figure 5.1(a)) whereas both the land covers are clearly separable in the MS image (refer Figure 5.1(b)).
Similarly, built-up areas are not easily separable from bare soil in MS image whereas the two land covers are easily separable in a PolSAR image.

### 5.3.3. Ground truth

Five land cover (LC) classes are considered namely bare soil (BS), tall vegetation (TV), short vegetation (SV), built-up, and water. Ground truth class data points are measured directly on the terrain based on visual inspection. Ground truth class data points collected for study in chapter 4 (refer section 4.2.3) are utilized here as well since the class labels are same and the study areas are also same. Details of the ground truth collected for both study areas are shown in Table 5.1. Also, samples images of land covers collected as ground truth are shown in Figure 3.3.

<figure><img src="image_48.png" /><figcaption>Figure 5.1 Datasets for Roorkee study area: (a) False colour composite (red = HH, green = HV, and blue = VV) of ALOS PALSAR-2 PolSAR data, and (b) False colour composite (red = NIR, green = Red, and blue = Green) of Lansat-8 multispectral data.</figcaption></figure>

<figure><img src="image_49.png" /><figcaption>Figure 5.2 Datasets for Haridwar study area: (a) False colour composite (red = HH, green = HV, and blue = VV) of ALOS PALSAR-2 PolSAR data, and (b) False colour composite (red = NIR, green = Red, and blue = Green) of Lansat-8 multispectral data.</figcaption></figure>
<figure><img src="image_50.png" /><figcaption>Figure 5.3 Flowchart of the experiment setup and CNN based land cover classification methodologies proposed with PolSAR-MS data</figcaption></figure>

<table><caption>Table 5.1 Summary of the ground truth collected for study. *BS = Bare soil, TV = Tall vegetation, SV = Short vegetation. All points are acquired directly on the field by visual inspection at selected GPS coordinates.</caption><thead><tr><th rowspan="2">Land Cover</th><th colspan="2">Roorkee</th><th colspan="2">Haridwar</th></tr><tr><th>Training and validation</th><th>Testing</th><th>Training and validation</th><th>Testing</th></tr></thead><tbody><tr><td>BS</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>TV</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>SV</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Built-up</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Water</td><td>200</td><td>40</td><td>200</td><td>40</td></tr><tr><td>Total</td><td>1000</td><td>200</td><td>1000</td><td>200</td></tr></tbody></table>

The experiment setup i.e. the study areas, the datasets, or the ground truth data points, is used for, first, the critical analysis of the performance of conventional CNN frameworks, and second, the development of novel CNN classification frameworks for land cover classification. The flowchart shown in Figure 5.3 depicts the experiment setup.

## 5.4. 1D, 2D, and 3D CNNs for Land Cover Classification with PolSAR-MS Data

### 5.4.1. 1D-CNNs for PolSAR-MS data.

One dimensional CNNs (1D CNNs) have been successfully used with unimodal satellite image data in the recent years. In [204], authors used stacked auto encoders for classification with hyperspectral satellite image data. They also used PCA for hyperspectral feature reduction before
classification as the number of input features is very high (say 176 for KSC dataset). A range i.e. from 4 to 8 principle components are considered as features during classification. PolSAR-MS data used in the present study only have 13 spectral bands and do not require feature reduction. In [100], authors used different 1D-CNNs for two popular hyperspectral satellite image data (the Salinas Valley and the Indian Pines dataset) classification. In [101], authors used 1D-CNN for hyperspectral satellite image data classification. All these references report that 1D-CNNs are good feature extractors for high-dimensional hyperspectral satellite image data. Most of these studies used some form of feature reduction technique before feeding data to the 1D-CNN in order to reduce the computational complexity of the model. The reduced number of bands are on the same order as the number of bands present in the PolSAR-MS data considered here in this study. This argument justifies the direct use of 1D-CNNs for PolSAR-MS satellite image data.

Two variants of 1D CNNs are considered here. In the first 1D CNN model i.e. 1D-CNN-v1, the kernel size of the filter in the first convolutional layer i.e. C1 layer is equal to the spectral depth of PolSAR-MS data i.e. 13. One dimensional spectral convolution is shown in Figure 5.4(b). In the second 1D CNN model, i.e. 1D-CNN-v2, the kernel size is less than the spectral depth of PolSAR-MS data. One dimensional local spectral convolution is visualized in Figure 5.4(c). Models similar to 1D-CNN-v2 has been successfully tested with hyperspectral data [96], [100] The present study focuses on the analysis of the performance of 1D CNNs; both 1D-CNN-v1 and 1D-CNN-v2 with PolSAR-MS data.

The selected hyper-parameter settings are listed in Table 5.2. Apart from kernel size, the rest of the parameters are similar for both models. Rectified linear unit (ReLU) and Softmax activation functions are used in the intermediate and output layers respectively. A dropout strategy is considered for regularization and dropout fraction values of 0.2 for C1 and 0.1 for C2 and FC layer are set. Pooling is not considered because it hinders the full usability of the spectral variation. The loss function used in both models is the categorical cross entropy function. It optimizes the classification accuracy based on probabilities of each class during classification. The learning rate and the number of epochs are set to 0.01 and 200 respectively.
<figure><img src="image_51.png" /><figcaption>Figure 5.4 Different convolution strategies: (a) Data cuboid with M by N pixels and P bands, (b) Convolution with $1 \times 1 \times P$ convolutional filter where $1 \times 1$ represents to the spatial extent and P represents total number of bands in the data. Each filter convolves with all the bands as all times. This strategy is employed in 1D-CNNs during per pixel classification of satellite images. (c) Convolution with $1 \times 1 \times 3$ convolutional filters where $1 \times 1$ represents the spatial extent and 3 represents the number of adjacent bands to be convolved simultaneously. One filter can convolve with three adjacent bands only. This strategy is also employed in 1D-CNNs. (d) Convolution with $1 \times 1 \times 3$ convolutional filters where $1 \times 1$ represents to the spatial extent and 3 represents the number of arbitrarily selected bands to be involved. This strategy is employed in the proposed Perm-LS-CNN framework.</figcaption></figure>

### 5.4.2. 2D-CNNs for PolSAR-MS data

The use of spectral information without spatial information by 1D-CNNs undermines the full potential of satellite image data. Adding spatial information provides contextual aid to the classifiers, especially CNNs in the classification process. Two dimensional or 2D-CNNs have also been successfully used with unimodal satellite image data in the recent years. In [112], authors used patch based 2D-CNN model for land cover classification with multispectral data. They opted a $5 \times 5 \times 8$ Landsat-8 image patch and a $3 \times 3$ 2D convolutional kernel. In [82], authors used 2D-CNNs for land cover classification with PolSAR data. Many studies used 2D-CNNs for classification with hyperspectral satellite image data (refer section 2.1). The need for dimension reduction in these studies is already discussed in section 5.4.1. These studies support the utilization of 2D-CNNs with PolSAR-MS data for land cover classification. In [72], authors used 2D-CNNs for crop type classification with PolSAR and multispectral data. They used Landsat-8 multispectral and Sentinel-1 C-Band PolSAR data. Although in [72] authors have studied the PolSAR-MS data potential for crop type classification with 2D-CNNs, the current study differs on: – first, using PolSAR-MS on a single time stamp with a focus on spectral and spatial features. In [72], authors used time series PolSAR-MS data with a focus on temporal and spectral features; and – second, Quad PolSAR data is considered in the present study in contrast to dual PolSAR used in [72]. The increased channels add complementary information which in turn add an indirect weighting of the spatial features. Analysis of fully polarimetric PolSAR with multispectral data for classification is therefore beneficial.
<table><caption>Table 5.2 1D-CNN-v1, 1D-CNN-v2, 2D-CNN-v1, and 3D-CNN-v1 CNN model configurations, hyperparameter and training settings.</caption><thead><tr><th rowspan="2">Model</th><th>Layer</th><th rowspan="2">Convolutional layer 1 (C1)</th><th rowspan="2">Convolutional layer 2 (C2)</th><th rowspan="2">Fully connected layer (FC)</th><th rowspan="2">Output layer</th></tr><tr><th>Input</th></tr></thead><tbody><tr><td>1D-CNN-v1</td><td>Input data size = 1×13<br>Normalization = Batch Normalization</td><td>Filters = 20<br>Kernel size = 1×13<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×1<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td></td><td></td></tr><tr><td>1D-CNN-v2</td><td>Input data size = 1×13<br>Normalization = Batch Normalization</td><td>Filters = 20<br>Kernel size = 1×4<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×4<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td>Nodes = 16<br>Activation = ReLU<br>Dropout fraction = 0.1</td><td>Labels = 5<br>Activation = Softmax</td></tr><tr><td>2D-CNN-v1</td><td>Input data size = 3×3×13<br>Normalization = Batch Normalization</td><td>Filters = 20<br>Kernel size = 3×3<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×1<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td></td><td></td></tr><tr><td>3D-CNN-v1</td><td>Input data size = 3×3×13<br>Normalization = Batch Normalization</td><td>Filters = 20<br>Kernel size = 3×3×4<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×1×4<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td></td><td></td></tr><tr><td>Training parameters</td><td></td><td>Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam [211]<br>10 fold cross validation strategy is employed during training</td><td></td><td></td><td></td></tr></tbody></table>
<figure><img src="image_52.png" /><figcaption>Figure 5.5: Different convolution strategies: (a) Data cuboid with M by N pixels and P bands, (b) Convolution with 3 × 3 × P convolutional filter where 3 × 3 represents the spatial extent and P represents the total number of bands in the data. Each filter convolves with all the bands. This strategy is employed in 2D CNNs, (c) Convolution with 3 × 3 × 3 convolutional filters where 3 × 3 represents the spatial extent and 3 represents the number of adjacent bands to be convolved simultaneously. One filter can convolve with three adjacent bands only. This strategy is employed in 3D-CNNs [96], and (d) Convolution with 3 × 3 × 3 convolutional filters where 3 × 3 represents the spatial extent and 3 represents the number of arbitrarily selected bands to be involved. This strategy is employed in the proposed Perm-LSS-CNN framework.</figcaption></figure>

A 2D-CNN model namely 2D-CNN-v1 is proposed here. 2D-CNN-v1 configuration and hyperparameter settings considered are listed in Table 5.2. The idea of a local spatial convolution is shown in Figure 5.5 (b). The configuration for the intermediate and output layers and learning parameters are the same as in section 5.4.1 and are summarized in Table 5.2.

### 5.4.3. 3D-CNNs for PolSAR-MS data

The 2D-CNN-v1 model discussed in section 5.4.2 is successful because of its spatial contextual awareness. The localized spatial convolutions provide local connectivity, which controls the impact of neighbouring pixels on the pixel to classify. Parameter sharing between local filters also makes the learning faster and less cumbersome, in contrast to fully connected neural networks. In satellite image data applications, spectral information is as important, if not more, as spatial information. Local connectivity in spectral dimension also provides the same benefits. Advantages of local spectral convolutions and local spatial convolutions together can be exploited using 3D-CNNs. The 3D-CNNs have been successfully used with hyperspectral (unimodal) satellite image data in the recent years. Authors in [96], [108], and [203] used 3D-CNNs for spectral spatial information extraction for fusion and classification applications.

The proposed 3D CNN model 3D-CNN-v1 architecture and hyperparameters considered are listed in Table 5.2. The C1 layer of 2D-CNN-v1 in section 5.4.2 uses the complete spectral depth, i.e. 13 bands, during convolutions whereas 3D-CNN-v1 uses only a subset during convolutions. The idea of local spectral and local spatial convolution is shown in Figure 5.5 (c).
For example, a first filter (in yellow) convolves with bands 1, 2, and 3 whereas a second filter (in blue) convolves with bands 5, 6, and 7. This provides an improvement in classification accuracy and robustness compared to other 3D-CNN models [96], [203] Configuration and learning parameters are the same as in section 5.4.1 and are summarized in Table 5.2.

## 5.5. Proposed Permuted Local Spectral CNN (Perm-LS-CNN) and Permuted Local Spectral-Spatial CNN (Perm-LSS-CNN)

In 1D-CNN-v2, a local convolution is performed over the spectral dimension. The convolution filter size in the spectral dimension i.e. 4 is less than the total number of input bands i.e. 13. This local spectral convolution improves the classification performance as discussed in [96]. A pictorial representation of local spectral convolution is depicted in Figure 5.4(c). For example, a first filter (in yellow) convolves with bands 1, 2, and 3 whereas a second filter (in blue) convolves with bands 4, 5, and 6. However, an issue arises in the process of local spectral convolution with high dimensional satellite image data such as hyperspectral data or multisensor data e.g. PolSAR-MS data. While performing local spectral convolution, various combinations of spectral bands are not convoluted together and therefore, under-utilizing the full potential of the spectral information present in the data. For example, it is evident from Figure 5.4(c), that the second spectral band and the tenth spectral band cannot not get convoluted together using 1D-CNN-v2 and hence the spectral information is under-utilized. The issue needs more attention for multisensor data processing with CNNs. For PolSAR-MS data, some polarimetric channels might not convolute together with some multispectral bands. For example, out of the 13 bands of the PALSAR-2-Landsat-8 dataset considered in the present study, only 4 consecutive bands get convoluted together with a convolutional kernel window of 4. Moreover, stacking PolSAR and MS data in a common feature set induces an arbitrary ordering and irrelevant band neighbourhood relationship which might impact the performance of the local spectral convolution strategy employed in 1D-CNN-v2. Different band ordering might result in different classification results. MS neighbour bands may be related by some wavelength ordering relation, but the boundary between PolSAR and MS bands is irrelevant. Therefore, it is important to perform convolutions in a manner which compensates the ordering of PolSAR-MS bands.

To provide a solution which compensates for the under-utilization of the spectral information due to PolSAR-MS band ordering, the present study proposes an ensemble approach which attempts to improve the utilization of spectral information. To present a proof of concept, this study introduces a permuted local spectral CNN model namely Perm-LS-CNN which is based on the 1D-CNN-v2 discussed in section 5.4.1. Perm-LS-CNN performs local spectral convolutions on permuted sets of the PolSAR-MS data. The permuted sets of the PolSAR-MS
data provide increased combinations of PolSAR-MS bands that are missing from using only the
original set. The idea of local spectral convolution with permuted PolSAR-MS bands can be
realized in Figure 5.4(d). For example, first filter (in yellow) convolves with bands 1, 3, and 7
whereas second filter (in blue) convolves with bands 2, 5, and 6. The proposed Perm-LS-CNN
model configurations are provided in Table 5.3. A permutation layer before the first convolution
layer is added to provide permuted sets of the original PolSAR-MS data. Although the optimal
number of permuted sets required for full utilization of the spectral information is not derived in
this study, 20 permuted sets are used as a proof of concept. Other configuration and
hyperparameters are similar to 1D-CNN-v2 and are listed in Table 5.3.

Further, based on the Perm-LS-CNN, another novel classification framework for land
cover classification using PolSAR-MS data is introduced here. This framework extends the idea
of Perm-LS-CNN to spatial context. The proposed permuted local spectral-spatial CNN or the
Perm-LSS-CNN utilizes the spatial feature extraction capability of CNNs, along with the idea of
permuted local spectral convolution. The added spatial context helps CNNs to extract more
complex and abstract features, which improves classification results. The idea of permuted
spectral-spatial convolution is presented in Figure 5.5(d). For example, a first filter (in yellow)
convolves with bands 1, 2, and 4 whereas a second filter (in blue) convolves with bands 5, 7, and
10. The proposed Perm-LSS-CNN model configurations are provided in Table 5.3. Similar to
Perm-LS-CNN, 20 permuted sets are created in the Perm-LSS-CNN model. All hyperparameters
are set as for 3D-CNN-v1 and are listed in Table 5.3.

The advantage and significance of the permuted local spectral-spatial convolutions used
in Perm-LSS-CNN over the 2D-CNN-v1 and 3D-CNN-v1 is evaluated based on a class
separability measure [175]. The separability measure or separability index (SI) is computed
using equation (3.4). The class separabilities are measured on the ground truth class samples of
Roorkee study area (refer Table 5.1) and at the first convolutional layer i.e. C1, of the 2D-CNN-
v1, 3D-CNN-v1 and the proposed Perm-LSS-CNN models. For each model and class pair, the
highest SI value achieved using equation (3.4) is given in Table 5.4. For example, in the BS-
water class pair case, the highest SI value of; 1.82 for 2D-CNN-v1, 1.99 for 3D-CNN-v1, and
2.87 for the Perm-LSS-CNN are obtained. In other example, in the tall vegetation (denoted as
TV), short vegetation (denoted as SV) class pair case, the highest SI value of; 1.46 for 2D-CNN-
v1, 1.56 for 3D-CNN-v1, and 2.77 for the Perm-LSS-CNN are obtained. Overall, it is observed
from Table 5.4 that the Perm-LSS-CNN model provides a feature space where the classes are
more significantly separable than what is achieved by the 2D-CNN-v1 and the 3D-CNN-v1
models. The experiment establishes the advantage of permuted local convolution in the spectral
dimension along with the spatial context.
<table><caption>Table 5.3 Perm-LS-CNN and Perm-LSS-CNN model configurations, hyperparameters and training settings.</caption><thead><tr><th rowspan="2">Model</th><th colspan="6">Layer</th></tr><tr><th>Input</th><th>Permutation layer (P1)</th><th>Convolutional layer 1 (C1)</th><th>Convolutional layer 2 (C2)</th><th>Fully connected layer (FC)</th><th>Output layer</th></tr></thead><tbody><tr><td>Perm-LS-CNN</td><td>Input image size = 1×13<br>Normalization = Batch<br>Normalization</td><td>Permuted sets = 20<br>Kernel size = 1×13<br>Activation = Linear</td><td>Filters = 20<br>Kernel size = 1×4<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×4<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td>Nodes = 16<br>Activation = ReLU<br>Dropout fraction = 0.1</td><td>Labels = 5<br>Activation = Softmax</td></tr><tr><td>Perm-LSS-CNN</td><td>Input image size = 3×3×13<br>Normalization = Batch<br>Normalization</td><td>Permuted sets = 20<br>Kernel size = 1×1×13<br>Activation = Linear</td><td>Filters = 20<br>Kernel size = 3×3×4<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td>Filters = 20<br>Kernel size = 1×1×4<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Pooling size = 0</td><td></td><td></td></tr><tr><td>Training parameters</td><td colspan="6">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam [211]<br>10 fold cross validation strategy is employed during training</td></tr></tbody></table>
<table><caption>Table 5.4 Separability index (SI) values for all the possible class pairs for the 2D CNN-v1, 3D CNN-v1, and ‘Perm-LSS-CNN filters.</caption><thead><tr><th rowspan="2">Class pairs</th><th colspan="3">Highest Separability Index (SI) value</th></tr><tr><th>2D CNN-v1</th><th>3D-CNN-v1</th><th>Perm-LSS-CNN</th></tr></thead><tbody><tr><td>BS vs Water</td><td>1.82</td><td>1.99</td><td>2.87</td></tr><tr><td>BS vs Built-up</td><td>2.09</td><td>2.35</td><td>2.90</td></tr><tr><td>BS vs TV</td><td>2.98</td><td>3.82</td><td>3.75</td></tr><tr><td>BS vs SV</td><td>1.29</td><td>1.69</td><td>1.87</td></tr><tr><td>Water vs Built-up</td><td>3.05</td><td>3.04</td><td>3.57</td></tr><tr><td>Water vs TV</td><td>2.59</td><td>3.75</td><td>4.38</td></tr><tr><td>Water vs SV</td><td>1.43</td><td>1.29</td><td>1.64</td></tr><tr><td>Built-up vs TV</td><td>2.81</td><td>3.64</td><td>5.47</td></tr><tr><td>Built-up vs SV</td><td>1.70</td><td>2.10</td><td>2.27</td></tr><tr><td>TV vs SV</td><td>1.47</td><td>1.56</td><td>1.77</td></tr></tbody></table>

## 5.6. Results and Discussions

### A. Quantitative analysis

The models 1D-CNN-v1, 1D-CNN-v2, 2D-CNN-v1, 3D-CNN-v1, Perm-LS-CNN, and Perm-LSS-CNN are trained and cross validated on ground truth class data points provided in Table 5.1. Classification results are shown in Figure 5.6 and Table 5.5 for the Roorkee data set, and in Figure 5.7 and Table 5.6 for the Haridwar data set. Figure 5.6(a) and Figure 5.7(a) show Google Earth images for the Roorkee and the Haridwar data sets respectively for reference purpose. Authors in [212] reports a detailed comparison of deep neural networks with SVMs for land cover classification with satellite image data. They deplore the lack of case studies where a quantitative and qualitative comparison of deep neural networks with SVMs with multi or high spectral satellite image data. Therefore, two non-CNN classifiers namely radial basis function-support vector machine (RBF-SVM), and random forest (RF) [33] are also considered for comparisons. The RBF-SVM classifier parameters settings are; (a). Regularization parameter is set to 12.7, and (b). Kernel influence parameter is set to 0.056568. The random forest classifier parameters settings are; (a) The number of features to consider when looking for the best split is set to 4, and (b) The number of trees in the forest is set to 41. From Table 5.5 and Table 5.6, it is evident that the proposed Perm-LSS-CNN provides the best performance for both the datasets with a 10-fold cross validation overall accuracy of 97.8 and quadratic kappa of 0.972 for the Roorkee data set, overall accuracy of 97.9 and quadratic kappa of 0.974 for the Haridwar data set. Individual class precision (a.k.a. user) and recall (a.k.a. producer) accuracies are also provided in Table 5.5 and Table 5.6 for the Roorkee and the Haridwar data sets. The difference between precision and recall averaged over all land cover classes (termed as AD in Table 5.5 and Table 5.6 is computed to evaluate the generalization ability and consistency of the classifiers. AD is minimum with Perm-LSS-CNN for Roorkee data and second minimum for the Haridwar
data. It is also observed that the AD is consistent for both data sets, i.e. around 3 whereas the AD
for other classifiers fluctuates with data sets. This indicates the good generalization ability of the
proposed classifier. Further, Table 5.5 and Table 5.6 also show that even if the overall accuracy
of the proposed Perm-LSS-CNN is best, the other classifiers have shown good overall accuracies
as well.

## B. Qualitative analysis

A detailed analysis of the classification performance of all the classifiers considered here
is however required to support these tabular results. A visual analysis of the classified images
from RF, RBF-SVM, 2D-CNN-v1, 3D-CNN-v1, and Perm-LSS-CNN is performed. Figure 5.8
shows the classification results on selected areas for the five models. The selected areas are
marked as A, B, C, D, and E in Figure 5.6(a). They contain the points that are taken as ground
truth.

In Figure 5.8, Figure 5.8(a) show a river and man-made riverbed markers. In Figure 5.8(b, c, d, e and f), the man-made riverbed structures are correctly classified by each method. But, in Figure 5.8(b, and c), water flow in the river is not consistent which is undesirable, whereas, in Figure 5.8(d, e and f), water flow is consistent. Figure 5.8(g) shows an agriculture field with a building located in the upper right corner. In Figure 5.8(h, and i), the building is classified as SV by RF and RBF-SVM, whereas it is correctly classified as built-up by 2D-CNN-v1, 3D-CNN-v1, and the Perm-LSS-CNN in Figure 5.8(j, k and l). Figure 5.8(m) shows a dried up river, a canal, and a bridge over the canal. The river is partially classified as SV in Figure 5.8(n, and o) whereas, it is correctly classified as BS in Figure 5.8(p, q and r). The bridge is well off the ground and oriented perpendicular to the orbit of the SAR sensor, hence exhibits double bounce scattering (only from one side). It is consistently classified as a built-up structure in Figure 5.8(p and r). The same is wrongly classified in Figure 5.8(n, o and q). Observing the above two examples suggest that RF and RBF-SVM methods are unable to capture subtle changes in land covers. Also, 3D-CNN-v1 classified the bridge as Bare Soil, missing the double bounce nature inherited in the PolSAR features. Figure 5.8(s) show a peculiar TV pattern within a golf ground. The structural and spectral identity are better preserved by Perm-LSS-CNN (refer Figure 5.8(x)) as compared to other classifiers. In Figure 5.8(t, and u), most of the ground areas (bare soil or short vegetation) is wrongly classified as water. In Figure 5.8(v), the golf ground is approximately well classified, although the shape is a partially deteriorated. Figure 5.8(y) shows a tree line. Again, the structural and spectral identity is most significantly preserved by Perm-LSS-CNN as shown in Figure 5.8(d1)).

Overall observation of Figure 5.8 suggest that the proposed Perm-LSS-CNN method
yields better classification performance both in terms of classification accuracy and
generalization ability by capturing the structural and spectral identity of land covers. In fact, classification accuracies alone cannot be used as reliable indicator for comparing classification methods because of overfitting. For example, RF and RBF-SVM provides good classification accuracies (refer Table 5.5 and Table 5.6) but lacks generalization performance (refer Figure 5.8). The generalization accuracy on well-known ground truth is the real test. The reason for the inferior generalization ability of the 2D-CNN-v1 is that the 2D convolutional filter is convolving with all the spectral features, which reduces the possibility for generation of local spectrally significant features (refer Figure 5.5(b)). The proposed Perm-LSS-CNN generates such combinations of spectral and spatial properties and hence shows improved generalization performance.

In summary, the analysis of the results in Table 5.5, Table 5.6, and Figure 5.8 indicate the improved classification performance of the proposed Perm-LSS-CNN over other classifiers considered for comparison. The discussion also indicates an increase in the generalization ability of the Perm-LSS-CNN because of the ensemble learning scenario.

<figure><img src="image_53.png" /><figcaption>Figure 5.6 Classification image: (a) Google Earth image of the Roorkee study area, India. Classified images are shown for different classifiers: (b) 1D-CNN-v1, (c) 1D-CNN-v2, (d) Perm-LS-CNN, (e) 2D-CNN-v1, (f) 3D-CNN-v1, (g) Perm-LSS-CNN, (h) RBF-SVM, and (i) RF. Colour scheme; red = built-up, light green = short vegetation, green = tall vegetation, blue = Water, and yellow = bare soil.</figcaption></figure>
<figure><img src="image_54.png" /><figcaption>Figure 5.7 Classification image: (a) Google Earth image of the Haridwar study area, India. Classified images are shown for different classifiers: (b) 1D-CNN-v1, (c) 1D-CNN-v2, (d) Perm-LS-CNN, (e) 2D-CNN-v1, (f) 3D-CNN-v1, (g) Perm-LSS-CNN, (h) RBF-SVM, and (i) RF. Colour scheme; red = built-up, light green = short vegetation, green = tall vegetation, blue = water, and yellow = bare soil.</figcaption></figure>
<table><caption>Table 5.5 Classification performance of various classifiers considered for PolSAR-MS Roorkee dataset. *AD is the difference between precision and recall, averaged over all the land cover classes considered. BS = bare soil, TV = tall vegetation, SV = short vegetation, BU = built-up, W = water.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">OA (%)</th><th rowspan="2">Kappa</th><th colspan="5">User accuracy (%)</th><th colspan="5">Producer accuracy (%)</th><th rowspan="2">AD*</th></tr><tr><th>BS</th><th>TV</th><th>SV</th><th>BU</th><th>W</th><th>BS</th><th>TV</th><th>SV</th><th>BU</th><th>W</th></tr></thead><tbody><tr><td>1D-CNN-v1</td><td>95</td><td>0.938</td><td>94.5</td><td>100</td><td>85.2</td><td>100</td><td>97.5</td><td>93.4</td><td>92.4</td><td>97</td><td>97.5</td><td>96.3</td><td>4.9</td></tr><tr><td>1D-CNN-v2</td><td>95.2</td><td>0.939</td><td>96.4</td><td>100</td><td>84.5</td><td>100</td><td>97.1</td><td>93</td><td>91.8</td><td>96.6</td><td>98.1</td><td>98.4</td><td>5.3</td></tr><tr><td>Perm-LS-CNN</td><td>95.7</td><td>0.946</td><td>97</td><td>100</td><td>86.5</td><td>100</td><td>97.1</td><td>94.4</td><td>92.1</td><td>94.2</td><td>98.1</td><td>98.4</td><td>4.3</td></tr><tr><td>2D-CNN-v1</td><td>95.3</td><td>0.941</td><td>96.4</td><td>100</td><td>83.9</td><td>100</td><td>98.4</td><td>94.1</td><td>92.7</td><td>98</td><td>96.9</td><td>96.3</td><td>5.8</td></tr><tr><td>3D-CNN-v1</td><td>96.3</td><td>0.933</td><td>98.8</td><td>100</td><td>84.5</td><td>100</td><td>99.6</td><td>95.6</td><td>91</td><td>100</td><td>98.7</td><td>98.4</td><td>6.1</td></tr><tr><td>Perm-LSS-CNN</td><td>97.8</td><td>0.972</td><td>99.7</td><td>99.6</td><td>91.1</td><td>100</td><td>99.3</td><td>94.3</td><td>98.3</td><td>99.6</td><td>98.7</td><td>99</td><td>3.3</td></tr><tr><td>RBF-SVM</td><td>94.6</td><td>0.93</td><td>94.6</td><td>99.3</td><td>90.2</td><td>96.2</td><td>94</td><td>92.6</td><td>94</td><td>93.5</td><td>98</td><td>96.7</td><td>3.2</td></tr><tr><td>RF</td><td>91.7</td><td>0.932</td><td>86.8</td><td>96.8</td><td>88.2</td><td>96.8</td><td>93.1</td><td>92.1</td><td>97.1</td><td>65</td><td>88.6</td><td>90.6</td><td>7.9</td></tr></tbody></table>

<table><caption>Table 5.6 Classification performance of various classifiers considered for PolSAR-MS Haridwar dataset. *AD is the difference between precision and recall, averaged over all the land cover classes considered. BS = bare soil, TV = tall vegetation, SV = short vegetation, BU = built-up, W = water.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">OA (%)</th><th rowspan="2">Kappa</th><th colspan="5">User accuracy (%)</th><th colspan="5">Producer accuracy (%)</th><th rowspan="2">AD*</th></tr><tr><th>BS</th><th>TV</th><th>SV</th><th>BU</th><th>W</th><th>BS</th><th>TV</th><th>SV</th><th>BU</th><th>W</th></tr></thead><tbody><tr><td>1D-CNN-v1</td><td>94.1</td><td>0.925</td><td>100</td><td>100</td><td>70.2</td><td>96.6</td><td>99.3</td><td>97.1</td><td>87.1</td><td>98.9</td><td>95</td><td>99.3</td><td>9.2</td></tr><tr><td>1D-CNN-v2</td><td>95</td><td>0.937</td><td>100</td><td>100</td><td>73.2</td><td>99.1</td><td>99.3</td><td>99.2</td><td>87.8</td><td>100</td><td>95.9</td><td>99.3</td><td>8.6</td></tr><tr><td>Perm-LS-CNN</td><td>96.1</td><td>0.95</td><td>100</td><td>100</td><td>77.8</td><td>100</td><td>100</td><td>100</td><td>89.2</td><td>100</td><td>97.5</td><td>100</td><td>7.1</td></tr><tr><td>2D-CNN-v1</td><td>94.7</td><td>0.933</td><td>100</td><td>100</td><td>70.9</td><td>99.6</td><td>100</td><td>97.8</td><td>89.2</td><td>100</td><td>92.9</td><td>99.3</td><td>9.9</td></tr><tr><td>3D-CNN-v1</td><td>96.5</td><td>0.955</td><td>100</td><td>100</td><td>80.1</td><td>99.1</td><td>100</td><td>98.5</td><td>90.3</td><td>99</td><td>100</td><td>100</td><td>6.2</td></tr><tr><td>Perm-LSS-CNN</td><td>97.9</td><td>0.974</td><td>100</td><td>100</td><td>89.3</td><td>99.1</td><td>100</td><td>99.2</td><td>96</td><td>99.1</td><td>96.7</td><td>100</td><td>3.4</td></tr><tr><td>RBF-SVM</td><td>96.1</td><td>0.95</td><td>98.5</td><td>100</td><td>86.2</td><td>95.8</td><td>97.2</td><td>97.8</td><td>92.7</td><td>95.7</td><td>96.6</td><td>100</td><td>4.3</td></tr><tr><td>RF</td><td>94.4</td><td>0.928</td><td>97</td><td>98.6</td><td>86.2</td><td>91.6</td><td>95.1</td><td>97.7</td><td>95.9</td><td>84.9</td><td>91.6</td><td>100</td><td>1.9</td></tr></tbody></table>
<figure><img src="image_55.png" /><figcaption>Figure 5.8 Results from different classification techniques applied on the Roorkee data set. The areas include data points collected as ground truth.</figcaption></figure>
## 5.7. Conclusion

This chapter presents a critical analysis of the potential and performance of the one, two, and three dimensional CNNs, where dimensionality refers to the convolved features: spectral, spatial or both. These CNNs are applied to land cover classification with multisensor (PolSAR-MS) satellite image data. Two novel models are introduced, where local filters in either spectral domain (Perm-LS-CNN) or spatial-spectral domains (Perm-LSS-CNN) are applied after a permutation layer. The permutation layer generates an ensemble of band combinations, which are drawn upon by the CNN in order to generate powerful features for land cover classification. Classification results demonstrate the improved generalization ability of these models compared to both alternative CNN architectures and other machine learning models (SVM and random forests). Perm-LSS-CNN also fully uses both PolSAR and multispectral information, improving results compared to other popular methods for PolSAR-MS data based land cover classification. A separability measure is used for the model significance evaluation. Separability analysis states that the Perm-LSS-CNN transforms the input data space into a feature space where separability among classes increases. This increase in separability is achieved due to the inclusion of permuted local spectral convolution along with the local spatial convolution which is not available in the 1D-CNN-v1, 2D-CNN-v1, or 3D-CNN-v1 CNN models. A fine analysis on ground truth data provides an extra qualitative assessment which complements the quantitative classification results. These qualitative assessments demonstrate that the proposed Perm-LSS-CNN model is able to generalize much better while keeping high classification accuracies.
# NOVEL CNN, RNN, AND CNN-RNN MODELS FOR MULTISENSOR SATELLITE TIME SERIES IMAGE BASED CROP CLASSIFICATION

Crop classification is an important task in many crop monitoring applications. Satellite image based monitoring provides an easy, techno-economical, reliable, stable and fast approaches to crop monitoring tasks. Numerous techniques have been reported for satellite time series image based crop classification. However, with the availability of more and more high quality satellite images from new and powerful sensors, which are increasing by the day, it has become a challenge for techniques to keep up in terms of performance, high dimensional operability, scalability, and generalizability. The new technologies must have a unified framework to be able to process plethora of multi-resolution, multisensor, and high dimensional satellite time series images. In the context, CNNs and recurrent neural networks (RNNs) are the most recent and trending deep neural network frameworks for satellite time series image based agriculture crop monitoring applications. Both, CNN and RNN models have been developed in, standalone and synergistic modes for agriculture crop classification by many researchers. However, the CNN and RNN models are still young, promising, and growing techniques in the satellite image proceeding community especially in multisensor satellite time series image processing and therefore further exploration of the potential of these models is motivating. Also, certain limitations such as presence of mixed land cover scenario still exist. Mixed land cover scenario in crop classification refers to the presence of other land covers (built-up, forests, or grasslands) in-between and around the croplands in the study region (e.g. in India). Therefore, it is still a challenge for existing DNN based techniques to synergistically exploit the spatial, spectral, and temporal information of a crop present in the multimodal and/or multisensor satellite time series images and provide accurate and reliable crop type information in mixed land cover scenarios.

This chapter presents a study in which an attempt is made; to analyse conventional, and to develop novel, CNN and RNN models based agriculture crop classification models with multisensor/multimodal satellite time series imagery. This chapter is an extension of chapter 5 in a sense that the concept of permuted spectral input is used here with time series images and how DNN models exploit the satellite images with spectral, spatial, and temporal information.
## 6.1. Introduction

Recent reports indicate the initiatives taken by authorities worldwide towards development of sustainable agriculture science and management strategies and policies [213]–[215]. Satellite image based crop monitoring is a key technology in acquiring crop information such as crop type, or crop phenology[14], [216]–[218]. Satellite image based crop monitoring is easy, fast, reliable, techno-economical, and efficient [219]–[221]. In the recent years, India is strongly leaning towards the use of satellite images for crop monitoring and crop information extraction [222], [223] but still needs to go far.

Optical and PolSAR satellite images have been successfully used for agricultural applications in India such as crop mapping/classification [224], crop yield forecasting [225], crop stress and drought stress monitoring [226], mapping of crop type/cycle changes [227], crop phenology assessment [228], and biomass estimation [229] in the last two to three decades. The use of satellite images in these applications is increasing day by day and new applications such as precision agriculture [230] are being discovered as well. In particular, with the availability of open source high spatial and/or spectral and/or temporal resolution multisensor satellite images via programs such as the Copernicus Sentinel satellite series, the performance and efficiency of the aforementioned applications has increased. Also, new and advanced studies in crop monitoring have surfaced [72], [231], [232]. Recent studies on Indian croplands are also using Sentinel series data for crop monitoring [233], [234].

Moreover, time series satellite images are rich source of crop phenology information and have become popular, efficient, and crucial for agriculture crop monitoring including crop identification and classification. For agriculture, it has been shown that the temporal evolution of the signal measured via satellite imaging enables a better separation between crop types classes [235]. Satellite time series image based agriculture crop classification is one of the most cost-effective, reliable, and successful framework. Also, different image modalities have been successfully used in crop classification. Satellite PolSAR imaging is a weather independent technology and time series PolSAR images have been successfully used in crop classification applications[236]–[240]. On the other hand, time series optical imagery provides crop phonology information via spectral-temporal features. Time series optical imagery has also been successfully used for crop classification in the recent past [241], [242] The reduced number of optical data based crop classification studies is because of the effect of clouds on optical satellite images. PolSAR and optical image modalities have also been used synergistically for crop classification to improve the performance of methodologies [53], [67], [243]–[248].
In context to satellite time series image processing techniques for crop classification, numerous techniques have been proposed and/or utilized in the past. The popular techniques are maximum likelihood classifier [238], [239], [243], [245], neural networks [239], [243], [248], decision trees [239], [243], random forest [67], [238], [241], [242], [246], [247], temporal fusion classification [53], and hierarchical classification strategy [244]. It is observed that random forest, maximum likelihood classifier, and neural networks are the most popular methods utilized in satellite time series image based crop classification applications. However, with the availability of more and more high quality satellite images from new and powerful sensors, which are increasing by the day, it has become a challenge for these techniques to keep up in terms of performance, high dimensional operability, scalability, and generalizability. Therefore, it is important to level up the technologies. The technologies must be able to process plethora of multi-resolution, multisensor, and high dimensional satellite time series images.

In this regard, recently, deep learning with deep neural networks is one of the most promising techniques. Numerous successful approaches have been reported in the recent years which are based on deep neural networks for satellite time series image applications. In particular, CNNs and recurrent neural networks (RNNs) are the most popular deep neural network types utilized in satellite time series image based agriculture crop monitoring applications. Further, novel CNN and RNN models have also been developed and utilized in both, standalone and synergistic, modes for agriculture crop classification by many researchers. A detailed review on popular CNN and RNN models for agriculture crop monitoring and classification applications is presented in chapter 2 (refer section 2.3). The review indicated to the success and significance of deep neural network models in satellite time series image based crop classification applications. However, the technology is still young and growing, and there is scope for much more. In particular, the DNN model based studies for time series satellite image processing discussed in section 2.3 are focused more on either the spectral, spatial, or temporal characteristic or at most more focussed on two characteristics at a time ignoring the third characteristic. For example, some CNN and RNN studies focused on spectral features more than temporal or spatial features during classification which lead to under-utilization of temporal and spatial features during crop classification. Similarly, some other studies focused more on temporal and spatial characteristics and less on spectral characteristic which again lead to under-utilization of spectral features of the crops. This imbalance, in priority, requires investigation on the subject and demands novel CNN and RNN model based approaches that can address this issue. To this day, only few studies focussed on utilizing all three characteristics have been reported but a trend is underway. Other major limitation during crop classification is the presence of mixed land cover scenario i.e.
presence of other land covers (built-up, forests, or grasslands) in-between and around the croplands in the study region (e.g. in India). Therefore, it is still a challenge for existing DNN based techniques to synergistically exploit the spatial, spectral, and temporal information of a crop present in the multimodal and/or multisensor satellite time series images and provide accurate and reliable crop classification.

The above arguments motivate towards new studies focused on using multisensor satellite time series images, especially optical and PolSAR, for crop classification and development of novel CNN and RNN models which focus on the judicious exploitation of the spectral, temporal and spatial characteristics of crops to achieve improved classification results.

In the study presented in this chapter, an attempt is made; to analyse the conventional CNN and RNN models, and to develop novel CNN and RNN (in standalone as well as synergistic modes) models for agriculture crop classification with multisensor/multimodal satellite time series (MSTS) imagery. For this purpose, two sub tasks are set in this study, *first*, critical analysis of conventional CNN and RNN models for MSTS images based crop classification, and *second*, development of novel CNN and RNN models for MSTS images based crop classification. In the first sub task i.e. *sub task 1*, conventional CNN and RNN models are analysed and evaluated for the task of MSTS images based crop classification. 1D, 2D, and 3D CNNs, RNNs, and combinations of these are considered for analysis. Then, based on the critical analysis of the working and performance of these conventional models over MSTS images based crop classification, few novel models are proposed and developed in *sub task 2*.

The novel models involve permuted spectral and/or spatial convolutions and/or bi-directional recurrent layers. The Bidirectional RNNs capture the temporal profile and the permuted spectral and/or spatial CNNs capture the spectral and/or spatial profile of a particular crop. Bidirectional strategy in RNNs enables the influence of the past, present, and future inputs on the current output. Bidirectional RNNs are beneficial in time series images based crop classification because crop classification can utilize the bidirectional process i.e. classification result can be observed from any time node. Also, the permuted spectral band stacking strategy developed in chapter 5 is also used in this study and presumed to improve the significance of the spectral information present in multisensor satellite time series images. In particular, a synergistic utilization of permuted spectral-spatial CNNs and Bidirectional RNNs helps in better exploitation of the multisensor satellite time series images.

This chapter is organized as follows. Section 6.2 provides conceptual background on CNNs and RNNs. Further, section 6.3 provides the information on the experiment setup i.e. study areas, datasets, and ground truth. Section 6.5 and 6.7 provide the implementation details on the
conventional and novel CNN-RNN models Section 6.8 discusses on results obtained using these models and its analysis. Finally, section 6.9 concludes the study.

## 6.2. Theoretical Background

Basic knowledge of CNNs and RNNs is required in this chapter as CNNs can exploit the spectral and/or spatial information and RNNs can exploit the temporal information, present in the MSTS images. The brief concepts of CNNs are presented in chapter 3 (refer section 3.2.2) and chapter 5 (refer section 5.2.1). In addition, the basic concepts on RNNs are discussed in this section.

### 6.2.1. Recurrent neural networks

Recurrent neural networks, also known as RNNs, are a class of neural networks that allow previous outputs to be used as inputs while having hidden states. The RNNs have the ability to process input of any length while keeping the model size unchanged. RNNs also share weights across time along with space. Therefore, the output depends on historical information. A typical architecture of a RNN is shown in Figure 6.1.

For each timestep t, the activation $a^{<t>}$ and the output $y^{<t>}$ are expressed as follows [249].

$$a^{<t>} = g_1(W_{aa}a^{<t-1>} + W_{ax}x^{<t>} + b_a) \qquad (6.1)$$

$$y^{<t>} = g_2(W_{ya}a^{<t>} + b_y) \qquad (6.2)$$

where $W_{aa}$, $W_{ax}$, $W_{ya}$, $b_a$, and $b_y$ are coefficients that are shared temporally, $g_1$ is an activation function (usually a hyperbolic tangent), and $g_2$ is a softmax activation function [249].

<figure><img src="image_56.png" /><figcaption>Figure 6.1 Typical architecture of a recurrent neural network [250].</figcaption></figure>
<figure><img src="image_57.png" /><figcaption>Figure 6.2 Signal flow graph in a simple recurrent cell [250].</figcaption></figure>

## A. Recurrent cell

A recurrent cell is the smallest unit in RNNs which is able to memorize activation representing historical information. There are different variants of recurrent cells. The popular recurrent cell architectures are discussed here.

### Simple recurrent cell

Architecture and signal flow of a simple recurrent cell is shown in Figure 6.2. The cell takes $x^{<t>}$ i.e. current input and $a^{<t-1}>$ i.e. activation from previous cell, as inputs and outputs $a^{<t>}$ i.e. activation for next RNN cell and $y^{<t>}$ i.e. prediction from current state. Although the cell is able to capture short historical information but is unable to retain longer historical information. There are two popular variants of the basic RNN cell that are able to capture longer historical relations within a signal.

### Long short term memory (LSTM) recurrent cell

Recurrent neural networks with simple recurrent cell have problems associated with vanishing and exploding gradients [251]. This is a common problem in neural network updates where successive multiplication by the weight matrix W is inherently unstable; it either results in the gradient disappearing during backpropagation, or in it blowing up to large values in an unstable way. In RNNs, this type of instability is the direct result of successive multiplication with the (recurrent) weight matrix at various time-stamps. One way of viewing this problem is that a neural network that uses only multiplicative updates is good only at learning over short sequences, and is therefore inherently endowed with good short-term memory but poor long-term memory [251]. This problem can be addressed with the use of a long short-term memory
(LSTM) cell for longer memory. The operations inside the LSTM cell are designed to have fine-grained control over the data written into this long-term memory. Figure 6.3 shows the internal architecture and signal flow in a LSTM recurrent cell. The LSTM recurrent cell has a new variable $c^{<t>}$, which stands for memory cell at time t. The memory cell will provide memory to remember activations which can be used during predictions at later time steps.

The governing equations inside a LSTM recurrent cell are as follows [252].

$$ \tilde{c}^{<t>} = \tanh(W_c[\Gamma_r * a^{<t-1>} , x^{<t>}] + b_c) \quad (6.3) $$

$$ c^{<t>} = \Gamma_u * \tilde{c}^{<t>} + \Gamma_f * c^{<t-1>} \quad (6.4) $$

$$ a^{<t>} = \Gamma_0 * c^{<t>} \quad (6.5) $$

Where, $\tilde{c}^{<t>}$ is a candidate for being a current memory cell state which is based on, relevance gate i.e. $\Gamma_r$ status, previous cell activation i.e. $a^{<t-1>}$, and current input i.e. $x^{<t>}$. $W_c$ and $b_c$ are trainable parameters. $c^{<t>}$ is the current memory cell state which is based on, update gate i.e. $\Gamma_u$ status, forget gate i.e. $\Gamma_f$ status, candidate for memory current cell state i.e. $\tilde{c}^{<t>}$, and old memory cell state i.e. $c^{<t-1>}$. $a^{<t>}$ is current cell activation which is based on, output gate i.e. $\Gamma_0$, and current memory cell state i.e. $c^{<t>}$ [252].

## Gated recurrent unit (GRU) recurrent cell

The Gated Recurrent Unit (GRU) can be viewed as a simplification of the LSTM, which does not use explicit cell states. Another difference is that the LSTM directly controls the amount of information changed in the hidden state using separate forget and output gates. On the other hand, a GRU uses a single reset gate to achieve the same goal. The architecture and signal flow in a GRU recurrent cell is shown in Figure 6.4.

The governing equations inside a GRU cell are as follows [253].

$$ \tilde{c}^{<t>} = \tanh(W_c[\Gamma_r * a^{<t-1>} , x^{<t>}] + b_c) \quad (6.6) $$

$$ c^{<t>} = \Gamma_u * \tilde{c}^{<t>} + (1 - \Gamma_u) * c^{<t-1>} \quad (6.7) $$

$$ a^{<t>} = c^{<t>} \quad (6.8) $$

Where, all the parameters have usual meanings. The GRU cell replaces the forget gate $\Gamma_f$ with $1 - \Gamma_u$ and does not utilizes the output gate $\Gamma_0$ [253].
<figure><img src="image_58.png" /><figcaption>Figure 6.3 A LSTM recurrent cell [250].</figcaption></figure>

<figure><img src="image_59.png" /><figcaption>Figure 6.4 A GRU recurrent cell [250].</figcaption></figure>

## B. Learning in RNNs

The key concept behind learning in RNNs is the ‘backpropagation through time’ algorithm. In backpropagation through time, the negative logarithms of the softmax probability of the correct predictions at the various time-stamps are aggregated to create the loss function. If the softmax probability at time t is represented as $p^{<t>}$, the loss L is defined as follows [252].

$$L = -\sum_{t=1}^{T} \log(\hat{p}^{<t>}) \qquad (6.9)$$

## C. RNN architectures

RNN models are mostly used in the fields of time series signal processing. Overall, there are four variants of RNN architectures.

i. One to one

ii. One to many

iii. Many to one

iv. Many to many

The many to one and many to many versions of RNNs shown in Figure 6.5 are popularly used for crop classification in many studies.
<figure><img src="image_60.png" /><figcaption>Figure 6.5 RNN architectures used popularly in satellite time series image based crop classification: (a) Many to one, and (b) Many to many RNN architectures [250].</figcaption></figure>

In the current study, multiple versions of RNNs, alone and in combination with CNNs are explored. Details of the CNN and RNN models developed and utilized in the study are provided in further sections.

## 6.3. Experiment Setup

This section provides information about the resources collected and utilized during the study. It includes information of study areas, datasets, and ground truth selected or collected for the study.

### 6.3.1. Study area

The study area selected for study is a cropland region in the Haridwar district of the Uttarakhand, India where common seasonal crops are practiced. The region is considered based on accessibility to ground truth collection, crop season, and generic land cover distribution. The geographical extents of the region are provided in Table 6.1. Figure 6.6 shows the Google Earth image of the study area. The region is a land parcel which is managed by local farmers. Seasonal crops and sugarcane are grown in this region. The study area is approximately a 6km by 7km land parcel, covering 42km² of area. The region also has forests, grasslands, deciduous tree plantations along with various crops in the study area. This makes the region more realistic environment and mixed class scenario for algorithm testing. The presence of multiple land covers will evaluate the generalization ability of the classifiers. Areas marked in the image (1, 2, 3, and 4) are subsets of study area used for qualitative and visual evaluation of the classification performance of classifiers developed in the study.

<table><caption>Table 6.1 Geographical extent of study area.</caption><thead><tr><th colspan="2">Coordinates (upper left)</th><th colspan="2">Coordinates (lower right)</th></tr><tr><th>Latitude</th><th>Longitude</th><th>Latitude</th><th>Longitude</th></tr></thead><tbody><tr><td>29.84183</td><td>77.02115</td><td>29.78843</td><td>78.09080</td></tr></tbody></table>
<figure><img src="image_61.png" /><figcaption>Figure 6.6 Google Earth image of the study area (GE Image Copyright 2018). Subset areas selected for ground truth via visual inspection. 1: GTI-1, 2: GTI-2, 3: GTI-3, and 4: GTI-4 (highlighted in yellow boxes).</figcaption></figure>

### 6.3.2. Dataset

Multisensor satellite images are considered for crop classification in this study. Sentinel-1 PolSAR and Sentinel 2 multispectral satellite sensor images are used. Sentinel-1 C band dual pol SAR images are available at 10 m spatial resolution and at a revisit time of 6 days (12 if pass direction is considered). Similarly, Sentinel-2 multispectral images are available at 10m, 20m, and 60m spatial resolutions and at a revisit time period of 5 days Both sensor datasets are freely available for public use. Based on the growing season of wheat, Sentinel-1 ground range detected (GRD) and Sentinel-2 L1C satellite time series images are collected every month throughout the wheat growing season for year 2017-18. Acquisition dates of each month for both sensors are listed in Table 6.2. Also,

Table 6.3 lists the specifications of bands considered from Sentinel-1 and Sentinel-2. Overall, 10 multispectral bands and 2 PolSAR channels are considered for the study. Also, 7 time steps from October 2017 to April 2018 are considered for the study. Figure 6.7 shows a false colour composite of the dataset for the study area and highlights the varied crop types.

<table><caption>Table 6.2 Sentinel-1 and Sentinel-2 data acquisition dates.</caption><thead><tr><th rowspan="2">Month</th><th colspan="2">Acquisition dates</th></tr><tr><th>Sentinel-1</th><th>Sentinel-2</th></tr></thead><tbody><tr><td>October</td><td>3/10/2017</td><td>7/10/2017</td></tr><tr><td>November</td><td>20/11/2017</td><td>21/11/2017</td></tr><tr><td>December</td><td>2/12/2017</td><td>1/12/2017</td></tr><tr><td>February</td><td>12/2/2018</td><td>9/2/2018</td></tr><tr><td>March</td><td>8/3/2018</td><td>6/3/2018</td></tr><tr><td>March</td><td>1/4/2018</td><td>31/3/2018</td></tr><tr><td>April</td><td>13/4/2018</td><td>15/4/2018</td></tr></tbody></table>
<table><caption>Table 6.3 Band specifications of Sentinel-1 and Sentinel-2.</caption><thead><tr><th>Sensor</th><th>Band selection</th><th>Spatial resolution (meters)</th></tr></thead><tbody><tr><td rowspan="2">Sentinel-1</td><td>VH</td><td rowspan="2">10</td></tr><tr><td>VV</td></tr><tr><td rowspan="10">Sentinel-2</td><td>Band 2: Blue</td><td rowspan="4">10</td></tr><tr><td>Band 3: Green</td></tr><tr><td>Band 4: Red</td></tr><tr><td>Band 8: NIR</td></tr><tr><td>Band 5: Vegetation Red Edge</td><td rowspan="5">20</td></tr><tr><td>Band 6: Vegetation Red Edge</td></tr><tr><td>Band 7: Vegetation Red Edge</td></tr><tr><td>Band 8A: Vegetation Red Edge</td></tr><tr><td>Band 11: SWIR-1</td></tr><tr><td>Band 11: SWIR-2</td><td></td></tr></tbody></table>

<figure><img src="image_62.png" /><figcaption>Figure 6.7 False colour composite from the Sentinel-2 time series dataset. Colour scheme; red: Band 4 (November 21, 2017), green: Band 4 (February 9, 2018), blue: Band 4 (March 31, 2018). Image subset areas selected for ground truth via visual inspection (highlighted in black boxes). 1 = GTI-1, 2 = GTI-2, 3 = GTI-3, and 4 = GTI-4.</figcaption></figure>

### 6.3.3. Ground truth

Based on the study regions and crop season selection, wheat is the major crop grown in the area. However, the region also grows sugarcane throughout the year. The region also has evergreen forests, deciduous commercial plantations such as Salicaceae (Popular Eucalyptus), grasslands, and villages which must be considered during crop classification as it is a usual landscape (mixed
land cover) in India. Therefore, overall, six classes are considered namely: wheat, sugarcane, forests, deciduous plantation mostly 'Popular' plantations, grassland, and built-up. The ground truth samples are either measured directly on the terrain or are collected via visual inspection from the dataset. Summary of ground truth samples collected for all considered classes is provided in Table 6.4. Few images from the ground truth survey are also shown in Figure 6.8. Four subset areas from the study area are selected for visual and qualitative evaluation. Figure 6.6 and Figure 6.7 shows the four marked areas. The subset areas are termed as ground truth image (GTI) 1 through 4 i.e. GTI-1, GTI-2, GTI-3, and GTI-4.

<table><caption>Table 6.4 Summary of ground truth samples collected during ground truth survey.</caption><thead><tr><td>Class</td><td>Ground truth samples</td></tr></thead><tbody><tr><td>Wheat</td><td>300</td></tr><tr><td>Sugarcane</td><td>300</td></tr><tr><td>Forest</td><td>300</td></tr><tr><td>Popular</td><td>300</td></tr><tr><td>Grassland</td><td>300</td></tr><tr><td>Built-up</td><td>300</td></tr></tbody></table>

<figure><img src="image_63.png" /><figcaption>Figure 6.8 Sample images of different land covers collected during ground truth survey. (a) Sugarcane cropland, (b) Wheat cropland, (c) Grasslands, (d) Popular plantations.</figcaption></figure>
### 6.3.4. Preprocessing

Both Sentinel-1 and Sentinel-2 datasets require preprocessing before classification. Sentinel-1 GRD datasets are terrain corrected and calibrated. Terrain correction is performed using the SRTM 1 arc-second digital elevation model (DEM) available for public use. Speckle filtering is not performed as speckle noise is already suppressed in GRD products. Sentinel-2 L1C datasets are atmospherically corrected using the 'sen2cor' processor [254]. Further the 20 m spatial resolution bands in the bottom of atmosphere (BOA) Sentinel-2 datasets are resampled to 10 m using the 'super-resolution' approach proposed in [255]. Overall, 2 polarimetric channels i.e. VH, and VV from the terrain corrected and calibrated Sentinel-1 and 10 multispectral bands from resampled bottom-of-atmosphere Sentinel-2 datasets are selected and stacked together to form a PolSAR-multispectral multisensor dataset. Overall, 7 multisensor datasets are created corresponding to the 7 time steps. Further, the 7 multisensor datasets are stacked along the temporal dimension creating a multisensor time series dataset. This PolSAR-multispectral multisensor satellite time series (MSTS) image dataset is utilized for crop classification.

## 6.4. Generalized DNN Framework for Satellite Time Series Images based Crop Classification

Although, numerous studies reported on satellite time series image processing have developed and utilized different DNN models, these models can be consolidated into a generalized framework. Figure 6.9 depicts a generalized DNN framework for time series satellite image based crop classification. The framework is composed of CNNs (1D, 2D or, 3D), RNNs (Simple, LSTMs or, GRUs), and fully connected (FC) modules. Different DNN architectures can be created using either one, more than one or, all three modules. For example, 1D-CNNs can be used as spectral or temporal feature extractors with or without FC layers. Further, these 1D features can be utilized with a classifier (e.g. a softmax layer, SVM or any other) to achieve classification. Alternatively, the 1D CNNs can also be integrated with RNNs as, the former acts as spectral feature extractors and the latter acts as temporal feature extractors. The classification stage can be a softmax layer.

Similarly, 2D-CNNs can replace 1D-CNNs in the aforementioned framework. The only difference is that 2D-CNNs provide spatial features in contrast to 1D-CNNs (which provide only spectral features). If 2D-CNNs are used with RNNs then spatial-temporal features are generated which can be used with a classifier to achieve classification. 1D- and 2D-CNNs can also be used in parallel and cascade fashion to obtain both spectral and spatial or, spectral-spatial features
respectively. These spatial and spectral or spectral-spatial features can be used with RNNs to obtain, spectral, spatial, and temporal features, spectral-spatial-temporal features or, any combination of these. These strategies lead to full exploitation of time series images with good spatial and spectral information.

However, the cascade fashion of 1D- and 2D-CNNs can be replaced with 3D-CNNs. 3D-CNNs have the ability to extract spectral and spatial features simultaneously from satellite images. 3D-CNNs can be used with or without FC layers prior to classification. Also, 3D-CNNs can be utilized with RNNs to capture all three i.e. spectral, spatial, and temporal information from satellite time-series images. The classification stage is usual.

The 1D-CNNs can be employed at pixel level for classification whereas the 2D- and 3D-CNNs can only be employed at image patch or image window level. Per-pixel classification is not an option with 2D- and 3D-CNNs. Also, the 1D-, 2D-, and 3D-CNNs have to be employed in time distributed fashion before using RNNs as temporal feature extractors. RNNs can also be used in standalone mode to extract temporal features directly from satellite time series images at pixel level.

The CNNs, RNNs, and the FC modules can be used in numerous other fashions to achieve classification. The next few sections present few such possibilities and configurations for multisensor satellite time series image based crop classification.

Although the current study used multisensor satellite time series (MSTS) images, however, the architectures discussed above are independent of thee notion of multisensor. This means that all or any other variation of the aforementioned DNN architectures will work with multisensor satellite time series images in the similar fashion.

<figure><img src="image_64.png" /><figcaption>Figure 6.9 A generic framework for DNN models for satellite time series images based crop classification.</figcaption></figure>
## 6.5. Sub Task 1: Conventional CNN and RNN models for multisensor satellite time series image based crop classification

In this section, various conventional CNN and RNN models, in standalone or synergistically, are used for multisensor satellite time series (MSTS) image based crop classification. The section provides information about the architecture, and training parameters of the models used.

### 6.5.1. 1D-CNN model for MSTS images based crop classification

The present study focuses on the analysis of the performance of 1D-CNNs for MSTS image based crop classification. In the 1D-CNN model namely 1D-CNN-v1 presented here, one dimensional convolution can be performed in either the spectral, or temporal dimension. Since convolution in the spectral dimension is already covered in chapter 5, here, one dimensional convolution is performed in the temporal dimension. The kernel size of the filter in the convolutional layer i.e. C1 layer is equal to the temporal depth of MSTS image i.e. 7. One dimensional temporal convolution is similar to one dimensional spectral convolution in the implementation aspect (refer Figure 5.4(b)). However, the implications and interpretations are clearly different. One dimensional local temporal convolution is not considered as the temporal depth is low.

The 1D-CNN-v1 architecture and hyper-parameter settings are listed in Table 6.5. Rectified linear unit (ReLU) and softmax activation functions are used in the intermediate and output layers respectively. A dropout strategy is considered for regularization and a dropout fraction value of 0.2 for C1 is set. Pooling is not considered because it hinders the full usability of the temporal variation. The loss function used is the categorical cross entropy function. It optimizes the classification accuracy based on probabilities of each class during classification. The learning rate and the number of epochs are set to 0.01 and 200 respectively. It is to be noted here that the models (conventional or novel) discussed in this chapter have similar hyperparameters.

### 6.5.2. 2D-CNN model for MSTS images based crop classification

The input MSTS image data has 1 spectral, 2 spatial and 1 temporal dimension. It is not possible to realize 2D-CNNs, in standalone mode, to process this 4-dimensional data directly. Therefore, 2D-CNN model is not explored further.
### 6.5.3. 3D-CNN model for MSTS images based crop classification

The 3D-CNN model proposed here operates (window based convolution) along the time and spatial dimensions. The spectral dimension corresponds to the channels in the input data during the convolution.

In the 3D-CNN model namely 3D-CNN-v1, the size of the three dimensional kernel of the filter in the convolutional layer i.e. C1 layer is set equal to the temporal depth of MSTS image i.e. 7 in the temporal dimension and is set equal to 3×3 in the spatial dimension. Three dimensional temporal-spatial convolution is similar to spectral-spatial convolution shown in Figure 5.5(c) with the difference that spectral dimension is replaced with temporal dimension. Localized convolution is not considered in the temporal dimension as the temporal depth is already low. The present study also focuses on the analysis of the performance of 3D-CNNs for MSTS image based crop classification. The 3D-CNN-v1 architecture and hyper-parameter settings are listed in Table 6.5.

### 6.5.4. RNN model for MSTS images based crop classification

A conventional RNN model is considered here for analysis. The input to this RNN model namely RNN-v1 is a 7×12 matrix where 7 indicates the temporal depth (time stamps) and 12 indicates to the spectral depth in the multisensor image data in each time stamp. Since, RNNs do not accommodate spatial contextual information, per-pixel approach is considered here. A simple LSTM unit is selected as recurrent cell. Since the temporal depth is 7, seven LSTM units constitute the R1 recurrent layer. The architecture is used in a many-to-many fashion. It means the input to the recurrent layer is more than one dimensional (in this case, 7) and output from the recurrent layer is more than one dimensional (in this case, 7). Also, the flow of activation from each RNN cell in the recurrent layer is considered in a bidirectional manner. It means that the activation from particular recurrent cell can move to next recurrent cell (next time stamp, usual) and previous recurrent cell (previous time stamp). This helps in the crop classification and its significance is discussed in the results and discussion section later in this chapter. Further a FC layer with 20 nodes and output layer with 6 outputs is set.

The RNN-v1 architecture and hyper-parameter settings are listed in

Table 6.6. Hyperbolic tangent and softmax activation functions are used in the LSTM recurrent cells for activation and output layer respectively. A dropout strategy is considered for regularization and dropout fraction values of 0.1 and 0.1 for R1 and FC are set respectively.
<table><caption>Table 6.5 Architecture and hyperparameter settings of the 1D-CNN-v1, 2D-CNN-v1, and 3D-CNN-v1 models for MSTS images based crop classification.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Layer</th></tr><tr><th>Input</th><th>Convolutional layer (C1)</th><th>Fully connected layer (FC)</th><th>Output layer</th></tr></thead><tbody><tr><td>1D-CNN-v1</td><td>Input data size = 7×12 (time, spectral)<br>Normalization = Batch Normalization</td><td>Filters = 32<br>Kernel size = 7 (time)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td><td rowspan="2">Not used</td><td rowspan="2">Labels = 6<br>Activation = Softmax</td></tr><tr><td>3D-CNN-v1</td><td>Input data size = 7×3×3×12 (time, spatial, spatial, spectral)<br>Normalization = Batch Normalization</td><td>Filters = 32<br>Kernel size = 7×3×3 (time, spatial, spatial)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0</td></tr><tr><td>Training parameters</td><td colspan="4">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam [211]<br>10 fold cross validation strategy is employed during training</td></tr></tbody></table>

<table><caption>Table 6.6 Architecture and hyperparameter settings of the RNN-v1 model for MSTS images based crop classification.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Layer</th></tr><tr><th>Input</th><th>Recurrent layer (R1)</th><th>Fully connected layer (FC)</th><th>Output layer</th></tr></thead><tbody><tr><td>RNN-v1</td><td>Input data size = 7×12 (time, spectral)<br>Normalization = Batch Normalization</td><td>Filters = 10<br>Kernel size = 7 (time)<br>Activation = Tanh<br>Dropout fraction = 0.1<br>Pooling size = 0<br>Bidirectional: Yes<br>Input-output format: Many-to-many</td><td>Nodes= 20<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Time-distributed: Yes</td><td>Labels = 6<br>Activation = Softmax<br>Time-distributed: Yes</td></tr><tr><td>Training parameters</td><td colspan="4">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam [211]<br>10 fold cross validation strategy is employed during training</td></tr></tbody></table>

### 6.5.5. CNN-RNN models for MSTS images based crop classification

In this section, CNN and RNN architectures are used synergistically to form CNN-RNN models for crop classification with MSTS images. The previous standalone CNN or RNN models are not able to completely exploit the spectral, spatial, and temporal information simultaneously. At best, the standalone 3D-CNN model i.e. 3D-CNN-v1 is able to exploit spectral-spatial information or temporal-spatial simultaneously but not all three simultaneously. Alternatively, the standalone
RNN model i.e. RNN-v1 could benefit from the temporal information but is unable to extract
spatial or spectral features. Therefore, a synergistic utilization of CNN and RNN models is
straightforward beneficial. In the novel models developed in the next few sections, the CNNs
are used to extract spectral and/or spatial features and RNNs are used to extract temporal features.
The CNN architectures are used prior to RNN architectures in a time-distributed fashion to
preserve the temporal information of the MSTS images.

6.5.5.1. 1D-CNN-RNN model for MSTS images based crop classification

At first, a 1D-CNN architecture is used here to extract spectral features as the temporal
information is exploited using a RNN architecture. The kernel size of the filter in the
convolutional layer i.e. C1 layer is equal to the spectral depth of MSTS image i.e. 12. One
dimensional spectral convolution is shown in Figure 5.4(b).

Then, a conventional RNN model is considered. Since, 1D-CNNs or RNNs cannot
accommodate spatial contextual information, per-pixel approach is considered here. A simple
LSTM unit is selected as recurrent cell similar to the RNN-v1 model. Since the temporal depth
is 7, seven LSTM units constitute the R1 recurrent layer. Here again, the RNN architecture is
used in a many-to-many fashion. It means the input to the RNN layer is more than one
dimensional (in this case, 7) and output from the RNN layer is more than one dimensional (in
this case, 7). Also, the flow of activation from each recurrent cell in the recurrent layer is
considered in a bidirectional manner. It means that the activation from particular recurrent cell
can move to next recurrent cell (next time stamp, usual) and previous RNN cell (previous time
stamp). This helps in the crop classification and the significance is discussed in the results and
discussion section later in this chapter. Further a FC layer with 20 nodes and output layer with
6 outputs is set.

The consolidated CNN-RNN model is termed as 1D-CRNN-v1 hereafter in this chapter.
The architecture and hyper-parameter settings are listed in Table 6.7. ReLU, hyperbolic tangent
(Tanh) and softmax activation functions are used in the convolutional & FC layers, LSTM
recurrent cells, and output layer respectively. A dropout strategy is considered for regularization
and dropout fraction values of 0.2, 0.1 and 0.1 for C1, R1 and FC are set respectively.
<table><caption>Table 6.7 Architecture and hyperparameter settings of CNN-RNN based models for MSTS images based crop classification.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">Layer</th></tr><tr><th>Input</th><th>Convolutional layer (C1)</th><th>Recurrent layer (R1)</th><th>Fully connected layer (FC)</th><th>Output layer</th></tr></thead><tbody><tr><td>1D-CRNN-v1</td><td>Input data size = 7×12 (time, spectral)<br>Normalization: Batch<br>Normalization</td><td>Filters = 32<br>Kernel size = 12 (spectral)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Time-distributed: Yes</td><td>Filters = 10<br>Kernel size = 7 (time)<br>Activation = Tanh<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Bidirectional: Yes<br>Input-output format: Many-to-many</td><td>Nodes= 20<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Time-distributed: Yes</td><td>Labels = 6<br>Activation = Softmax<br>Time-distributed: Yes</td></tr><tr><td>2D-CRNN-v1</td><td>Input data size = 7×3×3×12 (time, spatial, spatial, spectral)<br>Normalization: Batch<br>Normalization</td><td>Filters = 32<br>Kernel size = 3×3 (spatial)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Time-distributed: Yes</td><td>Filters = 10<br>Kernel size = 7 (time)<br>Activation = Tanh<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Bidirectional: Yes<br>Input-output format: Many-to-many</td><td>Nodes= 20<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Time-distributed: Yes</td><td>Labels = 6<br>Activation = Softmax<br>Time-distributed: Yes</td></tr><tr><td>3D-CRNN-v1</td><td>Input data size = 7×3×3×12 (time, spatial, spatial, spectral)<br>Normalization: Batch<br>Normalization</td><td>Filters = 32<br>Kernel size = 3×3×12 (spatial, spatial, spectral)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Time-distributed: Yes</td><td>Filters = 10<br>Kernel size = 7 (time)<br>Activation = Tanh<br>Dropout fraction = 0.2<br>Pooling size = 0<br>Bidirectional: Yes<br>Input-output format: Many-to-many</td><td>Nodes= 20<br>Activation = ReLU<br>Dropout fraction = 0.1<br>Time-distributed: Yes</td><td>Labels = 6<br>Activation = Softmax<br>Time-distributed: Yes</td></tr><tr><td>Training parameters</td><td colspan="5">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam [211]<br>10 fold cross validation strategy is employed during training</td></tr></tbody></table>
### 6.5.5.2. 2D-CNN-RNN model for MSTS images based crop classification

The 2D-CNN architecture is used here to extract spatial features as the temporal features are capture by the RNN architecture. Since, 2D-CNNs can accommodate spatial contextual information, patch based approach is considered here. The kernel size of the filter in the convolutional layer i.e. C1 layer is a 3×3 window. The 2D-CNN architecture is employed in a time-distributed fashion in order to preserve the temporal information. Then, a conventional RNN model is considered which is identical to the RNN model used in 1D-CRNN-v1. This consolidate CNN-RNN model is termed as 2D-CRNN-v1 hereafter in this chapter. The architecture and hyper-parameter settings are listed in Table 6.7.

### 6.5.5.3. 3D-CNN-RNN model for MSTS images based crop classification

The 3D-CNN architecture is used here to extract spectral-spatial features as the temporal features are capture by the RNN architecture. Since, 3D-CNNs can also accommodate spatial contextual information, patch based approach is considered here. The kernel size of the filter in the convolutional layer i.e. C1 layer is a 3×3×12 window. The 3D-CNN architecture is employed in a time-distributed fashion in order to preserve the temporal information.

After the convolutional layer, a conventional RNN model is considered similar to the RNN models used in 1D-CRNN-v1 and 2D-CRNN-v1. This consolidate CNN-RNN model is termed as 3D-CRNN-v1 hereafter in this chapter. The architecture and hyper-parameter settings are listed in Table 6.7.

## 6.6. The Concept of Permuted Local Spectral Convolutions and its Synergistic Use with RNNs for MSTS images based Crop Classification

The concept of permuted local spectral convolution is introduced and discussed in chapter 5 (refer section 5.5). Chapter 5 also establishes the significance and advantage of permuted local spectral convolutions over the conventional spectral convolutions or local spectral convolutions. In this chapter, the concept is analysed in presence of two rather one complementary information i.e. spatial and temporal. Usually, the DNN models developed for high dimensional time series image data, focus on temporal and/or spatial information more than the spectral information present. However, in context to satellite based earth observation images, the spectral information is as important as the spatial and temporal or even more. Therefore, the idea of utilizing the spectral information more efficiently along with the spatial and temporal information motivates to development of novel DNN models.
Therefore, in this chapter, the study and analysis of the performance of synergistic use of CNNs with permuted local spectral convolution and RNNs for MSTS based crop classification is carried out. It is postulated here that the RNNs might take advantage of permuted local spectral convolutions in MSTS image applications. The more generalized features obtained using the permuted local spectral convolutions are more abstract in nature and the CNN-RNN models built on top of this concept might improve the performance of these models.

## 6.7. Sub Task 2: Novel CNN and RNN Models with Permut ed Local Spectral Convolutions for MSTS Image based Crop Classification

In this section, various novel CNN and RNN models are developed for MSTS image based
crop classification which use the concept explained in the previous section. This section
provides information about the architecture, and training parameters of the novel models
developed.

6.7.1. 1D-CNN model with permuted local spectral convolution for MSTS Images based crop classification

In this 1D-CNN model namely Perm-1D-CNN-v1, first, a permutation layer which generates 10 permuted versions of spectral bands as explained in chapter 5 (refer section 5.5). Then, the 10 permuted versions are stacked together to form an ensembled spectral band stack. This stack of bands (120, 10 permutations with 12 bands in each permutation set) is used as input in the further models.

After the ensembled input is generated, one dimensional convolution layers are used.
Since, the temporal depth is unaltered, one dimensional convolution in the temporal dimension
will yield similar results as 1D-CNN-v1, it is insignificant to replicate the model. However, the
spectral depth has now increased from 12 to 120, local spectral convolutions should yield more
and may be different spectral features. Therefore, a series of spectral convolutions are
performed over the spectral dimension with the help of one dimensional convolutional filters.

Three convolutional layers in time-distributed fashion are employed. The kernel size of the filters in the three convolutional layers i.e. C1, C2, and C3 layers are set to 7, 21, and 63 respectively. The kernel sizes are set based on experiments. Then, a FC layer with 50 nodes is used. Finally, an output layer with 6 classes is used. The FC and the output layer are also employed in time-distributed fashion.
The Perm-1D-CNN-v1 architecture and hyper-parameter settings are listed in Table 6.8. Rectified linear unit (ReLU) and softmax activation functions are used in the convolutional & FC layers and output layers respectively. A dropout strategy is considered for regularization and dropout fraction values of 0.2 for C1, C2, and C3 are set. A dropout fraction value of 0.1 is set for the FC layer. The loss function used is the categorical cross entropy function. It optimizes the classification accuracy based on probabilities of each class during classification. The learning rate and the number of epochs are set to 0.01 and 200 respectively.

### 6.7.2. 2D-CNN model with permuted input for MSTS images based crop classification

Since the input MSTS image data has 1 spectral, 2 spatial and 1 temporal dimension, it is not possible for 2D-CNNs to process this 4-dimensional data simultaneously and directly. Therefore, the permuted local spectral convolution version of the 2D-CNN model is not explored further.

### 6.7.3. 3D-CNN model with permuted local spectral convolution for MSTS images based crop classification

The 3D-CNN model proposed here namely Perm-3D-CNN-v1 operates window based convolution along the spectral and spatial dimensions in a time distributed fashion.

The first layer is a permutation layer similar to the Perm-1D-CNN-v1 with the difference in the spatial dimension. The Perm-3D-CNN-v1 uses a 3×3 window in the spatial dimensions input in contrast to Perm-1D-CNN-v1 which uses per-pixel approach. After the ensembled input is generated, a series of three dimensional convolutions are employed in a time distributed fashion. The three dimensional convolutions operate over the spectral and spatial dimensions.

Then, three convolutional layers are employed. The kernel size of the filters in the three convolutional layers i.e. C1, C2, and C3 layers are set to 3×3×7, 1×1×21, and 1×1×63 respectively where 3×3 represents the spatial extents and 7, 21, or 63 represents the spectral extents of the convolutional filters. The kernel sizes are set based on experiments. Then, a FC layer with 50 nodes is used. Finally, an output layer with 6 classes is used. The FC and the output layer are also employed in time-distributed fashion. The Perm-3D-CNN-v1 architecture and hyper-parameter settings are listed in Table 6.8.
<table><caption>Table 6.8 Architecture and hyperparameter settings of CNN models with permuted spectral convolution for MSTS images based crop classification.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">Layer</th></tr><tr><th>Input</th><th>Permutation layer</th><th>Convolutional layer (C1, C2, C3)</th><th>Fully connected layer (FC)</th><th>Output layer</th></tr></thead><tbody><tr><td>Perm-1D-CNN-v1</td><td>Input data size = 7×12 (time, spectral)<br>Normalization = Batch<br>Normalization</td><td>Permutated sets = 10<br>Kernel size = 12 (spectral)<br>Activation = Linear<br>Time-distributed: Yes</td><td>Filters = 20, 40, 60<br>Kernel size = 7, 21, 63 (spectral)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Time-distributed: Yes</td><td>Nodes= 50<br>Activation = ReLU<br>Dropout fraction = 0.1</td><td>Labels = 6<br>Activation = Softmax</td></tr><tr><td>Perm-3D-CNN-v1</td><td>Input data size = 7×3×3×12 (time, spatial, spectral)<br>Normalization = Batch<br>Normalization</td><td>Permutated sets = 10<br>Kernel size = 12 (spectral)<br>Activation = Linear<br>Time-distributed: Yes</td><td>Filters = 20, 30, 40<br>Kernel size = 3×3×7, 1×1×21, 1×1×63 (spatial, spatial, spectral)<br>Activation = ReLU<br>Dropout fraction = 0.2<br>Time-distributed: Yes</td><td>Time-distributed: Yes</td><td></td></tr><tr><td>Training parameters</td><td colspan="5">Loss function = Categorical cross entropy<br>Learning rate = 0.01<br>Number of epochs = 200<br>Optimizer = Adam<br>10 fold cross validation strategy is employed during training</td></tr></tbody></table>