# A Mutual Information-Based Self-Supervised Learning Model for PolSAR Land Cover Classification

Bo Ren, Yangyang Zhao, Biao Hou, Jocelyn Chanussot, Licheng Jiao

► To cite this version:

Bo Ren, Yangyang Zhao, Biao Hou, Jocelyn Chanussot, Licheng Jiao. A Mutual Information-Based Self-Supervised Learning Model for PolSAR Land Cover Classification. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (11), pp.9224-9237. 10.1109/TGRS.2020.3048967. hal-03429688

HAL Id: hal-03429688

https://hal.science/hal-03429688v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Mutual Information-Based Self-Supervised
Learning Model for PolSAR Land
Cover Classification

Bo Ren<sup>1,*</sup>, Member, IEEE, Yangyang Zhao, Biao Hou, Member, IEEE, Jocelyn Chanussot<sup>1,†</sup>, Fellow, IEEE, and Licheng Jiao<sup>1,‡</sup>, Fellow, IEEE

**Abstract**—Recently, deep learning methods have attracted much attention in the field of polarimetric synthetic aperture radar (PolSAR) data interpretation and understanding. However, for supervised methods, it requires large-scale labeled data to achieve better performance, and getting enough labeled data is a time-consuming and laborious task. Aiming to obtain a good classification result with limited labeled data, we focus on learning discriminative high-level features between multiple representations, which we call mutual information. As PolSAR data have multi-modal representations, there should have strong similarity between multi-modal features of the same pixel. In addition, each pixel has its own unique geocoding and scattering information. Hence, every pixel has great difference from other pixels in a specific representation space. Based on the above observations, this article proposes a mutual information-based self-supervised learning (MI-SSL) model to learn an implicit representation from unlabeled data. In this article, the self-supervised learning idea is first applied to PolSAR data processing. Furthermore, a reasonable pretext task, which is suitable for PolSAR data, is designed to extract mutual information for classification tasks. Compared with the state-of-the-art classification methods, experimental results on four PolSAR data sets demonstrate that our MI-SSL model produces impressive overall accuracy with fewer labeled data.

**Index Terms**—Land cover classification, mutual information, polarimetric synthetic aperture radar (PolSAR), self-supervised learning (SSL).

## I. INTRODUCTION

POLARIMETRIC synthetic aperture radar (PolSAR) is an active sensor by transmitting and receiving backscattering

Manuscript received July 21, 2020; revised November 23, 2020; accepted December 21, 2020. This work was supported in part by the National Natural Science Foundation of China under Grant 62001355, Grant 61671350, Grant 61771379, and Grant 61836009; in part by the Foundation for Innovative Research Groups of the National Natural Science Foundation of China under Grant 61621005; in part by the Key Research and Development Program in Shaanxi Province of China under Grant 2019ZDLGY03-05; and in part by the China Postdoctoral Science Foundation Funded Project under Grant 2018M633468. (Corresponding author: Bo Ren.)

Bo Ren, Yangyang Zhao, Biao Hou, and Licheng Jiao are with the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an 710071, China (e-mail: asdwer2046@126.com; yangzyyk@163.com; avcodec@163.com; lchjiao@mail.xidian.edu.cn).

Jocelyn Chanussot is with Inria, CNRS, Grenoble INP, LJK, Université Grenoble Alpes, 38000 Grenoble, France (e-mail: jocelyn.chanussot@gipsa-lab.grenoble-inp.fr).

Color versions of one or more figures in this article are available at https://doi.org/10.1109/TGRS.2020.3048967.

Digital Object Identifier 10.1109/TGRS.2020.3048967

waves from land covers. It can efficiently enhance the ability to acquire information from targets. Compared to conventional SAR systems, PolSAR can receive more information from the targets to describe properties. Exploiting these polarization characteristics sufficiently has a strong relationship with PolSAR image interpretation. For PolSAR land cover classification task, the main method at present is to assign each pixel to a specific category, which can be applied to sea monitoring, fine agriculture, geological mapping, government public decision-making, and so on [1].

Many researchers have proposed a series of PolSAR classification methods. These methods fall into the following three categories. The first category of methods uses the inherent physical scattering characteristics to classify the data. Most of these methods are based on target decomposition mechanisms, including H/A/α decomposition [2], Pauli decomposition [3], Freeman decomposition [4], Huynen decomposition [5], Cameron decomposition [6], and Krogager decomposition [7]. The second methods mainly use the statistical characteristics of PolSAR data. It has been shown that the polarimetric coherency matrix and covariance matrix follow the complex Wishart distribution. Therefore, the Wishart distribution-derived distance is proposed for PolSAR land cover classification [8]. These methods based on the Wishart distance measure are independent of polarization and have good robustness. The third category is machine learning. These methods learn a classifier by studying the characteristics of the data structure and reorganizing the knowledge content [9]–[31]. Among these methods, deep learning-based models have shown outstanding performance in feature extraction for PolSAR data.

Therefore, various classifiers based on neural networks, such as real-valued convolutional neural networks (RV-CNN) [16], complex-valued convolutional neural networks (CV-CNN) [17], Wishart deep belief networks (WDBN) [18], Wishart deep stacking networks (WDSN) [19], and graph semi-supervised learning (GSS) [32], have been proposed. As we know, the basic framework of these methods is the convolutional neural network (CNN) model. Although these methods can achieve high classification performance, they still have three disadvantages.

1. These methods just employ different forms of CNN without considering scattering mechanisms and the multi-modal PolSAR data representation. They only take
the single polarimetric feature representation as input for
the networks.

2) These methods need great amounts of labeled data sets, which is a great expense of time and manpower.

3) Training with the limited ground truth of PolSAR data can easily result in overfitting. This means that these classification models with poor robustness in the case of few labeled data.

PolSAR data own multiple raw features for learning deep classification models. Specifically, the coherency matrix and covariance matrix are widely used for feature representation. The components from target decompositions are commonly used in classification tasks. Besides, some features from natural images have been used in classification tasks, such as color features [33] and texture features [34]. Multiple feature representations can describe the target more completely from various perspectives. Furthermore, a reasonable combination of these features can effectively reduce the impact of single description loss and improve the classification accuracy [33]. However, multiple features of the same sample also means that there is information redundancy. If multiple features are just cascaded in classification tasks, redundant information can even reduce the classification accuracy. This inspires us to remove redundant information from multiple feature representation and extract mutual information for PolSAR land cover classification.

The main purpose of this article is to improve the accuracy of PolSAR land cover classification with limited labeled data. Since PolSAR data have multiple raw features and rich information on targets, we seek to extract high-level feature representation from the PolSAR feature space in a self-supervised learning (SSL) model. SSL utilizes inherent attributes and relationships in raw data to obtain high-level semantic representation, and this process does not require the participation of manually annotated data. The extracted mutual information provides a prior knowledge for CNN classification model, which greatly reduces the labeled data usage and prevents the overfitting of the training model. In this article, a two-stream network is established for a self-supervised learning procedure to learn representation from the multi-modal feature space. After this, the learned parameters can be transferred to build a neural network classifier to improve classification performance. The crucial contributions of this article are highlighted as follows.

1) The idea of SSL is first proposed for PolSAR data processing. The high-level semantic feature hidden between multiple modalities of PolSAR data is extracted without manually annotation data. When dealing with PolSAR data classification tasks with limited label data, the learned information can effectively improve the classification accuracy and avoid overfitting.

2) We designed a reasonable pretext task to extract mutual information from multiple modalities of raw PolSAR data. Furthermore, the learned high-level features can provide prior knowledge for the CNN classification model to achieve better performance. Besides, we increased the number of layers of the network based

on the work [16], and the classification accuracy of the model is further improved.

3) To validate the superiority of our mutual information-based self-supervised learning (MI-SSL) model, the experiments were carried out on four benchmark PolSAR data sets. When the number of available labeled samples is not enough, our MI-SSL model gets higher classification accuracy than some state-of-the-art classification methods. Besides, comparison of the MI-SSL model with a random initialization CNN model demonstrates that the designed pretext task can effectively extract high-level features.

The rest of this article is organized as follows: A brief review of the CNN classifier for PolSAR data and self-supervised learning is briefly given in Section II. The details of the proposed MI-SSL model are described in Section III. Benchmark PolSAR data and experimental analysis are shown in Section IV. Finally, we give our conclusion and forecast the prospects for future research work in Section V.

## II. RELATED WORK

### A. CNN for PolSAR

Deep learning is a new widely used data analysis technique in the machine learning community. In recent years, deep learning has made outstanding achievements in computer vision tasks. One of the important reasons is that deep learning can automatically extract effective feature representation and avoid the complex process of manually designing the desired features. Through a series of convolution and pooling procedures, it can efficiently reduce the dimension of a large number of image data. In this process, the parameters in the network are learned, and the high feature representations are automatically extracted. Some network models based on CNN have achieved high classification accuracy in the image benchmark data sets, such as AlexNet [35], GoogLeNet [36], and the very deep convolutional networks proposed by Visual Geometry Group [37].

The basic CNN model consists of convolution, activation, and pooling. When dealing with the classification task, we take the feature map of CNN output as the input of the full connection layer network. In this way, we connect the full connection layer to complete the mapping from the raw data to the labeled set. Convolution is the basic operation in CNN. Through convolution kernel filtering, local features of the input data are extracted. After convolution, the bias and activation functions need to be added. The role of the activation function is to provide nonlinear modeling ability for the network. Pooling is a kind of down-sampling operation. Its main goal is to remove redundant information, compress features, and reduce the resolution of the feature map. The common pooling methods include average pooling and maximum pooling.

More information and features need to be effectively exploited in PolSAR image classification by CNN, which is different from the optical image classification tasks. In recent years, many researchers have applied the CNN network model to PolSAR land cover classification and achieved success
along this path. The feature representations of PolSAR images obtained by the CNN model are more discriminative and have strong generalization ability. Zhou et al. [16] first proposed the CNN classifier for PolSAR land cover classification. Furthermore, Zhang et al. [17] extended the application of CNN to the complex domain. This change makes CNN more suitable for PolSAR data and improves the classification accuracy. Meanwhile, Wen et al. [28] applied the Wishart distance measure to construct the loss function for CNN. It further combined the characteristics of PolSAR data and improved the classification performance.

With the advancement of graphics processing units (GPUs), CNN-based models have attracted more and more attention from researchers. Although the methods mentioned above have achieved amazing results, they all need a lot of manually annotated data to train the classifier. For instance, the works in [18] and [19] use 5% labeled pixels to build a training data set for the CNN model. Meanwhile, the works in [16], [17], and [20] exploit 10% labeled pixels to train the classifier. However, when the labeled data are limited, CNN often has difficulty in obtaining more robust results. In order to deal with the aforementioned issues, we consider how to make full use of the multi-modal representations of PolSAR data to train better classifiers with fewer labeled data, meanwhile enhancing the robustness of the classification model.

## B. Self-Supervised Representation Learning

CNN achieves high classification performance in supervised learning, which needs large-scale manually annotated data. It is time- and labor-consuming to obtain sufficient annotation data. In some cases, especially in the field of remote sensing data analysis, the label information of the data cannot even be obtained. In order to solve this problem, some self-supervised learning methods have been proposed to extract feature representation without the participation of labeled information. Self-supervised learning is a special kind of unsupervised method that uses the inherent characteristics of data as supervision signals, such as color, texture, context information, and so on. These methods define a pretext task and objective function related to data characteristics. In the process of training neural networks to complete pretext tasks, feature representation can be obtained. For example, some methods are designed to complete the missing information of data themselves [38]–[40]. Some methods scramble the original structure of the data and then reconstruct the data to obtain the feature representation [41]–[44]. There are also some methods to build pretext tasks through adversarial training, such as [45] and [46].

Self-supervised representation learning for image classification tasks mainly consists of the following three parts: First, a reasonable pretext task related to data structure is designed, which is the key to the whole self-supervised learning procedure. The quality of pretext tasks determines the extraction ability of feature representation. Second, the high-level features can be obtained by training neural network with the designed pretext task and labeled data are not required in this process. Finally, the extracted feature representations are

used for classification tasks through transfer learning. With a self-supervised representation learning process, the classifier is more robust and has stronger generalization ability.

Compared with optical images, PolSAR data not only have the characteristics of the natural image, such as texture, structure, and color information, but also have polarimetric properties, including backscattering and target decomposition information. In addition, the high-dimensional nature of PolSAR data provides much useful information for designing pretext task. The self-supervised learning method can effectively extract the mutual information from these raw features and learn the high-level semantic feature representations of the target, which will help us to better understand the PolSAR data and provide a more solid foundation for PolSAR classification and target recognition tasks.

## III. METHODOLOGY

In this section, we present the detailed implementation of our MI-SSL model for PolSAR land cover classification. We emphatically introduce the pretext task designed for PolSAR data. In addition, the mutual information model is first introduced based on a multiple raw PolSAR feature space. The detailed method is as follows.

### A. Raw Feature Extraction and Analysis of PolSAR Data

The data acquired by PolSAR can be represented by a 2 × 2 Sinclair matrix that reflects the polarimetric scattering characteristics of each resolution unit on the ground. This polarization scattering matrix S is defined to characterize a given target from the radar equation. The scattering matrix is formed as

$$S = \begin{bmatrix} S_{HH} & S_{HV} \\ S_{VH} & S_{VV} \end{bmatrix}. \qquad (1)$$

In the case of reciprocal backscattering $S_{HV} = S_{VH}$, we can get the coherency matrix

$$T = \langle k_p k_p^\top \rangle = \begin{bmatrix} T_{11} & T_{12} & T_{13} \\ T_{21} & T_{22} & T_{23} \\ T_{31} & T_{32} & T_{33} \end{bmatrix} \qquad (2)$$

where “$k_p$” refers to the scatter vector and “$\top$” denotes the conjugate transpose. Moreover, the covariance matrix C is also used to express PolSAR data, which has a linear relation with the coherency matrix.

In general, raw PolSAR data can be represented in many ways. One is to simply use the elements of the above matrices or their transformations. Specifically, the coherency matrix T can be extracted to form a 9-D real number vector a, that is, $a = [T_{11}, T_{22}, T_{33}, \text{real}(T_{12}), \text{real}(T_{13}), \text{real}(T_{23}), \text{image}(T_{12}), \text{image}(T_{13}), \text{image}(T_{23})]$. It is one of the most commonly used features in PolSAR classification task. Another way to describe the PolSAR data is from those components of polarimetric target decomposition. In addition, we can obtain the color characteristics of PolSAR data from pseudocolor image that is acquired by components in the target decomposition mechanism.

In order to verify the generalization of the designed MI-SSL model, we choose three typical decomposition methods from
<table><caption>TABLE I
SELECTED RAW FEATURES FOR OUR METHODS</caption><thead><tr><th>Features</th><th>Description</th></tr></thead><tbody><tr><td>T<sub>11</sub>, T<sub>22</sub>, T<sub>33</sub>, real(T<sub>12</sub>), real(T<sub>13</sub>), real(T<sub>23</sub>), image(T<sub>12</sub>), image(T<sub>13</sub>), image(T<sub>23</sub>)</td><td>Elements from the coherency matrix</td></tr><tr><td>|a|<sup>2</sup>, |b|<sup>2</sup>, |c|<sup>2</sup></td><td>Pauli decomposition</td></tr><tr><td>H, A, α</td><td>H/A/α decomposition</td></tr><tr><td>P<sub>hs</sub>, P<sub>hd</sub>, P<sub>hv</sub></td><td>Freeman decomposition</td></tr></tbody></table>

polarization decomposition mechanism to construct three groups of input data. Pauli decomposition [3], H/A/α decomposition [2], and Freeman decomposition [4] are selected as the source of raw features for the MI-SSL model. The reasons for choosing these decomposition methods are as follows. Pauli decomposition provides the reliable visualization features of PolSAR data. Compared with other features, the images of the Pauli decomposition have a high degree of contrast and identification. As for Freeman decomposition, we extract the features of the scattering mechanism, which is based on the physical property of radar scattering, not a purely mathematical construction. H/A/α decomposition is a statistical model based on eigenvalue decomposition whereby statistical features are obtained in this method. Combined with the coherency matrix **T**, three pairs of raw features are formed as input to our self-supervised learning model.

In conclusion, we select three typical features, including visual features, physical features, and statistical features, to form three types of input data. The three selected raw features are listed in Table I. According to the different features used here, we can subdivide MI-SSL as follows: MI-SSL with Pauli decomposition (MI-SSL-P), MI-SSL with Freeman decomposition (MI-SSL-F), and MI-SSL with H/A/α decomposition (MI-SSL-H).

## B. Self-Supervised Model for PolSAR Land Classification

The PolSAR data have abundant inherent characteristics that can be represented by multi-modal features. This enlightens us with the idea of using these raw data to get more useful mutual information for classification tasks. The self-supervised learning paradigms provide an effective way to extract hidden knowledge from large volumes of raw PolSAR data.

For self-supervised learning, given a set of raw PolSAR data $X$, for each $x_i \in X$, there is a $y_i$ automatically generated from the raw data themselves. $y_i$ is generated in the process of designing pretext tasks and does not need to rely on any label information. For a set of training sets with $N$ samples $D = \{x_i\}_{i=1}^N$, the object function of the self-supervised learning model can be defined as follows:

$$ \text{loss}(D) = \min \frac{1}{N} \sum_{i=1}^{N} \text{loss}(x_i, y_i). \quad (3) $$

Fig. 1 shows the flow of a self-supervised representation learning model for PolSAR data. Specifically, the green box

<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of our self-supervised learning feature extraction model for PolSAR land classification.</figcaption></figure>

is the structure of the self-supervised pipeline. In this stage, no labeled data are required. First and foremost, a pretext task is designed for the CNN model. At the same time, as designing pretext tasks, supervisory signals for the training network are generated from the raw data. Furthermore, the neural network model is trained to solve pretext tasks without label information. Meanwhile, the high-level semantic feature representation of data is produced in this process. After this, the learned representations are transferred to the supervised classified network fine-tuning (the blue box). Labeled data are required for this process. In the process of classification, a convolutional neural network initialized by advanced semantic features can reduce the number of labeled data required, improve the classification accuracy of CNN, and reduce overfitting.

## C. Mutual Information Extraction

Each pixel of PolSAR data can be expressed as a $3 \times 3$ coherency matrix **T**, which is commonly used in the literatures for PolSAR land cover classification. Also, features derived from the target decomposition mechanism, texture, color, and spatial relationship between pixels are commonly used. However, it is difficult for neural networks to deal with multiple features. There is redundant information between multiple PolSAR features, which may reduce classification performance.

While people focus on the study of the physical characteristics of each PolSAR data set, few people pay attention to the information contained in the multi-modal feature representations. The high-level representation between multiple features with discrimination is crucial to PolSAR land cover classification, and we call it mutual information. Usually, CNN is trained on only one of these features, which contains limited information and cannot completely represent all the characteristics of the data. The methods of multiple feature fusion and multiple feature selection can make up for the deviation caused by the training set. In order to learn how to eliminate the unbiased representation of unnecessary changes, we suggest training a self-supervised representation learning model from multiple features of PolSAR data.

Given a PolSAR data set, we can get more than one representation that contains semantically overlapping but
different information. Based on multiple-domain learning, a better PolSAR data representation can be learned in a self-supervised way. In other words, a multi-modal PolSAR feature space can represent the characteristics of PolSAR data more comprehensively. Hence, it is feasible to extract mutual information from the feature space to acquire better representation of PolSAR data. The advantages of this method are to enrich the diversity of data, reduce the built-in deviation of a single data set, utilize shared information, and allow knowledge migration across multiple fields. Based on this observation, we can extract mutual information that cannot be obtained from any one field alone.

## D. Pretext Task and Loss Function

Based on the above description of self-supervised learning idea, our proposed methods learn representation by extracting mutual information from raw PolSAR data. The design of our pretext task is based on the following observation facts. In PolSAR data, each sample contains the polarization scattering information, geocoding information, and other information of the polarimetric target, so the representation between the samples should have some differences. Wu et al. [47] pointed out that the CNN model can not only extract the difference among classes, but also capture discriminate feature from individual instances. This enlightens us to extract mutual information by distinguishing different instances. In addition, the unique characteristics of PolSAR data are multi-modal representation. The single representation is noisy and incomplete, so we use multiple modalities to depict the data. Hence, the representations between each modality should have some similarity. CNN is employed to extract the similarity mutual information.

In the multi-modal representation of PolSAR data, the 9-D real number vector obtained from the coherency matrix T is denoted as u. The representation vector obtained from polarization decomposition mechanisms are denoted as v. Two pixels are randomly selected, the ith pixel and the jth pixel, to form a set of input data pairs, denoted as $(u_i, v_i)$ and $(u_j, v_j)$. Two different CNN models $f(\cdot)$ and $g(\cdot)$ are used to map the input data to an embedded space. The details of two models are described in Section III-E. After being mapped to another subspace, two modal representations of a sample maintain similarity, while different samples are far from each other. In order to quantify the similarity, we introduce the cosine distance d into the embedded space. It can be expressed by

$$d(\mathbf{m}, \mathbf{n}) = 1 - \frac{\mathbf{m} \cdot \mathbf{n}}{|\mathbf{m}| \times |\mathbf{n}|} \quad (4)$$

where m and n are vector representations in embedded space. And · is the dot product operator.

First, we extract the similarity mutual information between two modal representations of the same instance. Similarity means that the two representations are close to each other in the embedding space. In other words, the similarity is measured by forcing the distance d between two representations as small as possible. As described in the following formulation,

the similarity loss $L_1$ is given by

$$L_1 = \frac{1}{2} [d(f(u_i), g(v_i)) + d(f(u_j), g(v_j))] \quad (5)$$

Second, we extract the difference mutual information between the two instances of the same modal representation. The difference means that the two representations are far away from each other in the embedding space. In other words, the difference between two representations is measured by forcing the distance between two representations as large as possible. As described in the following formulation, the difference loss $L_2$ is given by

$$L_2 = -\frac{1}{2} [d(f(u_i), f(u_j)) + d(g(v_i), g(v_j))] \quad (6)$$

To sum up, two loss functions are weighted and summed to get the final loss function. For two types of losses, we use the same weight to measure loss contribution

$$L_{\text{total}} = L_1 + L_2. \quad (7)$$

## E. Architecture of Self-Supervised Learning Model

Advanced deep CNN models, such as VGG-16 [37] and CaffeNet [48], have made outstanding achievements in image classification tasks. This shows that, to some extent, the more layers in the neural network, the better feature representation are obtained. Meanwhile, as computing power increases, it becomes easier to train deep neural networks. Therefore, compared with RV-CNN, it is reasonable to apply deep CNN to PolSAR land cover classification.

The structure and parameters of the two CNN models are the same except for the input layer. The input layer of $f(\cdot)$ is a 9-D vector, while the input layer of $g(\cdot)$ is a 3-D vector. The remaining settings for the network are as follows. The architecture mainly contains five convolutional layers interleaved with three max poolings. The shape of the input data is the same as in [49]. A sliding window of 15 × 15 is applied to generate a significant number of patches as input data. In detail, the proposed network adopts six-layer architecture, including five convolutional layers and one fully connected layer. For each convolutional layer, there are 96, 256, 384, 384, and 256 filter parameters, respectively. Besides, an identically sized 3 × 3 convolution kernel is employed. In order to reduce the complexity of network operation, three pooling layers with the 3 × 3 max pooling are adopted. Meanwhile, the rectified linear unit is used as the activation function. In addition, dropout with a ratio of 0.5 is used in two fully connected layers to prevent overfitting. All self-supervised pretext task learning procedures and fine-tuning procedures are trained using stochastic gradient descent (SGD) with momentum of 0.9. Batch sizes for the input data is 100. Through calculation and analysis, the floating point operations per second (FLOPs) of our network model is 0.77 giga FLOPs.

Based on the above CNN model, a self-supervised learning network model for our pretext task is designed, as shown in Fig. 2. Besides, a trainable CNN network structure model
<figure><img src="image_3.png" /><figcaption>Fig. 2. Overview of our MI-SSL model.</figcaption></figure>

with two-stream input is adopted. For ease of training, except for the first layer, two-stream CNN models share their structure and weight parameters. The proposed method randomly selects a pair of pixels from the raw data as input. And each pixel can be represented by two kinds of features, such as the features from coherency matrix and Pauli decomposition. Two different presentations can be denoted as $u \in U$ and $v \in V$. This information is easy to obtain in multiple representations of PolSAR data. After training with unlabeled data, we transfer the learned mutual information from the self-supervised model to the classification task. In the experimental part of the article, the quality of the learned mutual information is evaluated based on its performance on the classification task.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Flevoland data from AIRSAR. (a) Pauli RGB image. (b) Ground-truth image.</figcaption></figure>

## IV. EXPERIMENT RESULTS AND ANALYSIS

In this section, some experiments are carried out to show the superiority of the MI-SSL model. Four real PolSAR images from three different areas, including Flevoland, the Netherlands, the western Xi'an area, China, and the San Francisco Bay Area, California, USA, are used in the experiments. Besides, the refined Lee filter is utilized to all the data sets. Six typical classification methods are used for comparison, including a support vector machine (SVM) [13], an autoencoder (AE) model, a CNN model, a convolutional autoencoder (CAE) model, a Wishart autoencoder (WAE) model [28], and a Wishart convolutional autoencoder (WCAE) model [28]. Furthermore, we use three methods to verify the effectiveness of our model, including MI-SSL-P, MI-SSL-H, and MI-SSL-F. All experiments were conducted 10 times independently on a single GeForce 2080Ti GPU with the PyTorch library.

### A. Results on Flevoland From AIRSAR

The first PolSAR data set is from Flevoland. It is obtained from NASA/JPL AIRSAR data in 1989. There are 15 identified classes in this image in total, which are widely used in the study of PolSAR land cover classification. The size of the selected data is 1024 x 750 pixels. Fig. 3 shows the Pauli image and ground-truth map of the selected scene. In SVM and CNN classification model experiments, 1% labeled data are randomly selected to build the training data set, while the rest are used as testing data. The other seven methods are slightly different from the first two methods. We first conduct these methods on 5% unlabeled data to get prior knowledge in an unsupervised way. After that, 1% labeled data are used to fine-tune the classifier.
<table><caption>TABLE II
CLASSIFICATION PERFORMANCES OF FLEVOLAND FROM AIRSAR WITH DIFFERENT METHODS</caption><thead><tr><th>Method</th><th>SVM</th><th>AE</th><th>WAE</th><th>CAE</th><th>WCAE</th><th>CNN</th><th>MI-SSL-P</th><th>MI-SSL-H</th><th>MI-SSL-F</th></tr></thead><tbody><tr><td>Used unlabeled data</td><td>—</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>—</td><td>5%</td><td>5%</td><td>5%</td></tr><tr><td>Used labels</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td></tr><tr><td>Stembeans</td><td>94.37±0.77</td><td>92.42±2.73</td><td>90.82±4.69</td><td>78.98±2.54</td><td>78.89±4.42</td><td>99.32±0.76</td><td><strong>99.95±0.04</strong></td><td>99.69±0.41</td><td>99.85±0.18</td></tr><tr><td>Potatoes</td><td>86.35±0.86</td><td>92.34±3.60</td><td>93.70±2.92</td><td>98.48±0.43</td><td>96.30±1.84</td><td>96.07±6.87</td><td><strong>99.97±0.01</strong></td><td>99.96±0.04</td><td>99.97±0.02</td></tr><tr><td>Grasses</td><td>56.83±1.65</td><td>71.08±22.27</td><td>92.39±3.18</td><td>98.64±0.40</td><td>96.60±2.93</td><td>52.00±23.40</td><td>99.51±0.75</td><td><strong>99.60±0.36</strong></td><td>99.24±0.88</td></tr><tr><td>Water</td><td>84.99±2.94</td><td>87.91±2.17</td><td>86.89±2.39</td><td>92.20±1.24</td><td>90.37±1.29</td><td>87.34±18.21</td><td>100.00±0.01</td><td>99.99±0.02</td><td><strong>100.00±0.00</strong></td></tr><tr><td>Wheat</td><td>84.00±0.36</td><td>89.86±2.73</td><td>91.23±1.60</td><td>96.12±2.63</td><td>94.76±2.07</td><td>95.30±3.58</td><td>99.82±0.16</td><td><strong>99.87±0.14</strong></td><td>99.83±0.15</td></tr><tr><td>Rapeseed</td><td>62.66±3.41</td><td>76.74±6.27</td><td>78.80±5.20</td><td>83.32±0.34</td><td>72.90±1.24</td><td>52.18±42.88</td><td>99.64±0.52</td><td><strong>99.77±0.16</strong></td><td>99.49±0.39</td></tr><tr><td>Forest</td><td>85.52±0.93</td><td>94.38±1.59</td><td>93.48±2.03</td><td>88.49±1.91</td><td>87.56±2.56</td><td>98.50±1.50</td><td><strong>99.98±0.01</strong></td><td>99.96±0.07</td><td>99.89±0.15</td></tr><tr><td>Lucerne</td><td>90.82±0.84</td><td>93.14±4.12</td><td>93.52±3.59</td><td>93.83±2.91</td><td>92.13±2.07</td><td>93.52±11.99</td><td><strong>99.98±0.04</strong></td><td>99.93±0.09</td><td>99.86±0.26</td></tr><tr><td>Wheat3</td><td>94.08±0.60</td><td>82.32±7.93</td><td>85.69±1.79</td><td>85.02±3.37</td><td>79.52±2.24</td><td>69.67±27.90</td><td><strong>99.95±0.06</strong></td><td>99.95±0.07</td><td>99.94±0.03</td></tr><tr><td>Wheat2</td><td>44.32±2.01</td><td>90.46±2.16</td><td>90.62±1.21</td><td>85.93±1.63</td><td>87.60±0.88</td><td>62.48±28.02</td><td><strong>99.75±0.07</strong></td><td>99.62±0.28</td><td>99.73±0.31</td></tr><tr><td>Buildings</td><td>68.68±11.91</td><td>79.86±1.89</td><td>85.39±2.28</td><td>72.99±0.81</td><td>59.74±3.20</td><td>96.95±1.69</td><td>96.71±3.42</td><td><strong>99.73±0.22</strong></td><td>99.48±0.55</td></tr><tr><td>Barley</td><td>96.89±0.51</td><td>94.96±1.56</td><td>93.04±1.40</td><td>97.76±0.85</td><td>97.11±1.79</td><td>43.82±51.45</td><td><strong>99.60±0.38</strong></td><td>99.34±0.98</td><td>97.55±2.60</td></tr><tr><td>Peas</td><td>89.25±0.82</td><td>66.46±17.19</td><td>78.20±7.20</td><td>49.22±15.79</td><td>41.94±22.66</td><td>82.06±29.60</td><td><strong>99.86±0.16</strong></td><td>99.73±0.16</td><td>99.83±0.12</td></tr><tr><td>Beet</td><td>92.51±0.67</td><td>87.59±2.56</td><td>89.06±2.09</td><td>96.72±0.23</td><td>94.37±0.50</td><td>86.70±23.65</td><td><strong>99.95±0.03</strong></td><td>99.95±0.04</td><td>99.94±0.04</td></tr><tr><td>Bare soil</td><td>10.24±19.92</td><td>97.39±2.47</td><td>96.06±3.26</td><td>79.24±3.52</td><td>76.70±2.78</td><td>18.91±37.77</td><td>99.43±0.85</td><td><strong>99.70±0.56</strong></td><td>99.15±0.97</td></tr><tr><td>OA</td><td>80.14±0.40</td><td>88.73±1.47</td><td>89.97±0.43</td><td>89.22±0.27</td><td>86.21±0.65</td><td>78.24±8.24</td><td>99.84±0.13</td><td><strong>99.84±0.08</strong></td><td>99.70±0.21</td></tr><tr><td>KAPPA</td><td>0.79±0.01</td><td>0.87±0.02</td><td>0.89±0.01</td><td>0.88±0.01</td><td>0.85±0.01</td><td>0.76±0.09</td><td><strong>0.99±0.01</strong></td><td><strong>0.99±0.01</strong></td><td><strong>0.99±0.01</strong></td></tr><tr><td>Pre-training time(s)</td><td>—</td><td>63.99</td><td>75.44</td><td>108.22</td><td>129.40</td><td>—</td><td>1191.41</td><td>1186.09</td><td>1183.16</td></tr><tr><td>Fine-tune time(s)</td><td>—</td><td>0.51</td><td>0.62</td><td>42.25</td><td>21.15</td><td>—</td><td>309.57</td><td>308.86</td><td>308.44</td></tr><tr><td>Total Training time(s)</td><td>—</td><td>64.50</td><td>76.06</td><td>150.47</td><td>150.55</td><td>307.52</td><td>1500.98</td><td>1494.95</td><td>1491.60</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 4. OA values with different ratio of training data on Flevoland area data.</figcaption></figure>

To avoid the influence of the unlabeled data ratio on the experimental results, the unlabeled data sets ratio from 0.1% to 10% is employed in the proposed model to learn mutual information. Then 0.1%–10% labeled data are applied to fine-tune the network. In addition, the randomly initialized CNN classification model, with the same network structure, is used to further verify the effectiveness of the MI-SSL. The overall

classification accuracy (OA) results are shown in Fig. 4. In this figure, we find that the OA of MI-SSL-P is much higher than the random initialization CNN model training by a small number of labeled data. Besides, as the ratio of unlabeled data increasing, the classification accuracy becomes higher. This experimental result confirms that our self-supervised training process is effective for a PolSAR image, and these features are beneficial for a deep CNN model to get better performance on classification tasks. After pre-training the network with more than 1% unlabeled data, even if 0.1% labeled data are used to fine-tune the classifier, the OA can be up to 99%. In order to compare with other methods, we use 1% labeled data to verify the superiority of the proposed method.

The experimental results of our MI-SSL model and the comparison methods are shown in Table II and Fig. 5. Obviously, the MI-SSL model has the highest classification accuracy. Besides, the analysis about standard deviations reveals that the MI-SSL model has stronger robustness than the random initialization CNN classifier. However, by comparing the experimental time, our method needs more time to train the classifier. The main time consumption is used for mutual information extraction in the pretraining process. The process of extracting mutual information from unlabeled data takes a long time. In addition, the network parameters updating needs a large number of instances to compare with each other.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Classification results of different methods. (a) SVM. (b) AE. (c) WAE. (d) CAE. (e) WCAE. (f) CNN. (g) MI-SSL-P. (h) MI-SSL-H. (i) MI-SSL-F.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6. Xi’an-area data from RADARSAT-2. (a) Pauli RGB image. (b) Ground-truth image.</figcaption></figure>

<caption>Fig. 7. OA values with different ratio of training data on Xi’an-area data.</caption>

## B. Results on Xi’an Area From RADARSAT-2

The second PolSAR data set is from the western Xi’an area in China. It is produced by RADARSAT-2. Fig. 6 shows the

Pauli color-coded image and its ground truth of three categories, including benchland, urban, and river. We choose a subimage whose size is 512 × 512. In the proposed MI-SSL
<table><caption>TABLE III
CLASSIFICATION PERFORMANCES OF XI'AN AREA FROM RADARSAT-2 WITH DIFFERENT METHODS</caption><thead><tr><th>Method</th><th>SVM</th><th>AE</th><th>WAE</th><th>CAE</th><th>WCAE</th><th>CNN</th><th>MI-SSL-P</th><th>MI-SSL-H</th><th>MI-SSL-F</th></tr></thead><tbody><tr><td>Used unlabeled data</td><td>—</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>—</td><td>5%</td><td>5%</td><td>5%</td></tr><tr><td>Used labels</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td></tr><tr><td>Bench land</td><td>90.54±0.44</td><td>86.09±0.47</td><td>91.12±0.52</td><td>91.50±2.39</td><td>92.76±2.54</td><td>93.63±2.75</td><td>96.10±0.39</td><td><b>96.72±0.33</b></td><td>96.16±0.26</td></tr><tr><td>Urban</td><td>83.65±1.31</td><td>84.84±0.68</td><td>81.17±0.86</td><td>85.59±4.85</td><td>87.75±3.05</td><td>91.08±6.32</td><td>98.37±0.12</td><td><b>98.44±0.27</b></td><td>98.21±0.15</td></tr><tr><td>River</td><td>89.44±0.97</td><td>85.04±2.44</td><td>89.46±0.31</td><td>93.89±0.85</td><td>95.76±0.41</td><td>82.27±10.07</td><td>95.83±0.63</td><td><b>96.17±0.55</b></td><td>95.64±0.66</td></tr><tr><td>OA</td><td>88.015±0.80</td><td>85.49±0.24</td><td>87.45±0.37</td><td>89.76±2.75</td><td>91.44±2.28</td><td>91.03±1.71</td><td>96.86±0.15</td><td><b>97.24±0.06</b></td><td>96.80±0.12</td></tr><tr><td>KAPPA</td><td>0.80±0.01</td><td>0.76±0.01</td><td>0.79±0.01</td><td>0.83±0.04</td><td>0.86±0.04</td><td>0.85±0.03</td><td>0.95±0.01</td><td><b>0.96±0.01</b></td><td>0.95±0.01</td></tr><tr><td>Pre-training time(s)</td><td>—</td><td>88.49</td><td>95.28</td><td>100.09</td><td>126.64</td><td>—</td><td>1664.72</td><td>1656.97</td><td>1654.51</td></tr><tr><td>Fine-tune time(s)</td><td>—</td><td>1.15</td><td>3.16</td><td>9.02</td><td>16.77</td><td>—</td><td>430.96</td><td>430.21</td><td>429.84</td></tr><tr><td>Total training time(s)</td><td>—</td><td>89.64</td><td>98.44</td><td>109.11</td><td>143.41</td><td>430.33</td><td>2095.69</td><td>2087.18</td><td>2084.36</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Fig. 8. Classification results of different methods. (a) SVM. (b) AE. (c) WAE. (d) CAE. (e) WCAE. (f) CNN. (g) MI-SSL-P. (h) MI-SSL-H. (i) MI-SSL-F.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 9. San Francisco Bay data From RADARSAT-2. (a) Pauli RGB image. (b) Ground-truth image.</figcaption></figure>

models, 5% unlabeled data are used for self-supervised pre-training and 1% labeled data are used for fine-tuning the network.

Fig. 7 shows the classification results, taking seven typical unlabeled pixel ratio (0.1%, 0.5%, 1.0%, 2.5%, 5.0%, and 10.0%) as the proposed MI-SSL-P model input. Besides, the randomly initialized CNN classification model with the same network structure is applied for comparison. As we can see, in the case where 2.5% unlabeled data used, even if 0.5% labeled data used for the MI-SSL-P model, the accuracy is as high as 95%. Meanwhile, the classification accuracy of CNN networks using random initialization parameters without

self-supervised training is only about 85% in the case of using 0.5% labeled data. So, we can draw the conclusion that this self-supervised learning process could extract effective mutual information from PolSAR data and improve the classification accuracy with limited labeled data.

The experimental results are shown in Table III and Fig. 8. It is obvious that the MI-SSL model achieves the highest OA. Besides, the classification accuracy of the three proposed models is almost the same. This demonstrates that our proposed model can extract more information from PolSAR data, which is beneficial for improving classification precision. As we can see from Fig. 8, the results of our proposed methods have fewer misclassified points and looked cleaner.

### C. Results on San Francisco Bay From RADARSAT-2

The third PolSAR data set is from the San Francisco Bay Area. It is produced by RADARSAT-2 sensor in April 2008. There are five identified classes in this image in total, including water, vegetation, developed, high-density urban, and
<figure><img src="image_10.png" /><figcaption>Fig. 10. Classification results of different methods. (a) SVM. (b) AE. (c) WAE. (d) CAE. (e) WCAE. (f) CNN. (g) MI-SSL-P. (h) MI-SSL-H. (i) MI-SSL-F.</figcaption></figure>

<table><caption>TABLE IV
CLASSIFICATION PERFORMANCES OF SAN FRANCISCO BAY FROM RADARSAT-2 WITH DIFFERENT RATIO OF LABELED DATA</caption><thead><tr><th>Used labels</th><th>Methods</th><th>Water</th><th>Vegetation</th><th>Low-Density Urban</th><th>High-Density Urban</th><th>Developed</th><th>OA</th><th>KAPPA</th></tr></thead><tbody><tr><td rowspan="2">0.1%</td><td>Random initialization</td><td>99.96</td><td>93.38</td><td>81.81</td><td>84.20</td><td>59.92</td><td>91.30</td><td>0.88</td></tr><tr><td>MI-SSL-H</td><td>99.99</td><td>96.03</td><td>97.43</td><td>98.16</td><td>92.21</td><td>98.34</td><td>0.98</td></tr><tr><td rowspan="2">0.5%</td><td>Random initialization</td><td>99.99</td><td>93.85</td><td>91.59</td><td>92.92</td><td>85.52</td><td>95.80</td><td>0.94</td></tr><tr><td>MI-SSL-H</td><td>99.99</td><td>96.48</td><td>97.84</td><td>97.34</td><td>95.67</td><td>98.50</td><td>0.98</td></tr><tr><td rowspan="2">1%</td><td>Random initialization</td><td>99.99</td><td>92.46</td><td>93.27</td><td>94.15</td><td>87.57</td><td>96.23</td><td>0.94</td></tr><tr><td>MI-SSL-H</td><td>99.98</td><td>96.68</td><td>98.16</td><td>97.10</td><td>94.51</td><td>98.39</td><td>0.98</td></tr><tr><td rowspan="2">2.5%</td><td>Random initialization</td><td>99.99</td><td>92.25</td><td>93.22</td><td>92.45</td><td>85.52</td><td>95.83</td><td>0.94</td></tr><tr><td>MI-SSL-H</td><td>99.98</td><td>96.72</td><td>98.11</td><td>97.21</td><td>94.15</td><td>98.49</td><td>0.98</td></tr><tr><td rowspan="2">5%</td><td>Random initialization</td><td>99.99</td><td>96.33</td><td>96.64</td><td>97.51</td><td>89.00</td><td>97.98</td><td>0.97</td></tr><tr><td>MI-SSL-H</td><td>99.99</td><td>95.99</td><td>98.20</td><td>97.16</td><td>93.58</td><td>98.40</td><td>0.98</td></tr><tr><td rowspan="2">10%</td><td>Random initialization</td><td>99.98</td><td>95.48</td><td>97.50</td><td>98.85</td><td>97.40</td><td>98.61</td><td>0.98</td></tr><tr><td>MI-SSL-H</td><td>99.97</td><td>96.04</td><td>97.32</td><td>98.32</td><td>95.42</td><td>98.48</td><td>0.98</td></tr></tbody></table>

low-density urban. The size of this image is 1380 × 1800. The Pauli pseudocolor image and ground-truth map are shown in Fig. 9. Due to the large number of pixels in this PolSAR image, we randomly select 10% of the pixels for the experiment.

In order to compare the effect of random initialization and our proposed methods, we apply 0.1%–10% labeled data to train the models. The MI-SSL-H with 5% unlabeled data pretraining is adopted to ensure the reliability and stability of experiment results. The comparison are shown in Table IV.

From this table, we can see that in the case of randomly initializing the network, we need to use about 10% of the labeled data to get the highest accuracy. However, using our method, only 0.1% of the labeled data is needed to achieve the same training results. The experimental results, including OA and kappa value, are listed in Table V. From Fig. 9, we can see the output of each classification method easily. Compared with CNN networks without self-supervised training, our method increases the classification accuracy by about 2%.
<table><caption>TABLE V
CLASSIFICATION PERFORMANCES OF SAN FRANCISCO BAY FROM RADARSAT-2 WITH DIFFERENT METHODS</caption><thead><tr><th>Method</th><th>SVM</th><th>AE</th><th>WAE</th><th>CAE</th><th>WCAE</th><th>CNN</th><th>MI-SSL-P</th><th>MI-SSL-H</th><th>MI-SSL-F</th></tr></thead><tbody><tr><td>Used unlabeled data</td><td>—</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>—</td><td>5%</td><td>5%</td><td>5%</td></tr><tr><td>Used labels</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td></tr><tr><td>Water</td><td>99.79±0.14</td><td>99.51±0.27</td><td>99.60±0.18</td><td>98.78±0.88</td><td>98.34±1.74</td><td>99.99±0.01</td><td>99.99±0.01</td><td>99.98±0.01</td><td><b>100.00±0.01</b></td></tr><tr><td>Vegetation</td><td>87.62±1.27</td><td>84.18±3.05</td><td>83.52±1.85</td><td>86.66±6.01</td><td>82.40±8.74</td><td>92.46±1.27</td><td>96.04±0.42</td><td><b>96.68±2.63</b></td><td>96.59±0.43</td></tr><tr><td>Low-Density Urban</td><td>66.02±1.50</td><td>74.81±1.54</td><td>77.74±3.01</td><td>85.92±1.77</td><td>80.86±2.49</td><td>93.27±3.08</td><td>97.21±0.98</td><td><b>98.16±0.98</b></td><td>97.68±0.31</td></tr><tr><td>High-Density Urban</td><td>84.06±1.12</td><td>71.35±2.61</td><td>64.43±3.07</td><td>67.09±16.22</td><td>60.35±14.29</td><td>94.15±3.96</td><td><b>97.92±0.69</b></td><td>97.10±1.17</td><td>97.21±0.72</td></tr><tr><td>Developed</td><td>44.26±3.24</td><td>63.51±7.16</td><td>61.98±2.61</td><td>48.82±12.62</td><td>29.64±9.55</td><td>87.57±4.31</td><td>94.54±1.51</td><td>94.51±3.07</td><td><b>94.94±1.95</b></td></tr><tr><td>OA</td><td>86.67±0.33</td><td>86.69±0.42</td><td>86.03±0.19</td><td>88.45±2.16</td><td>83.82±2.44</td><td>96.23±0.43</td><td>98.37±0.03</td><td>98.39±0.44</td><td><b>98.43±0.05</b></td></tr><tr><td>KAPPA</td><td>0.81±0.01</td><td>0.81±0.01</td><td>0.80±0.01</td><td>0.83±0.03</td><td>0.76±0.04</td><td>0.94±0.01</td><td>0.98±0.01</td><td>0.98±0.01</td><td><b>0.98±0.01</b></td></tr><tr><td>Pre-training time(s)</td><td>—</td><td>70.79</td><td>90.37</td><td>92.77</td><td>131.80</td><td>—</td><td>1289.55</td><td>1276.01</td><td>1277.41</td></tr><tr><td>Fine-tune time(s)</td><td>—</td><td>1.11</td><td>0.35</td><td>8.43</td><td>32.48</td><td>—</td><td>334.05</td><td>335.11</td><td>334.23</td></tr><tr><td>Total training time(s)</td><td>—</td><td>71.90</td><td>90.72</td><td>101.20</td><td>164.28</td><td>334.39</td><td>1623.60</td><td>1611.12</td><td>1611.64</td></tr></tbody></table>

<table><caption>TABLE VI
CLASSIFICATION PERFORMANCES OF FLEVOLAND FROM RADARSTA-2 WITH DIFFERENT RATIO OF LABELED DATA</caption><thead><tr><th>Used labels</th><th>Methods</th><th>Urban</th><th>Water</th><th>Forest</th><th>Cropland</th><th>OA</th><th>KAPPA</th></tr></thead><tbody><tr><td rowspan="2">0.1%</td><td>Random initialization</td><td>74.14</td><td>99.97</td><td>46.33</td><td>90.29</td><td>86.42</td><td>0.79</td></tr><tr><td>MI-SSL-F</td><td>94.93</td><td>99.62</td><td>97.23</td><td>98.58</td><td>98.60</td><td>0.98</td></tr><tr><td rowspan="2">0.5%</td><td>Random initialization</td><td>84.99</td><td>84.19</td><td>87.47</td><td>98.20</td><td>89.95</td><td>0.85</td></tr><tr><td>MI-SSL-F</td><td>94.30</td><td>99.53</td><td>96.77</td><td>98.72</td><td>98.51</td><td>0.98</td></tr><tr><td rowspan="2">1%</td><td>Random initialization</td><td>80.75</td><td>99.90</td><td>62.77</td><td>98.93</td><td>92.57</td><td>0.89</td></tr><tr><td>MI-SSL-F</td><td>94.02</td><td>99.82</td><td>96.26</td><td>98.96</td><td>98.62</td><td>0.98</td></tr><tr><td rowspan="2">2.5%</td><td>Random initialization</td><td>90.81</td><td>97.20</td><td>96.66</td><td>98.89</td><td>97.39</td><td>0.96</td></tr><tr><td>MI-SSL-F</td><td>95.82</td><td>99.96</td><td>97.19</td><td>98.44</td><td>98.73</td><td>0.98</td></tr><tr><td rowspan="2">5%</td><td>Random initialization</td><td>93.43</td><td>99.58</td><td>96.39</td><td>97.81</td><td>98.08</td><td>0.97</td></tr><tr><td>MI-SSL-F</td><td>95.67</td><td>99.83</td><td>97.23</td><td>98.45</td><td>98.67</td><td>0.98</td></tr><tr><td rowspan="2">10%</td><td>Random initialization</td><td>94.50</td><td>99.93</td><td>94.23</td><td>99.32</td><td>98.50</td><td>0.98</td></tr><tr><td>MI-SSL-F</td><td>94.38</td><td>98.91</td><td>97.21</td><td>98.59</td><td>98.28</td><td>0.97</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 11. Flevoland data From RADARSTA-2. (a) Pauli RGB image. (b) Ground-truth image.</figcaption></figure>

## D. Results on Flevoland From RADARSTA-2

The last PolSAR data set is from Flevoland, the Netherlands. It is produced by RADARSAT-2 sensor in April 2008.

There are four identified classes in this image in total, including forest, cropland, water, and urban. The size of the selected scene is 1635 × 2375 pixels. The corresponding Pauli image and ground-truth class labels are shown in Fig. 11. As in the previous experiment, we randomly selected 10% of samples to verify the effectiveness of our method.

On this data set, the MI-SSL-F method with 5% unlabeled data pretraining is compared with the randomly initialized CNN model. We apply 0.1%–10% labeled data to train the models, and the quantitative result is presented in Table VI. Using the table given above, it can be seen that the MI-SSL-F method can still achieve high classification accuracy with fewer labeled data. The comparison and results with other advanced methods, including OA and kappa value, are listed in Table VII. From Fig. 12, we can see the visual effects of each classification method. The OA of the proposed methods is better than those of all the comparison methods. Compared with CNN networks without self-supervised training, our methods approximately increase the classification accuracy by 9%. From Fig. 12, it appears that our proposed methods greatly improve the classification accuracy for urban areas.
<table><caption>TABLE VII
CLASSIFICATION PERFORMANCES OF FLEVOLAND FROM RADARSTA-2 WITH DIFFERENT METHODS</caption><thead><tr><th>Method</th><th>SVM</th><th>AE</th><th>WAE</th><th>CAE</th><th>WCAE</th><th>CNN</th><th>MI-SSL-P</th><th>MI-SSL-H</th><th>MI-SSL-F</th></tr></thead><tbody><tr><td>Used unlabeled data</td><td>—</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>—</td><td>5%</td><td>5%</td><td>5%</td></tr><tr><td>Used labels</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td><td>1%</td></tr><tr><td>Urban</td><td>73.67±2.85</td><td>25.88±14.93</td><td>53.28±4.45</td><td>55.51±2.25</td><td>53.41±7.09</td><td>80.75±2.63</td><td><strong>94.97±1.14</strong></td><td>90.24±2.39</td><td>94.02±1.15</td></tr><tr><td>Water</td><td>99.50±0.08</td><td>98.68±1.10</td><td>99.62±0.04</td><td>99.89±0.04</td><td>99.78±0.08</td><td>99.90±8.71</td><td>99.90±5.97</td><td>97.27±2.69</td><td><strong>99.82±0.12</strong></td></tr><tr><td>Forest</td><td>83.98±1.55</td><td>56.41±24.44</td><td>78.88±1.00</td><td>87.51±1.31</td><td>81.55±2.39</td><td>62.77±14.01</td><td><strong>97.83±1.69</strong></td><td>95.25±1.21</td><td>96.26±1.52</td></tr><tr><td>Cropland</td><td>92.20±0.67</td><td>83.52±7.00</td><td>93.09±0.44</td><td>85.87±0.95</td><td>86.74±2.43</td><td>98.93±0.82</td><td><strong>98.99±0.29</strong></td><td>98.90±0.13</td><td>98.96±0.27</td></tr><tr><td>OA</td><td>92.92±0.37</td><td>82.38±7.26</td><td>91.38±0.18</td><td>90.37±0.37</td><td>89.53±1.45</td><td>92.57±13.99</td><td>98.61±2.62</td><td>97.17±1.18</td><td><strong>98.62±0.21</strong></td></tr><tr><td>KAPPA</td><td>0.89±0.00</td><td>0.72±0.12</td><td>0.87±0.00</td><td>0.85±0.00</td><td>0.84±0.02</td><td>0.89±0.20</td><td>0.98±0.04</td><td>0.96±0.02</td><td><strong>0.98±0.01</strong></td></tr><tr><td>Pre-training time(s)</td><td>—</td><td>103.60</td><td>201.10</td><td>104.07</td><td>127.10</td><td>—</td><td>1797.99</td><td>1794.77</td><td>1784.90</td></tr><tr><td>Fine-tune time(s)</td><td>—</td><td>0.38</td><td>0.32</td><td>13.93</td><td>36.48</td><td>—</td><td>446.16</td><td>448.78</td><td>446.55</td></tr><tr><td>Total training time(s)</td><td>—</td><td>103.98</td><td>201.42</td><td>118.00</td><td>163.58</td><td>446.64</td><td>2244.15</td><td>2243.55</td><td>2231.45</td></tr></tbody></table>

<figure><img src="image_12.png" /><figcaption>Fig. 12. Classification results of different methods. (a) SVM. (b) AE. (c) WAE. (d) CAE. (e) WCAE. (f) CNN. (g) MI-SSL-P. (h) MI-SSL-H. (i) MI-SSL-F.</figcaption></figure>

Due to the limited training data of the urban areas, the classification accuracy of the other comparison methods is relatively low. However, the proposed model has an accuracy of up to 92%. This further proved that the method proposed in this article takes advantage of the mutual information from the unlabeled data. Furthermore, it greatly improves the classification results.

## V. CONCLUSION

In this article, an MI-SSL model is established for PolSAR land cover classification tasks. We propose using self-supervised learning to solve the problem of training deep

CNN with limited labeled data. Before the classification with labeled data, we design a reasonable pretext task to extract mutual information in a latent feature space without any human annotation. Experimental results on four real PolSAR data sets show the superiority of the proposed method. Besides, our MI-SSL model achieves remarkable classification results with fewer labeled data compared with the state-of-the-art methods. However, there is more unexplored hidden information in multi-view PolSAR data forms. Our future work will be focused on how to use this information more effectively to improve the classification performance in a self-supervised way.
ACKNOWLEDGMENT

The authors would like to thank Wen et al. [28] for providing code for our comparative experiments.

REFERENCES

[1] J.-S. Lee, M. R. Grunes, T. L. Ainsworth, L.-J. Du, D. L. Schuler, and S. R. Cloude, "Unsupervised classification using polarimetric decomposition and the complex wishart classifier," *IEEE Trans. Geosci. Remote Sens.*, vol. 37, no. 5, pp. 2249-2258, Sep. 1999.

[2] S. R. Cloude and E. Pottier, "An entropy based classification scheme for land applications of polarimetric SAR," *IEEE Trans. Geosci. Remote Sens.*, vol. 35, no. 1, pp. 68-78, Jan. 1997.

[3] E. Pottier, "Dr. J. R. Huynen's main contributions in the development of polarimetric radar techniques and how the 'radar targets phenomenological Concept' becomes a theory," *Proc. SPIE*, vol. 1748, pp. 72-85, Feb. 1993.

[4] A. Freeman and S. L. Durden, "A three-component scattering model for polarimetric SAR data," *IEEE Trans. Geosci. Remote Sens.*, vol. 36, no. 3, pp. 963-973, May 1998.

[5] J. R. Huynen, "Physical reality of radar targets," *Proc. SPIE*, vol. 1748, pp. 86-96, Feb. 1993.

[6] W. L. Cameron and L. K. Leung, "Feature motivated polarization scattering matrix decomposition," in *Proc. IEEE Int. Conf. Radar*, vol. 1, May 1990, pp. 549-557.

[7] E. Krogager, "New decomposition of the radar target scattering matrix," *Electron. Lett.*, vol. 26, no. 18, pp. 1525-1527, Aug. 1990.

[8] J.-S. Lee, M. R. Grunes, and R. Kwok, "Classification of multi-look polarimetric SAR data based on complex Wishart distribution," *Int. J. Remote Sens.*, vol. 15, no. 11, pp. 2299-2311, Jul. 1994.

[9] J. A. Kong, A. A. Swartz, H. A. Yueh, L. M. Novak, and R. T. Shin, "Identification of terrain cover using the optimum polarimetric classifier," *J. Electromagn. Waves Appl.*, vol. 2, no. 2, pp. 171-194, Jan. 1988.

[10] M. Tao, F. Zhou, Y. Liu, and Z. Zhang, "Tensorial independent component analysis-based feature extraction for polarimetric SAR data classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 5, pp. 2481-2495, May 2015.

[11] E. Pottier and J. Saillard, "On radar polarization target decomposition theorems with application to target classification, by using neural network method," in *Proc. 7th IEE ICAP*, 1991, pp. 265-268.

[12] O. Antropov, Y. Rauste, H. Astola, J. Praks, T. Hame, and M. T. Hallikainen, "Land cover and soil type mapping from spaceborne PolSAR data at L-Band with probabilistic neural network," *IEEE Trans. Geosci. Remote Sens.*, vol. 52, no. 9, pp. 5256-5270, Sep. 2014.

[13] C. Lardeux et al., "Support vector machine for multifrequency SAR polarimetric data classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 47, no. 12, pp. 4143-4152, Dec. 2009.

[14] A. Masjedi, M. J. Valadan Zoej, and Y. Maghsoudi, "Classification of polarimetric SAR images based on modeling contextual information and using texture features," *IEEE Trans. Geosci. Remote Sens.*, vol. 54, no. 2, pp. 932-943, Feb. 2016.

[15] K. Ersahin, I. G. Cumming, and R. K. Ward, "Segmentation and classification of polarimetric SAR data using spectral graph partitioning," *IEEE Trans. Geosci. Remote Sens.*, vol. 48, no. 1, pp. 164-174, Jan. 2010.

[16] Y. Zhou, H. Wang, F. Xu, and Y.-Q. Jin, "Polarimetric SAR image classification using deep convolutional neural networks," *IEEE Geosci. Remote Sens. Lett.*, vol. 13, no. 12, pp. 1935-1939, Dec. 2016.

[17] Z. Zhang, H. Wang, F. Xu, and Y.-Q. Jin, "Complex-valued convolutional neural network and its application in polarimetric SAR image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 55, no. 12, pp. 7177-7188, Dec. 2017.

[18] F. Liu, L. Jiao, B. Hou, and S. Yang, "POL-SAR image classification based on wishart DBN and local spatial information," *IEEE Trans. Geosci. Remote Sens.*, vol. 54, no. 6, pp. 3292-3308, Jun. 2016.

[19] L. Jiao and F. Liu, "Wishart deep stacking network for fast POLSAR image classification," *IEEE Trans. Image Process.*, vol. 25, no. 7, pp. 3273-3286, Jul. 2016.

[20] L. Zhang, W. Ma, and D. Zhang, "Stacked sparse autoencoder in PolSAR data classification using local spatial information," *IEEE Geosci. Remote Sens. Lett.*, vol. 13, no. 9, pp. 1359-1363, Sep. 2016.

[21] B. Ren, B. Hou, J. Chanussot, and L. Jiao, "PolsAR feature extraction via tensor embedding framework for land cover classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 4, pp. 2337-2351, Apr. 2020.

[22] Y. Wu, K. Ji, W. Yu, and Y. Su, "Region-based classification of polarimetric SAR images using wishart MRF," *IEEE Geosci. Remote Sens. Lett.*, vol. 5, no. 4, pp. 668-672, Oct. 2008.

[23] H. Bi, F. Xu, Z. Wei, Y. Xue, and Z. Xu, "An active deep learning approach for minimally supervised PolSAR image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 11, pp. 9378-9395, Nov. 2019.

[24] C. Yang, B. Hou, B. Ren, Y. Hu, and L. Jiao, "CNN-based polarimetric decomposition feature selection for PolSAR image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 11, pp. 8796-8812, Nov. 2019.

[25] J. Zhao, M. Datcu, Z. Zhang, H. Xiong, and W. Yu, "Contrastive-regulated CNN in the complex domain: A method to learn physical scattering signatures from flexible PolSAR images," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 12, pp. 10116-10135, 2019.

[26] B. Ren, B. Hou, J. Chanussot, and L. Jiao, "Modified tensor distance-based multiview spectral embedding for PolSAR land cover classification," *IEEE Geosci. Remote Sens. Lett*, vol. 17, no. 12, pp. 2095-2099, Dec. 2020.

[27] B. Rasti et al., "Feature extraction for hyperspectral imagery: The evolution from shallow to deep: Overview and toolbox," *IEEE Geosci. Remote Sens. Mag.*, vol. 8, no. 4, pp. 60-88, Dec. 2020.

[28] W. Xie et al., "POLSAR image classification via wishart-AE model or wishart-CAE model," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 10, no. 8, pp. 3604-3615, Aug. 2017.

[29] A. Vali, S. Comai, and M. Matteucci, "Deep learning for land use and land cover classification based on hyperspectral and multispectral Earth observation data: A review," *Remote Sens.*, vol. 12, no. 15, p. 2495, Aug. 2020.

[30] X. X. Zhu et al., "Deep learning in remote sensing: A comprehensive review and list of resources," *IEEE Geosci. Remote Sens. Mag.*, vol. 5, no. 4, pp. 8-36, Dec. 2017.

[31] M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, "Deep learning classifiers for hyperspectral imaging: A review," *ISPRS J. Photogramm. Remote Sens.*, vol. 158, pp. 279-317, Dec. 2019.

[32] H. Bi, J. Sun, and Z. Xu, "A graph-based semisupervised deep learning model for PolSAR image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 4, pp. 2116-2132, Apr. 2019.

[33] S. Uhlmann and S. Kiranyaz, "Integrating color features in polarimetric SAR image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 52, no. 4, pp. 2197-2216, Apr. 2014.

[34] G. D. De Grandi, J.-S. Lee, and D. L. Schuler, "Target detection and texture segmentation in polarimetric SAR images using a wavelet frame: Theoretical aspects," *IEEE Trans. Geosci. Remote Sens.*, vol. 45, no. 11, pp. 3437-3453, Nov. 2007.

[35] A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep convolutional neural networks," in *Proc. Neural Inf. Process. Syst.*, 2012, pp. 1097-1105.

[36] C. Szegedy et al., "Going deeper with convolutions," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jun. 2015, pp. 1-9.

[37] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in *Proc. Int. Conf. Learn. Represent.*, 2015, pp. 1-14.

[38] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jun. 2016, pp. 2536-2544.

[39] G. Larsson, M. Maire, and G. Shakhnarovich, "Colorization as a proxy task for visual understanding," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jul. 2017, pp. 840-849.

[40] R. Zhang, P. Isola, and A. A. Efros, "Split-brain autoencoders: Unsupervised learning by cross-channel prediction," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jul. 2017, pp. 645-654.

[41] C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction," in *Proc. IEEE Int. Conf. Comput. Vis. (ICCV)*, Dec. 2015, pp. 1422-1430.

[42] T. N. Mundhenk, D. Ho, and B. Y. Chen, "Improvements to context based self-supervised learning," in *Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.*, Jun. 2018, pp. 9339-9348.

[43] M. Noroozi and P. Favaro, "Unsupervised learning of visual representations by solving jigsaw puzzles," in *Proc. Eur. Conf. Comput. Vis.*, 2016, pp. 69-84.

[44] M. Noroozi, H. Pirsivavash, and P. Favaro, "Representation learning by learning to count," in *Proc. IEEE Int. Conf. Comput. Vis. (ICCV)*, Oct. 2017, pp. 5899-5907.

[45] J. Donahue, P. Krähenbühl, and T. Darrell, "Adversarial feature learning," in *Proc. Int. Conf. Learn. Represent.*, 2017, pp. 1-18.
[46] S. Jenni and P. Favaro, "Self-supervised feature learning by learning to spot artifacts," in *Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.*, Jun. 2018, pp. 2733-2742.

[47] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination," in *Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.*, Jun. 2018, pp. 3733-3742.

[48] Y. Jia et al., "Caffe: Convolutional architecture for fast feature embedding," in *Proc. 22nd ACM Int. Conf. Multimedia*, Nov. 2014, pp. 675-678.

[49] S.-W. Chen and C.-S. Tao, "PolSAR image classification using polarimetric-feature-driven deep convolutional neural network," *IEEE Geosci. Remote Sens. Lett.*, vol. 15, no. 4, pp. 627-631, Apr. 2018.

<figure><img src="image_13.png" /></figure>

**Jocelyn Chanussot** (Fellow, IEEE) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP), Grenoble, France, in 1995, and the Ph.D. degree from the Université de Savoie, Annecy, France, in 1998.

In 1999, he was with the Geography Imagery Perception Laboratory for the Delegation Generale de l'Armement (DGA-French National Defense Department). Since 1999, he has been with Grenoble INP, where he is a Professor of Signal and Image Processing. He is conducting his research at the Grenoble Images Speech Signals and Automatics Laboratory (GIPSA-Lab). He has been a Visiting Scholar at Stanford University, Stanford, CA, USA; KTH Royal Institute of Technology, Stockholm, Sweden; and National University of Singapore (NUS), Singapore. Since 2013, he has been an Adjunct Professor with the University of Iceland. From 2015 to 2017, he was a Visiting Professor with the University of California at Los Angeles (UCLA), Los Angeles, CA, USA. His research interests include image analysis, multicomponent image processing, nonlinear filtering, data fusion, and machine learning in remote sensing.

Dr. Chanussot is the founding President of IEEE Geoscience and Remote Sensing French chapter (2007-2010), which received the 2010 IEEE GRS-S Chapter Excellence Award. He was a co-recipient of the NORSIG 2006 Best Student Paper Award, the IEEE GRSS 2011 and 2015 Symposium Best Paper Award, the IEEE GRSS 2012 Transactions Prize Paper Award, and the IEEE GRSS 2013 Highest Impact Paper Award. He is a member of the IEEE Geoscience and Remote Sensing Society AdCom. He was a member of the Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society (2006-2008). He is also a member of the Institut Universitaire de France (2012-2017) and a 2018 Highly Cited Researcher (Clarivate Analytics). He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote sensing (WHISPERS). He was the Chair (2009-2011) and Co-Chair of the GRS Data Fusion Technical Committee (2005-2008). He was the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing (2009). He is an Associate Editor of the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING and the IEEE TRANSACTIONS ON IMAGE PROCESSING. He was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING (2011-2015). In 2013, he was a Guest Editor of the PROCEEDINGS OF THE IEEE, and a Guest Editor of *IEEE Signal Processing Magazine* in 2014.

<figure><img src="image_14.png" /></figure>

**Bo Ren** (Member, IEEE) received the B.S. degree in telecommunications engineering from Northwest University, Xi'an, China, in 2011, and the Ph.D. degree in circuits and systems from Xidian University, Xi'an, in 2017.

Since 2018, he has been an Assistant Professor with the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, Xidian University. He is a Visiting Scholar with the Grenoble Images Parole Signal Automatique (GIPSA-Lab), Grenoble, France. His research interests include synthetic aperture radar image interpretation and understanding, and sparse representation and machine learning in remote sensing images.

<figure><img src="image_15.png" /></figure>

**Yangyang Zhao** was born in Yulin, Shaanxi, China, in 1994. He received the B.S. degree from Northwestern Polytechnical University, Xi'an, China, in 2015.

He is a Graduate Student with the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an. His research interest includes the polarimetric synthetic aperture radar image processing.

<figure><img src="image_16.png" /></figure>

**Biao Hou** (Member, IEEE) received the B.S. and M.S. degrees in mathematics from Northwest University, Xi'an, China, in 1996 and 1999, respectively, and the Ph.D. degree in circuits and systems from Xidian University, Xi'an, in 2003.

Since 2003, he has been with the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, where he is a Professor. His research interests include compressive sensing and synthetic aperture radar image interpretation.

<figure><img src="image_17.png" /></figure>

**Licheng Jiao** (Fellow, IEEE) was born in Shaanxi, China, in October 1959. He received the B.S. degree from Shanghai Jiao Tong University, Shanghai, China, in 1982, and the M.S. and Ph.D. degrees from Xi'an Jiaotong University, Xi'an, China, in 1984 and 1990, respectively.

From 1984 to 1986, he was an Assistant Professor with the Civil Aviation University of China, Tianjin, China. From 1990 to 1991, he was a Post-Doctoral Fellow with the National Key Laboratory of Radar Signal Processing, Xidian University, Xi'an, where he is the Director of the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education. His research interests include signal and image processing, nonlinear circuits and systems theory, learning theory and algorithms, optimization problems, wavelet theory, and machine learning.