<figure><img src="image_1.png" /></figure>

HAL
open science

# SAR Scene Classification Based on Self-supervised Jigsaw Puzzles

Zhongle Ren, Yiming Lu, Hanxiao Wang, Yu Zhang, Biao Hou

<figure><img src="image_2.png" /></figure>

► To cite this version:

Zhongle Ren, Yiming Lu, Hanxiao Wang, Yu Zhang, Biao Hou. SAR Scene Classification Based on Self-supervised Jigsaw Puzzles. 5th International Conference on Intelligence Science (ICIS), Oct 2022, Xi'an, China. pp.334-343, 10.1007/978-3-031-14903-0_36. hal-04666421

HAL Id: hal-04666421

https://hal.science/hal-04666421v1

Submitted on 1 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# SAR Scene Classification Based on Self-supervised
## Jigsaw Puzzles

Zhongle Ren¹, Yiming Lu¹, Hanxiao Wang¹, Yu Zhang¹, Biao Hou¹
¹Key Laboratory of Intelligent Perception and Image Understanding of Ministry of
Education of China
Xidian University, Xi'an 710071, P. R. China,
zlren@xidian.edu.cn

**Abstract.** Scene classification is a hot issue in the field of SAR image interpretation. Many SAR image interpretation tasks can be promoted with the development of highly credible scene classification methods. But the fussy steps of traditional methods and the imperious demands of labeled samples in deep learning-based methods restrict the effective feature learning in SAR scene classification. Hence, a self-supervised learning method based on Jigsaw puzzles is proposed to address the problems. Concretely, the Jigsaw puzzle reassembly of the SAR image block is firstly taken as the upstream task without manual labels. Once the correct spatial arrangement is obtained from it, the learned high-level feature from the upstream task is used as the pre-training model for the downstream task, which is then fine-tuned with only a few labeled samples to enhance the performance of the SAR scene classification task. Experimental results on 25-class real SAR scenes confirm the proposed method can greatly improve the scene classification performance than directly training the network with the same number of labeled samples used in the downstream task.

**Keywords:** SAR, Self-supervised Learning, Jigsaw Puzzles, Scene Classification.

## 1 Introduction

Synthetic Aperture Radar (SAR), is a system for observing the surface of the earth, which does not gets affected by sunlight, cloud cover and weather conditions. It is
very important to interpret SAR images correctly and effectively, for instance, the SAR image classification, which is the most rudimentary issue in interpreting SAR images. SAR image classification has been proved to play a great role in agricultural detection [1], fuel pollution [2] and terrain surface classification [3].

The traditional classification method is cumbersome. For instance, the methods based on SVM (Support Vector Machine) [4] or random forest classifier [5] need to manually extract features and bring them into the classifier, which requires a high cost in manual experience. With the development of technology, many methods based on deep learning have emerged. Zhao et al [6] proposed a SAR image classification method based on the multi-scale local Fisher pattern, which uses the implicit generation and discrimination information of each Fisher vector to classify. Zhang et al [7] used both the deep features learned from the convolutional neural network and the spatial constraints between superpixels to achieve classification, but this method requires many samples to extract the deep features. To relieve the demand for the data, Li et al [8] combined the convolutional neural network with metric learning, but the computational cost of this method is high. Many deep learning methods are inseparable from a large number of samples. However, due to insufficient open-source data for SAR images, these deep learning methods cannot obtain human enough labeled data for experiments, thus limiting the performance.

The datasets are not enough in many fields, and the cost of manually labeled data is also very high, which consumes a lot of human and financial resources. Hence, a new method is produced, which is called self-supervised learning. Self-supervised learning mainly uses upstream tasks to excavate its supervision information from large-scale unlabeled data, then gets a pre-trained model. Only a small number of labeled samples are used to fine-tune the parameters in the downstream task. This learning strategy is the latest variant of unsupervised learning topics, which can take advantage of tags provided by data “for free”.

In this paper, an algorithm for self-supervised SAR image scene classification is proposed, using Jigsaw puzzles [9,10] as the upstream task. The algorithm is to complete the Jigsaw puzzles problem in the upstream task and migrate the learned features to the downstream classification task. The remainder of this article is arranged as follows. Section 2 describes the background knowledge and the proposed method. Section 3 introduces the experimental details and results in upstream and downstream tasks. Finally, the conclusion and future work are drawn in Section 4.
# 2 Methodology

## 2.1 Jigsaw Puzzle Problem

Jigsaw puzzles have always been closely related to learning. By solving Jigsaw puzzles, we can learn the relationship between various blocks and understand their internal characteristics. It is also indicated by psychological research that Jigsaw puzzles can be used to evaluate human visual spatial processing [11]. Our work is to use the convolution neural network to learn the characteristics of images in the process of solving Jigsaw puzzles. The direct method of solving Jigsaw puzzles is to stack the blocks of the puzzle along the channel, then increase the filter depth of the first convolution layer in the convolution neural network accordingly. The problem with the solution is that the network will learn the correlation between low-level texture information among the blocks, which is one-sided for understanding the overall structure, and it may not be conducive to solving the downstream tasks of self-supervised learning. Hence, a network is proposed, which can calculate characteristics according to each block pixel, and find the permutation of the puzzle through the whole connection layer by using these characteristics. In this way, the network can learn representative and distinguishing characteristics without using any label.

## 2.2 Network Structure

A convolution neural network is built to complete Jigsaw puzzles. We make changes to the AlexNet [12] convolutional layer to obtain a network structure with a smaller number of parameters and design the fully connected layer (as shown in Fig.1). The change index and parameter quantity are shown in Table 1 and Table 2, the network shares weights between the convolutional layer part and the first fully connected layer. The output of fc6 in the first layer of all full connection layers is connected as the output of fc7. After a full connection layer and softmax classifier, the final output value is obtained. The overall network structure is called the context-free network.

<table><caption>**Table 1.** Number of convolution cores.</caption><tr><td>Conv1</td><td>Conv2</td><td>Conv3</td><td>Conv4</td><td>Conv5</td></tr></table>
<table><tr><td>AlexNet</td><td>96</td><td>256</td><td>384</td><td>384</td><td>256</td></tr><tr><td>Ours</td><td><b>64</b></td><td><b>192</b></td><td><b>384</b></td><td><b>256</b></td><td><b>256</b></td></tr></table>

At the beginning of the experiment, the size of the input image is adjusted to 255 × 255 and divided into 3 × 3 image blocks, each size of which is 85 × 85. Then a 64 × 64 pixel image block is randomly selected from each 85 × 85 image block. To make the input conform to the structure of the network, we resize it to 75 × 75. The nine image blocks are reordered by randomly selected permutations in the predefined permutation set and sent to the network. Our ultimate aim is to predict the index of the selected permutation.

<table><caption>**Table 2.** Number of parameters.</caption><thead><tr><th></th><th>Conv</th><th>Fc6</th></tr></thead><tbody><tr><td>AlexNet</td><td>3.7M</td><td>37.7M</td></tr><tr><td>Ours</td><td><b>2.4M</b></td><td><b>2.3M</b></td></tr></tbody></table>

<figure><img src="image_5.png" /></figure>

**Fig. 1.** Flowchart of proposed methods. The sample is cropped into blocks (red grid), and the pixel value is set to 255 for the part beyond the sample. This cutting method is more conducive in practice. 64×64 regions are randomly cut from the nine blocks respectively and resized to 75×75. After that, they are fed into the network, and the index value of the selected permutation is predicted accordingly. The stride of the first convolution layer in the upstream task is set to 2 to adapt to the size of the input samples. The network structure in the downstream task is consistent with this flowchart before the layer of fc6.

## 2.3 Network Training

For the 3×3 puzzle, there are 9!= 362880 possible permutations in total. The
selection of permutation set also has a certain impact on the performance of the experiments, which will be explained in 2.4. An index is assigned to a predefined permutation set. Then one permutation is randomly selected to be sent to the network. It is also required to return a vector containing the probability value of each index.

The output of network can be regarded as the conditional probability density function of object space permutation

$$p(S|A_1, A_2, \dots, A_g) = p(S|F_1, F_2, \dots, F_g) \prod_{i=1}^{g} p(F_i|A_i) \quad (1)$$

where S is the permutation selected above, $A_i$ represents the i-th image block, and $F_i$ forms an intermediate feature representation. Our aim is to make feature $F_i$ have semantic attributes that can recognize the relative position between image blocks.

If only one Jigsaw puzzle was generated for each sample, it would be probable to learn only the information of absolute position, which will be irrelevant to our requirements. Hence, it is necessary to generate multiple Jigsaw puzzles for a sample.

If a location list $S = (L_1, L_2, \dots, L_g)$ is generated for each part of the configuration S, then $p(S|F_1, F_2, \dots, F_g)$ can be written as:

$$p(L_1, L_2, \dots, L_g|F_1, F_2, \dots, F_g) = \prod_{i=1}^{g} p(L_i|F_i) \quad (2)$$

In this way, the position of each block is determined by the corresponding features.

## 2.4 Shortcuts Prevention

In case of using shortcuts while solving Jigsaw puzzles, a series of measures are made. In the process of randomly extracting from the image block with the size of 85 × 85, the average gap of 21 pixels is specially divided, which can prevent learning low-level information. As mentioned above, an image is divided into 9 image blocks, with a total of 362880 different segmentation methods. Some block methods are very similar to the original image, but what we need are some methods that Jigsaw
puzzles permutation is quite different from the original image. Hence, Hamming
distance is required. Hamming distance indicates the number of different characters in
the corresponding position of two strings of the same length. According to the
Hamming distance before and after the reshuffle, we select the first 100 permutations
as the way to choose the puzzle.

## 2.5 Downstream Tasks

After the upstream task, the learned features are used for the pre-trained weight of the scene classification task on the SAR image dataset, and we apply the weight of the upstream task network to initialize all the convolutional layers of the downstream network. The convolution layer of the downstream task network is the same as that of the upstream task network, but the full connection layer is different. Hence, we choose to copy only the weights of the convolution layers. However, unlike the upstream task, the downstream task sets the stride of the first layer of convolution to 4, which conforms to the standard AlexNet structure. Then we turn to train the rest of the network to make scene classification for SAR image data.

# 3 Experiments

## 3.1 Dataset And Experiment Settings

The data used in this experiment are shown in Fig.2. The dataset of the SAR scene contains 25 different classes, and each class has 400 samples [13]. They are shot in Rosenheim, Toronto, Java, Colorado, Beijing and Hong Kong airport, in which abundant scenes can be found, such as large areas of farmland and forest areas, water surfaces and residential areas with different densities. These 25 scenes are named C1 to C25 respectively. Specifically, the 25 types of scenes are wasteland, airport runway, 3 types of water area, agriculture, 4 types of architecture, sparse residence, 4 types of dense residence, skyscraper, 2 types of river, 2 types of road, farmland, forest, grassland residence, railway track and vegetation farmland mixture. The size of each sample is 200 x 200. In the task of the Jigsaw puzzle reassembly of the SAR image block, the training set and test set are divided according to the ratio of 7:3, with 7000 training samples and 3000 test samples.

All the experiments are implemented on the Pytorch 1.8.0. The interface is Python 3.8 and the platform is an Intel Xeon(R) CPU, and an NVIDIA GeForce RTX 2080 Ti
under Ubuntu 18.04. The learning rate used in the upstream task is 0.01, and we set the training samples batch size and the test samples batch size to 64 and 32 respectively. The training epoch is 300. Additionally, the adopted attenuation way is to reduce the learning rate to 0.5 times the original one per 50 epochs.

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** SAR scene dataset used in our experiments. Each class is illustrated with one sample. C1 for wasteland; C2 for airport runway; C3, C4 and C5 for water area; C6 for agriculture; C7, C8, C9 and C10 for architecture; C11 for sparse residence; C12, C13, C14 and C15 for dense residence; C16 for skyscraper; C17 and C18 for river; C19 and C20 for road; C21 for farmland; C22 for forest; C23 for grassland residence; C24 for railway track; C25 for a mixture of vegetation and farmland.</figcaption></figure>

## 3.2 Upstream Task Results

The upstream task is regarded as a sorting task to judge whether the correct sorting is found. By solving Jigsaw puzzles, the network learned how to recognize each image block as an object part and how to assemble it into the object, from which the network learns the feature that is beneficial for the downstream task. We train the upstream task following the same experimental setting described in Section 3.1. The accuracy of the upstream task is calculated in the following style: first, the index value of a permutation randomly selected from the pre-defined permutation set is obtained as the label. Then the image is scrambled according to this index value and the scrambled image blocks are fed into the network, obtaining a probability vector. Finally, the index value corresponding to the largest value in the probability vector is regarded as the predicted result and compared with the label to calculate the accuracy.

The experimental results are shown in Table 3.

<table><caption>**Table3.** Training parameters of the upstream task.</caption><thead><tr><td>Upstream Tasks</td><td>Epoch</td><td>Training Time</td><td>Training/Testing Batch Size</td><td>Accuracy</td></tr></thead></table>
In the upstream task training, according to the number of training and test samples, we set the batch size of training and test to 64 and 32 respectively, and the initial learning rate is set to 0.01. It is found that the network converges after 300 epochs, and the accuracy of the upstream task is 72%.

## 3.3 Downstream Task Results

The goal of the downstream task is to fulfill SAR scene classification with labeled samples as few as possible. Hence, in the downstream task, a very small amount of samples is used to fine-tune the model. We select 1, 2 and 3 samples from each class to train the downstream scenes. The two samples in each class will include the situation of selecting one sample, and the three samples in each class will include the situation of selecting two samples, which constitutes incremental comparison settings to show the effect of choosing different samples on the experiment. As shown in Figure 3, the downstream task is fine-tuned. Hence, we set the learning rate to 0.001 and verify the influence of different batch sizes on the experimental results. The test set of the upstream task is still considered as the selected test set, and we test whether it is classified into the corresponding scene class. Then it is compared with the method that does not take the weight obtained from the upstream task as the pre-trained weight, to verify the effectiveness of this experimental method. The results obtained are shown in Table 3.

<figure><img src="image_7.png" /><figcaption>Fig. 3. The training samples for each class in downstream task.</figcaption></figure>
<table><caption>**Table4.** Experimental results of scene classification using 1 sample of each class for training in the downstream task.</caption><thead><tr><th>Number of Selected Samples Per Class</th><th>Whether to Use Upstream Task Training Weight</th><th>Training Batch Size</th><th>Classification Accuracy</th></tr></thead><tbody><tr><td>1</td><td>No</td><td>2</td><td>13.6%</td></tr><tr><td>1</td><td>No</td><td>4</td><td>12.6%</td></tr><tr><td>1</td><td>No</td><td>8</td><td>8.2%</td></tr><tr><td>1</td><td>No</td><td>16</td><td>7.5%</td></tr><tr><td>1</td><td>Yes</td><td>2</td><td><strong>25.0%</strong></td></tr><tr><td>1</td><td>Yes</td><td>4</td><td>17.3%</td></tr><tr><td>1</td><td>Yes</td><td>8</td><td>11.7%</td></tr><tr><td>1</td><td>Yes</td><td>16</td><td>10.8%</td></tr></tbody></table>

<table><caption>**Table5.** Experimental results of scene classification using 2 samples of each class for training in the downstream task.</caption><thead><tr><th>Number of Selected Samples Per Class</th><th>Whether to Use Upstream Task Training Weight</th><th>Training Batch Size</th><th>Classification Accuracy</th></tr></thead><tbody><tr><td>2</td><td>No</td><td>2</td><td>14.7%</td></tr><tr><td>2</td><td>No</td><td>4</td><td>15.0%</td></tr><tr><td>2</td><td>No</td><td>8</td><td>10.7%</td></tr><tr><td>2</td><td>No</td><td>16</td><td>6.5%</td></tr><tr><td>2</td><td>Yes</td><td>2</td><td><strong>33.4%</strong></td></tr><tr><td>2</td><td>Yes</td><td>4</td><td>30.2%</td></tr><tr><td>2</td><td>Yes</td><td>8</td><td>18.4%</td></tr><tr><td>2</td><td>Yes</td><td>16</td><td>11.8%</td></tr></tbody></table>

Table 4, 5 and 6 show the quantitative results of the different number of training samples, batch size and whether use the feature learned in the upstream task respectively. By comparing the results in Table 4, 5 and 6, it is obvious that the classification accuracy increases with the number of the sample, whether using the weights from the upstream task or not. This is also true for different batch size settings.

From each table, it can be observed that under the same parameter setting, the classification accuracy of the downstream task using weights learned from the upstream task is all higher than without it, which indicates the promotion of the
Jigsaw puzzle task to the scene classification task. To be pronounced, the highest accuracy using the weights from the Jigsaw puzzle task of each table is higher than that acquired from the scratch training by over 10 points. This is a promising indicator to improve performance with related upstream tasks, instead of directly using lots of labeled samples in the task.

<table><caption>**Table6.** Experimental results of scene classification using 3 samples of each class for training in the downstream task.</caption><thead><tr><th>Number of Selected Samples Per Class</th><th>Whether to Use Upstream Task Training Weight</th><th>Training Batch Size</th><th>Classification Accuracy</th></tr></thead><tbody><tr><td>3</td><td>No</td><td>2</td><td>23.2%</td></tr><tr><td>3</td><td>No</td><td>4</td><td>23.2%</td></tr><tr><td>3</td><td>No</td><td>8</td><td>16.5%</td></tr><tr><td>3</td><td>No</td><td>16</td><td>10.1%</td></tr><tr><td>3</td><td>Yes</td><td>2</td><td><strong>38.1%</strong></td></tr><tr><td>3</td><td>Yes</td><td>4</td><td>37.8%</td></tr><tr><td>3</td><td>Yes</td><td>8</td><td>23.6%</td></tr><tr><td>3</td><td>Yes</td><td>16</td><td>12.8%</td></tr></tbody></table>

It can be noticed from Table 4, 5 and 6 that the performance decreases as the batch size increases. One possible explanation for this result is that the generalization of the model decreases with the batch size increasing, especially in the case of a few training samples. Specifically, the accuracy drops a few from 2 to 4, but a lot when the batch size is chosen as 8 and 16. Considering both accuracy and efficiency, picking 4 as the default batch size is a good choice for future research.

## 4 Conclusions

In this paper, a self-supervised learning framework consisting of Jigsaw puzzles sorting learning and scene classification is constructed to achieve SAR scene classification with few samples. By solving Jigsaw puzzles, the network learns feature which is suited for the downstream task. The experimental results show a notable improvement for SAR scene classification with an appropriate upstream task. What's more, the results obtained with few samples have a huge potential in real SAR
applications. However, this is an initial research in this direction. Abundant experiments are still needed under more parameter settings and a larger dataset.

## 5 Acknowledgement

This work was supported in part by the National Natural Science Foundation of China under Grant 62171347,61877066, 61771379,62001355,62101405; the Foundation for Innovative Research Groups of the National Natural Science Foundation of China under Grant 61621005; the fundamental Research Funds for the Central Universities under Grant XJS211904; the Key Research and Development Program in Shaanxi Province of China under Grant 2019ZDLGY0305 and 2021ZDLGY0208; the Science and Technology Program in Xi'an of China under Grant XA2020-RGZNTJ-0021; 111 Project.

## 6 References

1. Wang H, Magagi R, Goita K. Polarimetric decomposition for monitoring crop growth status[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(6): 870-874.

2. Del Frate F, Latini D, Scappiti V. On neural networks algorithms for oil spill detection when applied to C-and X-band SAR[C]//2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, 2017: 5249-5251.

3. Dumitru C O, Cui S, Schwarz G, et al. Information content of very-high-resolution SAR images: Semantics, geospatial context, and ontologies[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 8(4): 1635-1650.

4. Chapelle O, Haffner P, Vapnik V N. Support vector machines for histogram-based image classification[J]. IEEE transactions on Neural Networks, 1999, 10(5): 1055-1064.

5. McNairn H, Kross A, Lapen D, et al. Early season monitoring of corn and soybeans with TerraSAR-X and RADARSAT-2[J]. International Journal of Applied Earth Observation and Geoinformation, 2014, 28: 252-259.

6. Zhao Z, Jia M, Wang L. High-resolution SAR image classification via multiscale local Fisher patterns[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(12): 10161-10178.

7. Zhang A, Yang X, Fang S, et al. Region level SAR image classification using deep features and spatial constraints[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 163: 36-48.
8. Li Y, Li X, Sun Q, et al. SAR image classification using CNN embeddings and metric learning[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 19: 1-5.

9. Doersch C, Gupta A, Efros A A. Unsupervised visual representation learning by context prediction[C]//Proceedings of the IEEE international conference on computer vision. 2015: 1422-1430.

10. Noroozi M, Favaro P. Unsupervised learning of visual representations by solving jigsaw puzzles[C]//European conference on computer vision. Springer, Cham, 2016: 69-84.

11. Richardson J T E, Vecchi T. A jigsaw-puzzle imagery task for assessing active visuospatial processes in old and young people[J]. Behavior Research Methods, Instruments, & Computers, 2002, 34(1): 69-82.

12. Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[J]. Advances in neural information processing systems, 2012, 25.

13. Zhao Z, Jia M, Wang L. High-Resolution SAR Image Classification via Multiscale Local Fisher Patterns[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(12): 10161-10178.