Year: 2021

Abstract:
As a specific semantic segmentation task, aerial imagery segmentation has been widely employed in high spatial resolution (HSR) remote sensing images understanding. Besides common issues (e.g. large scale variation) faced by general semantic segmentation tasks, aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. There have been some recent efforts that attempt to address this issue by proposing sophisticated neural network architectures, since they can be used to extract informative multi-scale feature representations and increase the discrimination of object boundaries. Nevertheless, many of them merely utilize those multi-scale representations in ad-hoc measures but disregard the fact that the semantic meaning of objects with various sizes could be better identified via receptive fields of diverse ranges. In this paper, we propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multiscale representations generated by widely adopted neural network architectures. Particularly, a learnable module, called Adaptive Confidence Mechanism (ACM), is proposed to determine which scale of representation should be used for the segmentation of different objects. Comprehensive experiments show that AF2 has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.

heading: 1 INTRODUCTION
text: Understanding geospatial objects, such as plants, buildings, vehicles, etc., in high spatial resolution (HSR) remote sensing images plays a vital role in land cover monitoring, urban management, and civil engineering. Aerial imagery segmentation, as a specific semantic segmentation task that assigns a semantic category to each image pixel, has been widely leveraged in HSR remote sensing images understanding since it can provide semantic and location information for objects of interest.
Nonetheless, in addition to some common issues in most semantic segmentation datasets (Caesar et al., 2018; Cordts et al., 2016; Zhou et al., 2019), including large scale variation (Kirillov et al., 2019; Long et al., 2015; Ronneberger et al., 2015), complex scene (Chen et al., 2017; Zhao et al., 2017), and indistinguishable object boundaries (Cheng et al., 2020; Kirillov et al., 2020; Zhen et al., 2020), aerial imagery segmentation has its own challenges, the most critical one among which lies in foreground-background imbalance (Deng et al., 2019; 2018; Li et al., 2021; Pang et al., 2019; Waqas Zamir et al., 2019; Xia et al., 2018; Zheng et al., 2020). Taking images in Fig. 1 as examples, the foreground proportion can be extremely small, e.g. less than 1% for the leftmost image. Such acute imbalance could drastically increase the difficulty of object recognition, as even human eyes can hardly recognize them from the image. Moreover, the larger intra-class variance of the background objects may significantly increase the risk of false positive results (Li et al., 2021; Zheng et al., 2020).
Existing general semantic segmentation methods mainly pay attention to designing sophisticated neural network architectures that can obtain informative multi-scale feature representations (Chen et al., 2018; He et al., 2016; Kirillov et al., 2019; Sun et al., 2019; Szegedy et al., 2017) and highlight the object boundaries (Kirillov et al., 2020; Li et al., 2021; Zhen et al., 2020). To further address the foreground-background imbalance challenge of aerial imagery segmentation, some recent efforts have created more delicate modules in the neural networks to obtain superior results. For instance, Foreground-Aware Relation Network (FarSeg) (Zheng et al., 2020) introduces a foreground-scene
relation module to enhance the discrimination of foreground features as well as a foreground-aware optimization to alleviate foreground-background imbalance problem. More recently, PointFlow (Li et al., 2021) designs a dual point matcher to select points from the salient area and object boundaries.
While these previous studies have demonstrated the effectiveness of sophisticated neural network architecture design in both general and aerial imagery segmentation, most of them ignore the other side of the coin, i.e. how to efficiently utilize the multi-scale representations generated by complex neural networks. Intuitively, to identify the semantic meaning of a large object, it is more important for the model to leverage the representations obtained by wider receptive fields since they can represent the semantics of the whole large object rather than confusing the discrimination with interior details. On the other hand, for small objects, it is more efficient for the model to exploit the representations obtained by more concentrated receptive fields because they can focus on the discrimination without being distracted by noisy context. Unfortunately, most of the existing general or aerial segmentation methods merely employ multi-scale representations in ad-hoc ways, either simply concatenating them or arbitrarily using the final layer. This inevitably limits the potential of aerial imagery segmentation.
Inspired by the adaptively-focusing process of the human eye (Artal et al., 2006; Wikipedia contributors, 2021), we propose a novel Adaptive Focus Framework (AF2) in this paper, which adopts a hierarchical segmentation procedure for aerial imagery segmentation. The general idea of this framework is shown in Fig. 2. Particularly, through any widely employed model structure (e.g. encoder-decoder structure), AF2 can obtain hierarchical representation maps based on different levels of receptive fields. The semantic segmentation procedure starts from the representation map created by the largest receptive field. After obtaining the segmentation result on this level, AF2 will filter out the pixels of low confidence on segmentation and then carry out another round of segmentation
procedure on a finer-grained feature map. The whole process will be conducted repeatedly until all the pixels have been predicted, or until there is no finer-grained feature map. In AF2, an Adaptive Confidence Mechanism (ACM) is proposed to deal with pixel filtration. The confidence of a pixel is defined as the highest value among the probabilities that the pixel belongs to each category, while an adaptive updated threshold is set to filter out low confidence pixels.
In summary, the main contributions of this paper include:
• We propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations generated by widely adopted neural network architectures.
• In the pixel filtering process for each scale’s representation, the Adaptive Confidence Mechanism (ACM) is adopted to dynamically decide which pixels need to use finer-grained features. This mechanism ensures the performance and robustness of the framework.
• Extensive experiments and analysis demonstrate the advantage of AF2. It has significantly improved the accuracy on three typical aerial benchmarks. At the same time, its convergence speed and inference speed are also the fastest.
It is worth noting that AF2 is a general framework, focusing particularly on efficient representation utilization, which can be leveraged to benefit any other semantic segmentation task.

heading: 2 RELATED WORKS

heading: 2.1 GENERAL SEMANTIC SEGMENTATION
text: Fully-convolutional networks (FCNs Long et al. (2015)) are the earliest method of using deep learning to model semantic segmentation problem. The backbone models (He et al., 2016; Simonyan & Zisserman, 2014; Sun et al., 2019; Szegedy et al., 2017; 2016; Xie et al., 2017) are used to generate a lower resolution output than the input image and use bilinear up-sampling to recover the original image resolution. Employing the dilated convolution (Chen et al., 2017; 2018; Yu & Koltun, 2015) to replace the down-sampling operation will have a better performance at the expense of more memory and computation cost. The spatial context information can overcome the limited receptive field of convolution layer to a certain extent such as Atrous Spatial Pyramid Pooling (ASPP Chen et al. (2017; 2018)), Pyramid Pooling Module (PPM Zhao et al. (2017)) , Densely connected Atrous Spatial Pyramid Pooling (DenseASPP Yang et al. (2018)) , Relation-Augmented fully convolutional network (RA-Net Mou et al. (2019)), etc. PointRend (Kirillov et al., 2020) performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. Some other works (Li et al., 2020a; Yuan et al., 2020; Zhang et al., 2020) propose architectures specific for segmentation boundary that is difficult to predict.
The encoder-decoder architectures (Chen et al., 2018; Kirillov et al., 2019; Lin et al., 2017a; Ronneberger et al., 2015; Takikawa et al., 2019; Li et al., 2019) progressively upsample the high-level features and combine them with the features from lower levels, ultimately generating high-resolution features. For instance, Deeplab v3+ (Chen et al., 2018) combines dilated convolutions with an encoder-decoder structure to produce the output on a grid 4x sparser than the input. SemanticFPN (Kirillov et al., 2019) merges the information from all levels of the Feature Pyramid Network (FPN) pyramid into a single output and produces a dense prediction.

heading: 2.2 SEMANTIC SEGMENTATION FOR AERIAL IMAGERY
text: In recent years, employing deep learning to accelerate the understanding of aerial image has received widespread attention (Kaiser et al., 2017; Kussul et al., 2017; Marcos et al., 2018; Marmanis et al., 2018; Scott et al., 2017), and benefits a lot of applications such as agriculture vision (Arakeri et al., 2016; Kamilaris & Prenafeta-Boldú, 2018; Patrício & Rieder, 2018), road extraction (Bastani et al., 2018; Ji et al., 2018), land cover mapping (Li et al., 2016; Malkin et al., 2018; Robinson et al., 2019), forest monitor (Jiao et al., 2019; Yuan et al., 2015), etc. They often design delicate structures to ensure that general semantic segmentation migrates well for specific application scenarios. For instance, Relation Augmented network (RA-Net Mou et al. (2019)) proposes a spatial relation module and a channel relation module to explicitly model global relations. Foreground-Aware Relation Network
(FarSeg Zheng et al. (2020)) enhances the discrimination of foreground features and proposes a balanced optimization based on focal loss (Lin et al., 2017b). PointFlow (Li et al., 2021) designs a dual point matcher to select points from the salient area and object boundaries.

heading: 3 METHOD
text: In this section, we discuss details of the proposed AF2. As shown in Fig. 3, AF2 consists of three main parts: hierarchical features extractor, predictor, and adaptive confidence mechanism (ACM). The pseudo-code of Adaptive Focus Framework is shown in appendix.
• Hierarchical features extractor: Given an input image, this is used to extract the hierarchical feature maps of the image. The higher the level of feature map, the coarser the granularity of feature map and the larger the receptive field.
• Preliminary predictor: For each level of the feature map, there is a preliminary predictor to output the categories of the pixels that still cannot be confidently predicted with higher-level feature maps.
• Adaptive confidence mechanism for prediction selection: The adaptive confidence mechanism (ACM) is employed to judge whether the prediction for each pixel is sufficiently confident or not in each level. The predictions with high enough confidence will be accepted for the corresponding pixels. Otherwise, the pixels will be passed down to the next hierarchy for further prediction with finer-grained features, until there is no pixels for prediction or the lowest level is reached.

heading: 3.1 HIERARCHICAL FEATURES EXTRACTOR
text: Hierarchical feature extractor is the fundamental part of the AF2. It prepares different levels of feature maps for further pixel-level classification. The higher the level of feature map, the larger the receptive field. Since AF2 is independent of the particular structure of the network, any mainstream feature extraction network can be adopted as long as it is capable of extracting a hierarchy of feature maps.
The typical model structures, such as Fully-Convolutional Network (FCN Long et al. (2015)), Feature Pyramid Networks (FPN Lin et al. (2017a)) , Semantic FPN Kirillov et al. (2019), etc., can be employed here. In this section, we take FPN (Lin et al., 2017a) with Atrous Spatial Pyramid Pooling module (ASPP) (Chen et al., 2017) as an example. Specifically, ResNet (He et al., 2016) is chosen as the backbone network for basic feature extraction. We denote the different level of feature maps generated by ResNet as C = { Cl ∣∣Cl ∈ Rdl×H2l×W2l , l ∈ [Lmin, Lmax]}, where H
and W represent the image’s original height and width. l is the level of feature map, while Lmin and Lmax are the lowest and highest levels (e.g. if the output stride of ResNet is 25, its Lmax
is 5. The lowest feature map’s output stride is always 22, and its Lmin is 2.). Furthermore, let F = { Fl ∣∣Fl ∈ Rdl×H2l×W2l , l ∈ [Lmin, Lmax]}, which stands for the the feature map set from the
decoder part. Fl is defined as
Fl = { f (ASPP(Cl), Cl | θl) , l = Lmax f (UP2(Fl+1), Cl | θl) , otherwise. (1)
where f(·, · | θl) represents the top-down feature fusion function. ASPP represents Atrous Spatial Pyramid Pooling Module (Chen et al., 2017). UP2 represents the bilinear up-sampling function with a scale factor 2.

heading: 3.2 PRELIMINARY PREDICTOR
text: Multiple pixel-level preliminary predictors are assigned for the different feature maps, and the goal of the predictor is to output the category of the pixels that cannot be confidently predicted with higher-level features. Specifically, the feature map Fl is fed into its corresponding predictor for level l to obtain the prediction results. Since the results belong to Rn× H 2l ×W
2l (n is the category number), the bilinear up-sampling method is employed to generate a prediction with the same size as the original image. The whole process can be formulated as1:
Pl = SOFTMAX (UP2l (g (Fl | θl))) , (2)
where g(· | θl) is the prediction function based on multi-layer perceptron (MLP), and UP2l is the bilinear up-sampling method with a scale factor 2l.
In fact, only a part of the prediction results will be selected from Pl. Specifically, we denote pl,i,j as the prediction results vector in Pl for pixel (i, j) ∈ Vl, where Vl is the pixel set in the original image that still cannot be confidently predicted in higher levels like Eq. equation 4. Only these selected prediction results, i.e. {pl,i,j | (i, j) ∈ Vl}, will be processed by ACM to determine whether they are adopted as the final results.

heading: 3.3 ADAPTIVE CONFIDENCE MECHANISM FOR PREDICTION SELECTION
text: The purpose of adaptive confidence mechanism (ACM) is to determine which pixels can be identified and which pixels need to be fed into the lower level for prediction. To achieve the above, the metric for each pixel is defined and is named as the pixel’s confidence. Then, the filtration function is adopted based on this metric. The process of ACM is depicted in Fig. 4.
Firstly, the pixel’s confidence is defined to evaluate whether the corresponding prediction result is reliable enough (i.e. the larger the pixel’s confidence, the more reliable the prediction). For the
1Certainly, Pl = SOFTMAX (f (UP2l (Fl) | θ)) is another option which performs up-sampling first.
pixel (i, j) ∈ Vl, the prediction result pl,i,j is an n-dimensional vector to indicate the probabilities belonging to the n categories. We define the highest value among these probabilities as its confidence2, that is cfl,i,j = max (pl,i,j) , (i, j) ∈ Vl. Secondly, we filter out some of the pixels to be fed into the lower level according to their confidence. A straightforward way is to select the pixels with the top k-lowest confidence in each image. However, since the variance among different images, especially for aerial images, is very large, the top-k approach cannot efficiently handle such complicated situation. For instance, pixels in complicated images usually have low confidence, while pixels in simple images have relatively higher confidence. Therefore, we choose to use a uniform threshold for all images. Since the accuracy of prediction is improved during training, the threshold should also be learnable and adaptively updated with model training. More concretely, the statistical information of the confidence for correctly predicted pixels in each level is employed to help update the threshold adaptively. That is
τ t+1l = γ · τ t l + (1− γ) · QUANTILEr {cfl,i,j | (i, j) ∈ Vl ∧ argmax(pl,i,j) = labeli,j} , (3)
where τ tl is the threshold in step t for level l, γ is soft update factor, and QUANTILEr represents the r-quantile method. It should be noted that the threshold is fixed without update in inference.
Thus, compared with threshold τl, the pixels with a lower confidence will be fed into the level l − 1 like Eq. equation 4 except for the lowest level. Meanwhile, the categories of the rest pixels will be determined based on the prediction results of this layer like Eq. equation 5. In the lowest level Lmin, we set τLmin = 0 so that all the pixels VLmin will be determined.
Vl−1 = {(i, j) | (i, j) ∈ Vl ∧ cfl,i,j < τ tl }, l 6= Lmin, (4) Ol = {(i, j, argmax(pl,i,j)) | (i, j) ∈ Vl ∧ cfl,i,j ≥ τ tl }, (5)
where Ol is the confident prediction in each level.
Since there are a number of pixels whose categories could be determined in each level, i.e., Ol for lmin ≤ l ≤ lmax, the final prediction results is a union of all these pixels O = ⋃Lmax l=Lmin Ol.

heading: 3.4 OPTIMIZATION
text: The optimization objective function is based on cross-entropy loss. Due to the hierarchical prediction procedure adopted in AF2, the overall loss is accumulated at each level of the hierarchical predictions. Since pixels in Vl are fed into predictor in level l, the total loss function can be formulated as follow:
J = ∑
l∈[Lmin,Lmax]
 1 ||Vl|| ∑ (i,j)∈Vl CROSS-ENTROPY (pl,i,j , labeli,j)  . (6)

heading: 4 EXPERIMENTS
text: To evaluate the proposed method, we carry out comprehensive experiments on three aerial imagery datasets: iSAID, Vaihingen and Potsdam. We first introduce these datasets. Then, we show the experimental results on those four datasets. After that, we conduct further analysis to examine the importance of each components of AF2. The implementation details are shown in appendix.

heading: 4.1 DATASETS
text: iSAID. iSAID (Waqas Zamir et al., 2019; Xia et al., 2018) is the largest dataset for instance segmentation in the HSR remote sensing imagery. It contains 2,806 high-resolution aerial images with 655,451 instance annotations from 15 categories. iSAID is distinguished from other semantic segmentation datasets for its significant imbalance between the annotated instances and background as well as its scale variation even for the instances from the same category. In our experiments, we follow its default dataset split where 1,411 images are used for training, 458 for validation, and 937 for testing. The original images can be as large as 4000×13000 pixels. Following previous work (Zheng et al.,
2The probability distribution entropy, the difference value between the first and second largest probability value, etc. are other optional choices for pixel’s confidence.
2020; Li et al., 2021), these images are cropped into patches with a fixed size of 896×896 with a sliding window striding 512 pixels, and these models are trained with 16 epochs on cropped images for all experiments. We employ the mean intersection over union (mIoU) as evaluation metric.
Vaihingen and Potsdam. Vaihingen includes 33 aerial images with 2494×2064 pixels. Potsdam includes 38 aerial images with 6000×6000 pixels. 6 categories are defined for both of them. Following the previous work (Li et al., 2021), images are cropped into patches with fixed sizes of 768×768 and 896×896, respectively. These models are trained with 200 epochs for all experiments. We use mIoU and mean-F1 metrics to evaluate the proposed method.

heading: 4.2 RESULT COMPARISON
text: In this section, we conduct detailed experiments on iSAID, Vaihingen and Potsdam datasets to compare the final prediction performance and the inference efficiency of AF2 with several mainstream methods. The base model for hierarchical feature extractor adopted is a combination of FPN (Lin et al., 2017a) and ASPP (Chen et al., 2018), abbreviated as AFPN for convenience. Our whole implementation is denoted as AF2-AFPN.
Result Comparison on iSAID. As shown in Table. 1, AF2-AFPN achieves the state-of-the-art result among all previous works with r = 0.32 in ACM. The results also show a significant improvement among some categories, which demonstrates again that AF2 can identify the tiny instance with high accuracy. Meanwhile, compared with previous methods, It shows that AF2-AFPN has exceeded most mainstream models in terms of running efficiency. More details can be found in the appendix.
It is worth noting that the architecture re-design methods, such as FarSeg (Zheng et al., 2020) and the best model before PointFlow (Li et al., 2021), can be easily integrated with our model to further improve the performance. However, tweaking the architecture is not the focus of this work. Meanwhile, the source code of some models, e.g. PointFlow, is not available yet. As one piece of future work, we will verify the effectiveness of the combination of AF2 and other more advanced architectures.
Result Comparison on Vaihingen and Potsdam. To further verify the performance of our framework, we also conduct experiments on two well-known aerial imagery datasets, i.e. Vaihingen and Potsdam. Compared with iSAID dataset, the categories are relatively balanced in terms of instance shape and category ratio in these two datasets. The quantitative results listed in Table 1 show that AF2 outperforms all methods except PointFlow. More details can be found in the appendix.

heading: 4.3 SENSITIVITY ANALYSIS
text: In this section, we conduct thorough analysis over iSAID to study the importance of each modules of AF2. To better demonstrate the analysis results, we first present the process of image prediction over some examples from the validation set of iSAID in Figure 5. In this figure, columns 3 to 5 indicate the accepted prediction results at level 4 to 2, respectively, while the last column is the stacked final results. As we can see, at higher levels, it is more often the inner areas of the background or foreground that are predicted successfully. Lower level finer-grained features are primarily used to improve the prediction results of the edges. This hierarchical prediction method is independent to the proportion of foreground and background and thus can effectively solve the problems existed in the previous method.
Study on the Effect of the ACM. ACM, as a core part of AF2, is designed to filter out the low confidence pixels. In ACM, the quantile ratio is a key factor to control the filtration threshold. To study its influence, we set different values of r in ACM. As shown in Table 2, the models integrated with AF2 achieve significant improvement over the basic AFPN model. In addition, the experimental results show that the setting of r’s value has a great influence on the results. Particularly, either too large or too small r will skew the model toward finer- or coarser- grained representations. Experimental results show that 0.3 is a compromise value on the iSAID dataset, which can greatly improves the prediction accuracy over the baseline method. It is worth noting that, in the case of r = 0.9, where most pixels (96.0%) are segmented in level-2, the mIoU value has been improved by 2.1%. This result indicates that the loss function for coarse-grained features is beneficial.
Study on the Effect of Employed Levels. In this experiment, we study the influence on performance when employing feature maps from only a subset of levels in AF2. By default, feature maps from level-2, level-3, and level-4 are employed in the framework to produce the final result. However, as shown in Table3, when using only two levels’ feature maps, the segmentation mIoU dropped from 67.4 to 66.8 even in the best case. Note that only employing feature maps from level-2 and level-4 is equivalent to FPN and FCN respectively. Experimental results demonstrate that the multi-scale feature maps utilization is crucial.
Table 2: Study of ACM. The prediction ratio means how many pixels have completed prediction at this level (i.e. ‖Ol‖
H×W ). The foreground ratio means foreground pixels proportion of the current pixel
sets (i.e. ‖{labeli,j is fg. | (i,j)∈Vl}‖‖Vl‖ ).
Method r Prediction Ratio (%) Foreground Ratio (%) mIoU (%)level-4 level-3 level-2 level-4 level-3 level-2 AFPN - - - 100 - - 3.3 63.3
AF2-AFPN 0.9 2.2 1.8 96.0 3.3 3.5 3.6 65.4 AF2-AFPN 0.7 9.7 16.3 74.0 3.3 3.8 4.3 65.8 AF2-AFPN 0.5 32.7 41.2 26.0 3.3 4.7 8.8 66.4 AF2-AFPN 0.3 73.7 20.0 6.3 3.3 8.8 27.6 67.4 AF2-AFPN 0.1 91.4 7.0 1.6 3.3 22.7 41.3 64.7
Hierarchical Features Extractor with Different Neural Architectures. In this experiment, we aim to evaluate the extendibility and flexibility of the proposed AF2 in terms of feature extraction. Specifically, we select 3 typical architectures: FCN (Long et al., 2015), SemanticFPN (Kirillov et al., 2019), FPN+ASPP (Chen et al., 2017; Lin et al., 2017a) as the hierarchical features extractor, which are named as FCN, SFPN and AFPN for convenience. For FCN, we combine the feature of each level with all of its higher level feature maps to introduce richer semantic information into lower levels. As shown in Table 4, we find that the different architectures have different degrees of score improvement with the assistance of AF2. AF2 narrows the gap among different architectures. Even for the the FCN architecture, which only has a bottom-up feature generation, AF2 can achieve about 8.4 improvement on mIoU score. This result further illustrates the extendibility and superiority of AF2.
Results on general segmentation benchmark We further verify our approach on general segmentation dataset Cityscapes. As show in Table 5, the experiment shows that our method has about 2% mIoU improvement. The training, validation, testing data is 2975, 500, and 1525 respectively. We only use the fine-data for training. During training, data augmentation contains random horizontal flip, random cropping with the size 768×768. We train a totally 50k iterations with a minibatch 16. We use ResNet-50 with weights pretrained on ImageNet.

heading: 5 CONCLUSION
text: In this paper, we argue that the lack of efficient utilization of multi-scale representations could be a bottleneck for accurate semantic segmentation on HSR aerial imagery, which is characterized by the huge scale variation of objects and the imbalance between foreground and background. We present AF2, i.e. Adaptive Focus Framework, to alleviate this critical but long-standing concern. AF2 is independent of the specific architecture and is capable of adaptively utilizing multi-scale feature representations and producing the final result through the proposed Adaptive Confidence Mechanism. Extensive experiments and analyses have demonstrated its remarkable advantages in boosting segmentation accuracy and have proved its universality on common architectures and datasets.

heading: A APPENDIX
text: A.1 PSEUDO-CODE OF ADAPTIVE FOCUS FRAMEWORK
Pseudo-code is shown in Alg. 22 which summarizes and describes how various stages work in training or testing mode.
Algorithm 1 Pseudo-code of Adaptive Focus Framework
Input: Image: X ∈ Rc×H×W Output: Image segmentation
1: Initialize: VLmax = {(i, j) | i ∈ [1, H], j ∈ [1,W ]} 2: Mode: train or test 3: Feature map from FPN: {Fl ∣∣Fl ∈ Rdl×H2l×W2l , l ∈ [Lmin, Lmax]} 4: Preliminary predictor result: Pl = Softmax (Up2l (g (Fl | θl))) 5: for l ∈ {Lmax, Lmax−1, ..., Lmin} do: 6: for each (i, j) ∈ Vl do: 7: select pixel (i,j) ’s preliminary prediction from Pl: pl,i,j = Pl,(i,j) 8: calculate the confidence value: cfl,i,j = max (pl,i,j) , (i, j) ∈ Vl 9: if cfl,i,j > τl and l 6= Lmin then:
10: put (i, j, argmax(pl,i,j)) into Ol 11: else 12: put (i, j) into Vl−1 13: end if 14: if mode is train then: 15: Adaptively update the threshold:τl 16: end if 17: end for 18: end for 19: Final prediction is: O = ⋃Lmax l=Lmin
Ol 20: if mode is train then: 21: optimize objective function: J = ∑ l 1 ||Vl|| ∑ (i,j)∈Vl CrossEntropy(pl,i,j , labeli,j) 22: end if
A.2 IMPLEMENTATION DETAILS
Following Chen et al. (2018); Li et al. (2021); Mou et al. (2019); Zheng et al. (2020), we use ResNet-50 with weights pretrained on ImageNet in all the experiments for fair comparison except for HRNet (Sun et al., 2019). The output stride of the backbone is adjusted to 16 by setting the convolution stride of the last layer to 1 and employing dilated convolution following DeepLab v3+(Chen et al., 2018). The feature maps from the backbone, i.e. C2, C3, C4, are the outputs of the conv2_x, conv3_x, and conv5_x of the ResNet-50 respectively. All the confidence thresholds are initialized to be 0.5. In the threshold update, we set the soft intensity γ to 0.9 and r to 0.3 for QUANTILEr. The learning rate decreases from 0.01 to 0.0001 following the poly policy, i.e. lrstep = lrinit(1 − stepmax_step )
power, where power is 0.9. We employ synchronized SGD over 4 GPUs with each mini-batch containing 16 cropped patches, weight decay of 0.0001 and momentum of 0.9. The synchronized batch normalization is enabled for cross-GPU communication. Following Li et al. (2021); Mou et al. (2019); Zheng et al. (2020), for iSAID, we adopt data augmentation during the training, which includes horizontal flip, vertical flip, and rotations of 90, 180, and 270 degrees. Note that, for each experimental setting, we run the experiment 5 times and report the average score to remove the influence of randomness.
A.3 RESULTS ON ISAID DATASET
We show the detailed results on iSAID Dataset in Tab. 6, and AF2 achieves state-of-the-art for most categories.
Efficiency Comparison: we compare the model size and inference speed on validation set in Figure 6. It shows that AF2-AFPN has exceeded most mainstream models in terms of running efficiency.
A.4 RESULTS ON VAIHINGEN DATASET
Vaihingen contains 33 images (of different sizes) and 6 categories have been defined. Following previous work (Li et al., 2021), we adopt large patches as the iSAID dataset. We utilize 16 images for training and the rest 17 images for testing. For training set, the image IDs are 1, 3, 5, 7, 11, 13, 15, 17, 21, 23, 26, 28, 30, 32, 34, 37. For validation set, the images IDs are 2, 4, 6, 8, 10, 12, 14, 16, 20, 22, 24, 27, 29, 31, 33, 35, 38. We crop the images into 768×768 with a sliding window striding 512 pixels, and all the experiments are trained with 200 epochs. Like previous work, we use the mIoU and m-F1 (i.e. the harmonic mean of precision and recall) as the main metric.
The detailed results on Vaihingen Dataset are shown in Tab. 7. The process of prediction is shown in Fig. 7. In Vaihingen dataset, the imbalance between foreground and background is relatively slight. Among all categories, only the cars are tiny, while the background is complex. Our framework also has a big improvement in these categories, which demonstrates the flexibility of AF2.
A.5 RESULTS ON POTSDAM DATASET
Potsdam contains 38 images (of different sizes) and 6 categories have been defined. Following (Li et al., 2021), we utilize 24 images for training, and the image IDs are 2_10, 2_11, 2_12, 3_10, 3_11, 3_12, 4_10, 4_11, 4_12, 5_10, 5_11, 5_12, 6_7, 6_8, 6_9, 6_10, 6_11, 6_12, 7_7, 7_8, 7_9, 7_10, 7_11, 7_12. We utilize the another 14 images for test, and the image IDs are 2_13, 2_14, 3_13, 3_14,
4_13, 4_14, 4_15, 5_13, 5_14, 5_15, 6_13, 6_14, 6_15, 7_13. We crop the images into 896×896 with a sliding window striding 512 pixels, and all the experiments are trained with 80 epochs. Like previous work, we use the mIoU and m-F1 as the main metric.
We show detailed results in Tab. 8 and the process of prediction in Fig. 8 for Potsdam dataset.

Review summaries:
This paper proposes a hierarchical segmentation procedure for aerial imagery semantic segmentation. The proposed method adaptively filters pixels in each scale representation by dynamically adjusting the filtering confidence to decide pixels will be segmented by the next level. Experiments have shown the proposed method can boost existing model architectures' performance to achieve SotA on several datasets.

This paper propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multiscale representations generated by widely adopted neural network architectures. The main contribution is a learnable module, called Adaptive Confidence Mechanism (ACM) to determine which scale of representation should be used for the segmentation of different objects. The method achieves the state-of-the-art results on two ariel datasets(iSAID, Vaihingen).

This paper proposes a pipeline for image segmentation. The  approach is validated  experimentally for remote sensing applications. A hierarchical segmentation procedure corresponding to a Feature Pyramid Networks (FPN) with Atrous  Spatial Pyramid is used.  A preliminary detector take advantage of these feature maps for each resolution level (=layer) for computing class prediction to each pixel. In addition a confidence measurement to each pixel level prediction is also computed (Adaptive confidence mechanism). The architecture is optimized using the cross entropy. Results show that the proposed pipeline improves the mIoU  scores compared to the state of the art techniques on three standard dataset.