AF$_2$: Adaptive Focus Framework for Aerial Imagery Segmentation
Authors: Lin Huang
Qiyuan Dong
Jia Zhang
Lijun Wu
Jiang Bian
Tie-Yan Liu
Venue: ICLR 2022 Submitted (2022)

Abstract: As a specific semantic segmentation task, aerial imagery segmentation has been widely employed in high spatial resolution (HSR) remote sensing images understanding. Besides common issues (e.g. large scale variation) faced by general semantic segmentation tasks, aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. There have been some recent efforts that attempt to address this issue by proposing sophisticated neural network architectures, since they can be used to extract informative multi-scale feature representations and increase the discrimination of object boundaries. Nevertheless, many of them merely utilize those multi-scale representations in ad-hoc measures but disregard the fact that the semantic meaning of objects with various sizes could be better identified via receptive fields of diverse ranges. In this paper, we propose Adaptive Focus Framework (AF$_2$), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations generated by widely adopted neural network architectures. Particularly, a learnable module, called Adaptive Confidence Mechanism (ACM), is proposed to determine which scale of representation should be used for the segmentation of different objects. Comprehensive experiments show that AF$_2$ has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.


Under review as a conference paper at ICLR 2022
2: ADAPTIVE FOCUS FRAMEWORK FOR AERIAL IMAGERY SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
As a specific semantic segmentation task, aerial imagery segmentation has been widely employed in high spatial resolution (HSR) remote sensing images under- standing. Besides common issues (e.g. large scale variation) faced by general semantic segmentation tasks, aerial imagery segmentation has some unique chal- lenges, the most critical one among which lies in foreground- background imbalance. There have been some recent efforts that attempt to address this issue by propos- ing sophisticated neural network architectures, since they can be used to extract informative multi-scale feature representations and increase the discrimination of object boundaries. Nevertheless, many of them merely utilize those multi-scale representations in ad-hoc measures but disregard the fact that the semantic meaning of objects with various sizes could be better identified via receptive fields of diverse ranges. In this paper, we propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi- scale representations generated by widely adopted neural network architectures. Particularly, a learnable module, called Adaptive Confidence Mechanism (ACM), is proposed to determine which scale of representation should be used for the segmentation of different objects. Comprehensive experiments show that AF2 has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.
INTRODUCTION
Understanding geospatial objects, such as plants, buildings, vehicles, etc., in high spatial resolution (HSR) remote sensing images plays a vital role in land cover monitoring, urban management, and civil engineering. Aerial imagery segmentation, as a specific semantic segmentation task that assigns semantic category to each image pixel, has been widely leveraged in HSR remote sensing images understanding since it can provide semantic and location information for objects of interest.
Nonetheless, in addition some common issues in most semantic segmentation datasets (Caesar ct al. 2018 Cordts et al. 2016 Zhou et al. 2019). including large scale variation (Kirillov et al.
2019 Long et al. (2055) Ronneberger et al. 2015), complex scene (Chen al.
2017
ZZao 2Zao et al.
2017), , and indistinguishable object boundaries (Cheng et al. 2020 Kirillov et al.
2020
Zhen et al.
2020). aerial imagery segmentation has its own challenges. the most critical one among which lies
in bnreground-baccgroumm imbalance (Deng et al., 2019 2018 Li et al. 2021 Pang et al. 2019
Waqas Zamir et al. 2019 Xia al. 2018 Zheng al. 2020). Taking images in Fig. as examples, the foreground proportion can be extremely small, e.g. less than 1% for the leftmost image. Such acute imbalance could drastically increase the difficulty of object recognition, as even human eyes can hardly recognize them from the image. Moreover, the larger intra-class variance of the background objects may significantly increase the risk of false positive results (Li et al. 202T Zheng et al.) 2020).
Existing general semantic segmentation methods mainly pay attention to designing sophisticated neural network architectures that can obtain informative multi-scale feature representations (Chen
al. 2018 He et al. 2016 Kirillov et al. 2019 Sun et al. 2019 SZegedy et al. 2017) and highlight the object boundaries (Kirillov et al. 22200] Li et al. 2021 Zhen et al. 2020). To further address the foreground-background imbalance challenge of aerial imagery segmentation, some recent efforts have created more delicate modules in the neural networks to obtain superior results. For instance, Foreground- Aware Relation Network (FarSeg) (Zheng al. 2020) introduces a foreground-scene

---

Under review as a conference paper at ICLR 2022
Ground Thuth
Foregmond
0.11
0.28
Figure I: Illustration of aerial imagery segmentation.
Reprssmatiin uncer
Pael predetctt reults
Conegory
Canagery
Uncertain
Prodidied
Figure 2: Diagrammatic representation of Adaptive Focus Framework.
relation module to enhance the discrimination of foreground features as well as a foreground-aware optimization to alleviate foreground -ackuuuuuu imbalance problem. More recently, PointFlow (Li al. 2021 designs dual point matcher to select points from the salient area and object boundaries.
While these previous studies have demonstrated the effectiveness sophisticated neural network architecture design in both general and aerial imagery segmentation, most of them ignore the other side of the coin, i.c. how to efficiently utilize the multi-scale representations generated by complex neural networks. Intuitively, identify the semantic meaning of a large object, it more important for the model to leverage the representations obtained by wider receptive fields since they can represent the semantics of the whole large object rather than confusing the discrimination with interior details. On the other hand, for small objects, it is more efficient for the model to exploit the representations obtained by more concentrated receptive fields because they can focus the discrimination without being distracted by noisy context. Unfortunately, most of the existing general or aerial segmentation methods merely employ multi-scale representations ad-hoe ways, either simply concatenating them or arbitrarily using the final layer. This inevitably limits the potential of aerial imagery segmentation.
Inspired by the Shaatively-focusing process the human eye (Artal et al. 2006 Wikipedia con- tributors 2021), we propose novel Adaptive Focus Framework (AF2) in this paper, which adopts a hierarchical segmentation procedure for aerial imagery segmentation. The general idea of this framework is shown in Fig. Particularly, through any widely employed model structure (e.g. incoder-decoder structure), AF can obtain hierarchical representation maps based on different levels of receptive fields. The semantic segmentation procedure starts from the representation map created by the largest receptive field. After obtaining the segmentation result on this level, AF, will filter the pixels of low confidence segmentation and then carry out another round of segmentation
2

---

Under review as a conference paper at ICLR 2022
procedure finer- aner-grained feature map. The whole process will be conducted repeatedly until all the pixels have been predicted, or until there no finer aver-grained feature map. In AF2. an Adaptive Confidence Mechanism (ACM) is proposed to deal with pixel filtration. The confidence of pixel is defined as the highest value among the probabilities that the pixel belongs to each category, while an adaptive updated threshold is set filter low confidence pixels.
In summary, the main contributions of this paper include:
We propose Adaptive Focus Framework (AF2), which adopts hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations generated by widely adopted neural network architectures.
In the pixel filtering process for each scale's representation, the Adaptive Confidence Mechanism (ACM) is adopted to dynamically decide which pixels need to use finer-grained features. This mechanism ensures the performance and robustmess of the framework.
Extensive experiments and analysis demonstrate the advantage of AF2. It has significantly improved the accuracy on three typical aerial benchmarks. At the same time, its convergence speed and inference speed are also the fastest.
is worth noting that AF2 is a general framework, focusing particularly on efficient representation utilization, which can be leveraged to benefit any other semantic segmentation task.
2
RELATED WORKS
2.1 GENERAL SEMANTIC SEGMENTATION
2015)) are the earliest method using deep learning to model semantic segmentation problem. The backbone models (He et al. 2016 [Simonyan] & Zisserman 2014) Sun et al., 22119] Szegedy et al.,
Fully-convolutional networks (FCNs Long et al.
2017 2016 Xic et al. 2017) are used to generate lower resolution output than the input image and use bilinear up-sampling to recover the original image resolution. Employing the dilated convolution (Chen et al. 2017: 2018 Yu & Koltun| 2015) to replace the down-sampling operation will have better performance the expense of more memory and computation cost. The spatial context information can overcome the limited receptive field of convolution layer to a certain extent such as Atrous Spatial Pyramid Pooling (ASPP[Chen
Chen et al. (2017 2018)). Pyramid Pooling Module (PPM Zhao et al. (017 Densely connected Atrous Spatial -aaatinn-Aggmetted fully convolutional network (RA-Net Mou et al. (2019)), etc. PointRend (Kirillov et al.) 2020) performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. Some other
Pyramid Pooling (DenseASPP) Yang al. (2018)
works (Li et al., |200aa] Yuan et al.) 2020 Zhang et al. segmentation boundary that is difficult to predict.
2020 propose architectures specific for
The encoder-decoder architectures (Chen al. 2018 Kirillov et al. 2019 Lin et al. 2017a; Ron-
neberger et al., 2015] Takikawa al. 2019:] Li et al.] 2019) progressively upsample the high-level features and combine them with the features from lower levels, ultimately generating righrresolution features. For instance, Deeplab v3+ (Chen et al., 2018) combines dilated convolutions with an encoder-decoder structure produce the output
grid 4x sparser than the input. SeManticFPN (Kirillov et al.) 2019) merges the information from all levels of the Feature Pyramid Network (FPN) pyramid into single output and produces a dense prediction.
2.2
SEMANTIC SEGMENTATION FOR AERIAL IMAGERY
In recent years, employing deep learning to accelcrate the understanding of acrial image has received widespread attention (Kiiser et al. 2017 7 Kussul et al. 2017 Marcos al.) 20188] Marmanis et al.
2018
Scott et al.] (017), and benefits lot of applications such as agriculture vision (Arakeri et al.
2016 Kamilaris & Prenafcta-Boldi 2018 Patrício Rieder 2018), 1), road extraction (Bastani ct al.
2018 Ji al.) 2018). land cover mapping (Li et al. 2016 Malkin et al. 2018 Robinson et al. 2019) forest monitor (iioo et al. 2019 Yuan et al. 2015), etc. They often design delicate structures to ensure that general semantic segmentation migrates well for specific application scenarios. For instance, Relation Augmented network (RA-Net Mou et al. (2019)) proposes a spatial relation module and a channel relation module to explicitly model global relations. Foreground-Awarr Relation Network
3

---

Under review as a conference paper at ICLR 2022
Prefiminary Predictor
Adaptive Confidence Mechonism
Features
ASPP
Predictor4
P
Predictor3
sowpisgnping
Upvampling
Figure 3: Adaptive Foeus Framework. Cj and Fj represent different hierarchy feature maps. V means the pixel set fed into level I and we show them in gray. The adaptive confidence mechanism (ACM) employed to judge whether the prediction for each pixel sufficiently confident or not in each level, and is the sign of confidence. The detail of the red dashed box is shown in Fig. 4
(FarSeg Zheng et al. (2020) enhances the discrimination of foreground features and proposes
balanced optimization based on focal loss (Lin et al.) 2017b). PointFlow (Li et al. 202[ dual point matcher to select points from the salient area and object boundaries.
designs
METHOD
In this section, we discuss details proposed 2F2. As shown in Fig. 3 AF2 consists of three main parts: hierarchical features extractor, predictor, and adaptive confidence mechanism (ACM). The pseudo-code of Adaptive Focus Framework is shown appendix.
Hierarchical features extractor: Given an input image, this is used to extract the hier- archical feature maps of the image. The higher the level of feature map, the coarser the granularity of feature map and the larger the receptive field.
Preliminary predictor: For each level of the feature map, there a preliminary predictor to output the categories of the pixels that still cannot be confidently predicted with higher-level feature maps.
Adaptive confidence mechanism for prediction selection: The adaptive confidence mech- anism (ACM) is employed to judge whether the prediction for each pixel is sufficiently confident or not in each level. The predictions with high enough confidence will be accepted for the corresponding pixels. Otherwise, the pixels will be passed down to the next hierarchy for further prediction with finer-grained features, until there is no pixels for prediction or lowest level is reached.
3.1 HIERARCHICAL FEATURES EXTRACTOR
Hierarchical feature extractor is the fundamental part of the AF2. It prepares different levels of feature maps for further pixel-level classification. The higher the level of feature map. the larger the receptive field. Since AF2 is independent of the particular structure of the network, any mainstream feature extraction network can be adopted as long as it capable of extracting a hierarchy of feature maps.
The typical model structures, such as Fully-Convolttional Network (FCN et al. (2015)) Feature Pyramid Networks (FPN Lin al. (2017a)) Semantic FPN Kirilov al. (2019). etc., can be employed here. In this section, we take FPN (Lin et al., 2017a) with Atrous Spatial Pyramid Pooling module (ASPP) (Chen et al., (017) as an example. Specifically, ResNet (He et al. 2016) is chosen as the backbone network for basic feature extraction. We denote the different level of feature maps generated by ResNet C = [G|C R4x and W represent the image's 's original height and width. is the level of feature map, while Lmin and Tanax the lowest and highest levels (e.g. if the output stride of ResNet is 25, its I Tanax
I [Tmiin= /mmax].. where H
4

---

Under review as a conference paper at ICLR 2022
Confidence distribution
V4
2=label?
argmax
Update
Threshold
Fj
Predictor
max
Pt.
cua
Figure 4: The process of Adaptive Confidence Mechanism. Firstly, the corresponding feature map of pixels in Vi are fed into predictor to get predicted probabilities. Then, the threshold employed to judge whether the pixel should go down the hierarchy to get lower-grained features or output the prediction. Meanwhile, the threshold is periodically updated according the confidence distribution of correctly predicted pixels. Note that the threshold will be fixed without update during the inference.
5. The lowest feature map's output stride is always 2 and its Liinn is 2.). Furthermore, let F == { Fi E pdx # x # [Lmins Lmax] J. which stands for the the feature map set from the decoder part. Fi is defined as
f (ASPP(C)), Ci 0:), Lmax f (UP2(Fi-1)).) 0,), otherwise.
Fi =
where f(,,, Oi) represents the top-down feature fusion function. ASPP represents Atrous Spatial Pyramid Pooling Module (Chen et al.) 2017). UP2 represents the bilinear up-sampling function with scale factor 2.
3.2 PRELIMINARY PREDICTOR
Multiple pixel-level preliminary predictors are assigned for the different feature maps, and the goal of the predictor to output the category of the pixels that cannot be confidently predicted with higher-level features. Specifically, the feature map Fi is fed into its corresponding predictor for level ( to obtain the prediction results. Since the results belong to nx # = (n is the category number), the bilinear up-sampling method is employed to generate prediction with the same size the original image. The whole process can be formulated a
Pt SOFTMAX (UP22 (g (Fi 0:))),
(2)
where g(: 0z) is the prediction function based on multi-layer perceptron (MLP), and UP is the bilinear up-sampling method with scale factor 2
In fact, only part of the prediction results will be selected from P;. Specifically, we denote Prii.j as the prediction results vector in for pixel ( E Vi, where is the pixel set in the original image that still cannot be confidently predicted in higher levels like Eq. equation 4 Only these selected prediction results, i.e. {pt...) (i. E Vi), will be processed by ACM to determine whether they are adopted as the final results.
3.3 ADAPTIVE CONFIDENCE MECHANISM FOR PREDICTION SELECTION
The purpose of adaptive confidence mechanism (ACM) is to determine which pixels can be identified and which pixels need to be fed into the lower level for prediction. To achieve the above, the metric for each pixel is defined and is named as the pixel's confidence. Then, the filtration function is adopted based on this metric. The process of ACM is depicted in Fig.
Firstly, the pixel's confidence is defined to evaluate whether the corresponding prediction result is reliable enough (i.e. the larger the pixel's confidence, the more reliable the prediction). For the
Certainly, P. SOFTMAX (f (UPg) (Fi) ()) another option which performs up-sampling first.
5

---

Under review as a conference paper at ICLR 2022
pixel (i, Vi, the prediction result Pt. is n-dimensional vector to indicate the probabilities belonging to the categories. We define the highest value among these probabilities its confidence that is cft, = max (Pr,as) : (i.5) E Vi.
Secondly, we filter out some of the pixels to be fed into the lower level according their confidence. A straightforward way is to select the pixels with the top k-lowest confidence in each image. However, since the variance among different images, especially for aerial images, is very large, the top-k approach cannot efficiently handle such complicated situation. For instance, pixels in complicated images usually have low confidence, while pixels in simple images have relatively higher confidence. Therefore, we choose to use a uniform threshold for all images. Since the accuracy of prediction is improved during training, the threshold should also be learnable and adaptively updated with model training. More concretely, the statistical information of the confidence for correctly predicted pixels in each level employed to help update the threshold adaptively. That is
(3) where of is the threshold in step t for level I, is soft update factor, and QUANTILE, represents the r-quantile method. should be noted that the threshold is fixed without update in inference.
7f = of ()) QUANTILE, {cft.a, (t,j) Vi A max((.....))) label
Thus, compared with threshold m, the pixels with a lower confidence will be fed into the level - 1 like Eq. equation | except for the lowest level. Meanwhile, the categories the rest pixels will be determined based on the prediction results of this layer like Eq. equation 5 In the lowest level Lmnn+
we set Timin
0 so that all the pixels Vimm will determined.
Vi-1 ((1,5) (1,5) E Vi Affas <tt,, Linnn O [(i,j, arg max(pi,s)) (i,5) VI AAAff. 7
(5)
where Oj is the confident prediction in each level.
Since there number of pixels whose categories could be determined in each level, i.e., Ot for min < < Imax the final prediction results is a union of all these pixels O Ul=Amtn O₁.
3.4
OPTIMIZATION
The optimization objective function is based on cross-entropy loss. Due to the hierarchical prediction procedure adopted in AF2, the overall loss is accumulated at each level of the hierarchical predictions. Since pixels in Vi are fed into predictor in level 1, the total loss function be formulated follow:
== Σ
1
Σ (i.))V))
CROSS-ENTROPY (Pr,s label;s)
(6)
[E(Temina Iemax)
EXPERIMENTS
To evaluate the proposed method, we carry out comprehensive experiments on three aerial imagery datasets: iSAID, Vaihingen and Potsdam. We first introduce these datasets. Then, we show the experimental results on those four datasets. After that, we conduct further analysis examine the importance of each components of AF2. The implementation details are shown in appendix.
4.1 DATASETS
iSAID. iSAID Waqas Zamir et al., 2019 Xia et al. 2018) is the largest dataset for instance segmenta- tion in the HSR remote sensing imagery. It contains 2,806 high-resolution aerial images with 655,451 instance annotations from 15 categories. iSAID is distinguished from other semantic segmentation datasets for its significant imbalance between the annotated instances and background well as its scale variation even for the instances from the same category. In our experiments, we follow its default dataset split where 1,411 images are used for training, 458 for validation, and 937 for testing. The original images can be as large x000x 13000 pixels. Following previous work (Zheng et al.
Thh probability distribution entropy, the difference value between the first and second largest probability value, etc. are other optional choices for pixel's confidence.

---

Under review as a conference paper at ICLR 2022
Table 1: Experiments 3 aerial imagery datasets. For iSAID vol set, we show the mloU score and the category with significant improvement, such as: baseball court (BC), large vehicle (LV), helicopter (HC), swimming pool (SP) and roundabout (RA). All the experiments use ResNet-50 with weights pretrained ImageNet as backbone for fair comparison except HRNet (Sun et al. 2019).
Method
(S)
Vathingen (%)
Potsdam (8)
mIoU
BC
LV
mloU
m-F;
mIoU
FCN (Long 2015
83.1
PSPNet Zhao
2017
60.3
61.1
58.0
46.8
68.6
65.1
76.8
73.9
83.9
Ocnet
Yuan 2018
65.7
77.4
74.2
84.1
DenseASPP
Yang
2018
57.3
54.8
55.6
33.4
37.5
53.4
64.7
76.4
73.9
83.9
Deeplabv3+
CChen al
2018
61.5
56,6
60.3
34.5
41.4
65.1
64.3
76.0
74.1
83.9
SemanticFPN
Kiniloy et 2019
62.1
54.1
61.0
37.4
42.8
70.2
66.3
77.6
74.3
84.0
RefineNet
Deag et a. 20.00
60.2
61.1
58.2
23.0
43.4
65.6
UPerNet
Xiao et a..
2018
63.8
55.3
61.3
30.3
45.7
68.7
66.9
78.7
74.3
84.0
HRNet
Sun ml. 2019
61.5
59.4
62.1
14,9
44.2
52.9
66.9
78.2
73.4
83.4
GSCNN
TTkikaaw et 10
63.4
56.1
63.8
33.8
48.8
58.5
67.7
79.5
73.4
84.1
SFNet al.
20200
64.3
58.8
62.9
30.4
47.8
69.8
67.6
78.6
74.3
84.0
RANet Mou et
62.1
53.2
60.1
38.1
41.8
70.5
73.8
83.9
PointRend Kirilloy al.
2020
62.8
55.4
62.3
82.7
FarSog Zheng
63.7
62.1
60.6
35.8
83.3
PointFlow [1 nl. 2021
66.9
62.2
64.6
37.9
50.1
71.7
70.4
75.4
84.8
AF2-AFPN
67.8
66.2
67.3
38.9
53.1
77.0
70.5
82.1
74.9
84.4
2020 Li et al. 2021), these images are cropped into patches with fixed size of x96x896 with sliding window striding 512 pixels, and these models are trained with 16 epochs on cropped images for all experiments. We employ the mean intersection over union (mloU) evaluation metric.
Vaihingen and Potsdam. Vaihingen includes 33 acrial images with 2494 x 2064 pixels. Potsdam includes 38 aerial images with 6000 6000 pixels. categories are defined for both of them. Following the previous work (Li et al., 2021). images are cropped into patches with fixed sizes of 768 x 768 and 896 x 896, respectively. These models are trained with 200 epochs for all experiments. We use mloU and mean-Fi metrics to evaluate the proposed method.
4.2 RESULT COMPARISON
In this section, we conduct detailed experiments on iSAID, Vaihingen and Potsdam datasets to compare the final prediction performance and the inference efficiency of AF2 with several mainstream methods. The base model for hierarchical feature extractor adopted is combination of FPN (Lin al. 2017a and ASPP (Chen et al.) 2018). abbreviated AFPN for convenience. Our whole implementation is denoted as AFP-AFPN.
Result Comparison iSAID. As shown in Table. [ AF2-AFPN achieves the state-of-the-arr result among all previous works with r = 0.32 in ACM. The results also show significant improvement among some categories, which demonstrates again that AF, can identify the tiny instance with high accuracy. Mcanwhile, compared with previous methods, It shows that AF2-AFPN has exceeded most mainstream models terms of running efficiency. More details can be found in the appendix.
It is worth noting that the architecture re-design methods, such FarSeg (Zheng et al. 2020) and the best model before PointFlow (Li et al.) 2021). can be easily integrated with our model to further improve the performance. However, tweaking the architecture is not the focus of this work. Meanwhile, the source code of some models, e.g. PointFlow, not available yet. As one piece of future work, will verify the effectiveness of the combination of AF2 and other more advanced architectures.
Result Comparison Vaihingen and Potsdam. To further verify the performance of our frame- work, we also conduct experiments on two well-known aerial imagery datasets, i.e. Vaihingen and Potsdam. Compared with iSAID dataset, the categories relatively balanced in terms of instance shape and category ratio these two datasets. The quantitative results listed in Ta show that AF2 outperforms all methods except PointFlow. More details can be found in the appendix.

---

Under review as a conference paper at ICLR 2022
Final Prediction (Stacked)
Image
Ground Truth
Level
Lovel
Lovel
Iri rt-
Figure 5: Image prediction process show iSAID validation set with r 0.32. Different color means different category. Black means the background. Specifically, white means that the pixel has not been predicted has been predicted in higher level.
4.3 SENSITIVITY ANALYSIS
In this section, we conduct thorough analysis over iSAID to study the importance each modules of AF2. To better demonstrate the analysis results, we first present the process of image prediction over some examples from the validation set of iSAID in Figure 5 In this figure, columns to indicate the accepted prediction results level 4 to respectively, while the last column is the stacked final results. As we can see, at higher levels, it is more often the inner areas of the background or foreground that are predicted successfully. Lower level finer-grained features are primarily used to improve the prediction results of the edges. This hierarchical prediction method is independent the proportion of foreground and background and thus can effectively solve the problems existed previous method.
Study on the Effect of the ACM. ACM, as a core part of AF2, is designed to filter out the low confidence pixels. In ACM, the quantile ratio is key factor control the filtration threshold. To study its influence, we set different values of r in ACM. As shown in Table 2 the models integrated with AF₂ achieve significant improvement over the basic AFPN model. In addition, the experimental results show that the setting of r's value has a great influence on the results. Particularly, either too large or small will skew the model toward finer- or coarser- grained representations. Experimental results show that 0.3 a compromise value on the iSAID dataset, which can greatly improves the prediction accuracy over the baseline method. It is worth noting that, in the case of r = 0.9, where most pixels (96.0%) are segmented in 12vel-2, the mloU value has been improved by 2. %. This result indicates that the loss function for coarse gaansee features is beneficial.
Study on the Effect of Employed Levels. In this experiment, we study the influence on perfor- mance when employing feature maps from only a subset of levels in AF2. By default, feature maps from level-2, level-3, and level-4 are employed in the framework to produce the final result. However, as shown in Table3] when using only two levels' feature maps, the segmentation mloU dropped from 67.4 to 66.8 even in the best case. Note that only employing feature maps from level-2 and level-4 equivalent to FPN and FCN respectively. Experimental results demonstrate that the multi-scale feature maps utilization is crucial.
8

---

Under review as a conference paper at ICLR 2022
Table Study of ACM. The prediction ratio means how many pixels have completed prediction at
10.1 this level (i.e. #x xxW W ). The foreground ratio means foreground pixels proportion of the current pixel sets (i.c. 1 fg. (
Vill
Prediction Ratio (%)
Foreground Ratio (%)
Method
level-4
level-3
level-2
level-4
level-3
level-2
(9))
AFPK
100
63.3
AF2-AFPN
2.2
1.8
96.0
3.3
3.5
3.6
65.4
AFy-AFPN
0.7
9.7
16.3
74.0
3,3
3.8
4,3
63.8
AFPN
32.7
4.7
66,4
AFPN
0.3
73.7
20.0
6.3
3.3
8.8
27.6
67.4
AFP-APNN
0.1
91.4
7.0
1.6
3.3
22.7
41.3
64.7
Hierarchical Features Extractor with Different Neural Architectures. In this experiment, we aim to evaluate the extendibility and flexibility of the proposed AF2 in terms of feature extraction. Specifically, select 3 typical architectures: FCN (Long et al. (2015), SemanticFPN (Kirillov et al.) 2019), FPN+ASPP (Chen et al. 2017 Lin et al. 2017a) as the hicrarchical features extractor, which named FCN, SFPN and AFPN for convenience. For FCN, combine the feature of each level with all of its higher level feature maps to introduce richer semantic information into lower levels. As shown in Table with the assistance of AF2. AF2 narrows the gap among different architectures. Even for the the FCN architecture, which only has bottom-up up feature generation, AF2 can achieve about 8.4 improvement on mloU score. This result further illustrates the extendibility and superiority of AF2.
find that the different architectures have different degrees of score improvement
Table 3: Study of Employed Levels.
Table 4: Study of Hierarchical Features Extractor.
Method Level-4 Level-3 Level-2 mloU
Method
mloU
AFE-ARPN
57.9
FCN
57.9
AF2-APPN
60.7
SEPN
62.1
AFP-ARPN
63.1
AFPN
63.3
AFE-APPN
V
66.0
FFNN
66.3
AFE-APPN
v
66.7
SFPN
66.8
AFP-AFPN
V
66.8
AF2-AFPN
AFPNAPPN
V
67.4
Results on general segmentation benchmark We further verify our approach on general segmenta- tion dataset Cityscapes. show in Table[5] the experiment shows that our method has about 2% mloU improvement. The training, validation, testing data is 2975, 500, and 1525 respectively. We only use the fine-data for training. During training, data augmentation contains random horizontal flip, random cropping with the size 768 x 768. We train a totally 50k iterations with minibatch 16. use ResNet-50 with weights pretrained on ImageNet.
Table 5: Experiment on Cityscapes validation.
Methods mloU road SW BD wall fence pole TL TS VG terrain sky rider
truck bus train MT bicyole 0.95 0.82 0.63 0.95 0.71 0.82 0,58 0.68 0.77
SPPN 0,76 0.98 0.85 0.92 0,52 0.61 0.63 0.69 0.78 0.92 0.64
AF2-SFPN 0.78 0.98 0.86 0.93 0.57 0.62 0.66 0.71 0.80 0.93 0.65 0.95 0.83 0.66 0.95 0.76 0.84 0.70 0.67 0.78
SW. BD. TL, TS, VG, MT represents sidewalk, building. traffic light, traffic sign, vegetation, and motorcycle, respoctively.
CONCLUSION
In this paper, we argue that the lack of efficient utilization of multi-scale representations could be a bottleneck for accurate semantic segmentation on HSR aerial imagery, which is characterized by the huge scale variation of objects and the imbalance between foreground and background. We present AF2, i.e. Adaptive Focus Framework, to alleviate this critical but long-standing concern. AF₂ is independent of the specific architecture and is capable of adaptively utilizing multi-scale feature representations and producing the final result through the proposed Adaptive Confidence Mechanism. Extensive experiments and analyses have demonstrated its remarkable advantages in boosting segmentation accuracy and have proved its universality on common architectures and datasets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Megha P Arakeri et al. Computer vision based fruit grading system for quality evaluation of tomato in agriculture industry. Procedia Computer Science, 79:426-433 2016.
Pablo Artal, Antonio Benito, and Juan Tabernero. The human eye is an example of robust optical design. Journal of vision, 6(1):1-1, 2006.
Favyen Bastani, Songtao He, Sofiane Abbar, Mohammad Alizadeh, Hari Balakrishnan, Sanjay Chawla, Sam Madden, and David DeWitt. Roadtracer: Automatic extraction road networks from aerial images, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4720 4728, 2018.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05588 2017.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801 11888,8.1 2018.
Ho Kei Cheng, Jihoon Chung, Yu- Li-g Tai, and Chi- Chi-Keung Tang. Cascadepsp: Toward class agnossic and very high-resolution segmentation via global and local refinement. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8890-8899, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Zhipeng Deng, Hao Sun, Shilin Zhou, Juanping Zhao, Lin Lei, and Huanxin Zou. Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS journal of photogrammetry and remote sensing, 145:3-22 22, 2018
Zhipeng Deng, Hao Sun, Shilin Zhou, and Juanping Zhao. Learning deep ship detector in sar images from scratch. IEEE Transactions Geoscience and Remote Sensing, 57(6):4021-400 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shunping Shiqing Wei, and Meng Lu. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set. IEEE Transactions on Geoscience and Remote Sensing, 57(1):574 586. 2018
Zhentian Jiao, Youmin Zhang, Jing Xin, Lingxia Mu, Yingmin Yi, Han Liu, and Ding Liu. A deep learning based forest fire detection approach using uav and yolov3. In 2019 1st International Conference Industrial Artificial Intelligence (IAI), pp. 1-5. IEEE, 2019.
Pascal Kaiser, Jan Dirk Wegner, Aurélien Lucchi, Martin Jaggi, Thomas Hofmann, and Konrad Schindler. Learning aerial image segmentation from online maps. IEEE Transactions Geo- science and Remote Sensing, 1):605446606655 2017.
Andreas Kamilaris and Francese X PeenafetaBooddt Deep learning in agriculture: A survey. Com- puters and electronics in agriculture, 147:70-90, 2018
Alexander Kirillov, Ross Girshick, Kaiming He. and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408 2019.
10

---

Under review as a conference paper at ICLR 2022
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9799-9808, 2020.
Natalia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 14(5):778-782, 2017.
Weijia Li, Haohuan Fu, Le Yu, Peng Gong, Duole Feng, Congcong Li, and Nicholas Clinton. Stacked autoencoder- based deep learning for remote-sensing image classification: a case study of african land-cover mapping. International journal of remote sensing, 37(23):5632-5566 2016.
Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, and Kuiyuan Yang. Gff: Gated fully fusion for semantic segmentation. arXiv preprint arXiv:1904.014 01003, 2019.
Xiangtai Li. Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and
Yunhai Tong. Improving semantic segmentation via decoupled body and edge supervision. arXiv preprint arXiv:2007. 10035, 2020a.
Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In European Conference on Computer Vision, pp. 775-793. Springer, 2020b.
Xiangtai Li, Hao Hc, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semantics through points for acrial image segmentation. arXiv preprint 2XXi:2/03306656: 2021.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2017b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1431-3440, 2015.
Kolya Malkin, Caleb Robinson, Le Hou, Rachel Soobitsky, Jacob Czawlytko, Dimitris Samaras, Joel Saltz, Lueas Joppa, and Nebojsa Jojie. Label super-resolution networks. In International Conference on Learning Representations, 2018.
Diego Marcos, Michele Volpi, Benjamin Kellenberger, and Devis Tuia. Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models. ISPRS journal of photogranmetry and remote sensing, 105:96-177, 2018.
Dimitrios Marmanis, Konrad Schindler, Jan Dirk Wegner, Silvano Galliani, Mihai Datcu, and Uwe Stilla. Classification with edge: Improving semantic image segmentation with boundary detection. ISPRS Journal of Photogrammetry and Remote Sensing, 135:158-172, 2018.
Lichao Mou, Yuansheng Hua, and Xiao Xiang Zhu. A relation-augmentie fully convolutional network for semantic segmentation in aerial scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12416-12425 2019.
Jiangmiao Pang, Cong Li, Jianping Shi, Zhihai Xu, and Huajun Feng. R2-cnn: Fast tiny object detection in surge-scale remote sensing images. IEEE Transactions Geoscience and Remote Sensing, 57555555225 2019.
Diego Inácio Patrício and Rafael Rieder. Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review. Computers and electronics in agriculture, 153: 69-81, 2018.
11

---

Under review as a conference paper at ICLR 2022
Caleb Robinson, Le Hou, Kolya Malkin, Rachel Soobitsky, Jacob Czawlytko, Bistra Dilkina, and Nebojsa Jojic. Large scale high-resolution land cover mapping with multi-resolution data. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12726-12735, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. 234rinr,, 2015.
Grant J Scott, Matthew R England, William A Starms, Richard A Marcum, and Curt H Davis. Training deep convolutional neural networks for land-cover classification of high-resolution imagery. IEEE Geoscience and Remote Sensing Letters, 14(4):549-553, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 5693-5703, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-ve v4, inception- resnet and the impact of residual connections learning. Proceedings the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape cnns for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5229-5238, 2019.
Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in acrial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 28-33 28-37,22 2019.
Wikipedia contributors. Autofocus Wikipedia, the free encyclopedia, 2021. URL https: / en wikipedia owg/w/ accessed 24-April-202 2021].
/ wttp:///ttiiittttoooooooooooooo-
[Online;
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Tete Xiao, Yingeheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pp.
434, 2018
Saining Xie, Ross Girshick. Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmen- tation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3684-3692, 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 122 2015.
Chi Yuan, Youmin Zhang, and Zhixiang Liu. A survey technologies for automatic forest fire monitoring, detection, and fighting using unmanned aerial vehicles and remote sensing techniques. Canadian journal of forest research, 45(7):783-792, 2015.
12

---

Under review as a conference paper at ICLR 2022
Yuhui Yuan, Lang Huang. Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint S2t1:18090009000 2018.
Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refine- ment for segmentation. In European Conference on Computer Vision, pp. 489-506. Springer, 2020.
Li Zhang, Dan Xu, Anurag Arna,, and Philip HS Torr. Dynamic graph message passing networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
3726-3735, 2020.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881 2890, 2017.
Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
13666- 13675, 2020.
Zhuo Zheng, Yanfei Zhong, Junjue Wang, and Ailong Ma. Foreground relation network for geospatial object segmentation in high spatial resolution remote sensing imagery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 122,3) 2019.
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 PSEUDO-CODE OF ADAPTIVE FOCUS FRAMEWORK
Pseudo-code is shown in Alg. 22 which summarizes and describes how various stages work in
training testing mode.
Algorithm 1 Pscudo-code of Adaptive Focus Framework
Input: Image: X R
Output: Image segmentation
Initialize: VLmaz = {(i,j) [1, H.. [1]
2: Mode: train test
[Tmins Lmnax] 4: Preliminary predictor result: Pt Softmat (U'z (g (Fi 01)))
3: Feature map from FPN: (Fi Fi E R4
5: for I E {Imary Lmmar-1- -l.u.. Lmin do:
6: for each (1,j) E Vi do:
7: select pixel (i.j) 's preliminary prediction from Pf: Pt.ij Pr.(
8: calculate the confidence value: eft, max (Pta.), (i,j) E Vi
9: if cfi..., > 77 and # Lmin then:
10:
put (i,j, arg max(p(..g) into Or
else
12:
put (i,j) into Vi-1 end
13:
14:
if mode is train then:
15:
Adaptively update the threshold:7,
16: end if
17: end for
18: end for
19: Final prediction is: O = U Lmmax
Oj
Immrriin Tamin
20: if mode train then:
21:
optimize objective function: J = El |[Vil Eisee CrossEntropy(pL.. labelis)
22: end if
A.2 IMPLEMENTATION DETAILS
Following Chen al. (0088 ; Li al. (2021 Mou al. (2019): Zheng al. (2020).
ResNet-50 with weights pretrained on ImageNet in all the experiments for fair comparison except for HRNet (Sun et 2019). The output stride of the backbone adjusted to 16 by setting the convolution stride of the last layer to and employing dilated convolution following DeepLab v3+(Chen al., 2018). The feature maps from the backbone, i.e. C2. Cs, Ca, are the outputs of the conv2 x, comv3 and conv5_ x of the ResNet-50 respectively. All the confidence thresholds are initialized to be 0.5. In the threshold update, we set the soft intensity 7 to 0.9 and r to 0.3 for QUANTILFF- The learning rate decreases from 0.01 to 0.0001 following the poly policy, i.e. slep power where power is 0.9, We employ synchronized SGD over GPUs with each mini-batch containing 16 cropped patches, weight decay 0.0001 and momentum 0.9. The synchronized batch normalization is enabled for cross-GPU communication. Following Liet al. Mou et al. (2019): Zheng et al. 22020), for iSAID, we adopt data augmentation during the training, which includes horizontal flip, vertical flip, and rotations of 90, 180, and 270 degrees. Note that, for each experimental setting, we run the experiment 5 times and report the average score remove the influence of randomness.
Irstep Ir
mar mar__stpp
A.3 RESULTS ON ISAID DATASET
We show the detailed results on iSAID Dataset Tab. 0 and AF, achieves state-of-the-art for most categories.
14

---

Under review as a conference paper at ICLR 2022
Table 6: Experimental results on iSAID val set. The bold values in each column represent the best entries. The category are defined as: ship (Ship), storage tank (ST), baseball diamond (BD), tennis court (TC), baseball court (BC), ground field track (GTF), bridge (Bridge), large vehicle (LV), small vehicle (SV), helicopter (HC), swimming pool (SP), roundabout (RA), soccerball field (SBF), plane (Plane), harbor (Harbor). All the models are trained under the same setting following the FarSeg ZZhnng et al. 2020) and PointFlow (Li et al. 1021).
IoU cutegary(%)
Method
backbone
mloU((%)
Ship BD
ST
TC
BC
GTF
Bridge LV
SV
HC
SP
RA
SBF
Plane
Harhor
PSPNei (iiao
Resherki
60.3
52.1
75.7
85.6
61.2
12.5
430
10.9
46.8
71.
79.5
54.3
DemcASPP
Yang ei ml.
2018
RonNer50
57.3
55.7
63.5
67.2
81.7
54.8
52.6
34.7
55.6
36.3
33.4
37.5
53.4
73.3
74,7
46.7
3
Chen
ResNa50
34.2
41.
73.8
81.0
52.3
SemanticPNN
Kinlay et al
2
ResNer50
62.1
68.9
62.0
72.
85.4
54.1
48.9
44.9
61.0
48.6
37.4
42.8
70.2
61.6
81.7
54.9
RefineNet
Bong
ResNa10
60.2
32.6
12.4
13.4
79,9
51.
UPerNet
Xtao ot
201%
ResNec50
63.8
68.7
71.
73.1
85.5
55.3
57.3
43.0
61.3
45.6
30.3
45.7
68.7
75.1
84.3
56.2
HRNet
erall
STV
HRNeWW18
61.5
68.9
740
86.9
59.
33.8
4699
44.2
756
81.7
52.2
GSCNN
Takikawa el
2019
ResNet50
63.4
65,9
71.2 72.6
85.5
56.1
58.4
40.7
63.8
51.1
33.8
48.8
58.5
72.5
83,6
54.4
SFN
Li al
2020
ReNNa0
61.3
68.8
71. 72.
85.6
58.8
60.9
13.1
62.9
17.7
30
47.8
69.8
75.1
83.1
57.3
RANet
Mou
2019
ResNet50
62.1
67.1
61.3
72.5
85.1
53.2
47.1
45.3
60.1
49.3
38
41.8
70.5
58.8
83,1
55.6
PointRend
Kinllov dlall 1020
ReNNa50
62.8
64.4
69.9 73.7
82.9
55-4
61.1
38.5
62.3
18
29.8
45.0
66.0
72.7
80.7
51.0
FarSeg
20:0
ResNet50
63.7
65.4
61.
77.7
86.4
62.1
56.7
36.7
60.6
46.
51.
72.5
82.0
53.9
PrintPow
[3 0 al. 2121
ResNer50
66.9
70.3
74.7
77.8
87.7
62.2
59.5
45.2
64.6
50.
37.9
50.1
71.7
75.4
85.0
59.3
AFP-AFPN
ResNet50
67.8
73.7
89.9
52.0
53.1
74A
84.5
59.6
Efficiency Comparison: we compare the model size and inference speed on validation set in Figure 6 shows that AF2-AFPN has exceeded most mainstream models in terms of running efficiency.
DerwASPP
iPorama
KSOM
FSPiie
SerurecfPM
55
FPS
32 32
Figure 6: Speed (FPS) versus accuracy (mloU) on iSAID val set.
A.4 RESULTS ON VAIHINGEN DATASET
Vaihingen contains 33 images (of different sizes) and 6 categories have been defined. Following
previous work (Li et al. 2021), adopt large patches the iSAID dataset. We utilize 16 images for
training and the rest 17 images for testing. For training set, the image IDs are 1, 3, 5, 7, 11, 13, 15, 17, 21, 23, 26, 28, 30, 32, 34, 37. For validation set, the images IDs are 2, 4, 6, 10, 12, 14, 16, 20, 22, 24, 27, 29, 31, 33, 35, 38. We crop the images into 768 x 768 with a sliding window striding 512 pixels, and all the experiments are trained with 200 epochs. Like previous work, we use the mloU and m-Fi (i.e. the harmonic mean of precision and recall) as the main metric.
The detailed results on Vaihingen Dataset are shown in Tab. 7 The process of prediction is shown in Fig. 7 In Vaihingen dataset, the imbalance between foreground and background is relatively slight. Among all categories, only the cars are tiny, while the background is complex. Our framework also has a big improvement in these categories, which demonstrates the flexibility of AF,.
A.5 RESULTS ON POTSDAM DATASET
Li Potsdam contains 38 images (of different sizes) and categories have been defined. Following
et al. 2021), we utilize 24 images for training, and the image IDs are 2, 10. 2 11, 2, 12,
10, II.
12, 10, 4 11, 4_12, 5_ 10, 5_11, 5__551111 12, 6_7,6667666666ee6 6_8,6688 6_9, 6 10, 6 11, 12, 7 7_8, 7_9,
7_ 7 12. We utilize the another 14 images for test, and the image IDs are 2, 13, 2 34,
15

---

Under review as a conference paper at ICLR 2022
Table Experimental results on the Vaihingen Dataset. The results are reported with single scale input. The bold values in each column represent the best entries. The category are defined as: impervious surfaces (Imp.surf.), buildings (Build), low vegetation (Low veg), trees (Tree), cars (Car), cluster/background (Cluster). All the models are trained under the same setting following the PointFlow (Li al. 2021h.
Fi per category
Method
ml)
mean-F Fj
Imp.surf.
Build.
Low veg.
Tree
Car
Cluster
FCN Long el al. 2015
91.6
84.6
73.5
PSPNet Zhao et al 2017
65.1
76.8
88.4
92.8
79.2
85.9
73.5
41.0
OCNet(ASP-OCT Yuun ct al. 201
65.7
77.4
88.8
92.9
79.2
85.8
73.9
43.8
Denseaspp Yang et al.
2018
64.7
76.4
87.3
91.1
76.2
83.4
77.1
43.3
Deeplabv3+ Chen ct al.
2018
64.3
76.0
88.7
92.8
78.9
85.6
72.4
37.6
SemanticFPN
Kirillov ct
2019
66.3
77.6
89.6
93.6
79.7
86.3
75.7
40.7
UPerNet Xiao el
201
66.9
78.7
89.2
93.0
79.4
86.0
74,9
49.7
HRNet-WIX Sun
92.6
78.7
85.7 77.1
GSCNN (aaikaaa et al. 2019
4
92.6
85.4
77.9
52.9
SFNer (Li al. 2020h
67.6
78.6
90.0
94.0
80.3
86.5
78.9
41.9
RANet Mou et al. 2111
66.1
78.2
88.0
92.3
79.1
86.0
78.8
53.1
PointRend Kirillov et al. 2020
65.9
78.1
88.2
92.4
78.9
84.5
73.5
51.1
FarSeg Zheng al
2020
65.7
78.0
88.0
92.0
78.2
85.2
73.3
51.5
PointFlow
L1 al.
2021
70.4
81.9
90,1
93.6
77.7
85.4
80.0
64.6
AFA-APPN
70.5
82.1
89.6
93.4
79.8
86.2
79.8
63.5
Final Prediction
Image
Ground Truth
Level-4
Level-3
Level-2
(Stacked)
TAL
Figure 7: Image Prediction Process Show Case on Vaihingen val set.
16

---

Under review as a conference paper at ICLR 2022
Table 8: Experimental results on the Potsdam Dataset. The results are reported with single scale input. The bold values in each column represent the best entries. The category are defined as: impervious surfaces (Imp.surf.), buildings (Build), low vegetation (Low veg), trees (Tree), cars (Car), cluster/backkround (Cluster). All the models are trained under the same setting following the PointFlow (Li al. 2021).
Method
ml)U(())
mean-FF Fj
category
Imp.surf.
Build.
Low veg.
Tree
Car
Cluster
FCN (Log et
2015
73.1
83.1
90.2
94.7
84.1
85.6
89.2
54.8
PSPNet Yang et al
73.9
83.9
90.8
95.4
84.5
86.1
88.6
OCne((ASP-OCC)
Yuan el al. 2018
74.2
84.1
90.9
95.5
84.8
86.0
89.2
58.2
Denscaspp Yang ct
2018
73.9
83.9
90.8
95.4
84.6
86.0
88.5
58.1
Decplabv3+ Chen ct al.
2018
74.1
83.9
91.0
95.6
84.6
86.0
90.0
56.2
SemanticFPN
Kirillvv et 2019
74.3
84.0
91.0
95.5
84.9
85,9
90.4
56.3
UPerNer Xiao of al.
2018
74.3
84.0
90.9
95.7
85.0
86.0
90.2
56.2
HRNet-W18 (Sun
2019
73.4
83.4
90.4
94.9
84.2
85.4
90.0
55.5
GSCNN Takikaaa
2019
73.4
84.1
84.8
85.8 91.2
55.9
SFNei eL 20200
74.3
91.0
85.1
86.0 90.9
55.5
RANet Mou et al. 2019
73.8
83.9
90.8
92.1
84.3
86.8 88.9
PointRend Kinlllv el al. 2020
72.0
82.7
89.8
94.6
82.8
85.2
85.2
58.6
FarSeg Zheng ci al.
2020
73.4
83.3
90.7
95.2
84.3
85.3 90.2
54.1
PointFlow ([. et al.
202T
75.4
84.8
91.5
95.9
85.4
86.3
91.1
58.6
AF.-AFPN
74.9
84.4
91.1
95.7
85.0
86.2
91.5
57.0
Final Prediction
Image
Ground Truth
Level-1
Level-3
Level-2
(Stacked)
mnn MAT
ARD
Figure Image Prediction Process Show Case on Potsdam val set.
4_13, 4_15, 5 14, 5_15, 6_13,6133600 6_14, 6_15, 7_13. We crop the images into 896 896 with sliding window striding 512 pixels, and all the experiments are trained with 80 epochs. Like previous work, we use the mloU and m-F, as the main metric.
We show detailed results in Tab, g and the process of prediction in Fig. g for Potsdam dataset.
17


Reviews:
[{"review_id": "nE5pgBMDlOY", "reviewer": "Reviewer_VRMT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper proposes  Adaptive Focus Framework (AF2) to deal with semantic segmentation task of aerial imagery. The strengths of this paper is that the writing is clear and easy to understand. The structure of this paper is ok. The weakness is that the idea is simple and not novel enough. In addition, the experiments cannot verify the effectiveness of the proposed method.", "summary_of_the_paper": "In this paper, the Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multiscale representations generated by widely adopted neural network architectures, is proposed. Particularly, a learnable module, called Adaptive Confidence Mechanism (ACM), is proposed to determine which scale of representation should be used for the segmentation of different objects.", "summary_of_the_review": "The proposed AF2 tries to solve HSR aerial imagery semantic segmentation by making use of multi-scale representations. The whole  Adaptive Focus Framework (AF2) is simple, including Hierarchical features extractor, Preliminary predictor and Adaptive confidence mechanism for prediction selection (ACM).  \nAs the first two modules are popular in semantic segmentation and other related works, the main contribution of this work is the proposed ACM, which judges whether the prediction for each pixel is sufficiently confident or not in each level.  But actually, the proposed ACM is simple lacking novelty. The experiments in this work cannot verify the effectiveness of proposed method solidly. In Table 1, AF2-AFPN brings slightly better performance compared with PointFlow. On Potsdam, its performance is even worse. For general segmentation benchmark, only SFPN method is reported and compared, but in PointFlow, it has mIoU 80.3% on Cityscape. \n\nAll in all, i think this paper is not novel and solid enough to be accepted by ICLR.\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SHri1GvrZ3", "reviewer": "Reviewer_464N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe strength of the approach is that experimental results show improvement against state of  the art for mIoU score.\nIntuitively a multi-resolution approach (pyramid) to segment objects of various size is sound. \n\nHowever the paper lacks clarity and coherence:\n- foreground/background are mentioned extensively in the paper but these are not defined. How are these relating to the classes defined with the dataset used in the experiments?\n- class imbalance (between foreground/ background )   is mentioned as an issue in the abstract but it is unclear how this method addresses class imbalance.\n- the abstract also indicates that many techniques \"disregard the fact that the semantic meaning of objects with various sizes could be better identified via receptive fields of diverse ranges\" but it is unclear how the presented work addresses this shortcoming:  how semantic meaning of objects and association with size is used in the proposed pipeline? What are the objects segmented ? how are their real sizes translating into number of  pixels in the dataset used (  in aerial images are geolocated and correspond to an area (e.g. pixel == 50cm x 50cm) ) and relation to receptive fields of filters used?   In the results reported what objects correspond to \"small object\" and \"large object\" (e.g. a few objects are mentioned in Tab 1 , and as well in Tab.5)? \n- Several sentences lack clarity e.g.  \"For instance, pixels in complicated images usually have low confidence, while pixels in simple images have relatively higher confidence.\" What is a simple/complicated image?\n- mathematics are difficult to follow e.g. Eq 3\n", "summary_of_the_paper": "This paper proposes a pipeline for image segmentation. The  approach is validated  experimentally for remote sensing applications.\nA hierarchical segmentation procedure corresponding to a Feature Pyramid Networks (FPN) with Atrous  Spatial Pyramid is used.\n A preliminary detector take advantage of these feature maps for each resolution level (=layer) for computing class prediction to each pixel. In addition a confidence measurement to each pixel level prediction is also computed (Adaptive confidence mechanism).\nThe architecture is optimized using the cross entropy.\nResults show that the proposed pipeline improves the mIoU  scores compared to the state of the art techniques on three standard dataset.\n", "summary_of_the_review": "\nThe proposed method looks promising in improving segmentation results in remote sensing application. \nHowever clarity has to be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EMqCsONEbnA", "reviewer": "Reviewer_wDBh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points:\n\n1,  The paper is easy to follow and understanding.\n2,  The proposed methods achieve the state-of-the-art results on two ariel datasets(iSAID, Vaihingen).\n\n\nWeakness:\nSorted by the importance.  Here are the details:\n\n1, Several related works are not cited and compared.  These work may decrease the novelty of this paper. \nThe difference of this work and these dynamic network only lies on the prediction source.\nThis work adpot the predictions as route gating which are used in [1][2][3][4].\n\n[1] Learning Dynamic Routing for Semantic Segmentation CVPR-2020\n[2] Fine-Grained Dynamic Head for Object Detection NIPS-2020\n[3] Dynamic Routing Networks arxiv-2019 \n[4] Improving Video Instance Segmentation via Temporal Pyramid Routing arxiv-2021\n\nUsing predictions for further refine is not very novel and may bring extra computation cost.\n\n\n2, Several statements are not true. \n\"e.g. PointFlow, is not available yet\"\nI search for the Github. https://github.com/lxtGH/PFSegNets This work is already opensourced about half year ago.\n\n\n3, Several statements are not necessary which may be decribed in short sentences.\n\nSec.3.1 and Sec.3.4 may be not necessary since these are the common knowledges for segmentation. \n\n4,  Several detailed design are missing. The choices of ASPP should be claimed. \nWhy use ASPP in the head?\nWhat about the GFlops and FPS of proposed methods?\nWhat is the baseline in Tab3?\n\n5, What about the generalization of Adaptive Confidence Mechanism? Will it still work well on other scene understanding tasks or datasets?\nIf not, it may be not suitable for the ICLR submission.\n\n ", "summary_of_the_paper": "This paper propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multiscale representations generated by widely adopted neural network architectures.\nThe main contribution is a learnable module, called Adaptive Confidence Mechanism (ACM) to determine which scale of representation should be used for the segmentation of different objects. The method achieves the state-of-the-art results on two ariel datasets(iSAID, Vaihingen).", "summary_of_the_review": "Compared with exsiting methods on scene understanding, the proposed Adaptive Confidence Mechanism is little weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3LiMiZWMNu", "reviewer": "Reviewer_hDsQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. To my understanding, this paper essentially proposes a new strategy to merge multi-scale predictions, by choosing the high-confidence output of earlier levels, and passing low-confidence output to downstream levels. Similar strategy could be seen as early as Viola-Jones cascade classifier, but in the domain of semantic segmentation, I do not remember seeing similar design, so in that sense this paper does have some small novelty, though it's incremental.\n\n2. That said, the ideas of merging multi-scale predictions in semantic segmentation have already been studied in many existing work, from the early FCN [1], to more focused studies on this such as [2] and [3]. The proposed AF2 can be considered a hand-crafted strategy to decide which output scale to take, which could be better than naive method like average-pooling, but I'm not convinced it will work better than learnable attention such as in [2].\n\n   2.1. Even comparing to max-pooling, it's possible that AF2 will settle on one class in early scale after passing the threshold, but later scale may reveal an even higher confidence from a different class.\n\n   2.2. The main problem of the paper is it failed to compare to any of these multi-scale merging strategy. For example, [2] claimed the attention method can boost baseline model result by 6%, which is higher than AF2's boost on AFPN (these are not really comparable due to different settings, but I meant the paper should have a fair comparison on this aspect).\n\n   - [1] Fully Convolutional Networks for Semantic Segmentation, CVPR 2015\n\n   - [2] Attention to Scale: Scale-aware Semantic Image Segmentation, CVPR 2016\n\n   - [3] Hierarchical Multi-Scale Attention for Semantic Segmentation, 2020\n\n3. Besides, a few other issues or questions of the paper:\n\n   3.1. The paper highlights the \"adaptive\" threshold and claims top-k won't be effective on aerial images, but there is no experiment designed specifically to validate how much improvements the adaptive mechanism can bring over baselines like fixed threshold or top-k.\n\n   3.2. How is the \"soft update factor\" in Eq.(3) decided?\n\n   3.3. Is r set to a fixed value for experiments on all datasets?\n\n   3.4. In Sec. 4.3, the paper states \"This hierarchical prediction method is independent to the proportion of foreground and background and thus can effectively solve the problems existed in the previous method.\" - I don't understand the logic here. I agree the hierarchical design is independent to the foreground/background ratio, but then how can it also help solving imbalance issue if they're independent?", "summary_of_the_paper": "This paper proposes a hierarchical segmentation procedure for aerial imagery semantic segmentation. The proposed method adaptively filters pixels in each scale representation by dynamically adjusting the filtering confidence to decide pixels will be segmented by the next level. Experiments have shown the proposed method can boost existing model architectures' performance to achieve SotA on several datasets.", "summary_of_the_review": "The paper proposed a method to selectively merge multi-scale predictions, but failed to compare to any other methods aiming for similar goal (e.g. max-pooling, attention). There is no experiment to validate the \"adaptive\" design of the mechanism either (compared to fixed threshold or top-k). Therefore, I recommend reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four experts reviewed this paper and all recommended rejection. There was no rebuttal. The reviewers raised many concerns regarding the paper, such as missed citations, lack of comparison with related methods, and some presentation issues. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time.  The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject