AF$_2$: Adaptive Focus Framework for Aerial Imagery Segmentation

1 INTRODUCTION . Understanding geospatial objects , such as plants , buildings , vehicles , etc. , in high spatial resolution ( HSR ) remote sensing images plays a vital role in land cover monitoring , urban management , and civil engineering . Aerial imagery segmentation , as a specific semantic segmentation task that assigns a semantic category to each image pixel , has been widely leveraged in HSR remote sensing images understanding since it can provide semantic and location information for objects of interest . Nonetheless , in addition to some common issues in most semantic segmentation datasets ( Caesar et al. , 2018 ; Cordts et al. , 2016 ; Zhou et al. , 2019 ) , including large scale variation ( Kirillov et al. , 2019 ; Long et al. , 2015 ; Ronneberger et al. , 2015 ) , complex scene ( Chen et al. , 2017 ; Zhao et al. , 2017 ) , and indistinguishable object boundaries ( Cheng et al. , 2020 ; Kirillov et al. , 2020 ; Zhen et al. , 2020 ) , aerial imagery segmentation has its own challenges , the most critical one among which lies in foreground-background imbalance ( Deng et al. , 2019 ; 2018 ; Li et al. , 2021 ; Pang et al. , 2019 ; Waqas Zamir et al. , 2019 ; Xia et al. , 2018 ; Zheng et al. , 2020 ) . Taking images in Fig . 1 as examples , the foreground proportion can be extremely small , e.g . less than 1 % for the leftmost image . Such acute imbalance could drastically increase the difficulty of object recognition , as even human eyes can hardly recognize them from the image . Moreover , the larger intra-class variance of the background objects may significantly increase the risk of false positive results ( Li et al. , 2021 ; Zheng et al. , 2020 ) . Existing general semantic segmentation methods mainly pay attention to designing sophisticated neural network architectures that can obtain informative multi-scale feature representations ( Chen et al. , 2018 ; He et al. , 2016 ; Kirillov et al. , 2019 ; Sun et al. , 2019 ; Szegedy et al. , 2017 ) and highlight the object boundaries ( Kirillov et al. , 2020 ; Li et al. , 2021 ; Zhen et al. , 2020 ) . To further address the foreground-background imbalance challenge of aerial imagery segmentation , some recent efforts have created more delicate modules in the neural networks to obtain superior results . For instance , Foreground-Aware Relation Network ( FarSeg ) ( Zheng et al. , 2020 ) introduces a foreground-scene relation module to enhance the discrimination of foreground features as well as a foreground-aware optimization to alleviate foreground-background imbalance problem . More recently , PointFlow ( Li et al. , 2021 ) designs a dual point matcher to select points from the salient area and object boundaries . While these previous studies have demonstrated the effectiveness of sophisticated neural network architecture design in both general and aerial imagery segmentation , most of them ignore the other side of the coin , i.e . how to efficiently utilize the multi-scale representations generated by complex neural networks . Intuitively , to identify the semantic meaning of a large object , it is more important for the model to leverage the representations obtained by wider receptive fields since they can represent the semantics of the whole large object rather than confusing the discrimination with interior details . On the other hand , for small objects , it is more efficient for the model to exploit the representations obtained by more concentrated receptive fields because they can focus on the discrimination without being distracted by noisy context . Unfortunately , most of the existing general or aerial segmentation methods merely employ multi-scale representations in ad-hoc ways , either simply concatenating them or arbitrarily using the final layer . This inevitably limits the potential of aerial imagery segmentation . Inspired by the adaptively-focusing process of the human eye ( Artal et al. , 2006 ; Wikipedia contributors , 2021 ) , we propose a novel Adaptive Focus Framework ( AF2 ) in this paper , which adopts a hierarchical segmentation procedure for aerial imagery segmentation . The general idea of this framework is shown in Fig . 2 . Particularly , through any widely employed model structure ( e.g . encoder-decoder structure ) , AF2 can obtain hierarchical representation maps based on different levels of receptive fields . The semantic segmentation procedure starts from the representation map created by the largest receptive field . After obtaining the segmentation result on this level , AF2 will filter out the pixels of low confidence on segmentation and then carry out another round of segmentation procedure on a finer-grained feature map . The whole process will be conducted repeatedly until all the pixels have been predicted , or until there is no finer-grained feature map . In AF2 , an Adaptive Confidence Mechanism ( ACM ) is proposed to deal with pixel filtration . The confidence of a pixel is defined as the highest value among the probabilities that the pixel belongs to each category , while an adaptive updated threshold is set to filter out low confidence pixels . In summary , the main contributions of this paper include : • We propose Adaptive Focus Framework ( AF2 ) , which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations generated by widely adopted neural network architectures . • In the pixel filtering process for each scale ’ s representation , the Adaptive Confidence Mechanism ( ACM ) is adopted to dynamically decide which pixels need to use finer-grained features . This mechanism ensures the performance and robustness of the framework . • Extensive experiments and analysis demonstrate the advantage of AF2 . It has significantly improved the accuracy on three typical aerial benchmarks . At the same time , its convergence speed and inference speed are also the fastest . It is worth noting that AF2 is a general framework , focusing particularly on efficient representation utilization , which can be leveraged to benefit any other semantic segmentation task . 2 RELATED WORKS . 2.1 GENERAL SEMANTIC SEGMENTATION . Fully-convolutional networks ( FCNs Long et al . ( 2015 ) ) are the earliest method of using deep learning to model semantic segmentation problem . The backbone models ( He et al. , 2016 ; Simonyan & Zisserman , 2014 ; Sun et al. , 2019 ; Szegedy et al. , 2017 ; 2016 ; Xie et al. , 2017 ) are used to generate a lower resolution output than the input image and use bilinear up-sampling to recover the original image resolution . Employing the dilated convolution ( Chen et al. , 2017 ; 2018 ; Yu & Koltun , 2015 ) to replace the down-sampling operation will have a better performance at the expense of more memory and computation cost . The spatial context information can overcome the limited receptive field of convolution layer to a certain extent such as Atrous Spatial Pyramid Pooling ( ASPP Chen et al . ( 2017 ; 2018 ) ) , Pyramid Pooling Module ( PPM Zhao et al . ( 2017 ) ) , Densely connected Atrous Spatial Pyramid Pooling ( DenseASPP Yang et al . ( 2018 ) ) , Relation-Augmented fully convolutional network ( RA-Net Mou et al . ( 2019 ) ) , etc . PointRend ( Kirillov et al. , 2020 ) performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm . Some other works ( Li et al. , 2020a ; Yuan et al. , 2020 ; Zhang et al. , 2020 ) propose architectures specific for segmentation boundary that is difficult to predict . The encoder-decoder architectures ( Chen et al. , 2018 ; Kirillov et al. , 2019 ; Lin et al. , 2017a ; Ronneberger et al. , 2015 ; Takikawa et al. , 2019 ; Li et al. , 2019 ) progressively upsample the high-level features and combine them with the features from lower levels , ultimately generating high-resolution features . For instance , Deeplab v3+ ( Chen et al. , 2018 ) combines dilated convolutions with an encoder-decoder structure to produce the output on a grid 4x sparser than the input . SemanticFPN ( Kirillov et al. , 2019 ) merges the information from all levels of the Feature Pyramid Network ( FPN ) pyramid into a single output and produces a dense prediction . 2.2 SEMANTIC SEGMENTATION FOR AERIAL IMAGERY . In recent years , employing deep learning to accelerate the understanding of aerial image has received widespread attention ( Kaiser et al. , 2017 ; Kussul et al. , 2017 ; Marcos et al. , 2018 ; Marmanis et al. , 2018 ; Scott et al. , 2017 ) , and benefits a lot of applications such as agriculture vision ( Arakeri et al. , 2016 ; Kamilaris & Prenafeta-Boldú , 2018 ; Patrício & Rieder , 2018 ) , road extraction ( Bastani et al. , 2018 ; Ji et al. , 2018 ) , land cover mapping ( Li et al. , 2016 ; Malkin et al. , 2018 ; Robinson et al. , 2019 ) , forest monitor ( Jiao et al. , 2019 ; Yuan et al. , 2015 ) , etc . They often design delicate structures to ensure that general semantic segmentation migrates well for specific application scenarios . For instance , Relation Augmented network ( RA-Net Mou et al . ( 2019 ) ) proposes a spatial relation module and a channel relation module to explicitly model global relations . Foreground-Aware Relation Network ( FarSeg Zheng et al . ( 2020 ) ) enhances the discrimination of foreground features and proposes a balanced optimization based on focal loss ( Lin et al. , 2017b ) . PointFlow ( Li et al. , 2021 ) designs a dual point matcher to select points from the salient area and object boundaries . 3 METHOD . In this section , we discuss details of the proposed AF2 . As shown in Fig . 3 , AF2 consists of three main parts : hierarchical features extractor , predictor , and adaptive confidence mechanism ( ACM ) . The pseudo-code of Adaptive Focus Framework is shown in appendix . • Hierarchical features extractor : Given an input image , this is used to extract the hierarchical feature maps of the image . The higher the level of feature map , the coarser the granularity of feature map and the larger the receptive field . • Preliminary predictor : For each level of the feature map , there is a preliminary predictor to output the categories of the pixels that still can not be confidently predicted with higher-level feature maps . • Adaptive confidence mechanism for prediction selection : The adaptive confidence mechanism ( ACM ) is employed to judge whether the prediction for each pixel is sufficiently confident or not in each level . The predictions with high enough confidence will be accepted for the corresponding pixels . Otherwise , the pixels will be passed down to the next hierarchy for further prediction with finer-grained features , until there is no pixels for prediction or the lowest level is reached . 3.1 HIERARCHICAL FEATURES EXTRACTOR . Hierarchical feature extractor is the fundamental part of the AF2 . It prepares different levels of feature maps for further pixel-level classification . The higher the level of feature map , the larger the receptive field . Since AF2 is independent of the particular structure of the network , any mainstream feature extraction network can be adopted as long as it is capable of extracting a hierarchy of feature maps . The typical model structures , such as Fully-Convolutional Network ( FCN Long et al . ( 2015 ) ) , Feature Pyramid Networks ( FPN Lin et al . ( 2017a ) ) , Semantic FPN Kirillov et al . ( 2019 ) , etc. , can be employed here . In this section , we take FPN ( Lin et al. , 2017a ) with Atrous Spatial Pyramid Pooling module ( ASPP ) ( Chen et al. , 2017 ) as an example . Specifically , ResNet ( He et al. , 2016 ) is chosen as the backbone network for basic feature extraction . We denote the different level of feature maps generated by ResNet as C = { Cl ∣∣Cl ∈ Rdl×H2l×W2l , l ∈ [ Lmin , Lmax ] } , where H and W represent the image ’ s original height and width . l is the level of feature map , while Lmin and Lmax are the lowest and highest levels ( e.g . if the output stride of ResNet is 25 , its Lmax is 5 . The lowest feature map ’ s output stride is always 22 , and its Lmin is 2. ) . Furthermore , let F = { Fl ∣∣Fl ∈ Rdl×H2l×W2l , l ∈ [ Lmin , Lmax ] } , which stands for the the feature map set from the decoder part . Fl is defined as Fl = { f ( ASPP ( Cl ) , Cl | θl ) , l = Lmax f ( UP2 ( Fl+1 ) , Cl | θl ) , otherwise . ( 1 ) where f ( · , · | θl ) represents the top-down feature fusion function . ASPP represents Atrous Spatial Pyramid Pooling Module ( Chen et al. , 2017 ) . UP2 represents the bilinear up-sampling function with a scale factor 2 .

Summary: This paper propose Adaptive Focus Framework (AF2), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multiscale representations generated by widely adopted neural network architectures. The main contribution is a learnable module, called Adaptive Confidence Mechanism (ACM) to determine which scale of representation should be used for the segmentation of different objects. The method achieves the state-of-the-art results on two ariel datasets(iSAID, Vaihingen).