In-Domain Representation Learning For Remote Sensing

Given the importance of remote sensing , surprisingly little attention has been paid to it by the representation learning community . To address it and to establish baselines and a common evaluation protocol in this domain , we provide simplified access to 5 diverse remote sensing datasets in a standardized form . Specifically , we investigate in-domain representation learning to develop generic remote sensing representations and explore which characteristics are important for a dataset to be a good source for remote sensing representation learning . The established baselines achieve state-of-the-art performance on these datasets using in-domain representations . 1 INTRODUCTION . Remote sensing via computer vision and transfer learning is an important domain to address climate change as outlined by Rolnick et al . ( 2019 ) . Among others , research in remote sensing promises to help in solving challenges in food security ( precision farming ) , water sustainability , disaster prevention ( floods/landslides/earthquake forecasting ) , deforestation or wild fire detection , urban planning , and monitoring of carbon stocks and fluxes or the air quality . The number of Earth observing satellites is constantly increasing , with currently over 700 satellites monitoring many aspects of the Earth ’ s surface and atmosphere from space , generating terabytes of imagery data every day . However the ground truth data acquisition is costly , usually requiring extensive campaign preparation , people and equipment transportation , and in-field gathering of the characteristics under question . While there are remote sensing communities working on applying general deep learning methods to remote sensing problems , this domain has received relatively little attention from the representation learning community . Given its importance , it is still in an early development stage in comparison to the progress made in representation learning on natural and medical images ( eg . by Raghu et al . ( 2019 ) ) . Some reasons for this are the diversity of data sources ( satellite types , data acquisition modes , resolutions ) , the need of domain knowledge and special data processing , and the wide and scattered field of applications . The scarcity of standard recognized benchmark datasets and evaluation frameworks is another . For a long time there were only small labeled remote sensing datasets available . Only recenctly new large-scale datasets ( eg . by Zhu et al . ( 2018 ) ; Sumbul et al . ( 2019 ) ) have been generated for remote sensing problems . However , a consistent evaluation framework is still missing and the performance is usually reported on non-standard splits and with varying metrics , making reproduction and quick research iteration difficult . To address this , we provide five representative and diverse remote sensing datasets in a standardized form for easy reuse . In particular , we explore the importance of in-domain representation learning for remote sensing at various data sizes and establish new state-of-the-art baseline results . The main goal of this work is to develop general remote sensing representations that can be applied by researchers to other unseen remote sensing tasks . By providing these standardized datasets , common problem definition and baselines , we hope this work will simplify and enable faster iteration of research on remote sensing and inspire general representation learning experts to test their newest methods in this critical domain . In summary , the main contributions of this work are as follows : 1 . Exploring in-domain representation learning to train generic remote sensing representations . 2 . Generating 5 existing remote sensing datasets in a standardized format and establishing a common evaluation protocol . Publishing the best trained representations for easy reuse in transfer learning applications1 . 3 . Establishing state-of-the-art baselines for the BigEarthNet , EuroSAT , RESISC–45 , So2Sat , and UC Merced datasets . 2 RELATED WORK . REPRESENTATION AND TRANSFER LEARNING . Shortly after AlexNet ( Krizhevsky et al. , 2012 ) was trained on ImageNet ( Deng et al. , 2009 ) , representations obtained from it were used as off-the-shelf feature extractor networks ( Razavian et al. , 2014 ) . Fine-tuning approaches were also explored and led to better results than random-initialization even when the tasks or domains differ ( Yosinski et al. , 2014 ; Kornblith et al. , 2019 ) . These approaches extend beyond natural images and have been used the in medical domain , leading Raghu et al . ( 2019 ) to question on how they work . Simultaneously there have been many improvements on training representations . Mahajan et al . ( 2018 ) explored training bigger models on larger datasets with weaker labels and ( Ngiam et al. , 2018 ; Cui et al. , 2018 ) showed gains by closer matching the domains . Very promising are the more sample-efficient approaches using semi- or self-supervision ( Zhai et al. , 2019a ) . DEEP LEARNING IN REMOTE SENSING . Common remote sensing problems include land-use and land cover ( LULC ) classification , physical and bio-geo-chemical parameter estimation , target detection , time-series analysis , pan-sharpening and change detection . Many of these tasks can be solved or helped by deep learning approaches related to classification , object detection , semantic segmentation , or super-resolution . Reviews of some of these approaches can be found for instance in ( Ball et al. , 2017 ; Zhu et al. , 2017 ; 2019 ) . Remote sensing data is acquired in different modes : optical , multi- and hyper-spectral , synthetic aperture radar ( SAR ) , lidar , spectrometer ( Elachi & Van Zyl , 2006 ) . Each of these modes has its own acquisition geometry and specific characteristics providing unique and complimentary information about the Earth ’ s surface . In dependence of the instrument , the remote sensing imagery is available from very high resolutions at centimeter scale ( aerial optical or radar , eg . for urban monitoring ) to very low resolution at kilometer scale ( eg . for atmosphere and ocean surface monitoring ) . Another important satellite data characteristic is the revisit time ( how fast does a satellite revisit the same location , and constructs a time series ) , which can range from daily to multiple months . The majority of deep learning approaches in remote sensing are currently based on optical imagery at high ( 0.3-1 m ) to medium ( 10-30 m ) resolution , obtained from aerial imagery ( such as seen on Google Earth ) or publicly available satellite data ( eg . NASA ’ s Landsat or ESA ’ s Sentinel-2 satellites ) , respectively . Though , multi-spectral , hyper-spectral , and radar imagery is increasingly being used as well . REPRESENTATION/TRANSFER LEARNING IN REMOTE SENSING . Since labeling remote sensing data is expensive , for a long time there was no equivalent to ImageNet and most benchmark datasets were small . The probably most used remote sensing benchmarking dataset , UC Merced ( Yang & Newsam , 2010 ) , has only 2100 images with 100 images per class . 1To be released upon publication . Because it was not possible to train large state-of-the-art network on a small dataset from scratch , remote sensing researchers started to build smaller networks , as for instance in Luus et al . ( 2015 ) . Another direction was to use models pre-trained on natural images . For instance , Marmanis et al . ( 2016 ) used the extracted CNN features from the pre-trained Overfeat model ( Sermanet et al. , 2013 ) to feed to another CNN model . Castelluccio et al . ( 2015 ) used GoogLeNet ( Szegedy et al. , 2015 ) pre-trained on ImageNet to fine-tune UC Merced . Similarly , Nogueira et al . ( 2017 ) used pre-trained models on natural imagery to fine-tune five models based of cross–validation splits . Afterwards , they trained an additional SVM on the fine-tuned features to achieve best performance . Although there are works in using pre-trained representations , there is hardly any work on training representations specific for solving tasks on this domain . An example is the multi-layer generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) approach for unsupervised representation learning presented by Lin et al . ( 2017 ) . Xie et al . ( 2015 ) developed a transfer learning pipeline for mapping poverty distribution based on transfering the nighttime lights prediction task . Not specifically addressing representation learning , but a still related application using conditional GANs ( Mirza & Osindero , 2014 ) was demonstrated for cloud removal from RGB images by fusing RGB and near infrared bands in Enomoto et al . ( 2017 ) , or by fusing multi-spectral and synthetic aperture radar data ( Grohnfeldt et al. , 2018 ) . 3 DATASETS . For this work , five diverse datasets were selected . We prioritized newer and larger datasets that are quite diverse from each other , address scene classification tasks , and include at least optical imagery . Image data comes either from aerial high-resolution imagery or from satellites . Three datasets include imagery from the European Space Agency ’ s ( ESA ) Sentinel-2 satellite constellation , that provides medium resolution imagery of the Earth ’ s surface every three days . The multi-spectral imager on Sentinel-2 delivers next to the 3 RGB channels additional channels at various frequencies ( see Appendix A.1 ) . One dataset includes co-registered imagery from a dual-polarimetric synthetic aperture radar ( SAR ) instrument of ESA ’ s Sentinel-1 . Besides the differences in data sources , number of training samples , number of classes , image sizes and pixel resolutions ( summarized in Table 1 ) , the datasets are also quite diverse across : • Intra- and inter-class visual diversity : some datasets have high in-class and low betweenclasses diversity and vice versa . • Label imbalance : some datatasets are perfectly balanced , while others are highly unbalanced . • Label domain : land-use land cover ( LULC ) , urban structures , fine ecological labels . • Label quality : from fine human selection to weak labels from auxiliary datasets2 . While having only 5 datasets might not allow us to completely disentangle the confounding factors of remote sensing representation learning , it should still help us in understanding the important factors for good remote sensing representation learning datasets . 3.1 INDIVIDUAL DATASETS . In addition to the descriptions of the individual datasets in this section , Appendix A shows example images for all datasets and the distribution of classes . BigEarthNet ( Sumbul et al. , 2019 ) is a challenging large-scale multi-spectral dataset consisting of 590,326 image patches from the Sentinel-2 satellite . 12 frequency channels ( including RGB ) are provided , each covering an area of 1.2 × 1.2 km with resolutions of 10 m , 20 m , and 60 m per pixel . This is a multi-label dataset where each image is annotated by multiple land-cover classes . The label distribution is highly unbalanced ranging from 217k images of “ mixed forest ” label to only 328 images with label “ burnt areas ” . About 12 % of the patches are fully covered by seasonal 2Some datasets and images are affected by not filtered-out cloud and snow coverage that makes the correct classification of the samples difficult . snow , clouds or cloud shadows . The only available public baseline metrics include precision and recall values of 69.93 % and 77.1 % , respectively , for using a shallow CNN on a reduced dataset , after removing the snow and cloud affected samples . EuroSAT ( Helber et al. , 2019 ) is another recently published dataset containing 27,000 images from Sentinel-2 satellites . All 13 frequency bands of the satellite are included . Each image covers an area of 640× 640 meters and is assigned to one of 10 LULC classes , with 2000 to 3000 images per class . Because the classes are quite distinctive , very high accuracies can be achieved when using the entire dataset for training . NWPU RESISC–45 ( Cheng et al. , 2017 ) dataset is an aerial dataset consisting of 31,500 RGB images divided into 45 scene classes . Each class includes 700 images with a size of 256 × 256 pixels . This is the only dataset with varying spatial resolution ranging from 20 cm to more than 30 meters . The data covers a wide range of countries and biomes . During the construction , the authors paid special attention to have classes with high same-class diversity and between-class similarity to make it more challenging . So2Sat LCZ-42 ( Zhu et al. , 2018 ) is a dataset consisting of co-registered SAR and multi-spectral 320 × 320 m image patches acquired by the Sentinel-1 and Sentinel-2 remote sensing satellites , and the corresponding local climate zones ( LCZ ) ( Stewart & Oke , 2012 ) labels . The dataset is distributed over 42 cities across different continents and cultural regions of the world . This is another challenging dataset and it is intended for learning features to distinguish various urban zones . The challenge of this dataset is the relatively small image size ( 32×32 ) and the relatively high inter-class visual similarity . UC Merced Land-Use Dataset ( Yang & Newsam , 2010 ) is a high-resolution ( 30 cm per pixel ) dataset that was extracted from aerial imagery from the United States Geological Survey ( USGS ) National Map over multiple regions in the United States . The 256 × 256 RGB images cover 21 land-use classes , with 100 images per class . This is a relatively small datasets that has been widely benchmarked for remote sensing scene classification task since 2010 and for which nearly perfect accuracy can be achieved with modern convolutional neural networks ( Castelluccio et al. , 2015 ; Marmanis et al. , 2016 ; Nogueira et al. , 2017 ) .

Summary: This paper explored the representation learning problem on remote sensing domain. In particular, it provided several standardized remote sensing data sets with standard train/validation/test splits and various metrics for fairly comprehensive evaluation. It showed that compared to fine-tuning on ImageNet or learning from scratch, in-domain representation could produce better baseline results for remote sensing at various training data sizes.