TorchGeo: deep learning with geospatial data

1 INTRODUCTION . With the explosion in availability of satellite and aerial imagery over the past decades , there has been increasing interest in the use of imagery in remote sensing ( RS ) applications . These applications range from precision agriculture [ 39 ] and forestry [ 34 ] , to natural and man-made disaster monitoring [ 60 ] , to weather and climate change [ 48 ] . At the same time , advancements in machine learning ( ML ) , larger curated benchmark datasets , and increased compute power , have led to great successes in domains like computer vision ( CV ) , natural language processing ( NLP ) , and audio processing . However , the wide-spread success and popularity of machine learning—particularly of deep learning methods—in these domains has not fully transferred to the RS domain , despite the existence of petabytes of freely available satellite imagery and a variety of benchmark datasets for different RS tasks . This is not to say that there are not successful applications of ML in RS , but that the full potential of the intersection of these fields has not been reached . Indeed , a recent book by Camps-Valls et al . [ 8 ] thoroughly details work at the intersection of deep learning , geospatial data , and the Earth sciences . Increasing amounts of research on self-supervised and unsupervised learning methods specific to remotely sensed geospatial imagery [ 35 ; 3 ; 29 ] bring the promise of developing generic models that can be tuned to various downstream RS tasks . Recent large-scale efforts , such as the creation of a global 10 m resolution land cover map [ 31 ] or the creation of global 30 m forest maps [ 47 ] , pair the huge amount of available remotely sensed imagery with modern GPU accelerated models . To reach the full joint potential of these fields , we believe that we need tools for facilitating research and managing the complexities of both geospatial data and modern machine learning pipelines . We describe the challenges of this below , and detail our proposed solution , TorchGeo . One major challenge in many RS tasks is the large amount of diversity in the content of geospatial imagery datasets compared to datasets collected for traditional vision applications . For example , most conventional cameras capture 3-channel RGB imagery , however most satellite platforms capture different sets of spectral bands . The Landsat 8 satellite [ 50 ] collects 11 bands , the Sentinel-2 satellites [ 16 ] collect 12 bands , and the Hyperion satellite [ 44 ] collects 242 ( hyperspectral ) bands all measuring different regions of the electromagnetic spectrum . The exact wavelengths of the electromagnetic spectrum captured by each band can range from 400 nm to 15 µm . In addition , different sensors capture imagery at different spatial resolutions : satellite imagery can range from 4 km/px ( GOES [ 43 ] ) to 30 cm/px ( Maxar WorldView satellites [ 56 ] ) , while imagery captured from drones can go as low as 7 mm/px [ 1 ] . Depending on the type of orbit a satellite is in , imagery can be continuous ( for geostationary orbits ) or daily to biweekly ( for polar , sun-synchronous orbits ) . Machine learning models or algorithms developed for one of these platforms will not generalize across inputs collected by the others , and , as a consequence of this , it is not possible to publish a single set of pre-trained model weights that span imaging platforms . In contrast , ImageNet [ 15 ] pretrained models have been proven to be useful in a large number of transfer learning tasks [ 65 ] . Researchers and practitioners can often start with ImageNet pre-trained models in a transfer learning setup when presented with vision problems to reduce the overall amount of effort needed to solve the problem . Further , it isn ’ t clear whether the inductive biases built into common modeling approaches for vision problems are immediately applicable to remotely sensed imagery . Large neural architecture search efforts [ 32 ; 72 ] produce models that are optimized for , and indeed , outperform hand designed architectures on vision tasks , but it is an open question whether these transfer to remotely sensed imagery . Most machine learning libraries have not been designed to work with geospatial data . For example , the Python Imaging Library ( PIL ) [ 11 ] , used by many libraries to load images and perform data augmentation , does not support multispectral imagery . Similarly , deep learning models implemented by the torchvision library only support 3 channel ( RGB ) inputs , and must be adapted or re-implemented to support multispectral data . Datasets of geospatial data can be made up of a heterogenous mix of files with differing file formats , spatial resolutions , projections , and coordinate reference systems ( CRS ) . Libraries such as rasterio [ 22 ] and fiona [ 21 ] can interface with most types of geospatial data , however further abstractions for using such data in arbitrary deep learning pipelines are limited . Indeed , the gap between loading geospatial data from disk , and using it in a modeling pipeline , is large for all of the reasons mentioned above . As illustrated in Figure 1 , users will often need pixel-aligned crops from multiple layers of data : imagery from different points in time over the same space , imagery and corresponding label masks , high-resolution and low resolution imagery from the same space , etc . In contrast , there are a wide variety of software libraries at the intersection of machine learning and other domains . The PyTorch ecosystem alone has torchvision [ 38 ] , torchtext [ 67 ] , torchaudio [ 63 ] , Hugging Face ’ s Transformers [ 62 ] , PyTorch Geometric [ 18 ] , TorchMeta [ 14 ] , and PyTorch Video [ 17 ] , that provide the tools necessary for abstracting the peculiarities of domain-specific data away from the details of deep learning training pipelines . To address these challenges we propose TorchGeo , a Python package that allows users to transparently use heterogenous geospatial data in PyTorch-based deep learning pipelines . Specifically , TorchGeo provides : 1. data loaders for common geospatial datasets from the literature 2. data loaders for combining arbitrary geospatial raster and vector data layers with the ability to sample pixel-aligned patches on-the-fly 3. augmentations that are appropriate for multispectral imagery and geospatial data 4. data samplers appropriate for geospatial data 5. pre-trained models for many common remotely sensed imagery sources We formally describe TorchGeo , propose and test methods for sampling from large geospatial datasets , and test the effect of ImageNet pretraining versus random weight initialization on several benchmark datasets . We achieve close to state-of-the-art results on all experimental datasets , despite focusing only on creating simple and reproducible results to serve as baselines for future work to build on . We further find that ImageNet pre-training significantly improves spatial generalization performance in a land cover mapping task . We believe that these results are interesting in their own right , and that they highlight the importance of TorchGeo to the larger machine learning community . TorchGeo is open source and available on GitHub : redacted . 2 DESIGN . Remotely sensed imagery datasets are usually formatted as scenes , i.e . tensorsX∈RH×W×C , whereH is height , W is width , and C is the number of spectral channels , with corresponding spatial and temporal metadata . This metadata includes a coordinate reference system ( CRS ) that maps pixel coordinates to the surface of the Earth , a spatial resolution ( the size of each pixel when mapped onto the surface of the Earth ) , spatial bounds ( a bounding box representing the area on Earth that the data covers ) , and a timestamp or time range to indicate when the data was collected . We say that two datasets , X1 andX2 are pixel-aligned ifX1i , j andX 2 i , j represent data from the same positions on Earth for all i , j . Most pairs of datasets are not aligned by default . For example , X1 andX2 can be captured by two satellites in different orbits and will only have partially overlapping spatial bounds , orX1 will be satellite imagery whileX2 will be from a dataset of labels with a different CRS ( see Figure 2 ) . However , deep learning model training requires pixel-aligned patches of imagery—i.e . smaller crops from large scenes . Most models are trained with mini-batch gradient descent and require input tensors in the formatB×H×W×C whereB is the number of samples in a mini-batch andW andH are constant over all samples in the batch . At a higher level , training semantic segmentation models requires pairs of pixel-aligned imagery and masks . Aligning two datasets requires reprojecting the data from one in order to match the CRS of the other , cropping the data to the same spatial bounds , and resampling the data to correct for differences in resolution or to establish the same underlying pixel grid . Typically , these are performed as pre-processing steps using GIS software such as QGIS , ArcGIS , or tools provided by GDAL—see Section A.4 for an example of a GDAL command to align two data layers . This requires some level of domain knowledge to perform correctly and does not scale to large datasets as it requires creating duplicates of the layer to be aligned . Further , this approach still requires an implementation of a dataset or data loader that can sample patches from the pre-processed imagery . In TorchGeo , we facilitate this process by performing the alignment logic on-the-fly to create pixel-aligned patches of data sampled from larger scenes . Specifically , we implement the alignment logic in custom PyTorch Dataset classes that are indexed in terms of spatiotemporal coordinates . Given a query in spatiotemporal coordinates , a desired destination CRS , and a desired spatial resolution , the custom dataset is responsible for returning the corresponding reprojected and resampled data for the query . We further implement geospatial data samplers that generate queries according to different criteria ( e.g . randomly or in a regular grid pattern ) . See Section 3 for a discussion on the implementation of these . As our datasets are indexed by spatiotemporal coordinates we can easily compose datasets that represent different data layers by specifying a valid area to sample from . For example , if we have two datasets , D1 and D2 , we may want to sample data from the intersection of the layers , D1∩D2 , if one layer is imagery and the other layer is labels , or the union of the layers , D1∪D2 , if both layers are imagery . The latter is particularly powerful for use in applications like multimodal learning [ 4 ] and data fusion [ 68 ] . As we describe in the following section , we implement generic dataset classes for a variety of common remotely-sensed datasets ( e.g . Landsat imagery ) that can be composed in this way . This allows users of the library to create their own multimodal datasets without having to write custom code . Most importantly , these abstractions that TorchGeo creates—geospatial datasets and samplers—can be combined with a standard PyTorch data loader class to produce fixed size batches of data to be transferred to the GPU and used in training or inference . See Listing 1 for a working example of TorchGeo code for setting up a functional data loader that uses Landsat and Cropland Data Layer ( CDL ) dataset implementations . Our approach trades off required storage space for data loading time compared to pre-processing all data layers , but , crucially , does not require knowledge of GIS tooling . We benchmark our implementations in Section 4.3 .

Summary: The paper presents a new PyTorch library, TorchGeo, specifically for processing deep learning models on geospatial datasets. The library majorly provides a variety of benchmark geospatial datasets, along with pre-trained models on multi-spectral and SAR images. The library also provides a set of pre-processing steps which can pixel-align patches of data on-the-fly. Apart from these major contributions, the library has functions for augmenting (transforming) and sampling geospatial datasets. The paper is overall, well written and easy to understand.