TorchGeo: deep learning with geospatial data

1 INTRODUCTION . With the explosion in availability of satellite and aerial imagery over the past decades , there has been increasing interest in the use of imagery in remote sensing ( RS ) applications . These applications range from precision agriculture [ 39 ] and forestry [ 34 ] , to natural and man-made disaster monitoring [ 60 ] , to weather and climate change [ 48 ] . At the same time , advancements in machine learning ( ML ) , larger curated benchmark datasets , and increased compute power , have led to great successes in domains like computer vision ( CV ) , natural language processing ( NLP ) , and audio processing . However , the wide-spread success and popularity of machine learning—particularly of deep learning methods—in these domains has not fully transferred to the RS domain , despite the existence of petabytes of freely available satellite imagery and a variety of benchmark datasets for different RS tasks . This is not to say that there are not successful applications of ML in RS , but that the full potential of the intersection of these fields has not been reached . Indeed , a recent book by Camps-Valls et al . [ 8 ] thoroughly details work at the intersection of deep learning , geospatial data , and the Earth sciences . Increasing amounts of research on self-supervised and unsupervised learning methods specific to remotely sensed geospatial imagery [ 35 ; 3 ; 29 ] bring the promise of developing generic models that can be tuned to various downstream RS tasks . Recent large-scale efforts , such as the creation of a global 10 m resolution land cover map [ 31 ] or the creation of global 30 m forest maps [ 47 ] , pair the huge amount of available remotely sensed imagery with modern GPU accelerated models . To reach the full joint potential of these fields , we believe that we need tools for facilitating research and managing the complexities of both geospatial data and modern machine learning pipelines . We describe the challenges of this below , and detail our proposed solution , TorchGeo . One major challenge in many RS tasks is the large amount of diversity in the content of geospatial imagery datasets compared to datasets collected for traditional vision applications . For example , most conventional cameras capture 3-channel RGB imagery , however most satellite platforms capture different sets of spectral bands . The Landsat 8 satellite [ 50 ] collects 11 bands , the Sentinel-2 satellites [ 16 ] collect 12 bands , and the Hyperion satellite [ 44 ] collects 242 ( hyperspectral ) bands all measuring different regions of the electromagnetic spectrum . The exact wavelengths of the electromagnetic spectrum captured by each band can range from 400 nm to 15 µm . In addition , different sensors capture imagery at different spatial resolutions : satellite imagery can range from 4 km/px ( GOES [ 43 ] ) to 30 cm/px ( Maxar WorldView satellites [ 56 ] ) , while imagery captured from drones can go as low as 7 mm/px [ 1 ] . Depending on the type of orbit a satellite is in , imagery can be continuous ( for geostationary orbits ) or daily to biweekly ( for polar , sun-synchronous orbits ) . Machine learning models or algorithms developed for one of these platforms will not generalize across inputs collected by the others , and , as a consequence of this , it is not possible to publish a single set of pre-trained model weights that span imaging platforms . In contrast , ImageNet [ 15 ] pretrained models have been proven to be useful in a large number of transfer learning tasks [ 65 ] . Researchers and practitioners can often start with ImageNet pre-trained models in a transfer learning setup when presented with vision problems to reduce the overall amount of effort needed to solve the problem . Further , it isn ’ t clear whether the inductive biases built into common modeling approaches for vision problems are immediately applicable to remotely sensed imagery . Large neural architecture search efforts [ 32 ; 72 ] produce models that are optimized for , and indeed , outperform hand designed architectures on vision tasks , but it is an open question whether these transfer to remotely sensed imagery . Most machine learning libraries have not been designed to work with geospatial data . For example , the Python Imaging Library ( PIL ) [ 11 ] , used by many libraries to load images and perform data augmentation , does not support multispectral imagery . Similarly , deep learning models implemented by the torchvision library only support 3 channel ( RGB ) inputs , and must be adapted or re-implemented to support multispectral data . Datasets of geospatial data can be made up of a heterogenous mix of files with differing file formats , spatial resolutions , projections , and coordinate reference systems ( CRS ) . Libraries such as rasterio [ 22 ] and fiona [ 21 ] can interface with most types of geospatial data , however further abstractions for using such data in arbitrary deep learning pipelines are limited . Indeed , the gap between loading geospatial data from disk , and using it in a modeling pipeline , is large for all of the reasons mentioned above . As illustrated in Figure 1 , users will often need pixel-aligned crops from multiple layers of data : imagery from different points in time over the same space , imagery and corresponding label masks , high-resolution and low resolution imagery from the same space , etc . In contrast , there are a wide variety of software libraries at the intersection of machine learning and other domains . The PyTorch ecosystem alone has torchvision [ 38 ] , torchtext [ 67 ] , torchaudio [ 63 ] , Hugging Face ’ s Transformers [ 62 ] , PyTorch Geometric [ 18 ] , TorchMeta [ 14 ] , and PyTorch Video [ 17 ] , that provide the tools necessary for abstracting the peculiarities of domain-specific data away from the details of deep learning training pipelines . To address these challenges we propose TorchGeo , a Python package that allows users to transparently use heterogenous geospatial data in PyTorch-based deep learning pipelines . Specifically , TorchGeo provides : 1. data loaders for common geospatial datasets from the literature 2. data loaders for combining arbitrary geospatial raster and vector data layers with the ability to sample pixel-aligned patches on-the-fly 3. augmentations that are appropriate for multispectral imagery and geospatial data 4. data samplers appropriate for geospatial data 5. pre-trained models for many common remotely sensed imagery sources We formally describe TorchGeo , propose and test methods for sampling from large geospatial datasets , and test the effect of ImageNet pretraining versus random weight initialization on several benchmark datasets . We achieve close to state-of-the-art results on all experimental datasets , despite focusing only on creating simple and reproducible results to serve as baselines for future work to build on . We further find that ImageNet pre-training significantly improves spatial generalization performance in a land cover mapping task . We believe that these results are interesting in their own right , and that they highlight the importance of TorchGeo to the larger machine learning community . TorchGeo is open source and available on GitHub : redacted . 2 DESIGN . Remotely sensed imagery datasets are usually formatted as scenes , i.e . tensorsX∈RH×W×C , whereH is height , W is width , and C is the number of spectral channels , with corresponding spatial and temporal metadata . This metadata includes a coordinate reference system ( CRS ) that maps pixel coordinates to the surface of the Earth , a spatial resolution ( the size of each pixel when mapped onto the surface of the Earth ) , spatial bounds ( a bounding box representing the area on Earth that the data covers ) , and a timestamp or time range to indicate when the data was collected . We say that two datasets , X1 andX2 are pixel-aligned ifX1i , j andX 2 i , j represent data from the same positions on Earth for all i , j . Most pairs of datasets are not aligned by default . For example , X1 andX2 can be captured by two satellites in different orbits and will only have partially overlapping spatial bounds , orX1 will be satellite imagery whileX2 will be from a dataset of labels with a different CRS ( see Figure 2 ) . However , deep learning model training requires pixel-aligned patches of imagery—i.e . smaller crops from large scenes . Most models are trained with mini-batch gradient descent and require input tensors in the formatB×H×W×C whereB is the number of samples in a mini-batch andW andH are constant over all samples in the batch . At a higher level , training semantic segmentation models requires pairs of pixel-aligned imagery and masks . Aligning two datasets requires reprojecting the data from one in order to match the CRS of the other , cropping the data to the same spatial bounds , and resampling the data to correct for differences in resolution or to establish the same underlying pixel grid . Typically , these are performed as pre-processing steps using GIS software such as QGIS , ArcGIS , or tools provided by GDAL—see Section A.4 for an example of a GDAL command to align two data layers . This requires some level of domain knowledge to perform correctly and does not scale to large datasets as it requires creating duplicates of the layer to be aligned . Further , this approach still requires an implementation of a dataset or data loader that can sample patches from the pre-processed imagery . In TorchGeo , we facilitate this process by performing the alignment logic on-the-fly to create pixel-aligned patches of data sampled from larger scenes . Specifically , we implement the alignment logic in custom PyTorch Dataset classes that are indexed in terms of spatiotemporal coordinates . Given a query in spatiotemporal coordinates , a desired destination CRS , and a desired spatial resolution , the custom dataset is responsible for returning the corresponding reprojected and resampled data for the query . We further implement geospatial data samplers that generate queries according to different criteria ( e.g . randomly or in a regular grid pattern ) . See Section 3 for a discussion on the implementation of these . As our datasets are indexed by spatiotemporal coordinates we can easily compose datasets that represent different data layers by specifying a valid area to sample from . For example , if we have two datasets , D1 and D2 , we may want to sample data from the intersection of the layers , D1∩D2 , if one layer is imagery and the other layer is labels , or the union of the layers , D1∪D2 , if both layers are imagery . The latter is particularly powerful for use in applications like multimodal learning [ 4 ] and data fusion [ 68 ] . As we describe in the following section , we implement generic dataset classes for a variety of common remotely-sensed datasets ( e.g . Landsat imagery ) that can be composed in this way . This allows users of the library to create their own multimodal datasets without having to write custom code . Most importantly , these abstractions that TorchGeo creates—geospatial datasets and samplers—can be combined with a standard PyTorch data loader class to produce fixed size batches of data to be transferred to the GPU and used in training or inference . See Listing 1 for a working example of TorchGeo code for setting up a functional data loader that uses Landsat and Cropland Data Layer ( CDL ) dataset implementations . Our approach trades off required storage space for data loading time compared to pre-processing all data layers , but , crucially , does not require knowledge of GIS tooling . We benchmark our implementations in Section 4.3 .

Summary: Optimized Data downloading, processing, and sampling especially for domain scientists can be cumbersome. These are often steps that are routinely taken and can be optimized well once so that the entire community benefits from the ML-readiness. This paper tries to create generalized pipelines that both domain scientists and AI researchers can use to quickly get up and running with their experiments. The paper gathers existing datasets and makes them available through TorchGeo, a PyTorch domain library.