TorchGeo: deep learning with geospatial data
Authors: Adam J Stewart
Caleb Robinson
Isaac A Corley
Anthony Ortiz
Juan M Lavista Ferres
Arindam Banerjee
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
remote sensing
geospatial data

Abstract: Remotely sensed geospatial data are critical for earth observation applications including precision agriculture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many earth observation tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. However, the variance in data collection methods and handling of geospatial metadata make the application of deep learning methodology to remotely sensed data nontrivial. For example, satellite imagery often includes additional spectral bands beyond red, green, and blue and must be joined to other geospatial data sources that can have differing coordinate systems, bounds, and resolutions. To help realize the potential of deep learning for remote sensing applications, we introduce TorchGeo, a Python library for integrating geospatial data into the PyTorch deep learning ecosystem. TorchGeo provides data loaders for a variety of benchmark datasets, composable datasets for generic geospatial data sources, samplers for geospatial data, and transforms that work with multispectral imagery. TorchGeo is also the first library to provide pre-trained models for multispectral satellite imagery, allowing for advances in transfer learning on downstream earth observation tasks with limited labeled data. We use TorchGeo to create reproducible benchmark results on existing datasets, benchmark our proposed method for preprocessing geospatial imagery on-the-fly, and investigate the differences between ImageNet pre-training and in-domain self-supervised pre-training on model performance across several datasets. We aim for TorchGeo to become a new standard for reproducibility and for driving progress at the intersection of deep learning and remotely sensed geospatial data.

Under review as a conference paper at ICLR 2022
TORCHGEO: DEEP LEARNING WITH GEOSPATIAL DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Remotely sensed geospatial data are critical for applications including precision agri- culture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many re- mote sensing tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. However, the variance in data collection methods and handling of geospatial metadata make the application of deep learning methodology to remotely sensed data nontrivial. For example, satellite imagery often includes additional spectral bands beyond red, green, and blue and must be joined to other geospatial data sources that can have differing coordinate systems, bounds, and resolutions. To help realize the potential of deep learning for remote introduce TorchGeo, Python library for integrating goospatial sensing applications, data into the Py PyTorch deep learning ecosystem. TorchGeo provides data loaders for a variety of benchmark datasets, composable datasets for generic geospatial data sources, samplers for geospatial data, and transforms that work with multispectral imagery. TorchGeo is also the first library provide pre-trained models for multispectral satellite imagery (e.g. models that use all bands from the Sentinel 2 satellites), allowing for advances in transfer learning on downstream remote sensing tasks with limited labeled data. We use TorchGeo to create reproducible benchmark results on existing datasets and benchmark our proposed method for preprocessing geospatial imagery on-the-fly. TorchGeo is open source and available on GitHub: redacted
INTRODUCTION
With the explosion in availability of satellite and acrial imagery over the past decades, there has been increasing interest in the use of imagery in remote sensing (RS) applications. These applications range from precision agriculture (39 and forestry [4 to natural and man-made disaster monitoring (60), to weather and climate change 48 At the same time, advancements in machine learning (ML), larger curated benchmark datasets, and increased compute power, have led to great successes in domains like computer vision (CV), natural language processing (NLP), and audio processing. However, the wide-spread success and popularity of machine domains has not fully transferred to the RS domain, despite the existence of petabytes of freely available satellite imagery and variety of benchmark datasets for different RS tasks. This is not say that there not successful applications of ML in RS, but that the full potential of the intersection of these fields has not been reached. Indeed, a recent book by Camps-Val Valls et al. [8] thoroughly details work at the intersection of deep learning, geospatial data, and the Earth sciences. Increasing amounts of research on self-supervised and unsupervised leamingg methods specific remotely sensed geospatial imagery 335 3 29 bring the promise of developing generic models that can be tuned to various downstream RS tasks. Recent
paatiinggarttuularlly of deep leamingg methods in these
large-scale scale efforts, such as the creation of a global 10 m resolution land cover map [3 or the creation
of global 30 m forest maps [7). pair the huge amount of available remotely sensed imagery with modern GPU accelerated models. To reach the full joint potential these fields, we believe that we need tools for facilitating research and managing the complexities of both geospatial data and modern machine learning pipelines. We describe the challenges of this below, and detail our proposed solution, TorchGeo.
One major challenge in many RS tasks is the large amount of diversity in the content of geospatial imagery datasets compared to datasets collected for traditional vision applications. For example, most conventional cameras capture 3-channel RGB imagery, however most satellite platforms capture different sets of spectral bands. The Landsat 8 satellite [50] collects 11 bands, the Sentinel-2 satellites [16] collect 12 bands, and the Hyperion satellite
44 collects 242 (hyperspectral) bands all measuring different regions

---

Under review as a conference paper at ICLR 2022
A
Landsat 8 Scene LPSG
c
D
Cropland Data Layer
FPSG 5372
Figure 1: An illustration of the challenges in sampling from heterogeneous geospatial data layers. (A and B) show example geospatial data layers that a user may want to sample pixel-aligned data from. As these layers have differing coordinate reference systems, patches of imagery (C and D) sampled from these layers that cover the same area will not be pixel-aligned. TorchGeo transparently performs the appropriate alignment steps (reprojecting and resampling) during data loading such that users can train deep learning models without having to manually align the data layers.
of the electromagnetic spectrum. The exact wavelengths of the electromagnetic spectrum captured by each band can range from 400 nm to 15 pmm In addition, different sensors capture imagery at different spatial resolutions: satellite imagery can range from 4 km/px (GOES 4 to 30 cm/px (Maxar WorldView satellites (56)), while imagery captured from drones can go as low as 7 mm/px (. Depending on the type of orbit satellite is in, imagery can be continuous (for geostationary orbits) daily to biweekly (for polar, sun-synchronous orbits). Machine learning models or algorithms developed for one of these platforms will not generalize across inputs collected by the others, and, as a consequence of this, it not possible to publish a single set of pre-trained model weights that span imaging platforms. In contrast, ImageNet [5] pretrained models have been proven to be useful in large number of transfer learning tasks 65. Researchers and practitioners can often start with ImageNet pre-trained models in transfer learning setup when presented with vision problems to reduce the overall amount of effort needed to solve the problem. Further, it isn't 't clear whether the inductive biases built into common modeling approaches for vision problems are immediately applicable to remotely sensed imagery. Large neural architecture search efforts [32 72 produce models that optimized for, and indeed, outperform hand designed architectures on vision tasks, but is an open question whether these transfer to remotely sensed imagery.
Most machine learning librarics have not been designed to work with geospatial data. For example, the Python Imaging Library (PIL) [1], used by many libraries to load images and perform data augmentation, does not support multispectral imagery. Similarly, deep learning models implemented by the torchvision library only support channel (RGB) inputs, and must be adapted or re-implemented to support multispectral data. Datasets of geospatial data can be made up of heterogenous mix of files with differing file formats, spatial resolutions, projections, and coordinate reference systems (CRS). Libraries such as rasterio (22) and fiona (2l) can interface with most types of geospatial data, however further abstractions for using such data in arbitrary deep learning pipelines are limited. Indeed, the gap between loading geospatial data from disk, and using in a modeling pipeline, is large for all of the reasons mentioned above. As illustrated in Figure [ users will often need pixel-aligned crops from multiple layers of data: imagery from different points in time over the same space, imagery and corresponding label masks, high-resolution and low resolution imagery from the same space, etc. In contrast, there are a wide variety of software libraries at the intersection of machine learning and other domains. The PyTorch ecosystem alone has torchvision (8), torchtext [67], torchaudio 633. Hugging Face's 's Transformers [62]. Py Torch Geometric [8]. TorchMeta [14), and Py PoTch Video [7]. that provide the tools necessary for abstracting the peculiarities of domain-specific data away from the details of deep learning training pipelines.
2

---

Under review as a conference paper at ICLR 2022
Landsat 8
Cropland Data Layer
NAIP Imagery
CC Land Cover
Different projections and spatial resolution
Reprojected and resampled
Figure 2: Different layers of geospatial data often have differing coordinate reference systems and spatial resolutions. (Top row) The same physical area cropped from four raster layers with different coordinate reference systems and spatial resolutions- this data is not pixel-aligned and cannot yet be used in modelling pipelines. (Bottom row) The same data as after reprojecting into the same coordinate system and resampling to the highest spatial resolution this data pixel aligned and can serve inputs or masks to deep neural networks.
To address these challenges we propose TorchGeo, a Python package that allows users to transparently use heterogenous geospatial data in Py HyToch--base deep learning pipelines. Specifically, TorchGeo provides:
1. data loaders for common geospatial datasets from the literature
2. data loaders for combining arbitrary geospatial raster and vector data layers with the ability to sample pixel-aligned patches on-the-fly
3. augmentations that are appropriate for multispectral imagery and geospatial data
4. data samplers appropriate for geospatial data
5. pre-trained models for many common remotely sensed imagery sources
We formally describe TorchGeo, propose and test methods for sampling from large geospatial datasets, and test the effect of ImageNet pretraining versus random weight initialization on several benchmark datasets. We achieve close to state-of-the-art results on all experimental datasets, despite focusing only on creating simple and reproducible results to serve as baselines for future work to build on. We further find that ImageNet pre-training significantly improves spatial generalization performance a land cover mapping task. We believe that these results interesting in their own right, and that they highlight the importance of TorchGeo to the larger machine learning community. TorchGeo is open source and available on GitHub: redacctd
2 DESIGN
Remotely sensed imagery datasets usually formatted scenes, ie. tensors X HxWXC where H height, W width, and C is the number of spectral channels, with corresponding spatial and temporal metadata. This metadata includes coordinate reference system (CRS) that maps pixel coordinates to the surface of the Earth, a spatial resolation (the size of each pixel when mapped onto the surface of the Earth), spatial boundy (a bounding box representing the area on Earth that the data covers), and timestamp or time range to indicate when the data was collected. We say that two datasets, X and X are pixel- rixel-aligned if X and X2, represent data from the same positions on Earth for all i.j. Most pairs of datasets are not aligned by default. For example, X and X can be captured by two satellites in different orbits and will only have partially overlapping spatial bounds, or X will be satellite imagery while X will be from a dataset of labels with a different CRS (see Figure 2 However, deep learning model training requires pixel-aligned patches of imagery- ie. smaller crops from large scenes. Most models are trained
3

---

Under review as a conference paper at ICLR 2022
with mini-bateh gradient descent and require input tensors in the format B X x C where B is the number of samples in a mini-batch and W and H are constant over all samples in the batch. At a higher level, training semantic segmentation models requires pairs of pixel-aligned imagery and masks.
Aligning two datasets requires reprojecting the data from one in order to match the CRS of the other, cropping the data to the same spatial bounds, and resampling the data to correct for differences in resolution or to establish the same underlying pixel grid. Typically, these are performed as pre-proeee preeeesocsiingggg steps using GIS software such as QGIS, AreGIS, or tools provided by GDAL-see Section A.4 for an example of a GDAL command to align two data layers. This requires some level of domain knowledge to perform correctly and does not scale to large datasets as requires creating duplicates of the layer to be aligned. Further, this approach still requires an implementation of a dataset or data loader that can sample patches from the pre-processed imagery.
In TorchGeo, we facilitate this process by performing the alignment logic on-the-fly to create pixel-aligned patches of data sampled from larger scenes. Specifically, we implement the alignment logic in custom PyTorch Dataset classes that are indexed in terms of spatiotemporal coordinates. Given a query in spatiotemporal coordinates, a desired destination CRS, and a desired spatial resolution, the custom dataset is responsible for returning the corresponding reprojected and resampled data for the query. We further implement geospatial data samplers that generate queries according to different criteria (e.g. randomly or in regular grid pattern). See Section 3 for discussion the implementation of these.
As our datasets are indexed by spatiotemporal coordinates we can easily compose datasets that represent different data layers by specifying a valid area to sample from. For example, if we have two datasets, D and D2, we may want to sample data from the intersection of the layers, D O D2, if one layer is imagery and the other layer is labels, the union of the layers, DIUD* if both layers are imagery. The latter is particularly powerful for use in applications like multimodal learning [4] and data fusion [68]. As we describe in the following section, we implement generic dataset classes for variety of common remotely-sensed datasets (e.g. Landsat imagery) that can be composed in this way. This allows users of the library to create their own multimodal datasets without having write custom code.
Most importantly, these abstractions that TorchGeo cvrates-geospattin datasets and samplers can be com- bined with standard Py ToTrch data loader class to produce fixed size batches of data to be transferred to the GPU and used in training or inference. See Listing T for a working example of TorchGeo code for setting up a functional data loader that uses Landsat and Cropland Data Layer (CDL) dataset implementations. Our ap- proach trades off required storage space for data loading time compared to pre-processing all data layers, but, crucially, does not require knowledge of GIS tooling. We benchmark our implementations in Section 4.3
3 IMPLEMENTATION
The implementation of TorchGeo follows the design of other PyTorch domain libraries to reduce the amount of new concepts that a user must learn to integrate it with their existing workflow. We split TorchGeo up into the following parts:
Datasets Our dataset implementations consist of both benchmark datasets that allow users to interface with common datasets used in RS literature and generic datasets that allow users interface with common geospatial data layers such as Landsat or Sentinel-2 imagery. Either of these types of datasets also be geospatial datasets, i.e. datasets that contain geospatial metadata and can be sampled as such. These are part of the core contribution of TorchGeo and we describe them further in the following section.
Samplers
We implement samplers for indexing into any of our geospatial datasets. Our geospatial datasets indexed by bounding boxes in spatiotemporal coordinates (as opposed to standard fixed-length datasets of images which are usually indexed by an integer). The samplers generate bounding boxes according specific patterns: randomly across all scenes in dataset, random batches from single scenes at a time, or in grid patterns over scenes. Different sampling patterns can be useful for different model training strategies, or for running model inference over datasets.
Models
Most existing model implementations (e.g. in torchvision) are fixed to accept 3 channel inputs which are not compatible with multispectral imagery. We provide implementations (or wrappers around well-established implementations) of common deep learning model architectures with variable-sized inputs and pre-trained weights. For example, models that use all of the Sentinel-2
4

---

Under review as a conference paper at ICLR 2022
multispectral bands imputs. We also implement architectures from recent geospatial ML work such as the Fully Convolutional Siamese Network [12].
Transforms Similar to existing model implementations, some existing deep learning packages do not support data augmentation methods for multi-spectral imagery. We provide wrappers for augmentations the Kornia (45] library (which does support augmentations over arbitrary channels) and implement transforms specific geospatial data.
Trainers Finally, we implement model training recipes using the PyTorch Lightning library [61). These include both dataset-specific training code, and general training routines- for example, an implementation of the BYOL self-supervision method [2
3.1 DATASETS
We organize datasets based whether they generic datasets benchmark datasets and based whether or not they contain geospatial metadata-i.c. are a geospatial dataset.
Benchmark datasets are datasets released by the community that consist of both inputs and target labels for specific type of task (scene classification, semantic segmentation, instance segmentation, etc.). These may or may not also contain geospatial metadata that allows them be joined with other sources of data. In our opinion, one of the strongest components of existing deep learning domain libraries is the way that they make the use of existing datasets trivial. We aim to replicate this and, for example, include options that let users automatically download the data for corresponding dataset. Table [ lists the set of benchmark datasets that TorchGeo currently supports.
Dataset
Task
Source
# Samples
Categories
Size (px)
Resolution (m)
Bands
ADVANCE
27
C
Google Earth, Freesound
5,075
13
512x512
0.5
RGB
BigFartNett
C
Sentinel- 1/2
590,326
19 43
120x120
10
SAR, MSI
EuroSAT
25
C
Sentinel-2
27,000
64x64
10
PutterNett
70
C
Google Earth
30,400
38
256x256
0.06-5
RGB
RESISC45 10
C
Google Earth
31,500
45
256x256
0.2-30
RGB
So2Sat 7
Sentinel-122
400,673
17
32x32
10
SAR, MSI
UC Merced 64
USGS National Map
21,000
21
256x256
0.3
RGB
CSUAV AFRL, ISPRS, LINZ, AGRC
COWC A0
C.R
388,435
2
256x256
0.15
RGB
Tropical Cyclone
Wind Estimation 27
R
GOES 8
108,110
256x256
4K 8K
MSI
SEN12MS 51
S
Sentinel 1/2, MODIS
180,662
33
256x256
10
SAR, MSI
CV4A Kenya
Crop Type 19
S
Sentinel-2
4,688
3,035x2.016
10
MSI
ETCI 2021
Flood Detection
41
S
Sentinel 1
66,810
2
256x256
20
SAR
GID 15
57
S
Gaofen
150
15
6,800x7.200
3
RGB
LandCover.ai
Aerial
10,674
512x512
0.25
Smallholder Cashew
Plantations in Benin
30
Airbus Pléiades
1,2
0.5
Potsdam
Acrial
6,000x6.000
0.05
Vaihingen 40
Acrial
33
1,281-3.816
0.09
RGB
NWPU VHR-10 1003-013000
0
I
Google Earth, Vaihingen
800
10
358-1,728
0.08-2
RGB
WorldView-22 2/3,
SpaceNet 2,
889-28.728
102-900
0.5-4
MSI
ZucriCrop 58
LT
Sentinel-2
116K
48
24x24
10
MSI
Scasonal Contrast
35
T
Sentinel-2
100K-IM
264x264
10
MSI
LEVIR-CD+ 52
D
Google Earth
985
2
1.024x1.02 .024
0.5
RGB
OSCD L3
D
Sentinel-2
2
40-1,180
60
x View2
24
Maxar
3.732
1,024x1.024
0.8
RGB
= classificatoo, R= regression, S = semantic segmentation, instance segmentation, T time series, D change detection
Table 1: Benchmark datasets implemented in TorchGeo.
5

---

Under review as a conference paper at ICLR 2022
Generic datasets not created with specific task in mind, but instead represent layers of geospatial data that can be used for any pur- pose. For example, we implement datasets for representing collections of scenes of Landsat imagery that let users index into the imagery and use it in arbitrary PyTocc-bassed pipelines. These are not limited to imagery. for exam- ple we also implement dataset representing the Cropland Data I Layer labels, an annual US wide raster layer that gives the estimated crop type or land cover at 30 m/px resolution (see T.
Type
Dataset
Landsat 50
Image Source
Sentinel [6
National Agriculture
Imagery Program 20
Cropland Data Layer
Labels
Chesapeake Land Cover 46
Canadian Buildings Footprints 555
Table 2: Generic datasets implemented in TorchGeo.
3.2 SAMPLERS
As our geospatial datasets are indexed with bounding boxes using spatiotemporal coordinates and do have concept of a dataset "length", they cannot be sampled from by choosing a random integer. We provide three types of samplers for different situations: a "RandomGeeSampler that returns fixed-sized bounding box from the valid spatial extent of dataset uniformly random, a "RandooBaceGGGeSSappeer" that returns set of randomly positioned fixed-sized bounding boxes from random scene within dataset, and a "ampleeeSampler" that returns bounding boxes a grid pattern over subsequent scenes within dataset. These samplers are benchmarked in Section 4.2 This abstraction also allows for methods that rely on specific data sampling pattemn.. For example, Tile2Vec [29] relies on sampling triplets of imagery where two of the images are close each other in space while the third is distant. This logic can be implemented in several lines of code as a custom sampler class, that would then operate over any of the generic imagery datasets. Finally, all TorchGeo samplers compatible with Py PoToch data loader objects so can be into any PyTorch-basee pipeline.
4 EXPERIMENTS AND RESULTS
4.1 DATASETS
We use the following datasets our experiments:
Landsat and Cropland Data Layer (CDL) The Landsat and CDL dataset consists of multispectral imagery from 114 Landsat 8 50 collection 2 level 2 scenes taken in 2019 that intersect with the continental United States and the 2019 Cropland Data Layer (CDL) dataset 77. This data is 151 GB on disk and is stored in cloud optimized GcoTIFF (COG) format. We use this dataset to benchmark our proposed GeoDataset and sampler implementations.
So2Sat The So2Sat-I.C7A dataset 71 is a classification dataset that consists of 400,673 image patches classified with one of 42 different local climate zone labels. The patches are 30 30 pixels in size with 18 channels consisting of Sentinel 1 and Sentinel 2 bands. They are sampled from different urban areas around the globe. We use the second version of the dataset described on the project's GitHub page"] in which the training split consists of data from 42 cities around the world, the validation split consists of the western half of 10 other cities, and the testing split covers the eastern half of the 10 remaining cities.
LandCover.ai
The LandCoverai dataset is a semantic segmentation dataset 5 that consists of high-resolution (0.5 m/px and 0.25 m/px) RGB aerial imagery from 41 tiles over Poland where each pixel has been classified as one five classes (four land cover classes and a "background" class). The scenes are divided into 10,674 512 x 512 pixel patches and partitioned into pre-defined training, validation, and test splits according to the script on the dataset webpag !
Chesapeake Land Cover The Chesapeake I Land Cover dataset (46 is semantic segmentation dataset that consists of high-resolution (1 m/px) imagery from the US Department of Agriculture's National Agriculture Imagery Program (NAIP) and high-resolution (1 m/px) 6-class land cover
httpps://anndooverr.o

---

Under review as a conference paper at ICLR 2022
labels from the Chesapeake Conservancy. Specifically, the dataset contains imagery and land cover masks for parts of six states in the Northeastem US: Maryland, Delaware, Virginia, West Virginia, Pensylvania, and New York. The data for each state is split into ~ 7 6 km tiles and then divided into pre-defined train, validation, and test split F
RESISC45
The RESISC45 dataset is a classification dataset (69] that consists of 31,500 256 x 256 pixel RGB image patches of varying spatial resolutions where each patch is classified into one of 45 classes. As the dataset does not have official splits, we use the train/val/test splits defined in 42
ETCI 2021 The ETCI 2021 dataset [11] is a semantic segmentctiion dataset used in a flood detection competition. It consists 66,810 256x256 pixel Sentinel-1 SAR images. We use the official train/test splits, and we further randomly subdivide the train split 80/20 into train/val splits.
EuroSAT The EuroSAT dataset [25] is classification dataset consisting of 27,000 64x64 pixel Sentinel-2 images and 10 target classes. As the dataset does not have official splits, we use the train/val/test splits defined in 42..
UC Merced The UC Merced dataset 64 is a classification dataset consisting of 21,000 256x256 pixel RGB images from the USGS National Map and 21 target classes. As the dataset does not have official splits, we use the train/val/tess splits defined in 42
4.2 DATA LOADER BENCHMARKS
We first benchmark the speed at which TorchGeo can sample patches of imagery and masks from the Landsat and CDL dataset. We believe this dataset is typical of a large class of geospatial machine learning mroblesswwhen
have access to large amount of satellite imagery scenes covering broad spatial extent and, separately, per-pixel label masks where each scene is not necessarily projected in the same coordinate reference system. The end goal of such problem of imagery and label masks as described in Section 2 As such, we measure the rate at which our dataset and sampler implementations can provide patches to GPU for training and inference.
train a model with pixel-aligned patches
In Figure Bai we calculate the rate at which samples of varying batch size can be drawn from a "GeoDataset" using various "GGooSmmper" implementations. Compared to the other samplers, "GrridGeoSampler" sig- nificantly faster due to the repeated access of samples that are already in GDAL's least recently used (LRU) cache, For small batch sizes, "RandonGeeSanplee" and BrandomBBachhGGeSSmpleee" are almost identical, since overlap between patches is uncommon. However, for larger batch sizes, "RanoomBacchGGoSSmmpp. starts to outperform "RandonGeoSSmplee" as the cache is used more effectively.
In Figure 3b we demonstrate the difference that preprocessing and caching data makes. This is most easily demonstrated by "GridGeoSampler" and to lesser extent the other samplers. GDAL's LRU cache only saves raw data loading times, warping must always be done on-the-fly if the dataset CRSs resolutions
(a) Sampler performance batch size
(b) Effect of preprocessing and caching
900
700
Raw Data, Not Cached Preprocessed, Not Cached
800
(patchas/sec)
Raw Data, Cached
(patches/tec)
600
Preprocessed, Cached
Random pler
Sampling
Sampling
500
Random BatchGeo mpler
400
400
Gri GrimleeSampler
300
300
100
100
SampeerSampler
RandomBatchGee RandomeeeSamppees
16
32
64 Batch size
128
256
Figure 3: Sampling performance of various "GeoSampler" implementations. (a) Solid lines represent average sampling rate, while shaded region represents minimum and maximum performance across random seeds. (b) Average sampling rate under different data loading conditions.
7

---

Under review as a conference paper at ICLR 2022
don't match. When the necessary storage is available, preprocessing the data ahead of time can lead to significantly faster sampling rates. Although "RandonGesSumpler" and "Ramdom are much slower than "GridGeoSampler" most users will only need to use "GridGeoSampler" for inference due to our pre-trained model weights.
4.3 DATASET BENCHMARKS
We also use TorchGeo to create simple, reproducible benchmark results 7 of the datasets described in Section 4.1 We report the uncertainty in the results such that future work can evaluate whether a proposed improvement is due methodological innovation or variance in results due the stochastic nature of training deep learning models. To ensure reproducibility, we include model training and evaluation framee- work TorchGeo based around the Py PyTorch Lightning library and release all of our results. To quantify uncertainty we report the mean and standard deviation metrics calculated over 10 training runs with different random seeds. The torchmetries library 6 was used to compute all performance values (overall top-1 accu- racy for the classification datasets, and mean intersection over union for the semantic segmentation datasets).
Our main results are shown in 3 3 We find that our simple training setup achieves competitive results on several datasets. The in-domain pre-training method in 42] trains models, starting with ImageNet weights, further on remote sensing (in domain) datasets before actually training on the target dataset and finds that this performs better than simply starting from ImageNet weights. In contrast, our best result across the RESISC45, EuroSAT, and UC Merced datasets come from simply using ImageNet pre-traine pre-trained models and training on the target dataset with low learning rate. The difference between these approaches is likely entirely due to different learning rate selection in the hyperparameter search, and data augmentation in the in-domain pre-training setup. On the So2Sat dataset find that the ImageNet pre-trained models able to achieve similar results as in-domain pretraining, but only when using all Sentinel-2 bands. The previously
Dataset
Method
Weight Initialization
Bands
Performance
ResNet50
ImageNet
RGB
95.42 0.23%
ResNet18
random
RGB
79.90 0.25%
RESISC45
ResNet50 v2
domain
RGB
96.86 %
VIT B/16 [53
ImageNet-2Ik
RGB
96.80%
ResNet50 66
Sup- Sun-RRtationn10009
RGB
96.30%
ResNet50
ImageNet random)
MSI
63.99
1.38%
ResNet50
random
56.82
4.32%
ResNet50
ImageNet
RGB
59.82
0.94%
So2Sat 71
ResNet50
random
RGB
49.46 2.67%
ResNeXt+ CBAM 71
random
MSI
61%
ResNe150 v2 42
In domain
RGB
63.25%
U-Net, ResNet50 encoder
ImageNet
RGB
84.81 =:: 00.21%
LandCover.ai
U-Net, ResNet50 encoder
random
RGB
79.73 + 00.67%
DeepLabv3+, Xception71 with
Cityscapes
RGB
85.56 %
DPC encoder [S]
Chesapeake Land Cover
U-Net, ResNet50 encoder
ImageNet random)
MSI
69.40
1.39%
Delaware split
U-Net, ResNet18 encoder
random
68.99
0.84%
ETCI 2021
41
U-Net, ResNet50 encuder
random
SAR
45.77 +: 3.19%
ResNet50
ImageNet (+ random)
MSI
97.86 0.23%
ResNet50
random
MSI
96.07 0.28%
AT 25
ResNet50
ImageNet
RGB
98.11 + 0.31%
ResNet50
random
RGB
87.33 + 0.76%
ResNet50 v2 42
In domain
RGB
99.20% %
ResNet50
ImageNet
RGB
98. 15% + 0.46%
UC Merced 64
ResNet50 v2 42
In domain
RGB
99.61 %
Table 3: Benchmark results comparing TorchGeo trained models to previously reported results over seven datasets. RESISC45, So2Sat, FuroSat, and UC Merced results are reported as overall top-1 accuracy while reported mean class IoU. Results
LandCover.ai, Chesapeake Land Cover, and ETCI 2021 results from TorchGeo models are reported as the mean with one standard deviation over training runs from different random seeds. Results from related work reported as is.
8

---

Under review as a conference paper at ICLR 2022
reported baseline methods the LandCover.ai dataset all use DeepLabV3+ segmentation model with Xception71 + Dense Prediction Cell (DPC) encoder that has been pre-trained on Cityscapes. We are able to achieve result within 0.75% mloU of this setup using simple U-Net and ResNet50 encoder pre-trained on ImageNet. Finally, we report the first set of basic benchmark results on the Chesapeake Land Cover dataset.
4.4 EFFECT OF IMAGENET PRE-TRAINING ON GENERALIZATION PERFORMANCE
Two of the datasets we test with, So2Sat and Chesapeake Land Cover, contain splits that are designed measure the generalization performance of a model. The validation and test splits from So2Sat include data from urban areas that are not included in the training split while the Chesapeake Land Cover dataset contains separate splits for six different states. In these setting observe large performance boost when training models from ImageNet weights versus from a random initialization, however we do not observe the boost on in-domain data. Table shows the performance of models that are trained on the Delaware split and evaluated on the test splits from every state. The in in-oomiin performance (i.e. in Delaware) of the ImageNet pre- pre-trained model and model trained from scratch are the same, however in every other setting the ImageNet pre-trained model performs better. In all cases but Maryland, this difference is greater than points of mloU with the most extreme difference being 12 points Virginia. In the So2Sat case find that most models are not able significantly reduce validation loss (where in this case validation data is out-of-domain), while, unsurprisingly, achieving near perfect results on the training data. Despite this overfitting, there remains large gap between the best and worst models, with ImageNet pre-trained models achieving +7% and + +10% accuracy over randomly initialized models. These results extend existing lines of research in computer vision that show how pre-training can improve out-of-domain performance [26], and how, specifically, ImageNet pretraining is useful for transfer learning tasks 28
Weight init
Delaware
Maryland
New York
Pennsylvania
Virginia
West Virginia
ImageNet (+ random)
69 49.40. 1. 1.39%
59.57 1 0.70%
57.95 1 1.10%
55.13 : 1.25%
45.56 1 1.54%
20.76 1: 1.95% 14.95 2.72%
random
68.99 0.84%
57.30 0.78%
49.26 + 2.40%
47.67 + 2.0%
33.14 3.73%
Table 4: Mean IOU performance of models trained in Delaware, with and without ImageNet weight initialization, on the test splits from Chesapeake Land Cover dataset.
DISCUSSION
We introduce TorchGeo, a Python package for enabling deep learning with geospatial data. TorchGeo provides data loaders for common geospatial datasets, composable data loaders for arbitrary goospatial raster and vector data, samplers appropriate for geospatial data, models, transforms, and model trainers. Importantly, TorchGeo allows users to bypass the common pre-processing steps necessary align geospatial data with imagery and performs this processing on-the-fly. We benchmark TorchGeo dataloader speed, and demonstrate how TorchGeo can be used create reproducible benchmark results in several geospatial datasets.
Finally, TorchGeo serves as a platform for performing geospatial machine learning research. Existing works in self-supervised learning with geospatial data rely on spatiotemporal metadata and can be naturally implemented in TorchGeo and scaled over large amounts of geospatial imagery without the need for pre- processing steps. Similarly, augmentation methods appropriate for training with geospatial imagery are under-explored, however can integrated easily with TorchGeo. Other interesting directions include building inductive biases appropriate for geospatial imagery into deep learning models (similar to the work done on rotation equivariant networks (36)), data fusion techniques (e.g. how to incorporate spatial information into models, appropriately use multi-modal layers), and learning shape-based models. Finally, TorchGeo exposes a catalog of benchmark geospatial datasets (Table T through a common interface, and, with the results in this paper, has begun to include corresponding benchmark results. This makes it easy for researchers to compare new ideas to existing work without having to repeat expensive computations. We hope TorchGeo can help drive advances at the intersection of machine learning and remote sensing.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Machine learning with remotely sensed data has the potential to impact many aspects of society in both positive and negative ways. For example, satellite imagery can be used model and track progress towards roughly 20% of the indicators of the United Nation's 's sustainable development goals [2]. as well as inform more effective responses to natural disasters. At the same time, remotely sensed imagery can be used for unwarranted tracking of individuals. For example, academic papers and datasets about human and vehicle re-identification from high-resolution imagery exist and pose serious ethical concerss. Further, the biases of models that are over satellite imagery not well understood, and, if used in decision making, could systematically exacerbate existing disparities. TorchGeo makes easier for researchers to develop machine leaming models with remotely sensed data. Users of TorchGeo, and the wider machine learning and remote sensing communities, should consider the ethical consequences of their research when publishing work.
REFERENCES
[1] Wingtra AG. WingtraOne mapping drone and Sony RXIR II camera, 2021.
URL
https:/ /tippss//ingtta....
[2] Stephan Arnold, Jun Chen, and Olav Eggers. Global and complementary (non-authoritative) geospatial data for SDGs: Role and utilisation. Technical report, Inter-agency and Expert Group on Sustainable Development Goal Indicators (IAEG-SDGs), United Nations Statistical Commission, March 2019.
[3] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-awan self-supervised learning. In Procedings of the IEFF/CVF International Conference on Computer Vision, pp. 10181-10190, 2021.
[4] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Phillpee Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern PatternAnalyee Analysis and Machine Intelligence, (1): 423 2019. doi: 00.10099TPAMM1220827779886
[5] Adrian Boguszewski, Dominik Batorski, Natalia ZiembbaJankowsker Tomasz Dziedzic, and Anna Zambrzycka. LandCoverai: Dataset for automatic mapping of buildings, woodlands, water and roads from aerial imagery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Putlern Recognition, pp. 1102-1110, 2021.
[6] Jirka Borovec et al. TorchMetries: Machine learning metrics for distributed, scalable PyTorch applications, 2020.
[7] Claire Boryan, Zhengwei Yang. Rick Mueller, and Mike Craig. Monitoring US agriculture: the US Department of Agriculture, National Agricultural Statistics Service, Cropland Data Layer Program. Geocarto International, 26(5):341-355 2011.
[8] Gustau Camps- Valls, Devis Tuia, Xiao Xiang Zhu, and Markus Reichstein. Deep learning for the Earth Sciences: comprehensive approach to remote sensing, climate science and geosciences. John Wiley & Sons, 2021.
[9] Gong Cheng, Junwei Han, Peicheng Zhou, and Lei Guo. Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS Journal of Photogramme- try and Remote Sensing, 98: 19-132, 2014. ISSN 0924-2716. doi: 210.100/.. 10.000..
[10] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-188,, 2017. doi: 10.1 N0.110//POCC.227...7599999
[11] Alex Clark et al. Pillow: The friendly PIL fork (Python Imaging Library), 2010. URL
https://gitthen www www.pyotoonnopeeoocooocoooo wwwpahonniplooo/PPP1.. low/Pillow
[12] Rodrigo Caye Daudt, Bertr Le Saux, and Alexandre Boulch. Fully convolutional Siamese networks for change detection. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 4063-4067. IEFE, 2018.
10

---

Under review as a conference paper at ICLR 2022
[13] Rodrigo Caye Daudt, Bertr Le Saux, Alexandre Boulch, and Yann Gousseau. Urban change detection for multispectral earth observation using convolutional neural networks. In IGARSS 2018-2018 IEFE International Geoscience and Remote Sensing Symposium, pp.
IEEE, 2018.
[14] Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta:
A meta-learning library for PyTorch. arXiv preprini 190.000799 2019.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. lece, 2009.
[16] Matthias Drusch, Umberto Del Bello, Sébastien Carlier, Olivier Colin, Veronica Femaandzz Ferran Gascon, Bianca Hoersch, Claudia Isola, Paolo Laberinti, Philippe Martimort, et al. Sentinel-2: ESA's optical high-resolution mission for GMES operational services. Remote sensing of Environment, 120:25-36 36, 2012.
[17] Haoqi Fan, Tullie Murrell, Heng Wang. Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong. Nikhila Ravi, Meng Li, Haichuan Yang. Jitendra Malik, Ross Girshick, Matt Feiszli, Aaron Adcock, Wan- Yen Lo, and Christoph Feichtenhofer. PyTorchiideo: deep learning library for video understanding. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
[18] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with PyTorch Geometric. arXiv preprini M.Xiv:9933.0222 2019.
[19] Radiant Earth Foundation. CV4A competition Kenya crop type dataset, 2020. Version 1.0, Radiant MLHub.
[20] USDA Farm Service Agency (FSA). National Agriculture Imagery Program (NAIP). USDA Geospatial Data Gateway, 2015.
[21] Sean Gillies et al. Fiona: Fiona reads and writes goographic data files, 2011. URL
https:://qtthui
[22] Sean Gillies al. Rasterio: Rasterio reads and writes geospatial raster data, 2013.
URL
https:://i htt.ttttthhttgggggg com httpc//rattttrooorrseeeoooo
[23] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach self-supervised learning. arXiv preprint arXiv:2006.0773. 2020.
[24] Ritwik Gupta, Richard Hosfelt. Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xBD: A dataset for assessing building damage from satellite imagery, 2019.
[25] Patrick Helber, Benjamin Bischke. Andreas Dengel, and Damian Borth. EuroSAT: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2288 2019.
[26] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pp. 2712-2721. PMLR, M., 2019.
[27] Di Hu, Xuhong Li, Lichao Mou, Pu Jin, Dong Chen, Liping Jing, Xiaoxiang Zhu, and Dejing Dou. Cross-task transfer for geotagged audiovisual aerial scene recognition. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 68-84, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58586-0.
[28] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes ImageNet good for transfer learning? arXiv preprint arXiv:1608, 08614, 2016.
[29] Neal Jean, Sherrie Wang. Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tele2Vec: Unsupervised representation leamnigg for spatially distributed data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3967-3974, 2019.

---

Under review as a conference paper at ICLR 2022
[30] Z. Jin, C. Lin, C. Weigl. J. Obarowski, and D. Hale. Smallholder cashew plantations in Benin, 2021. Radiant MLHub.
[31] Krishna Karra, Caitlin Kontgis, Zoe Statman- Weil, Joseph C Mazzariello, Mark Mathis, and Steven P Brumby. Global land use/land cover with Sentinel 2 and deep learning. In 2021 IEEE International Geoscience and Remole Sensing Symposium IGARSS, pp. 4704-4707, IEEE, 2021.
[32] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp. 19-34, 2018.
[33] Iyy Loshchilov and Frank Hutter. Decoupled
weight decay regularization. arXiv preprint
arXiv:1711.05100, 2017.
[34] Dengsheng Lu, Qi Chen, Guangxing Wang, Lijuan Liu, Guiying Li, and Emilio Moran. A survey of remote sensing-based aboveground biomass estimation methods in forest ecosystems. International Journal of Digital Earth, 9(1):33-100 2016.
[35] Oscar Mañas, Alexandre Lacoste, Xavier Giro-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. arXiv preprint 1:Xiv2210311600) 2021.
[36] Diego Marcos, Michele Volpi, Benjamin Kellenberger, and Devis Tuia. Land cover mapping at very high resolution with rotation equivariant CNNs: Towards small yet accurate models. ISPRS Journal of Photogrammetry and Remote Sensing, 145:96-107, 2018.
[37] Manil Maskey, Rahul Ramachandran, Muthukumaran Ramasubramanian, Iksha Gurung, Brian Freitag, Aaron Kaulfus, Drew Bollinger, Daniel J. Cecil, and Jeffrey Miller. Deepti: Deep-learning based tropical cyclone intensity estimation system. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing. 13:4471-4288) 2020. doi: 10. 109/SSTARS 202.SSTTRRS.220....00 2020.301 1907.
[38] Francisco Massa et al. torchvision: Datasets, transforms and models specific to computer vision, 2016. URL htt.t///ee////ggtt....o..o/rooo.
[39] David J. Mulla. Twenty five years of remote sensing in precision agriculture: Key advances and re- maining knowledge gaps. Biosystems Engineering. 114(4):358-317 2013. ISSN 1537-5110. doi: 10. 1016/6 bos.booyseeessen22000........00 00.... Special Issue: Sensing Technologies for Sustainable Agriculture.
[40] T Nathan Mundhenk, Goran Konjevod, Wesam A Sakla, and Kofi Boakye. A large contextual dataset for classification, detection and counting of cars with deep learning. In European Conference on Computer Vision, pp. 785-800. Springer, 2016.
[41] NASA-IPPACT. ETCI 2021 competition on flood detection, 2021.
URL
https
7/nasa- impact.im.......mmm www.tgggithhhieoeeee.eeeee http:ti22//ccc/22/////
[42] Maxim Neumann, Andre Susano Pinto, Xiaohua Zhai, and Neil Houlsby. In-domain representation leaming for remote sensing. arXiv preprint arXiv: /91 /911119711111111111111111111 2019.
[43] NOAA Office of Satellite and Product Operations. NOAA geostationary operational environmental satellite (GOES) I-M and N-P series imager data. NOAA National Centers for Environmental Information, 1994.
[44] Jay S Pearlman, Pamela S Barry, Carol C Segal, John Shepanski, Debra Beiso, and Stephen Carman. Hyperion, space-based imaging spectrometer. IEEE Trunsactions on Geoscience and Remote Sensing, 41(6):1160-1177 2003.
[45] Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for Py PyToch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3674-3683, 2020.
[46] Caleb Robinson, Le Hou, Kolya Malkin, Rachel Soobitsky, Jacob Czawlytko, Bistra Dilkina, and Nebojsa Jojic. Large scale nigh-resouution land cover mapping with multi rullitiesooution data. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 12726-12735. 2019.
12

---

Under review as a conference paper at ICLR 2022
[47] John Rogan, Janet Franklin, Doug Stow, Jennifer Miller, Curtis Woodcock, and Dar Roberts. Mapping lave-cover modifications over large areas: A comparison of machine learning algorithms. Remote Sensing of Environment, 112(5):2272-222 2008.
[48] David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Galojevic-Dupome Natasha Jaques, Anna Waldman BBoww,, et al. Tackling climate change with machine learning. arXiv preprint arXiv: 1tt.//906009333 2019.
[49] Franz Rottensteiner, Gunho Sohn, Jaewook Jung. Markus Gerke, Caroline Baillard, Sebastien Benitez, and Uwe Breitkopf. The ISPRS benchmark on urban object classification and 3D building reconstruction. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1-3 (2012), Nr. 1, 1(1):293 2982 2012.
[50] David P Roy, Michael A Wulder, Thomas R Loveland, Curtis Woodcock, Richard G Allen, Martha C Anderson, Dennis Helder, James R Irons, David M Johnson, Robert Kennedy, et al. Landsat-8: Science and product vision for terrestrial global change research. Remote sensing of Environment, 145:154-177 172, 2014.
[51] Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, and Xiao Xiang Zhu. SENI2MS a curated dataset georeferenced multi-spectral Sentinel-12 1/2 imagery for deep learning and data fusion. arXiv preprint arXfn:1906.07789, 2019.
[52] Li Shen, Yao Lu, Hao Chen, Hao Wei, Donghai Xie, Jiabao Yue, Rui Chen, Yue Zhang, Ao Zhang, Shouye Lv, et al. S2looking: A satellite side-looking dataset for building change detection. arXiv preprint 20X7::200 2021. 2021.
[53] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? data, augmentation, and regularization in vision transformers. arXiv preprint 2tl::2000.1227700 2021.
[54] Gencer Sumbul, Marcela Charfuelan, Begümm Demir, and Volker Markl. Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. IGARSS 2019 2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 5901-5904, 2019. doi:
10.1109/GARSSS 8900532..
[55] Timu Sumisu. Canadian building footprints, 2019. URL httpp:::::/// http:://ggttuub..commmmmoooooo
[56] Maxar Technologies. WorldView satellite imagery, 2021.
[57] Xin-Yi Tong, Gui-Song Xia, Qikai Lu, Huanfeng Shen, Shengyang Li, Shucheng You, and Liangpei Zhang. Land-cover classification with high-resolution remote sensing images using transferable deep models. Remote Sensing of Environmnnt, 237:111322, 2020.
[58] Mehmet Ozgur Turkoglu, Stefano D' D'Aronco, Gregor Perich, Frank Liebisch, Constantin Streit, Konrad Schindler, and Jan Dirk Wegner. Crop mapping from image time series: deep learning with multi-scale label hierarchies. arXiv preprint arXiv:2102.088 08200, 2021.
[59] Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow. SpaceNet: A remote sensing dataset and challenge series. arXiv preprint arX.vv1807.01232, 2018.
[60] Cees J Van Westen. Remote sensing and GIS for natural hazards assessment and disaster risk management. Treatise on geomorphology, 3:259-298, 2013.
[61] Falcon William and The PyTorch Lightning team. PyTorch Lightning, March 2019. URL
htttps://github..rt http:://g..ghhaa...o/////ttttttr
[62] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. HuggingFace's transformers: 2019.
State-of-the-ar natural language processing. arXiv preprint
13

---

Under review as a conference paper at ICLR 2022
[63] Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching- Feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang. Jason Lian, Jay Mahadeokar, Jeff Hwang. Ji Chen, Peter Goldsborough, Prabhat Roy. Sean Narenthi- ran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville Beean,, and Yangyang Shi. TorchAudio: Building blocks for audio and speech processing. arXiv preprint arXiv:21 15018, 2021.
[64] Yi Yang and Shawn Newsam. Fag-of-visuallwordd and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pp. 270-279, 2010
[65] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features deep neural networks? arXiv preprint 14X1::4411.179 2014.
[66] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[67] Guanheng Zhang et al. torchtext: Data loaders and abstractions for text and NLP, 2016. URL
[68] Jixian Zhang. Multi-source remote sensing data fusion: status and trends. International Journal of Image and Data Fusion, 1(1):5-24, 2010.
[69] Wei Zhang. Ping Tang, and Lijun Zhao. Remote sensing image scene classification using CNN-CapsNet. Remote Sensing, 11(5):494, 2019.
[70] Weixun Zhou, Shawn Newsam, Congmin Li, and Zhenfeng Shao. PatterNNe: A benchmark dataset for performance evaluation of remote sensing image retrieval. ISPRS Journal of Photogrammetry and Remote Sensing, 145:197-209, 2018.
[71] Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang. Lichao Mou, Hossein Bagheri, Matthias Häberle, Yuansheng Hua, Rong Huang, al. So2Sat LCZ42: A benchmark dataset for global local climate zones classification. arXiv preprint arXiv: 12111,,, 2019.
[72] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 2016.
A APPENDIX
A. DATA LOADER BENCHMARKING
In order benchmark the performance of our data loaders, we download 114 Landsat 8 scenes and I Cropland Data Layer (CDL) file for the year of 2019, All files are stored as Cloud Optimized GeoTIFFS (COGs) and kept in their original CRS (Albers Equal Area for CDL and UTM for Landsat). Files are uncompressed with block size of 512. Experiments are run on Microsoft Azure with a 6-core Intel Xeon E5-2690 CPU. All data stored on local SSD attached to the compute node, Batch size and random seed are varied while the remaining hyperparameters are kept fixed. Total epoch size is 4096, patch size is 224, stride is 112, and number of workers for parallel data loading is set to 6.
A.2 BENCHMARK DATASET EXPERIMENTS
For experiments, we the pre-defined training, validation, and testing splits in all datasets. We perform a small hyperparameter search with a single fixed random seed on each dataset. Specifically we search over the best validation performance over the grid: learning rate E (0.01_0.001,00000 function E {cross entropy, jaccard], weight initialization e {random, ImageNet] and model architecture (ResNet18, ResNet50] [ With the So2Sat and EuroSAT datasets we also run experiments that use all the Sentinel 2 bands vs. only the RGB bands. In the cases where we use ImageNet weights with imagery
loss
*For the classification problems we use the ResNets as is and for the semantic segmentation datasets use the ResNets the encoder model in U-Net.
14

---

Under review as a conference paper at ICLR 2022
that has more than RGB bands randomly initialize the non-RGB kernlls in the first convolutional layer of the network and denote this setting as ImageNet (+ random). In all cases we use the AdamW optimizer [33], reduce learning rate validation loss plateaus, and early stop based on validation loss. We repeat the training process with 10 different seeds using the best performing hyperparameter configuration and report the test set performance over these models.
A.3 TORCHGEO CODE EXAMPLE
TorchGeo is designed to be simple and easy to use, and provides familiar API for users who have experience using libraries like torchvision [38] In Listing[] we provide an example code snippet showing how to use TorchGeo. In this example, we first instantiate dataset objects for Landsat 7 and 8 [50]. We then take the union of these two datasets, treating them as equivalent such that doesn't 't matter which satellite the imagery comes from. Next we instantiate a Cropland Data Layer (CDL) 7 dataset. We perform an intersection between our Landsat and CDL datasets so that we sample only from the regions of overlap between the two. We also instantiate PyTorch sampler, allowing us to sample from this dataset using spatiotemporal queries. All use of latitude/longitudee bounding boxes, coordinate reference systems (CRS), and spatial resolution abstracted away from the user, allowing non-remote sensing experts to work effectively with geospatial data. All datasets and samplers are compatible with PyTorch's 's Datal Loader class, making easy modify existing Py PyTorch workflow to handle geospatial data.
from torch. utils, data import Datalcader from torchgeo..
datasets corchgeo.datasets import Landsat?, Landsat8, CDL from torchgeo. samplers import RandenGeoSampler
Take the union of all Landsat 7 and 8 imagory landsat7 Landsat (roct="...")
Landsat8 Landsat 8 (root="...")
landsat landsat7 | landsat
Take the intersection of Landsat and CDL data cdl = (DL(root="....
dataset = landsat edi
Sample km"2 2 image patches from the intersection of the datasets sampler RandomGeoSampler (dataset, size=1000, length-512)
Use the dataset and sampier normal in Pyforch Datalloader dataloader DataLoader (dataset, sampler-sampler, batch_size=32) for batch in dataloader:
Train modeT, or make predictions using a pre-trained model
Example TorchGeo code for creating a joint Landsat 7/8 50 and Cropland Data Layer (CDL) dataset, and using such a dataset with standard Py DataLoader class.
Listing 1:
A.4 PRE-PROCESSING ALIGNMENT WITH GDAL
As an example of an alignment pre-processing workflow, we assume that have a Landsat 8 scene and a Cropland Data Layer (CDL) raster ("cdl.tif") which completely covers the extent of the Landsat scene. We would like create pixel-aligned version of these two layers. Given that the Landsat 8 scene has CRS of "EPSG:32619". height of 8011 pixels, a width of 7891 pixels, and spatial bounds of (186585, 4505085, 423315, 4745415), the corresponding GDAL command would create a cropped version of the CDL layer that is aligned to the Landsat layer:
gdalwarp \
stas EPSG: 32619
15

---

Under review as a conference paper at ICLR 2022
-of COG
-te 186585 4505085 423315 4745415 -ts 7891 8011 \
cdl.tif aligned aeiiied_ddl.ttte cif.tif
The spatial metadata of the Landsat scene can be determined through other GDAL command-ling tools (gdalinfo command), geospatial data packages such as the "rasterio" package in Python, or through GIS software such as QGIS or AreGIS.
16


Reviews:
[{"review_id": "VRfUwZrxzgw", "reviewer": "Reviewer_xaxq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper review for manuscript “TORCHGEO: DEEP LEARNING WITH GEOSPATIAL DATA”\n\nThis paper describes a newly created library for a geospatial data processing. The library is based on PyTorch and combines the functionality of common geospatial libraries. \n\n\nThe proposed Python package is designed to facilitate usage of deep learning modles with geospatial data. TorchGeo provides data loaders for common geospatial datasets, composable data loaders for arbitrary geospatial raster and vector data, samplers appropriate for geospatial data, models, transforms, and model trainers. TorchGeo allows users to bypass the common pre-processing steps necessary to align geospatial data with imagery and performs this processing on-the-fly. We benchmark TorchGeo dataloader speed and demonstrate how TorchGeo can be used to create reproducible benchmark results in several geospatial datasets. TorchGeo can serve as a platform for performing geospatial machine learning research. \n\nMain paper contributions are stated to be as following:\n1. providing data loaders for common geospatial datasets from the literature;\n2. composable data loaders for arbitrary geospatial raster and vector data with the ability to create pixel-aligned patches of data on-the-fly;\n3. augmentations that are appropriate for multispectral imagery;\n4. data samplers appropriate for geospatial data.\n\n== Comparison to previous work ==\nPreviously, the same result (loading the data, aligning the bands, loading datasets) could be achieved by using a combination of a few packages, including gdal, rasterio and geopandas and common Python libraries (TF/Pytorch, etc.). The authors propose a more convenient way to do the same things, combined in one useful library. This library optimises data loading and aligning.\n==Theory==\nTheoretical part is not included in this research, it is more engineering paper.\n==Method==\nThe propose package includes common ML methods, used with GIS data: U-net, Res net etc. Some models are pre-trained on Image net, which improves their performance.\n=== Pros === \nThis is a great contribution, useful for beginner researchers in the field of ML for Earth Observation data. It provided useful tools for working with EO data and loading the dataset with an improved speed. \n===Cons===\nThe novelty of the proposed research is limited since it presents a package for data loading and model training rather than a new research method, a compilation of approaches presented before.\n\n=== Basis for recommendation ===\nI am borderline regarding this paper since it has limited scientific novelty. This is a good software engineering paper, however I’m not sure if this kind of paper is suitable for the main ICLR conference.\n=== Questions to address in the revision ===\nI suggest authors remove the “Transforms” part from section 3 (page 5, top) since it presents Index calculation (which is are simple arithmetic operations on numpy arrays of different combination of bands): (a) it’s too simplistic for a serious research paper; (b) Please decide if you want to use word “index” in explaining NDVI and other indices or not, currently it’s inconsistent. \n=== Minor comments and additional feedback (not necessary to address in the revision)===\np.1 “composable data loaders” requires explanation (or reference to the section when t’s explained)\n\n\n\n", "summary_of_the_paper": "This paper describes a newly created library for a geospatial data processing. The library is based on PyTorch and combines the functionality of common geospatial libraries.  ", "summary_of_the_review": "I believe that this is a great work and useful tool for scientific research, but I doubt that it is suitable for ICLR main conference: it's rather a software engineering paper than a novel research. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "IxGBMVCXUH-", "reviewer": "Reviewer_Xe57", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n\n* Provides a good description of domain challenges when working with data indexed by geospatial coordinates.\n\n* Provides ready-to-use torch datasets for 18 different benchmark geospatial datasets.\n\n* Provides mean/std over 10 runs to quantify uncertainty in experimental metrics in 4 of these dataset.\n\n* The torchgeo package abstracts away much of the geospatial domain knowledge and data-munging to make it easier for CV practitioners to experiment on geospatial datasets.\n\n* The question of if the inductive biases from ImageNet will transfer to geospatial domains is interesting and a question I've had myself. I think the results in this section are generally interesting wrt to how well ImageNet generalizes to other domains.\n\n\n\n## Weaknesses\n\n* I's suggest using a different acronym than EO when talking about Earth Observation. Perhaps just remote sensing (RS) is better? Working within this domain, I find EO is more strongly associated with Electro-Optical than Earth Observation, but this is just a suggestion and not a big deal. \n\n* The abstract claims it is is the first package to publish pretrained models with MSI imagery, but there are many variants of MSI imagery, so I think it is important to say what bands are used. I assume these are the Sentinel 2 bands? I think it would be sufficient to say that. \n\n* Indexing in terms of geo-spatial coordinates still requires domain knowledge. IMO a more natural (wrt to a computer-vision researcher) alternative method might be to index by pixels within a geospatial region at a given GSD (pixel resolution). But there are certainly advantages to the former. The point is that it is still domain knowledge.\n\n* Torch datasets return both data and target label, the choice of data and label will depend heavily on the task to be learned (e.g. patch category labels, segmentations). While this is fine for providing data loaders for pre-defined datasets, this does not provide a way to ingest custom geo-spatial datasets. This may be out of the scope of this paper, so I'm not sure how big of a weakness this is.\n\n* Benefit of a torch-specific library are by design restricted only to torch users. \n\n* Many of the challenges dealing with geospatial data (large image sizes, multiple input bands) are not unique to the geospatial domain. The medical domain has similar challenges, but the tooling for these domains is often developed independently without considering common core infrastructure issues that could be jointly addressed (e.g. openslide vs cog formats).  \n\n* It is unclear what \"cached\" means in Figure 3. I assume it's used wrt to GDAL's cache? I use GDAL heavily, but I'm unfamiliar with how it's internal cache works, but I would expect that it's some sort of LRU cache. In that case it's not clear how cached versus not cached is measured. The other possibility I considered is that perhaps the raw data is not stored as a COG, so there is some cache that creates a tile-based format for efficient patch sampling. That would make more sense to me wrt to this cache, but again it's unclear and I would encourage the authors to clarify here.\n\n* I disagree strong with the idea from Section 4.3 that controlling for the pseudorandom number generator is necessary to determine if benefits are caused by design decisions or just getting a good seed. This would imply that we should never train with non-deterministic algorithms like those employed in CUDNN. Instead I think it underscores the importance of gathering statistics with quantified uncertainty bounds about a method over multiple runs before determining if there is an improvement or not. If a PRNG is causing a spike in performance, you likely have a sharp-non-generalizing model.\n\n* The model evaluation metrics used are unclear. If this paper is going to claim reproducible results, I would like to know what software is used to calculate quality metrics. Are metrics baked into torchgeo? Is the pycocoutils implementation used? Is it something else?\n\n* I don't think you can claim SotA on RESISC45 if you don't have the same train/validation/test split as the original paper. Have you contacted the authors to try and get their split? Furthermore, you should ensure that you publish your train/val/test split in torchgeo so that future work can compare to yours. If this is already done, mention it and say that you were unable to acquire the original split, so caveat any potential SotA claim with that, and then say that your split is published in torchgeo.\n\n* The paper briefly mentions SAR data, but then seems to focus on mostly on RGB data in experiments, with the exception being So2Sat with includes some MSI data. There is no discussion of how augmentations might differ from RGB augmentations when applied to MSI or SAR, or at least what examples of some augmentations appropriate or not appropriate for each might be.\n\n* In the appendix I would like to know the COG settings used in this package. If data is not\n  provided in COG or some tiled format, is it cached as a COG? If so what\ncompression and blocksize settings are used?\n\n* In the appendix it's unclear how I would go about merging MSI datasets with different bands / resolutions. If I was loading and merging L8 and an S2 dataset, what would happen?\n\n* How does pixel alignment work when faced with Maxar imagery that may require RPC transformations with a digital elevation model in order to get accurate geo-localization?  Does torch-geo take that into account? Does it attempt to always return orthorectified data?\n\n* Does torch geo do anything to align images over time, say for change-detection datasets like XView2? Can I ask a dataset for all patches in a region within a certain time frame?\n\n* What are the license restrictions on the datasets that torch geo provides loaders for? Does it automatically download the datasets like torchvision does? I imagine this is why some datasets are not supported (e.g. xview datasets)? \n\n* It's unclear to me what the backend used to maintain the datasets and annotations are. Are they different for each loader or does torchgeo use a common data format to index all of the datasets.\n\n* At the end of section 2 it says the approach trades off space for time, and I can guess at what that means, but I think the authors should state it more explicitly.\n\n* The experiments in 4.2 show the effect of different dataloaders, but does not compare against any baseline naive method of sampling. It might help to underscore the importance of domain-aware dataloading methods if you compared to something like a torchvision ImageFolder dataset. I imagine it's lack of ability to work with tiled formats like COGs would clearly show the importance of your dataloaders.", "summary_of_the_paper": "Introduces the torchgeo Python module, which provides torch Datasets, augmentations, samplers, and pretrained models to provide reproducible data ingestion for deep learning on standard geospatial datasets.\n\nThese datasets are indexed in terms of geo-spatial coordinates, which make it natural to express the desired sampling pattern. \n\nThe paper investigates the question of how well ImageNet RGB models transfer to multispectral geospatial models. \n\nThe paper investigates the efficiency of different sampling strategies and provides experimental measurements. \n\nTrains baseline models and achieves competitive or state-of-the-art results for 4 datasets considered. ", "summary_of_the_review": "I work with geospatial data, and this paper does a good job at describing the challenges, and I think it's important that packages like this are developed and publicized. The paper doesn't introduce anything particularly novel, but I think the goal of providing reproducible methods for experimentation is technically significant enough to warrant publication. \n\nThere are a few issues in the weaknesses that I would like to see resolved, but if these are addressed I think this is a above the acceptance threshold. Particularly what I'd like to see is more discussion of how torchgeo handles MSI/SAR data and how (or if) these disparate modalities can be rectified and combined when merging different datasets.\n\nLastly, I'd like to mention that I've done a lot of work in this domain, and after the review period is over I'd be interested in connecting with the authors to share ideas and software in order to help make this package as generally useful as possible.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F_d0TCSVaH", "reviewer": "Reviewer_W3nK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Establishes an easily accessible and optimized pipeline for data processing that is often esoteirc \n- A wide range of datasets are available \n\nWeakness: \n- Can detail the optimization (CPU vs GPU) process, availability and ease of adding new metrics, plots, figures etc. \n- Give examples of tasks\n- Detail how this pipeline will actually be used, eg reduced time, reduced lines of code etc \n- Encourage people to add their own datasets to this library, and detail the process for doing so \n- Have datasheets for datasets also published", "summary_of_the_paper": "Optimized Data downloading, processing, and sampling especially for domain scientists can be cumbersome. These are often steps that are routinely taken and can be optimized well once so that the entire community benefits from the ML-readiness. This paper tries to create generalized pipelines that both domain scientists and AI researchers can use to quickly get up and running with their experiments. The paper gathers existing datasets and makes them available through TorchGeo, a PyTorch domain library. ", "summary_of_the_review": "The paper establishes a new domain specific library that is ML-ready", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AE42v1dFTqF", "reviewer": "Reviewer_6sDp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths : The authors implement various architecture combined with popular datasets\n\nWeaknesses: The technical novelty of the paper is low (mathematical modeling, deep learning architecture, processing, etc..)", "summary_of_the_paper": "The paper introduces TorchGeo, a Python library for integrating geospatial data into the PyTorch deep learning ecosystem. TorchGeo provides data loaders for a variety of benchmark datasets, composable datasets for generic geospatial data sources, samplers for geospatial data, and transforms that work with multispectral imagery", "summary_of_the_review": "Even though the library is interesting, the overall contribution of the paper is low and does not advance the field", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Y9MJJ0_eyq", "reviewer": "Reviewer_stxS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and has a good flow. The authors work upon a genuine issue of processing geo-spatial data with deep learning models. Geo-spatial datasets can be challenging compared to regular RGB imagery given the spectral bands, along with the embedded coordinate reference systems, that they have. The paper provides useful functions for indexing geo-spatial datasets on the basis of their bounding boxes, and pixel-aligning multi-sensor images, as well as data augmentations for multispectral images. The authors state that the library is also compatible with PyTorch dataloaders, which is good. \n\nTorchGeo right now provides a set of benchmark datasets, consisting of MSI and RGB images. I assume the library would have the potential to be expanded later on to consist of hyperspectral and UAV imageries, along with Lidar as well as DEM data. Having said that, I believe providing benchmark geospatial datasets would be incomplete without providing access to some ISPRS benchmarks (https://www.isprs.org/education/benchmarks.aspx) such as Potsdam and Vaihingen datasets, especially in Section 4.1 and Table 3.\n\nMy major concern is what authors mention in the second last line of the Introduction, (also evident from Tables 3 & 4) - “ImageNet pretraining significantly improves spatial generalization performance” - why is this so? Why does ImageNet pre-training perform better than in-domain training? Is this solely because of the difference in chosen hyperparameters? A more detailed discussion on this observation from the perspective of vision (covering concepts such as image features, edges, scale of the ImageNet images vs remotely sensed images etc.) would be helpful to the community. Please expand Section 4.4 in this regard.\n\nTable 2: Please give references to the labelled datasets (CDL, Chesapeak Land Cover etc.)\n\nTable 3: The table does not provide pre-trained models on multispectral or SAR images. It only provides pre-trained models for mostly RGB imageries. As your contribution is towards geospatial datasets such as multispectral and SAR images, you should also provide pre-trained models on these datasets. Example: datasets presented in Table 1.\n\nSection 3.2 and 4.2: Does any past work, such as in GDAL, also implement sampling methods? Can you compare the performance of your methods, their utility, as compared to the past methods? \n\nMinor Comments:\n\n - Please cite Figure 1 and Table 2 in the main text\n - There’s no need to embolden so many words, repeatedly, in Section 2.\n - As Earth is a noun, please capitalize the first letter of the word wherever it appears.", "summary_of_the_paper": "The paper presents a new PyTorch library, TorchGeo, specifically for processing deep learning models on geospatial datasets. The library majorly provides a variety of benchmark geospatial datasets, along with pre-trained models on multi-spectral and SAR images. The library also provides a set of pre-processing steps which can pixel-align patches of data on-the-fly. Apart from these major contributions, the library has functions for augmenting (transforming) and sampling geospatial datasets. The paper is overall, well written and easy to understand. ", "summary_of_the_review": "The paper makes processing geospatial datasets easier with deep learning algorithms, and is worthy of publication at another venue after responding to some concerns. The paper does not contribute to any aspect of representation/deep learning, and hence ICLR would not be a right fit. The contribution, the TorchGeo package, is more relevant for the remote sensing field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a Python library for geospatial data based on Pytorch, TorchGeo. TorchGeo is a useful tool for applying deep learning methods to geospatial data. The reviewers agrees the contribution of this library. It will help machine learning researchers to use geospatial data and help geospatial researchers to apply machine learnig methods. However, the technical contribution is low, and the novelty is not high enough since the results can be achieved by a combination of existing packages.
Decision: Reject