Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a newly created library for a geospatial data processing. The library is based on PyTorch and combines the functionality of common geospatial libraries.  
main_review: Paper review for manuscript “TORCHGEO: DEEP LEARNING WITH GEOSPATIAL DATA”

This paper describes a newly created library for a geospatial data processing. The library is based on PyTorch and combines the functionality of common geospatial libraries. 


The proposed Python package is designed to facilitate usage of deep learning modles with geospatial data. TorchGeo provides data loaders for common geospatial datasets, composable data loaders for arbitrary geospatial raster and vector data, samplers appropriate for geospatial data, models, transforms, and model trainers. TorchGeo allows users to bypass the common pre-processing steps necessary to align geospatial data with imagery and performs this processing on-the-fly. We benchmark TorchGeo dataloader speed and demonstrate how TorchGeo can be used to create reproducible benchmark results in several geospatial datasets. TorchGeo can serve as a platform for performing geospatial machine learning research. 

Main paper contributions are stated to be as following:
1. providing data loaders for common geospatial datasets from the literature;
2. composable data loaders for arbitrary geospatial raster and vector data with the ability to create pixel-aligned patches of data on-the-fly;
3. augmentations that are appropriate for multispectral imagery;
4. data samplers appropriate for geospatial data.

== Comparison to previous work ==
Previously, the same result (loading the data, aligning the bands, loading datasets) could be achieved by using a combination of a few packages, including gdal, rasterio and geopandas and common Python libraries (TF/Pytorch, etc.). The authors propose a more convenient way to do the same things, combined in one useful library. This library optimises data loading and aligning.
==Theory==
Theoretical part is not included in this research, it is more engineering paper.
==Method==
The propose package includes common ML methods, used with GIS data: U-net, Res net etc. Some models are pre-trained on Image net, which improves their performance.
=== Pros === 
This is a great contribution, useful for beginner researchers in the field of ML for Earth Observation data. It provided useful tools for working with EO data and loading the dataset with an improved speed. 
===Cons===
The novelty of the proposed research is limited since it presents a package for data loading and model training rather than a new research method, a compilation of approaches presented before.

=== Basis for recommendation ===
I am borderline regarding this paper since it has limited scientific novelty. This is a good software engineering paper, however I’m not sure if this kind of paper is suitable for the main ICLR conference.
=== Questions to address in the revision ===
I suggest authors remove the “Transforms” part from section 3 (page 5, top) since it presents Index calculation (which is are simple arithmetic operations on numpy arrays of different combination of bands): (a) it’s too simplistic for a serious research paper; (b) Please decide if you want to use word “index” in explaining NDVI and other indices or not, currently it’s inconsistent. 
=== Minor comments and additional feedback (not necessary to address in the revision)===
p.1 “composable data loaders” requires explanation (or reference to the section when t’s explained)




summary_of_the_review: I believe that this is a great work and useful tool for scientific research, but I doubt that it is suitable for ICLR main conference: it's rather a software engineering paper than a novel research.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new PyTorch library, TorchGeo, specifically for processing deep learning models on geospatial datasets. The library majorly provides a variety of benchmark geospatial datasets, along with pre-trained models on multi-spectral and SAR images. The library also provides a set of pre-processing steps which can pixel-align patches of data on-the-fly. Apart from these major contributions, the library has functions for augmenting (transforming) and sampling geospatial datasets. The paper is overall, well written and easy to understand. 
main_review: The paper is well written and has a good flow. The authors work upon a genuine issue of processing geo-spatial data with deep learning models. Geo-spatial datasets can be challenging compared to regular RGB imagery given the spectral bands, along with the embedded coordinate reference systems, that they have. The paper provides useful functions for indexing geo-spatial datasets on the basis of their bounding boxes, and pixel-aligning multi-sensor images, as well as data augmentations for multispectral images. The authors state that the library is also compatible with PyTorch dataloaders, which is good. 

TorchGeo right now provides a set of benchmark datasets, consisting of MSI and RGB images. I assume the library would have the potential to be expanded later on to consist of hyperspectral and UAV imageries, along with Lidar as well as DEM data. Having said that, I believe providing benchmark geospatial datasets would be incomplete without providing access to some ISPRS benchmarks (https://www.isprs.org/education/benchmarks.aspx) such as Potsdam and Vaihingen datasets, especially in Section 4.1 and Table 3.

My major concern is what authors mention in the second last line of the Introduction, (also evident from Tables 3 & 4) - “ImageNet pretraining significantly improves spatial generalization performance” - why is this so? Why does ImageNet pre-training perform better than in-domain training? Is this solely because of the difference in chosen hyperparameters? A more detailed discussion on this observation from the perspective of vision (covering concepts such as image features, edges, scale of the ImageNet images vs remotely sensed images etc.) would be helpful to the community. Please expand Section 4.4 in this regard.

Table 2: Please give references to the labelled datasets (CDL, Chesapeak Land Cover etc.)

Table 3: The table does not provide pre-trained models on multispectral or SAR images. It only provides pre-trained models for mostly RGB imageries. As your contribution is towards geospatial datasets such as multispectral and SAR images, you should also provide pre-trained models on these datasets. Example: datasets presented in Table 1.

Section 3.2 and 4.2: Does any past work, such as in GDAL, also implement sampling methods? Can you compare the performance of your methods, their utility, as compared to the past methods? 

Minor Comments:

 - Please cite Figure 1 and Table 2 in the main text
 - There’s no need to embolden so many words, repeatedly, in Section 2.
 - As Earth is a noun, please capitalize the first letter of the word wherever it appears.
summary_of_the_review: The paper makes processing geospatial datasets easier with deep learning algorithms, and is worthy of publication at another venue after responding to some concerns. The paper does not contribute to any aspect of representation/deep learning, and hence ICLR would not be a right fit. The contribution, the TorchGeo package, is more relevant for the remote sensing field.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Introduces the torchgeo Python module, which provides torch Datasets, augmentations, samplers, and pretrained models to provide reproducible data ingestion for deep learning on standard geospatial datasets.

These datasets are indexed in terms of geo-spatial coordinates, which make it natural to express the desired sampling pattern. 

The paper investigates the question of how well ImageNet RGB models transfer to multispectral geospatial models. 

The paper investigates the efficiency of different sampling strategies and provides experimental measurements. 

Trains baseline models and achieves competitive or state-of-the-art results for 4 datasets considered. 
main_review: ## Strengths


* Provides a good description of domain challenges when working with data indexed by geospatial coordinates.

* Provides ready-to-use torch datasets for 18 different benchmark geospatial datasets.

* Provides mean/std over 10 runs to quantify uncertainty in experimental metrics in 4 of these dataset.

* The torchgeo package abstracts away much of the geospatial domain knowledge and data-munging to make it easier for CV practitioners to experiment on geospatial datasets.

* The question of if the inductive biases from ImageNet will transfer to geospatial domains is interesting and a question I've had myself. I think the results in this section are generally interesting wrt to how well ImageNet generalizes to other domains.



## Weaknesses

* I's suggest using a different acronym than EO when talking about Earth Observation. Perhaps just remote sensing (RS) is better? Working within this domain, I find EO is more strongly associated with Electro-Optical than Earth Observation, but this is just a suggestion and not a big deal. 

* The abstract claims it is is the first package to publish pretrained models with MSI imagery, but there are many variants of MSI imagery, so I think it is important to say what bands are used. I assume these are the Sentinel 2 bands? I think it would be sufficient to say that. 

* Indexing in terms of geo-spatial coordinates still requires domain knowledge. IMO a more natural (wrt to a computer-vision researcher) alternative method might be to index by pixels within a geospatial region at a given GSD (pixel resolution). But there are certainly advantages to the former. The point is that it is still domain knowledge.

* Torch datasets return both data and target label, the choice of data and label will depend heavily on the task to be learned (e.g. patch category labels, segmentations). While this is fine for providing data loaders for pre-defined datasets, this does not provide a way to ingest custom geo-spatial datasets. This may be out of the scope of this paper, so I'm not sure how big of a weakness this is.

* Benefit of a torch-specific library are by design restricted only to torch users. 

* Many of the challenges dealing with geospatial data (large image sizes, multiple input bands) are not unique to the geospatial domain. The medical domain has similar challenges, but the tooling for these domains is often developed independently without considering common core infrastructure issues that could be jointly addressed (e.g. openslide vs cog formats).  

* It is unclear what "cached" means in Figure 3. I assume it's used wrt to GDAL's cache? I use GDAL heavily, but I'm unfamiliar with how it's internal cache works, but I would expect that it's some sort of LRU cache. In that case it's not clear how cached versus not cached is measured. The other possibility I considered is that perhaps the raw data is not stored as a COG, so there is some cache that creates a tile-based format for efficient patch sampling. That would make more sense to me wrt to this cache, but again it's unclear and I would encourage the authors to clarify here.

* I disagree strong with the idea from Section 4.3 that controlling for the pseudorandom number generator is necessary to determine if benefits are caused by design decisions or just getting a good seed. This would imply that we should never train with non-deterministic algorithms like those employed in CUDNN. Instead I think it underscores the importance of gathering statistics with quantified uncertainty bounds about a method over multiple runs before determining if there is an improvement or not. If a PRNG is causing a spike in performance, you likely have a sharp-non-generalizing model.

* The model evaluation metrics used are unclear. If this paper is going to claim reproducible results, I would like to know what software is used to calculate quality metrics. Are metrics baked into torchgeo? Is the pycocoutils implementation used? Is it something else?

* I don't think you can claim SotA on RESISC45 if you don't have the same train/validation/test split as the original paper. Have you contacted the authors to try and get their split? Furthermore, you should ensure that you publish your train/val/test split in torchgeo so that future work can compare to yours. If this is already done, mention it and say that you were unable to acquire the original split, so caveat any potential SotA claim with that, and then say that your split is published in torchgeo.

* The paper briefly mentions SAR data, but then seems to focus on mostly on RGB data in experiments, with the exception being So2Sat with includes some MSI data. There is no discussion of how augmentations might differ from RGB augmentations when applied to MSI or SAR, or at least what examples of some augmentations appropriate or not appropriate for each might be.

* In the appendix I would like to know the COG settings used in this package. If data is not
  provided in COG or some tiled format, is it cached as a COG? If so what
compression and blocksize settings are used?

* In the appendix it's unclear how I would go about merging MSI datasets with different bands / resolutions. If I was loading and merging L8 and an S2 dataset, what would happen?

* How does pixel alignment work when faced with Maxar imagery that may require RPC transformations with a digital elevation model in order to get accurate geo-localization?  Does torch-geo take that into account? Does it attempt to always return orthorectified data?

* Does torch geo do anything to align images over time, say for change-detection datasets like XView2? Can I ask a dataset for all patches in a region within a certain time frame?

* What are the license restrictions on the datasets that torch geo provides loaders for? Does it automatically download the datasets like torchvision does? I imagine this is why some datasets are not supported (e.g. xview datasets)? 

* It's unclear to me what the backend used to maintain the datasets and annotations are. Are they different for each loader or does torchgeo use a common data format to index all of the datasets.

* At the end of section 2 it says the approach trades off space for time, and I can guess at what that means, but I think the authors should state it more explicitly.

* The experiments in 4.2 show the effect of different dataloaders, but does not compare against any baseline naive method of sampling. It might help to underscore the importance of domain-aware dataloading methods if you compared to something like a torchvision ImageFolder dataset. I imagine it's lack of ability to work with tiled formats like COGs would clearly show the importance of your dataloaders.
summary_of_the_review: I work with geospatial data, and this paper does a good job at describing the challenges, and I think it's important that packages like this are developed and publicized. The paper doesn't introduce anything particularly novel, but I think the goal of providing reproducible methods for experimentation is technically significant enough to warrant publication. 

There are a few issues in the weaknesses that I would like to see resolved, but if these are addressed I think this is a above the acceptance threshold. Particularly what I'd like to see is more discussion of how torchgeo handles MSI/SAR data and how (or if) these disparate modalities can be rectified and combined when merging different datasets.

Lastly, I'd like to mention that I've done a lot of work in this domain, and after the review period is over I'd be interested in connecting with the authors to share ideas and software in order to help make this package as generally useful as possible.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Optimized Data downloading, processing, and sampling especially for domain scientists can be cumbersome. These are often steps that are routinely taken and can be optimized well once so that the entire community benefits from the ML-readiness. This paper tries to create generalized pipelines that both domain scientists and AI researchers can use to quickly get up and running with their experiments. The paper gathers existing datasets and makes them available through TorchGeo, a PyTorch domain library. 
main_review: - Establishes an easily accessible and optimized pipeline for data processing that is often esoteirc 
- A wide range of datasets are available 

Weakness: 
- Can detail the optimization (CPU vs GPU) process, availability and ease of adding new metrics, plots, figures etc. 
- Give examples of tasks
- Detail how this pipeline will actually be used, eg reduced time, reduced lines of code etc 
- Encourage people to add their own datasets to this library, and detail the process for doing so 
- Have datasheets for datasets also published
summary_of_the_review: The paper establishes a new domain specific library that is ML-ready
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces TorchGeo, a Python library for integrating geospatial data into the PyTorch deep learning ecosystem. TorchGeo provides data loaders for a variety of benchmark datasets, composable datasets for generic geospatial data sources, samplers for geospatial data, and transforms that work with multispectral imagery
main_review: Strengths : The authors implement various architecture combined with popular datasets

Weaknesses: The technical novelty of the paper is low (mathematical modeling, deep learning architecture, processing, etc..)
summary_of_the_review: Even though the library is interesting, the overall contribution of the paper is low and does not advance the field
review_title: null

Meta-review: This paper develops a Python library for geospatial data based on Pytorch, TorchGeo. TorchGeo is a useful tool for applying deep learning methods to geospatial data. The reviewers agrees the contribution of this library. It will help machine learning researchers to use geospatial data and help geospatial researchers to apply machine learnig methods. However, the technical contribution is low, and the novelty is not high enough since the results can be achieved by a combination of existing packages.
Decision: Reject