Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This dataset (CropHarvest) contains nearly 90,000 geographically-diverse samples of the earth’s agricultural lands, labeled and made ready for ML-based applications by linkages with four remote sensing data products. This dataset is spatially and semantically comprehensive as well as coherent. The authors test out four sample different models to evaluate the performance of the model across three sample locations – Togo, Brazil, and Kenya – and find that model performance varies by location but that the global dataset offers some useful performance improvements even for these localized tasks. The dataset offers significant value in reducing the amount of pre-processing needed for spatially explicit ag research in a way that can accelerate research on applications relating to global food security, climate change, and agricultural development. 
strengths: As someone who works in the agricultural/remote sensing nexus, I can attest that a lack of harmonized labels and substantial effort to make EO data “ML-ready” is an enormous challenge. It is because of this that only some of the most well-funded groups have been able to pursue analyses in these areas, and many projects have withered on the vine for lack of ability to use such data. This is an important and timely contribution that I suspect will prove very useful for advancing research in this area and be well cited.

I further applaud the team for encouraging additional contributions to the dataset and providing instructions for new contributions to enhance its utility across an even broader set of actors and regions as new data emerges. Bravo!

weaknesses: 
Unless I’m missing detail, it seems the authors may be conflating the definition of land use (economic uses of land) with land cover (different types of crops). It seems that perhaps land cover may be a closer fit. This might seem like a minor issue, and many times land use and land cover appear together (like in the NASA LCLUC program of which the authors I’m sure are connected to), but several people get rather itchy around the specific definition of each of those terms, so at least I’d recommend the authors review which they intend. 

Although I understand the desire to include meteorological data as part of the product options, the 31km spatial resolution seem much too coarse a resolution be useful for most agricultural applications. However, I am open to being wrong on this and will be curious to see what other researchers do with the data at larger scales. The other attached remotely sensed datasets (sentinel-1 SAR data, least cloudy Sentinel-2 images, and the SRTM DEM) are very reasonable.

Also it’s an important first step to have land cover labels – an additional hope is that a similar effort can be done for evaluating agricultural yields. 

correctness: The dataset appears to be constructed in a thoughtful way and is well documented.
clarity: The paper is clearly written and easy to understand.
relation to prior work: The manuscript reflects a clear linkage to the existing problems in the literature and a knowledge of what their dataset has the potential to enable. 
additional feedback: I’m looking forward to this paper and the dataset being published so I can share it with others. Kudos on an excellent and important effort. I foresee this being well-cited.
review_title: CropHarvest unifies a variety of previously fractured public and private datasets to accelerate machine-learning based applications of global agricultural production.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: + CropHarvest is a satellite dataset of nearly 90,000 geographically-divers samples with agricultural labels. This dataset harmonizes 20 datasets with crop type labels, including existing public datasets and new datasets released with the paper.  
+ An ML-friendly API inspired by the torchvision package is released for accessing the dataset.
+ There are three benchmark tasks for evaluating the performance of models in a range of agro-ecologies and dataset-size regimes. 
+ Data is very rich. There are Multispectral Optical Images, Synthetic  Aperture  Radar  (SAR)  Data, Meteorological Data, and Topographic Data. 

strengths: + Data richness.
+ Broad interest for both, pure ML researchers and also for ML researchers interested in climate change and sustainable development. 
+ Global coverage (world level).
+ Accessibility. 
+ It will also enable the research into methods that incorporate geographic or spatial information since the location metadata is provided in addition to the input data and labels.
weaknesses: + There are some limitations of the dataset related to the representativeness of the data, but this is a minor issue once it has been identified. 
correctness: The dataset is constructed in a sound way and a large effort has been devoted to lowering the existing barriers to access this kind of data. 
The evaluation of the benchmark task is well done and the conclusions are very informative. 
clarity: The paper is well written. 
relation to prior work: Existing work has been extensively reported in the paper. 
additional feedback: Good work!
review_title: CropHarvest

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The author presents the CropHarvest—a satellite dataset of nearly 90,000 geographically-diverse samples with agricultural labels.
The author makes reasonable use of existing satellite products to obtain the large dataset.

strengths: The author makes reasonable use of existing satellite products to obtain the large dataset.
The experimental analysis is reasonable
weaknesses: The format of the article needs to be improved
correctness: yes
clarity: yes
relation to prior work: yes
additional feedback: no
review_title: a interesting global satellite dataset for crop type classification

Meta-review: All reviewers recommend acceptance and the AC has no reason to overturn the reviewers' decision. What an exciting dataset!! Congratulations to authors!
Decision: Accept