Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The DENETHOR dataset provides satellite imagery from Planet Fusion (3m/px, daily revisit, RGB + near-infrared), Sentinel-1 (10m/px, 6-day revist, radar), Sentinel-2 (10m/px, 5-day revisit, 12 spectral bands) for two non-overlapping 24km-by-24km regions in Germany. Labels include crop type (9 classes) for 4,500 crop fields, thresholded at at least 1000 square meters in size. Data for the first region are from 2018, while data for the second region are from 2019.

Notably, DENETHOR is the first publicly available benchmark dataset with daily, analysis-ready remote sensing data, since Planet Fusion imagery is typically not free for public use. Experimental results on crop type classification suggest the need to develop new machine learning techniques to take advantage of the high temporal resolution, multi-modal data and improve cross-domain generalization.
strengths: **S1) First publicly available benchmark dataset with daily, analysis-ready remote sensing data**

Especially since Planet Fusion imagery is typically not free for public use, DENETHOR provides researchers in the machine learning and remote sensing communities access to the latest generation of satellite imagery for analysis. It is reasonable to expect that such imagery becomes more common in the future, so developing models now can help us take advantage of the future imagery.

**S2) Positive societal implications**

Crop type mapping is critical for many use cases, including food security estimation.

**S3) Provides ML community with research tasks**

DENETHOR encourages development of machine learning methods that can handle:
- multi-modal data (Planet Fusion + Sentinel-1 + Sentinel-2)
- geospatio-temporal data
- domain shift (geospatial and temporal)
- noisy input data (clouds in Sentinel-2 imagery)
weaknesses: **W1) Difficulty untangling spatial vs. temporal generalization**

The training and testing data differ both in location and in year. This makes it difficult to disentangle the drop in generalization performance due to spatial differences vs. temporal differences. If data for both 24km-by-24km tiles could be provided for both years (2018 and 2019), then it would be easier to perform ablation studies.

**W2) Limited geographic coverage**

The dataset only includes crop data from two 24-by-24km regions in Germany, and the training set only includes one such region. Therefore, it is difficult to expect models trained on DENETHOR to perform well in other geographic regions. Especially given that crop type mapping is most critical in developing countries where data is scarce, the limited geographic coverage limits the utility of models trained on DENETHOR.


correctness: The benchmark seems well-designed. The authors took care to ensure no overlap between the training and test sets. Code for reproducing baseline models is available on GitHub.
clarity: No major clarity issues identified. However, the paper seems to use a large amount of remote-sensing jargon, which the broader machine learning community may be less familiar with.
relation to prior work: R1) As mentioned in (S1), this is the first work to publicly and freely release daily analysis-ready satellite imagery.

R2) While the paper discusses some specific benchmark datasets for crop type classification, the paper does not seem to highlight other sources of crop type classification data, such as the US Cropland Data Layer (CDL). Although the CDL is not necessarily "benchmark-ready,"  a discussion of how DENETHOR is different from the CDL and related datasets would be appreciated.
additional feedback: N/A
review_title: Valuable Contribution to ML for Remote Sensing Community

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a dataset that provides daily and high resolution analysis ready remote sensing data for crop type monitoring. They collect the data through a commercial provide of high-resolution satellite imagery which is not typically freely available. They show that different deep-learning baselines do not perform well on this new data, suggesting it may require new methods to fully exploit the new data.
strengths: This dataset may have good and useful social implications. It provides the research community with high quality remote sensing data, which is typically not free, to develop algorithms for monitoring crop.
weaknesses: One of the main contributions of this paper is that it suggests that we may need new approaches to fully exploit the data. But I do not think the experiments provide a strong support for it. The authors evaluate existing baselines on the new dataset and show that they only slightly outperform random forest. It would be better if the authors can also compare existing baselines with random forest on existing dataset. Without such comparisons, it is hard to tell whether this dataset is really more challenging and warrants new approaches. Furthermore, one of the baselines achieves an accuracy of 67.25% on the test set which seems to be quite high already, considering that the model is small as it can be trained with a single mid-tier GPU from a few years ago.

It would be great if the authors can provide more details on the data, such as the cost of the data if they are allowed to disclose such information. This would be beneficial to the community if they find the data useful and would like to get data upon it.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: Not applicable
review_title: A new dataset for daily crop monitoring from space.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - Introduce and make available a novel, analysis-ready dataset (de-clouded and harmonized) of daily (time-series) remote sensing data for crop type classification in addition to S1 and S2 for 4,500 fields at 3m spatial resolution.
- Discuss new challenges that need to be tackled in order to effectively utilize the proposed dataset. This includes domain-shift, arable land classification without field boundaries, tackling class imbalance, and utilizing increased temporal density.
- Baseline experiments investigate domain shift due to temporal variation (out-of-year-generalization).
- Suitably describes limitations of this work (and the niche for this application in general).
strengths: - The dataset is analysis-ready and also includes S1 and S2 inputs to the train and test tiles allowing multi-modal experiments. It is the first "daily" dataset in remote sensing enabling new methods for crop detection in greater temporal density.
- The dataset is of high quality (ARD, QA) and in compliance with EU and regulatory bodies.
- Dataset does not include field boundaries making it a useful challenge in Arable land classification that could lead to works that effectively segment crops directly. 
- Logically sound baseline experiments and comparison with deep and classical ML methods. Authors show that conventional methods can still surpass deep methods (except with pixel-set encoding and self-attention). But, they also state that "current methods do not fully exploit temporal data at scale".

weaknesses: - A general tone of the work has been that new challenges (Eg: class imbalance, domain shift, temporal shift, etc) are to be tackled by new (to be investigated) methods rather than by modifying the data collection or by suggesting specific solutions/ideas.
- Collection of only 4 spectral bands (RGB+NIR) as compared with Sentinel (12 bands).
- Can the authors discuss how training models on their dataset can enable generalizations to crops grown in other regions of the world especially when the spectral signatures captured will vary, and this dataset captures only 4 bands leading to a lower scope in indices-based analysis.
correctness: Yes, the submission is a dataset constructed in a manner useful for the remote sensing and machine learning communities. The evaluation methods and experiments provide insights on limitations in current machine learning approaches and show that new directions of research are necessary. The dataset and the code were shared.
clarity: Yes, the paper is clear and well written.
relation to prior work: Yes, this work suitably discusses prior art and also explains limitations in previous methodologies that make this contribution useful for the community. Compared with datasets in Table 1, the size is understandably higher, and the dataset covers 4,500 fields in 3m spatial resolution from Northern Germany.
additional feedback: Minor edit:
- L8: Operabel -> operable
review_title: Useful and novel real-world remote-sensing dataset that also sheds light on inadequacies in current learning algorithms

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a dataset that includes daily remote sensing data for crop monitoring. 
The dataset contains daily Planet Fusion data, Sentinel-1 radar, and Sentinel-2 optical time series for crop type classification in Northern Germany. The proposed dataset could be useful for developing novel machine learning approaches for crop type monitoring.
strengths: 1. The paper is well motivated and well written. The authors demonstrate the usefulness of the temporal dimension for crop type classification.
2.  The paper provides a daily remote sensing dataset, which could facilitate research on crop type classification.
weaknesses: 1. The dataset is mainly in Northern Germany and is not geographically diverse. Thus, the results and conclusions might not be generalizable to other countries.
2. The dataset documentation on the website needs to be improved. 


correctness: The dataset is constructed in a sound way. The experiment design looks reasonable.
clarity: Yes.
relation to prior work: Yes. 
additional feedback: It would be great to expand the dataset to include data from different parts of the world.
review_title: A dataset for monitoring crop

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: the author introduced a daily remote sensing data for crop type classification: the DynamicEarthNET dataset for Harmonized, inter-Operabel, analysis-Ready, daily crop monitoring from space.
strengths: 1. The data is not only geographically precise but also of high quality.
2. The experiment is sufficient.
weaknesses: The layout and format of the paper still need to be improved
correctness: yes
clarity: yes
relation to prior work: yes
additional feedback: no
review_title: An interesting daily remote sensing data for crop type classification

Meta-review: All reviewers recommended accept. AC finds no grounds to overturn the consensus.
Decision: Accept