Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a large scale satellite imagery dataset (WorldStrat). The dataset involves high-resolution satellite images covering 10,000 sq km of unique locations of various land uses across the world. Each high-resolution image is paired with multiple low-resolution images of the same location. The dataset can directly facilitate the computer vision task of image super-resolution for satellite imagery. This paper has also conducted benchmark experiments with three compute-efficient baselines. 
strengths: 1. The proposed dataset covers large area of locations and various types of land-use across the world, which is novel with respect to scale and diversity comparing with existing satellite imagery datasets.  
2. The dataset is fully released and can be accessed with ease. The accompanying document and code is also complete.  
3. The dataset has potential to benefit research community of both satellite imagery and computer vision.
weaknesses: 1. The writing may need be further refined. Some parts of the main text are a little redundant. It seems like a report with unimportant details rather than a succinct article. On the other hand, details and analysis of the benchmark experiments are not sufficient.  
  a. For example in Section 3.3, it is difficult for me to follow the redundant text together with the coarse figure. Figure 4 has inconsistent X-axis and strange number on the grey bar, making it difficult to get meaningful information.  
  b. In Section 4.1, the metric of 95% confidence interval (CI) should be explained at least briefly. You can't simply ask the reader to refer to other paper without any explanation.  
  c. In Figure 6, the right part looks strange that the three different baselines have exactly the same value on two metrics. The current explanation makes no sense for me. Further discussion is needed.  
2. The submission is not formatted in a review mode; there is no line number.
correctness: The claims are almost correct.
clarity: As commented above, the writing needs further refinement.
relation to prior work: The difference from related datasets is discussed in Introduction. It may be better to add a seperate section of Related Work to introduce related work of satellite imagery and image super-resolution in a more organized way.
additional feedback: Please see my comments above.
In addition, please refine the text carefully.   
  a. Please write full name when an abbreviation first apperars, e.g., IPCC and LCCS in Section 2.1.  
  b. In Figure 2, it is written 5000 $km^2$ for Underrepresented location, however, there is only 3895 $km^2$ right?  
  c. In the beginning of Section 2, what do you mean by The JIF dataset?
review_title: Review of "The WorldStrat Dataset" by Reviewer Sf2W

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a curated high-resolution satellite imagery dataset along with implementations of super resolution using this dataset. 
strengths: The main strength of this dataset is the availability in an easy to obtain manner and example code to implement super resolution techniques. The authors go to great lengths to address data selection and curation methodology and also address ethical concerns arising from the use of data from some under-served regions of the planet. One application of this dataset is discussed briefly.
weaknesses: There a few weaknesses in this paper 
- The paper uses many acronyms which are not always clearly defined when first used, making the paper difficult to read. For example, having to search the internet for acronyms that may be common knowledge to domain experts or authors themselves makes it harder to go through the paper. 
- There is not much discussion of the super resolution application. For instance, the paper discusses three architectures for benchmarking super resolution approaches but does not include any metrics that will help the reader understand the effectiveness of existing approaches. Similarly, there is no mention of actual metrics at all other than referring to another paper. 
- It would also be beneficial to highlight how the broader NeurIPS/AI/ML community can leverage this dataset for other applications in machine learning. 
correctness: The dataset is constructed appropriately. However, only one potential application is discussed and experiments are not described in great detail. The paper will benefit from a more detailed discussion of the experiments performed and metrics used.
clarity: The liberal use of acronyms throughout the paper made it a little hard to read, specifically because the reader needed to lookup definitions too often. Having clearly defined all acronyms when using them for the first time in the paper would go a long way towards making the paper clearer and easier to understand. At times, the Supplementary Material was actually better organized for understanding the paper. I understand that page limits pose a challenge, but perhaps there's a way to tighten up the structure of the paper and include some additional detail from the supplementary materials. (For instance, removing Figure 5 and using some of that space to discuss the super resolution benchmark comparison metrics)
relation to prior work: Yes, the authors do a good job of discussing prior contributions and how this work differs.
additional feedback: No additional feedback
review_title: Free High-Resolution Satellite Imagery: The WorldStrat Dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This dataset consists of high resolution Spot 6/7 data (1.5m/pixel resolution), paired with multiple Sentinel-2 revisits (10m/pixel resolution).

This dataset was specifically sampled to a) represent a variety of urban settings and land cover classes, b) cover areas of interest to charities such as Amnesty International.

The stated goal of this dataset (and demonstrated example) is to encourage the development of globally representative super-resolution modelling, allowing Sentinel-2 data to be upsampled to 1.5m resolution.
strengths: - Principled and well-motivated sampling means that models developed against this dataset are likely to be useful in a variety of geographical regimes and use cases. In particular, the combined spatial (under-represented) and semantic (land-use) sampling is of particular interest
- In addition to being well sampled, the dataset is large and allows for the exploration and utilization of high-resolution data to supplement a free data source.
- The dataset is well documented. In particular, the release of exploratory notebooks and code with which to train a model mean that others can begin building on top of this dataset
weaknesses: - The over-sampling of urban environments is a significant design decision which introduces a bias in the dataset; the motivation for this could be further discussed (since there are plenty of machine learning + remote sensing applications which aren't focussed on urban areas). In particular, by using 'urban' as a proxy for 'settled', the "settlement" portion of the dataset seems to significantly over-sample Europe and North America relative to other parts of the world (Figure 2).

- Related to the above, it would be nice to see some quantification on the geographic distribution of the dataset, since this can influence the applications of the dataset. In addition, worldwide representativity is an explicit goal of this dataset according to the paper.
correctness: The dataset appears to be thoughtfully constructed, and is well documented.
clarity: The paper and datasheet are clear and easy to read. 
relation to prior work: The paper clearly describes prior attempts to solving this problem, and their limitations. In addition, it considers the specific use case that this dataset can solve, framed in previous attempts to solve these problems.
additional feedback: None
review_title: An interesting and well considered dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: I worked on a Satellite Image project where each image has over 10B pixels. Such images may be illegal to put online.
Your project is very important and I believe it would have a great scientific value.

However, it may have legal issues.  Did you check with lawyers?  In the paper, it mentioned "European Space Agency’s Phi-Lab as part of the ESA-funded QueryPlanet project".  Besides laboratory regulations, there may be national-wide laws. Please check with lawyers.
strengths: Your project is very important and I believe it would have a great scientific value.
weaknesses: However, it may have legal issues. 
correctness: Please check with lawyers.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Please check with lawyers.
review_title: Could you please explain the regulation issues?

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The submission provides a **novel dataset for super-resolution in optical satellite images**. The dataset consists of repeated measures of medium resolution Sentinel 2 and a high resolution SPOT 6/7 reference satellite observation per sample. Key contributions are highlighted in **bold** font.

This is a solid contribution and I recommend for acceptance. The work may advance super-resolution approaches on satellite imagery---which is a popular topic, underlining the community’s needs for higher-resolution data. The authors do a great job of documenting their data and making it easily accessible, which raises the expectations that the submission may become of interest to the broader computer vision community. Overall, the idea and the motif underlying this submission may be well-established and there exists related prior work, but the execution of curating and documenting the dataset at hand is outstandingly well done.
strengths: ### Significance:
The submission provides a dataset of high resolution satellite data, which is attractive for remote sensing researchers (commonly limited by access constraints to low-to-medium resolution Earth observations) as well as the computer vision community (oftentimes focusing on more familiar very-high resolution UAV RGB data). High resolution satellite imagery is beneficial yet resource-consuming to obtain. Hence, the work’s significance is twofold: in making such images accessible over diverse and interesting areas, as well as in providing a super-resolution benchmark dataset on said observations.

### Relevance:
* Super-resolution is a competitive and active field in satellite imagery, underlining the community’s needs for observations of a finer spatial resolution than is commonly available. 
* The authors provide a global sampling following an in-depth geospatial stratification of areas, in order to accomplish a representative, diverse and interesting distribution of sites. The geospatial sampling is motivated in great detail and also regards typically under-represented land cover classes. The wide and heterogeneous coverage ensures the gathered data’s relevance for researchers focused on particular regions as well as those interested in whole-planet observations. 
* Machine Learning for social good is becoming a theme of increasing interest, and I am positive that the community will find value in the dataset’s curated high-resolution samples over critical sites, as identified by e.g. UNHCR and Amnesty.

### Accessibility and Accountability:
The authors do an outstandingly great job at making their contribution accessible. The dataset is hosted on zenodo, which I consider an appropriate platform. The submission comes with an associated python package, facilitating the distribution and access of the dataset. The provided documentation  follows the [“Datasheets for Datasets”](https://arxiv.org/abs/1803.09010) template, is in great detail and goes beyond the standards of comparable submissions. This is welcome, as multi-spectral remote sensing may still be novel to (parts of) the computer vision community and hence benefit of further explanations as provided therein.

### Ethical and social implications:
* The authors clarify that the utilized land use and land cover products, albeit containing potentially sensitive information (e.g. wr.t. refugee settlements), have readily been released by the corresponding parties (e.g. UNHCR). Furthermore, the authors consulted an independent to assess providing high-resolution imagery of potentially sensitive land cover may not entail any negative social implications.
weaknesses: ### Significance:
* The core idea and motif of the submitted dataset is fairly well-established. Submissions like [Märtens et al., 2019] and the very recent work of Michel et al (2022) are closely related. While the improvements on prior work are thus of a rather quantitative nature, the submitted dataset convinces in its quality of execution (e.g. careful stratification of areas etc).
* The experimental evaluation in the submission is kept fairly brief. With 3 models evaluated on the proposed data, the benchmarking is relatively compact, compared to other dataset submissions. The authors reference related multi-frame methods in section 4.1 and there’s additional single-frame methods that are of relevance [i, ii]. Super-resolution is a competitive and active field, and any extensive and readily available benchmarking may facilitate the prospects of the community adapting a novel dataset. Hence I would appreciate the author’s comments on whether they plan to extend the current evaluations (by e.g. benchmarking further models and conducting ablation studies on data) following an eventual acceptance or in the scope of future works.

### Relevance:
* The submission focuses on super-resolution in satellite imagery, which may be a more specific and rather novel domain for the broader super-resolution in computer vision community. But the submission does a good job of making such data more approachable and thereby contributing to the growing relevance of Earth observation to a broader audience.
* Earth observation imagery often has spatio-spectral characteristics specific to a particular instrument. That is, the provided SPOT 6/7 high-resolution data may be significantly different from e.g. the 0.5m resolution Maxar images provided in the SpaceNet challenge datasets [Van Etten et al., 2018] or the 5m resolution VENμS satellites in Michel et al. [2022]. Likewise, the spatio-spectral properties of Sentinel-2 differ from e.g. the commonly utilized Landsat missions’ sensors. In that sense, any upsampling facilitated by the submission will necessarily be sensor-specific and does not straightforwardly apply to other instruments.

###  Accessibility and Accountability:
While the authors do a good job making the dataset easily accessible and thereby facilitating its future adaptation, I do have some concerns about the data volume and whether the entirety of the included data is properly motivated. For instance, it is not apparent what the added benefits of the total 8+8 medium resolution observations are over e.g. a smaller set of associated observations. At best, this surplus of data may have benefited from a proper motivation backed by ablation experiments. At worst, this makes the dataset unnecessarily bloated and voluminous in file size without significant benefits. To avoid accidentally excluding researchers from working on the curated dataset due to potential computational constraints, I would suggest the authors address this issue by making it an option to solely download and extract single-frame data. The authors have readily provided a ‘Core’ version of their dataset on kaggle, containing only level 2A images and 8 rather than 16 medium-resolution samples. Further ablation experiments to motivate the full dataset and an option to access only single-frame data for works the likes of [i, ii] would be very much welcome. 

### Ethical and social implications:
As super-resolution is an ill-posed problem, reconstructing the signal could result in biased outcomes as [previously](https://twitter.com/Chicken3gg/status/1274314622447820801) reported in the community. I do consider satellite surveying being a potential risk application in these regards. While the authors did not specifically discuss the raised concern, I do however think that their data curation approach (at least to a sufficient extent) should resolve this issue: a) the dataset is sampled globally to obtain very heterogeneous areas b) including under-represented areas (shelters etc). And finally, c) multi-temporal medium-resolution observations may act as regularizers to learning the reverse mapping. I am positive the authors may provide further arguments to address the raised concern.


[i] Lanaras, C., Bioucas-Dias, J., Galliani, S., Baltsavias, E., & Schindler, K. (2018). Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network. ISPRS Journal of Photogrammetry and Remote Sensing, 146, 305-319.

[ii] Pouliot, D., Latifovic, R., Pasher, J., & Duffe, J. (2018). Landsat super-resolution enhancement using convolution neural networks and Sentinel-2 for training. Remote Sensing, 10(3), 394.
correctness: The claims are correct and sustained. However, I would like to add the following minor remark: Copernicus Sentinel 2, according to [ESA’s user guide](https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/overview) does actually come with 13 rather than 12 bands. Hence, the curated dataset does technically not contain the fulls spectrum of Sentinel 2 imagery, other than stated in the submission. As far as I may infer, the missing band is 10 (Cirrus), which may have been dropped by to SentinelHub’s level 2A preprocessing pipeline. However, I think this does not overly impact the submission, as the the missing band has a 60 m resolution and its primary purpose of atmospheric screening is mostly irrelevant for the super-resolution task.  Nonetheless, I suggest the authors either add the 13th band or clarify this in the main text. As I do consider this work of potential interest for the wider computer vision community, which may not be familiar with remote sensing instruments, it would be very welcome to avoid potential future confusions about the 12/13-bands matter. 
clarity: The paper is well written. While always remaining clear and understandable, the writing occasionally follows a rather informal style and appears somewhat quirky at times. Its okay as is, but I nonetheless suggest the authors make some minor edits to e.g. the Introduction section’s writing style.
relation to prior work: Relevant publications are cited and discussed in sections 1 and 4.1. The closest prior work is probably a dataset on single-image super-resolution by Michel et al. (2022), which was published very recently and close to the deadline however. While the given submission is similar in its spirit, it considerably extends on the scope, volume and diversity of preceding datasets.
additional feedback: The authors make the design choice to include medium-resolution satellite data featuring clouds. While arguably better reflecting real use cases, this somewhat mixes a pure super-resolution task with a denoising and image reconstruction task. I find this approach principally justified, but have some follow-up questions:
* What are the statistics on cloud coverage like, what are the mean and variance of cloud coverage? Do you observe considerable differences across some sampled regions or land cover types?
* Are you planning to provide pixel-wise cloud mask annotations for mask-guided reconstruction and super-resolution approaches?
* It would be great to provide code for evaluating on a distinct subset of the dataset exclusively containing cloud-free samples, in order to allow for focusing on the super-resolution task in isolation. Controlling for sample numbers while involving cloudy observations in a separate condition would make for an interesting comparison. This may provide insights about the difficulty added by the cloud-covered observations.
review_title: A solid super-resolution dataset, not only fitting the urging needs of the remote sensing community but also being attractive to a broader audience

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Cornebise, Orsolic, and Kalaitzis in "Free High-Resolution Imagery: The WorldStrat Dataset" present a medium-resolution (approx. 6m per pixel), multi-spectral (RGB, NIR) SPOT-satellite imagery dataset including a high-resolution (approx. 1.5m per pixel) equivalent through pansharpening, and a spatially-matched, temporally SPOT-centered timeseries of 16 twelve-band, 10m (RGB, NIR) to 60m (others) per pixel multi-spectral images as captured by the Sentinel-2 satellites. Sampling of data is with focus over our planet's land  based on three generic use case scenarios for Earth observation applications. The dataset will ship with code to pull the data online, and to perform an image super-resolution benckmark utilizing 3 corresponding deep neural network architectures.
strengths: - standardized code (PyTorch Lightning + integration to existing EO-Learn Python library) allows to verify and flexibly extend in future the dataset
- well-designed geospatial sampling strategy to maintain diversity of satellite imagery
weaknesses: As noted by the authors in Sect. 5, the dataset is limited to little cover features over water bodies, oceans, in particular. Consequently, it limits applications for vessel detection, fish farming, oil spills, etc.
Further, as it stands, without additional layers for e.g. semantic segmentation, the dataset is most valuable for super-resolution or unsupervised learning approaches. Regarding the term *high-resolution*, I suggest the authors expliticly note that Sentinel-2 and SPOT do not significantly differ without pansharpening techniques, i.e. 10m vs. 6m.
Moreover, I recommend to cut down the abstract and the main text to the point, reducing loose phrases in favor for more crisp statements, e.g. compare the lengths of the introduction, Sect. 1.1 vs. the size of the conclusions, Sect. 5.
correctness: please:
- drop (illegal?) Meteosat signal streaming story
- reference or drop vague statements such as *possibly the most widely used satellite imagery datasets*
- minor: please adjust typo *percent o will*
clarity: kindly:
- In the interest of a neutral research approach, please remove emotional phrases such as:
    * _Computer vision and satellite imagery seem to be a match made in heaven._, What does define the term *heaven*?
    * _from this admittedly elaborate figure_, What criterion does define *elaborate*? 
    * avoid the use of exclamation marks
- increase of annotation size in Fig. 3 improves reading
- please simplify & summarize Fig. 4
- properly introducing acronyms such as AOI (area of interest), POI (point of interest), etc. for easier access of researchers outside the scope of remote sensing
relation to prior work: I suggest:
- cite review of computer vision in remote sensing for multi-spectral imagery, e.g. from IEEE Geoscience & Remote Sensing journal
- list available geospatial data platforms beyond Google Earth
- a more comprehensive list of super-resolution datasets in general
- inclusion of recent developments in deep learning on super-resolution techniques is recommended;
  in particular this year's CVPR did present relevant works wrt. pansharpening
additional feedback: I highly value the efforts of the authors to make openly available a curated remote sensing data set as presented by Cornebise et al. However, the work needs adjustments according to the above before publication. The manuscript's scope for NeurIPS is rather restrictive to super-resolution of satellite imagery. Applications beyond require additional efforts such as geospatial data fusion in order to assemble the required labels. In particular, such efforts pose a barrier to researchers not skilled in processing earth observation data - even when convenient libraries such as EO-learn do exist.
review_title: assessment of paper as potential NeurIPS dataset contribution

Meta-review: metareview: This paper had 6 reviews. The reviewers’ comments were overall fine, except concerns and the writing style (ao84, Sf2W and 33Vk). Instead of addressing these concerns and showing that they were taken into consideration, the authors responded aggressively, e.g., to reviewer 33Vk. 

It has been noted that legal concerns have been addressed in the response to the reviewers’ comments, but in order to prevent any lawsuits against Neurips, I recommend that the authors include a footnote or a specific section in their revised paper to address the legal concerns (in accordance with their response to the reviewers' comments).

In terms of significance, although the work and effort are considerable to construct this dataset, “as it stands, without additional layers for e.g., semantic segmentation, the dataset is most valuable for super-resolution or unsupervised learning approaches” [33Vk]. Maybe future work could include semantic segmentation or add a section to explain how the dataset can be used for other tasks. 

Dataset is available on various platforms and documentation seems appropriate.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept