Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This submission presents a new remote sensing data set - the "Catalonia Multiresolution Land Cover Dataset (CatLC)" -  composed of images from aircraft, satellites and from existing topographic maps of the region of Catalonia. The aim is to provide a data set with good resolution covering a large and heterogeneous area to the research community to foster the development of deep learning based image segmentation models for remote sensing, especially for land cover maps. The data set, along with tutorials, code and descriptions is provided at a number of links hosted by the Institut Cartografic i Geologic de Catalunya.
strengths: - The CatLC data set indeed provides a novel, labelled collection of remote sensing images of unprecedented area coverage, directly available multi-layers and high resolution of 1 m, which can be used in the development of machine learning models for large scale computer vision tasks.

- It seems, after checking the provided tutorials and code, that the data set has indeed been curated in such a way that it can be mostly readily used, also by a non-expert in specific remote-sensing techniques. It thus should be directly usable for the broader computer vision community for development.
 
- While the main focus of CatLC seems to be to foster the development of models for land cover map segmentation, the range of different layers provided in the data set open up opportunities to explore sensor fusion techniques or focus on different aspects of the problem.
weaknesses: - It is not entirely clear to the reviewer what the exact aims of publishing the CatLC data set in this fashion are. It is clear that the general goal is to foster the development of deep learning based image segmentation methods for remote sensing. However some of the more specific goals are not clearly collected in one place in the text, such as whether it is to improve the automated labelling of land cover maps or to generally stimulating research and development, or whether the main audience is supposed to be general computer vision experts interested in remote sensing ,or remote sensing experts with knowledge in computer vision.

- In addition, the first benchmark study is very short and only using one type of images in the data set, which raises the question what the exact goals are for the specific combination of data types found in CatLC. 

- Unfortunately, the main part of the paper - the description of the CatLC data set itself - has some weaknesses in presentation, especially of details in the data, which could also be remedied by general better documentation (see below). A few major examples:
    * The land cover map containing the ground truth labels is not well explained, at least to a non-expert in land cover   remote sensing. How excatly have the original labels been obtained? What was the exact procedures used to work out the changes from 2009 to 2018, and is the 81% thematic accuracy obtained in the test sample a good value in this field?

    * The Orthophoto data set is described to brief and without enough detail. How many images at what sizes are available? What are the digital make-up tasks performed? Why the 1% percentage threshold?

    * For the sentinel-1 data, the reason struck the reviewer as odd that the authors suggested for why they left out part of the data because it would be better to reduce the information. Since CatLC is essentially a collection of different sub data sets, why not let the user decide how much information to use?  

- The benchmark study is very short and there is not enough room given for proper descriptions of the procedures and results (which may be just a presentation issue, since the code  - very briefly reviewed - seems quite solid). A range of questions are a bit unclear and left open for the reader: 

    * How was the evaluation done exactly? Has there been cross-validation procedures used? What was the validation data used for? Why not also display the accuracy? 

    * It also is a bit unclear how the results of that first benchmark are considered by the authors themselves. How do the authors judge the highly different performance over class labels, for instance? Is that a clear results of scarcity of some labels? Further, a comparison with typical results for this tasks in land cover maps is completely missing thus rendering the worth of the benchmark study a bit unclear. 
		
    * It is a bit misleading to consider this a benchmark of the whole data set, since essentially this is just a study on one part - the orthophoto data.

    * It would have been nice to see at least some discussion of possibilities here, e.g. other architectures beyond the U-net, or how to make use of the full data set together.

- While the data set has indeed been curated by remote sensing experts for a more general audience with tutorials and code are provided, the presentation in the paper is sometimes still very focused on specific terminology and techniques from remote sensing and thus not always easy to follow for a non-expert in that field, for instance in Section 3.3.
correctness: To the best of the knowledge and understanding of the reviewer the basic construction of the data set is fine and correct. Also the benchmark study itself seems to have been conducted correctly on the technical level judging by a brief overview of the code. The issues of the reviewer are more with presentation and discussion (see weaknesses and documentation).
clarity: There are some problems with clarity and structure in the paper that could be easily addressed and are thus not a major issue. Main examples:

- sometimes there are wording and grammar issues hindering the clarity of statements, e.g. in line 27-28, or 144 the use of the word "open" (has been considered? Open to the user?).  
 
- Image captions could be improved. For instance, it was quite unclear at first glance that all pictures cover the same area. 

- The aforementioned lack in clarity of the exact goals and aims of the CatLC data set. A short and concise bullet list in the introduction could be helpful, for example.

relation to prior work: The paper contains a section on previously published similar labelled land cover data sets and mentions the main features of those data sets. While the reader can compare the characteristics themselves, a clear comparison of the CatLC data set to these previous sets by the authors is lacking in the rest of the paper
additional feedback:  A specific question that the reviewer was wondering about is whether the random assignment of tiles over all classes to train, test and validation data in the benchmark study is really preserving the i.i.d. principle since there are probably quite some correlation between nearby regions in the map?

**** Final Comment after paper revision ****

The authors addressed many of the concerns raised in my review in their revised version. While I still think that there would be still more potential in presentation and analytical analysis of the data and benchmark study i clearly put my emphasis on the data set and the clear effort the authors put into collecting and presenting it to the wider computer vision community. I changed the rating accordingly to an accept.
review_title: Review of the NeurIPS 2021 Track Datasets and Benchmarks Submission "CatLC: Catalonia Multiresolution Land Cover Dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper describes and presents a public available aerial multi-modal image-based dataset of the Spanish region of Catalonia.
The dataset is labelled with a total of 41 classes from four main topics covering agriculture, forest, urban areas, and water bodies.
The authors evaluate the performance of a segmentation network on the RGB orthophotos from the dataset.
strengths: * The dataset can be of interest for communities working with image-based geographical analysis.
* The dataset is made public available with ground truth annotations of 41 labelled classes.
* The dataset consists of multi-modal images, which allow for a more in-depth analysis of the data, compared to e.g., RGB images alone.
* Compared to the presented related work, this dataset covers a significantly larger land area.
weaknesses: 1) Throughout the paper the authors make claims, which I do not find sufficiently backed.
* An example is on line 80-91 where the authors use the size of the land area and the quality and variety of the images as arguments for how suited the dataset is for "AI and deep learning techniques". This is followed by a vague description of the Catalonian landscape and how it is more heterogeneous compared to Alaska and the Saraha desert and therefore better suited for more applications. However,  this is only words and there are no apparent analysis to back up the claims.
* I miss an in-depth analytic evaluation of the proposed dataset, e.g., describing the distribution of each of the classes, such that it is comparable to other similar datasets.
2) The datasets presented in the related work section are described by somewhat varying terms, which makes them difficult to compare. The first two datasets [4] are described by "tiles", "spatial resolution", "bands", and "classes", but the next [5] is presented by "area", "spatial resolution", and "classes". The next is simply "8000 km of roads and four cities", and so on. I would suggest to standardize the description of the individual datasets (as good as possible) , such that they can be more easily and fairly compared to the proposed dataset. E.g., summarizing the findings in a table together with the proposed dataset, thereby providing both an overview of current similar datasets and a clear display of how the proposed dataset stands out.
* I would suggest the authors present the proposed dataset in a manner where it is more easily understandable for computer vision researchers (to make it more accessible). It is extremely difficult to comprehend how many images are in the dataset when only the spatial resolution of 1-10m/px and a total area of 32,000 sq km are provided.
3) The authors state that the "Slope and Aspect" maps (section 3.5.2) are calculated directly from the "Digital Elevation Model". They present no evaluation or discussion of how this additional information can be used to enhance performance of some arbitrary method. In other words, the "Slope and Aspect" maps contain redundant information as it is already available through the "Digital Elevation Model" images.
4) In section "Classification" (section 4.1, line 226-227) the authors state that they "... show the most optimal way to train a model with these data.", but they simply train a public available segmentation network and measure the performance, which has nothing to do with how to train a model in the most optimal way.
5) In section "Classification" (section 4.1) the authors provide an "initial benchmark". However, the authors state that the dataset is randomly divided into train, validation, and test splits. Are these splits published? Otherwise it can hardly be considered a benchmark since it cannot be replicated.
* Furthermore, the authors only use the RGB orthophotos and simply ignore the rest of the dataset. The evaluation could have been more informative if it was conducted in a way where the pros and cons of the various sensor types were illustrated.
* It is difficult to get anything from the results in table 2, when there are no information on the distribution of the classes. How are the classes distributed in the three splits? How are they distributed overall? Do we see overfitting on some of the classes?
correctness: To some degree.
Several claims may be correct, but is not backed up by sufficient scientific proof. See my comments.

clarity: To some degree. The language is understandable, but there are several claims, which are not backed by proof. See my comments  here and under "weaknesses" for examples.

1) I suggest to make it clear that the three figures in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, and Figure 10 are of the same area.
* Also, caption sub-figures (throughout the entire paper), such that it is possible to reference the individual sub-figures.
2) Remember dots at the end of captions.
3) I find the descriptions of the sensors/dataset parts (section 3.1 -> 3.5) of varying quality. 
* In section 3.5 "Topographic Maps" it says that there are five different topographic maps, but they are described in 4 subsections, which may be a bit confusing.
* Why are there no image examples of 3.5.2 and 3.5.3?
* Line 144-145 could the VH polarization provide additional valuable information? The dataset is already 1.4 TB, so the reasoning of not taking it into account in order to minimize the amount of information seems a bit odd.
* Line 149-162 is better suited for supplementary material in my opinion.
* Line 171-179 are also for the suppl. mat. in my opinion. I would also suggest to present images before and after they have been corrected, otherwise it is difficult to comprehend for people not familiar with e.g., level-2a images.



relation to prior work: I think the authors could make it more clear how their work differs from previous published datasets. See my comments in "Weaknesses"
additional feedback: I find the proposed dataset both interesting and relevant and it is my impression that the authors have put a lot of time and effort into constructing it.
However, I find it problematic that there is (more or less) no analysis of the dataset. E.g., how are the 41 classes distributed, how do the sensors support each other, etc.
I don't find it suitable for NeurIPS in it's current state.


** Comments after reading the second and updated version of the paper:
The authors have improved several parts of the paper based on the comments from the reviewers.
The analytic parts of the paper could still be improved, however, I choose to put more weight on the dataset itself, which I think could be interesting and relevant for the community. Based on the improvements I have changed my rating from "rejection" to "marginally above acceptance threshold".
review_title: Interesting aerial dataset lacking relevant analytical information

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a large multi sensor arial dataset consisting of a land cover map (segmentation labels), Orthophoto from airplane photos (RGB + IR), Satellite Photos (Sentinel 1, Sentinel 2) and Topography maps. The dataset covers Catalonia, Spain. The main motivation of the dataset is change detection and land cover predicting, where the authors present a simple baseline model for predicting the latter.  

strengths: - The dataset seems very interesting. It combines a lot of different sensors and seems to address relevant problems.
- The authors propose a simple baseline method for land cover segmentation.
weaknesses: - Unfortunately, the paper is poorly written. There are many places, where I was unsure about what was meant. I tried to elaborate on some of issues in the clarity section, but these are just a few examples.
- Evaluation: It seems weird that some of the classes are not present in the test set. How can you evaluate the performance on these classes if they are not in the test set? 
- Experiments: It is unclear from the experimental results if the many sensor modalities can improve the predictions and why they are relevant. I undertand that the authors wants the community to figure this out, but I believe that it would be a strong (and rather simple) point to show that multi sensors can improve the segmentation quality over the simple baseline.
- Experiments: Authors motivate two research problems for the dataset. (1) change detection (2) land cover segmentation. They do not give a baseline method for (1) nor explain how the proposed dataset can be used for change detection nor specify what changes users would be interested in. 
- Related work is incomplete (see below for more comments)
correctness: - It seems like beaches, port areas and sea are not in the test set. It is quite strange to have classes in the training set that are not in the test set, as you cannot know how well you perform on these classes if they are not in the test set. I would change the way you create the test/val/train division such that all classes are represented in each set, such that you can evaluate on all performance on the classes.
- It would be good with single number that sum up table 2. Mean IoU could be an idea? This is similar to COCO: https://cocodataset.org/#stuff-eval
clarity: - The introduction is poorly written. You can assume that the reader are familiar with CNN, classification and segmentation. I believe that the references to cat/dog does not add  valuable information, but rather confuses the reader. You could simply write that “pixel-wise human annotation of satellite images are challenging and dubious, and thus often requires manual annotation with tools like GSV”.
- I would recommend to delete the current introduction, and use section 4 “Applications” as a new introduction.  Make section 4.1 a separate section called something like “Experiments”.
- “The original images were taken with a resolution of 25 centimeters, but because the land cover map has a resolution of 1 meter, we have decided to rescale the orthophoto raster layer also to 1 meter.” Would it be possible also to release the original resolution. This might become relevant for some applications down the line, e.g. can the land cover map be refined wit the high resolution image?
-  I think the following sentence is difficult to understand: “The slope indicates the steepness of each pixel: The lower its value, the flatter the terrain is”. Does it mean “the pixel intensity/color indicates the terrain’s steepness: yellow means flatter terrain and blue mean more steep terrain”? 
- Weird to class this section “Classification” when you do “Segmentation”. These are two different tasks. The goal of classification is to find a label per image, and in segmentation you want to find a label per pixel.
-  “both to have a starting point and to show the most optimal way to train a model with these data”, be careful with stating that it is the most optimal way… if so it limits the use case of the dataset for the research community, but there is probably other ways to train.
- I assume that the input to the U-Net is just the RGB from the Orthophotos? It would be interesting to see what happens if you gave all the different sensor modalities as input to the U-Net? Hopefully the network could learn to use different modalities for different classes, which would be a nice motivation for the dataset. With the current experimental work, it is a bit difficult to see why we need all these sensor modalities?
relation to prior work: -  Seems incomplete. I believe this list of additional arial/satellite datasets would be valuable for the related work section: https://github.com/chrieke/awesome-satellite-imagery-datasets
- It would be beneficial with a table comparing the different datasets especially if the table could make it clear where the CatLC dataset differentiates compared to the other datasets.
additional feedback: - It would be interesting with some stats on how large each class of pixels are, e.g. there is probably very few pixels with “port”  and a lot of pixels with “Vineyards”.
- Table 2 is difficult to digest. It could be better to show e.g a bar plot. The performance for some classes seems very low. Is this because these classes are very poorly represented in the training set?
- As I understand change detection is also one of the motivations of the paper. I would therefore like to see a simple baseline for change detection also. As I understand you have labels for different seasons, which you could use a GT. You could create a simple baseline as an autoencoder with two encoders (one for spring image, and one for fall image) and one decoder, where information are fused in the bottleneck.
- Another simple, but relevant experiment would be to use the U-Net for different input modalities and see which modality is best for predicting land cover. One would expect the orthophotos as they have highest resolution, but how much worse are the satellite images?

###################

Additional comment after author responds


The authors have answered most of my points. I think the dataset is interesting enough to be accepted, and have therefore changed my vote to marginally accept. I believe especially the method section could be improved to better highlight the relevance of the dataset.
review_title: CatLC: Catalonia Multiresolution Land Cover Dataset

Meta-review: The paper presents a new land cover dataset comprising aerial images of Catalonia together with a number of semantic and geographic annotations. All reviewers felt that the dataset was interesting and likely to be useful; its scale is particularly impressive, as it covers the entirety of Catalonia.

The main weaknesses identified by all reviewers have to do with the clarity of the writing and presentation in the paper. All reviewers gave detailed examples of unclear passages, and suggested a large number of edits and improvements. The authors implemented many of these suggestions in their revised draft, which caused all reviewers to raise their scores to recommend acceptance. In their final comments, all reviewers mentioned that their final rating was ultimately weighted more toward the utility of the dataset, but that the paper itself still needs improvement.

After reviewing the reviewer comments and the paper, the AC agrees that the dataset itself is novel, interesting, impressive, and worthy of publication. The AC was impressed by the willingness of the authors to incorporate feedback from the reviewers to improve the paper; however even after revision the quality of presentation and writing is not up to the standard we expect from NeurIPS publications. 

Therefore after consulting with the Program Chairs, we have decided to reject the paper for now. However we emphasize that this decision is solely based on the clarity and presentation of the written paper; the dataset itself is of high quality and is exactly the sort of contribution that we intend to publish in this track. We therefore encourage the authors to make another round of revision and rewriting to the paper, then resubmit to the 2nd round of the Datasets and Benchmarks Track at the end of August.

The reviewers have already given great suggestions for improving the paper. The AC would like to echo the recommendation of making the paper more accessible to readers familiar with computer vision and machine learning, but not land cover or remote sensing. Some of Figures 4-12 might be slightly shrunk and combined in order to give more space for additional definitions, explanations, or diagrams that are currently relegated to the supplementary material or website.

We understand that this may not be the outcome that authors expected. However we truly believe that more clearly presenting and describing the dataset for the NeurIPS community will ultimately lead to more users, and lead to more impact overall in the long run.

Decision: Reject