Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces the first publicly available dataset to study the evolution of spatial apartheid, using a large number of high-resolution satellite images of 95 provinces in South Africa.  It includes pixel-wise labels for 4 classes of neighborhoods. 
The paper presents some baseline experiments with state-of-the-art models commonly used in land use segmentation tasks. 
strengths: + The proposed task is interesting for ML researchers and results can be of interest to researchers of other areas. From this point of view, the expected impact of this dataset goes beyond the strict field of ML. 
+ There are several land-use datasets that represent similar tasks, but this one corresponds to a poorly represented environment.
+ Ethical and social implications are clearly identified. 

weaknesses: + Access limitations, depending on the way they are applied, could severely limit the potential impact of the dataset. The access form is not available. It is necessary to define in clear terms which applications are legitimate and which are not. 
+ Modern land-use datasets are richer, in terms of features, than the one presented in this paper because of the limitation of satellite images (resolution and spectral properties). It is not clear if the provided data is sufficient to develop better ML models to increase classification performance. On the other side, to study the effects of spatial apartheid, there is also a lack of additional data. Because of this, the presented dataset needs more data to maximize its potential impact. 
correctness: + The dataset creation methodology is well described and there are no concerns related to this point. 
+ Evaluation methods are well selected. 

clarity: The paper is well written. 
relation to prior work: The most related class of datasets are land-use datasets, but some of the most popular datasets are not considered: 
+ U.S. Geological Survey. National Land Cover Database, New York.https://cugir.library. cornell.edu/catalog/cugir-009031, 2016.
+ Seyed  Majid  Azimi,  Corentin  Henry,  Lars  Sommer,  Arne  Schumann,  and  Eleonora  Vig. Skyscapes  fine-grained semantic understanding of aerial scenes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
+ Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize to any city? the Inria aerial image labeling benchmark. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, 2017.
+ Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow.  Spacenet:  A remote sensing dataset and challenge series.arXiv preprint arXiv:1807.01232, 2018.

additional feedback: + Complete related work with a broader survey of existing land-use datasets.
+ Consider the enrichment of the dataset with pixel level, open-source, geographical data.
+ Define in clear terms legitimate uses of this dataset.
review_title: Visual Dataset to Study the Effects of Spatial Apartheid in South Africa

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: To briefly summarise, the submission is a dataset "to study the evolution of spatial apartheid", in the form of "geo-referenced satellite images covering the entire country of South Africa and a corresponding mask of neighborhoods labelled according to their type". As this dataset covers the entirety of South Africa, the submission provides a wide coverage, while at the same time provides a high-enough resolution for the research community.

The contribution is strong, built using relatively open data sources, and has the potential to contribute to meaningful cross-disciplinary research, beyond ML --  from geographic studies, to social science, to economics and public policy, etc. -- due to its potential ease of use, and above all, thoroughness.

strengths: The strengths of this paper are 3-fold, and are highlighted here. (Please refer subsequent sections for further details).

* It is built using a collection of open data - per the `Datasheet` PDF: South African National Space Agency satellite imagery + Eskom  building count dataset + EA dataset (the latter is public South African census data).
* The preprocessing workflow is adaptable to continually updated satellite images -- per e.g. Section 5.2, Supplementary Figure 23 -- which will appeal to inter- and cross-disciplinary studies as highlighted in the Summary And Contributions above. To elaborate, as the data is provided in image format, it is easily processed by researchers in other disciplines merely using standard bitmap image processing techniques (as the authors have provided an example in Section 5.2 - the mere difference/delta in pixel intensity over time can detect changes in population density patterns).
* The workflow is describe in a clear, concise manner; and the authors have provided a thorough examination of use cases and (most importantly) ethical and societal issues in the supporting material.
weaknesses: There are several issues which may need clarification, mainly the process of assembling and preprocessing the dataset. 
These are intended to improve the quality of the dataset and the documentation, should the dataset be recommended for acceptance.

* Meanings of colour maps - In Section 3, in the course of documenting the data processing steps, it would be preferable if the use of colour was standardised throughout, and if possible, a colour-blind/colour vision deficiency-friendly palette was employed. 
For example, when following the processing pipeline in Section 3 onwards, Figure 2 (EA polygons) indicate *farms* in a shade of cyan/turquoise. In Figure 3(b)(top), *farms* are a light purple, while in Figure 3(b)(bottom)/3(c)(left), and `Datasheet` (point #4) *farms* are in a shade of bright green, and finally for Figure 3(c)(right), it goes back to a brighter/pastel shade of purple. This makes the section a bit hard to follow (unless the reader has, beforehand, studied the Supplementary Material to identify the trends of colour palettes for different stages).

* Combining the sources of mask and map data - In Section 3, satellite images from SPOT are restricted to the year 2011, to correspond with the year the Enumeration Areas (EAs) dataset was created in 2011. However, the "Geo-referenced buildings ... shapefile dataset was created by Eskom ... and consists of building count data in South Africa from 2006 to 2016". This is unclear. Further, the Wikipedia-based disambiguation process will need to be elaborated. (will be expanded in Correctness below). 
correctness: The overall construction of the dataset is performed in a sound way. 
* The Supplementary Material section has done a good job at explaining the entire pipeline, alongside a generous collection of images to illustrate the dataset in practical use. 
* Figure 9 was especially useful in visualising the train/test sample split for the neighbourhood classiﬁcation model.
* The authors have also taken great care in ensuring the geo data projections are standardised, and ensuring that the classes are balanced (Section 4).
Hence, overall, the processing pipeline for constructing, validating, and testing the dataset appears sound.  

Two issues in the 'source data' can be addressed further.
One of the key items that the reviewer would like the authors to address is the use of the "Geo-referenced buildings dataset" (hereinafter, the "shapefile" dataset) - as discussed in the main manuscript, Section 4, "consists of building count data in South Africa from 2006 to 2016". However, it is not clear that the dataset is 
(a) built in the ten year time period, whereby it runs the risk of having buildings > 2011 *not reflected* in the choice of EA + SPOT images; or
(b) similar in versioning as the SPOT dataset, i.e. one per year, and that the 2011 version was chosen to align with the latest possible EA year.
The `Datasheet` file, under Collection Process->point 18, mentions further details about the shapefile dataset including a link to a technical report. It would be optimal if the authors can cite this report in the main manuscript, and provide some extra details about the specifics of the shapefile construction (strengths/weaknesses/considerations), as Section 3 currently lacks details (when compared to, say, the EA dataset and the SPOT dataset).

Another issue is also in Section 3: "One shortcoming of the EA dataset is that townships are grouped with suburbs... which does not allow us to distinguish between them... each EA is associated with a name... looked up a list of South African townships on Wikipedia and labelled the EAs". The use of Wikipedia to label the EAs is one way to get past the limitation; however more information needs to be provided on this process - e.g. how many total EAs require Wikipedia disambiguation, with examples of the disambiguation process? Further for the recruitment of "10 volunteer students who grew up in townships to aid in labelling townships" (per `Datasheet`), some comments on the methodology will be preferable (e.g. each of the X ambiguous townships was labelled by Y students by consensus/majority vote)?
clarity: The paper is very well written. No major issues or typographical errors found.
Some ideas on improving the visualisation of the dataset (including intermediate steps) are discussed in the Correctness section above.

Kudos to the authors for a clear and thorough documentation in the Supplementary Material to anticipate questions that reviewers/dataset users might have.
relation to prior work: Yes, how this work differs from previous contributions is clearly discussed in Section 2.
additional feedback: I commend the authors for a strong submission to NeurIPS 2021 Track Datasets and Benchmarks; where the empirical data can greatly benefit researchers from various disciplines in studying contemporary social issues. 

However, there are some issues above which will need to be resolved or clarified before it is ready for acceptance: the issues raised above are mainly for completeness (in terms of methodology), especially in terms of how the dataset is compiled. 

I wish the authors all the best for NeurIPS 2021.
review_title: 20210916 Review of the "Spatial Apartheid" dataset (Sefala et al, 2021) 

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The dataset is designed for studying the evolution of spatial apartheid in South Africa. 
This dataset contains 6,768 remote sensing images, covering 9 provinces for 4 classes of neighborhoods.
Several analyses have been performed to understand the characteristics and changes in South Africa.
However, this dataset seems to be designed for human geography thematic tasks, which may not be suitable for NeurIPS.
Besides, the benchmarked experiments are relatively weak and can not inspire future model development in machine learning.
strengths: 1. It is the first publicly available dataset to study the evolution of spatial apartheid.
2. The publicity of the dataset is helpful to the field of human geography, which may promote the relevant research of studying the evolution of neighborhoods in South Africa.
weaknesses: 1. The contributions are not very clear and are supposed to be highlighted in the Abstract.
2. Some novel insights require to be added referring to the experiments.
3. More advanced remote sensing semantic segmentation methods need to be extended.
4. Article organization needs adjustment. The descriptions of data preparation can be reduced or moved to the Appendix.
More statistical feature analysis needs to be clarified to enhance the value of this dataset.
5. The data generation process includes buffer and topology analysis. These parameter-sensitive operations will bring some noise. How to control the accuracy and quality of each step?
6. Why are the evaluation indicators in Table 7 and Table 8 different？ The $mAP_{50}$ needs to be clarified because this is an object detection metric. Are there any instances in the labels? 

correctness: The process of the dataset construction seems sound to me.
clarity: Please see 5,6 in Weaknesses
relation to prior work: Yes. It is the first publicly available dataset to study the evolution of spatial apartheid.
additional feedback: Please see Weaknesses
review_title: It is a human geography thematic dataset and is not suitable for machine learning or computer vision.

Meta-review: The data provided in the paper is of relevance for the NeurIPS data track. Based on reviewers opinions, and in particular after discussion with the authors, the paper achieves the minimum score required for publication at NeurIPS data track.
Decision: Accept