Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper releases a collection of benchmarks and datasets related to UN’s sustainable development goals (SGD). Monitoring SDGs is challenging since it requires ground based household surveys and domain knowledge. SUSTAINBENCH aims to define standard datasets and metrics to make prediction of SDG tasks using machine learning easier.

strengths: - UN’s sustainable development goals is a crucial area where machine learning community can help in monitoring if there are standardized benchmarks and SUSTAINBENCH tries to address this issue.
- Many different countries are covered in the dataset across different tasks and some of the data is gathered from different previous work. While this makes the data a bit scattered, it is a good start to collect them in one place for a common goal.
weaknesses: - For some tasks that uses DHS surveys like predicting Women BMI, women education and child mortality rate; it is not clear how using satellite images as in input can help predicting women education levels at all. Low r2 numbers on the baseline model shows this too (i.e. 0.01). And these are the only evaluation results that do not rely on previously published work.
- It’s not clear how some of the specific tasks defined helps with improving the SDG. For example, it’s not clear to me how does field delineation task helps with Zero Hunger SDG, it would have been good to provide more background and motivation on this for non domain experts. 
- While predicting poverty over space task includes both satellite and street level images, predicting poverty over time only contains satellite images. Since these two tasks are related, it would have been more complete if predicting poverty over time task included street level images over time as well. Figure 1 makes it look like it does include but in fact it does not.
- The leaderboard on the website does not contain any results/baselines. It would have been great if Table A7 included the model type used to evaluate as well.


correctness: The evaluation results in the paper are collected/gathered mostly from previously published peer-reviewed results. I added some comments about the ones that are not priorly published above.
clarity: Paper is clearly written and well presented. 
relation to prior work: Relation to prior work is discussed. SustainBench build on many prior work that focuses on one particular SDG related task, SustainBench aims to collect many tasks towards different SDGs and standardize towards a collective goal. It also adds new data which was not used in prior work before.
additional feedback: Overall I think it is a crucial area and I think the SustainBench could be valuable benchmark suite. Please address the questions in the weaknesses section.
review_title: Important area to address, could use more work

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors propose 15 benchmark tasks for inferring indicators of human/economic activity using (mostly) remote sensing data.  The benchmark tasks are quite unique in their explanatory (input) data and scoring metrics, however, they are assembled together in this benchmark because they are each useful for measuring progress towards one (or more) United Nations Sustainable Development Goal (SDG).  

I believe this work addresses an important open problem.  At present it is difficult for ML-focused researchers to understand and contribute to SDGs; and it is similarly difficult for non-ML researchers to navigate the scattered research involving the use of ML methods to monitor SDGs.  This benchmark and the attendant resources provided by the authors (e.g., large newly-released datasets, code for accessing these data) are a keystone resource to connect and coordinate the efforts of these communities. 

I would give a higher rating but the work seems somewhat incomplete.  The website seems incomplete, and I'm concerned by the limited inclusion/discussion of related models to solve some of the proposed benchmark tasks.  Similarly, I think further discussion of, and comparison with, related datasets for the proposed benchmark tasks is needed.   I explain these concerns further in the "Relation to Prior Work" section.   

This is an ambitious project and I appreciate the substantial work that has gone into this already.  I think it is a great challenge to prepare all of the necessary content to support this project.  However, I still think everything must be complete and thorough for publication.   With that in mind, my rating would be much higher if some of the aforementioned limitations were addressed.   

Also, this paper already contains a substantial amount of content, much of which is in a long supplement.  I think this project might be better-served by a longer-form journal publication, allowing for more explanation and detail in the main text.  



strengths: 1) Through this effort, the researchers are connecting a large community of non-ML researchers (e.g., in public policy, economics, and energy sector) with the ML research community.  The website can ultimately provide a central location for non-ML researchers to see the state/progress of ML-based tools for monitoring SDGs.   Historically efforts to develop ML-based tools for SDGs have been largely uncoordinated, and difficult to navigate by ML researchers and non-ML researchers alike.    
2) The authors release several new large datasets with labels. 
3) More importantly, the authors put all of these datasets in one place, and make them easily accessible via Python code.  I agree with authors that a major challenge for ML researchers is accessing and pre-processing these large datasets.  Having the datasets available via download packages is really helpful.

weaknesses: 1) The relationship to existing work is somewhat weak (see comments on "Relation To Prior Work") below.  

2) Related to (1), the number of benchmark methods included seems to be limited, despite the fact that there are (I believe, though could be incorrect) substantial existing work on some of these problems. 

3) The website appears to be somewhat incomplete. Some specific instances include the following: 
a) The "Get Started" page has just a single sentence on it (https://sustainlab-group.github.io/sustainbench/docs/get_started.html)
b) Some of the links to download the datasets do not work, or are empty.  I refer to some specific examples under my response for the "Correctness" enquiry. 
c) There are no benchmark results listed.  




  

correctness: Some of the links do download the datasets do not work or are empty, including those on the following pages of their website:
1) https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg15/out_of_domain_land_cover.html
2) https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg15/land_cover_representation.html
3) https://drive.google.com/drive/folders/1VvDQHTorD8sa6YJ6_Z9UoEFGu7QpR2dT  
4) https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg2/TBD
clarity: The paper is relatively well-written.
relation to prior work: The work seems to be incomplete when comparing to prior work.  For many of the included benchmark tasks, I believe (though I'm not sure in all cases) that there are some other existing resources (e.g., datasets) and existing methods (e.g., models).  When proposing a new benchmark, it is important that the authors explain how their proposed dataset and scoring methods compare to existing work, and how their particular dataset is advantageous or differs compared to existing resources.  Similarly, while the authors often provide some justification for their chosen baseline methods (e.g., it is simple, or it comes from an existing paper), they often don't seem to discuss other methods on the topic, or how their chosen baseline compares to them.  

For example, in D.3 could the authors clarify how their crop datasets compare to other crop datasets utilized in recent studies (e.g., in geographic coverage, number of classes, etc) and whether these prior works use public or private datasets?  Similarly, in E.2.2, E.2.3, and E.2.4 it would be helpful if the authors could explain how their proposed baseline methods for these problems compare to work by other authors on these topics?    I'm only somewhat familiar with this literature and so I could be mistaken, in which case the authors need only clarify that there are not existing methods/datasets beyond those they mention in their supplement.  




additional feedback: No additional feedback.
review_title: Important problem and a major contribution, but somewhat incomplete 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper provides a compilation of datasets for 15 benchmark tasks across 7 United Nation Sustainable Development Goals (SDGs). As the efforts towards the SDGs are highly interdisciplinary, they are less well accessible to the machine learning community. This paper is an effort to bridge the gap: it not only collects large-scale, cross-domain raw data for different SDG tasks, but also process them into a machine-learning-ready state, thereby creating a standardized resource that enables the machine learning researchers to contribute to these societal goals. 

Contributions:
- the largest compilation to date of datasets for SDG monitoring that offers standardized, machine-learning ready, data splits for benchmarking.
- well-documented baseline models/results, and a public leaderboard that encourage machine learning community to contribute to the SDG problems.
strengths: 1. The paper introduces the largest to date collection of datasets for monitoring the SDGs with machine learning, providing a standardized benchmark for various SDGs tasks, which is not previously achieved. 

2. The datasets are not only applicable to research on SDGs, but also a good resource to train/test machine learning algorithms on real-world data. Especially, the cross-task, multi-modal data is very valuable.

3. The authors have put a great effort minimizing the potential ethical issues regarding the dataset, have provided a detailed discussion on the limitation of the dataset, and have pointed out the potential usages and pitfalls of the dataset. All these are helpful guiding the community to use the dataset properly.
weaknesses: I find no obvious weakness in the paper. There's one minor concern: it would be beneficial to briefly discuss the baseline methods in addition to describing them in Appendix E (e.g., when applicable, how's their performance compared to those reported in the original papers? Does this tell the advantage / extra information of the new dataset compared to existing ones?)


correctness: Yes, the dataset is in general constructed in a sound way.
clarity: The paper is clearly-structured and well written. 
relation to prior work: Yes, the paper clearly analyzes the limitations of prior work and states the strengths of this work against existing ones.
additional feedback: Personally moving the table A7 (summary of SOTA methods on the benchmark tasks) to the main text would make the main paper more self-contained; though it is also understandable that describing such a large-scale dataset within the page limit is challenging.
review_title: A comprehensive compilation of datasets that boosts ML research towards Sustainable Development Goals

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors are releasing ~10 novel datasets and improving upon ~5 already available ones. I think they have dev/train/test splits, which is great. That's a lot of datasets, especially since they tend to cover countries that are poorer than e.g., US and Western Europe. It  should be very interesting for the community. The value and impact of the datasets is unknown (as far as the reader can tell). They are also heavily oriented around remote sensing.
strengths: Lots of datasets, the work looks well done.  Impact demonstrated with some of the follow on work, this looks pretty nice.
weaknesses: I think the biggest weakness is that I don't know a ton about  remote sensing and don't have an intuitive idea of how great each dataset is.
correctness: There are 15 datasets here. I'm going to assume yes, but frankly, I'm not a good person to assess this.
clarity: Generally yes.
relation to prior work: Yes
additional feedback: Thank you for incorporating feedback, I feel like this was very helpful and the result  is much clearer to me as a reader.  I'm changing my score to a 7.
review_title: A lot of datasets for one paper, looks interesting, needs better presentation, what's the impact?

Meta-review: Dear authors,

Thank you for submitting your paper and addressing the issues raised by the reviewers.

The paper proposes a set of 15 problems for monitoring Sustainable Development Goals (SDGs) with machine learning. They provide a python module with dataloaders similar to WILDS and a website with leaderboards for each dataset.

The main criticism was

Missing comparison with existing work Incompleteness of the website Some tasks were not described clearly

All three points were addressed by the authors in the rebuttal. Furthermore, this work allows the ML community easier access to data related to the highly important sustainable development goals.

Taking all this into account as well as the scores of 6,5,7,7 the paper is accepted.
Decision: Accept