Kati8/ICLR_2023_2025 huggingface dataset:
authors: Makkunda Sharma
authors: Fan Yang
authors: Duy-Nhat Vo
authors: Jack Gidney
authors: Matthew Sutcliffe
authors: Mengyan Zhang
authors: Esra Suel
authors: H Juliette T Unwin
authors: Swapnil Mishra
authors: Samir Bhatt
authors: Oliver Fiala
authors: William Rudgard
authors: Seth Flaxman
reviews: 
Comments: The paper proposes a dataset on child poverty featuring Sentinel-2 and Landsat imagery in Eastern and Southern Africa. Two dataset splits, a spatial and a temporal one, are proposed. The dataset is tested with several deep learning models such as MOSAIKS, Dinov2, and SatMAE, as well as two baseline approaches (mean regression and Gaussian process regression).

In general, the field benefits from challenging datasets. However, in this case, none of the factors constituting the target class, child poverty/prevalence of severe deprivation, are directly measurable from optical satellite imagery. Moreover, the dataset fails to incorporate key information such as nighttime light data that is typically used as a proxy for poverty. The paper also lacks visualizations that show examples of the satellite imagery, target value, and model outputs. Due to this and the fact that the spatial dataset split greatly simplifies the task, it is unclear whether models are capable of learning any information related to child poverty from this dataset.
Strengths: - The dataset focuses on an important task in the Global South, while most existing remote sensing datasets were developed for the Global North.

- The paper includes two simple baselines that help to understand the dataset.
Weaknesses: - The description of the satellite data is incomplete (I am aware that the satellite specifications are listed in the supplementary material but I am referring to the preprocessed satellite imagery used as model input). For example, key information such as the processing level, the selected spectral bands, and the spatial resolution of the satellite imagery are missing. This also applies to the abstract which does not provide any information on the satellite imagery.

- A literature review on relevant remote sensing studies mapping poverty is missing. Instead, the related work section cites several less relevant datasets such as xView2 or SpaceNet 7.

- Since nighttime light data is a key modality for poverty mapping, it is surprising that the authors did not include it. Therefore, I strongly suggest enriching the dataset with nighttime light data from, for example, SDGSAT-1 (ideal due to its high spatial resolution) or VIIRS.  

- The visualizations are not useful for understanding the spatial patterns the models produce since they cover extensive areas. Therefore, I request some examples of model outputs over metropolitan areas of cities known to have deprived neighborhoods  (e.g., Maputo, Mozambique). This will lead to great insight into the capability of models trained on KidSat. 
 
- For the spatial benchmark, I assume the clusters were randomly split into five folds. However, this split largely overlooks spatial autocorrelations and does not sufficiently test generalization ability across space (see Rolf et al., 2024). This limitation is also indicated by the performance of the Gaussian process regression baseline. Therefore, I recommend adding experiments for a region-based split.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper outlines a methodology to map poverty (i.e., severe deprivation) across southern Africa by encoding satellite images into feature representations with pre-trained models (Dino, SatMAE) , random kitchen sinks (MOSAIKS) and performing a linear regression on the target variable "severe deprivation". A baseline is the GP regression given the label data. 

While the problem is highly relevant and impacting, the implications for the Machine Learning community remain unclear. In particular, the discussion remains focused on poverty mapping and does not investigate an underlying methodological question of "which pre-training or feature representations are particularly useful for certain downstream tasks (like poverty mapping)". Currently, tested methods are used as is, and their feature vectors have not been investigated in depth. 

Hence, i believe that this paper would rather fit in an applied journal where reviewers can better evaluate the societal impact of poverty mapping with existing foundation models.
Strengths: * Impacting problem and extensive dataset.
Weaknesses: * methodological questions are not investigated in-depth. E.g., what impact does the pre-training algorithm have on downstream performance in general. Are (reconstruction-based) MAE features better than Dino (joint-embedding based) features (see for instance, Shekhar et al., 2023 for a comparison here) in general for poverty mapping? Do takeaways in that direction align with existing papers? 

Shekhar, S., Bordes, F., Vincent, P., & Morcos, A. (2023). Objectives matter: Understanding the impact of self-supervised objectives on vision transformer representations. arXiv preprint arXiv:2304.13089.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2