Subnational variations in the quality of household survey data in sub-Saharan Africa

Domain: earth_science
Published: 2025-04-22
Editorial outcome: major_revision

Abstract
Nationally representative household surveys collect geocoded data that are vital to tackling health and other development challenges in sub-Saharan Africa. Scholars and practitioners generally assume uniform data quality but subnational variation of errors in household data has never been investigated at high spatial resolution. Here, we explore within-country variation in the quality of most recent household surveys for 35 African countries at 5×5 km resolution and district levels. Findings show a striking heterogeneity in the subnational distribution of sampling and measurement errors. Data quality degrades with greater distance from settlements, and missing data as well as imprecision of estimates add to quality problems that can result in vulnerable remote populations receiving less than optimal services and needed resources. Our easy-to-access geospatial estimates of survey data quality highlight the need to invest in better targeting of household surveys in remote areas.

Introduction
The absence of dependable data on age, fertility, mortality, health, wealth, education, and nutrition in many low-income countries has been a long-standing concern for scientists and practitioners of human development1,2. Lacking reliable and representative census data, public health information, and/or birth registration systems, most African countries rely on multi-purpose household surveys for information on their citizen’s well-being3–5. The most widely used sources of nationally-representative data by practitioners and the research community are the Demographic and Health Surveys (DHS), Multiple Indicator Cluster Surveys (MICS), and Living Standards Measurement Surveys (LSMS). DHS data, the main example used in this study, have been used in about 6000 publications since 2010 (a conservative estimate)6.

Today, independent researchers and the DHS program routinely use geocoded household survey data to produce high-resolution modelled surfaces of health and development indicators7–14 often in conjunction with spatial data from satellite images and other secondary data sources, such as data on armed conflict15–18 or disease distribution data19. Other household survey programs are now following their lead, e.g., MICS20. Since policy decisions can take place at national or subnational levels, and the assessment of impacts is usually pursued at district level or lower, there is an increasing demand for small area estimates21. For example, the World Health Organization (WHO) frequently uses estimates from the Malaria Atlas Project (MAP)19,22, which constructs modelled surfaces of malaria prevalence from geocoded household survey data23. Moreover, GAVI, the Vaccine Alliance, and the Nigerian government are making use of such high-resolution modelled estimates to guide strategies around reaching zero-dose children24,25, while the Children’s Investment Fund Foundation (CIFF) are utilizing these approaches for monitoring, evaluation and learning applications26.

The spatial dimension of policymaking critically depends on the quality of underlying sub-national and regional data. However, the quality of household survey data across sub-national regions cannot be assumed to be uniform. Nutrition and public health professionals as well as government ministers have called for fixing data gaps and improving data quality7,27. Illness and malnutrition are defined as quantified conditions lying outside ‘healthy’ thresholds. Thus even modest inaccuracies, such as errors and artifacts in the measurement of age can give a false picture of, for example, child stunting (low height for age)28–31. Robust subnational information, on the other hand, enables health authorities to target and better fund health interventions32.

We define high quality data as having low statistical uncertainty and little systematic measurement bias and systematically missing data. Statistical uncertainty is mainly determined by sampling strategy (determining for example sample sizes) and can occur independently of systematic measurement error and missing data. While high statistical uncertainty typically implies decreasing predictive power by widening prediction intervals, taking into account statistical uncertainty is readily achieved in standard statistical frameworks for empirical analysis, for example using stochastic error terms. Systematic measurement error and missingness are more serious threats to the derivation of empirical conclusions. These data errors cannot be accounted for because their magnitude and spatial distribution are very often not known12,26,33–35. The DHS program and independent researchers have examined measurement errors and missing data in household survey data on aggregate levels only-across countries and across survey teams within countries28,36–41. This is unsatisfying as no quantification of spatial variation in household survey data quality exists for a large number of countries at highly granular levels such as the village level. As a result, even those practitioners and researchers who work extensively with national household surveys and who are aware of the potential for spatial variation in data quality, may not know the precise spatial distribution and the magnitude of measurement errors and missing data.

These data errors can occur despite rigorous efforts and enumerator training42–44. Errors may be introduced at any stage of the household survey beginning with survey design45, during data collection or data processing38. Existing studies point to a number of variables such as the age of respondents46, low numeracy and literacy or lack of birth registration30 that may be less reliable in remote areas relative to urban areas, but we lack spatial estimates of data errors for specific locations, which in turn adds to the already greater vulnerability of the so-called last-mile populations in low-income countries (defined by The World Bank as having less than $1145 gross national income per capita in 2023)37,47. Beyond the fields of health and demography, which are the focus of this study, development and agricultural economists have been working towards high-quality economic and health data from low- and middle-income countries (defined by The World Bank as having less than $4515 in gross national income per capita in 2023), and have improved data quality through several strategies, including by incorporating spatial and community-level data, using GPS technology, or engaging with local communities (Supplementary Note 1)48–51. Similarly, the DHS program and international donors made significant efforts to ensure high data quality by identifying potential sources of data errors36,42,52–55 and by making continuous improvements41,56–59.

The precise spatial distribution of measurement errors and missing data has not been quantified before at subnational level for a large group of countries41,43,59–62. Not knowing the spatial variation in these data errors is unsatisfactory, because decisions are increasingly made locally, for example at the level of districts, and because so many decisions are made on the assumption that local data are reliable. In addition, researchers and other data users who do not frequently working with household data may not be aware of, or simply ignore, the potential for spatial variation in the quality of data with which they are working63. Even researchers and practitioners who work extensively with DHS data, and who are aware of the potential for data errors, can improve causal inference with knowledge of the extent of variation in data quality in a particular region. Finally, the DHS program has highlighted the need for greater attention to spatial variation in data quality. Their goal is to improve the performance of survey teams working in locations prone to high data errors by providing them with, for example, better training or more resources37.

In this paper, we provide a high-resolution geospatial analysis of DHS data quality taken from surveys of central, eastern, western, and southern Africa between 2006 and 2022. Among large household surveys in Africa, the DHS program offered the most complete data covering 35 of Africa’s 46 countries. The number of DHS survey rounds varies across nations. In countries with more than one survey round since 2006, the most recent survey was used (Supplementary Table 9). From this set of household surveys, we derived three data quality indicators which are widely used by practitioners and researchers (Supplementary Note 2). ‘Incomplete age’ refers to the share of interviewed women (15–49 years) with either the year or the month of birth missing relative to all interviewed women. ‘Age heaping’ refers to the proportion of reported ages ending in 5 or 0 of all adults between 23 and 62. This is based on Whipple’s index definition, a value above 20% indicates age heaping. Finally, ‘flagged HAZ’ refers to missing or biologically implausible values for the measured heights of children (height-for-age z-scores) according to WHO (World Health Organization) standards.

In this work, we map predictions of data errors beyond data points in the analyzed DHS datasets. To do this, we employ Bayesian model-based geostatistics that combine spatially explicit data and covariates from gridded high-resolution datasets to produce 5×5 km gridded estimates of the three data quality indicators - a level of detail that is roughly equivalent to the size of villages in rural areas. We aggregate our estimates to districts and to national levels using population weights. We provide an online data visualization tool to provide easy access to our estimates for data users (https://apps.worldpop.org/SSA/data_quality/). Next, we combine original DHS data with freely available data on settlements and nighttime light emissions to explore non-random distribution patterns of data errors and chart the deterioration of DHS data quality with increasing distance to settlements. Finally, we contrast the spatial distribution of our predictions of data errors with the statistical uncertainties associated with related public health indicators in the DHS data, which may collectively threaten the quality of insights and decisions that can be drawn from DHS data.

Results
Subnational variability in data quality
Predictions of data quality varied substantially across 5×5 km cell levels and across district levels throughout the 35 included African countries. Within-country variation was apparent with all three data quality indicators for all 35 included countries (Fig. 1) and it was higher than what would be the product of chance. Our data quality predictions were obtained from a geostatistical model trained on age heaping, incomplete age and flagged HAZ derived from DHS data (see Methods for details). We provide a description of these three data quality indicators in the Supplementary Note 2. 5×5 km cell levels of geographical resolution are commonly used in studies using DHS data and by the DHS program8,10,11,13,14,64.

The Moran’s I statistic, a measure of spatial autocorrelation that we computed using raw DHS data at the district level, was recognizable and varied in value across countries and across our three indicators. Spatial autocorrelation at the district level was more identifiable for incomplete information on women’s age while the subnational variation in age heaping and flagged HAZ values appeared closer to random (Supplementary Tables 10–12).

Within-country variation in predictions of data quality was of larger magnitude in countries with lower average data quality (Supplementary Fig. 2). For example, our estimates of age heaping in Nigeria (national mean 39.8%) ranged from 62.1% (standard deviation 2.2%) in Danmusa district in Katsina state in northwest Nigeria to 25.4% (sd 5.6%) in Agege in Lagos State, where the terminal digit preference for 0 or 5 was just 5.4 percentage points above the expected natural occurrence of 20%. Estimates of the share of incomplete information on women’s age in Chad (national mean 67.3%) ranged from 91.6% (sd 2.7%) in Loug Chari in the south of Chad to only 8.1% (sd 2.4%) in Dar Tama, on the eastern border with Sudan. Flagged HAZ values in Madagascar (national mean 7.58%) ranged from 14.8% (sd 1.9%) in Bongolava to 4.5% (sd 0.95%) in Haute Matsiatra, two regions in the center of the island. These estimates demonstrate that analyzing systematic measurement errors at the country-level alone masks important local and regional patterns

There was little to moderately positive correlation between the three data quality indicators. The correlation coefficients between pairs of indicators ranged from 0.35 between age heaping and incomplete age to 0.71 between the two age-related indicators, age heaping and flagged HAZ (Supplementary Fig. 3). Data quality challenges are not uniform across the indicators we employed, and different correlation coefficients reflected different underlying potentials for systematic measurement error and missing data.

The only moderate and low correlation between the three data quality indicators reflected a distinctive subnational spatial pattern for each indicator. This suggests varying degrees of difficulty encountered in collecting the different types of measurements across localities and across DHS data collection teams37. For example, incomplete age records reported from Wajir West (53.3%, sd 5.9%) in northern Kenya were considerably worse than the national mean of 13.4% while in the same district flagged HAZ values (4.0%, sd 0.94%) were closer to the Kenyan national mean of 2.2%.

Deteriorating data quality in remote areas
To explore the potential of a collection bias in the spatial distribution of measurement and missing data, we combined the prevalence of errors by survey location in the raw DHS household data with distance to the nearest settlements. We conducted the analysis for each of the three data quality indicators to determine the extent to which they were subject to a remoteness penalty, that is, an increase in the prevalence of measurement errors, artifacts, and missingness with distance to settlements. Remoteness captures a mix of potential factors such as respondent literacy and numeracy, or more challenging conditions for survey collection teams. We did not intend to study these underlying causes. Our aim was to explore the magnitude of this problem and whether it affected the three data quality indicators to a similar extent. In terms of further empirical analysis, the data challenges systematically increasing with remoteness could threaten the insights drawn from the data in two ways. In the best case, measurement errors and missingness are random, leading to more noisy estimates; in the worst case, errors and missingness are systematic, leading to systematic bias in parameter estimates. Data users will need to assess on a case-by-case basis which of these two threats is more relevant to their study based on geographic location, point in time, and variables of interest.

The predictions underlying Fig. 2 were based on nighttime light (NTL) emissions of a specific luminosity (Digital Number, DN 15), which lies above the threshold of the luminosity of public street lights indicating a settlement with sufficient infrastructure for survey teams to restock equipment and rest65–67. However, the differences in data quality were similarly apparent with lower and brighter light emissions threshold levels (Supplementary Figs. 5, 6). The deterioration in data quality was also discernible with distance to settlements using two alternative high-resolution settlement datasets built from daylight and radar satellite imagery and electoral records. These alternative datasets includes smaller and unlit settlements with shorter geographical distances between them, which made this approach less informative relative to using NTL emissions but displayed similar trends across all three data quality indicators (Supplementary Figs. 7, 8)68. Finally, we found little evidence that the observed bias was based on one-time effects for the majority of the included countries with multiple survey rounds (Supplementary Fig. 9–11). Data from countries with more than one survey round since 2006 indicate that the spatial bias is a long-term problem for the large majority of the 24 countries with multiple survey rounds.

In the 35 included African countries, increasing distance from electrified settlements emitting NTL of DN 15 was associated with decreasing data quality across all regions. The differences in quality were relatively worse across West Africa, and relatively better in Central and Southern Africa (Supplementary Fig. 4). The bias affected all three data quality indicators. The bias was relatively stronger with age heaping (Fig. 2a) and incomplete age records of women (Fig. 2b). It is relatively less apparent for flagged HAZ values (Fig. 2c) which was consistent with cross-sectional studies specifically examining stunting47. For age heaping and incomplete age of women in particular, this association grew stronger within countries of medium to high overall levels of measurement errors (Supplementary Figs. 12–14). For example, the estimated share of mothers (15–49 years) with either the year or month of their infant’s birth missing (‘incomplete age’) in Togo (national mean 22.3%) was 20.4% at 50 km distance, increasing to 24.8% at 100 km and was 35.3% at 200 km. The estimated proportion of adult ages ending in 5 or 0 (‘age heaping’) increased within Kenya (national mean 26.7%) from 26.2% at 50 km from the nearest town or nighttime light emitting area to 26.7% at 100 km and 27.9% at 200 km distance. The differences in data quality were smallest within countries that had overall higher quality DHS data. For example, age heaping within South Africa (national mean 20.2%) was at 20.0% at 50 km distance to the nearest point of nighttime light of DN 15 value. It was 20.3% at 100 km distance and 21.0% at 200 km.

Data errors and sampling uncertainty
Figure 3 shows the spatial distribution of two of our estimates for data quality: ‘incomplete age’ and ‘flagged HAZ’ (in shades of blue), and it pairs them with the standard deviations of predicted estimates of related public health indicators: ‘contraceptive use’ and ‘stunted children’ (in shades of green). These indicators are related because contraceptive use is an important health indicator for women, as it relates to their overall health and ability to achieve their desired fertility. In addition, maternal health status is a major determinant of stunting. Standard deviations indicate statistical uncertainty, for example regarding the precision of estimates for the prevalence of contraceptive use among sexually active women (‘contraceptive use’) or stunting prevalence among children (‘stunted children’). High standard deviations are mainly the result of small sample sizes at the survey cluster level or from a distribution of survey locations that does not take into account geostatistical design considerations12. High uncertainties may even be consistent with the DHS sampling strategy, for example, to make economical use of resources. In terms of data quality, high uncertainties arising from small samples can for example reduce the predictive power of fitted models, but they are generally considered a lesser threat compared to systematic measurement bias and systematically missing data because they are typically directly accounted for in statistical modelling frameworks12,69–71. With only information on statistical uncertainty-i.e., without spatial quantifications of data errors such as those provided by our study-users of DHS data may be inclined to limit their assessment of data quality to easy-to-measure statistical imprecision and neglect the threat of measurement errors because they lack subnational information on the extent to which these two types of data problems overlap in the geographic area of their interest. By pairing for example, the sampling uncertainty of ‘stunted children’ with our estimates of missing or biologically implausible data (‘flagged HAZ’), we inform DHS data users in which localities these two types of data problems overlap. Our findings can also help inform the DHS program where resources could be allocated to maximize data quality. This includes for example training enumerators, improving tools and procedures, or considering a larger sample size or new sampling strategy.

We estimated correlation coefficients at the district level (Supplementary Fig. 15). Missing data for women’s age was moderately negatively correlated with the standard deviation of the estimated share of contraception users (ρ=−0.402, p<0.0001). Flagged HAZ values were slightly negatively correlated with the standard deviation of the estimated prevalence of stunting (ρ=−0.109, p<0.0001).

Some regions were challenged with either high standard deviations of both ‘contraceptive use’ and ‘stunting’ (for example in Namibia, Nigeria, parts of the Democratic Republic of Congo, Malawi, parts of South Africa and parts of Tanzania) or with high estimates of both ‘incomplete age’ and ‘flagged HAZ’ values (for example in Angola, parts of Democratic Republic of the Congo, parts of Ethiopia, parts of Kenya, and Chad). On the country level, measurement errors paired with large sampling uncertainty were widely present only in Madagascar (national mean ‘incomplete age’ 16.3%, national mean ‘flagged HAZ’ 7.6%) and in Niger (national mean ‘incomplete age’ 84.6%, national mean ‘flagged HAZ’ 13.9%). On the local level, single districts were challenged by both higher error estimates and large standard deviations of estimates. N’gauma district in north-western Mozambique reported estimates for ‘incomplete age’ of 21.5% and a standard deviation of ‘contraceptive use’ at 2.7% (mean 14.6%). In the same district, the estimate of ‘flagged HAZ’ was 6.6% and the standard deviation of ‘stunted children’ was 3.5% (mean 45.6%). The Mozambiquan national means of ‘incomplete age’ and ‘flagged HAZ’ were 6.7% and 5.8%, respectively. No single country exhibited low rates of errors and high statistical certainty nationwide. However, districts with high data quality appeared in clusters across the subcontinent. One example was several districts in Uíge Province in Northwest Angola (national mean of ‘incomplete age’ 4.4%), including Songo (‘incomplete age’ 0.5%, standard deviation of ‘contraceptive use’ 1.63), Buengas (‘incomplete age’ 0.6%, standard deviation of ‘contraceptive use’ 1.3), and Damba (‘incomplete age’ 0.6%, standard deviation of ‘contraceptive use’ 1.4). Similarly, near error-free HAZ values and high statistical certainty concentrated in districts surrounding Thiès such as Mbour (‘flagged HAZ’ 2.8%, standard deviation of ‘stunted children’ 1.02% (mean 15.3%) in western Senegal (national mean of ‘flagged HAZ’ 3.4%).

Methods
DHS data
We compiled a database of publicly available geocoded DHS data for 35 African countries. DHS data are best suited for the scope of the study, because they are publicly available, geocoded, and more comprehensive in time and space than data from other large survey programs such as MICS or LSMS6,80,81. To demonstrate the deteriorating data quality in remote areas we directly calculated the prevalence of data errors in the most recent DHS survey round to avoid reporting a purely artificial, mechanical relationship arising from regressing predictions generated by a geostatistical model, which are a function of distance to night light, on distance to night light. To illustrate within-country variation in data quality as well as the relationship between data errors and sampling uncertainty, we moved beyond the DHS survey locations and employed Bayesian model-based geostatistics, using covariates from gridded high-resolution datasets to predict data quality beyond surveyed areas. The geostatistical model (see below) produced 5×5 km gridded estimates (roughly equivalent to villages in rural areas) of three widely used data quality indicators, which we aggregated to district (‘admin-2’) levels and national levels using population weights. The number of DHS survey rounds varied across African countries. In 24 countries with more than one survey round since 2006, we used the most recent survey (Supplementary Table 10).

We extracted relevant variables from the DHS children file, from the individual women’s file and the household roster file. In DHS data, the individual and household survey questionnaires are linked to a survey location (termed ‘cluster’ or ‘primary sampling unit’) that represents a set of neighboring households or a village structure. The data we derived from the most recent surveys came from 20,077 DHS clusters. For most clusters, geolocation information in the form of GPS coordinates were available. For the data from the Republic of the Congo (equivalent to 1.6% of the sample size), no geoinformation were available. To impute likely locations for these primary sampling units, we sampled 1000 candidate locations for each ‘latent’ survey location, based on population density rasters for each sub-national unit. The resulting candidate positions were then clustered using a k-means procedure and the centroids of the resulting clusters were used as pseudo-location. For each of these pseudo-locations, the sub-national average value of the indicator of interest was assigned. This procedure closely followed the imputation strategy that has been used for similar problems in recent literature10.

Covariate data
The modeling framework includes the following geospatial covariates which we constructed from freely available gridded datasets: population density (1), malaria incidence (2), terrain ruggedness (3), night light intensity (NTL) and settlement data (4). We believe a larger set of covariates would add no substantial explanatory power or change the argument of the study. For all covariates, data from the closest grid cell and matching the respective survey year were used for estimation. Where these were unavailable, data from the year closest to the respective survey year were included in the modeling framework. For prediction, covariates closest to the year 2013 were used. Prior to introducing the variables in our statistical modeling framework, standard temporal and spatial pre-processing steps have been conducted (Supplementary Tables 1 and 2).

Spatial processing. Both malaria incidence and population density are available at a resolution finer than the resolution of the night light data. To align the resolution covariate rasters, the input covariates ‘ruggedness’, ‘malaria incidence’ and ‘population density’ were resampled from their native resolution to match the native resolution of the night light intensity rasters (roughly 5×5 km at the equator line). Bilinear interpolation was used as resampling technique in all cases.

Administrative boundaries: We used GADM shapefiles in version 4.1 to define the national and sub-national boundaries used for spatial analysis82. In general, we used the smallest subdivisions available, which were equivalent to admin-2 for most countries and to admin-1 for Lesotho.

Gridded population density: We obtained the gridded data on population density from the Gridded Population of the World (GPW) database, version 483. The data are available in steps of five years. In general, we selected the year closest to the relevant year (e.g., survey year or year of predictions) for the analysis.

Malaria Incidence: We used a standard dataset on malaria incidence, obtained from the Malaria Atlas Project (MAP)19. Specifically, we used data on the incidence of Plasmodium falciparum. We included the closest available year to the respective survey round in the analysis. The year 2013 was used for predictions.

Ruggedness: Terrain ruggedness (3) was extracted from a dataset assembled and collected by Nathan Nunn and Diego Puga in 2012. The raw data are available from the personal website of Diego Puga84. The data come in form of a 30 arc-second grid covering the globe. We assumed that terrain ruggedness is time-invariant.

Nighttime light emissions: Satellite images recording nighttime light (NTL) emissions are frequently used to estimate economic activity11. We used the temporally NTL dataset generated by Li, Zhou, Zhao & Zhao (2020) which harmonizes inter-calibrated NTL data from the Defense Meteorological Satellite Program Defense Operational Line Scanner (DMSP-OLS) and simulated DMSP-OLS like NTL observations from the newer Visible Infrared Imaging Radiometer (VIIRS) data85. DMSP-OLS data are composite images in which each 30 arc second pixel (about 1 km at the equator) gives the annual average brightness level in units of digital numbers (DN) ranging from 0 (no light) to 63 (maximum reported light). For model estimation, we used the corrected DMSP NTL data covering observations from 1992 to 2013 across different satellites (F10, F12, F14, F15, F16, and F18), falling in the closest year that was available to the respective year closest to each survey year. For prediction, we used the DMSP NTL data for the year 2013. The improved data provided by Li and co-authors are temporally consistent correcting for varied atmospheric conditions, satellite shift or sensor degradation. In the analysis our ‘distance to nightlight’ variable measures the distance from the nearest NTL emissions of DN 15. We built binomial regression models with logistic link, regressing the respective data quality indicators computed from raw DHS data on distance to nightlight and fixed effects. We used raw DHS data to avoid reporting a purely artificial, mechanical relationship arising from regressing the predictions of data quality generated by a geostatistical model. Across 35 African countries, the average distance to the nearest settlement emitting NTL of DN 15 or higher was 64.8 km and 90% of clusters were located within 163.1 km from a settlement of DN 15 or higher. Thresholds delineating rural from urban areas vary across space and time66. Large rural settlements may remain dark on NTL maps for lack of access to electricity86. NTL emissions of DN 15, our main threshold, lie well above the lowest identified NTL thresholds of about DN 5 emitted from villages in Senegal and Mali with more than 40 streetlights67. DN 15 is below DN 20 assumed for urban, built environments such as in smaller cities66. In line with this literature, we concluded that in most of sub-Saharan Africa thresholds above DN 15 very likely include settlements with public streetlights and electrified buildings indicating a minimum degree of urban infrastructure and economic activity65.

Settlement data: We replaced NTL emissions with other measures of human settlements to see if our findings remained robust. The bias in the distribution of the errors remained apparent with auxiliary a) high-resolution settlement data and b) data on urban African agglomerations. Satellite-based high-resolution settlement data were taken from Marconcini et al.68. These data do not distinguish in the geographic extension of settlements and include tiny hamlets and suburbs. The average distance to the nearest settlement was 2.5 km. 90% of clusters were located within 5.1 km from a settlement. The deterioration of data quality was apparent in most sub-Saharan African countries even for these short distances. In contrast, Africapolis data collect larger urban agglomerations of at least 10,000 inhabitants87. The average distance to the nearest urban agglomeration of at least 10,000 inhabitants was 61.6 km. 90% of clusters were located within a distance of 167.1 km. Data from the Africapolis dataset are partly constructed from census data or population records which may be outdated or of varying reliability.

Covariate analysis: All covariates were examined for multicollinearity in a non-spatial modelling framework for each indicator. We calculated the variance inflation factor (VIF) for each covariate in each case and concluded that there was no evidence of multicollinearity when the VIF value is < 4.0. This was done for each modelling region, namely Central Africa (CA), East Africa (EA), Southern Africa (SA) and West Africa (WA). Where covariate information was missing at the cluster and grid level, we imputed using kriging interpolation based on covariate information from nearby locations. This was done for Lesotho, for which malaria prevalence data are not available. Both cluster and grid level data for this country were imputed using cluster level data from neighboring countries.

Geostatistical model
Each of our three data quality indicators was modelled in each of the four geographic regions, and the outputs of the regional models were combined to form a continuous map for each indicator. We describe each indicator in detail in the Supplementary Note 2. Countries were assigned to one of the four regions (Supplementary Fig. 16). This sped up computation and leverages regional relationships between the indicators and the geospatial covariates. Our INLA-SPDE approach, which we explain in the exploratory data analysis, requires the specification of a fine triangulation mesh to approximate the spatial random effect. An example of a spherical mesh for West Africa is shown in Supplementary Fig. 1.

Exploratory data analysis
To examine the relationships between the covariates and each of the indicators, we used the empirical logit transformation of the indicators given by Eq. (1):

y*(s) = log (y(s_i) + 0.5) / (m(s_i) - y(s_i) + 0.5), i=1, ..., n, (1)

where y(s_i) denotes the number of individuals possessing the attribute being modelled at spatial locations (represented using the longitude and latitude coordinates) and m(s_i) is the number of individuals sampled from that location. For example, for ‘incomplete age’, y(s_i) is the number of individuals with missing age out of m(s_i) individuals sampled from locations i. To improve the linear associations between the covariates and the logit-transformed indicators as is assumed in a binomial regression context, we log-transform the covariates, considering that these have heavily skewed distributions. We fit non-spatial binomial regression models and compute the Variance Inflation Factor (VIF) to check for (multi)collinearity88. A VIF value greater than 4.0 is indicative of the presence of (multi)collinearity among the covariates. Also, using the residuals from the non-spatial models, we calculate the empirical variograms of the indicators to check for the presence of residual spatial correlation. Assuming the presence of considerable spatial correlation in the residuals, we specified a spatial model for the analysis, described in the next section. All the covariates were standardized to have a mean of zero and variance of one prior to model-fitting.

Geostatistical model, model-fitting and prediction
To predict the data quality indicators at 5×5 km resolution and the district level, we fitted a Bayesian geostatistical model with a binomial likelihood. Using similar notation as before, the first level of the model can be expressed as Eq. (2):

Y(s_i) ~ Binomial(m(s_i), p(s_i)) (2)

where p(s_i) is the underlying true proportion of individuals possessing the attribute being modelled, e.g., incomplete age, at locations. We model p(s_i) using the logistic regression model expressed as Eq. (3):

logit(p(s_i)) = x(s_i)^T * beta + omega(s_i) + epsilon(s_i) (3)

where x(s_i) is a vector of covariate data associated with locations, beta are the corresponding regression coefficients, epsilon(s_i) is an independent and identically distributed (iid) Gaussian random effect with variance sigma_epsilon^2 used to model non-spatial residual variation, and omega(s_i) is a Gaussian spatial random effect used to capture residual spatial correlation in the model; i.e. omega = (omega(s_1), ..., omega(s_n))^T ~ N(0, Sigma_omega). Sigma_omega is assumed to follow the Matérn covariance function24 given by

omega(s_i, s_j) = (sigma^2 / (2^(nu-1) * Gamma(nu))) * (kappa * ||s_i - s_j||)^nu * K_nu(kappa * ||s_i - s_j||) (4)

where ||.|| denotes the distance between cluster locations s_i and s_j, sigma^2 is the marginal variance of the spatial process, kappa is a scaling parameter related to the range r = sqrt(8 * nu) / kappa – the distance at which spatial correlation is close to 0.1, and K_nu is the modified Bessel function of the second kind and order nu > 0. Further, for identifiability reasons, we set nu = 1. We fit model (1) for each region (Supplementary Fig. 16). A Bayesian approach is adopted for the analysis, which is implemented using the integrated nested Laplace approximation – stochastic partial differential equation (INLA-SPDE) approach89,90. To complete the model specification, we place a non-informative N(0, 10^3 * I) prior on the regression coefficients, beta. A penalized complexity (PC) prior introduced in Simpson et al. 2017 is set on sigma_epsilon such that p(sigma_epsilon > 3) = 0.0191,92. Similarly, following Fuglstad et al. 2018, a joint PC prior is placed on the covariance parameters of the spatial random effect, omega93. These are: p(r < r_0) = 0.01 and p(sigma > 3) = 0.01, with r_0 chosen to be 5% of the size of the region. The SPDE approach involves a triangulation of the spatial domain to approximate omega. A spherical mesh was constructed for this approximation using the boundary points and the data locations falling within each region. The maximum triangle edge length is set to 15 km in the inner mesh and 100 km (50 km in Southern Africa) in the outer mesh, with an offset of 500 km in each case. With the spherical mesh, the distance matrix needed to fit the Matérn covariance function is calculated as the great circle distance along the surface of the Earth. To implement this in practice, the data locations and boundary points for each region are first converted to three-dimensional coordinates on the unit sphere. These are then used to construct a mesh on a spherical manifold, S^2, on which the SPDE is defined. From each of the fitted models, we generate 1000 samples from the posterior distributions of the parameters of the model, as well as from the posterior predictive distributions of the indicators for each of the prediction locations, i.e., the 5 × 5-km grid cells. The latter are then used to calculate the district level estimates as population-weighted averages taken over all the grid cells falling within each district.

Model validation
To assess the performance of the fitted models for out-of-sample prediction, we adopted k-fold cross-validation, setting k=10. The cross-validation folds were created as random splits of all the data locations falling within each region. The following model evaluation metrics were computed using the observed (p(s_i)) and predicted values (p^(s_i)) of the indicators (on the probability scale) for i=1, ..., m validation locations and averaged over the 10 subsets:

Percentage bias (%Bias) = 100 * (sum(p^(s_i) - p(s_i)) / sum(p(s_i))) (5)

root mean square error (RMSE = sqrt(sum((p^(s_i) - p(s_i))^2 / m))) (6)

mean absolute error (MAE = sum(|p^(s_i) - p(s_i)| / m)) (7)

and the Pearson correlation coefficient. The closer the values of %Bias, RMSE and MAE are to zero, the better the predictions. Correlation values close to one indicate better predictive ability.

We found no substantial non-linear associations between the covariates and the indicators in each region (Supplementary Fig. 17 for ‘age heaping”). There was also no evidence of (multi)collinearity in the data as the VIF values of the covariates are estimated below 4.0 in each case. For all the indicators, we found evidence of significant residual spatial correlation in the non-spatial regression models fitted using all four covariates (Supplementary Fig. 18), justifying the use of geostatistical models for prediction.

Predictive performance: Our primary goal was to provide predictions of data quality indicators and associated uncertainties across sub-Sahara Africa at a high resolution. This means that we focused on the predictive performance of the fitted models rather than causal interpretability. Based on the correlations between observed and predicted values (Supplementary Table 3) the best predictive performance was observed in Central Africa, where the estimated correlations were greater than 0.58 for all the indicators (except ‘flagged HAZ’) and the worst predictive performance was observed in Southern Africa (SA). Other validation metrics indicate that the fitted models produced reasonably accurate predictions. The RMSE values were <= 0.18 in most cases while the MAE were <= 0.15 in most cases. The maximum RMSE was 0.24 (‘stunting prevalence’-SA and ‘incomplete age’-WA), the maximum MAE was 0.20 (‘stunting prevalence’; SA) and the maximum % Bias (in absolute value) was 2.97 (‘stunting prevalence’; SA).

Interpretation of Estimated Parameters: The covariates were more frequently significant predictors of ‘incomplete age’, ‘stunting prevalence’ and ‘flagged HAZ’, compared to other indicators and were hence most informative for these indicators (Supplementary Tables 4–8). In terms of the relative magnitudes of the estimated regression coefficients, no covariate uniformly has the largest coefficient (in absolute value) in all the regions for any of the indicators. We note that while most of the estimated relationships between the covariates and the outcome indicators are intuitively in the expected direction, others are not. There are a number of reasons for this, including undetected collinearity or measurement error in the covariates. However, as noted earlier, this is not a concern as this study focuses on prediction. In all cases the estimates of the spatial range (r^), the spatial variance (sigma^2) and the iid error variance (sigma_epsilon^2) were all significant, confirming the contributions of the spatial term omega and the non-spatial errors epsilon(s_1), ..., epsilon(s_n) to explaining residual variation in the fitted models.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This paper addresses a perennial concern of all scientists working on human development in Sub-Saharan Africa (SSA): the quality of data. A decade ago, Devarajan (2013) argued that SSA faced a \"statistical tragedy\" in that \"GDP accounts use old methods, population censuses are out of date, and poverty estimates are infrequent and often not comparable over time. The proximate reasons have to do with weak capacity, inadequate funding, and a lack of coordination of statistical activities.\" See also the extensive discussion in Jerven (2013).\nMany of these shortcomings remain, although there has been progress. For instance, over the past decade the scope of micro-level data has expanded in terms of country coverage, in terms of time coverage, and in terms of spatial coverage, including the geolocation of entities surveyed.\nExamples that are now regularly used by researchers include the Living Standards Measurement Study (LSMS) surveys (LSMS-ISA includes Agriculture), Demographic and Health Surveys (DHS), Household Income and Expenditure Surveys (HIES), Labor Force Surveys (LFS) and the AfroGrid data which now cover 1989-2020 and provide socio-economic, climatic and conflict data at the .5 x .5 degree grid cell level. In addition, satellite data has become available, and many countries implemented reforms within their statistical agencies, and updated/ recalibrated their basic economic data (e.g. Ghana, South Africa).\nStill, many shortcomings remain, and amongst these are measurements errors. Development economists have done quite some work on these, including on the sub-national level. It is well known in development policy circles that data in remote areas are less reliable and that this contribute to the potential greater vulnerability of populations in such remoter areas - indeed the concepts of data gaps, data divides, data justice and data inequality are well explored (e.g. Adebisi and Lucero-Prisno, 2022).\nThis is mentioned as it seems that the current paper seems unaware of this work and tackles the issue of measurement errors only from a health / demographic perspective (illness and malnutrition), and seems to suggest that policy makers are unaware of data gaps and data divides. As such, it misses much that has been learned in development economics. Back in 2009 Naudé et al (2009) for instance argued that the error term in a regression of sub-national income levels on Human Development Indicators in South Africa reflects unexplained variations in human development that carries potentially important information for sub-national policy interventions.\nMore recent examples of work on measurement errors in micro-level surveys in SSA in this regard includes : Abay (2020), Carletto et al (2017), Carletto et al (2013), Desiere & Jolliffe (2018), Gollin and Udry (2021), Gibson and Kim (2010), Hyslop, and Imbens (2001). This is just a modest selection. Much of the studies cited has a been concerned with understanding and correcting for classical and nonclassical measurement errors in agricultural surveys (e.g. LSMS-ISA) - particularly relevant given the importance of agriculture as livelihood across the spatial economy of SSA.\nBrück et al (2022) constructed a spatial fragility exposure index using Kenyan data, calling for taking into account spatial heterogeneity in exposure to conflict to the reliability of data. This is mentioned since the current paper also ignores conflict, and claims that \"The modeling framework includes the following geospatial covariates which we constructed from freely available gridded datasets: population density (1), malaria incidence (2), terrain ruggedness (3), nightlight intensity (NTL) and settlement data (4). We believe a larger set of covariates would add no substantial explanatory power or change the argument of the study.\" Certainly, many authors working on conflict in SSA, the nature of vulnerability and fragility on the continent, and the impact of conflict on health, poverty, and migration, would strongly argue that the authors miss a very important spatial determinant of outcomes, but moreover a variable that affects the quality of data collected - in obvious ways (and perhaps this is even more important in Central Africa, where the authors found the best predictive performance of their model....)\nHence, it is not entirely accurate as the authors claim that \"we still lack knowledge about the spatial distribution and magnitudes of data errors where it matters most - on subnational and district levels.\" The studies cited have resulted in good knowledge about measurement errors in agriculture, the role of conflict in measurement errors, etc.\nHaving mentioned this blindspot however, it has to be emphasised that the current paper does make an interesting contribution to this growing and evolving debate. Their core arguments - that the spatial dimension of policy making requires good sub-national and regional data, and that the reliability of data across sub-national regions are not always equally reliable - are worth stressing. Moreover, their methodology is both creative and rigorous in terms of trying to quantify the measurement errors across sub-national regions in SSA.\nThe only methodological issues I would flag is the likely high correlation between conflict across space and measurement errors in their variables of interest as already mentioned. Related to this, is their lack of using longitudinal data, which has typically been a central strategy used by economists to reduce the impact of unobserved heterogeneity across time where these have been time invariant. For instance, some of the geo-variables that the current study control for, such as terrain ruggedness, are time invariant; to the extent that it would affect nutritional outcomes, using longitudinal data would control for this. The changes in measurement errors over time is therefore a non trivial issue and somehow not discussed.\nFinally, given that it is already well known that data divides and gaps exists, that remote populations are more vulnerable as a result, and that work on measurements errors in surveys are growing and ongoing, the questions that the current paper do not fully answer are \"so what? what should researchers and policy makers be aware of? Can their methods shed light on how to reduce measurement errors?\" The policy and research implications should be highlighted, including being stated upfront. Otherwise the reader will come away from the paper impressed by the data analytics, but with the feeling that not much has been found that was not already known or can be reasonably assumed.\nREFERENCES:\nAbay, K.A. (2020). Measurement errors in agricultural data and their implications on marginal returns to modern agricultural inputs, Agricultural Economics, 51 (3): 323-341; https://doi.org/10.1111/agec.12557\nAdebisi YA, Lucero-Prisno DE 3rd. (2022). Fixing Data Gaps for Population Health in Africa: An Urgent Need. Int J Public Health. Nov 10;67:1605418. doi: 10.3389/ijph.2022.1605418. PMID: 36439279; PMCID: PMC9684304.\nCarletto, G., Gourlay, S., Murray, S., & Zezza, A. (2017). Cheaper, faster, and more than good enough: Is GPS the new gold standard in land area measurement? Survey Research Methods, 11(3), 235–265.\nCarletto, C., Savastano, S., & Zezza, A. (2013). Fact or artefact: The impact of measurement errors on the farm size–productivity relationship. Journal of Development Economics, 103, 254–261.\nDesiere, S., & Jolliffe, D. (2018). Land productivity and plot size: Is measurement error driving the inverse relationship? Journal of Development Economics, 130(1), 84–98\nDevarajan, Shantayanan. \"Africa's statistical tragedy.\" Review of Income and Wealth 59 (2013): S9-S15.\nGollin, D. and Udry, C. (2021). Heterogeneity, Measurement Error, and Misallocation: Evidence from African Agriculture, Journal of Political Economy, 129 (1).\nGibson, J., & Kim, B. (2010). Non-classical measurement error in long-term retrospective surveys. Oxford Bulletin of Economics and Statistics, 72, 687–695.\nHyslop, D. R., & Imbens, G. W. (2001). Bias from classical and other forms of measurement error. Journal of Business & Economic Statistics, 19(4), 475–481.\nJerven, M. (2013). Poor numbers: How we are misled by African development statistics and what to do about it. Ithaca, NY: Cornell University Press.\nNaude, W., McGillivray, M. and Rossouw, S. (2009). Measuring the Vulnerability of Subnational Regions in South Africa, Oxford Development Studies, 37:3, 249-276, DOI: 10.1080/13600810903085800", "reviewer_2": "Subnational variations in the quality of household survey data in sub-Saharan Africa\nSummary:\nIn general, the manuscript presents a comprehensive examination of the quality of survey data, particularly DHS surveys, shedding light on heterogeneity in subnational distribution of sampling and measurement errors. The authors employed a systematic and rigorous approach to create awareness on the quality and utility of survey data for subnational level estimation, particularly at lower level of administration units, drawing upon a rich body of literature and empirical evidence from 33 countries in sub-Saharan Africa. While the manuscript contributes valuable insights to the field of survey research to generate policy- and program-related evidence, there are several areas that require attention and refinement.\nStrengths:\nComprehensive Literature Review: The manuscript provides a thorough review of existing literature on surveys related to their quality and utility in low and middle-income countries. This review helps readers contextualize the research within the broader domain of survey methodology and serves as a solid foundation for further utility of these data at subnational level.\nMethodological Rigor: The manuscript employs a methodologically sound approach to assess quality of survey data to perform subnational level analysis. The authors discuss various quality indicators: (1) incomplete age, (2) age heaping, and (3) ‘flagged HAZ’, providing a holistic view of the challenges faced in subnational level analysis of survey data.\nEmpirical Evidence: The inclusion of empirical data from about 33 countries in sub-Saharan countries strengthens the manuscript's credibility and practical relevance. Such exhaustively analyzed data provide valuable insights into the complexities and nuances of survey data quality issues when it comes to subnational level analysis.\nIncorporation of visual aids: Visual aids, such as tables, charts, and graphs, could be employed to illustrate key findings and trends. Visual representations can help convey complex information more effectively and make the manuscript more engaging. The authors did a great job on this.\nRecommendations: The manuscript offers practical recommendations for researchers to better understand the causes of heterogeneity in data quality (such as lack of birth registrations, lower access to education, migration, or poverty). The manuscript highlights that remote communities are likely the ones who are missed out from such surveys and hence are being misrepresented at subnational level of analysis. The paper recommends that local contexts need to be considered when using survey data to inform policies and interventions, particularly in lower level of administrative units. The authors of the manuscript also made available their estimates of the magnitude of errors, which will be of significant value to researchers, health practitioners and agencies in planning future surveys. These recommendations are well-grounded in the analysis and synthesis of existing literature and survey data from 33 countries.\nAreas for Improvement:\nClarity of Presentation: The manuscript could benefit from improved clarity in its presentation of analysis results on the distribution of errors. Some paragraphs are densely packed with information, making it challenging for readers to follow the arguments and findings. Clearer organization (such as using tables) and more concise language would enhance readability.\nMethodological transparency: While the manuscript discusses various quality indicators, there is room for greater transparency in the methodology used for data analysis and the justification in selecting the type of survey (DHS over MICS), geography (Sub-Saharan over all countries worldwide), year of survey, and subnational level of analysis (level-2 admin over level-3 and level-4 admin for some countries). Providing more details on the criteria used for assessing data quality from these perspectives would enhance the manuscript's rigor.\nGiven DHS surveys have become more robust in terms of sample size and methodology over time, it is not clear why authors chose to limit their analysis to surveys between 2006-2016 (for some countries goes to 2018/19).\nAuthors indicated that, in countries with more than one survey round since 2006, the most recent survey entered the study (completed DHS rounds and selected datasets in Supplementary Table S9). However, this is not the case for some countries. For example, the following countries have more recent data than the one described in table S9, and I recommend that authors consider exploring these datasets.\n• Burkina Faso – 2021\n• Cote D'Ivoire - 2021\n• Ethiopia – 2019\n• Gabon – 2019/21\n• Ghana – 2019\n• Guinea – 2021\n• Kenya – 2022\n• Liberia – 2019/20\n• Madagascar – 2021\n• Mauritania – 2019/21\n• Rwanda – 2019/20\nFor the results to be more informative, as the study suggested, countries with recent DHS survey data with geographic details and perhaps with more sample size and greater reach to remote areas need to be incorporated in the analysis.\nAddressing limitations: The manuscript should acknowledge its own limitations and potential sources of bias. An honest appraisal of the study's limitations would enhance the manuscript's credibility and assist readers in interpreting the findings.\nPractical applications: While the manuscript offers recommendations for improving survey data quality in future, it would be valuable to provide concrete examples of how these recommendations can be applied in real research scenarios. This would assist readers in translating theory into practice. For instance, most of these countries have started collecting routine facility data, and therefore, it would be more helpful to expand the discussion on how to leverage such datasets as appropriate, particularly for policy, planning and programming purposes.\nFurthermore, it is likely that many development-related factors (such as increased access to roads, education, and telecommunication, etc.) brought opportunities to increase the geographic reach of DHS sampling. Given most countries have multiple surveys over the past two decades, it would be worthwhile to expand the discussion using results from Figure S9 to highlight whether or DHS sampling strategies took advantage of increased access to remote areas.\nOverall, this manuscript makes a valuable contribution to the field of survey research by offering a comprehensive analysis of survey data quality at subnational level. With some revisions to improve clarity, transparency, and the incorporation of more recent datasets, it has the potential to become an even more valuable resource for researchers and practitioners. Additionally, addressing the limitations of the study and enhancing the practical applications of the recommendations would further strengthen the manuscript's impact.", "reviewer_3": "Report on “Subnational variations in the quality of household survey data in sub-Saharan Africa”\nThe paper analyzes unambiguous errors in the microdata of Demographic and Health Surveys from 33 African countries.\nThe authors use measures of age heaping, biologically impossible height-for-age Z-scores, and missing data as their error indicators. They show substantial subnational variation in these errors, which is increasing with remoteness, but imperfectly correctly with traditional sampling uncertainty.\nThe basic premises of the paper strike me as valid, and a worthy motivation for the paper: in many low- and lower-middle income countries, DHS takes the place of administrative data as the gold standard for vital statistics. Furthermore, researchers (including myself) typically ignore subnational variation in data reliability.\nI’ll divide my comments between the three main results in the paper.\n---\nResult 1: Subnational variability of data quality\nTo my mind, this result really consists of two steps, and I’m a bit skeptical of the second:\nMeasuring errors in geocoded DHS data.\nExtrapolating those errors down to a level of geographic specificity that the DHS does not (and is arguably not designed to) provide.\nMy doubts here are less with the method used to impute errors at, say, the district level, and more with the motivation and utility of doing so. The paper offers the following motivation:\n“Assessments of household surveys have so far compared data quality across provinces or states (admin-1 level) or between rural and urban populations. This is unsatisfactory, because decisions are often made locally or on the level of districts (admin-2) and should ideally be based on reliable information from the same locality.”\nYes, policy decisions happen on a sub-national basis, of course. But DHS data users are typically discouraged from producing or using estimates at the district level. The DHS program, as the authors know very well, typically only claims the surveys are representative at a higher administrative level, typically admin-1 or even higher.\nIt would be useful for the paper to illustrate the kinds of decisions currently being made on the basis of DHS data at the district level. At the risk of overstating my point, at present this section reads to me as “here’s something you were never supposed to do with DHS data, and sure enough, it seems dubious to do so.”\nFinally, while the maps are useful, it would also be nice to have a clearer analytical statement of the “result” here. For instance, is the variation higher than would be produced by chance? I struggle to translate the finding into concrete words.\n---\nResult 2: Deteriorating data quality in remote areas\nThis section provides useful evidence that the errors documented above are not random, but systematically correlated with observable geographic features.\nThe section refers to “measurement bias”, but I’m unclear exactly what bias this is. If analysts use the DHS to compute regional estimates of, say, child mortality, would the patterns of measurement error documented here bias the results somehow? Perhaps so, but it doesn’t seem obvious to me what kind of bias there would be. The paper would benefit from some analytical account of this, and/or an illustration.\n(As an aside, in line with my comments on result 1, I can’t resist noting that the authors rely on raw DHS data, rather than their model imputation above, to study the relationship between remoteness and errors. This reinforces my doubts about the need for such imputations of error levels at high levels of geographic disaggregation!)\n---\nResult 3: Measurement errors and sampling uncertainty\nI was surprised (in a good way) by the suggestion that measurement error and sampling uncertainty are not concentrated in the same geographic places. As with result 1, however, I didn’t quite see a clear statistical statement of this finding. Even a simple correlation coefficient would help (if this is reported and I missed it, apologies). Once again, the maps are an imperfect substitute for a clear, unambiguous statement or statistic that can be tested or compared to findings from elsewhere.\nLastly, while interesting for someone like me who often uses DHS data, the motivation for exploring this correlation was again not super clear to me. I am tempted to treat the (implied) low correlation between errors and uncertainty as good news, spreading the source of confusion more evenly across geographic space. But that’s a casual speculation, and I’d benefit from more concrete explanation of what we should conclude from this."}, "author_response": ""}, {"round": 2, "reviewer_comments": {"reviewer_1": "In my opinion they have adequately addressed these, and I am satisfied that the paper is suitable for publication.", "reviewer_4": "Dear editor,\nI am writing to peer review article number NCOMMS-23-43116A, titled “Subnational variations in the quality of household survey data in sub-Saharan Africa”.\nThis paper quantifies spatial variation in data quality in the most recent Demographic and Health Survey of 35 African countries, focusing on incomplete age, age heaping, and flagged HAZ as three examples of poor data quality. The paper then creates 5 km x 5 km surfaces of these indicators covering the 35 countries in order to help users of DHS data in academic and policy spaces know which regions of which countries likely have poorer data quality. Finally, the authors show that data quality worsens at longer distances from nightlights (using a DN 15 as the threshold), and they show that there is no correlation between their measures of data quality and measures of statistical uncertainty from sampling.\nI will first offer more overarching comments and then offer more minor comments.\nFirst, I enjoyed the paper and the focus on spatial variation in data quality in the DHS. I concur with the authors that the DHS has emerged as a very frequently used data source not only in academia but also in development and policy for within-country targeting of resources for health and development. Any novel work on the DHS that affects its use by this wide audience is therefore of important general interest. I therefore welcome the idea of quantifying measurement errors at subnational level for a large number of countries for users of the DHS.\nMy first question regards what follows the primary contribution. The paper could have gone in the direction of exploring what drives differences in data quality, as another referee has suggested, but this causal framework would call for different kinds of statistical analysis (exploiting longitudinal data, fixed effects, etc.). The authors have chosen not to go in that direction, which I don’t think is a problem, but it does mean that the remaining contributions of the paper should be substantive. The authors offer two add-ons to the surface mapping of measurement error: (a) they document that the problem becomes more severe as distance from urban areas (proxied by NTL) increases, and (b) the sampling variation is uncorrelated to the measurement error.\nRegarding (a), as the authors already state, it is already known that “…[DHS] data in remote areas can be less reliable relative to urban areas” (line 51). If the result is not too novel, perhaps the authors can deepen the analysis by adding other variables predictive of measurement error. These could be household assets and characteristics, ethnicity and language, etc. Knowing whether other variables observable to DHS enumerators and DHS users are predictive of measurement error would be useful for targeting additional resources to improve data quality, or for analysts, to statistically account for lower data quality using other DHS variables.\nRegarding (b), the authors could improve the motivation. On line 201 they explain that result 3 was explored because they were “curious to know to which extent high sampling uncertainty of stunting overlapped with the biologically implausible height-to-age measurements taken in the same population.” A better motivation would explain why answering this question is important, and how the overlap (or lack thereof) is important for users of the DHS. High statistical uncertainty in rural places with small population (and sample) sizes is to be expected, and is in line with the DHS sampling strategy. So concluding that more sampling is needed there is counter to the DHS strategy. Moreover, investing in increased sample size and improved data quality might be at odds from a resources point of view. In summary, the documented correlations between uncertainty and measurement error, therefore, need to be better motivated for the reader.\nA second question regards the authors’ assertion of bias generated by the measurement error. For example on line 147 when discussing the second result, the authors argue that lower data quality in more rural areas generates a bias. Similarly, the paragraph beginning on line 265 discusses “non-randomly” distributed errors documented by the study. While this could be true if calculating a national estimate of population statistics, the relevant use case here is for localized statistics. For this purpose, it’s unclear to me why there is necessarily bias, since missingness occurring randomly within the data distribution of rural observations (even if more frequent than more urban observations) would not generate a bias, only noisier estimates. Just because there is more missing data in rural areas does not mean it is non-random within the rural sample, which is what would affect bias in calculating population statistics for that region, or causal studies on household-level data. In fact, the authors cite a study by Finaret et al. [2018] which finds that bias from unusable HAZ data is likely small. This then begs the question: how important is this issue that the authors are documenting? Are these measurement errors consequential for DHS-based estimates of population-level characteristics? (Finaret et al. [2018]) suggests not in the case of stunting, but what about age?). Are they consequential for existing studies that use raw DHS data? The authors need to do more to convince the reader that this kind of measurement error matters.\nFinally, I would agree with another referee that conflict needs to be explored as an important predictor of data quality (even if the paper does not move in the direction of a causal estimate on what drives measurement error). Even if left to a supplementary exercise (for example if conflict data do not match DHS survey years), conflict is an easily observable (and likely important) potential predictor of data quality for DHS operations. The authors state that AfroGrid data ends in 2020, but the Uppsala georeferenced conflict dataset extends to 2023: https://ucdp.uu.se/downloads/index.html#ged_global\nMore minor comments:\n1) Page 5: It’s unclear to the reader whether the Moran’s I index was calculated on district-level estimates that are in turn aggregated from the 5x5 geostatistical model result. If that is correct, the high values for Moran’s I are hardly surprising, as the structure of the geostatistical model presupposes spatial autocorrelation in order to create a surface interpolation. More appropriate, I think, would be a spatial autocorrelation test that uses the raw DHS data and cluster locations.\n2) Line 124: “In some areas, pockets of comparable data quality spread across national borders. While this was partially a result of the supranational modeling framework we applied…”: If the authors want to make a statement about whether data quality transcends national borders, this can easily be made into a formal test. The creation of the surface could have been done separately for each country, and a regression discontinuity framework can test for statistical differences in the data quality across international borders. An example of this method is Burke et al. (2016). Otherwise statements like this are too loose.\n3) Line 130: “Furthermore, moderate to little correlation between the three data quality indicators themselves (Supplementary Fig. S3) resulted in distinct geographic patterns for each type of measurement error within countries.”: This is somewhat puzzling for the reader. If these three variables selected from the authors are to be interpreted as broad proxies for data quality, what explains the low correlation between them? Or are these three measures picking up different kinds of data problems? In fact, that correlation coefficients are not so small (.49, .71, and .35).\n4) Line 150: Please provide a citation for a DN of 15 being “above the threshold of the luminosity of public streetlights”\n5) Line 190: Why is contraceptive use a “related public health indicator” to women’s age?\n6) Line 335: The paper refers a few times to “data quality predictions”, I would make clear to the reader that these are the predictions generated in Result 1. In fact this sentence could go after the explanation of the geostatistical model so that the reader better understands what is being referred to when authors explain that result 2 is not using the output of that model.\n7) Figure S2: Is the y-axis share of observations in DHS clusters? This is unclear to the reader.", "reviewer_5": "I have read this manuscript with interest. The authors document interesting facts about the prevalence of three indicators of data quality in DHS data. DHS are cross-sectional data that cover many LMICs, often for several years, and are very widely used in different fields, including demography, public health, and economics.\n- There is of course a vast literature on measurement error (ME) in survey data, also in LMICs. That ME is often made worse when respondents have low numeracy / literacy is also well known. One study in particular that the authors should cite is Larsen, Headey, and Masters (2019). \"Misreporting Month of Birth: Diagnosis and Implications for Research on Nutrition and Early Childhood in Developing Countries\", Demography 56:707–728. This paper, like this manuscript, analyses ME in many DHS data set, in particular (unlike this manuscript) in relation to errors in reported age in month of children. The paper proposes and tests concrete explanations for the patterns of ME, and shows how ME can have important implications for the estimation of causal effects of events that require exact information on the time of birth of children. Also because of this paper I think you need to tone down the statement that (267-68) \"While survey experts or field enumerators may have an intuitive understanding that, for example, age-related information is hard to collect among remote populations, the exact magnitude of the problem has NEVER been assessed.\" Larsen et al indeed evaluate the exact magnitude of the problem, and go actually beyond that by finding explanations and exploring consequences. One key difference relative to your work is that they do NOT look at fine-grained variation in ME, although they do look at cross-country variation (also, they look at child age, while you look at adults). Measurement error in respondent's age in DHS is also analyzed in Rosenzweig (2021), \"Age is measured with systematic measurement error in developing country surveys: A diagnosis and analysis of consequences\" see https://journals.sagepub.com/doi/10.1177/20531680211044068. This also uses data from DHS.\n- I am confused by the idea that \"measures of sampling error\" are \"commonly used to indicate data quality in the literature\" (bottom of p4). I do not think this is true. Sampling error usually exists even with no measurement error, and even if data are being collected with extreme care. Sampling error is not about \"data quality\" but rather about statistical precision, a very different concept. Of course you know this, and you write (263-264) that \"[u]ncertainty of estimates and data quality itself are two different concepts and measuring statistical uncertainty alone, which is the common practice, does not necessarily address low data quality.\" This is of course true, but frankly I also think this is obvious and I don't recall ever seeing confusion about the two concepts, so I do not find this result very interesting and I see no reason why one should have strong priors about the relationship between sampling error and ME. Related: in your Result 3 you write that \"standard deviations are commonly used as a measure for the statistical uncertainty of key public health measures,\" but standard deviations are just by definition a measure of statistical variability, so the statement above seems trivial (a similarly trivial statement would have been that \"sample means are commonly used to measure the average of key public health measures\"). Also, and related, you write that \"sampling errors can often be statistically evaluated and corrected\" (p8, line 198). While sampling errors are routinely evaluated, I don't think you can say that they can be \"corrected\". What you can do is express uncertainty in terms of sampling errors, e.g. through confidence intervals, but the sampling error remains and is not \"corrected\". Perhaps you can can say that there are methods to \"correctly\" take them into account when gauging the degree of uncertainty around estimates.\n- That your measures of data quality deteriorate with \"remoteness\" is perhaps interesting, but remoteness is surely correlated with many other factors, especially literacy / numeracy of respondents. You are careful not to make causal claims about the correlations you document, but on the other hand it was not clear to me why \"remoteness\" was chosen as a key correlate of ME.\nMinor issues / typos\n- P3, line 49, should be \"Practitioners and researchers who work extensively with DHS data...\" (drop \"rely on\"). Line below \"data errors can be DISTRIBUTED ACROSS regions\" sounds awkward / unclear, please rephrase.\n- P4, line 62, should be \"for example AT the level of districts\""}, "author_response": "Response to Reviewers | Household Survey Data Quality\nIn this document we compile all reviewer comments. Reviewer comments are summarized in bold and followed by direct quotations and summaries of the reviewers’ concerns.\nReviewer 1\n1. Reference and use of relevant research. Reviewer 1 had concerns that some key research was left out of the literature review and motivation section for this paper, particularly the development economics research on measurement error. Reviewer 1 stated, “This is mentioned as it seems that the current paper seems unaware of this work and tackles the issue of measurement errors only from a health / demographic perspective (illness and malnutrition), and seems to suggest that policy makers are unaware of data gaps and data divides. As such, it misses much that has been learned in development economics.” Reviewer 1 provided a list of helpful references.\nThank you for your comment. We agree that the study would benefit from additional references to existing work on measurement error and data quality in the development and agricultural economics literature. We found particularly interesting studies that estimate measurement error in microdata and agricultural surveys. Among others, key references in the background/literature review section include works by Devarajan (2013), Jerven (2013), Naude (2014), Adebisi and Lucero-Prisno (2022), and Gibson & Kim (2010). Due to limited space in the main manuscript, we provided a brief discussion of this literature in the Supplementary Information. These additional references exemplify existing explorations of data quality in fields beyond health and demography, which is our focus. The significant additions to the literature review/background have enhanced the manuscript and made its specific contribution clearer.\n2. Incorporating conflict as a variable in the model – conflict as a blind spot in the model. Reviewer 1 stated, “Certainly, many authors working on conflict in SSA, the nature of vulnerability and fragility on the continent, and the impact of conflict on health, poverty, and migration, would strongly argue that the authors miss a very important spatial determinant of outcomes, but moreover a variable that affects the quality of data collected - in obvious ways (and perhaps this is even more important in Central Africa, where the authors found the best predictive performance of their model....).” Hence, it is not entirely accurate as the authors claim that \"we still lack knowledge about the spatial distribution and magnitudes of data errors where it matters most - on subnational and district levels.\" The studies cited have resulted in good knowledge about measurement errors in agriculture, the role of conflict in measurement errors, etc.”\nThank you for this comment. We agree that the presence of conflict and civil insecurity is likely to affect data quality. We did not see it as feasible to include because the AfroGrid data ends in 2020, and the vast majority of surveys we include are now from 2021 and 2022 after updating the DHS surveys. We are also concerned that the conflict data are relatively volatile and are subject to measurement error themselves (i.e., missing events, also difficult to connect the timing and spacing of conflict with the timing and spacing of the surveys accurately). In addition, a consensus-driven and generalizable “conflict exposure” variable does not yet exist, although we do think that there are some interesting research questions about this issue and how it intersects with data quality. We expect that such a data linkage with conflict data might be possible for a particular region if the conflict data were particularly rich and detailed in terms of timing, and if we felt we could develop a spatially aggregated indicator of conflict exposure. However, this is beyond the scope of our current study. We have removed the phrase “we still lack knowledge,” which as reviewer 1 has pointed out, is not entirely accurate. We have enhanced the discussion of the development/agriculture economics literature as described above. We also know that DHS does not operate in regions or countries where it is unsafe for survey enumerators, such as in Somalia.\n3. Methodological issues of no longitudinal data and correlation between conflict and measurement error in the variables of interest: Reviewer 1 stated: “The only methodological issues I would flag is the likely high correlation between conflict across space and measurement errors in their variables of interest as already mentioned. Related to this, is their lack of using longitudinal data, which has typically been a central strategy used by economists to reduce the impact of unobserved heterogeneity across time where these have been time invariant. For instance, some of the geo-variables that the current study control for, such as terrain ruggedness, are time invariant; to the extent that it would affect nutritional outcomes, using longitudinal data would control for this. The changes in measurement errors over time is therefore a non-trivial issue and somehow not discussed.”\nThank you for this comment. We think that it would be methodologically useful to examine changes over time in data quality, and to use fixed-effects for time-invariant factors which may influence data quality like terrain. This would allow us to explore more about the time-variant factors and their role as determinants of data quality, particularly conflict and civil insecurity in some areas. As currently stands, our model is examining the most recent survey in each included country and thus is a cross-sectional model, which we find useful to give a snapshot in time of an important construct (of data quality). In the future, we plan to extend our analysis to include longitudinal data and answer questions about changes in measurement error over time. A full panel data model may be computationally infeasible due to the significant increase in the amount of data used, but we agree that it would be a relevant future research pathway. Additionally, a cross-sectional approach has been used because many countries only had data available for one round of the Standard DHS. A longitudinal analysis would exclude those countries. We do show that data quality is relatively stable across time, for those countries with multiple surveys available in result 2 (Figs 9-12 in the Supplementary Information). Therefore, while this repeat cross-sectional analysis is not as informative as a panel study, it is unlikely that the results are due to cross-sectional variation at a point in time.\n4. Addressing the “so what” question: Reviewer 1 stated, “Finally, given that it is already well known that data divides and gaps exists, that remote populations are more vulnerable as a result, and that work on measurements errors in surveys are growing and ongoing, the questions that the current paper do not fully answer are \"so what? what should researchers and policy makers be aware of? Can their methods shed light on how to reduce measurement errors?\" The policy and research implications should be highlighted, including being stated upfront. Otherwise the reader will come away from the paper impressed by the data analytics, but with the feeling that not much has been found that was not already known or can be reasonably assumed.”\nThank you for making this point. We have revised the text to highlight the relevance of the findings and the implications for policy and research in the beginning of the paper, and clarified the implications of the study in the discussion section of the paper. Here we will clarify the implications of our paper and we have included this explanation in the manuscript as well.\nThe main motivation for this study is that even practitioners and researchers who work a lot with DHS data don't know how large the variation in data quality really is. The most recentDHS methodological report on this issue described implausible values and concluded that \"future research is needed to examine variation in team performance at the regional level to explore the impact of data quality on the interpretation of anthropometric estimates\" (Benedict et al. 2022, DHS Methodological Report 33, p. 44 and 45). We take this as indication that while DHS – like many other researchers and data users as well - are aware of a potential problem, our work is the first to estimate the extent of the errors at district level and at finer resolutions. The larger part of researchers, journals and reviewers simply trust DHS and sometimes look at standard deviations (for measures of uncertainty). We aim to raise awareness with this latter group and quantify the problem for the former group.\nOne of the main implications is that (1) future DHS rounds can improve in areas where we report low data quality – a recommendation raised in many DHS reports including the most recent one (MR 33). This potential for improvement in key geographic areas (and focused on specific challenges – like reducing non-response if that is what is needed in the region) is an implication that we discuss in the manuscript but have revised for clarity in the discussion section. Another implication is that (2) practitioners can understand more about the quality of the DHS data that they are relying on, in the moment, because they often rely on DHS surveys for micro-level data about their populations. With better DHS data also in remote areas, more practitioners would also be able to use them. Another implication is that (3) researchers, journals, and donor agencies will benefit from more awareness and caution when inferring causalities based on DHS data, especially in remote areas. Finally, for applied researchers in particular, another implication is that (4) it is very clear that data quality needs to be considered when analyzing data from remote populations. These implications come directly from our results, which show that relying on just reporting statistical uncertainty is inadequate because there is a bias in the distribution (result 2) and because standard deviations and our reported data errors do not overlap (result 3). These implications are now all made clearer throughout the discussion section as requested by Reviewer 1.\nReviewer 2\n1. Clarity of Presentation: Reviewer 2 stated, “The manuscript could benefit from improved clarity in its presentation of analysis results on the distribution of errors. Some paragraphs are densely packed with information, making it challenging for readers to follow the arguments and findings. Clearer organization (such as using tables) and more concise language would enhance readability.”\nThank you for this comment. In revision we have been able to clear up certain spots where the language was not as concise as possible. We have also kept various results to support the main findings in the Supplemental Information so that the manuscript itself is not too long.\n2. Methodological transparency: Reviewer 2 stated: “While the manuscript discusses various quality indicators, there is room for greater transparency in the methodology used for data analysis and the justification in selecting the type of survey (DHS over MICS), geography (Sub-Saharan over all countries worldwide), year of survey, and subnational level of analysis (level-2 admin over level-3 and level-4 admin for some countries). Providing more details on the criteria used for assessing data quality from these perspectives would enhance the manuscript's rigor.”\nThank you for this comment. We have chosen to focus on Sub-Saharan Africa because DHS data often serve to fill data gaps in many countries in Sub-Saharan Africa, and the DHS program is the most widely used household survey program in the world. As stated in the Methods section: “DHS data are best suited for the scope of the study, because they are publicly available, geocoded, and more comprehensive in time and space than data from other large survey programs such as MICS or LSMS.” In addition, several of the public health outcomes of interest are of particular concern in Sub-Saharan Africa, such as child stunting and contraception use, making it an important region to study the quality of data used to build these indicators. We provide detailed background information on the data quality indicators and how they were developed. In the interest of space in the manuscript, some of these details are located in the Supplemental Information.\n3. Using updated surveys: Reviewer 2 stated, “Given DHS surveys have become more robust in terms of sample size and methodology over time, it is not clear why authors chose to limit their analysis to surveys between 2006-2016 (for some countries goes to 2018/19). For the results to be more informative, as the study suggested, countries with recent DHS survey data with geographic details and perhaps with more sample size and greater reach to remote areas need to be incorporated in the analysis.” Reviewer 2 provided a list of the countries with more recent surveys.\nThank you for this comment. We agree that using the most up-to-date surveys available is ideal for this project, and we have now included them. The data now cover the most recent Standard DHS surveys through 2022. We have also been able to expand the number of included acountres to 35 with Gambia and Mauritania as new additions. Some of the surveys that Reviewer 2 has suggested were not the Standard DHS surveys, such as for maternal health or were “mini” or interim surveys, and we did not include those. These “mini” surveys are smaller in scope, without sufficient data for our model which is the reason why we could not include them. However, all standard, country-wide DHS survey rounds are included.\n4. Addressing limitations: Reviewer 2 stated: “The manuscript should acknowledge its own limitations and potential sources of bias. An honest appraisal of the study's limitations would enhance the manuscript's credibility and assist readers in interpreting the findings.\nThank you for this comment. Our limitations section has been updated to describe additional study limitations, and now reads as follows: “Our modeling framework is, like any statistical model, an abstraction of reality and subject to several limiting simplifications. Spatial data gaps between DHS survey locations may introduce varying degrees of uncertainty in our estimates. These considerations also apply – to a lesser extent – to the aggregated analyses and are relevant to any geolocated household-survey based data set. Also, survey locations are randomly displaced in space for data confidentiality reasons, further increasing the level of spatial uncertainty. However, the covariate layers we use to extract external information for each DHS cluster do not vary substantially at this high resolution and we expect that our overall findings will remain essentially unchanged. Other limitations include that we choose one model specification out of many, which holds potential influence on the predictions. Our estimates include uncertainty which we do not show, as the maps are merely point estimates.”\n5. Practical applications: Reviewer 2 stated: “While the manuscript offers recommendations for improving survey data quality in future, it would be valuable to provide concrete examples of how these recommendations can be applied in real research scenarios. This would assist readers in translating theory into practice. For instance, most of these countries have started collecting routine facility data, and therefore, it would be more helpful to expand the discussion on how to leverage such datasets as appropriate, particularly for policy, planning and programming purposes. Furthermore, it is likely that many development-related factors (such as increased access to roads, education, and telecommunication, etc.) brought opportunities to increase the geographic reach of DHS sampling. Given most countries have multiple surveys over the past two decades, it would be worthwhile to expand the discussion using results from Figure S9 to highlight whether or DHS sampling strategies took advantage of increased access to remote areas.”\nThank you for this comment. We agree that concrete applications are important. We think that the online visualization tool we built (https://apps.worldpop.org/SSA/data_quality/) will be especially useful for enumerators and data managers in the field and in the office to explore previous data quality challenges in the area that they are working. To address this reviewer concern, which overlaps with a concern from Reviewer 1, we also enriched the discussion of the practical implications of the findings in the discussion section of the paper.\nReviewer 3\n1. DHS surveys were not designed to explore errors or data at such a fine geographic specificity: Reviewer 3 states, “Extrapolating those errors down to a level of geographic specificity that the DHS does not (and is arguably not designed to) provide. My doubts here are less with the method used to impute errors at, say, the district level, and more with the motivation and utility of doing so…. Yes, policy decisions happen on a sub-national basis, of course. But DHS data users are typically discouraged from producing or using estimates at the district level. The DHS program, as the authors know very well, typically only claims the surveys are representative at a higher administrative level, typically admin-1 or even higher. It would be useful for the paper to illustrate the kinds of decisions currently being made on the basis of DHS data at the district level. At the risk of overstating my point, at present this section reads to me as ‘here’s something you were never supposed to do with DHS data, and sure enough, it seems dubious to do so.’”\nThank you for this comment. We have clarified this important point in the text, describing some of the spatial modeling that is used for various health indicators from the DHS and elsewhere. While the DHS program was not designed specifically to develop modelled surfaces, the DHS Program themselves routinely produces modelled surfaces using similar methods at the grid-square level, as can be seen, for example, on this website: https://spatialdata.dhsprogram.com/modeled-surfaces/.\nThere is also a growing amount of work using these spatial approaches including the Malaria Atlas Project which is used by the World Health Organization, and we have now included examples like this into the revised manuscript to highlight this increasing usage. We would argue that using cluster-level DHS data in empirical analysis is an increasingly common practice. The DHS Program itself also produce contextual variables at the DHS cluster level – DHS Spatial Covariate data files for download, so that researchers can merge these spatial covariates into the DHS survey data. Moreover, the MICS program at UNICEF are continuing to develop similar outputs, including geospatial measures at the cluster level (e.g. https://www.youtube.com/watch?v=WITZG-KvgjY).\nIn the realm of nutrition specifically, Buckland et al. 2020 argue through an extensive survey that “Periodic, population-based surveys form the backbone of most national nutrition information systems. Commonly available household surveys include the DHS, MICS, as well as a variety of national and sub-national surveys…” Buckland et al. (2020) also find 82% of their respondents did not have access to data at the level of geographic disaggregation that would be necessary to support their work. In addition, the 51st Session of the Committee on World Food Security in 2023 stated that “governments should strive to…. Increase and sustain responsible investment and adequate resources for the production of timely, quality, disaggregated, where relevant, reliable and consistent (food security and nutrition) data…”\nAnother benefit of examining data quality at this resolution is that many researchers use DHS cluster level data to estimate the impact of climate or conflict shocks on cluster-level outcomes, so they would benefit from knowing about a finer resolution of data quality. Therefore, even if we cannot assume that cluster-level indicators are representative, researchers and the DHS Program itself uses cluster-level data in many other kinds of ways for which this work on data quality becomes relevant. In general, given that many countries are working to track progress towards the Sustainable Development Goals, using modeled surfaces like in this paper is an essential strategy to fill data gaps and explore how data quality can be improved in the future.\n2. Clearer analytical statement of the result: Reviewer 3 states about Result 1, “Finally, while the maps are useful, it would also be nice to have a clearer analytical statement of the “result” here. For instance, is the variation higher than would be produced by chance? I struggle to translate the finding into concrete words.”\nThank you for this comment. We agree that there should be a clearer analytical statement of the result here. We have therefore estimated the Moran’s I statistic for all three indicators at the district and grid levels. The Moran’s I statistic measures spatial autocorrelation and its values range from -1.0 to +1.0. The null hypothesis when using Moran’s I for inference is that the geographic attribute being measured is randomly distributed. The results are now in the text as follows: “We see substantial spatial autocorrelation across neighboring districts, summarized using Moran’s I at the district level for age heaping (I=0.971, p<0.0001), the incomplete age information for women (0.896, p<0.0001), and flagged HAZ values (I=0.949, p<0.0001).” Therefore, the use of Moran’s I in the case of this study confirms the non-randomness of data errors across space. Tables S10-S12 provide all Moran's I statistics at the district level with p-values indicating the variation in spatial autocorrelation.\n3. The term ‘measurement bias’ is unclear: Reviewer 3 states, “The section refers to ‘measurement bias’, but I’m unclear exactly what bias this is. If analysts use the DHS to compute regional estimates of, say, child mortality, would the patterns of measurement error documented here bias the results somehow? Perhaps so, but it doesn’t seem obvious to me what kind of bias there would be. The paper would benefit from some analytical account of this, and/or an illustration. (As an aside, in line with my comments on result 1, I can’t resist noting that the authors rely on raw DHS data, rather than their model imputation above, to study the relationship between remoteness and errors. This reinforces my doubts about the need for such imputations of error levels at high levels of geographic disaggregation!)”\nThank you for this comment. We agree that the term “measurement bias” was misleading. We have changed wording and clarified that what we are studying is a bias in the geographic distribution of errors in DHS raw data. We have also revised it to “data quality” and have defined data quality in the introduction as: “We define data quality as a combination of high precision and high accuracy, meaning that measurements have low random error and low systematic error.” We have clarified in the methods section why we prefer using DHS raw data in result 2: “We used raw DHS data to avoid reporting a purely artificial, mechanic relationship arising from regressing data quality predictions (that are a function of distance to nightlight) on distance to nightlight. “\n4. Clear statistical statement of the finding (e.g., a correlation): Reviewer 3 states, “I was surprised (in a good way) by the suggestion that measurement error and sampling uncertainty are not concentrated in the same geographic places. As with result 1, however, I didn’t quite see a clear statistical statement of this finding. Even a simple correlation coefficient would help (if this is reported and I missed it, apologies). Once again, the maps are an imperfect substitute for a clear, unambiguous statement or statistic that can be tested or compared to findings from elsewhere.”\nThank you for this comment. We agree that a clear statistical result of the finding for Result 3 would improve clarity and provide helpful additional information to the maps. Therefore we have estimated correlation coefficients for each indicator of data quality (missing data and standard deviations) for each included indicator (contraception use and stunting prevalence). The results were very similar at the grid and at the district level. In the main article we report on district level results which now reads in the text as follows: “We estimated correlation coefficients between the two types of data quality for each indicator at the district level. Missing data for women’s age was moderately negatively correlated with the standard deviation of contraception use (ρ=-0.402, p<0.0001). Flagged HAZ values were slightly positively correlated with the standard deviation of the prevalence of stunting (ρ=-0.109, p<0.0001).” Figure S15 in the Supplementary Information provides the related charts. We concluded that the missing data and the standard deviation of data quality indicators are not strongly correlated at the district levels.\n5. Clarity of meaning for Result 3: Reviewer 3 states, “Lastly, while interesting for someone like me who often uses DHS data, the motivation for exploring this correlation was again not super clear to me. I am tempted to treat the (implied) low correlation between errors and uncertainty as good news, spreading the source of confusion more evenly across geographic space. But that’s a casual speculation, and I’d benefit from more concrete explanation of what we should conclude from this.”\nThank you for this comment. Our reason for showing Result 3 is that commonly used standard deviations to explore precision do not necessarily address the whole story when it comes to data quality. Low precision/uncertainty as indicated by a high standard deviation of an indicator is a different problem (stemming for example from survey design and sample size) than low data quality (data errors such as measurement errors or missing data). We concluded in the discussion that “uncertainty of estimates and data quality itself are two different concepts and measuring statistical uncertainty alone, which is the common practice, does not necessarily address low data quality.”\nOf those two threats to analytical work, analyzing data quality particularly challenges researchers who are using secondary data and for which they cannot go collect detailed information on sub-samples to assess systematic bias, for example. Consulting the map for which type of data quality problem (uncertainty, error, or both, or neither) would help them assess what they needed to do in their own modeling and research.\nREVIEWER COMMENTS (received Sept 25th 2024)\nWe would like to thank all the authors who took the time and effort to suggest extremely useful corrections. We have also revised substantial parts of the manuscript and believe it has improved significantly. Thank you for your contributions.\n-------\nReviewer #1 (Remarks to the Author):\nIn my opinion they have adequately addressed these, and I am satisfied that the paper is suitable for publication.\nResponse: Thank you. The manuscript has been improved thanks to your comments and suggestions.\n-------\nReviewer #4 (Remarks to the Author):\nDear editor,\nI am writing to peer review article number NCOMMS-23-43116A, titled “Subnational variations in the quality of household survey data in sub-Saharan Africa”.\nThis paper quantifies spatial variation in data quality in the most recent Demographic and Health Survey of 35 African countries, focusing on incomplete age, age heaping, and flagged HAZ as three examples of poor data quality. The paper then creates 5 km x 5 km surfaces of these indicators covering the 35 countries in order to help users of DHS data in academic and policy spaces know which regions of which countries likely have poorer data quality. Finally, the authors show that data quality worsens at longer distances from nightlights (using a DN 15 as the threshold), and they show that there is no correlation between their measures of data quality and measures of statistical uncertainty from sampling. I will first offer more overarching comments and then offer more minor comments.\nFirst, I enjoyed the paper and the focus on spatial variation in data quality in the DHS. I concur with the authors that the DHS has emerged as a very frequently used data source not only in academia but also in development and policy for within-country targeting of resources for health and development. Any novel work on the DHS that affects its use by this wide audience is therefore of important general interest. I therefore welcome the idea of quantifying measurement errors at subnational level for a large number of countries for users of the DHS.\nMy first question regards what follows the primary contribution. The paper could have gone in the direction of exploring what drives differences in data quality, as another referee has suggested, but this causal framework would call for different kinds of statistical analysis (exploiting longitudinal data, fixed effects, etc.). The authors have chosen not to go in that direction, which I don’t think is a problem, but it does mean that the remaining contributions of the paper should be substantive. The authors offer two add-ons to the surface mapping of measurement error: (a) they document that the problem becomes more severe as distance from urban areas (proxied by NTL) increases, and (b) the sampling variation is uncorrelated to the measurement error.\nRegarding (a), as the authors already state, it is already known that “…[DHS] data in remote areas can be less reliable relative to urban areas” (line 51). If the result is not too novel, perhaps the authors can deepen the analysis by adding other variables predictive of measurement error. These could be household assets and characteristics, ethnicity and language, etc. Knowing whether other variables observable to DHS enumerators and DHS users are predictive of measurement error would be useful for targeting additional resources to improve data quality, or for analysts, to statistically account for lower data quality using other DHS variables.\nResponse: Thank you for your comments. Indeed, we are not aiming for a causal analysis and we agree that this means the remaining contributions must be substantive. We believe our statement that \"…it is already known that [DHS] data in remote areas can be less reliable\" was formulated too strongly, as no spatial quantification of data problems has ever been conducted at the subnational level. We have rewritten this section of the introduction to better place our work (a spatial quantification of data errors) in the existing literature (only country / survey level), and to better highlight the relevance for those researchers and practitioners who work a lot with DHS data, and those who so far have taken DHS data without much knowledge of potential limitations. For experienced researchers, they can get the first ever subnational estimates of DHS data errors from this paper. For researchers who are less experienced with DHS data, they can benefit from the maps in Result 1 which illustrate the extent of the problem for three most used variables.\n(a) After much deliberation in the team, we decided not to add additional variables. Instead, we worked on softening our statements, e.g., \"it is already known that...\", which was too strongly worded. We now clearly state what is already known (e.g., that there are data errors in more remote areas) and what is not yet known (the extent of these errors and where they occur using 5x5 km grid estimates). We have made this distinction clearer in the manuscript by stating, \"As a result, even those practitioners and researchers who work extensively with national household surveys and who are aware of the potential for spatial variation in data quality, do not know the precise spatial distribution and the magnitude“\nWe use distance to NTL DN15 and distance to urban settlements, because they capture the effect of many known potential factors such as household assets, institutional quality etc and our intent is to illustrate the vast spatial variation in data quality and that data quality deteriorates non-randomly. We hope that future work in this area will be able to disaggregate the effect of specific variables predictive of measurement errors which would be a significant extension of the presented statistical analysis.\nOur main point is that there is a measurable spatial bias in data errors. Testing a set of variables that explain this at the subnational level, in our view, requires a significantly extended analysis, because these variables are highly correlated and the relationships are complex. We therefore use NTL and distance to settlements as they capture a number of potentially predictive variables.\nRegarding (b), the authors could improve the motivation. On line 201 they explain that result 3 was explored because they were “curious to know to which extent high sampling uncertainty of stunting overlapped with the biologically implausible height-to-age measurements taken in the same population.” A better motivation would explain why answering this question is important, and how the overlap (or lack thereof) is important for users of the DHS. High statistical uncertainty in rural places with small population (and sample) sizes is to be expected and is in line with the DHS sampling strategy. So concluding that more sampling is needed there is counter to the DHS strategy. Moreover, investing in increased sample size and improved data quality might be at odds from a resources point of view. In summary, the documented correlations between uncertainty and measurement error, therefore, need to be better motivated for the reader.\n(b) Thank you for this comment. We agree that the motivation for developing result 3 should be improved with some clarification. Therefore, we have replaced the phrase \"...curious to know...\" with a more clearly articulated argument placed at the end of the introduction, for why this study is important to DHS users and the DHS program. Our motivation for result 3 is not simply to suggest increasing the sample size in areas of higher statistical uncertainty.\nIn the introduction, we define data quality quite narrowly - as having both low statistical uncertainty and as little collection bias in measurement and missing data as possible since each problem can affect the insights that can be drawn from household survey data, even if in different ways. Therefore, result 3 suggests that future assessments of data quality may benefit from looking at both dimensions of data quality, not just uncertainty, which may well be the result of DHS's sampling strategy.\nA second question regards the authors’ assertion of bias generated by the measurement error. For example on line 147 when discussing the second result, the authors argue that lower data quality in more rural areas generates a bias. Similarly, the paragraph beginning on line 265 discusses “non-randomly” distributed errors documented by the study. While this could be true if calculating a national estimate of population statistics, the relevant use case here is for localized statistics. For this purpose, it’s unclear to me why there is necessarily bias, since missingness occurring randomly within the data distribution of rural observations (even if more frequent than more urban observations) would not generate a bias, only noisier estimates. Just because there is more missing data in rural areas does not mean it is non-random within the rural sample, which is what would affect bias in calculating population statistics for that region, or causal studies on household-level data.\nResponse: Thanks for pointing this out. We have clarified in result 2 and in the discussion why we believe our findings are relevant also for smaller than national geospatial estimations. First, we also consider noisy, uncertain estimates as problematic for drawing insights (see our definition of data quality) and second scholars studying remote populations will be made aware of different potentials for the distribution of data errors across countries and across DHS variables (Figure 2). We have made this a stronger argument in result 2 and in the discussion. Thanks for this important point.\nIn fact, the authors cite a study by Finaret et al. [2018] which finds that bias from unusable HAZ data is likely small. This then begs the question: how important is this issue that the authors are documenting? Are these measurement errors consequential for DHS-based estimates of population-level characteristics? (Finaret et al. [2018]) suggests not in the case of stunting, but what about age?). Are they consequential for existing studies that use raw DHS data? The authors need to do more to convince the reader that this kind of measurement error matters.\nResponse: Thank you for this comment. We believe that result 2 is important for data users, as it is the first subnational analysis of remoteness affecting three different data quality indicators. We find that the \"remoteness penalty\" impacts DHS variables to different extents. It is a novel finding that missing data and measurement errors change in different patterns with increasing distance to settlements. HAZ (stunting) is the least affected variable, as we expected based on the findings of Finaret 2018. Compared to Finaret et al. 2018, who conduct cross-sectional analysis comparing surveys – assessing one variable – HAZ – for its quality, we extend and deepen the analysis of data errors at the subnational level and find that two other indicators deteriorate more intensely with increased distance to settlements. Our findings may motivate data users who study remote populations to assess the quality of DHS data the work with on a “case by case” or “variable by variable” basis as we find different extents across our three data quality indicators (which are built on DHS data variables). We clarified this in the introduction of result 2 and towards the end of the discussion.\nFinally, I would agree with another referee that conflict needs to be explored as an important predictor of data quality (even if the paper does not move in the direction of a causal estimate on what drives measurement error). Even if left to a supplementary exercise (for example if conflict data do not match DHS survey years), conflict is an easily observable (and likely important) potential predictor of data quality for DHS operations. The authors state that AfroGrid data ends in 2020, but the Uppsala georeferenced conflict dataset extends to 2023.\nResponse: Thank you for this suggestion. In response to a previous reviewer, we explored conflict as a predictor and figured that it does not add much additional variation over the predictors already included in the model. Of course, we agree that conflict limits the reliability of household survey coverage, not least because of the inability to sample in unsafe and insecure areas. As a result, DHS enumerators omit conflict-affected areas in the national rounds of the survey that form the basis of our analysis. In addition, the predictors we use are likely to capture many of these risks and are also likely to be correlated with conflict. We acknowledged this in the manuscript and included studies of supporting evidence: “Conflicts limit the reliability of coverage from household-based surveys due to an inability to sample in unsafe and insecure areas.\"\nNew evidence/references:\nSbarra, A. N. et al. Estimating vaccine coverage in conflict settings using geospatial methods: a case study in Borno state, Nigeria. Sci Rep 13, (2023).\nKaraye, I. M., Stone, K. W. & Horney, J. A. Determinants of Under-Five Mortality in an Armed Conflict Setting: Empirical Findings from the Demographic and Health Surveys. Int J Environ Res Public Health 19, (2022).\nSchon, J. & Koren, O. Introducing AfroGrid, a unified framework for environmental conflict research in Africa. Sci Data 9, (2022).\nWagner, Z. et al. Women and children living in areas of armed conflict in Africa: a geospatial analysis of mortality and orphanhood. The Lancet Global Health 7, (2019).\nMore minor comments:\n1) Page 5: It’s unclear to the reader whether the Moran’s I index was calculated on district-level estimates that are in turn aggregated from the 5x5 geostatistical model result. If that is correct, the high values for Moran’s I are hardly surprising, as the structure of the geostatistical model presupposes spatial autocorrelation in order to create a surface interpolation. More appropriate, I think, would be a spatial autocorrelation test that uses the raw DHS data and cluster locations.\nResponse: Thank you for this valuable comment. We recalculated Moran's I with raw data and found still considerable but varying values across indicators and across data. We updated the text and the corresponding tables in the Supplementary Information.\n2) Line 124: “In some areas, pockets of comparable data quality spread across national borders. While this was partially a result of the supranational modeling framework we applied…”: If the authors want to make a statement about whether data quality transcends national borders, this can easily be made into a formal test. The creation of the surface could have been done separately for each country, and a regression discontinuity framework can test for statistical differences in the data quality across international borders. An example of this method is Burke et al. (2016). Otherwise statements like this are too loose.\nResponse: Thank you for this comment. We removed this section starting with “In some areas…” to reduce confusion about the motivation of the present paper and remove the loose language. We believe that examining trends and breaks at national borders and across DHS teams would be an interesting follow-up project, especially now that we have information on the subnational distribution of measurement errors. We also added the Burke et al. (2016) paper as a reference in this paragraph: “Following recent studies that have revealed considerable heterogeneity in important health and nutrition indicators of human well-being at the subnational level, which had previously been masked by country-wide measures, we hope to motivate more research into the causes of the subnational heterogeneity in data quality, including low birth registration, low access to education, migration, or high levels of poverty.“\nNew reference:\nBurke, M., Heft-Neal, S. & Bendavid, E. Sources of variation in under-5 mortality across sub-Saharan Africa: a spatial analysis. The Lancet Global Health 4, (2016).\n3) Line 130: “Furthermore, moderate to little correlation between the three data quality indicators themselves (Supplementary Fig. S3) resulted in distinct geographic patterns for each type of measurement error within countries.”: This is somewhat puzzling for the reader. If these three variables selected from the authors are to be interpreted as broad proxies for data quality, what explains the low correlation between them? Or are these three measures picking up different kinds of data problems? In fact, that correlation coefficients are not so small (.49, .71, and .35).\nResponse: Thank you for this comment. We corrected our wording and expanded this section to better explain that the underlying errors and patterns in the errors differ because data challenges are not the same across these indicators. We added that “There was little to moderately positive correlation between the three data quality indicators. The correlation coefficients between pairs of indicators ranged from 0.35 between age heaping and incomplete age to 0.71 between the two age-related indicators, age heaping and flagged HAZ (see Supplementary Figure S3). Data quality challenges are not uniform across the indicators we employed, and different correlation coefficients reflected different underlying potentials for systematic measurement error and missing data. … The only moderate and little correlation between the three data quality indicators resulted in distinctive subnational spatial patterns for each indicator. This reflects the varying degrees of difficulty encountered in collecting the different types of measurements across localities and across DHS data collection teams.“\n4) Line 150: Please provide a citation for a DN of 15 being “above the threshold of the luminosity of public streetlights”\nResponse: Thank you for this comment. We have added the relevant citations to the main manuscript. Before that they were only correctly cited in the Methods section: “DN 15 is below DN 20 assumed for urban, built environments such as in smaller cities. “\n5) Line 190: Why is contraceptive use a “related public health indicator” to women’s age?\nResponse: Thank you for this comment. We added the following to better explain this relationship: “…contraceptive use is an important health indicator for women, as it relates to their overall health and ability to achieve their desired fertility. In addition, maternal health status is a major determinant of stunting.“\n6) Line 335: The paper refers a few times to “data quality predictions”, I would make clear to the reader that these are the predictions generated in Result 1. In fact this sentence could go after the explanation of the geostatistical model so that the reader better understands what is being referred to when authors explain that result 2 is not using the output of that model.\nResponse: Thank you for this comment. We agree that the paper would benefit from improving the precision of our writing and definitions. We have worked on this throughout the manuscript and changed between, e.g., “data quality predictions” to “predictions of data quality as generated in Result 1” or “errors found in raw DHS data,” etc.\n7) Figure S2: Is the y-axis share of observations in DHS clusters? This is unclear to the reader.\nResponse: Thank you for this comment. The y-axis shows the minimum, maximum and the average (population weighted) values of data errors across DHS clusters. This was unclear before. We corrected the caption. Thanks for pointing that out.\n-------\nReviewer #5 (Remarks to the Author):\nI have read this manuscript with interest. The authors document interesting facts about the prevalence of three indicators of data quality in DHS data. DHS are cross-sectional data that cover many LMICs, often for several years, and are very widely used in different fields, including demography, public health, and economics.\n- There is of course a vast literature on measurement error (ME) in survey data, also in LMICs. That ME is often made worse when respondents have low numeracy / literacy is also well known. One study in particular that the authors should cite is Larsen, Headey, and Masters (2019). \"Misreporting Month of Birth: Diagnosis and Implications for Research on Nutrition and Early Childhood in Developing Countries\", Demography 56:707–728. This paper, like this manuscript, analyses ME in many DHS data set, in particular (unlike this manuscript) in relation to errors in reported age in month of children. The paper proposes and tests concrete explanations for the patterns of ME, and shows how ME can have important implications for the estimation of causal effects of events that require exact information on the time of birth of children. Also because of this paper I think you need to tone down the statement that (267-68) \"While survey experts or field enumerators may have an intuitive understanding that, for example, age-related information is hard to collect among remote populations, the exact magnitude of the problem has NEVER been assessed.\" Larsen et al indeed evaluate the exact magnitude of the problem, and go actually beyond that by finding explanations and exploring consequences.\nOne key difference relative to your work is that they do NOT look at fine-grained variation in ME, although they do look at cross-country variation (also, they look at child age, while you look at adults). Measurement error in respondent's age in DHS is also analyzed in Rosenzweig (2021), \"Age is measured with systematic measurement error in developing country surveys: A diagnosis and analysis of consequences\" see https://journals.sagepub.com/doi/10.1177/20531680211044068. This also uses data from DHS.\nResponse: Thank you for this comment. This was a similar comment to the first comment from Reviewer 4 as well. Thanks for the additional reference as well.\nWe agree that important work has been done including assessing the magnitude of specific types of measurement errors and studying potential explanations and consequences. We have clarified this in the discussion and deleted the sentence in question (“While survey experts…”).\nWe also made clearer that our contribution is the spatial quantification at the subnational levels, which has been lacking so far, not necessarily finding explanations. We have also have added low numeracy / literacy as potential factor for data errors in the text, adding Larsen et al. 2019 as citation. We also used Larsen et al. 2019 to improve the description of “flagged HAZ” in the Supplementary Information. Finally, we explicitly mentioned their important findings again in the Discussion of the main manuscript.\nWe also added Cummins (2013) as another example of a paper which examines data errors in the DHS, specifically related to child age.\nA key new paragraph addressing this reviewer comment is in the Discussion section: „Survey experts or field enumerators who have studied measurement errors in DHS data have created awareness for the potential of systematic measurement errors and missing data using cross-country and cross-survey analysis to assess the variation in measurement errors, their magnitude, as well as potential factors and consequences. However, the spatial dimension of systematic measurement errors at fine-grained local levels, where decisions and planning is taking place, has been unknown.“\nIn addition, throughout the study, we have made clearer that, different from Larsen et al. 2019 and similar studies, our contribution is the quantification of measurement errors at the subnational levels which has not been done before. We believe that cross-country and cross-survey measures of systematic measurement errors mask a heterogeneous spatial distribution as we aim to show in our results.\nThank you also for the Rosenzweig reference (2021). We have added this to our introduction and in the description of our indicator “age heaping” in the Supplementary Information.\nNew references:\nLarsen, A. F., Headey, D. & Masters, W. A. Misreporting Month of Birth: Diagnosis and Implications for Research on Nutrition and Early Childhood in Developing Countries. Demography 56, 707–728 (2019).\nRosenzweig, S. C. Age is measured with systematic measurement error in developing country surveys: A diagnosis and analysis of consequences. Research & Politics 8, (2021).\nCummins, J. On the Use and Misuse of Child Height-for-Age Z-Score in the Demographic and Health Surveys, (2013).\n- I am confused by the idea that \"measures of sampling error\" are \"commonly used to indicate data quality in the literature\" (bottom of p4). I do not think this is true. Sampling error usually exists even with no measurement error, and even if data are being collected with extreme care. Sampling error is not about \"data quality\" but rather about statistical precision, a very different concept. Of course you know this, and you write (263-264) that \"[u]ncertainty of estimates and data quality itself are two different concepts and measuring statistical uncertainty alone, which is the common practice, does not necessarily address low data quality.\" This is of course true, but frankly I also think this is obvious and I don't recall ever seeing confusion about the two concepts, so I do not find this result very interesting and I see no reason why one should have strong priors about the relationship between sampling error and ME. Related: in your Result 3 you write that \"standard deviations are commonly used as a measure for the statistical uncertainty of key public health measures,\" but standard deviations are just by definition a measure of statistical variability, so the statement above seems trivial (a similarly trivial statement would have been that \"sample means are commonly used to measure the average of key public health measures\"). Also, and related, you write that \"sampling errors can often be statistically evaluated and corrected\" (p8, line 198). While sampling errors are routinely evaluated, I don't think you can say that they can be \"corrected\". What you can do is express uncertainty in terms of sampling errors, e.g. through confidence intervals, but the sampling error remains and is not \"corrected\". Perhaps you can can say that there are methods to \"correctly\" take them into account when gauging the degree of uncertainty around estimates.\nResponse: Thank you for this comment. We agree that further clarification was needed. We have now explained what we mean by data quality (in the introduction), defining good data quality as: low statistical uncertainty and low systematic measurement error and missing data. We argue that both are necessary to derive high-quality insights from DHS data. We also specify early in the introduction that we consider systematic measurement errors/missing data to be the more serious data problem.\nIn result 3, we make clear that high uncertainties can threaten the predictive power of fitted models. A comparison with measurement errors makes sense because these problems are independent, and data users want to be informed when both occur, because both data characteristics threaten the insights we can derive from analyzing the data. In this sense, our predictions of measurement errors complement the analysis of data quality that is usually provided in the \"limitations\" sections of scientific studies, which so far have largely been concerned with classical measures of statistical uncertainty (such as standard errors) and have had to limit themselves to acknowledging the potential threat of missing data and measurement errors without precise knowledge in which localities and to which extent they occur.\nWe finally improved our wording of \" … taking into account statistical uncertainty…” in the introduction section.\nPlease note that your comment was similar to a comment from Reviewer 4, where we responded:\n“Thank you for this comment. We agree that the motivation for developing Result 3 could use some clarification. Therefore, we fixed the sentence “…curious to know…” and provided a better argument about why the present study is important for DHS users and for the DHS program. We have added some text to explain this: “Finally, in result 3, we contrasted the spatial distribution of measurement errors generated in result 1 with the statistical uncertainties associated with related public health indicators in the DHS data, which may collectively threaten the quality of insights that can be drawn from DHS data.“ We added this to emphasize that the motivation for Result 3 is not simply to increase the sample size in the areas where there is higher statistical uncertainty. Instead, we are aiming to define data quality as having both low statistical uncertainty and low bias (i.e., low measurement error and missing data). Result 3 suggests that future data quality assessments can benefit from looking at both dimensions of data quality, not just uncertainty (which can be improved by increasing sample size).“\n- That your measures of data quality deteriorate with \"remoteness\" is perhaps interesting, but remoteness is surely correlated with many other factors, especially literacy / numeracy of respondents. You are careful not to make causal claims about the correlations you document, but on the other hand it was not clear to me why \"remoteness\" was chosen as a key correlate of ME.\nResponse: Thank you for this comment. We added a clearer explanation of why we work with “remoteness” at the beginning of Result 2. The new text reads: “Remoteness captures a mix of potential factors such as respondent literacy and numeracy, or more challenging conditions for survey collection teams. We did not intend to study these underlying causes. Our aim was to explore the magnitude of this problem and whether it affected the three data quality indicators to a similar extent. In terms of further empirical analysis, the data challenges systematically increasing with remoteness could threaten the insights drawn from the data in two ways. In the best case, measurement errors and missingness are random, leading to more noisy estimates; in the worst case, errors and missingness are systematic, leading to systematic bias in parameter estimates. Data users will need to assess on a case-by-case basis which of these two threats is more relevant to their study based on geographic location, point in time, and variables of interest.“\n- P3, line 49, should be \"Practitioners and researchers who work extensively with DHS data...\" (drop \"rely on\"). Line below \"data errors can be DISTRIBUTED ACROSS regions\" sounds awkward / unclear, please rephrase.\n- P4, line 62, should be \"for example AT the level of districts\"\nResponse: Thank you for these comments – these issues have been fixed as well."}, {"round": 3, "reviewer_comments": {"reviewer_4": "The authors have adequately addressed my comments.\nTwo small quibbles that I leave as optional for the authors:\n1) Line 155: \"5.4% above\" should be \"5.4 percentage points above\". 20% to 25.4% is a 20% increase, in fact.\n2) Line 380: \"However, the predictors that we use likely capture many of these risks and are also likely correlated with conflict.\" I find this sentence frustratingly imprecise. Given that rich geolocated conflict data exist, one could easily test whether the predictors you use actually predict conflict. Better yet, you could have added conflict as a predictor and reported in the paper or in the response to the referees whether or not model fit improved. Suggest, at the very least, that the sentence beginning with \"However...\" on line 380 be removed.", "reviewer_5": "No further comments. Thanks for addressing my requests."}, "author_response": ""}]

Decision letter