SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Conduct comprehensive health modeling by relating medical prescriptions to environmental, image, and sociodemographic features at a fine-grained spatial level.
- Identify relevant indicators for population health and uncover regional health disparities.
- Understand trends and factors influencing population health over time, including the impact of pandemic years on relevant health factors.
The dataset was created by researchers from:
- Nokia Bell Labs, Cambridge, UK.
- Technical University of Munich, Munich, Germany.
- Wasdi platform (https://www.wasdi.cloud/).
The creation of the dataset was partly funded by the:
- Nokia Bell Labs,
- German Federal Ministry for Economic Affairs and Climate Action in the framework of the "national center of excellence ML4Earth" (grant number: 50EE2201C),
- Munich Center for Machine Learning,
- AI4EO Beyond Fellowship by The International AI for Earth Observation Future Lab at TUM/DLR,
ESA Network of Resources Initiative, and
- European Union's Horizon 2020 research and innovation programme under grant agreement No. 869764, awarded to the GoGreenRoutes project.
For the years covered (2019 and 2020), our dataset contains the following two types of instances:
- LSOA instance: Our dataset covers the entire of England by including an instance per every LSOA in England. These instances are represented by point features describing the LSOAs environmental and sociodemographic indicators.
- Sentinel-2 tile composite instance: For the benefits of using computer vision to extract rich visual concepts that define environmental health, we provide a set of composite Sentinel-2 satellite image tiles per meteorological season covering the entire of England.
In each year, there are 33755 LSOA instances (one per every LSOA in England) and  $4 \times 35$  composite tile instances (one set of tiles per every meteorological season).
Our dataset provides a comprehensive overview of the health landscape in England by incorporating data from all LSOAs throughout the country. The prescription outcomes are derived from all prescribed items by the NHS, encompassing all patients within the nation during the specified years. Additionally, the census data covers the entire population. In terms of environmental variables, we computed yearly average scores using either all non-cloudy observations or a substantial sample of over  $30\%$  of images (e.g., for DynamicWorld) due to computational constraints. This extensive coverage allows for a thorough investigation of health-related factors at a national level. However, it is essential to acknowledge that the representativeness of our dataset diminishes when examining larger geographical regions due to the absence of data from developing countries, and the inherent specificity present even among developed countries.
The LSOA instances are represented by feature vectors that contain environmental and sociodemographic indicators aggregated over the geographical area covered by an LSOA. In total, there are 43 environmental features describing air quality, greenery, climate, and land-cover distribution. Further, we include 111 sociodemographic variables that represent population counts, age group distribution, gender, ethnicity, religion, marital status, employment status, commuting habits, residence and housing, self-reported health, language skills, deprivation and income (the only non-census variable) levels. On the other hand, the Sentinel-2 tile composite instances contain 11 spectral bands that include aerosols, RGB bands, vegetation red edge bands, near-infrared bands and short-wave infrared bands. A detailed overview of the environmental and sociodemographic features and the Sentinel-2
image bands are provided in Figure 1 in the main manuscript and in the "MedSat Variables.csv" file in the public dataset directory.
The target variables in our dataset represent medical prescription prevalences on a yearly level. We included the following 7 target variables representing the quantity of:
- diabetes and hypertension prescriptions associated with metabolic conditions,
- depression and anxiety prescriptions associated with mental conditions,
- asthma prescriptions associated with respiratory conditions,
- opioids prescriptions which are predominantly prescribed for pain management, and
- total prescriptions as a proxy for general health and well-being.
In both years, 5,163 LSOA instances ( $\sim 15\%$  of the available instances) have missing feature values. The missing values for environmental features occur because some of the used satellite products do not provide these features for every LSOA. For example, there are 1,479 (4%) LSOAs for which the variables related to temperature, snow, or radiation derived from the European Centre for Medium-Range Weather Forecasts (ECMWF) Reanalysis v5 product [37] are missing. We observed that this product does not provide these features for LSOAs located near coastal regions. Next, there are 1,507 (4%) LSOAs for which information about the aerosol optical depth and PM2.5 particles is missing from the Copernicus Atmosphere Monitoring Service (CAMS) product [9]. In the future work, we aim to evaluate different satellite products that can provide the missing values for these features. There are 1,956 (6%) LSOAs with missing outcome data. This absence doesn't stem from a lack of prescription data from the NHS. Instead, it arises from a discrepancy between the LSOA shapefiles of the patient data used for outcome calculation from 2018 and the shapefiles from the 2021 census. We opted to utilize the most recent 2021 shapefiles. We await forthcoming releases of NHS patient data that align with these shapefiles, allowing us to incorporate the data for the missing LSOAs. For the sociodemographic indicators, we found that the net annual income is missing for 537 LSOAs as this data was not provided on an LSOA level in the UK census but we derived from the latest 2018 estimates of mean annual household income for Middle layer Super Output Areas (MSOAs) from the Family Resources Survey (and there is a mismatch for a small number of LSOAs and MSOAs due to the latest adjustments to the LSOAs boundaries in 2021).
Notably, we possess Census sociodemographic and image features data for all LSOAs. Depending on the specific analyses intended, the maximum missing value rate could be constrained by the outcomes, standing at  $5.7\%$ . For instance, this applies if a user does not require environmental variables from ECMWF or CAMS products.
To combine both types of instances, we provide shapefiles describing the LSOAs coordinates that allow extraction of image features for individual LSOAs from the composite Sentinel-2 satellite image. Based on these shapefiles, in our code repository we provide an implementation for the extraction of LSOA-specific image features out of the Sentinel-2 images. Further, the region column in the spatial data files determines the larger geographical area to which the LSOA belongs and can be used to cluster the LSOAs based on their spatial proximity and to perform health analysis over a larger geographical area.
We recommend the spatial data split described in Section E.2.2 where LSOAs are first clustered into blocks of sizes  $28\mathrm{km}\times 28\mathrm{km}$ . Next, those blocks are randomly assigned to 5 folds, each fold
containing an equal number of blocks. These procedures ensure that LSOAs belonging to a same block do not appear within the training and the test set.
The environmental and image features utilized in our dataset are derived from high-level satellite products, which undergo preprocessing to mitigate common errors associated with remote sensing acquisition. However, it is important to note that our subsequent processing and spatial aggregation of this data may introduce sources of noise. For instance, we employ a threshold of 0.2 for NDVI values to calculate the fraction of greenery pixels per LSOA. Consequently, different threshold values may yield varying results. Additionally, the prescription values are derived through probabilistic associations between patients and nearby LSOAs, as well as drug names and corresponding conditions. This process, while valuable, is not without imperfections and introduces some level of noise.
In terms of redundancies, our dataset contains many correlated features among the environmental and sociodemographic indicators. A subset of correlated environmental features is presented in Figure 1 where it can be seen that the temperature feature is strongly positively correlated with surface soil temperature, thermal radiation, and atmospheric features while being strongly negatively correlated with snow density. These correlations represent a challenge for many machine learning models and we encourage the researches to investigate approaches for feature selection that can lead to more efficient and accurate health modeling.
Figure 1: Example of correlated environmental features.
Our four complementary dataset components are derived from public data sources. As such, they are not confidential.
Our dataset does not contain any data that can be directly perceived as offensive or inappropriate. As mentioned in the limitations section, the only potential concern is the possibility of stigmatization at the level of LSOA. However, it is worth noting that such stigmatization can also be derived from other publicly available data sources.
Our dataset identifies a percentage of subpopulations by age, ethnicity, and gender across LSOAs derived from the UK census. However, these features do not point to any personally identifiable information because the census implements stringent privacy protection measures, including targeted record swapping and cell key perturbation, to ensure data confidentiality without compromising aggregated statistics [1]. The age group and ethnicity distributions are visualized in Figure 2 which shows that the age groups are balanced except for the age groups over 80 years which occur less frequently compared to the age other groups. When it comes to ethnicity and gender, the White ethnicity is dominant in the dataset while the gender distribution is balanced.
Figure 2: Age group and ethnicity distributions
No, for the reasons stated above.
Certainly, we have extracted sensitive information such as ethnicity, religious beliefs, and economic deprivation per LSOA from the census data. However, it is important to note that this information is publicly available through census records.
The environmental variables were already preprocessed by the satellite products, and we didn't perform further validation. They have varying frequencies, from daily to monthly, depending on the specific product. In our dataset, we created the environmental features by averaging the variables derived from these products on a yearly level. The Sentinel-2 tiles were derived by aggregating publicly available, cloud-free Sentinel-2 images per calendar season that were already preprocessed by the Sentinel-2 mission. Further, the sociodemographic variables are obtained as single values from the UK census data, and as such, do not require validation. The target variables include yearly aggregates of prescription quantities issued by the NHS health authorities. While there is limited condition prevalence data available at the LSOA level across England, our tests for London at the ward level showed moderate to high correlations between our prescription prevalence scores and condition prevalence, ranging from 0.76 for diabetes to 0.89 for depression. However, it is important to note that prescription scores and condition prevalence may not correlate perfectly, as they capture different health constructs.
In our data collection process, we developed our own parser for extracting prescription data. Additionally, for gathering environmental variables, we utilized the Google Earth Engine platform [14] for point data collection and the WASDI platform [44] for image data collection. The code for data collection and preprocessing is available on our GitHub repository.
The data collection process for this study was carried out solely by the co-authors, who were involved in the research as part of their respective roles.
All the data were collected in 2022.
As publicly available data was used, this was not needed.
Yes. The environmental features were averaged on a yearly level, the prescription quantities were normalized according to the number of patients residing in an LSOA, and the Sentinel-2 images were averaged per calendar season.
No, but the used data sources are available publicly from the NHS and ONS websites, and GEE and WASDI platforms.
Yes, on the provided github repository.
In the main manuscript, we have presented the potential value of our dataset in the context of prescription prediction, analysis of relevant health factors, and examination of health disparities. However, there have been no published works utilizing this dataset to date.
The geographical division within our dataset presents a challenge in developing machine learning models that can effectively generalize health findings across diverse regions. Additionally, the inclusion of two years of data in our dataset offers the potential for predicting future health outcomes based on historical health data.
We have meticulously collected and preprocessed the data in accordance with the highest ethical standards to prevent any misuse of our dataset. We strongly urge fellow researchers to conduct responsible analyses and utilize the dataset with integrity.
We caution against presenting the results from our dataset in any stigmatizing way, such as highlighting the worst areas based on certain attributes, without a clear purpose or intention to address the underlying issues. It is important to use the data responsibly and consider the potential implications of how the results are presented and interpreted.
The dataset will be publicly available.
As stated in Section B, the dataset is publicly available on TUMMedia through the following DOI: https://doi.org/10.14459/2023mp1714817
TUMMedia will host and maintain the dataset which ensures the long-term accessibility and citability of our dataset, facilitating its use and reference by the research community.
The email addresses of the authors will be available on the GitHub page.
As part of our ongoing efforts, we are currently developing a prescription parser that enables the calculation of prescription quantities on a monthly basis and going up to 10 years in the past. Additionally, we are enhancing the drug matching module by leveraging state-of-the-art Large Language Models (LLMs). Once these tasks are completed, we will be able to update the GitHub with the updated parser so that interested researchers can calculated additional target variables and enhance the temporal granularity of those features.
Yes, the code for collecting the four complementary components of the MEDSAT datasets is publicly available on our GitHub repository: https://github.com/sanja7s/MedSat.
D.1 Sociodemographic Features
All the sociodemographic variables are collected from the UK 2021 Census using https://www.nomisweb.co.uk/sources/census_2021, except for income, which is collected from Family Resources Survey 2018 https://www.gov.uk/government/collections/family-resources-survey--2.
D.2 Environmental Features
For this release of the MEDSAT dataset, we used the sources listed in Table 1 to derive our environmental features. We performed spatial reduction and averaging of yearly values at the LSOA level. When reducing image data to our areas of interest, we employed specific spatial resolution scaling values:  $1000\mathrm{m}$  for Sentinel-5, CAMS, and TOMS&OMI, and  $10\mathrm{m}$  for ESA-WorldCover and Sentinel-2. These choices were guided by the original products' resolutions and the processing limitations within the GEE platform.
Table 1: Sources and Statistics about of Environmental Variables. The links below should all be prepended with https://developers.google.com/earth-engine/datasets/catalog/
Type	Satellite/Source	Scale (m)	Resolution (m)	Link to data source
Air Q	Sentinel-5	1000	1113.2	COPERNICUS_S5P_OFFL_L3_NO2
Air Q	TOMS&OMI	1000	111000	TOMS_MERGED
Air Q	CAMS	5000	44528	ECMWF_CAMS_NRT
greenery	Sentinel-2	50	10	COPERNICUS_S2_SR
greenery	ERA5-ECMWF	100	11132	ECMWF_ERA5 LAND_MONTHLY(agGR
climate	ERA5-ECMWF	100	11132	ECMWF_ERA5 LAND_MONTHLY(agGR
land cover	DynamicWorld	200	10	GOOGLE_DYNAMICWORLD_V1
D.3 Image Features
To generate composite images, we obtained raw Sentinel-2 satellite data and processed it using the WASDI platform (https://www.wasdi.net), which allows access and online processing of both public and commercial datasets. For our analysis, we processed Sentinel-2 images corresponding to the years 2019 and 2020 by calculating the average values for each of the four meteorological seasons in the both years, as defined by the Met Office [32]. Each Sentinel-2 image consists of 13 bands, with four bands at a spatial resolution of  $10\mathrm{m}$ , six bands at  $20\mathrm{m}$ , and three bands at  $60\mathrm{m}$ . We focused on 11 specific bands, as described in main manuscript, which capture environmental factors relevant to health (e.g., B01 captures aerosols, representing air quality). We excluded bands B09 and B10, as the former is primarily used for cirrus cloud detection and the latter is used to map water vapor.
To generate composite images for each season, we collected and parsed all images within the corresponding three-month period (on average  $\sim 1000$  images per season). This amounted to parsing around 2TB per season and around 8TB per year (see Table 2). For image processing, we conducted several steps to ensure data consistency and quality. Firstly, we resampled the selected bands from their original resolutions of  $60\mathrm{m}$  and  $20\mathrm{m}$  to a uniform resolution of  $10\mathrm{m}$ . This resampling allowed for a consistent analysis across all bands. Next, we applied the cloud mask provided with each Sentinel-2 image to each of the 11 bands. This cloud mask effectively identified and excluded pixels affected by cloud cover, ensuring the accuracy and reliability of the data. Then, we computed a pixel-per-pixel average over time, considering only values observed in cloud-free conditions. This averaging process ensured that the final composite images represented the typical environmental characteristics for each season. Visualizations of a subset of composite bands for the summer of 2020 for LSOAs with the highest and lowest total prescriptions per capita are shown in Figure 3.
D.3.1 Extracting Image Features per LSOA
To extract per-LSOA features from Sentinel-2 composite images, we segmented each image into LSOA-specific imagelets with the procedure depicted in Figure 4. For every imagelet, we computed five descriptive statistics: min, max, mean, std, and median. Our examination of these statistics
highest prescription LSOA
lowest prescription LSOA
optical bands RGB
Figure 3: Visualization of the selected MEDSAT Sentinel-2 composite bands for the LSOAs with highest/lowest total prescriptions quantity per capita.
single band B06
single band B01
revealed inter-correlations, leading to three distinct classes of correlated bands for each image (e.g., cross-correlation of mean values between summer and winter composite images for 2020 can be seen in Figure 5). Further assessment of the statistics' distributions showed skewness in some (like min) and both skewness and extreme outliers in others (specifically max). Moreover, certain statistics, such as mean and median, exhibited inherent correlations.
D.4 Prescription Outcomes
D.4.1 Existing Public and Population Health Datasets
Fine-grained spatial and temporal indices for prevalence of medical conditions are rarely available. Public health agencies typically collect data infrequently through representative surveys (e.g., National health and nutrition examination survey (NHANES) [11], Health Survey for England (HSE) [29], or Behavioral Risk Factor Surveillance System (BRFSS) [33]) or on population samples with cohort studies (e.g., Framingham Heart Study [26], The Swiss National Cohort (SNC) [38]), or The UK Biobank [39]). Surveys come with well-known biases, such as sampling [7], non-response [36], recall [19], or question wording [40] bias. While cohort studies aim at limiting these biases by complementing participant questionnaire responses with their health records, or even genetic
Figure 4: Extracting Image Features: For each seasonal Sentinel-2 composite image, we extracted a set of five metrics per LSOA from each band  $B_{i} \in \{B_{01},\dots,B_{11}\}$ : mean, stdev, min, max, and median. While we show only 3 representative bands for visualization purposes, features were extracted from all 11 bands, resulting in a total of 55 features per seasonal image.
Table 2: The size of the data processed on WASDI platform while producing the environmental image features for each month.
date from	date to	GB
2018-12-01	2018-12-31	671.4277637
2019-01-01	2019-01-31	671.1432031
2019-02-01	2019-02-31	652.6567187
2019-03-01	2019-03-31	648.4059570
2019-04-01	2019-04-31	632.4159082
2019-05-01	2019-05-31	617.2984863
2019-06-01	2019-06-31	641.2775684
2019-07-01	2019-07-31	650.5995117
2019-08-01	2019-08-31	661.7470703
2019-09-01	2019-09-31	631.0063281
2019-10-01	2019-10-31	631.5184570
2019-11-01	2019-11-31	666.4724512
2019-12-01	2019-12-31	688.0771484
2020-01-01	2020-01-31	657.6159668
2020-02-01	2020-02-31	648.8922559
2020-03-01	2020-03-31	614.7591406
2020-04-01	2020-04-31	654.5712695
2020-05-01	2020-05-31	676.6889844
2020-06-01	2020-06-31	640.9450684
2020-07-01	2020-07-31	648.6113184
2020-08-01	2020-08-31	647.2279785
2020-09-01	2020-09-31	654.1471207
2020-10-01	2020-10-31	648.9410100
2020-11-01	2020-11-31	655.3504297
2020-12-01	2020-12-31	622.1216504
2021-01-01	2021-01-31	672.5485547
Figure 5: Cross-correlation matrix for mean band values extracted from summer (JJA) and winter (DJF) Sentinel-2 composites for 2020. We see that bands  $B01 - B05$ , and  $B06 - B8A$ , as well as  $B11 - B12$  form clusters of highly correlated features.
information, they are still limited in size, can be expensive and time-consuming to conduct, and participants may drop out of the study. The All of US Research Program (AoURP) [34] is a unique effort in the US recruiting over 1 million participants to study precision population health. However,
despite its size, AoURP still might face limited representativeness, and the success of the program will depend on long-term engagement and retention of participants. In summary, exhaustive health outcomes data are often of limited scope and granularity, and are prone to a variety of biases.
D.4.2 NHS Medical Prescriptions