SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Readiness. (Optimal) Data was in an appropriate state to apply learning algorithms.
Reliability. (Functional) Data collection bias since seal counting has only been attempted on some regions: "as we do not have image scenes of locations in which counting has been attempted but zero seals have been observed, we cannot generalise our results to other regions of ice".
Sensitivity. (Optimal) The data was open source and classified at Tier 1.
Sufficiency. (Functional) The variability on spatial points was restricted to two locations representing typical seal communities. In order to properly validate the model, it should be done by statistical testing in a new region outside of the given ones.
A.3 DWP: Department for Work and Pension
The UK Department of Work and Pensions (DWP) processes a large amount of personal data, including demographics, family circumstances, and financial details, which presents opportunities for large-scale data analysis and concomitant risks to personal privacy. This datathon (Deineha et al., 2021) evaluated the potential for generating synthetic instances from the original data while preserving utility. The DWP provided two datasets of Universal Credit claimant data. Both datasets were synthetic data, representing the original sensitive population, and are close in structure to existing DWP datasets.
Appropriateness. (Optimal) Since the challenge regarding the evaluation of the similarity of generated datasets, the provided target data and comparator-generated sets were highly appropriate to this task.
Readiness. (Functional) Additional cleaning and merging of the target datasets was necessary before processing; secondary datasets required alignment work.
Reliability. (Optimal) Sources of bias in the data collection methodology were identified due to the generation mechanisms for the target dataset. However, these were limited in their effect on the evaluation metrics used.
Sensitivity. (Functional) While the target data did not contain real personal identifiers, it was still considered confidential and a potential vector for data breaches.
Sufficiency. (Optimal) A large number of available target data rows exceeded the scope of previous challenges in synthetic data generation.
A.4 Dementia Research Institute and DEMON Network: Predicting Functional Relationship between DNA Sequence and the Epigenetic State
This datathon aimed to improve regulatory genomic predictions to understand the impact of genetic variants on gene expression and regulation in disease-relevant cell types, with a focus on dementias such as Alzheimer's disease, and potential implications for drug target development.(Phuycharoen et al., 2022). Four different datasets were provided, containing information about the DNA sequence and the associated chromatin states.
Appropriateness. (Optimal) The data provided contained the features and labels of interest. For example, "...the outputs are always trained to predict one of two epigenetic signals ...in 3 cell types".
Readiness. (Functional) Data required merging and aggregating between datasets to be able to extract insights "Most of the samples were provided with associated DNase-seq data [...] some cell lines [...] were provided with ATAC-seq data instead. Although both assays are broadly used to"
Reliability. (Functional) The datasets used were open-source and annotated by domain experts. Some examples of the datasets used were: EpiMap<sup>4</sup> and Human Genome  $38^{5}$  There were some gaps in the data which the authors were not able to account for. For example, "All profile data ... included occasional not a number (NaN) values."
Sensitivity. (Optimal) The datasets used were open-source and was mapped to tier 0.
Sufficiency. (Optimal) 4 well-labelled and large genomic datasets were used for the challenge.
A.5 Automating Perfusion Assessment of Sublingual Microcirculation in Critical Illness
This challenge sought to determine if a validated measure of microcirculatory perfusion, specifically the microcirculatory flow index, can be directly predicted from a video sequence obtained through darkfield microscopy. The goal is to enable automatic, real-time analysis of the videos, reducing the labour-intensive manual analysis process and facilitating the integration of microcirculatory targets into clinical trials for better patient outcomes (Tomlinson et al., 2022). The data comprised 800 grayscale videos of different lengths and the quality obtained via DFM from 52 patients monitored over four days
Appropriateness. (Optimal) The obtained data and labeled was specifically designed for the challenge. "These videos have been manually analysed to obtain perfusion parameters per short clip, labelling each data point...".
Readiness. (Functional) The dataset was ready but required minor renaming, merging and feature extraction.
Reliability. (Functional) There were cases in which a video had no perfusion parameters and vice versa. "some mismatches were identified ... cases in which a video had no perfusion parameters and vice versa".
Sensitivity. (Functional) The data was commercially sensitive due to containing videos from patients and was mapped to a Tier 2.
4http://compbio.mit.edu/epimap/
<sup>5</sup>https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.39
Sufficiency. (Developing) Since the data relied on manual labeling, the total amount of data was not sufficient to provide statistical validity for the method. "[...]dataset is comprised of a smaller number[...]
A.6 Entale: Recommendation Systems for Podcast Discovery
The datathon aimed to develop methods for capturing relationships between podcasts, incorporating information about the content discussed within them and using this information to produce podcast recommendations (Chan et al., 2021). The data provided included user data (episode and show listened by the users) and podcast data (episode transcripts and show descriptions).
Appropriateness. (Optimal) Since the challenge was to analyse podcast episode content and to produce episode recommendations, user history and podcast data (episode transcripts, episode and show meta-data) is highly appropriate.
Readiness. (Functional) Merging and cleaning of the two datasets was necessary before analysis.
Reliability. (Functional) Bias was induced by a small overlap between datasets, and since the participants were tasked with producing a recommender system, they had some measurement bias when evaluating their methodology since they should be evaluated with new data.
Sensitivity. (Functional) Data was provided by Entale, and so there was a commercial risk for the data.
Sufficiency. (Functional) The two datasets provided were large and enabled users to tackle the research question, but the overlap between user episodes listened to, and episode transcripts was only  $10\%$  making it challenging to incorporate user data with detailed episode content fully.
A.7 Odin Vision: Exploring AI Supported Decision Making for Early-Stage Diagnosis of Colorectal Cancer
Odin Vision is a UK company that has developed AI technology for detecting and characterising bowel cancer. The aim of this datathon was to explore methods that enhance the explainability of Odin-Vision's current machine learning models to aid clinical decision-making by (i) adding a measure of predictive uncertainty along with class prediction, (ii) making the classifications more understandable to the clinicians, and (iii) exploring methods that can automatically learn representations of features using generative models (Almarzouq et al., 2021). The provided data collated three public datasets of colorectal polyp images, obtained by merging data from white light endoscopy and narrow-band-imaging light sources.
Appropriateness. (Functional) A total of three image datasets were employed. Two of them were publicly available and contained both labelled and unlabelled images. The third one was a high-quality, non-public dataset that was not made available to the datathon participants but was employed by Odin Vision to train the models developed during the datathon.
Readiness. (Developing) Many of the images from the publicly available datasets were of low quality due to blur and the presence of reflective spots. A considerable number of images had to be either removed or pre-processed in order to alleviate this issue.
Reliability. (Developing) In addition to the positive-case bias resulting from unbalanced classes (see section on Sufficiency), the datathon participants concluded that the best approach was to train the models predominantly on Odin Vision's higher quality, non-public dataset, and to use a test set from the publicly available images. As the train and test sets were not drawn from the same distribution, the reported quantitative results might be affected by the distributional shift between the train and test datasets.
Sensitivity. (Optimal) The datathon participants only had access to open-source datasets and not to Odin Vision's non-public data, which made this a Tier 0 challenge.
Sufficiency. (Functional) Due to the nature of the datasets (real-world medical images), these were relatively small, with even fewer positive labelled data, which resulted in an insufficient amount of data for learning algorithms and drawing statistically significant answers.