How to Data in Datathons
Authors: Carlos Mougan
Richard Plant
Clare Teng
Marya Bazzi
Alvaro Cabrejas-Egea
Ryan Sze-Yin Chan
David Salvador Jasin
martin stoffel
Kirstie Jane Whitaker
JULES MANSER
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: datathon
hackathon
data
data framework
TL;DR: We provide a data analysis framework for data in datathons

Abstract: The rise of datathons, also known as data or data science hackathons, has provided a platform to collaborate, learn, and innovate quickly. Despite their significant potential benefits, organizations often struggle to effectively work with data due to a lack of clear guidelines and best practices for potential issues that might arise. Drawing on our own experiences and insights from organizing +80 datathon challenges with +60 partnership organizations since 2016, we provide a guide that serves as a resource for organizers to navigate the data-related complexities of datathons. We apply our proposed framework to 10 case studies.

How to Data in Datathons
Carlos Mougan
Richard Plant
Clare Teng The Alan Turing Institute
The Alan Turing Institute University of Southampton
The Alan Turing Institute Edinburgh Napier University
Marya Bazzi
Alvaro Cabrejas Egea Fujitsu Research Europe
Ryan Sze- Yin Chan The Alan Turing Institute
The Alan Turing Institute University of Warwick
David Salvador Jasin The Alan Turing Institute
Martin Stoffel The Alan Turing Institute
Kirstie Jane Whitaker The Alan Turing Institute
Jules Manser
The Alan Turing Institute jmanser@turing ac. uk
Abstract
The rise datathons, also known as data or data science hackathons, has provided a platform to collaborate, learn, and innovate in short timeframe. Despite their significant potential benefits, organizations often struggle to effectively work with data due lack of clear guidelines and best practices for potential issues that might arise. Drawing on our own experiences and insights from organizing 2 80 datathon challenges with 60 partnership organizations since 2016, provide guidelines and recommendations that serve as a resource for organizers to navigate the data-related complexities of datathons. We apply our proposed framework 10 case studies.
Introduction
Datathons or data hackathons, loosely defined as data or data science-centric hackathons (Anslow et al., 2016; Chau and Gerber, 2023), have become increasingly popular in recent years, providing a platform for participants and organisations to collaborate, innovate, and learn in the area of data short timeframe. This paper focuses data-related challenges in datathons, including: What does it mean for data to be "appropriate" for a datathon? How much data is "enough" data? How identify, categorise, and use "sensitive" data? We aim to offer set of guidelines and recommendations to prepare different types of data for datathons drawn from extensive experience in datathon organisation.
science
Working with and preparing data a datathon can challenging due to short time scales available prepare the data for analysis and answer specific set of questions. In research projects, there is significantly more time and room to explore and adapt the data and the questions. Other factors include the type of datathon, its potential breadth (c.g., data which is publicly available and has been previously analysed by researchers (Kitsios and Kamariotou, 2019; Concilio et al., 2017)), datasets from a company with 2019), or working with multiple data sources and formats. To effectively incorporate (varied) data, there is need have standardised processes and definitions that can account for the complexity and challenges that arise during the datathon selection and preparation process. In particular, our
and the
limited prior analysis (Nolte al., 2018; Nolte,
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

construct paper differentiates and focuses on the following data-related dimensions: appropriateness, readiness, reliability, sensitivity and sufficieney. Our main contributions are as follows:
Introduce a framework to analyse the data intended for datathons from several dimensions: appropriateness, readiness, reliability, sensitivity, and sufficiency.
Share insights and experiences from organising datathons with external organisations, mapping 10 data challenges and organisations to the proposed framework.
Provide recommendations and best practices for organisers to effectively select, prepare, and categorise data for datathon challenges.
Our paper is organised as follows. In section 2, we overview various aspects of hackathons and datathons, data assessment terminology and frameworks. In section 3, we describe Data Suudy Groups (DSGs), which form the methodology for this paper. In section 4, we introduce our data assessment framework, which details ten datathon use cases and shows how these can be mapped to the proposed framework. In particular, we map the three most recently completed DSGs to the framework propose. We offer some general recommendations on improving data quality in datathons as an event organiser in section 6. Finally, section 7 provides summary of the contribution.
map to existing terminology in the literature, where appropriate. Section 5
2 Related Work
2.1 Hackathons & Datathons
Previous research hackathons has provided valuable insights that enhance one's understand- ing of datathons and their effects on organisations and participants. Extensive literature reviews include Chau and Gerber (2023); Olesen and Halskov (2020). Chau and Gerber (2023) found that hackathon and datathon research tends to focus on four core areas: the event purpose, the event format, the processes undertaken and experienced by stakeholders, and the event outcomes (with some papers having more than one focus). They also add that most publications about processes focus on what happens during the event (e.g., intra-team dynamics in collaboration and project brainstorming) rather than before or after.
As mentioned in the introduction, datathons are data science data-centric hackathons. The work in Anslow et al. (2016) explores how these can enhance data science curriculum (e.g.. through improved data science, communication, and community engagement skills) and that in Kuter and Wedrychowiez (2020) on how to run a datathon with very limited resources. Aboab et al. (2016) focus on healthcare and emphasize the cross-disciplinary aspects of datathons, claiming they help address poor study design, paucity of data, and improve analytical rigour through increased transparency. Luo al. (2021) conducted a virtual datathon focusing on the early stages of the Covid-19 pandemic, highlighting the importance of interdisciplinary and diverse team compositions. Piza et al. (2018) found significant association between positive teamworking behaviors and affective learning in Sealtheare-focusee datathon, pointing out effective leadership as the key factor.
We propose add to existing literature that explores challenges and best practices in datathons, with a focus on data practices. More specifically, and following the terminology in Chau and Gerber (2023), we focus data as a "technical project material" in datathon formats, and examine key challenges that can arise in data evaluation and preparation during the "pre-hackathon" organisation process. To the best of our knowledge, prior research has focused on data-related challenges in the specific context of datathons.
2.2 Data Analysis Dimensions
Previous research data quality evaluation has focused general description frameworks and metrics, rather than datathon context-specific development of methods (Wang and Strong, 1996; Zaveri et al., 2016; Juran al., 1974; Schlegel et al., 2020).
Early work in the area (Wang and Strong, 1996) presented conceptual framework for assessing "data quality" (i.e. assessing issues that can affect the potentiality of the applications that use the data), which included dimensions grouped under four categories: intrinsie; contextual; representational and system- supporee. Indeed, "data quality" is commonly viewed as

---

that defines "fitness of use" and may depend on various factors (Zaveri et al., 2016; Juran et al., 1974), such as believability, accuracy, completeness. relevancy, objectivity, accessibility, amongst several others (Wang and Strong, 1996). The context-specifie methodology of Schlegel et al. (2020) focuses on defect diagnosis and prediction and addresses the following five dimensions, which they argued were the most important for assessing "data quality" (which they rather refer to as "data suitability"); relevancy; completeness; appropriate amount (of data); accessibility and interpretability. Their study focused on developing guidance for companies, using various metrics and evaluation levels, on how to gain insight into the suitability of the data collected in the context of defect diagnosis and prediction. Our work extends and builds upon these previous frameworks and methodologies by applying them specifically to the challenges and requirements of datathons. In particular, we consider the five following dimensions in the context of data- events: data appropriateness, readiness, reliability, sensitivity and sufficiency. In what follows, we link our proposed dimensions existing terminology in Wang and Strong (1996) and Schlegel et al. (2020) where possible, and mention other related research in the field.
mooblem-solving during the datathon
Appropriateness: This dimension considers the question "Is the data relevant to the data science questions posed (i.e., problem--peccifie)". and closely aligns with "relevancy" (Wang and Strong, 1996; Schlegel et al., 2020). Examples of context-specific uses of relevancy include Daly (2006), which provide a comprehensive set of guidelines for analysing specific climate datasets. In their guidelines, the authors outline the relevancy of specific climate factors for different types of analysis, and Schlegel et al. (2020) provides domain-specifi concepts to assess data suitability along the production chain for defect prediction.
Readiness: This dimension considers the questions "Is the data analysis-ready?" and includes "completeness" and "accessibility" (Wang and Strong, 1996; Schlegel et al., 2020). Technology readiness levels (Dunbar, 2017; Hirshorn et al., 2017) was developed in the 1970s as a standardized technology maturity assessment tool in the aerospace engineering sector. NASA developed scale with multiple degrees which has increased from 7, in the early years (Sadin et al., 1989), up to 15 degrees in the latest reviews (Olechowski et al., 2020).
Similarly, Lawrence (2017) proposed the use of data readiness levels, giving a rough outline of the stages of data preparedness and speculating on how formalization of these levels into a common language could facilitate project management. Building on this previous work, we specify four data readiness levels but specifically within the context of datathons challenges.
Afzal et al. (2021) outlines recommended task-agnostic 'Data Readiness Report' template for project's lifecycle, which includes different stakeholders involved such as final contributions by data scientists. Castelijns al. (2020) identifies 3 levels of data readiness and introduces PyWash, which is a semi-automatic data cleaning and processing Python package. Their levels also include a series of steps which includes feature processing and cleaning. In their works, the use of 'data readiness' refers to the end-to-end lifecycle of a data science project, including steps typically taken during a datathon and after. Other works focus specifically on 'big data' readiness and in country-spectie ecosystems which may not be easily extendable (Austin, 2018; Joubert et al., 2023; Hu et al., 2016 6; Zijlstra et al., 2017; Romijn, 2014). Since datathons are not confined to a single use-case, we believe it would be more appropriate to create a framework that can work across different applications.
This dimension considers the question "How biased is the data?", and includes "objec- tivity" (Wang and Strong, 1996). The reliability of data is crucial for Al applications as it directly impacts the accuracy and fairness of the outcomes. Biased data which does not represent the problem fairly returns unreliable results (Ntoutsi et al., 2020). Drawing from established frameworks such as Suresh and Guttag (2019) and Ntoutsi et al. (2020). we identify several potential sources of bias that can undermine the reliability of the data. Selected examples may include: (i) historical bias, which arises from past societal or systemic inequalities: (ii) representation bias, where certain groups or perspectives are underrepresented in the data; (iii) measurement bias, which stems from flawed or subjective data collection methods; and (fv) aggregation bias, which occurs when data is combined or summarised in a way that does not accurately represent each uub-group within the wider population.
Reliability:
This dimension considers the questions "How sensitive is the data?". We build on existing work from Arenas et al. (2019) by incorporating their data sensitivity analysis categorisation to our framework. They propose a defined bised system for sensitive data management in the cloud, which contain: tier O, datasets with no personal information or data which carries legal, reputational, or political risks; tier 1, data that can be analysed on personal devices or data that can be published in
Sensitivity:
3

---

the future without significant risks, such as anonymised or synthetic information; rier 2, data that is not linked directly to personal information, but may derive from it through synthetic generation or pseudonymisation; fier 3, data that presents a risk to personal safety, health, or security on disclosure. This may include pseudonymous or generated data with weak confidence in the anonymisation mechanism or commercially and legally confidential material; tier 4, personal information that poses a substantial threat to personal safety, commercial, or national security and is likely to be subject to attack.
The data used in the study may also need to adhere various regulations, including but not limited to the Data Protection Act (Government of the United Kingdom, 2018), GDPR (European Commission, 2016), and specific regulations governing special categories of data (Information Commissioner's Office (ICO), 2021; European Parliament, 2016). Additionally, different countries or sectors may have their own specific regulations in place. It important to note that the application of these laws is evaluated on a case-by-case basis, and this paper does not provide specific guidance on how to comply with them.
This dimension considers the question "How much data is enough data for given datathon?", and includes "appropriate amount" (Wang and Strong, 1996; Schlegel et al., 2020). This question is highly problem and method-specific, and has been formulated in many research fields. In an early study, Lauer (1995) explored a series of issues in analysing data requirements for statistical Natural Language Processing (NLP) systems and suggested the first steps toward theory of data requirements. Questions such as (i) "Given a limited amount of training data, which methods are more likely to be accurate?", or (ii) "Given a particular method, when will acquiring further data stop to improve predictive accuracy?", were addressed. In intelligent vehicles, Wang et al. (2017) proposed general method to determine the appropriate data model driver behavior from a statistical perspective. They point out that this question has not been fully explored in their field of research, but it is highly relevant given that insufficient data leads to inaccurate models, while excessive data can result in high costs and a waste of resources. Cho et al. (2015) presented a general methodology for determining the size of the training dataset necessary to achieve certain classification accuracy in medical image deep learning systems. They employed the learning curve method (Figueroa et al., 2012), which models classification performance as a function of the training sample size. They pointed out that, in medical image deep learning, data scarcity is a relevant problem. Developing a methodology to estimate how large training sample needs to be in order to achieve a target accuracy is therefore highly relevant. Since datathons include a wide range of applications and approaches, we adopt different take on "sufficiency" and propose general guidelines based experiences in prior datathons. In our case studies, we make case-specific observations (see section 5).
Sufficiency:
3 Data Study Groups
Data Study Groups (DSGs) are an award winniinn collaborative datathon event organised by The Alan Turing Institute, the UK's 's national institute for data science and artificial intelligence. Each DSGs consist of multiple datathons (or "challenges"), typically 3-4, that are worked on collaboratively by single team (rather than multiple teams competing with each other). The aim of DSGs is to provide opportunities for organisations and participants from academia and industry to work together to solve real-world challenges using data science and MI. methodologies. The DSGs are managed and prepared by a specialised internal team of event organisers and interdisciplinary academic support staff.
The events typically run for week, during which 8-12 participants work in teams to explore and investigate the challenges. The events are overseen by experienced researchers that act as "principal investigators" (PI) and "challenge owners" as subject-matter experts. Both provide technical guidance and support for the teams during the week. DSGs have tackled various challenges across different sectors, such as healthcare, finance, and transportation. Figure shows the breadth of challenges undertaken, with 2 69 challenge owners spanning 2 77 datathons across different domains. Note that we have carried out over 80 datathons, but not all datathons have available meta-data for analysis.
During the datathon, a technical report is written by the participants, and edited by the PI after the event. The report collates the work that was carried out by the datathon team and articulates successes, challenges, and recommendations for future analysis directions. The report is not intended to undergo
'bttps//www.prxxssuuill org www.....wwwwwwooeeeooo www.eewwsppoccybbggg/uuingggege data- ke- award winners

---

Environment
Health
Telecomnuntt
perense
Other
Fconomis
Gavermnot
99%
Manufacturina
Well-being
Analytics
Signncning
9.1%
Pharmarauzical
Planning
10.45
Policy
secr
Puplic Palty
Agpicuture
Modia
11.7%
Data Centric Engineering
11.7%
Data for Science
Urban And your
100
12
Defence & Security
Total Count 69
Cha Benge Owner Sector Distribution
Challenge Domain Distribution 77
Figure 1: Left: Bar chart showing the distribution sectors from the challenge owners (n 69). Right: Distribution of datathon topics (N = 77) for the Data Study Groups (DSGs).
traditional academic peer-review process to assess novelty. Rather, members of the DSG team at the Alan Turing Institute review the report for scientific rigour and clarity before publication.
The internal peer-review process is aligned with considerations of responsible research and innovation. DSG challenges hosted the Turing are all assessed for ethical considerations following guidance published by the Public Policy team (Leslie, 2019), therefore, ethics is not an additional data dimension our process,
a separate step that is comparable size data assessment. Although we recommend that all datathon projects be assessed for data protection and ethical considerations, these processes are outside this paper's s scope, as varies by country, organising institution and partners needing for in-depth discussion even within fields to be useful. Our data assessment framework complements institutional policies and can within data governance laws across different countries.
4 Data Assessment Matrix
Insufficient
Developing
Functional
Optimal
Target variables are inappropriate for the research question.
Features aron't entirely appropriate the research question.
Appropriateness
Data unrelated research question(e).
Clear relation between features target
Reediness
No data collected collection methodology.
Some collected data or clear collection methodology in
Data collected but requires marging unifying.
Date collected and cleaned.
Reliability
Strongly blased data.
Indeterminate source of blas.
Determinate source of bias which accounted for.
Weak or clear of bias.
Data does not present significant risk but may be confidential.
Limited risks associoted with the data, usually publicly evailable data.
Sensitivity
Highly risky sensitive data.
Requires extensive security messures.
Sufficiency
Prior experiance shows low success rate.
Does not meet the level previous succesful events.
Previous events have successfully operated at this level.
Extensive data excese of previous successful events.
Figure 2: Summary data assessment matrix for the five data dimensions proposed in this paper.
4.1 Appropriateness: Is the data relevant to the questions posed?
Data appropriateness refers to how relevant the data is for the challenge research questions at hand:
Data is not related to the challenge question is completely unrelated to the problem at hand. As an example, a datathon team tasked with predicting consumer behavior for a new product launch but is given data historical weather patterns instead.
Insufficient:
Developing: Data related to the challenge question, but the target variables needed to answer the research questions missing. For example, datathon may seek to predict stock price movements
5

---

from historical transaction data but without stock prices, it would be challenging to produce useful results.
Functional: Data is appropriate for the challenge but the features can notably be enhanced with data the partner organisation could provide. For example, datathon team is given a dataset on website traffic and user behavior, which is relevant to their challenge, but they identify that additional data on customer feedback would be helpful in making more accurate predictions.
Optimal: Data is highly appropriate and relevant to the challenge. A datathon team is given compre- hensive datasets which provide all the necessary information for making accurate predictions and developing effective solutions. Appropriate meta-data and collection processes are also provided.
4.2 Readiness: Is the data analysis ready?
Data readiness refers to the degree which data is prepared and available in datathon. This includes factors such as the quality and completeness of the data, the availability of relevant metadata, and the ease with which the data can be accessed and analysed. Ease and effectiveness of analysis are closely coupled to effective and thorough data documentation Gebru et al. (2021); Holland et al. (2018), which increase transparency, understandability. and accountability of data and model usage (Mitchell et al., 2019). Data readiness categories are as follows:
Insufficient: No data has been collected yet, and there is methodology for doing so. For example, the datathon involves collecting data from sensors, and there isn't It a clear view of where the data is being stored what it actually contains, the data readiness degree would be insufficient.
Developing: Some data has been collected, and firm methodology in place for doing so. However, there may still be some uncertainty around its quality and completeness. For example, if the datathon involves analysing customer data from a company, and even though they have access to their data, parts of it are missing, or inconsistent, the data readiness degree would be developing.
Functional: Data collected, but some merging and unifying still need be done during the event. The data may require some cleaning and preprocessing, but with minor processing can serve to answer the challenge questions. For example, if the datathon involves analysing public transportation data from different sources, and some data needs to be merged, aggregated, and formatted to fit into the analysis, the data readiness degree would be functional.
Optimal: Data is collated, clean, with no missing gaps, and can potentially be fed to a learning algorithm with minimal processing. The data is also documented with the data dictionary and data cards, such as the ones highlighted in previous research (Gebru et al., 2021; Pushkarna et al., 2022) but in the context of datathons. It is worth noting that dataset documentation in general remains a developing field of study, and datathon- -specific recommendations or guidelines are yet to be established. For example, if the datathon involves analysing a well-established dataset, such as the MNIST dataset, the data readiness degree would be optimal.
4.3 Reliability: How biased is the data?
Reliability here refers to the extent to which the data accurately represents the population or phe- nomenon without systematic data collection errors or distortions.
Insufficient: Data has significant known biases, and conclusions drawn from the analysis will be undermined. For example, in a datathon studying democratic voting patterns, the dataset only includes responses from small, self-selected group of participants, rendering it insufficient to make accurate inferences about the broader population's 's voting behavior.
Developing: Data has unknown or indeterminate roots of bias, and hence no firm conclusions can be made using only these data sources. For example, in a datathon analysing economic indicators of different countries, it is suspected that there geographical and temporal inconsistencies in the data collection. As a result, the dataset is considered to be in the developing stage of reliability.
Data has known biases that will impact the analysis and can be corrected or accounted for as a work limitation. For example, in a datathon analysing customer feedback surveys, is known that the surveys were only completed by customers who voluntarily provided feedback, introducing
Functional:

---

a self-selection bias. The organisers acknowledge this limitation and proceed with the analysis, highlighting the potential impact of the bias on the conclusions.
Optimal: Data does not suffer from a known source of biases with significance for the conclusions. For example, in datathon exploring climate change impacts, the dataset comprises meticulously collected climate measurements from a comprehensive network of sensors across multiple regions.
4.4 Sensitivity: Is the data private or confidential?
Data sensitivity refers to the level of confidentiality and privacy associated with specific types of data. It indicates the degree to which the data is considered private, personal. or valuable, requiring enhanced protection. Ensuring these safeguards may increase the difficulty of hosting a datathon. Therefore, map the sensitivity tiers described by Arenas et al. (2019) to analysis framework.
Insufficient (Tier 4): Highly sensitive data presents significant threats, making hosting datathon impractical. Due to potential legal and personal risks, datathons are not supported at this tier. For instance, handling datasets with commercial or national security sensitivities. This aligns with the UK government's "secret" classification (UK Cabinet Office, 2018). At Tier 4, the risk of malicious actors infiltrating the datathon team becomes notable.
Developing (Tier 3): Careful consideration and extensive security measures would be required prior to hosting datathon in this tier. Datathon participants would be restricted in their use of tools, location and network connections, increasing system friction. For example, in a medical context, a dataset containing highly sensitive patient health records, including names, addresses, and detailed medical histories, poses a substantial threat
Functional (Tier 2): Data in this tier does not present significant personal, legal commercial risk. However, sufficient measures must be taken to mitigate processing risks, such as (i) data de- anonymisation, (ii) competitive data leakage (iii) or inadvertent disclosure. These measures increase the difficulty of participants interacting with the data. For example, an educational dataset with anonymized student records that still hold the risk of re-identification. Hosting a datathon using this data would necessitate robust security measures to prevent any potential breach of student privacy and unauthorized access to sensitive academic information.
Optimal ('fer 0/1): Since risks associated with this data are limited, hosting datathons in virtual environments is usually unnecessary. Free data access to the participants reduces system friction. A dataset containing weather records from a public weather station that doesn't 't include any personal or sensitive information would fall under this tier.
More details, including a discussion on the classification process, as well as the technical requirements to build such environments can be found in (Arenas et al., 2019).
4.5 Sufficiency: How much data is enough data for given datathon?
Data sufficiency refers to the quantity of available data to investigate the challenge questions given a proposed approach. In addition to case-by-case (e.g., depending on the approach and the question) considerations, we propose below an assessment based on previous experiences of similar challenges. In section 5, which is problem-specifii we make case-specific comments on data sufficiency.
Prior experiences have shown that working with similar data quantities on projects with similar objectives hinders the overall success and outcomes of the datathon. For example, in medical research datathon aiming to predict patient outcomes, the provided dataset does not include information about clinical trials.
Insufficient:
Developing: Prior experience has shown that the quantity of data available is not substantial enough to address the research question comprehensively. This quantity of data raises questions about its ability to offer statistical evidence and comprehensive insights into the challenge questions, unlike the insufficient category, data may be present but not adequate for comprehensive insights. For example, a dataset containing a few time-series signals in an industrial failure detection manufacturing datathon not allowing distinguish between normal and failure modes (Bocincova al., 2022).
Functional: Although not surpassing previous successful datathons, prior experience has shown that the quantity of data is sufficient to provide useful insights and statistical evidence into the challenge
7

---

questions. Any insufficieccy can be accounted for as work limitation and discussed in future work. For example, datathon focused on developing an automated system for classifying sea ice and mapping seals to aid ecological monitoring, the geospatial data provided covered only two focal areas, which limits the potential generalizability of the developed system to broader regions (Acharya et al., 2020).
This category represents an amount data which surpasses previous successful datathons. The dataset(s) is extensive (potentially more than what was proposed initially), offering a rich source of information for participants explore and leverage as they tackle the challenge questions. For example, in datathon that aims to predict the functional relationship between DNA sequence and the epigenetic state, having access to comprehensive and well-labelled genomic datasets (Phuycharoen et al., 2022).
Optimal:
5 Case Studies
evaluated the quality of data past DSGs by referring to the final reports available on The Alan Turing Institute's website" and show aggregate summary of the results in Figure 3. It is important to acknowledge a publication bias in these reports, as only those that meet minimum quality criteria determined by the reviewing team made available. Moreover, the reports from early DSGs might not have been published due to problems ranging from unresolved sensitivity issues to misaligned expectations between organisers and challenge owners. Consequently, our focus was primarily on the later versions of the DSG.
This section presents the ten most recent datathons case studies, selected from over 80 datathons, in Figure we provide a summary the results of the data assessment the case studies. These case studies have data, challenge descriptions, and limitations extracted from the publicly available DSG Reports. They are divided between the main body of the report and the appendix.
insufficient
Developing
Functional
Optimal
Appropriate
0
2
3
5
Ready
0
2
5
3
Reliabllty
2
3
5
Sufficiency Sensilvity
0
6
Figure 3: Report count data assessment classification of the last 10 DSG reports
5.1 Cefas: Centre for Environment, Fisheries and Aquaculture Science
The datathon aimed to address the challenge of automating the classification of plankton species and detritus in millions of images collected by the Cefas Endeavour research vessel's 's Plankton Imager system, enabling efficient analysis and contributing to the understanding and conservation of marine ecosystems (Asthana et al., 2022).
"https://www turing.
wtwpettttetttiiitataattttaatatat study groups

---

Appropriateness. (Optimal) Plankton images and their corresponding labels were present in the data: "expert manual classification (labels) allowed challenge participants to verify the accuracy of the automated classification methods explored."
Readiness. (Functional) Images had faulty labels: "[...] we discovered that number of the images labelled as detritus were in fact empty."
Reliability, (Optimal) When writing the challenge proposal, we identified potential bias in the data collection method: all images were collected by the same imager system on the same vessel. However, Cefas did not assess this risk substantial, which was corroborated by the project outcome.
Sensitivity. (Optimal) The data is open access, collected by the Plankton Imager'.
Sufficiency. (Functional) Some species of plankton were under-represented. making an imbalanced problem which needed to be accounted for in the analysis (e.g. using data augmentation).
5.2 The University of Sheffield Advanced Manufacturing Research Centre: Multi-sensor based Intelligent Machining Process Monitoring
This challenge aimed to enhance process monitoring in modern manufacturing using machine learning techniques analyse multiple sensor measurements (Bocincova et al., 2022). Two datasets were provided for this challenge containing time-series process signals the machine operating in both normal and failure modes. The datasets were recorded during separate machining trials.
Appropriateness. (Developing) The target variables needed for the challenge were not present the data: "the provided dataset does not contain any data describing the transition period between the normal operation mode and failure modes"
Readiness. (Functional) The data requires aggregation before being used: "datasets were collected extremely high sampling frequencies ... resulted in excessively big datasets"
Reliability. (Functional) The data experiments were not recorded in the same instance so although measures taken to ensure that they could be accounted for: "The two provided datasets were recorded during two separate machining trials [...]. However, number of identical components were machined in each trial to act repeats for the dataset."
identical possible, there was known bias that
Sensitivity. (Functional) The data was commercially sensitive and was mapped as Tier 2.
Sufficiency. (Developing) Not enough data for the analysis: "There were still enough multivariate time series produced by different runs of the experiment to accurately use image classification."
5.3 CityMaaS: Making Travel for People in Cities Accessible through Prediction and Personalisation
The aim of this datathon was develop a platform that provides reliable information the accessi- bility of destinations (Aragones et al., 2021). ). In particular, participants were tasked to (i) predict the accessibility of points of interest (POIs) in a city; (ii) to develop personalised route-planning algorithm with accessibility constraints and for different "persona" types of their users. For (i). CityMaaS provided manually enhanced point data from OpenStreetMap (OSM) (OpenStreetMap contributors, 2021) and for (ii) information on (synthetic) potential end-user personas were provided (i.e. place of residence, civil status, profession, income, conditions, etc.).
Appropriateness. (Developing) The enhanced OSM dataset provided for the task (1) was appropriate for predicting POI accessibility. However, for task (fi), participants were only given information about the profile of potential users.
(Functional) Significant data cleaning was necessary before the analysis was performed; for example, participants noted the need to "homogenise" the POI data to avoid naming errors in the dataset.
Readiness.
httpps//daaa.. cefas ww/ /202/

---

Reliability. (Developing) The dataset for POI prediction had a class imbalance with 73% of POIs being labelled as accessible. POI data was taken from OSM where users are allowed input tags leading to possible errors and annotation bias.
Sensitivity. (Functional) There was commercial sensitivity for City in providing the enhanced OSM dataset provided and the (synthetic) user profiles.
Sufficiency. (Developing) In particular, for the routing task, no routing data was provided. Participants searched for additional open-source data sources to supplement the data provided by CityMaaS.
6 Recommendations
Recommendations for effectively working with data in datathons can be organised based on the preliminary stages of the event. The following suggestions draw from our investigation of the case studies and are supported by learning from previous datathons. Given that our paper aims to suggest small number of dimensions and recommendations that can be applied to a broad range of datathons, the recommendations below are general and broad-purpae purpose. Specific recommendations for moving from one tier to another or on what constitutes "good enough" will need to be assessed case-by-case, as illustrated in section 5.
Prior to the datathon, one important recommendation is for the event organisers to engage the challenge owners actively. Our framework can help focus discussions with challenge owners on dimensions where scores are below "functional". Their understanding of the problem and domain expertise can greatly improve the data, and early collaboration helps align the objectives and expectations on both sides, increasing the likelihood of a fruitful event. For example, in our Strathelyde challenge (Amaizu al., 2022), the initial proposal was misaligned in terms of readiness, and the project focus was shifted from the participants executing simulations at the datathon to the challenge owners providing them that the participants could focus on pattern- fiiniin..iggg..... Conversely, the Cefas challenge described in section 5.1 achieved greater success due to significant involvement from the challenge owners.
Preparatory phase:
As the datathon approaches, it beneficial to do sanity checks on data readiness and consider changing the challenge questions based input from the Pl. Again, our framework can help focus the efforts of PIs on dimensions where scores are below "functional", and they can case-by-case basis. This phase can help refine the
Refinement phase:
consider the viability of the datathon problem formulation and avoid appropriateness issues, ensuring alignment with the goals of the datathon. For example, by earlier involvement of PI, could have detected the lack of appropriate data or the insufficiency of the multivariate series in the Sheffield challenge in section 5.2.
At the datathon:
During the datathon itself, the PI will be present to help guide participants, having considered and reflected on any data issues and mitigants prior to the datathon. It is also essential to leverage the expertise and feedback of the participating data scientists. Their insights and perspectives can serve as valuable expert advice in identifying areas for improvement across the data quality dimensions. By actively seeking and incorporating participant feedback, organisers and participants can make small adjustments that enhance data utility, facilitating more effective and impactful analyses during and in future events. For example, in the CityMaas challenge of section 5.3, participants found data reliability issues (inconsistent labelling by users), or in Get Bristol Moving (Mougan et al., 2020) alternative sources of open-source data were found that allowed decreasing the sensitivity tier.
7 Conclusions
In this paper, we have analysed data in the context of datathons along five key dimensions: appropri- ateness, readiness, reliability, sensitivity and sufficiency. We then mapped 10 case studies from Data Study Groups to our framework and provided set of recommendations based on the lessons learned. By doing so, we hope to improve the handling of data for organisations prior to datathon events.
Our proposed qualitative analysis provides degree of data status across several perspectives; these degrees can adapted or extended, similar to the Technology Readiness Levels provided by NASA (Sadin et al., 1989), which have been extended through time and further work.
10

---

Acknowledgements
The DSG team wish to thank the past contributions of Dr Merve Alanyali, Dr Alex Bird, Rebecca Blower-Harris, Dr Ayman Boustati, Xander Brouwer, Dr Zhenzheng Hu, Dr Chanuki Illuskka Seresinhe, Catherine Lawrence, Dr Zhangdaihong Liu, Dr Bilal Mateen, Frank Murphy, Daisy Parry, Katrina Payne, and Dr Mahlet Zimeta. With special thanks to Prof Sebastian Vollmer for bringing the concept to us and starting DSGs the Turing back in 2016, and Dr Franz Kiraly for his work and input in the early years. We also wish to thank all past Challenge Owners, DSG Pls and participants for their invaluable contribution and for making DSGs possible.
Carlos Mougan was funded by the European Union's 's Horizon 2020 research and innovation program under the Marie Sklodowska- Actions (grant agreement number 860630) for the project: "NoBIAS Artificial Intelligence without Bias".
References
Aboab, J., Celi, L. A., Charlton, P., Feng, M., Ghassemi, M., Marshall, D. C., Mayaud, L.., Naumann, T., McCague, N., Paik, K. E., Pollard, J., Resche-Rigon, M., Salciccioli, J. D., and Stone, D. (2016). A datathon model to support cross-disciplinary collaboration. Science Translational Medicine, 8(333):333ps8-3333
Acharya, A., Laws, A., Tegho, C., Miller, E., Shone, F., Rincon, L. M. G., Baker, L., Deecke, L., Gill, P. Hurst, T., Pollington, T., and Sanchez-Silva, V. (2020). Seals from space: automated antarctic ecosystem monitoring via high-resolution satellite imagery. Technical report, The Alan Turing Institute, University of Exeter, Calipsa, University of Cambridge, Arup London, University of Warwick, University of Glasgow, University of Edinburgh, British Antarctic Survey, Heriot Watt University.
Afzal, S., Rajmohan, C., Kesarwani, M., Mehta, S., and Patel, H. (2021). Data Readiness Report. In 2021 IEEE International Conference on Smart Data Services (SMDS), pages 42-51.
Almarzouq, B., Cabrera-Arnau, C., Ditter, E.-J., Duckworth, P., Jewsbury, R., Kazmierski, M., Li, Z., Lin, S., Lines, D., Mun, T. S. H., Yallup, D., and Zanisi, (2021). Odin vision: Exploring ai supported decision-making for early-stage diagnosis of colorectal cancer. Technical report, The Alan Turing Institute.
Amaizu, M., Paul, O., Francesco, D. D., Nevin, J., Dirik, A., Sekkat, C., Matavalam, A. R. R., Papadopoulos, P., Tzelepis, D., Nakas, G., and Koukoura, S. (2022). Data study group final report: University of strathclyde and supergen energy networks hub. Technical report, The Alan Turing Institute.
Anslow, C., Brosz, J., Maurer, F., and Boyes, M. (2016). Datathons: an experience report of data hackathons for data science education. In Proceedings of the 47th ACM Technical Symposium Computing Science Education, pages 615-620. ACM.
Aragones, M., Bedogni, L.. Deshpande, A., Gieschen, A., Huang, Z., McIntyre, F., Nawfee, S. M., Roumpani, F., Saad, A., and Vonnak, (2021). Citymaas: Making travel for people in cities accessible through prediction and personalisation. Technical report, The Alan Turing Institute.
Arenas, D., Atkins, J., Austin, C., Beavan, D., Egea, A. C., Carlysle- Davies. S., Carter, I., Clarke, R., Cunningham, J., Doel, T., Forrest, O., Gabasova, E., Geddes, J., Hetherington, J., Jersakova, R., Király, F. J., Lawrence, C., Manser, J., O' O'Reilly, M. T., Robinson, J., Sherwood-Taylor. H., Tierney, S., Vallejos, C. A., Vollmer, S. J., and Whitaker, K. J. (2019). Design choices for productive, secure, data-intensive research at scale in the cloud. CoRR. 0bss/1908.087
Asthana, M., Blackwell, R., Clinciu, M., Ding, J., Giering, S., Hewitt, C., Rehman, A. u., Sahoo, R., Salvador-Jasin D.,, Mchonic, A.. Coca- Castro, A., Corcoran, E.,, and Costa-Gomes, B. (2022). Data Study Group Final Report: Centre for Environment, Fisheries and Aquaculture Science (Cefas). Technical report, Alan Turing Institute.
Austin, C. C. (2018). A path to big data readiness. 2018 IEEE International Conference on Big Data (Big Data), pages 4844-4853, IEEE.

---

Bocincova, A.. Boniface, C., O'Driscoll, R., Finol. D., Liu, X., McQuire. J., Meech, J., Mucchielli, P., Organokov, M., Teng, C., and Torquato. M. (2022). Data Study Group Final Report: The University of Sheffield Advanced Manufacturing Research Centre. Technical report, The Alan Turing Institute.
Castelijns, L. A., Maas, Y., and Vanschoren, J. (2020). The ABC of Data: Classifying Framework for Data Readiness. In Cellier, and Driessens, K., editors, Machine Learning and Knowledge Discovery in Databases, pages 3-16, Cham. Springer International Publishing.
Chan, R. S. Y., Choudhari, J., Fitzgerald, J.. Loghmani, E.,, McGowan, J., Pope, V., Price, I., Roster, K. Zhang, L.,, and Gamper, J. (2021). Entale: Recommendation systems for podcast discovery. Technical report, The Alan Turing Institute.
Chau, C. and Gerber,
(2023). On Hackathons: A Multidisciplinary Literature Review. In CHI '2:: Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, pages 1-21. ACM.
Cho, J., Lee, K., Shin, E., Choy, G., and Do, S. (2015). How much data is needed to train medical image deep learning system achieve necessary high accuracy?.
Concilio, G., Molinari, F., and Morelli, N. (2017). Empowering Citizens with Open Data by Urban Hackathons. In 2017 Conference for E-Democracy and Open Government (CeDEM), pages 125-134.
Daly, C. (2006). Guidelines for assessing the suitability of spatial climate data sets.
Deineha, O., Grace, C., Mhatre, S., Treetanthiploet, T., Valdenegro, D., Zhou, L., Packer, E., Kohli, S., Thakerar, A., and Plant, R. (2021). Data Study Group Final Report: Department for Work and Pensions. Technical report, The Alan Turing Institute, London, UK.
Dunbar, B. (2017). Technology readiness levels demystified. NASA, NASA, Aug.
European Commission (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance).
European Parliament (2016). Directive (eu) 2016/1 148 of the european parliament and of the council 6 july 2016 concerning measures for a high common level of security of network and information systems across the union. Official Journal of the European Union
Figueroa, L., Zeng-Treee Treitler, Q., Kandula, S., and Ngo, L. H. (2012). Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making, 12(1).
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H. M., III, H. D., and Crawford, K. (2021). Datasheets for datasets. Commun. ACM, 64(12):86-92.
Government of the United Kingdom (2018). Data protection act 2018. Chapter 12.
Hirshorn, S. R., Voss, L. D., and Bromley, (2017). NASA systems engineering handbook. Technical report, National Aeronautics and Space Administration.
Holland, S., Hosny, A., Newman, S., Joseph, J., and Chmielinski, K. (2018). The dataset nutrition label: A framework to drive higher data quality standards. CoRR, abs/1805.03677.
Hosseini, K., Ardanuy, M. C., Patterson, D. J., Garcia- Velez, L., Castro-Gonzalez, L., Deecke, L., Saattrupielsen, D. N., Cho, S., Dwyer, L., Auyeung, V., Krause, A., Coca-Castro, A., Vonnak, R., Balcato Lizancos. A., Miron, A., Metzler, A. B., and Goldmann, K. (2020). Wwf smart monitoring for conservation areas. Technical report, The Alan Turing Institute.
Hu, Y., Bai, X., and Sun, S. (2016). Readiness assessment of open government data programs: A case of shenzhen. Proceedings of the 17th International Digital Government Research Conference on Digital Government Research, pages 97-103.
12

---

Information Commissioner's Office (ICO) (2021). Guide to the general data protection regulation (gdpr).
Joubert, A., Murawski, M., and Bick, M. (2023), Measuring the big data readiness of develop- ing countries index development and its application to Africa. Information Systems Frontiers, 25(1):327-335
Juran, M., Gryna, F. M., and Bingham, R. S. (1974). Quality control handbook, volume Rainbow- Rainbow-Bridge.
Kitsios, F. and Kamariotou, M. (2019). Beyond Open Data Hackathons: Exploring Digital Innovation Success. Information, 10(7):235, Number: 7 Publisher: Multidisciplinary Digital Publishing Institute.
Kuter, K. and Wedrychowicz, C. (2020). Hosting data science hackathon with limited resources. Stat, (10:e338).
Lauer, M. (1995). How much is enough?: Data requirements for statistical NLP.
Lawrence, N. D. (2017). Data Readiness Levels. CoRR, abs/1705.02245.
Leslie, D. (2019). Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of Al systems in the public sector. Technical report, The Alan Turing Institute.
Luo, E. M., Newman, S., Amat, M.. Charpignon, M.-L., Duralde, E. R., Jain, S., Kaufman, A. R., Korolev, Lai, Y., Lam, B. D., Lipesey, M., Martinez, A., Mechanic, O. J., Mlabasati, J., McCoy, L. G., Nguyen, F. T., Samuel, M., Yang, E., and Celi, L. A. (2021). MIT COVID-19 Datathon: data without boundaries. BMJ Innovations, 7(1). Publisher: BMJ Specialist Journals Section: Letter
Mitchell, M., Wu, S., Zaldivar. A., Barnes, P., Vasserman, L., Hutchinson, B.. Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model cards for model reporting, In FAT, pages 220-229. ACM.
Mougan, C., Fu, Q. Kolath, J., Tong, H., Dixit, S., Geffert, L., Shin, H., Rabuh, A., Robinson, C., and Gale, E. (2020). Data Study Group Network Final Report: Bristol City Council (Gel Bristol moving: tackling air pollution in Bristol city centre). Zenodo. Turing Network Data Study Group Bristol; Conference date: 05-08-200 2019 Through 09-08-2019.
Nolte, A. (2019). Touched by the Hackathon: study on the connection between Hackathon participants and start-up founders. In Proceedings of the 2nd ACM SIGSOFT International Workshop on Software-Imtensive Business: Start-ups, Platforms, and Ecosystems, IWSiB 2019, pages 31-36, New York, NY, USA. Association for Computing Machinery.
Nolte, A., Pe-Than, E. P. P., Filippova, A., Bird, C., Scallen, S., and Herbsleb, J. D. (2018). You Hacked and Now What? Exploring Outcomes of Corporate Hackathon. Proceedings of the ACM on Htaman-Compint Interaction, (2CCC)::29:11122925
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F, Papadopoulos, S., Krasanakis, E.,, Kompatsiaris, I., Kinder-Kurlanda K., Wagner, C., Karimi, F, Fernánde7, M., Alani, H., Berendt, B., Kruegel, T., Heinze, C., Broelemann, K., Kasneci, G., Tiropanis, T., and Staab, S. (2020). Bias data-driven artificial intelligence systems an introductory survey, WIREs Data Mining Knowl. Discov, 10(3).
Olechowski, A., Eppinger, S. D., Joglekar, N. R., and Tomaschek, K. (2020). Technology readiness levels: Shortcomings and improvement opportunities. Syst. Eng., 2354):395-408.
Olesen, J. F. and Halskov, K. (2020). 10 Years of Research With and On Hackathons. In DIS '20: Proceedings of the 2020 ACM Designing Interactive Systems Conferenc, pages 1073-1088. ACM.
OpenStreetMap contributors (2021). Planet dump retrieved from https://pannt..omm.org
Phuycharoen, M., Murphy, A., Morgenstern, C., Yaman, U., Mathis, S. V., V, S. T., Dinh, T. T., Abrahamian, M., Sharma, P., Tantiangco, H., Mandal, S., Seal, S., and team, D. S. G. (2022). Data Study Group Final Report: UK Dementia Research Institute and DEMON Network. Technical report, The Alan Turing Institute.
13

---

Piza, F. d. I., Celi. L. A., Deliberato, R. O.. Bulgarelli, L., de Carvalho, F. R. T., Filho, R. R., de La Hoz, M. A. A., and Kesselheim, J. (2018). Assessing team effectiveness and affective learning in a datathon. International Journal of Medical Informatics, 112:40-44.
Pushkarna, M., Zaldivar, A., and Kjartansson, O. (2022). Data Cards:
Purposeful and Transparent Dataset Documentation for Responsible AL 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1776-1826, Scoul Republic of Korea. ACM.
Romijn, J. (2014). Using big data in the public sector. Uncertainties and Readiness in the Dutch Public Executive Sector..
Sadin, S. R., Povinelli, F. P., and Rosen, R. (1989). The NASA technology push towards future space mission systems. In Space and Humanity, pages 73-77. Elsevier.
Schlegel, P., Buschmann, D., Ellerich, M., and Schmitt, (2020). Methodological Assessment of Data Suitability for Defect Prediction. Quality innovation prosperity/kvaliaa Inovácia Prosperita, 24(2).
Suresh, H. and Guttag, J. V. (2019). A framework for understanding unintended consequences of machine learning. CoRR, abs/1901.10002.
Tomlinson, C., Grols, J., Debnath, R., Johnson, S., Barton, M., Han, T., Khatami, S. N., Baldo, G., Ramesh, A., Cammarano, D., Rajpoot, K., and team, D. S. G. (2022). Automating perfusion assessment of sublingual microcireulation in critical illness. Data study group final report, The Alan Turing Institute.
UK Cabinet Office (2018). Government Security Classifications. Accessed: 03.05.2022
https://bit.11/220000
Wang, R. Y. and Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of management information systems, 12(4):5-33.
Wang, W., Liu, C., and Zhao, D. (2017). How much data enough? A statistical approach with case study on longitudinal driving behavior. IEEE Transactions on Intelligent Vehicles, 2(2):85-98.
Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., and Auer, S. (2016). Quality Assessment for Linked Data: A Survey. Semantic Web, 7(1):63-93.
Zijlstra, A. A., Vaira, C. L., and Boothe, (2017). Open data readiness assessment: Malaysia.
14

---

A Appendix: Case Studies
A.1 WWF: Smart Monitoring for Conservation Areas
The datathon focused on developing NLP system to automatically detect news articles reporting emerging threats to protected areas, enabling real-time monitoring and proactive conservation efforts by WWF and the wider conservation community (Hosseini et al., 2020). The dataset contained approximately 45, 50,00 files describing news stories relevant UNESCO World Heritage Sites, of which 135 were laheled as O-'No threat', 1- T-'Thr detected but not actors' and 2- 2-"Threat with actors'.
Appropriateness. (Optimal) The data was obtained using the Google News API and was manually labeled by WWF experts to be relevant to the challenge.
Readiness. (Developing) Even thought there was a data collection methodology in place, the team had to increase the number of data points available during the event: "... this dataset was expanded during the DSG week to 1,000 labelled data points".
Reliability. (Functional) Data was manually labelled by WWF experts and scraped using the Google News API, creating a before-hand known data collection bias.
Sensitivity. (Optimal) News articles were scraped using Google News API which made it an open source project of Tier O.
Sufficiency. (Developing) The number of news scraped was potentially sufficient, but the amount of labeled data was insufficient: "... initially a training dataset of 135 news articles annotated by the WWF experts were supplied".
A.2 British Antarctic Survey: Seals from Space
The datathon aimed create an automated system for classifying sea ice, mapping seals, and exploring envirommental factors to facilitate ecological monitoring and understanding of the Antarctic ecosystem (Acharya al., 2020). The data consisted of GeoTIFF files covering two focal areas, the Antarctic Peninsula (Crystal Sound and Marguerite Bay) and Signy Island. Location data from over 2000 manually-countee seals accompanied these images.
Appropriateness. (Functional) Even though the images and seal location data were provided, some important covariates such nursing locations, ice features, marine food availability, were not provided.
Readiness. (Optimal) Data was in an appropriate state to apply learning algorithms.
Reliability, (Functional) Data collection bias since seal counting has only been attempted on some regions: "as we do not have image scenes of locations in which counting has been attempted but zero seals have been observed, we cannot generalise our results to other regions of ice".
Sensitivity. (Optimal) The data was open source and classified at Tier
Sufficiency. (Functional) The variability on spatial points was restricted to two locations representing typical seal communities. order to properly validate the model, it should be done by statistical testing in new region outside of the given ones.
A.3 DWP: Department for Work and Pension
The UK Department of Work and Pensions (DWP) processes a large amount of personal data, including demographics, family circumstances, and financial details, which presents opportunities for large-scale data analysis and concomitant risks to personal privacy. This datathon (Deineha et al., 2021) evaluated the potential for generating synthetic instances from the original data while preserving utility. The DWP provided two datasets of Universal Credit claimant data. Both datasets were synthetic data, representing the original sensitive population, and close in structure to existing DWP datasets.
Appropriateness. (Optimal) Since the challenge regarding the evaluation of the similarity of generated datasets, the provided target data and comparator- generated sets were highly appropriate to this task.
15

---

Readiness. (Functional) Additional cleaning and merging of the target datasets was necessary before processing; secondary datasets required alignment work.
Reliability. (Optimal) Sources of bias in the data collection methodology were identified due to the generation mechanisms for the target dataset. However, these were limited in their effect on the evaluation metrics used.
Sensitivity. (Functional) While the target data did not contain real personal identifiers, it was still considered confidential and potential vector for data breaches.
Sufficiency. (Optimal) A large number of available target data rows exceeded the scope of previous challenges in synthetic data generation.
A.4 Dementia Research Institute and DEMON Network: Predicting Functional Relationship between DNA Sequence and the Epigenetic State
This datathon aimed to improve regulatory genomic predictions to understand the impact of genetic variants on gene expression and regulation disease-relevant cell types, with a focus on dementias such as Alzheimer's disease, and potential implications for drug target development. (hhuycharPn et al., 2022). Four different datasets were provided, containing information about the DNA sequence and the associated chromatin states.
Appropriateness. (Optimal) The data provided contained the features and labels of interest. For example, "... the outputs are always trained to predict one of two epigenetic signals ... in 3 cell types".
Readiness. (Functional) Data required merging and aggregating between datasets to be able to extract insights "Most of the samples were provided with associated DNase-seq data [...] some cell lines [...] were provided with ATAC-seq data instead. Although both assays are broadly used to"
Reliability. (Functional) The datasets used were open-source and annotated by domain experts. Some examples of the datasets used were: EpiMap" and Human Genome 38 There were some gaps in the data which the authors were not able to account for. For example, "All profile data included occasional not number (NaN) values."
Sensitivity. (Optimal) The datasets used were open-source and was mapped to tier 0. Sufficiency. (Optimal) 4 well-labelled and large genomic datasets were used for the challenge.
A.5
Automating Perfusion Assessment Sublingual Microcirculation in Critical Illness
This challenge sought to determine if validated measure of microcirculatory perfusion, specifically the microcireulatory flow index, can be directly predicted from a video sequence obtained through darkfield microscopy. The goal is enable automatic, real-time analysis of the videos, reducing the labour-intensive manual analysis process and facilitating the integration of microcirculatory targets into clinical trials for better patient outcomes (Tomlinson et al., 2022). The data comprised 800 grayscale videos of different lengths and the quality obtained via DFM from 52 patients monitored over four days
Appropriateness. (Optimal) The obtained data and labeled was specifically designed for the chal- lenge. "These videos have been manually analysed to obtain perfusion parameters per short clip, labelling each data point..."
Readiness. (Functional) The dataset was ready but required minor renaming, merging and feature extraction.
Reliability. (Functional) There were cases in which a video had no perfusion parameters and vice versa. "some mismatches were identified ... cases in which video had no perfusion parameters and vice versa".
Sensitivity. (Functional) The data was commercially sensitive due containing videos from patients and was mapped to a Tier
'nttp://compioo.r .it. moit..dd//epimsa. hhttp:://ww www.//wwwww......ccm/..........t nlm, nih, sww/aassmb////FFF.0000000..
16

---

Sufficiency. (Developing) Since the data relied on manual labeling, the total amount of data was not sufficient to provide statistical validity for the method." "[... Jdataset is comprised of smaller number[...]"
A.6 Entale: Recommendation Systems for Podcast Discovery
The datathon aimed to develop methods for capturing relationships between podcasts, incorporating information about the content discussed within them and using this information to produce podcast recommendations (Chan et al., 2021). The data provided included user data (episode and show listened by the users) and podcast data (episode transcripts and show descriptions).
analyse podcast episode content and to produce episode recommendations, user history and podcast data (episode transcripts, episode and show meta-data) is highly appropriate.
Appropriateness. (Optimal) Since the challenge
Readiness. (Functional) Merging and cleaning the two datasets was necessary before analysis.
Reliability. (Fumctional) Bias was induced by a small overlap between datasets, and since the participants were tasked with producing recommender system, they had some measurement bias when evaluating their methodology since they should be evaluated with new data.
Sensitivity. (Functional) Data provided by Entale, and so there commercial risk for the data.
Sufficiency. (Functional) The two datasets provided were large and enabled users to tackle the research question, but the overlap between user episodes listened to, and episode transcripts was only 10% making it challenging to incorporate user data with detailed episode content fully,
A.7 Odin Vision: Exploring AI Supported Decision Making for Early-Stage Diagnosis of Colorectal Cancer
Odin Vision is a UK company that has developed Al technology for detecting and characterising bowel cancer. The aim of this datathon was to explore methods that enhance the explainability of Odin-Visioe Vision's 's current machine learning models to aid clinical decision-making by (i) adding measure of predictive uncertainty along with class prediction, (ii) making the classifications more understandable to the clinicians, and (iii) exploring methods that can automatically learn representations of features using generative models (Almarzouq et al., 2021). The provided data collated three public datasets of colorectal polyp images, obtained by merging data from white light endoscopy and narrow- www.wwbad-iie hand-inaging light sources.
Appropriateness. (Fumctional) A total of three image datasets were employed. Two of them were publicly available and contained both labelled and unlabelled images. The third one was high- quality, non-public dataset that was not made available to the datathon participants but was employed by Odin Vision to train the models developed during the datathon.
Readiness.
(Developing) Many of the images from the publicly available datasets were of low quality due to blur and the presence of reflective spots. A considerable number of images had to be either removed or pre-processed in order to alleviate this issue.
Reliability. (Developing) In addition the positive-case bias resulting from unbalanced classes (see section on Sufficiency), the datathon participants concluded that the best approach was to train the models predominantly Odin Vision's 's higher quality, non-public dataset, and to a test set from the publicly available images. As the train and test sets were not drawn from the same distribution, the reported quantitative results might be affected by the distributional shift between the train and test datasets.
Sensitivity. (Optimal) The datathon participants only had access to open-source datasets and not to Odin Vision's non-public data, which made this Tier 0 challenge.
Sufficiency. (Functional) Due to the nature of the datasets (real-world medical images), these were relatively small, with even fewer positive labelled data, which resulted in an insufficient amount of data for learning algorithms and drawing statistically significant answers.
17


Reviews:
[{"review_id": "wiaq2TVLMP", "reviewer": "Reviewer_s7kZ", "summary": "", "questions": "", "limitations": "The focus of the paper is on organizing more effective and beneficial hackathons. I don’t consider an analysis of negative impacts necessary. \n\nThe recommendations section serves as helpful practical guidance for appropriate and effective use of the framework. \n", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The framework is straightforward and actionable.\n\nThe paper is anchored in real-world insights from the DSG. The 10 case studies are also publicly documented.\n\nThe problem of building datasets that empower time-bound, impactful problem solving in the context of hackathons is important.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ikEuBJD44Y", "reviewer": "Reviewer_CNLi", "summary": "", "questions": "", "limitations": "This paper leaves out any substantial ethical issues that arise in datathons, which is a fundamental and ongoing issue for organizers. The authors just note that \"they recommend all datathon projects are assessed for data protection and ethical considerations,[but] these processes are outside of the scope of this paper.\" As there is no details given as to why they are outside the scope, it leaves the rest of the framework with a considerable gap, and provides no guidance in this area. As the aim of this paper is to \"improve the handling of data for organisations prior to datathon events\", in order to be accepted it would need to have a detailed section around good data stewardship, including issues of privacy, ethics, and so on. ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper's strengths are in its practicality and case studies. It addresses real-world challenges faced during datathons, offering a high level of applicability for event organizers with concrete guidelines. Moreover, the authors introduce their own framework that assesses data from several dimensions such as appropriateness, readiness, reliability, sensitivity, and sufficiency. This multidimensional approach presents a practical path for other data hackathon organizaers. Lastly, the paper provides detailed analysis of ten datathon use cases, showing how this theoretical framework can be applied in real-world scenarios.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dka5MJta39", "reviewer": "Reviewer_ammT", "summary": "", "questions": "", "limitations": "It would be good to apply the framework to a datathon outside of the Alan Turing Institute (though I see no technical limitations for doing so).", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors have a unique perspective from organising as many datathons. They provide a useful framework but also use it to analyze ten datathons, which should be valuable for new organizers of datathons. The impact and relevance to the broader research community is hard to estimate: most people will not organize datathons (or maybe not even attend them), but improved quality of datathons might lead to better datasets (or understanding thereof) which have the potential to be useful in research (in addition to the inherent value the datathons provide for the challenge owner and participants).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Un76nrSPes", "reviewer": "Reviewer_ijbY", "summary": "", "questions": "", "limitations": "There is not any discussion of potential negative societal impact in the paper. That being said, I do not personally see any negative societal impact for this work. It would be nice to see the authors provide some discussion of the limitations of this framework somewhere in the paper.\n\nAlso, notably, the NeurIPS paper checklist appears to be missing from this submission.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Studying data hackathons is a useful avenue of work in which I haven’t come across much existing literature. In my experience, it seems like data hackathons would probably benefit from having some kind of standardization/framework to build from and fall back on when organizing.\n- In particular, this paper seems to be (to the best of my knowledge) novel in that they focus on the pre-hackathon organization and preparation.\n- It’s clear the authors have much experience in data hackathons. The case studies are interesting to read about.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UxnmduxMnM", "reviewer": "Reviewer_oVgp", "summary": "", "questions": "", "limitations": "Some limitations of the work are mentioned, but they should be discussed in more detail and their impact should be made more clear.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The literature review is very well written and clear, and covers a large range of relevant fields.\nWhile the proposed method may only be relevant to limited communities, the dimensions regarding data quality evaluation could spark the interest of a much larger community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a framework for preparing datasets for use in datathons. The authors distill these guidelines from years of datathon organization experience. The framework is clear, actionable, somewhat limited in scope as noted by several reviewers, though the proposed changes from the authors make progress towards addressing these limitations. 

Pros:
- Introduces a clear, actionable framework for preparing/selecting datasets for datathons
- Distilled from real-world insights
- The framework would be a useful resources for any organizations or individuals striving to organize data science hackathons or similar events

Cons:
- Limited in scope: does not include recommendations re: datathon ethics, impacts of datathons etc (addressed in rebuttal)
- Limited empirical evidence on whether the proposed guidelines/methods would improve datathons (out of scope for this work)

Major issues have been addressed by the authors and any remaining comments can be addressed in a camera-ready revision.
Decision: Accept (Poster)