SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Carlos Mougan
The Alan Turing Institute University of Southampton
Richard Plant
The Alan Turing Institute
Edinburgh Napier University
Clare Teng
Marya Bazzi
The Alan Turing Institute University of Warwick
Alvaro Cabrejas Egea
Fujitsu Research of Europe
Ryan Sze-Yin Chan
David Salvador Jasin
Martin Stoffel
Kirstie Jane Whitaker
Jules Manser
The Alan Turing Institute jmanser@turing.ac.uk
The rise of datathons, also known as data or data science hackathons, has provided a platform to collaborate, learn, and innovate in a short timeframe. Despite their significant potential benefits, organizations often struggle to effectively work with data due to a lack of clear guidelines and best practices for potential issues that might arise. Drawing on our own experiences and insights from organizing  $\geq 80$  datathon challenges with  $\geq 60$  partnership organizations since 2016, we provide guidelines and recommendations that serve as a resource for organizers to navigate the data-related complexities of datathons. We apply our proposed framework to 10 case studies.
Datathons or data hackathons, loosely defined as data or data science-centric hackathons (Anslow et al., 2016; Chau and Gerber, 2023), have become increasingly popular in recent years, providing a platform for participants and organisations to collaborate, innovate, and learn in the area of data science over a short timeframe. This paper focuses on data-related challenges in datathons, including: What does it mean for data to be "appropriate" for a datathon? How much data is "enough" data? How can we identify, categorise, and use "sensitive" data? We aim to offer a set of guidelines and recommendations to prepare different types of data for datathons drawn from extensive experience in datathon organisation.
Working with and preparing data for a datathon can be challenging due to problem-specificity and the short time scales available to prepare the data for analysis and answer a specific set of questions. In research projects, there is significantly more time and room to explore and adapt the data and the questions. Other factors include the type of datathon, its potential breadth (e.g., data which is publicly available and has been previously analysed by researchers (Kitsios and Kamariotou, 2019; Concilio et al., 2017)), datasets from a company with no or limited prior analysis (Nolte et al., 2018; Nolte, 2019), or working with multiple data sources and formats. To effectively incorporate (varied) data, there is a need to have standardised processes and definitions that can account for the complexity and challenges that arise during the datathon selection and preparation process. In particular, our
paper differentiates and focuses on the following data-related dimensions: appropriateness, readiness, reliability, sensitivity and sufficiency. Our main contributions are as follows:
- Introduce a framework to analyse the data intended for datathons from several dimensions: appropriateness, readiness, reliability, sensitivity, and sufficiency.
- Share insights and experiences from organising datathons with external organisations, mapping 10 data challenges and organisations to the proposed framework.
- Provide recommendations and best practices for organisers to effectively select, prepare, and categorise data for datathon challenges.
Our paper is organised as follows. In section 2, we overview various aspects of hackathons and datathons, data assessment terminology and frameworks. In section 3, we describe Data Study Groups (DSGs), which form the methodology for this paper. In section 4, we introduce our data assessment framework, which we map to existing terminology in the literature, where appropriate. Section 5 details ten datathon use cases and shows how these can be mapped to the proposed framework. In particular, we map the three most recently completed DSGs to the framework we propose. We offer some general recommendations on improving data quality in datathons as an event organiser in section 6. Finally, section 7 provides a summary of the contribution.
2.1 Hackathons & Datathons
Previous research on hackathons has provided valuable insights that can enhance one's understanding of datathons and their effects on organisations and participants. Extensive literature reviews include Chau and Gerber (2023); Olesen and Halskov (2020). Chau and Gerber (2023) found that hackathon and datathon research tends to focus on four core areas: the event purpose, the event format, the processes undertaken and experienced by stakeholders, and the event outcomes (with some papers having more than one focus). They also add that most publications about processes focus on what happens during the event (e.g., intra-team dynamics in collaboration and project brainstorming) rather than before or after.
As mentioned in the introduction, datathons are data science or data-centric hackathons. The work in Anslow et al. (2016) explores how these can enhance data science curriculum (e.g., through improved data science, communication, and community engagement skills) and that in Kuter and Wedrychowicz (2020) on how to run a datathon with very limited resources. Aboab et al. (2016) focus on healthcare and emphasize the cross-disciplinary aspects of datathons, claiming they help address poor study design, paucity of data, and improve analytical rigour through increased transparency. Luo et al. (2021) conducted a virtual datathon focusing on the early stages of the Covid-19 pandemic, highlighting the importance of interdisciplinary and diverse team compositions. Piza et al. (2018) found a significant association between positive teamwork behaviors and affective learning in a healthcare-focused datathon, pointing out effective leadership as the key factor.
We propose to add to existing literature that explores challenges and best practices in datathons, with a focus on data practices. More specifically, and following the terminology in Chau and Gerber (2023), we focus on data as a "technical project material" in datathon formats, and examine key challenges that can arise in data evaluation and preparation during the "pre-hackathon" organisation process. To the best of our knowledge, prior research has not focused on data-related challenges in the specific context of datathons.
2.2 Data Analysis Dimensions
Previous research on data quality evaluation has focused on general description frameworks and metrics, rather than datathon context-specific development of methods (Wang and Strong, 1996; Zaveri et al., 2016; Juran et al., 1974; Schlegel et al., 2020).
Early work in the area (Wang and Strong, 1996) presented a conceptual framework for assessing "data quality" (i.e. assessing issues that can affect the potentiality of the applications that use the data), which included 15 dimensions grouped under four categories: intrinsic; contextual; representational and system-supported. Indeed, "data quality" is commonly viewed as a multi-dimensional construct
that defines "fitness of use" and may depend on various factors (Zaveri et al., 2016; Juran et al., 1974), such as believability, accuracy, completeness, relevancy, objectivity, accessibility, amongst several others (Wang and Strong, 1996). The context-specific methodology of Schlegel et al. (2020) focuses on defect diagnosis and prediction and addresses the following five dimensions, which they argued were the most important for assessing "data quality" (which they rather refer to as "data suitability"): relevancy; completeness; appropriate amount (of data); accessibility and interpretability. Their study focused on developing guidance for companies, using various metrics and evaluation levels, on how to gain insight into the suitability of the data collected in the context of defect diagnosis and prediction. Our work extends and builds upon these previous frameworks and methodologies by applying them specifically to the challenges and requirements of datathons. In particular, we consider the five following dimensions in the context of data-driven problem-solving during the datathon events: data appropriateness, readiness, reliability, sensitivity and sufficiency. In what follows, we link our proposed dimensions to existing terminology in Wang and Strong (1996) and Schlegel et al. (2020) where possible, and mention other related research in the field.
Appropriateness: This dimension considers the question "Is the data relevant to the data science questions posed (i.e., problem-specific)?", and closely aligns with "relevancy" (Wang and Strong, 1996; Schlegel et al., 2020). Examples of context-specific uses of relevancy include Daly (2006), which provide a comprehensive set of guidelines for analysing specific climate datasets. In their guidelines, the authors outline the relevancy of specific climate factors for different types of analysis, and Schlegel et al. (2020) provides domain-specific concepts to assess data suitability along the production chain for defect prediction.
Readiness: This dimension considers the questions "Is the data analysis-ready?", and includes "completeness" and "accessibility" (Wang and Strong, 1996; Schlegel et al., 2020). Technology readiness levels (Dunbar, 2017; Hirshorn et al., 2017) was developed in the 1970s as a standardized technology maturity assessment tool in the aerospace engineering sector. NASA developed a scale with multiple degrees which has increased from 7, in the early years (Sadin et al., 1989), up to 15 degrees in the latest reviews (Olechowski et al., 2020).
Similarly, Lawrence (2017) proposed the use of data readiness levels, giving a rough outline of the stages of data preparedness and speculating on how formalization of these levels into a common language could facilitate project management. Building on this previous work, we specify four data readiness levels but specifically within the context of datathons challenges.
Afzal et al. (2021) outlines a recommended task-agnostic 'Data Readiness Report' template for a project's lifecycle, which includes different stakeholders involved such as final contributions by data scientists. Castelijns et al. (2020) identifies 3 levels of data readiness and introduces PyWash, which is a semi-automatic data cleaning and processing Python package. Their levels also include a series of steps which includes feature processing and cleaning. In their works, the use of 'data readiness' refers to the end-to-end lifecycle of a data science project, including steps typically taken during a datathon and after. Other works focus specifically on 'big data' readiness and in country-specific ecosystems which may not be easily extendable (Austin, 2018; Joubert et al., 2023; Hu et al., 2016; Zijlstra et al., 2017; Romijn, 2014). Since datathons are not confined to a single use-case, we believe it would be more appropriate to create a framework that can work across different applications.
Reliability: This dimension considers the question "How biased is the data?", and includes "objectivity" (Wang and Strong, 1996). The reliability of data is crucial for AI applications as it directly impacts the accuracy and fairness of the outcomes. Biased data which does not represent the problem fairly returns unreliable results (Ntoutsi et al., 2020). Drawing from established frameworks such as Suresh and Guttag (2019) and Ntoutsi et al. (2020), we identify several potential sources of bias that can undermine the reliability of the data. Selected examples may include: (i) historical bias, which arises from past societal or systemic inequalities; (ii) representation bias, where certain groups or perspectives are underrepresented in the data; (iii) measurement bias, which stems from flawed or subjective data collection methods; and (iv) aggregation bias, which occurs when data is combined or summarised in a way that does not accurately represent each sub-group within the wider population.
Sensitivity: This dimension considers the questions "How sensitive is the data?" We build on existing work from Arenas et al. (2019) by incorporating their data sensitivity analysis categorisation to our framework. They propose a defined tier-based system for sensitive data management in the cloud, which contain: tier 0, datasets with no personal information or data which carries legal, reputational, or political risks; tier 1, data that can be analysed on personal devices or data that can be published in
the future without significant risks, such as anonymised or synthetic information; tier 2, data that is not linked directly to personal information, but may derive from it through synthetic generation or pseudonymisation; tier 3, data that presents a risk to personal safety, health, or security on disclosure. This may include pseudonymous or generated data with weak confidence in the anonymisation mechanism or commercially and legally confidential material; tier 4, personal information that poses a substantial threat to personal safety, commercial, or national security and is likely to be subject to attack.
The data used in the study may also need to adhere to various regulations, including but not limited to the Data Protection Act (Government of the United Kingdom, 2018), GDPR (European Commission, 2016), and specific regulations governing special categories of data (Information Commissioner's Office (ICO), 2021; European Parliament, 2016). Additionally, different countries or sectors may have their own specific regulations in place. It is important to note that the application of these laws is evaluated on a case-by-case basis, and this paper does not provide specific guidance on how to comply with them.
Sufficiency: This dimension considers the question "How much data is enough data for a given datathon?", and includes "appropriate amount" (Wang and Strong, 1996; Schlegel et al., 2020). This question is highly problem and method-specific, and has been formulated in many research fields. In an early study, Lauer (1995) explored a series of issues in analysing data requirements for statistical Natural Language Processing (NLP) systems and suggested the first steps toward a theory of data requirements. Questions such as (i) "Given a limited amount of training data, which methods are more likely to be accurate?", or (ii) "Given a particular method, when will acquiring further data stop to improve predictive accuracy?", were addressed. In intelligent vehicles, Wang et al. (2017) proposed a general method to determine the appropriate data model driver behavior from a statistical perspective. They point out that this question has not been fully explored in their field of research, but it is highly relevant given that insufficient data leads to inaccurate models, while excessive data can result in high costs and a waste of resources. Cho et al. (2015) presented a general methodology for determining the size of the training dataset necessary to achieve a certain classification accuracy in medical image deep learning systems. They employed the learning curve method (Figueroa et al., 2012), which models classification performance as a function of the training sample size. They pointed out that, in medical image deep learning, data scarcity is a relevant problem. Developing a methodology to estimate how large a training sample needs to be in order to achieve a target accuracy is therefore highly relevant. Since datathons include a wide range of applications and approaches, we adopt a different take on "sufficiency" and propose general guidelines based on experiences in prior datathons. In our case studies, we make case-specific observations (see section 5).
3 Data Study Groups
Data Study Groups (DSGs) are an award-winning<sup>1</sup> collaborative datathon event organised by The Alan Turing Institute, the UK's national institute for data science and artificial intelligence. Each DSGs consist of multiple datathons (or "challenges"), typically 3-4, that are worked on collaboratively by a single team (rather than multiple teams competing with each other). The aim of DSGs is to provide opportunities for organisations and participants from academia and industry to work together to solve real-world challenges using data science and ML methodologies. The DSGs are managed and prepared by a specialised internal team of event organisers and interdisciplinary academic support staff.
The events typically run for a week, during which 8-12 participants work in teams to explore and investigate the challenges. The events are overseen by experienced researchers that act as "principal investigators" (PI) and "challenge owners" as subject-matter experts. Both provide technical guidance and support for the teams during the week. DSGs have tackled various challenges across different sectors, such as healthcare, finance, and transportation. Figure 1 shows the breadth of challenges undertaken, with  $\geq 69$  challenge owners spanning  $\geq 77$  datathons across different domains. Note that we have carried out over 80 datathons, but not all datathons have available meta-data for analysis.
During the datathon, a technical report is written by the participants, and edited by the PI after the event. The report collates the work that was carried out by the datathon team and articulates successes, challenges, and recommendations for future analysis directions. The report is not intended to undergo
<sup>1</sup> https://www.praxisauril.org.uk/news-policy/blogs/turing-data-study-groups-ke-award-winners
Figure 1: Left: Bar chart showing the distribution of sectors from the challenge owners ( $n = 69$ ). Right: Distribution of datathon topics ( $N = 77$ ) for the Data Study Groups (DSGs).
Figure 2: Summary data assessment matrix for the five data dimensions proposed in this paper.
a traditional academic peer-review process to assess novelty. Rather, members of the DSG team at the Alan Turing Institute review the report for scientific rigour and clarity before publication.
The internal peer-review process is aligned with considerations of responsible research and innovation. DSG challenges hosted at the Turing are all assessed for ethical considerations following guidance published by the Public Policy team (Leslie, 2019), therefore, ethics is not an additional data dimension in our process, it is a separate step that is comparable in size to data assessment. Although we recommend that all datathon projects be assessed for data protection and ethical considerations, these processes are outside this paper's scope, as it varies by country, organising institution and partners needing for an in-depth discussion even within fields to be useful. Our data assessment framework complements institutional policies and can fit within data governance laws across different countries.
4 Data Assessment Matrix
Insufficient	Developing	Functional	Optimal
Appropriateness	Data unrelated to the research question(s).	Target variables are inappropriate for the research question.	Features aren't entirely appropriate for the research question.	Clear relation between features and target variables.
Readiness	No data collected or collection methodology.	Some collected data or clear collection methodology in place.	Data collected, but requires some merging and unifying.	Data collected and cleaned.
Reliability	Strongly biased data.	Indeterminate source of bias.	Determine source of bias which can be accounted for.	Weak or clear of bias.
Sensitivity	Highly risky sensitive data.	Requires extensive security measures.	Data does not present significant risk but may be confidential.	Limited risks associated with the data, usually publicly available data.
Sufficiency	Prior experience shows low success rate.	Does not meet the level of previous successful events.	Previous events have successfully operated at this level.	Extensive data in excess of previous successful events.
4.1 Appropriateness: Is the data relevant to the questions posed?
Data appropriateness refers to how relevant the data is for the challenge or research questions at hand:
Insufficient: Data is not related to the challenge question or is completely unrelated to the problem at hand. As an example, a datathon team is tasked with predicting consumer behavior for a new product launch but is given data on historical weather patterns instead.
Developing: Data is related to the challenge question, but the target variables needed to answer the research questions are missing. For example, a datathon may seek to predict stock price movements
from historical transaction data but without stock prices, it would be challenging to produce useful results.
Functional: Data is appropriate for the challenge but the features can notably be enhanced with data the partner organisation could provide. For example, a datathon team is given a dataset on website traffic and user behavior, which is relevant to their challenge, but they identify that additional data on customer feedback would be helpful in making more accurate predictions.
Optimal: Data is highly appropriate and relevant to the challenge. A datathon team is given comprehensive datasets which provide all the necessary information for making accurate predictions and developing effective solutions. Appropriate meta-data and collection processes are also provided.
4.2 Readiness: Is the data analysis ready?
Data readiness refers to the degree to which data is prepared and available in a datathon. This includes factors such as the quality and completeness of the data, the availability of relevant metadata, and the ease with which the data can be accessed and analysed. Ease and effectiveness of analysis are closely coupled to effective and thorough data documentation Gebru et al. (2021); Holland et al. (2018), which increase transparency, understandability, and accountability of data and model usage (Mitchell et al., 2019). Data readiness categories are as follows:
Insufficient: No data has been collected yet, and there is no methodology for doing so. For example, if the datathon involves collecting data from sensors, and there isn't a clear view of where the data is being stored or what it actually contains, the data readiness degree would be insufficient.
Developing: Some data has been collected, and a firm methodology is in place for doing so. However, there may still be some uncertainty around its quality and completeness. For example, if the datathon involves analysing customer data from a company, and even though they have access to their data, parts of it are missing, or inconsistent, the data readiness degree would be developing.
Functional: Data is collected, but some merging and unifying still need to be done during the event. The data may require some cleaning and preprocessing, but with minor processing can serve to answer the challenge questions. For example, if the datathon involves analysing public transportation data from different sources, and some data needs to be merged, aggregated, and formatted to fit into the analysis, the data readiness degree would be functional.
Optimal: Data is collated, clean, with no missing gaps, and can potentially be fed to a learning algorithm with minimal processing. The data is also documented with the data dictionary and data cards, such as the ones highlighted in previous research (Gebru et al., 2021; Pushkarna et al., 2022) but in the context of datathons. It is worth noting that dataset documentation in general remains a developing field of study, and datathon-specific recommendations or guidelines are yet to be established. For example, if the datathon involves analysing a well-established dataset, such as the MNIST dataset, the data readiness degree would be optimal.
4.3 Reliability: How biased is the data?
Reliability here refers to the extent to which the data accurately represents the population or phenomenon without systematic data collection errors or distortions.
Insufficient: Data has significant known biases, and conclusions drawn from the analysis will be undermined. For example, in a datathon studying democratic voting patterns, the dataset only includes responses from a small, self-selected group of participants, rendering it insufficient to make accurate inferences about the broader population's voting behavior.