SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Developing:** Data has unknown or indeterminate roots of bias, and hence no firm conclusions can be made using only these data sources. For example, in a datathon analysing economic indicators of different countries, it is suspected that there are geographical and temporal inconsistencies in the data collection. As a result, the dataset is considered to be in the developing stage of reliability.
Functional: Data has known biases that will impact the analysis and can be corrected or accounted for as a work limitation. For example, in a datathon analysing customer feedback surveys, it is known that the surveys were only completed by customers who voluntarily provided feedback, introducing
a self-selection bias. The organisers acknowledge this limitation and proceed with the analysis, highlighting the potential impact of the bias on the conclusions.
Optimal: Data does not suffer from a known source of biases with significance for the conclusions. For example, in a datathon exploring climate change impacts, the dataset comprises meticulously collected climate measurements from a comprehensive network of sensors across multiple regions.
4.4 Sensitivity: Is the data private or confidential?
Data sensitivity refers to the level of confidentiality and privacy associated with specific types of data. It indicates the degree to which the data is considered private, personal, or valuable, requiring enhanced protection. Ensuring these safeguards may increase the difficulty of hosting a datathon. Therefore, we map the sensitivity tiers described by Arenas et al. (2019) to our analysis framework.
Insufficient (Tier 4): Highly sensitive data presents significant threats, making hosting a datathon impractical. Due to potential legal and personal risks, datathons are not supported at this tier. For instance, handling datasets with commercial or national security sensitivities. This aligns with the UK government's "secret" classification (UK Cabinet Office, 2018). At Tier 4, the risk of malicious actors infiltrating the datathon team becomes notable.
Developing (Tier 3): Careful consideration and extensive security measures would be required prior to hosting a datathon in this tier. Datathon participants would be restricted in their use of tools, location and network connections, increasing system friction. For example, in a medical context, a dataset containing highly sensitive patient health records, including names, addresses, and detailed medical histories, poses a substantial threat
Functional (Tier 2): Data in this tier does not present significant personal, legal or commercial risk. However, sufficient measures must be taken to mitigate processing risks, such as  $(i)$  data de-annotationisation,  $(ii)$  competitive data leakage  $(iii)$  or inadvertent disclosure. These measures increase the difficulty of participants interacting with the data. For example, an educational dataset with anonymized student records that still hold the risk of re-identification. Hosting a datathon using this data would necessitate robust security measures to prevent any potential breach of student privacy and unauthorized access to sensitive academic information.
Optimal (Tier 0/1): Since risks associated with this data are limited, hosting datathons in virtual environments is usually unnecessary. Free data access to the participants reduces system friction. A dataset containing weather records from a public weather station that doesn't include any personal or sensitive information would fall under this tier.
More details, including a discussion on the classification process, as well as the technical requirements to build such environments can be found in (Arenas et al., 2019).
4.5 Sufficiency: How much data is enough data for a given datathon?
Data sufficiency refers to the quantity of available data to investigate the challenge questions given a proposed approach. In addition to case-by-case (e.g., depending on the approach and the question) considerations, we propose below an assessment based on previous experiences of similar challenges. In section 5, which is problem-specific, we make case-specific comments on data sufficiency.
Insufficient: Prior experiences have shown that working with similar data quantities on projects with similar objectives hinders the overall success and outcomes of the datathon. For example, in a medical research datathon aiming to predict patient outcomes, the provided dataset does not include information about clinical trials.
Developing: Prior experience has shown that the quantity of data available is not substantial enough to address the research question comprehensively. This quantity of data raises questions about its ability to offer statistical evidence and comprehensive insights into the challenge questions, unlike the insufficient category, data may be present but not adequate for comprehensive insights. For example, a dataset containing a few time-series signals in an industrial failure detection manufacturing datathon not allowing to distinguish between normal and failure modes (Bocincova et al., 2022).
Functional: Although not surpassing previous successful datathons, prior experience has shown that the quantity of data is sufficient to provide useful insights and statistical evidence into the challenge
questions. Any insufficiency can be accounted for as a work limitation and discussed in future work. For example, in a datathon focused on developing an automated system for classifying sea ice and mapping seals to aid ecological monitoring, the geospatial data provided covered only two focal areas, which limits the potential generalizability of the developed system to broader regions (Acharya et al., 2020).
Optimal: This category represents an amount of data which surpasses previous successful datathons. The dataset(s) is extensive (potentially more than what was proposed initially), offering a rich source of information for participants to explore and leverage as they tackle the challenge questions. For example, in a datathon that aims to predict the functional relationship between DNA sequence and the epigenetic state, having access to comprehensive and well-labelled genomic datasets (Phuycharoen et al., 2022).
5 Case Studies
We evaluated the quality of data in past DSGs by referring to the final reports available on The Alan Turing Institute's website<sup>2</sup> and show an aggregate summary of the results in Figure 3. It is important to acknowledge a publication bias in these reports, as only those that meet minimum quality criteria determined by the reviewing team are made available. Moreover, the reports from early DSGs might not have been published due to problems ranging from unresolved sensitivity issues to misaligned expectations between organisers and challenge owners. Consequently, our focus was primarily on the later versions of the DSG.
This section presents the ten most recent datathons case studies, selected from over 80 datathons, in Figure 3 we provide a summary of the results of the data assessment of the case studies. These case studies have data, challenge descriptions, and limitations extracted from the publicly available DSG Reports. They are divided between the main body of the report and the appendix.
Figure 3: Report count data assessment classification of the last 10 DSG reports
5.1 Cefas: Centre for Environment, Fisheries and Aquaculture Science
The datathon aimed to address the challenge of automating the classification of plankton species and detritus in millions of images collected by the Cefas Endeavour research vessel's Plankton Imager system, enabling efficient analysis and contributing to the understanding and conservation of marine ecosystems (Asthana et al., 2022).
$^{2}$ https://www.turing.ac.uk/collaborate-turing/data-study-groups
Appropriateness. (Optimal) Plankton images and their corresponding labels were present in the data: "expert manual classification (labels) allowed challenge participants to verify the accuracy of the automated classification methods explored."
Readiness. (Functional) Images had faulty labels: “[...] we discovered that a number of the images labelled as detritus were in fact empty.”
Reliability. (Optimal) When writing the challenge proposal, we identified a potential bias in the data collection method: all images were collected by the same imager system on the same vessel. However, Cefas did not assess this risk as substantial, which was corroborated by the project outcome.
Sensitivity. (Optimal) The data is open access, collected by the Plankton Imager<sup>3</sup>.
Sufficiency. (Functional) Some species of plankton were under-represented, making it an imbalanced problem which needed to be accounted for in the analysis (e.g. using data augmentation).
5.2 The University of Sheffield Advanced Manufacturing Research Centre: Multi-sensor based Intelligent Machining Process Monitoring
This challenge aimed to enhance process monitoring in modern manufacturing using machine learning techniques to analyse multiple sensor measurements (Bocincova et al., 2022). Two datasets were provided for this challenge containing time-series process signals of the machine operating in both normal and failure modes. The datasets were recorded during separate machining trials.
Appropriateness. (Developing) The target variables needed for the challenge were not present in the data: "the provided dataset does not contain any data describing the transition period between the normal operation mode and failure modes"
Readiness. (Functional) The data requires aggregation before being used: "datasets were collected at extremely high sampling frequencies ... resulted in excessively big datasets"
Reliability. (Functional) The data experiments were not recorded in the same instance so although measures were taken to ensure that they were as identical as possible, there was a known bias that could be accounted for: "The two provided datasets were recorded during two separate machining trials [...] However, a number of identical components were machined in each trial to act as repeats for the dataset."
Sensitivity. (Functional) The data was commercially sensitive and was mapped as Tier 2.
Sufficiency. (Developing) Not enough data for the analysis: "There were still not enough multivariate time series produced by different runs of the experiment to accurately use image classification."
5.3 CityMaaS: Making Travel for People in Cities Accessible through Prediction and Personalisation
The aim of this datathon was to develop a platform that provides reliable information on the accessibility of destinations (Aragones et al., 2021). In particular, participants were tasked to  $(i)$  predict the accessibility of points of interest (POIs) in a city;  $(ii)$  to develop a personalised route-planning algorithm with accessibility constraints and for different "persona" types of their users. For  $(i)$ , CityMaaS provided manually enhanced point data from OpenStreetMap (OSM) (OpenStreetMap contributors, 2021) and for  $(ii)$  information on (synthetic) potential end-user personas were provided (i.e. place of residence, civil status, profession, income, conditions, etc.).
Appropriateness. (Developing) The enhanced OSM dataset provided for the task  $(i)$  was appropriate for predicting POI accessibility. However, for task  $(ii)$ , participants were only given information about the profile of potential users.
Readiness. (Functional) Significant data cleaning was necessary before the analysis was performed; for example, participants noted the need to "homogenise" the POI data to avoid naming errors in the dataset.
<sup>3</sup>https://data.cefas.co.uk/view/20507
Reliability. (Developing) The dataset for POI prediction had a class imbalance with  $73\%$  of POIs being labelled as accessible. POI data was taken from OSM where users are allowed to input tags leading to possible errors and annotation bias.
Sensitivity. (Functional) There was commercial sensitivity for CityMaaS in providing the enhanced OSM dataset provided and the (synthetic) user profiles.
Sufficiency. (Developing) In particular, for the routing task, no routing data was provided. Participants searched for additional open-source data sources to supplement the data provided by CityMaaS.
6 Recommendations
Recommendations for effectively working with data in datathons can be organised based on the preliminary stages of the event. The following suggestions draw from our investigation of the case studies and are supported by learning from previous datathons. Given that our paper aims to suggest a small number of dimensions and recommendations that can be applied to a broad range of datathons, the recommendations below are general and broad-purpose. Specific recommendations for moving from one tier to another or on what constitutes "good enough" will need to be assessed case-by-case, as illustrated in section 5.
Preparatory phase: Prior to the datathon, one important recommendation is for the event organisers to engage the challenge owners actively. Our framework can help focus discussions with challenge owners on dimensions where scores are below "functional". Their understanding of the problem and domain expertise can greatly improve the data, and early collaboration helps align the objectives and expectations on both sides, increasing the likelihood of a fruitful event. For example, in our Strathclyde challenge (Amaizu et al., 2022), the initial proposal was misaligned in terms of readiness, and the project focus was shifted from the participants executing simulations at the datathon to the challenge owners providing them so that the participants could focus on pattern-finding. Conversely, the Cefas challenge described in section 5.1 achieved greater success due to significant involvement from the challenge owners.
Refinement phase: As the datathon approaches, it is beneficial to do sanity checks on data readiness and consider changing the challenge questions based on input from the PI. Again, our framework can help focus the efforts of PIs on dimensions where scores are below "functional", and they can consider the viability of the datathon on a case-by-case basis. This phase can help to refine the problem formulation and avoid appropriateness issues, ensuring alignment with the goals of the datathon. For example, by earlier involvement of a PI, we could have detected the lack of appropriate data or the insufficiency of the multivariate series in the Sheffield challenge in section 5.2.
At the datathon: During the datathon itself, the PI will be present to help guide participants, having considered and reflected on any data issues and mitigants prior to the datathon. It is also essential to leverage the expertise and feedback of the participating data scientists. Their insights and perspectives can serve as valuable expert advice in identifying areas for improvement across the data quality dimensions. By actively seeking and incorporating participant feedback, organisers and participants can make small adjustments that enhance data utility, facilitating more effective and impactful analyses during and in future events. For example, in the CityMaas challenge of section 5.3, participants found data reliability issues (inconsistent labelling by users), or in Get Bristol Moving (Mougan et al., 2020) alternative sources of open-source data were found that allowed decreasing the sensitivity tier.
In this paper, we have analysed data in the context of datathons along five key dimensions: appropriateness, readiness, reliability, sensitivity and sufficiency. We then mapped 10 case studies from Data Study Groups to our framework and provided a set of recommendations based on the lessons learned. By doing so, we hope to improve the handling of data for organisations prior to datathon events.
Our proposed qualitative analysis provides a degree of data status across several perspectives; these degrees can be adapted or extended, similar to the Technology Readiness Levels provided by NASA (Sadin et al., 1989), which have been extended through time and further work.
The DSG team wish to thank the past contributions of Dr Merve Alanyali, Dr Alex Bird, Rebecca Blower-Harris, Dr Ayman Boustati, Xander Brouwer, Dr Zhenzheng Hu, Dr Chanuki Illushka Seresinhe, Catherine Lawrence, Dr Zhangdaihong Liu, Dr Bilal Mateen, Frank Murphy, Daisy Parry, Katrina Payne, and Dr Mahlet Zimeta. With special thanks to Prof Sebastian Vollmer for bringing the concept to us and starting DSGs at the Turing back in 2016, and Dr Franz Kiraly for his work and input in the early years. We also wish to thank all past Challenge Owners, DSG PIs and participants for their invaluable contribution and for making DSGs possible.
Carlos Mougan was funded by the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie Actions (grant agreement number 860630) for the project: "NoBIAS - Artificial Intelligence without Bias".
Aboab, J., Celi, L. A., Charlton, P., Feng, M., Ghassemi, M., Marshall, D. C., Mayaud, L., Naumann, T., McCague, N., Paik, K. E., Pollard, T. J., Resche-Rigon, M., Salciccioli, J. D., and Stone, D. J. (2016). A datathon model to support cross-disciplinary collaboration. Science Translational Medicine, 8(333):333ps8-333ps8.
Acharya, A., Laws, A., Tegho, C., Miller, E., Shone, F., Rincon, L. M. G., Baker, L., Deecke, L., Gill, P., Hurst, T., Pollington, T., and Sanchez-Silva, V. (2020). Seals from space: automated antarctic ecosystem monitoring via high-resolution satellite imagery. Technical report, The Alan Turing Institute, University of Exeter, Calipsa, University of Cambridge, Arup London, University of Warwick, University of Glasgow, University of Edinburgh, British Antarctic Survey, Heriot Watt University.
Afzal, S., Rajmohan, C., Kesarwani, M., Mehta, S., and Patel, H. (2021). Data Readiness Report. In 2021 IEEE International Conference on Smart Data Services (SMDS), pages 42-51.
Almarzouq, B., Cabrera-Arnau, C., Ditter, E.-J., Duckworth, P., Jewsbury, R., Kazmierski, M., Li, Z., Lin, S., Lines, D., Mun, T. S. H., Yallup, D., and Zanisi, L. (2021). Odin vision: Exploring ai supported decision-making for early-stage diagnosis of colorectal cancer. Technical report, The Alan Turing Institute.
Amaizu, M., Paul, O., Francesco, D. D., Nevin, J., Dirik, A., Sekkat, C., Matavalam, A. R. R., Papadopoulos, P., Tzelepis, D., Nakas, G., and Koukoura, S. (2022). Data study group final report: University of strathclyde and supergen energy networks hub. Technical report, The Alan Turing Institute.
Anslow, C., Brosz, J., Maurer, F., and Boyes, M. (2016). Datathons: an experience report of data hackathons for data science education. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pages 615-620. ACM.
Aragones, M., Bedogni, L., Deshpande, A., Gieschen, A., Huang, Z., McIntyre, F., Nawfee, S. M., Roumpani, F., Saad, A., and Vonnak, R. (2021). Citymaas: Making travel for people in cities accessible through prediction and personalisation. Technical report, The Alan Turing Institute.
Arenas, D., Atkins, J., Austin, C., Beavan, D., Egea, A. C., Carlysle-Davies, S., Carter, I., Clarke, R., Cunningham, J., Doel, T., Forrest, O., Gabasova, E., Geddes, J., Hetherington, J., Jersakova, R., Kiraly, F. J., Lawrence, C., Manser, J., O'Reilly, M. T., Robinson, J., Sherwood-Taylor, H., Tierney, S., Vallejos, C. A., Vollmer, S. J., and Whitaker, K. J. (2019). Design choices for productive, secure, data-intensive research at scale in the cloud. CoRR, abs/1908.08737.
Asthana, M., Blackwell, R., Clinciu, M., Ding, J., Giering, S., Hewitt, C., Rehman, A. u., Sahoo, R., Salvador-Jasin, D., Mehonic, A., Coca-Castro, A., Corcoran, E., and Costa-Gomes, B. (2022). Data Study Group Final Report: Centre for Environment, Fisheries and Aquaculture Science (Cefas). Technical report, Alan Turing Institute.
Austin, C. C. (2018). A path to big data readiness. In 2018 IEEE International Conference on Big Data (Big Data), pages 4844-4853. IEEE.
Piza, F. M. d. T., Celi, L. A., Deliberato, R. O., Bulgarelli, L., de Carvalho, F. R. T., Filho, R. R., de La Hoz, M. A. A., and Kesselheim, J. C. (2018). Assessing team effectiveness and affective learning in a datathon. International Journal of Medical Informatics, 112:40-44.
Pushkarna, M., Zaldivar, A., and Kjartansson, O. (2022). Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1776-1826, Seoul Republic of Korea. ACM.
Romijn, J. (2014). Using big data in the public sector. Uncertainties and Readiness in the Dutch Public Executive Sector. .
Sadin, S. R., Povinelli, F. P., and Rosen, R. (1989). The NASA technology push towards future space mission systems. In Space and Humanity, pages 73-77. Elsevier.
Schlegel, P., Buschmann, D., Ellerich, M., and Schmitt, R. H. (2020). Methodological Assessment of Data Suitability for Defect Prediction. *Quality innovation prosperity/Kvalita Inovácia Prosperita*, 24(2).
Suresh, H. and Guttag, J. V. (2019). A framework for understanding unintended consequences of machine learning. CoRR, abs/1901.10002.
Tomlinson, C., Grols, J., Debnath, R., Johnson, S., Barton, M., Han, T., Khatami, S. N., Baldo, G., Ramesh, A., Cammarano, D., Rajpoot, K., and team, D. S. G. (2022). Automating perfusion assessment of sublingual microcirculation in critical illness. Data study group final report, The Alan Turing Institute.
UK Cabinet Office (2018). Government Security Classifications. https://bit.ly/2xODCkq. Accessed: 03.05.2023.
Wang, R. Y. and Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of management information systems, 12(4):5-33.
Wang, W., Liu, C., and Zhao, D. (2017). How much data are enough? A statistical approach with case study on longitudinal driving behavior. IEEE Transactions on Intelligent Vehicles, 2(2):85-98.
Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., and Auer, S. (2016). Quality Assessment for Linked Data: A Survey. Semantic Web, 7(1):63-93.
Zijlstra, A. A., Vaira, C. L., and Boothe, R. (2017). Open data readiness assessment: Malaysia.
A Appendix: Case Studies
A.1 WWF: Smart Monitoring for Conservation Areas
The datathon focused on developing a NLP system to automatically detect news articles reporting emerging threats to protected areas, enabling real-time monitoring and proactive conservation efforts by WWF and the wider conservation community (Hosseini et al., 2020). The dataset contained approximately 45,000 files describing news stories relevant to UNESCO World Heritage Sites, of which 135 were labeled as 0-'No threat', 1-'Threat detected but not actors' and 2-'Threat with actors'.
Appropriateness. (Optimal) The data was obtained using the Google News API and was manually labeled by WWF experts to be relevant to the challenge.
Readiness. (Developing) Even thought there was a data collection methodology in place, the team had to increase the number of data points available during the event: "...this dataset was expanded during the DSG week to 1,000 labelled data points".
Reliability. (Functional) Data was manually labelled by WWF experts and scraped using the Google News API, creating a before-hand known data collection bias.
Sensitivity. (Optimal) News articles were scraped using Google News API which made it an open source project of Tier 0.
Sufficiency. (Developing) The number of news scraped was potentially sufficient, but the amount of labeled data was insufficient: "...initially a training dataset of 135 news articles annotated by the WWF experts were supplied".
A.2 British Antarctic Survey: Seals from Space
The datathon aimed to create an automated system for classifying sea ice, mapping seals, and exploring environmental factors to facilitate ecological monitoring and understanding of the Antarctic ecosystem (Acharya et al., 2020). The data consisted of GeoTIFF files covering two focal areas, the Antarctic Peninsula (Crystal Sound and Marguerite Bay) and Signy Island. Location data from over 2000 manually-counted seals accompanied these images.
Appropriateness. (Functional) Even though the images and seal location data were provided, some important covariates such as nursing locations, ice features, or marine food availability, were not provided.