Responsible handover: Report on findings
Creators: name: Abok, Dorothea
name: Trust, The Wellcome
Publisher: name: F1000 Research Limited
Year: 2025

description: Data science and AI have become part of major national health strategies in the UK<sup>1</sup> and internationally<sup>2</sup><sup>,3</sup><sup>,4</sup>. It has therefore become a question of how, not if, we adopt these technologies. There has been extensive discussion around the ethics and safe development of these tools, ranging from the global stage to institutional levels. However, we are still in a situation where there is shockingly low distribution of knowledge of these systems across those involved in their implementation. Therefore, there is a significant risk of harm being done by these tools, in parallel with a risk that these tools aren’t used to their maximum benefit because of fear of the former. We have sought to scope the extent of this issue, capture existing best practice and propose and test the Responsible Handover framework as a solution, that can work across sectors and differing levels of technical expertise.<br>
<br>
While the number of public discussions around AI has skyrocketed, there is growing frustration that these have largely resulted in high-level principles which are hard to operationalise.<br>
<br>
Commendable work has produced the following thus far: ‘codes of practice’ listing ethical principles that developers of data science tools, and particularly AI, must abide by (autonomy, justice, beneficence, non-maleficence, and explainability being the most common), technical tools for identifying specific issues (such as code libraries for identifying sources of bias in AI training datasets), algorithmic impact statements (such as that produced by the EU’s high- level expert group) that encourage developers to proactively think through the potential effects of their model on specific stakeholder groups, and – increasingly – audits that can be<br>
conducted by third parties to assess the ‘transparency’ of models. While all have a part to play in different steps of a tool’s pipeline from the point of a tool’s creation to its post- implementation auditing, there are no processes yet for the point of a tool being handed over.<br>
<br>
<i>“It's</i><i> </i><i>getting</i><i> </i><i>to</i><i> </i><i>the</i><i> </i><i>point</i><i> </i><i>where</i><i> </i><i>it's</i><i> </i><i>becoming</i><i> </i><i>reckless.</i><i> </i><i>It's</i><i> </i><i>like</i><i> </i><i>you're commissioning</i><i> </i><i>a</i><i> </i><i>bunch</i><i> </i><i>of</i><i> </i><i>self-driving</i><i> </i><i>projects,</i><i> </i><i>for</i><i> </i><i>cars.</i><i> </i><i>But</i><i> </i><i>you're</i><i> </i><i>giving nothing</i><i> </i><i>to</i><i> </i><i>civil</i><i> </i><i>authorities</i><i> </i><i>about</i><i> </i><i>signage,</i><i> </i><i>or</i><i> </i><i>wardens</i><i> </i><i>or</i><i> </i><i>police</i><i> </i><i>presence</i><i> </i><i>to </i><i>facilitate these experiments” – </i><b><i>Dr. Haris Shuaib, Head of Clinical Computing </i></b><b><i>at</i></b><b><i> </i></b><b><i>Guys</i></b><b><i> </i></b><b><i>and</i></b><b><i> </i></b><b><i>St</i></b><b><i> </i></b><b><i>Thomas’</i></b><b><i> </i></b><b><i>NHS</i></b><b><i> </i></b><b><i>trust</i></b><br>
<br>
1 Department for Science, innovation and Technology, Department of Health and Social Care, The Rt Hon Michelle Donelan, The Rt Hon Steve Barclay MP and The Rt Hon Rishi Sunak MP, 2023. New £100<br>
million fund to capitalise on AI’s game-changing potential in life sciences and healthcare, https://www.gov.uk/government/news/new-100-million-fund-to-capitalise-on-ais-game-changing-  potential-in-life-sciences-and-healthcare<br>
2 U.S Department of Health and Human Services, 2024. HHS Artificial Intelligence (AI) Strategy, https://www.hhs.gov/programs/topic-sites/ai/strategy/index.html<br>
3 Australian Digital Health Agency, 2023. National Digital Health Strategy, https://www.digitalhealth.gov.au/national-digital-health-strategy/about-the-strategy.<br>
4 European Medicines Agency, 2023. Artificial intelligence workplan to guide use of AI in medicines regulation, https://www.ema.europa.eu/en/news/artificial-intelligence-workplan-guide-use-ai- medicines-regulation.<br>
<br>
Moreover, efforts to address the whole value chain of a tool mostly split into either high-level abstraction that is tricky to turn into processes or are so domain and context specific, they’re difficult to adapt for other purpose. Thus, the impact of these various tools or mechanisms designed to translate the ‘what’ of data science ethics into the ‘how’ have had an extremely limited impact on the real world.<br>
There still exists no standardized or established mechanism for assessing or preventing the potential harms of data science with biomedical applications, and, consequently, it is challenging for research funders and healthcare trusts alike to mandate or regulate in a cohesive way.<br>
<br>
Rapid technological developments have happened before, and resulted in great benefit for society, however the scale and speed of development of data science and AI tools is what distinguishes it from historic advances. In the biomedical field, outcomes and risk of harm are measured for individual patients, health care outcomes and at a population level. Broadly speaking, the major risks posed by these tools are as follows (all are set out and explored further in the problem statement created at the outset of this project):

 Privacy infringement
 Loss of autonomy
 Poor quality data
 Poor quality evidence
 Bias and inequities
 Accountability

The above issues are somewhat inherent to the tools themselves, but if picked up by a different user these tools can present two types of harms: deliberate misuse or accidental misuse, the latter of which has not been discussed to the same level. An incredibly powerful and successful tool developed for one context, may be completely inappropriate to apply to a different context and thus accidentally cause harm. Every tool comes with built-in strengths, assumptions and limitations, but there are currently no established processes to transfer this information with the tool itself. The are also additional problems associated with local deployment over slow-speed and high-compute utilisation that may impact how well the tool can perform. This is also information that can be transferred in a rigorous handover conversation.<br>
<br>
<i>“Here is</i><i> </i><i>a</i><i> </i><i>risk of</i><i> </i><i>someone just</i><i> </i><i>applying</i><i> </i><i>the</i><i> </i><i>model</i><i> </i><i>blindly.</i><i> </i><i>Because</i><i> </i><i>what</i><i> </i><i>they </i><i>know</i><i> </i><i>is</i><i> </i><i>I</i><i> </i><i>need</i><i> </i><i>data,</i><i> </i><i>I</i><i> </i><i>have</i><i> </i><i>data</i><i> </i><i>that</i><i> </i><i>I</i><i> </i><i>can</i><i> </i><i>just</i><i> </i><i>apply</i><i> </i><i>it</i><i> </i><i>on</i><i> </i><i>the</i><i> </i><i>same</i><i> </i><i>model</i><i> </i><i>and </i><i>get</i><i> </i><i>some</i><i> </i><i>output,</i><i> </i><i>it</i><i> </i><i>won't</i><i> </i><i>tell</i><i> </i><i>you</i><i> </i><i>that</i><i> </i><i>this</i><i> </i><i>output</i><i> </i><i>is</i><i> </i><i>not</i><i> </i><i>appropriate…..</i><i> </i><i>If</i><i> </i><i>you </i><i>don't</i><i> </i><i>have</i><i> </i><i>a</i><i> </i><i>good</i><i> </i><i>understanding</i><i> </i><i>of</i><i> </i><i>the</i><i> </i><i>right</i><i> </i><i>tool,</i><i> </i><i>and</i><i> </i><i>background</i><i> </i><i>of</i><i> </i><i>what you're</i><i> </i><i>trying</i><i> </i><i>to</i><i> </i><i>do,</i><i> </i><i>most</i><i> </i><i>likely</i><i> </i><i>you</i><i> </i><i>might</i><i> </i><i>end</i><i> </i><i>up</i><i> </i><i>applying</i><i> </i><i>models</i><i> </i><i>in</i><i> </i><i>wrong</i><br>
<i>settings” – </i><b><i>Dr. Steve Bicko Cygu, Data Scientist at African Population Health </i></b><b><i>Research Centre</i></b><br>
<br>
Where these handover conversations are happening, it’s often in incredibly technical language<br>
that is hard for the non-expert to question and understand, leading to an assurance gap.<br>
Consequently, legitimate concerns that tools t could cause harm and a lack of actionable<br>
processes to ensure that ‘due diligence’ has been performed results in an opportunity cost. There are people in decision-making positions acting as gatekeepers to these tools being implemented that lack detailed technical knowledge, but also lack the confidence to say yes as there isn’t an equivalent assurance process that’s present in other areas of engineering, ranging from nuclear power to circuit boards.<br>
The point a tool is handed over may happen in several ways and thus the conversation will appear differently. But the structure of this conversation plus the key landmarks will carry across these different permutations. For example, a tool being deployed into hospitals for patient diagnosis will involve perhaps a lengthy and complex process between the developer and clinicians, compared to a tool shared via GitHub which will be picked up asynchronously by a researcher looking to use the tool without communication between the two parties.<br>
However, in both situations, there is space for both parties to ask themselves questions, provide assurances and documentation.<br>
<br>
Other sectors of engineering have established systems and extensive history of designing such processes, often called scripts. These scripts are suitable for multiple situations and can be adapted by an end-user to fit their appropriate context. These scripts are central to creating a system of distributed knowledge, which is commonplace across these sectors. For example, it is not solely the developer of a nuclear reactor who has responsibility to sign off on the safety of a plant. This and other applications are based on frameworks that have been developed for end-users to make domain specific scripts. This is an applicable concept across risk levels and complexity of information; the process of building a bridge can sometimes take as long as building it, with the involvement of multiple parties for sign off, however this process is largely governed by well-established scripts for this process.<br>
<br>
<i>“We</i><i> </i><i>have</i><i> </i><i>a</i><i> </i><i>pretty</i><i> </i><i>good</i><i> </i><i>framework</i><i> </i><i>for</i><i> </i><i>us</i><i> </i><i>to</i><i> </i><i>build</i><i> </i><i>questions</i><i> </i><i>for</i><i> </i><i>drugs…a </i><i>package insert is a set of questions. And you as a consumer would like to ask the manufacturer that and it's provided to you in excruciating detail. Now, we need a package insert equivalent.” – </i><b><i>Professor</i></b><b><i> </i></b><b><i>Salim</i></b><b><i> </i></b><b><i>Abdool</i></b><b><i> </i></b><b><i>Karim, Director </i></b><b><i>of</i></b><b><i> </i></b><b><i>Center</i></b><b><i> </i></b><b><i>for</i></b><b><i> </i></b><b><i>the</i></b><b><i> </i></b><b><i>AIDS</i></b><b><i> </i></b><b><i>Programme</i></b><b><i> </i></b><b><i>of</i></b><b><i> </i></b><b><i>Research</i></b><b><i> </i></b><b><i>in</i></b><b><i> </i></b><b><i>South</i></b><b><i> </i></b><b><i>Africa</i></b><b><i> </i></b><b><i>(CAPRISA)</i></b><br>
<br>
Types of handovers in the biomedical space vary widely, both in terms of their risk level and the mechanism for information transfer. They may resemble the above engineering commissioning process, with an intense and involved relationship between the two handover parties, but they may also entail a researcher uploading an open-access RNA-seq data analysis tool to GitHub, which results in an asynchronous handover of information to the end- user with little control over who might pick this tool up. In this situation, establishing a handover process is as, if not more important, as there is less room for back and forth or clarification, and therefore the Responsible Handover conversation might be more similar to a‘living will’.<br>
<br>
In situations where a researcher or research team is sharing a tool with other researchers, there may be a large difference in subject specific knowledge and in code expertise. This expands outside of just from the individual level to the research climate that a tool may be entering into, such as low- and no-code environments, where there is a lack of infrastructure or shared knowledge to support a researcher attempting to deploy these tools. There are also situations where researchers are handing over tools to non-research environments, e.g. commissioners or procurement, formal clinical settings or real-world community implementation to the people directly interacting with patients. In these latter examples, the tool in question may be a commercial product and the developer a private company, which may impact what information can be shared widely.<br>
<br>
To some extent, the communication of the results of the tool by a researcher also is a<br>
handover of sorts, even if the ‘end-user’ will not be using the tool themselves, but instead will be using the findings to act as a basis for further research. In this case, specific parts of the handover information are more important than others, e.g. information on the uncertainties and the assumptions.<br>
Despite the varied nature of these handovers, all are vulnerable to information loss that could lead to harm or an opportunity cost, stemming from the same central concepts and issues that are inherent to all data science and AI tools.