Title: Improving the evaluation of mental health digital health interventions in randomised controlled trials by understanding and incorporating user engagement
Funder: Department of Health and Social Care

Abstract: Background Development of digital health interventions for mental health is rapidly increasing, over 10,000 apps now available. They overcome traditional barriers, offering individuals a confidential way to get the support needed. However, with so many available it's an onerous task choosing the right one. This is made worse as most have no evidence on their efficacy. Research has started to catch up, but many aspects are still unknown. One of these components, essential in mental health, is engagement. Digital health interventions have high attrition rates, but little is known of what the right engagement level is. The problem is challenging as these interventions are multifaceted like complex interventions but with an added dimension of unconstrained access. Individuals involved in project development highlighted that understanding engagement and its impact on outcomes as a priority. This can be achieved using statistical methods already available in a randomised evaluation, but a definition of engagement is required. Use of these methods has increased recently in other complex intervention studies where engagement is well defined. This research will provide the foundations for defining engagement for digital health interventions. Research questions How to define engagement levels for mental health digital health interventions? How should trial analysis of digital health interventions adjust for the effective engagement level?Aims and objectives: Determine methods used to summarise engagement and incorporate this in clinical trial results Define engagement levels using indicators in two case studies; EFFIP and BRIGHTEN Define the effective engagement level for each case study and re-analyse the outcomes Develop a methodological framework for researchers to incorporate engagement in trials of other digital health interventionsMethods and timelines for delivery: Outlined below as four work packages: Work Package 1: A systematic review of randomised controlled trials (RCTs) of mental health digital health interventions (month 1 to 9) Work Package 2: Defined engagement levels using indicators, e.g. pageviews, in case studies using Principle Component Analysis (month 10 to 18) Work package 3: Define the effective engagement level and re-analyse case studies using complier average causal effect analysis and evaluate using simulations (month 19 to 27) Work Package 4: Methodological framework combining learnings from previous aims (month 28 to 36) The case studies: EFFIP Study - an RCT comparing COPe-support, a web-based multi-component virtual learning environment for carers of those with psychosis against an attention control. BRIGHTEN Study -h; An RCT comparing two apps to treat depression against an information control app.Impact and dissemination:I will address outstanding clinically important questions regarding how to define engagement to digital health interventions and the impact to outcomes in the trial analysis, an important issue raised by patients involved in project development. Beyond this, it provides essential methodology for trialists to evaluate other mental health digital health interventions, leading to better guidance and helping patients to make informed treatment choices. I will disseminate to researchers and statisticians at seminars, conferences and in open access journals. For an effective impact I will target three audiences; researchers in mental health, trialists and intervention developers.

Lay abstract: Background Apps and web-based tools (collectively digital health interventions) help individuals overcome traditional barriers to mental health support such as physical access and confidentiality. There are more than 10,000 apps available to manage mental health conditions and this number is rapidly increasing, but how these interventions work and in particular how individuals should use them is poorly understood. This makes it challenging for individuals to choose the right one. The amount individuals use these interventions is known as engagement which is crucial in mental health to change behaviours leading to improvements in an individuals well-being. However, research has shown that engagement for digital health interventions in mental health is very low and it's unclear how this affects how well they work. Individuals involved in project development shared that if the effect of engagement was better understood they'd have more motivation to engage. In studies where we evaluate new interventions, specifically randomised controlled trials (RCTs), engagement is typically ignored. However, if the level of engagement is poor, as we know it is in these interventions, then the true benefit for good engagement is underestimated. Aims: The project answers two research questions: How to define engagement levels for mental health digital health interventions? How should trial analysis of digital health interventions adjust for the effective engagement level?Addressed in the following aims: Determine methods in published clinical trial results used to summarise engagement and adjust for this in the analysis Define different engagement levels using data from two case studies; EFFIP and BRIGHTEN Define the effective engagement level for the digital health intervention case studies and re-analyse the trial outcomes Develop a step-by-step guide for other researchers to incorporate engagement into trial analysis of other digital health interventionsCase studies: EFFIP Study - an RCT comparing COPe-support, a web-based multi-component virtual learning environment for carers of those with psychosis against an attention control. BRIGHTEN Study -h; An RCT comparing two apps to treat depression against an information control app.Design and Methods I will examine previous clinical trials of digital health interventions in a systematic review (aim 1). I will describe how engagement is reported, how its visualised and what, if any, methods are described to incorporate engagement into the trial analysis. To define the different engagement levels (aim 2) and which is the effective engagement level (aim 3), I will analyse indicators (e.g. pageviews) from interventions in the case studies. To develop guidance for other researchers (aim 4), I will generate a step by step guide to summarise my learnings from aims 1, 2 & 3. Patient Involvement Individuals with lived experience of mental health illness were invaluable in project development. Engagement levels will be defined using 'indicators' such as pageviews. However, instead of defining the minimum level for each indicator as I originally planned, I will focus on how to define the effective engagement range. During the project I will work with a diverse group of individuals who are the target populations of the case study interventions. Their involvement will help me identify the effective engagement levels for each case study. Engagement ranges for each indicator that showed both clinical benefit and manageable patient burden will be considered for the effective engagement definition.Dissemination To ensure this project leads to improvement in digital health interventions I must reach three audiences; researchers who study these interventions, developers of these interventions and patients who use these interventions. I will publish and promote this work at conferences and through my wider network. I will also seek advice during my patient involvement on the most effective methods to disseminate findings amongst the patient population.

Research activity codes: 6.6 PSYCHOLOGICAL AND BEHAVIOURAL;8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health categories: MENTAL HEALTH