Title: Developing Statistical Methods for Empirically Optimising Complex Interventions: Applying ‘Design of Experiments’ Methods in Health and Social Care
Funder: Department of Health and Social Care

Abstract: BACKGROUND: Complex interventions 'contain several potentially interacting components', put together into an 'opaque box' to achieve desired changes in important outcomes. However, when evaluated in randomised controlled trials (RCTs), the RCT is often designed as if it is evaluating a simple intervention, estimating only the average effect of the 'box'. A small proportion of the possible versions of a complex intervention that may be given in routine practice are compared, using RCTs with parallel-group designs. This limits the relevance of randomised evidence to real-world practice, and means optimal/efficient combinations of components may be missed. Factorial designs, developed in the 'Design of Experiments' (DoE) field, enable randomisation of patients/clusters to intervention components, and are a potential solution, but have not been widely adopted in health or social care. Indeed, DoE has several complex factorial designs for optimising interventions with complex structures, incorporating intervention context. However, DoE methods are currently limited, not accommodating longitudinal outcomes or incomplete control over allocation of intervention components. A lack of uptake of these designs suggests the need to understand barriers and enablers to their adoption. AIMS AND OBJECTIVES:To improve the health of patients by making a step-change in the design and analysis of future NIHR complex intervention RCTs, by applying and adapting DoE factorial design methodology for use in optimising complex interventions. To provide worked examples and guidance on how to design, analyse and report complex factorial RCTs by re-designing eight RCTs using existing DoE literature; To ascertain scenarios in which complex factorial RCTs are superior/inferior using simulation studies; To adapt DoE literature on complex factorial designs to accommodate longitudinal outcomes address incomplete control over allocation of intervention components; To explore practical barriers and enablers to using factorial designs, and identify factors that might influence their implementation; To provide clear directions for further methodological RCT research building on, and extending, DoE literature.METHODS:WS1: Practical application: Redesign eight RCTs using literature on multi-tiered, split-plot and split-block designs, providing worked examples of complex factorial RCT designs; conduct simulation studies to identify scenarios where these RCTs are superior/inferior to conventional counterparts. WS2: Statistical adaptation: Adapt literature on multi-tiered, split-plot and split-block designs to accommodate:i) Longitudinal outcomes;ii) Partially-randomised designs;iii)Non-orthogonal designs.One exemplar for each type of design will be chosen to motivate and illustrate the new designs. WS3: Understanding implementation: Explore barriers/enablers to adoption of factorial RCTs, talking to key people (research teams, funders) experienced in development and evaluation of complex interventions. Actively seek project partners across the UK. TIMELINE FOR DELIVERY: I will complete the Fellowship over 75 months (0.8FTE) to give opportunities for development outside the Fellowship. I will recruit a full-time post-doc Research Assistant for months 16-51. WS1. Months 1-24 Worked Examples (months 1-24) Simulation Studies (months 7-24)WS2. Months 13-66 Longitudinal outcomes (months 13-36) Partially randomised designs (months 31-54) Non-orthogonal designs (months 49-66)WS3. Months 1-75 Stakeholder engagement (months 18, 25, 43, 60) Project partners (months 1-75)Dissemination. Months 1-75.

Lay abstract: BACKGROUND:In health and social care research, an intervention is anything put in place to try to improve outcomes for patients, staff, services, or systems of care (e.g., treatments, group-based therapy, staff training or new services). Often interventions are made up of a collection of parts and referred to as 'complex'. The effect of the whole intervention may be greater or less than the sum of the parts.The standard approach in the UK is to treat a complex intervention as a single 'opaque box'. Experiments, referred to as randomised controlled trials (or simply RCTs), investigate whether a new intervention (Box A) works better than a current standard intervention (Box B) in a specific context. This approach does not allow detailed understanding of how different parts (or different combinations of parts) of the interventions affect outcomes of importance to patients, staff or services delivering them. Nor does it support the tailoring of interventions to achieve the optimum result for individual patients. Therefore, results may be limited when applied to routine practice. This approach is also inefficient.This fellowship explores the use of 'factorial' RCTs (an RCT design taking into account the parts of a complex intervention) - an approach developed for use in agriculture and industry and drawing on a field of mathematical statistics called 'Design of Experiments' (DoE). Work has been done on factorial RCTs in the US but needs taking further for use in a health and social care setting. AIM:To improve health of patients by making a step-change in the design and analysis of complex intervention RCTs by developing and testing the use of DoE factorial design methodology.METHODS:There are three interlinked workstreams: Practical application: I will redesign eight completed RCTs using existing DoE literature to show how a new approach can be used for randomisation, statistical analysis and sample size calculations. I will conduct computer experiments to identify the scenarios in which existing complex factorial RCTs are better/worse than standard RCT designs. Statistical adaptation: I will adapt existing DoE literature on complex factorial RCTs to accommodate: the possibility of multiple follow-up time-points; practical constraints arising from the intervention context; inability to design RCTs so that the effects of every part can be estimated separately. Understanding implementation: I will explore barriers and enablers to the adoption of the factorial designs, by talking to key people (research teams and funders) experienced in the development and evaluation of complex interventions. I will also actively seek project partners to use my new approaches across the UK.PATIENT AND PUBLIC INVOLVEMENT: To inform this proposal, I have spoken to eight people who are PPI contributors on a range of RCTs. Each person shared their experiences of working on RCTs, and we discussed the challenges faced in undertaking RCTs (including the impact on patients), the opportunity for adopting new methods, and challenges I might encounter. These discussions changed this summary, the language used in the application, the design of Workstream 3, and future PPI plans. Training resources for recruiting patients, for PPI contributors and ethics committees were also suggested. PPI contributors will form a separate PPI panel, participating in project management and project advisory group meetings.DISSEMINATION: Beyond traditional academic outputs, I will host courses, develop a website and disseminate findings through lay summaries, considering possible diverse backgrounds. The promise of this work is that more interventions delivered in practice will have robust evidence; that the pace of health and social care innovations will increase so better interventions will be available sooner; and fewer patients will need to be recruited into RCTs over the research pathway of an intervention.

Research activity codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health categories: GENERIC HEALTH RELEVANCE