Title: Publication and related bias in health services and delivery research (HSDR): systematic review of literature, evaluation of empirical evidence and methodology, and key informant interviews
Funder: Department of Health and Social Care

Abstract: Publication bias refers to the phenomenon that the likelihood of a research study being published is related to the direction or statistical significance of its findings. High-profile cases of publication bias in clinical trials of medicinal products have led to public outcry and enforcement of mandatory trial registration. Other forms of related bias that occur between the generation of research evidence and its publication (collectively termed publication bias hereafter), such as p-hacking (repeating analyses of data until statistically significant results are obtained) and selective outcome reporting, have also been highlighted as possible causes of lack of reproducibility of research findings in various scientific disciplines. Despite its importance, publication bias seems to be infrequently mentioned/discussed in health services and delivery research (HSDR), which produces evidence on the quality, accessibility and organisation of health services. This was confirmed by our scoping review (see Project protocol) and our discussions with some of the leading HSDR experts and journal editors. There is no obvious reason why HSDR should be immune from publication bias, and reports of anecdotal cases exists in the literature. We posit that the paucity of comprehensive investigation into this subject may arise from features of HSDR making its investigation challenging, e.g. lack of comprehensive registration of non-randomised studies which constitute a large share of evidence, lack of a clear boundary between research and routine quality improvement activities, the need to investigate numerous processes and outcomes along the service delivery causal pathways and the inherent complexity and fluidity of HSDR interventions and the context in which they are applied. Despite these challenges, a systematic investigation of publication bias in HSDR is of crucial importance because HSDR frequently informs decisions at institutional and policy levels, and this, in turn, impacts on the NHS and the population it serves. This proposed research aims to address this important issue by gathering prima facie evidence of publication bias in HSDR, documenting current practice and exploring common methods for detecting and mitigating the bias during evidence synthesis and providing recommendations for future research and practice. These aims will be achieved through five inter-related work packages (WPs), including three literature-based WPs (WP1, 2 and 3), a retrospective study of a HSDR cohort (WP4) and key informant interviews (WP5). WP1 is a systematic review built upon our scoping review of empirical evidence and methodologies related to publication bias in HSDR. WP2 is an overview to document current practice, findings and issues in systematic reviews of HSDR that have quantitatively examined the effectiveness of HSDR interventions or the associations of different factors along the service delivery causal chain. WP3 consists of in-depth study of cases drawn from WP1 and 2 to explore the applicability of different methods used to detect and mitigate publication bias when synthesising evidence in HSDR. WP4 aims to observe publication bias directly by retrospectively following up a cohort of HSDR studies to ascertain any relationship between their findings and publication status. WP5 will complement the other four WPs by exploring the perceptions and experience of HSDR researchers, journal editors and commissioners.

Lay abstract: It is important that the people who are responsible for making important decisions about the ways our health services are delivered have the best and most reliable evidence available to them to make sure that they make the right choices. Sometimes, however, the evidence available doesn’t tell the ‘full story’ or the evidence available may be biased. One problem is ‘publication bias’, which happens when only certain studies, usually those which are more exciting or are ‘positive’ are published. There are other forms of bias, too. Sometimes, researchers might not get exciting or positive results, so they use other ways to analyse the data or analyse only parts of the full data set until they get a more interesting result. This is called ‘p-hacking’. There is also ‘outcome reporting bias’, which happens when a study collects lots of data, but only the ‘positive’ or interesting results are published. In this project, we use the term ‘publication bias’ broadly to cover these various forms of bias as well. It is important to study publication bias because if the people making decisions don’t have all the evidence available to them, they might make wrong decisions, which may have a negative effect on our health services and the way the services are delivered. In clinical research, which focuses on medical treatments for patients, there has been a lot of research on publication bias and examples such as the non-publication of data showing harmful effects from drug trials has hit the headline. It is very important to make sure that there is no publication bias as treatment decision based on biased information might be wasteful or even harmful. Because of the potentially serious consequences, all clinical trials now have to be registered so that people making decisions about treatments have all the information available to them to make the right choices. In Health Services and Delivery Research (HSDR), which looks at the best ways our services are delivered to patients, there hasn’t been much research on publication bias. This might be because health service delivery research is very complicated. In this study, we will look at publication bias in HSDR to see if it exists, what kinds of publication bias there are, and how we can detect or deal with it. We want to obtain and then share the information about publication bias so that researchers doing HSDR and users of findings from the research can take measures to make sure that the occurrence and impact of publication bias are minimised. Investigation of publication bias in HSDR is very important because its findings are often used to make policy decisions that will influence many people. This project will be led by experienced researchers from the University of Warwick who will work with colleagues with expertise in HSDR and/or publication bias from the University of Birmingham and the University of East Anglia along with international experts in health services research and representatives of patients and the public. The study has five work packages (WPs), which are all linked. These include: WP1 - a systematic literature review extending a preliminary review that we have already conducted on methodological research related to publication bias in HSDR; WP2 - an overview of systematic reviews (which are literature reviews that utilise transparent and scientific methods to summarise the best available evidence on specific topics) of HSDR to see how people have dealt with publication bias when summarising evidence from HSDR and what they found; WP3 – in-depth studies of most informative cases selected from WP1 and WP2 to explore how useful methods that have been used to find and protect against publication bias in other fields are in HSDR; WP4 – a study to follow up two groups of HSDR studies and directly observe whether publication bias occurred; WP5 – in-depth interviews to find out the perceptions and experience related to publication bias among researchers, journal editors and research commissioners in HSDR. We will disseminate findings from this project in key health service research conferences and two dissemination workshops, and will publish the findings in academic journals. In addition we will produce practical recommendations with regard to how to minimise, detect and deal with publication bias for different stakeholders of HSDR.

Research activity codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health categories: GENERIC HEALTH RELEVANCE