Award Title: Developing and evaluating a tool to protect patients from fabricated data in health systematic reviews
Funder: Department of Health and Social Care

Abstract: Background     When conducting systematic reviews, researchers identify all randomised controlled trials (RCTs) on a given topic, and combine the results to obtain a summary of the evidence that says whether or not a treatment is safe and effective. The results of systematic reviews are very influential. They are included in clinical guidelines, are used by healthcare payers to determine which treatments should be available, are used by clinicians to make treatment decisions, and are used by funders to set research priorities. However, it is now clear that there are a substantial number of fabricated RCTs in the scientific literature. These appear to describe genuine clinical trials, but in reality they are partially or wholly made up.  Because systematic reviews include all relevant RCTs, any fabricated studies are likely to be included. However, there are no established ways for deciding whether any of the included trials are fabricated, and usually no checks are done at all. This means that fabricated trials go undetected. They then contribute to the findings of reviews, meaning that health decision-makers rely on bogus information. This puts patients at risk.  Aim  To develop and evaluate a tool to detect fabricated RCTs during the production of systematic reviews, so that they can be excluded.     Objectives    To supplement an existing scoping review and qualitative study with a review of grey literature and survey of experts to produce a comprehensive list of methods for detecting fabricated RCTs.  To evaluate the impact and feasibility of identified methods by retrospectively applying them to a large sample of published systematic reviews.  To develop a novel tool for identifying fabricated RCTs in systematic reviews, using a Delphi process.  To establish the feasibility of implementing the novel tool into the process of conducting systematic reviews, and its impact upon review conclusions, by prospectively testing it in the production of systematic reviews.      Methods  WP1: We require a comprehensive list of methods for detecting fabricated data for subsequent WP. We will supplement an existing scoping review and qualitative study with a grey literature review and survey of experts.  WP2: We will apply the list of methods to 50 systematic reviews, to obtain empirical evidence about their impact on review conclusions.  WP3: We will enter the list of methods into a two-round Delphi study of relevant experts, to achieve consensus around the content of a novel tool for detecting fabricated RCTs in health systematic reviews.  WP4: In partnership with Cochrane, we will test the tool in the production of new systematic reviews. We will use surveys and a user workshop of authors and editors involved in testing to gather feedback around usability, and will use this to finalise the tool. We will produce training materials for the tool.     Impact  A validated, feasible tool for detecting fabricated RCTs in health systematic reviews will be produced, and made freely available, along with associated training materials. This will inform Cochrane policy, and will protect against harm caused by fabricated RCTs in many clinical areas.

Lay Abstract: Clinical trials are performed to test medical treatments. The results are usually published in medical journals. Systematic reviews identify all the clinical trials published on a given topic, and combine the results in a summary of the evidence that says whether or not a treatment works and is safe. The results of systematic reviews can be very influential, and are often included in clinical guidelines, which are then referred to by doctors when deciding which treatments to use. Unfortunately, sometimes people commit fraud, and publish reports of clinical trials that are partially or wholly made up. These reports appear to describe genuine clinical trials, but they do not.  There are no established ways for the authors of systematic reviews to decide whether trials appear to be fraudulent, and often no checks are done at all. This means that fraudulent trials contribute to the findings of reviews, and consequently that doctors rely on bogus information when deciding on the best way to treat patients. This could mean prescribing treatments that don t work, or are harmful. This puts patients at risk.  We will develop a checklist that researchers can use to detect fraudulent clinical trials, so that they can be excluded when producing systematic reviews. We will do this by getting experts to agree on criteria that can be used to say that a trial is untrustworthy. An example of a possible criterion would be a study claiming to have recruited a suspiciously large number of participants in a short timeframe. We will first identify the possible criteria by reviewing all of the literature on detecting fraud and by sending a survey to journal editors to see how they approach the problem at present. We will test the criteria on lots of systematic reviews, and will use an approach known as a Delphi process to find out which criteria are supported by expert opinion. This will allow us to select which criteria to use. We will combine the selected criteria into a checklist that can be used to identify the fraudulent trials.  We will then test this checklist in new systematic reviews, to see if it is easy to implement and what impact it has on the findings. We will ask people to use the checklist when producing systematic reviews and will gather feedback. The feedback will be presented at a final meeting of people involved in this testing, where we will discuss how the checklist could be improved. We will then make these improvements. The checklist can then be used in systematic reviews to make sure fraudulent trials do not influence the results. We will also make training materials so that researchers know how to use the checklist.

Research Activity Codes: 8.4 RESEARCH DESIGN AND METHODOLOGIES (HEALTH SERVICES)
Health Categories: GENERIC HEALTH RELEVANCE