Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys
Authors: Lucy Brading
Philippe Ravaud
Paula R Williamson
Bridget Young
Alice M Biggane
Journal: Trials (2018)
Discipline: CLINICAL MEDICINE
Keywords: Core outcome sets; Delphi; Patient engagement; Patient participation; Survey.

Abstract:  Background: There are numerous challenges in including patients in a core outcome set (COS) study, these can vary depending on the patient group. This study describes current efforts to include patients in the development of COS, with the aim of identifying areas for further improvement and study. Methods: Using the COMET database, corresponding authors of COS projects registered or published from 1 January 2013 to 2 February 2017 were invited via a personalised email to participate in a short online survey. The survey and emails were constructed to maximise the response rate by following the academic literature on enhancing survey responses. Personalised reminder emails were sent to non-responders. This survey explored the frequency of patient input in COS studies, who was involved, what methods were used and whether or not the COS development was international. Results: One hundred and ninety-two COS developers were sent the survey. Responses were collected from 21 February 2017 until 7 May 2017. One hundred and forty-six unique developers responded, yielding a 76% response rate and data in relation to 195 unique COSs (as some developers had worked on multiple COSs). Of focus here are their responses regarding 162 COSs at the published, completed or ongoing stages of development. Inclusion of patient participants was indicated in 87% (141/162) of COSs in the published completed or ongoing stages and over 94% (65/69) of ongoing COS projects. Nearly half (65/135) of COSs included patient participants from two or more countries and 22% (30/135) included patient participants from five or more countries. The Delphi survey was reported as being used singularly or in combination with other methods in 85% (119/140) of projects. Almost a quarter (16/65) of ongoing studies reported using a combination of qualitative interviews, Delphi survey and consensus meeting. Conclusions: These findings indicated that the Delphi survey is the most popular method of facilitating patient participation, while the combination of qualitative interviews, Delphi survey and consensus meetings is the most popular combination of methods. The increased inclusion of patient participants in the development of COSs is encouraging, as is the international approach to COS development that some developers are adopting. 

Retracted: No

PubPeer comments:
 The study authors point out in the discussion that Delphi surveys can be excessively long and intimidating. Potential strategies to improve the situation could include the following: - Delphi participants only shown a randomly selected set of n outcomes to vote on - Delphi participants given the option to vote on n outcomes and to get an email reminder x days later to vote on the next set of n outcomes until none are left - Delphi participants given the option to vote on a subset of outcomes (eg only harms), or from subset to subset - Multiple Delphi survey each focusing on subsets of outcomes (eg first group of 100 participants focus on harms, 2nd on y, 3rd on z) - Rolled-out progressive Delphi (eg vote on 10 outcomes each day for 7 days) - Adaptive Delphi voting (eg option to not see outcomes with mean score below given threshold after at least n votes on this outcome) - Combinations of the options suggested above , 
 #1 Thank you for taking the time to read our paper and provide suggestions on how to modify the Delphi survey. It is an area of exciting research and consideration with numerous potential solutions as you indicate in your post. Best wishes, Alice Biggane ,