Assessment of a novel hybrid Delphi and Nominal Groups technique to evaluate quality indicators.

The first step in assessing potential quality indicators is to establish face validity (Campbell et al. 2002). An indicator without face validity is unlikely to be utilized, yet the evaluation of face validity often relies on unstructured assessments that suffer from subjectivity, bias, and poor reliability. Consensus-based panel evaluation is one method of establishing the face validity of indicators, and two approaches are widely used: the Delphi technique and the Nominal Group (NG) technique. The Delphi technique utilizes multiple rounds of independent ratings (e.g., by mailed questionnaire). Although specifics of the method features vary by application (Campbell, Cantrill, and Roberts 2000; Garrouste-Orgeas et al. 2010; Guru et al. 2005), typically a panel of experts independently rates indicators and the ratings are then compiled, summarized, and distributed for review before another round of ratings. This process is continued until the ratings converge and stabilize. This process allows for a large panel, minimizing the influence of individual panelists, and maximizing reliability. However, because the exchange of opinions and information occurs via written documentation, there is no opportunity for interactive discussion. The NG technique also utilizes an initial independent rating, followed by the distribution of summarized results. At this point the panel then meets, traditionally in person and in some cases via conference call, to discuss opinions regarding the indicators. Panelists then rerate the indicators independently. This technique is based on the RAND appropriateness method (Jones and Hunter 1995; Murphy et al. 1998b; Campbell et al. 2002; Hutchings and Raine 2006) and has been widely used in quality indicator development (Asch et al. 2001; Asch et al. 2002; Guttmann et al. 2006; Mularski et al. 2006; Wang et al. 2006; Smith, Soriano, and Boal 2007; Grunfeld et al. 2008) including the Agency for Healthcare Research and Quality (AHRQ) Quality Indicators (Davies et al. 2001; McDonald et al. 2002, 2008). The NG process allows for efficient information exchange among panelists, which is particularly important when panelists offer unique points of view (e.g., different clinical specialties, types of practice). However, successful facilitation of an in-person or call-based panel limits the size, generally to under 15 individuals. Without effective moderation by the facilitator, one or two individuals can unduly influence the discussion. Because of the small panel size, interpanel reliability is limited (Hutchings et al. 2006; Murphy et al. 1998a). Both techniques provide advantages over unstructured methods (Murphy et al. 1998a; Campbell et al. 2002). First, structured evaluations attempt to combat cognitive biases in judgment. These biases have been well described by Tversky and Kahneman (1974) and are particularly influential in complex tasks. For example, anchoring bias can occur when panelists set their initial responses relative to the opinion of the group. The Delphi Group and NG methods both require an independent initial rating to anchor opinions based on an individual's own knowledge. Furthermore, structured methods focus the discussion on specific topics pertinent to the underlying validity of the indicator and allow all panelists to have access to similar information before evaluation. Finally, these methods allow for objectively quantifying the results for direct comparisons among indicators to better establish consensual validity. The relative advantages and disadvantages of these panel techniques for evaluations of indicators have not been studied (although other applications have been examined), and methodological modifications are often not assessed. Some recent studies have assessed the impact of different panel compositions and methods, although most studies examined decisions of “appropriateness” rather than assessments of quality measurement (Hutchings et al. 2006; Hutchings and Raine 2006). We sought to develop a method that could incorporate the advantages of each method, and thereby minimize the disadvantages of each. This paper describes our hybrid Delphi/NG method and presents an evaluation of this method as applied during an assessment of quality indicators.