Identifying Best Practices within the MOVE! Weight 

Management Program for Veterans: An Application of 

Qualitative Comparative Analysis

Presented by: 

Heather Kane, PhD

Megan A. Lewis, PhD

Pamela A. Williams, PhD

Leila C. Kahwati, MD MPH

December 15, 2011
Poll 1

Have you heard of Qualitative Comparative 

Analysis (QCA) methods?

1) Yes

2) No

3) Not sure

2
Roadmap for Presentation

• Background and Purpose

• Qualitative Comparative Analysis (QCA) Defined

• Steps for Conducting a QCA 

– Example using the MOVE! Best Practices Evaluation

• Advantages of QCA

• Challenges of QCA 

• Conclusion

3
Background and Purpose

• MOVE! Weight Management Program for 

Veterans 

• Evaluation purpose: identify “best practice” 

facility structures, processes, policies, and 

clinical and organizational features linked with 

better patient weight loss outcomes within the 

VHA’s MOVE! Weight Management Program. 

• Collaboration between VHA and RTI

4
QCA Defined

• Qualitative Comparative Analysis is an analytic 

technique that bridges qualitative and quantitative 

methods.

– Based on set-theory 

– Useful for small to medium N number of cases

– Focuses on a limited set of conditions/features

– Used when outcomes have more than one cause and/or 

the interaction of multiple causes is in play (i.e., causal 

complexity and identifies multiple solutions to achieving 

outcome (equifinality)

– Supports theory-building and testing

– Provides actionable policy information

5
QCA Results/Findings

• Findings from QCA are reported using 

outcomes that are quite different from 

traditional quantitative or qualitative studies.

• QCA findings are reported as conditions that 

are necessary and conditions that are 

sufficient for the outcome of interest.

• In the MOVE! Program example, the outcome 

of interest was facility-aggregated patient 

weight loss outcomes at 6 months. 

6
X: Buying a lottery ticket

Y: Winning the 

Lottery

Y: Winning the Lottery

X1+ X3: also 

has lucky # of 

the day, which 

doubles the 

jackpot

X1: all 

numbers 

match, 

jackpot!

X2: five 

number 

match, still a 

winner but at 

a lower payout

Necessary & Sufficient Conditions

Necessary condition: Necessary 

conditions are conditions that must be 

present for an outcome to occur. 

Sufficient condition: demonstrate whether the 

cause in question produces the outcome in 

question. To assess sufficiency, a researcher looks 

at cases with the same causal condition (“X1”, “X2)) 

(or combinations of causal conditions “X1 & X3”) 

and identifies cases that share the same outcome. 

7
Conducting a QCA

• Step 1: Identify the evaluation question and 

develop a conceptual model

• Step 2: Select cases

• Step 3: Choose conditions

• Step 4: Identify or develop data sources

• Step 5: Calibrate conditions

• Step 6: Manage the data and code the cases

• Step 7: Analyze data 

8
Step 1: Identify Evaluation Question and 

Develop a Conceptual Model

• Adapted Weiner, Lewis and Linnan (2009) 

Model of Implementation Effectiveness

Rival

Activities

NCP

Actions

Implementation 

Policies and 

Practices

MOVE!-Task

Fit

Implementation 

Climate

MOVE! Leadership

• Facility leadership

• VISN Coordinator

Management 

Support

Resource

 Availability

MOVE!-Values

Fit

MOVE!

Implementation 

Effectiveness

MOVE!

Clinical 

Effectiveness

9
Step 2: Select Cases

• Case selection driven by outcome of interest and theory

• Evaluation team used MOVE! patient outcomes aggregated 

at the facility level to guide selection:

– mean body weight change at 6 months

– percent of treated patients losing 5% or more of body weight

• Facility size, location, and complexity were also factored in 

to assure representativeness

• Total number of sites (n=22) chosen based on resources 

and ability to maintain “empirical intimacy” 

– 11 sites with larger patient weight loss outcomes

– 11 sites with smaller patient weight loss outcomes

• RTI investigators involved in QCA data collection were blind 

to facility weight loss outcomes

10
Step 3: Choose Conditions

• Prioritized “resource availability” and 

“implementation policies and practices” from the 

conceptual model, plus clinical program features 

that should be part of evidence-based weight 

management

• Created an exhaustive list of conditions

• Challenge of too many conditions (4-5 are 

generally best)

• Final list was narrowed down to 17 conditions 

(see next slide)

11
Step 3: Choose Conditions

• Implementation policies and practices

– Interface between screening and treatment –use of 

an orientation session or class

– Standard curriculum for program delivery

– Multidisciplinary team approach involving dietitian 

and at least one other discipline

– Program complexity- use of a group orientation or 

some other initial screening and a minimum of an 8-

week group session series plus a maintenance 

component (or a longer than 8-week active 

component)

– Type of care delivery - group, individual, or 

combination 

– Quality improvement (QI) strategies used for 

enhancing MOVE! program and resolving 

program challenges

– High facility accountability and internal reporting 

requirements

– High VISN accountability and external reporting 

requirements

• Resource Availability

–High level of staff effort providing MOVE! care to 

patients (above the median for number of FTE 

providing MOVE! care per patient seen with 

MOVE!)

–Facility self-reported resource sufficiency 

–Facility complexity level (Medical Center vs. 

Community Based Outpatient Clinic)

–Data tracking and analysis capacity

–Active physician champion involved in MOVE! 

program

• Clinical Program Features

–Tailored but structured dietary plans for patients 

used

–Tailored but structured physical activity plans for 

patients used

–Multiple behavioral modification strategies used

–Weight loss maintenance component used

12
Step 4: Identify or Develop 

Data Sources

• Program Summary Form (PSF)

– Collected diagrams of patient flow through MOVE! programs

– Collected organizational charts including supervisory 

hierarchies and functional relationships

– Collected detailed staffing effort

• Key informant interview data (1-2 MOVE! staff/site)

• Electronic Medical Record (EMR) abstraction

– Chart reviews from ~50 randomly selected patients treated 

with MOVE! at each facility

• Follow-up summary

– Sites reviewed summary of PSF and interview and provided 

additional feedback, clarification, or correction to ensure 

accurate representation of their program

13
Step 5: Calibrate Conditions

• Calibration:

– “The degree to which cases satisfy membership criteria [in 

a condition], which in turn are usually empirically 

determined, not inductively derived” (Ragin 2008:80)

– Example:

• Condition: “Active Physician Champion”

14
Step 5: Calibrate Conditions

• Crisp-set calibration used (as opposed to fuzzy set)

• Each condition is evaluated and rules established:

• present (+) OR absent (-) [absolute]

• represented to a high degree (+) OR low degree (-) [relative]

• fully in the set (+) OR fully out of the set (-) [QCA jargon]

• Drew on expert knowledge to develop decision rules 

for calibration

• Iterative refinement to data collection tools and 

decision rules based on piloting with two sites

• Advantages of crisp-set calibration

– Easier to interpret and make policy recommendations

• Disadvantages

– Forces a dichotomy, lose some granularity

15
Step 6: Manage the Data and Code Cases

• Abstracted data from sources into Excel table

• Reviewed data

• Managed conflicts in data sources

• Assigned calibration decision 

• VHA team not involved in data coding

Condition #1 Physician Champion level of involvement

Facility ID

Program 

Summary Form 

(see Qs 1, 2) 

Interview Data 

(see Qs 2, 3, 10)

EMR Data Site 

Summary 

Form/

Follow-up

Decision

(1 = fully in the set;

0 = fully out of the 

set)

Facility A

XYZ PDQ ABC DEF 1

Facility B XYZ MNO HIJ QRS 0

16
Step 7: Analysis

• Coding of cases results in a “truth table”

• Looked for single conditions that were 

necessary or sufficient

• Managed having too many conditions with 

“bottom-up QCA” approach

• Reviewed unusual finding and reexamined 

qualitative data

• Iterative analysis

17
Truth Table-simple example

Site Condition X1 Condition X2 Condition X3 Outcome

A - + - Absent

B + + - Present

C + - - Absent

D - - + Absent

E + - + Present

F - - - Absent

G - + - Absent

H + + + Present

I + - - Absent

J + + - Present

18 X1 is a necessary condition, X2 and X3 are sufficient conditions
Necessary Conditions for MOVE!

• To achieve larger patient weight loss outcomes 

the following are necessary:

– Use of a standard curriculum

– Delivery via group-based care

• These two conditions were present at all 11 sites 

with larger outcomes (and 5 sites with smaller 

outcomes). Therefore, the absence of these two 

conditions guarantees that a site will have smaller 

patient weight loss outcomes. 

19
Sufficient Conditions for MOVE!

• Four sufficient combinations were identified:

– High program complexity in combination with high staff 

involvement (n=5)

– Active physician champion in combination with low facility 

accountability (n=5) 

– Use of mixed delivery format (group and individual) in 

combination with low facility accountability (n=5)

– Use of quality improvement strategies in combination with NO 

waiting list (n=3)

Most sites had more than one sufficient combination of conditions present.

20
Summary of Findings

Site 2, 5, 7, 8, 13, 22

Necessary Conditions Present

N=17

• Use of a standard curriculum

• Use of a group care delivery format

Site 9, 14

Site 19

Site 1, 11

Site 21

Site 6

Site 4, 12, 20

Use of quality improvement & no use 

of wait list

High program 

complexity and 

high staff 

involvement

Use of group 

care delivery 

format & low 

program 

accountability to 

facility 

leadership

Site 15

Active physician 

champion & low program 

accountability to facility 

leadership

Necessary 

Conditions Absent

N=5

Site 3, 10, 16, 17, 18

Reprinted from Figure 1 Am J Prev Med 2011: 41(5):457-464 21
Limitations of this Evaluation

• Case selection limitations

– Crude weight loss outcomes used

– Outcomes from the prior year used

– Weight data from electronic EMR extracts

• Use of crisp-set QCA requires dichotomization 

of conditions under evaluation

22
Translation of Findings to Policy/Practice

• Evaluate local MOVE! programs to ensure:

– Use of a standard curriculum that has a group component

• A 12-session group curriculum is available at the MOVE! 

intranet website http://vaww.move.med.va.gov/GrpSessions.asp

– Have an active physician champion

– Have a high level of staff involvement and program complexity 

(orientation sessions, maintenance treatment, multiple disciplines 

involved)

– Use quality improvement strategies and avoid use of a waiting list

23
Challenges and Advantages of QCA

Challenges

• Not suitable for very small 

case studies (e.g., 3-5 cases)

• Intensive data collection 

and iterative analyses

• Limited number of 

conditions allowed

• Less useful when programs 

are not comparable

• Collaboration between 

program implementers and 

evaluators required

Advantages

• Offers useful analytic tool 

for studying organizational 

process or programs

• Allows testing of 

equifinality principle

• Can be applied in a 

healthcare setting with 

actionable results

• Supports theory building 

and theory testing

24
Poll 2

Now that you have learned more about QCA do 

you think it would be a helpful approach to use 

in your research?

1) Very helpful

2) Somewhat helpful

3) Possibly helpful

4) Not helpful at all

25
Citations

MOVE! Best Practices QCA Evaluation:

• Kahwati LC, Lewis MA, Kane H, Williams PA, Nerz P, Jones KR, Lance TX, Vaisey S, Kinsinger LS. 

Best practices in the Veterans Health Administration's MOVE! Weight management program. 

Am J Prev Med. 2011 Nov;41(5):457-64. 

Other Citations:

• Weiner BJ, Lewis MA, Linnan LA. Using organizational theory to understand the determinants 

of effective implementation of worksite health promotion programs. Health Educ Res 

2009;24(2):292-305.

• Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 

1999 Dec;34(5 Pt 2):1225-39.

• Rihoux Benoit RC, editor. Configurational Comparative Methods: Qualitative Comparative 

Analysis (QCA) and Related Techniques. Thousand Oaks: SAGE Publications; 2009.

• Ragin CC. Redesigning Social Inquiry: Fuzzy Sets and Beyond. Chicago, IL: University of 

Chicago Press; 2008.

• Dy SM, Garg P, Nyberg D, Dawson PB, Pronovost PJ, Morlock L, et al. Critical pathway 

effectiveness: assessing the impact of patient, hospital care, and pathway characteristics 

using qualitative comparative analysis. Health Serv Res. 2005 Apr;40(2):499-516.

26
Contact Information

Heather Kane, PhD

hkane@rti.org

Leila Kahwati, MD MPH

Leila.kahwati@va.gov

Megan Lewis, PhD

melewis@rti.org

27
Additional Slides

28
Solution Coverage

29

Table 2. Raw and unique solution coverage of the four sufficient condition combinations 

identified in the qualitative comparative analysis

Solution Raw Coverage Unique Coverage

n (%) n (%)

-High program complexity and high staff involvement 5(45%) 3(27%)

-Use of QI and no wait list 3 (27%) 2(18%)

-Active physician champion and low program 5 (45%) 1 (9%) 

accountability to facility leadership

-Use of group-care delivery format and low program 

accountability to facility leadership 5 (45%) 1 (9%) 

Total Solution Coverage 100%

Note: N = 17 because 5 cases lacking the necessary conditions were excluded from the analysis of sufficient 

conditions
12/15/2011 12:00pm Identifying Best Practices within the MOVE! Weight Management 

Program for Veterans: An Application of Qualitative Comparative 

Analysis

Timely Topics 

of Interest 

Kahwati, Leila

Kane, Heather

Lewis, Megan

Do you happen to know whether data collected at site level was based on primarily male patients, or 

male and female patients? Or, has anyone looked at how move program outcomes may vary by 

sex/gender of veteran and by site org characteristics? I am asking because of the sex differences in 

weight management.

The current MOVE! Outcomes Report is available at the VSSC Website. The direct link is here:

https://securereports2.vssc.med.va.gov/Reports/Pages/Report.aspx?ItemPath=%2fMove%2fMoveBMIo

utcomes. It is also accessible from the main VSSC home page under the Clinical Programs section , then 

the MOVE! section. 

Was there a relationship between size of facility (medical center vs CBOC) and weight loss outcome?

The QCA evaluation did not specifically evaluate this. I refer you to another publication that discusses 

weight loss outcomes (Kahwati et al. REAIM Evaluation of the MOVE! Weight Management Program for 

Veterans http://www.springerlink.com/content/rqu01k308rt314w2/fulltext.html ). In some of our 

unpublished, operationally-driven work, we have found no association between facility complexity level 

(Complexity category( IA, IB, 1C, 2, 3, and various CBOC sizes) and weight loss outcomes. 

You mentioned most sites had trouble making a female only move group. Where can we find that 

specific data on difficulties with MOVE by sex/gender? Or, should I email you directly?

Would recommend you contact Dr. Ken Jones, the National Program Director for Weight Management 

(Kenneth.jones6@va.gov). 

Were attrition rates reported or analyzed?

This was not an evaluation of clinical weight management care, so attrition rates are not applicable. I 

refer you to another publication that discusses weight loss outcomes and patient engagement in care 

(Kahwati et al. REAIM Evaluation of the MOVE! Weight Management Program for Veterans 

http://www.springerlink.com/content/rqu01k308rt314w2/fulltext.html ). 

The best practice for one site might not be applied to other sites due to other environmental factors. 

How do we make sure that the best practice can be applied universally?

The strength of QCA is that it determines all of the recipes for success , and recognizes that there are 

multiple paths to success (e.g., this is the concept of equifinality). So, sites can have a menu to choose 

from. Also, if sites want to maximize their odds of success, but only have resources to invest in one best 

practice, they should select the solution with the highest raw and unique coverage as high raw and 

unique coverage means the solution was found at multiple sites, versus a solution with low coverage, 

which might mean the solution is less translatable across different sites.