Making sense of the ideal and the real in 

PACT implementation using qualitative 

research 

 

Part 1: 

Methodology, methods, and research-operations partnership 

HSR&D PACT Cyberseminar 

September11, 2013 Solimeo, Hamilton & True 1 
National PACT Demonstration Laboratories PACT Demonstration Lab Coordinating Center 

 Directed by Stephan Fihn, MD, MPH, VHA Office of Analytics and Business Intelligence, 

Seattle, WA. 

VISN 4 Center for Evaluation of PACT (CEPACT) 

Directed by Rachel Werner, MD, Ph.D., Philadelphia VA Healthcare System in Philadelphia, 

PA.

VISN 11 PACT Demonstration Laboratory 

 Directed by Eve Kerr, MD, MPH, VA Ann Arbor VA Healthcare System (VAAAHS) in Ann 

Arbor, MI. 

VISN 20 PACT Demonstration Laboratory 

 Directed by Devan Kansagara, MD, Portland VA Healthcare System in Portland, OR. 

VISN 22 Veterans Assessment and Improvement Laboratory (VAIL) 

 Directed by Lisa Rubenstein, MD, MSPH, VA Greater Los Angeles at Sepulveda, Sepulveda, 

CA. 

VISN 23 PACT Demonstration Laboratory 

 Directed by Gary Rosenthal, MD, Iowa City VA Healthcare System in Iowa City, IA. 

September 11, 2013 Solimeo, Hamilton & True 2 
Balance 

September 11, 2013 Solimeo, Hamilton & True 

innovation standardization 

3 
Qualitative data draw insight from the field 

September 11, 2013 Solimeo, Hamilton & True 4 
Overview, Part 1 

Methodology, methods, and research-operations partnership. 

 Samantha Solimeo, PhD, MPH VISN 23 

 

Using mixed qualitative methods to evaluate implementation of 

PACT 

Alison Hamilton, PhD, MPH VISN 22 

 

Evaluating an Innovative Strategy for PACT Implementation 

and Spread: The Roles of Clinical-Research Partnerships and 

Mixed Methods 

 Gala True, PhD VISN 4 

 

September 11, 2013 Solimeo, Hamilton & True 5 
Cyberseminar qualitative methods resources 

September11, 2013 Solimeo, Hamilton & True 

 

Qualitative Data Collection: Introduction and 

Conducting the Interview (Part 1) and (Part 2). George Sayre, Jackie Grimesey Szarka 

 

Qualitative Research Proposals: Common Pitfalls 

(and Solutions). 

 George Sayre 

 

6 
Which is the “right” way to 

design a qualitative study? 

September11, 2013 Solimeo, Hamilton & True 

ethnography 

grounded theory complexity science 

 participatory/ action research 

phenomenology implementation science 

participant observation 

formative evaluation diary techniques 

material culture interviews

7 
Methodology or method? 

September 11, 2013 Solimeo, Hamilton & True 

From: Carter and Little. 2007. DOI: 10.1177/1049732307306927

8 
A wealth of qualitative data for 

understanding PACT 

September 11, 2013 

System team meeting agendas, 

PDSA, and minutes;

training materials;

PACT–related 

communications; 

implementation plans;

organizational charts;

PCMM/ Primary care 

Almanac; 

training attendance records 

Research site visit field notes;

interviews, group and 1:1;

PACT training field notes;

employee feedback web site

responses to open-ended 

survey items

Solimeo, Hamilton & True 9 
September 11, 2013 Solimeo, Hamilton & True 

processes 

 

methods 

 

dissemination 

10 

Qualitative research 

can be inherently 

collaborative 
September 11, 2013 

How do we understand why some PACT teams 

are more successful than others? 

Solimeo, Hamilton & True 11 
Using mixed qualitative methods to 

evaluate implementation of PACT 

 

Alison Hamilton, PhD, MPH VISN 22 

12 September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 
Veterans Assessment and Improvement 

Laboratory (VAIL) 

• Main research question: 

– Will systematic, technically-supported implementation 

of PACT features through evidence-based quality 

improvement (EBQI) methods, and the researchclinical partnerships they embody, be more successful 

than routine PACT implementation? 

13 September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 
• Evidence-Based Quality Improvement (EBQI) 

– A multi-level QI approach 

– Plan-do-study-act (PDSA) cycles 

• Quality improvement/system redesign 

– Based on a logic model 

• Social marketing 

• Leadership theory 

VAIL Theoretical Basis 

14 September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 
Phased innovation and quasi-experimental 

evaluation design: “Stepped Wedge” 

15 September 11, 2013 September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 

Phase 1 

•Target innovation 

practices in three 

facilities 

•Set up VAIL 

infrastructure 

Phase 2 

•Target next wave of 

innovation practices 

(+3 additional 

facilities) 

Phase 3 

•Spread VAIL to rest 

of Veterans 

Integrated Service 

Network (VISN) 

practices June 2010

FY12

Sept 2013
16

Patient Surveys 

Electronic 

measures 

Teamlet 

effectiveness 

evaluation 

Provider & Staff 

Surveys 

Economic 

Evaluation Implementation 

Evaluation 

Practice Surveys 
Implementation Evaluation Teamlet Effectiveness Evaluation 

Objective Evaluate VAIL and PACT 

implementation and impacts

Assess determinants of team 

effectiveness, changes in team 

composition and roles, 

organizational facilitators of and 

barriers to effective teamwork

Design Longitudinal convergent mixed methods (with other eval 

components): 4 waves of qual data collection

Sample Key stakeholders (VISN, VAIL, 

PC, and PACT leaders; clinic 

leads; VAIL Quality Council 

members, researchers)

Random sample of teamlet

members (PC providers, RNs,

LVNS, clerks/MSAs) across sites

Data collection • Semi-structured interviews

• Monthly brief web-based 

updates w/selected

stakeholders

• Continual document collection

& cataloguing

Semi-structured interviews

Qualitative Evaluation Components 

17

September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 
Wave 1 Wave 2 

Key stakeholder interviews n=58 n=47

Teamlet interviews n=41 n=43

Data collection and analysis to date 

• Wave 3 interviews underway (Sept-Oct 2013) 

• All interviews professionally transcribed 

• Preliminary analysis for rapid turn-around: 

−Templated transcript summariesmatrices

• Formal analysis: 

−Team-based analysis process

−Use of ATLAS.ti

−Hybrid deductive/inductive coding

18 September 11, 2013 Solimeo, Hamilton & True VAIL-PACT VISN22 
Evaluating an Innovative Strategy for 

PACT Implementation and Spread: 

The Roles of Clinical-Research 

Partnerships and Mixed Methods 

 

Gala True, PhD, CEPACT (VISN 4) 

 

Anneliese Butler, MSW and Mary Pelak, MSW, CEPACT 

David Macpherson, MD, MPH and Jen Skoko, MHA VISN 4 

September 11, 2013 Solimeo, Hamilton & True 19 
Timeline: Collaboration between 

CEPACT and VISN 4 PACT leaders 

September 11, 2013 Solimeo, Hamilton & True 

July 2010 

CEPACT Qual 

Core engaged in 

evaluation of 

National PACT 

spread strategies 

e.g., (Regional 

Learning Sessions, 

Learning Centers

~April 2011 

Partnership 

between VISN 4 

PACT 

Implementation 

Leadership and 

CEPACT Qual 

Core team

September 2011 

VISN 4 CMO 

began planning 

Virtual 

Collaborative (VC) 

to support PACT 

implementation 

and spread

October 2011 

PACT Learning 

Centers suspended

20 
CEPACT –VISN Collaboration: 

Serendipitous given changes in administration of 

national training 

September 11, 2013 Solimeo, Hamilton & True 

an important 

means of staying 

current on 

implementation 

climate 

facilitated 

relationships 

with frontline 

team members 

CEPACT VC primary 

means of 

bringing new 

teams onboard 

VC now a 

demonstration 

project for 

national 

implementation 

VISN 4 

PACT 

Leadership 

21 
VC: scale and elements 

Coatesville, PA

Erie, PA

Butler, PA

Pittsburgh, PA

VISN 4 Office

Clarksburg, WV

Wilkes-Barre, PA

Lebanon, 

PA

Philadelphia, PA

Wilmington, DE

Altoona, 

PA

Planning & oversight 

- VC Steering Committee meetings 

- VISN PACT Steering Committee 

- prep calls with presenting content 

experts and teams 

VC SharePoint 

 VC archives 

 Tools & resources 

Virtual learning sessions 

- 2x/month 

- schedules blocked per VISN directive 

- Alternating PACT & Clinical topics 

22 
VC: scale and elements 

• On-site coaching 

• Team assignments 

• Team meetings 

– Protected time (≥ 1 hr every 2 weeks), per VISN 

directive 

 

 

 

September 11, 2013 Solimeo, Hamilton & True 23 
Mixed methods, partnered research 

approach 

• Survey of Learner Experiences 

– fielded twice during first 9 months 

• Key contact phone interviews 

– baseline, mid-point, 9 months 

• Participant observation 

– Virtual Learning Sessions, VC Steering Committee 

calls, VISN 4 PACT Steering Committee calls 

• Process diary 

– AO to CMO , tracked implementation challenges and 

processes 

September 11, 2013 Solimeo, Hamilton & True 24 
September 11, 2013 

How do 

research-clinical 

partnerships grow and 

evolve over time? 

Solimeo, Hamilton & True 25 
Strengths and challenges of this 

approach 

September 11, 2013 

Challenges 

 • Survey design facilitated 

trust but limited ability to 

track change over time;

• difficulty getting finegrained picture at the 

clinic/team level;

• evaluation not designed 

to link findings to PACT 

performance measures

• Nimble and reactive to 

changing conditions in 

the field; 

• clinical partners open to 

negative findings & made 

changes in response to 

feedback; 

• researchers learned to 

turn around preliminary 

findings quickly & to 

identify actionable items

Solimeo, Hamilton & True 26 

Strengths 
Mixed methods, partnered research: 

lessons learned 

• Researchers must be prepared to go outside comfort zone 

• Qualitative investigators play a key role & require measure of 

autonomy to pursue unanticipated opportunities 

• Close involvement of others with relevant areas of expertise at 

the start enhances applicability of findings 

• Researchers and clinicians may be interested in different 

questions/goals, but both can be accommodated with careful 

planning 

• Benefits to both parties extend beyond one project 

September 11, 2013 Solimeo, Hamilton & True 27 
Which topics would you be interested in 

learning more about? 

• qualitative methodologies 

• practical, ethical, and policy implications of 

community-engaged research 

• human subjects standards for qualitative 

methods 

• application of medical anthropology theory to 

health services research 

• narrative methods in health services research 

 September 11, 2013 Solimeo, Hamilton & True 28 
Questions? 

Solimeo, Hamilton & True 29 
Acknowledgements: 

VISN 23 PACT Demonstration Lab 

VISN 23 Patient Aligned Care Team Demonstration Laboratory, Department of Veterans Affairs, 

Iowa City VA Healthcare System, Iowa City, IA 

Gary Rosenthal, MD Director 

David Katz, MD, MSc Co-Director 

Bonnie Wakefield, PhD, RN Co-Director 

Greg Stewart, PhD PI, Team and Formative Evaluation Core 

Sarah. S. Ono, PhD Director of the Qualitative Core 

Nichole Carroll, BA Research Coordinator 

Michelle Lampman, MA Associate Director of Operations 

Monica Paez, BA Qualitative Core Coordinator 

Kenda Stewart, PhD Health Research Science Specialist 

 

VISN 23 Primary and Specialty Medicine Service Line (P&SMSL) 

Jaime Bland, MS, RN HPDP/PACT Coord., VISN 23 P&SMSL, Omaha, NE. 

Marsha Herke, MPH, RD Asst. Admin. Officer, VISN 23 P&SMSL, Minneapolis MN 

Michael Koopmeiners, MD Medical Director, VISN 23 P&SMSL, Minneapolis, MN 

the instrumental participation of the VISN 23 PACT Collaborative team members. 

 

Center for Comprehensive Access & Delivery Research and Evaluation (CADRE), Department of 

Veterans Affairs, Iowa City VA Healthcare System, Iowa City, IA; 

Iowa City VA Health Care System, Iowa City, IA; and the 

Office of Patient Care Services. 

September 11, 2013 Solimeo, Hamilton & True 30 
Acknowledgements: 

VISN 22 Veterans Assessment 

and Improvement Laboratory 

Principal Investigator: Lisa Rubenstein, MD, MSPH 

Co-PI, Evaluation Lead: Elizabeth Yano, PhD, MSPH 

Innovation Leads: Lisa Altman, MD; Phillip Roos, MD; 

 Timothy Dresselhaus, MD VA HSR&D 

Study Project Director: Susan Stockdale, PhD Provi

Implementation Evaluation and Team Effectiveness Leads: 

 Alison Hamilton, PhD, MPH and Susan Stockdale, PhD 

Clinician/Staff Survey Lead: Lisa Meredith, PhD 

Economics Evaluation Lead: Jean Yoon, PhD 

Patient Survey Leads: Lisa Altman, MD & Jill Darling, BA 

Organizational Survey Lead: Stephanie Taylor, PhD, MPH 

Research Coordinator: John McElroy, BA 

September 11, 2013 Solimeo, Hamilton & True 31 

Center for the 

of Healthcare 

der Behavior
Acknowledgements: 

VISN 4 Center for Evaluation of Patient Aligned Care Teams 

(CEPACT) 

Organizations 

• Philadelphia VA Medical Center

• Center for Health Equity Research and 

Promotion (CHERP)

• University of Pennsylvania

• VA Healthcare-VISN 4 Offices

• VISN 4 Medical Centers

• Office of Patient Care Services

Individuals 

• Michael Moreland , FACHE

• David Macpherson, MD, MPH

• Jen Skoko, MHA

• PACT Implementers at VISN 4 sites

CEPACT 

• Rachel Werner, MD PhD Director 

• Judith Long, MD Co-Director 

• David Asch, MD, MBA Co-Director 

• Judy Shea, PhD 

• Anneliese Butler, MSS

• Anne Canamucio, MS

• Jennifer Gutierrez, MA

• Michele Lempa, DrPH

• Mary Pelak, MSW

September 11, 2013 Solimeo, Hamilton & True 32