Making sense of the ideal and the real in 

PACT implementation using qualitative 

research 

 

Part II: The importance of being there 

HSR&D PACT Cyberseminar 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 1 
How do we understand why some PACT 

teams are more successful than others? 

• 1:1 interviews

• focus groups

• observation

• material culture

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 2 

Image credit: http://tinyurl.com/m24eyuv
Qualitative data draw insight from the field 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 3 
October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 4 

Focus group participants’ feedback 

This opportunity to meet with our peers proved the most beneficial event 

of this conference. Please continue this. It was very helpful to share 

problems and experiences that our fellow clinics have had. 

This was greatly appreciated, to hear how other peers are working, 

struggling, growing. Very helpful! Safe environment with peers. 

Very beneficial to be able to meet with other RNs at CBOCs. 

This was awesome. I have felt all alone. I learned I am not. 

I have a better concept of where we are heading. 

Great pathway to communicate to VISN management. 
October 16th, 2013 5 

 

PACT transformation 

Solimeo, Forman, Harrod, Tuepker & Ono

Professional 

identity 

 

Organizational 

culture 

 
October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 6 

You get what you pay for: 

Selected benefits of “being there”

Rapport and relationship 

Out of sight, out of mind 

Seeing is believing 

Contextualizing performance 

Belief vs behavior 

Meaningful recruitment 

Purposeful sampling 

 
Getting to the Nuts and Bolts: Using Qualitative Methods to Evaluate a 

Locally-designed PACT Teamlet Coaching Model 

 

Jane Forman, ScD, MHS Research Scientist and Director Qualitative/Mixed Methods Core, VA Center 

for Clinical Management Research (CCMR) 

Qualitative Evaluation Group, VISN 11 PACT Demonstration Lab 

VA Ann Arbor Healthcare System 

Molly Harrod, PhD Research Scientist, VA Center for Clinical Management Research (CCMR) 

Qualitative Evaluation Group, VISN 11 PACT Demonstration Lab 

VA Ann Arbor Healthcare System 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 7 
Setting 

 

 Ann Arbor VAMC 

 Primary Care Clinic 

 

 Clinic Structure 

– Over 20,000 patients 

– 20 Teamlets 

– 70 PCPs and Residents = 20 FTEE 

 

 Average of 3.5 PCPs per teamlet 

 At least 1 resident per teamlet 

 

October 16th, 2013 Solimeo, Forman, Harrod Tuepker, & Ono 
History: 

• PACT implementation 

slow; took time to build 

the foundation so 

practice redesign could 

occur. 

• PCPs and staff had little 

opportunity for 

experiential learning. 

• Coaching sessions 

established to help 

teamlets develop and 

implement new work 

processes.

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 9 

Image credit: http://tinyurl.com/m24eyuv
Locally-designed PACT Teamlet Coaching 

Model 

Developed and delivered by clinic leadership; consistent with 

national requirements 

6 coaches •Including the Director of Primary Care and the Associate Chief of Staff for Ambulatory 

Care 

•Coaches meet quarterly 

Brings each teamlet together for 1-2 sessions (2 hours) per month 

Sessions are agenda-based with the current clinic-wide goals 

( e.g., to address access, continuity) 

Teamlets develop and pilot QI interventions 
Evaluation 

• Designed with input from clinical operations leadership 

• Formative evaluation with an eye to program 

improvement and spread to other sites 

• Looking at: 

– Session content and attendance 

– Teamlet interactions 

– Development and implementation of new work processes 

over time 

– Spread of best practices 

– Effect on coaching sessions on part-time provider 

engagement and PACT metrics 

 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 11 
Methods 

• Longitudinal observation of 9 out of 20 teamlets 

– 5 different coaches; 2 in leadership positions 

– 16+ hours of observation to date 

• Planned: semi-structured interviews with teamlet members 

and coaches 

• Outcomes measurement 

• Today’s presentation: Preliminary analysis of observation 

data, focusing on teamlet interactions during coaching 

sessions. 

 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 12 
Findings 

• Teamlet members learn from each other 

• Building relationships within the teamlet 

• Spreading best practices across teamlets

• Teaching and learning with coaches

 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 13 
Teamlet members learn from each other 

• Learning about each others’ roles 

• Working out new work processes 

• Taking the initiative 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 14 
Learning about each others’ roles Working out new work processes 

Taking the initiative 

 

Fieldnote: 

PT MD- what tasks do you enjoy doing, or what should I be giving to you? 

LPN1- I would like to not have to turf everything over to RN. I can’t even take 

blood pressure (BP) readings over the phone. I can’t document them. I can’t assess 

them or offer my opinion. 

LPN2- wouldn’t that help you RN1? (RN1- oh yeah). They act like we don’t know 

what a high BP is or what a high glucose is. 

LPN1- if I take someone’s blood sugar and even if it is normal, I have to hand it 

over to an RN. 

MD1- so you can’t call, take the information and put it in a note? (LPN1- nope). 

Clerk- they put us in such a bind, we can’t do anything. 

LPN1- why can’t I put numbers in and then co-sign RN1?

PT MD- I think you should be able to. You’re just documenting what the patient is 

telling you. You do that when they are in clinic. 

MD2- but will you get into trouble if you just do it? 

Coach- that’s what we have to find out. 

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 15 
Learning about each others roles 

Working out new work processes Taking the initiative 

Fieldnote: 

MD1-...I’ll work on a list like we did for labs that has my 

most used injections and what I usually give.

LPN1- if you could look at the schedule that day and 

write down any injections that you want I can have them 

ready and that way, you don’t have to go searching for 

the injection tray. We have a bunch of new Residents so 

you never know where they put things. It messes with the 

flow of the clinic. 

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 16 
Learning about each others roles 

Working out new work processes 

Taking the initiative 

Fieldnote: 

LPN1- what I would like to be able to do is print out the medicationreconciliation form beforehand, call the patient and tell them I am 

putting it in the mail and to be sure to include all over the counters, 

vitamins, creams, and then have them bring it in with them.

PT MD- LPN1 has picked up adherence issues with patients before I 

even did. She’s really good at this. 

Coach- that’s a really good idea. That would help because it takes 20 

minutes of the visit to go over it with them. 

MD1- it takes sooo long. This would help so much. And then we can 

have all the LPNs do it! (Laughter). Is this beyond your scope of 

practice? 

LPN1- no, it is not. 

Coach- it shouldn’t be because they do this on check-in.

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker, & Ono 17 
Building relationships within the teamlet 

• Affirmation within the teamlet 

• Advocating for teamlet members 

• Flattening the hierarchy 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 18 
Affirmation within the teamlet Advocating for teamlet members 

Flattening the hierarchy 

Fieldnote: 

MD- she’s (LPN) very good, she’s very efficient. It is 

literally impossible for me to do my job without these 

two (RN and LPN).

Fieldnote: 

MD2- having a dedicated LPN has been great.

MD1 to LPN1- you are making a difference. You’re 

getting rave reviews from patients.

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 19 
Affirmation within the teamlet 

Advocating for teamlet members Flattening the hierarchy 

Fieldnote: 

MD2- I’d be happy to facilitate the LPNs doing the BP 

teaching. Telling them (primary care leadership)things 

are going great and it’s working well. It’s about 

reallocating our resources. LPNs shouldn’t be doing all 

check ins and check outs. Doing patient education is 

totally appropriate.

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 20 
Affirmation within the teamlet 

Advocating for teamlet members 

Flattening the hierarchy 

Fieldnote: 

LPN1- that day you had a pap, I should have had 

everything ready beforehand so now I’m going to look 

ahead and have it ready. I had to go to the Women’s 

Clinic to get everything and that’s why it took so long.

MD2- I don’t even know what the procedure is for doing 

paps. 

LPN1- maybe on a slow day, we can sit with [Women’s 

Clinic] and go over everything. 

MD2 to LPN1- we’ll work on that together. 

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 21 
Conclusions: 

• Use of observation allows us to see process-in-action: 

– Document how change occurs - interviews can only give us an 

account-as-told, and not details of processes involved as they 

happen. 

– See the process through which all teamlet members are both 

teachers and learners. 

– Understand the nuances involved in teamlet transformation. 

• Delineating roles and responsibilities 

• Flattening hierarchy 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 22 
Using Qualitative Methods to Identify and 

Pursue Unanticipated Research Questions: 

Examples from VISN 20 Demo Lab’s Emerging Focus 

on Performance Metric Use and Development 

Anaïs Tuepker, PhD, MPH 

Core Investigator, Health Services R&D, Portland VAMC 

Assistant Professor, Division of General Internal Medicine, 

Oregon Health & Science University 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 23 
• Initial Demo Lab focus on PACT innovations to 

improve care for patients with complex chronic 

conditions 

• Focus groups and interviews with 242 employees in 

15 Primary Care clinics; no questions about 

performance metrics 

• Loose use of the Consolidated Framework for 

Implementation Research (CFIR) & PACT pillars 

to develop initial codebook 

• Iterative coding process allowed codes to change 

• Analysis across codes pulled together findings 

related to performance metrics 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 24 
“...on one hand we say we want the PACT team, we 

want to have more phone clinic, we want to have 

more kind of non face-to-face stuff so we can manage 

these people better...on the other side of the coin, it’s 

like, “Boy, that veteran really wants an appointment, 

and you had better give it to him on the day he wants 

it.” And so everybody gets an appointment and all 

the slots fill up...you’re given two directives that really 

do not, they just bounce off each other, you can’t 

make that fit. [PCP, Site C] 

Metrics are sometimes in conflict 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 25 
We started scrubbing our panels. I changed the 

ratio of in-person to telephone calls, things of that 

sort. Then we get this backlash from 

administration that we’re not allowing patients to 

come in... we received this memorandum that we 

will see a certain number in-person and we will do 

a certain number of phone calls, that it will be that 

way. [PCP, Site K] 

Metrics are sometimes in conflict – and 

who sets the priority? 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 26 
On one hand we’re saying we’re a PACT team and 

we’re a team, but ... we’re using...the individual’s 

performance as the metric for how well the team is 

doing....[So]to be able to go and say the PACT 

model doesn’t work because these people haven’t 

made their metrics, where’s the breakdown? Is it the 

individual provider that’s not really providing 

leadership and guidance, is it the NCM who’s not 

really good, is it that people don’t understand their 

roles very well? (PCP, Site N) 

Metrics are missing something essential 

about PACT (team based care) October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 27 
I’m also frustrated, when his [PCP’s] numbers are 

clean and better, he gets the pat on the back for it. 

And they just say, ok, well now that you’ve got 

that done, here’s what else I need you to do. And 

so that’s very frustrating for me too. Because, 

what am I chasing numbers for? [Clinical 

Associate, Site B] 

Metrics are missing something essential 

about PACT – and have unintended 

consequences (staff satisfaction) 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 28 
And then the clinic also has these things that say, “Well, 

oh, gee, nobody has um, this guy hasn’t had his cholesterol 

checked in 150 days. Let’s get that to your team to add.” 

And then that generates an appointment. Whether the 

patient even needs to have his cholesterol, or his A1c 

checked, or any of that nonsense. There’s still those little 

boxes that we’re still expected to check. And the patients 

are expected, either by themselves or by the clinic 

managers to come in once a year, whether or not they need 

an appointment. That’s a system thing. So we have this 

system, these system requirements that don’t speak to the 

PACT model. [PCP, Site C] 

 

Metrics might not be aligned with 

PACT (“is it patient centered?”) 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 29 
I think we’re reaching to some of those core 

performance measures, that we’re able to help our 

providers reach some of those goals and attain them. 

But it’s almost like, with some we’re able to, but 

others are being left behind. Other patients are being 

left behind. It’s like we can focus on some of them 

[but] not all of them.... it feels like maybe our 

diabetics and our congestive heart failures, 

COPDers, those kind of people, we’re attaining 

those goals with them. But it doesn’t seem like we’re 

giving the same amount of attention and care to all of 

them across the board. [Clinical Associate, Site D] 

M etrics carry an opportunity cost 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 30 
Improving metrics: what to do, why to 

do it, and how to do it 

Align 

Avoid conflicted 

goals

Identify opportunity 

costs

Tools to assess 

impacts across PACT 

pillars

Streamline 

Co-ordinate priorities

Identify opportunity 

costs

Communication 

tools

System 

transparency 

Redefine Capture PACT 

concepts

“New” work of PACT 

defined

Team performance measures

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 31 
Next steps 

• Interviews with teams about specific impact of 

actions intended to address performance metrics 

• Interviews with patients: how would they measure 

team performance? What matters to them? 

• Interviews with VISN and facility leadership: 

examining processes of metric selection and 

dissemination 

• More focused interview guide, same process of 

directed-to-inductive coding 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 32 
Some parting comments 

• Inductive and directed/theoretically driven 

analytic approaches are not mutually exclusive: 

much health services research combines both 

• Validity of qualitative themes comes through 

consensus; involve multiple researchers and 

perspectives (double coding, iterative 

discussion, participant validity checks) 

• The findings don’t emerge, you work at them. 

 

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 33 
Qualitative Insight into PACT 

Dynamics and the Role of Clerks 

Sarah Ono, PhD 

Qualitative Core Director, Center for Comprehensive 

Access & Delivery Research and Evaluation (CADRE) 

Co-investigator, VISN 23 PACT Demonstration Lab, 

Formative Evaluation Core 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 34 
Working With Qualitative Data 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 35 
Challenge of Collecting Clerk Data 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 36 

• Challenges to collecting data on 

Clerks: time and willingness

• Participation in research may 

mean something different to 

different role

• Nature of the interaction changes 

based on perceived status or 

power dynamic

Primary Care 

Provider

(MD, DO, APRN, 

NP, PA)

Clinical Associate

(LPN, LPN)

Clerical 

Associate

(MSA, PSA)

RN Care 

Manager

Patient 

Panel
One of these things is not like the others 

• Why people become Clerks is different than other team members.

• Clerks are the only role on a PACT team without clinical 

licensure. 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 37 
Finding #1: Clerks report 

high workload and low status 

Clerks need to perform a broader range of tasks, 

including: panel management, data collection, 

triaging phone calls, screening secure messages, and 

sending faxes

Clerks are viewed by PACT team members as being 

“overwhelmed at front desk” and “too busy to do a 

great job,” but also needing to “be more involved in 

data collection” and to provide “better customer 

service.” 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 38 
Finding #2: Spatial dislocation and timing of 

tasks leads to feeling disconnected 

“[Clerks] would interact more with rest of team if not spatially apart.”

 - PACT Provider

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 39 

• Out of sight, out of mind

• Clerk role creates isolation is two ways:

• Physically removed from team

• Time and tasks

Clerks are the “face of the office”

Well-incorporated Clerks make valuable 

contributions to their teams.
Being There as a Method 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 40 
Being There as a Method 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 41 
Which aspects of qualitative observation methods 

would you most be interested in learning? 

 Different approaches 

(e.g. direct, participant, 

structured, etc) 

 Selecting the best approach for 

your study 

 Strategies for gaining entry and 

developing rapport 

 Fieldnote recording and 

analysis 

 

 October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 42 
Questions? 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker & Ono 43 
For more information 

please contact individual presenters 

 

October 16th, 2013 

Samantha L. Solimeo 

“The importance of being there” 

Jane Forman and Molly Harrod 

 “Getting to the Nuts and Bolts: Using Qualitative Methods to Evaluate a 

Locally-designed PACT Teamlet Coaching Model” 

Anais Tuepker 

“Using Qualitative Methods to Identify and Pursue Unanticipated Research 

Questions: Examples from VISN 20 Demo Lab’s Emerging Focus on 

Performance Metric Use and Development” 

Sarah S. Ono and Samantha L. Solimeo 

“Qualitative Insight into PACT Team Dynamics and the Role of Clerks” 

Solimeo, Forman, Harrod, Tuepker & Ono 44 
Acknowledgements: 

VISN 23 PACT Demonstration Lab Formative Evaluation Core 

VISN 23 Patient Aligned Care Team Demonstration Laboratory, Department of Veterans Affairs, 

Iowa City VA Healthcare System, Iowa City, IA 

Gary Rosenthal, MD Director 

David Katz, MD, MSc Co-Director 

Bonnie Wakefield, PhD, RN Co-Director 

Greg Stewart, PhD PI, Team and Formative Evaluation Core 

Nichole Carroll, BA Research Coordinator 

Michelle Lampman, MA Associate Director of Operations 

Monica Paez, BA Qualitative Core Coordinator 

Kenda Stewart, PhD Health Research Science Specialist 

 

VISN 23 Primary and Specialty Medicine Service Line (P&SMSL) 

Jaime Bland, MS, RN HPDP/PACT Coord., VISN 23 P&SMSL, Omaha, NE. 

Marsha Herke, MPH, RD Asst. Admin. Officer, VISN 23 P&SMSL, Minneapolis MN 

Michael Koopmeiners, MD Medical Director, VISN 23 P&SMSL, Minneapolis, MN 

the instrumental participation of the VISN 23 PACT Collaborative team members. 

 

Center for Comprehensive Access & Delivery Research and Evaluation (CADRE), Department of 

Veterans Affairs, Iowa City VA Healthcare System, Iowa City, IA; 

Iowa City VA Health Care System, Iowa City, IA; and the 

Office of Patient Care Services. 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker, & Ono 45 
Molly Harrod, PhD, Qualitative Evaluation Group 

Claire Robinson, MPH, Qualitative Evaluation Group 

Jessica Ott, MPH, Qualitative Evaluation Group 

Myra Kim, ScD, Statistician 

Bree Holtz, PhD, Navigator System Development 

Wendy Morrish, BSN, RN, Navigator System Development 

Darcy Saffar, MPH, Project Manager 

Jennifer Burgess, MPH, Research Associate 

Ann Arbor VA Healthcare System Management, Providers, and Staff 

Acknowledgements: 

VISN 11 Patient Aligned Care Team Demonstration Laboratory 

(PRIISM ) Department of Veterans Affairs, Ann Arbor 

Healthcare System, Ann Arbor, MI

Eve Kerr, MD MPH, Principal Investigator and Director, Center for Clinical Management Research 

C. Leo Greenstone, MD, Co-Principal Investigator and Associate Chief of Staff for Ambulatory Care 

Sarah Krein, PhD, RN, Co-Principal Investigator 

Jane Forman, ScD, MHS, Co-Investigator and Lead, Qualitative Evaluation Group 

Denise Ramsey, MSN, RN, Co-Investigator and Associate Chief Nurse, Ambulatory Care 

Adam Tremblay, MD, Co-Investigator and Director, Primary Care 

Caroline Blaum, MD, MS, Co-Investigator 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker, & Ono 46 
Acknowledgements: 

VISN 20 Patient Aligned Care Team Initiative Demonstration Lab 

Devan Kansagara, MD Co-Principal Investigator 

Michael Alperin, MD, MBA Clinical Director, Primary Care Division, 

Portland VA Medical Center; Co-Principal Investigator 

 

Anaïs Tuepker, PhD, MPH , Co-Investigator 

Annette Totten, PhD , Co-Investigator 

Christina Nicolaidis, MD, MPH , Co-Investigator 

Sandra Joos, PhD , Co-Investigator 

Jeannette Richardson, MSRN, CNS , Clinical Nurse Specialist 

Eleni Skaperdas, BA , Research Assistant 

Jessica Robb, MPH , Research Assistant 

Jesse Byrd, BA , Research Assistant 

Suzanne Denker, MHA , Demo Lab Coordinator 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker, & Ono 47 
Disclaimer 

The views expressed in this presentation are 

those of the authors and do not necessarily reflect 

the position or policy of the Department of 

Veterans Affairs or the United States 

government. 

 

October 16th, 2013 Solimeo, Forman, Harrod, Tuepker, & Ono 48