QUERI Implementation Research- 10 -Department of Veterans Affairs

QIR041212

**Department of Veterans Affairs**

**QUERI Implementation Research**

**Cost Analysis in Implementation Research: Principles and Examples**

**Patsi Sinnott, PT, PhD**

**April 12, 2012**

Moderator:  Good morning, everyone.  Good afternoon, for those of you on the East Coast.  I’m an investigator with [Cyprus], and it is my pleasure to introduce Patsy Sinnott this morning for the Query Implementation Research series.  And, this lecture also serves as follow-up to the midlevel advanced curriculum from the Enhancing Implementation Science Conference back in the fall of 2011, for those of you who were participants in that conference.

Doctor Sinnott is a health economist with the VA Health Economics Resource Center — or the HERC — and she is also in Menlo Park, and she is a CHPT Core Associate.  She is also an assistant clinical professor in the UCSF San Francisco State University graduate program in physical therapy, where she teaches the health policy and evidence-based medicine sections in the Doctor of Physical Therapy program.

Her research interests include the evaluation of health care provider quality and efficiency performance; various aspects of workers’ compensation; and the economics of physical therapy and rehabilitation.

Before joining HERC, Doctor Sinnott was a senior manager at the Pacific Business Group on Health in San Francisco, where she was the project manager for an initiative to measure and report physician quality and efficiency performance.  While there, she coordinated the production of a report on the current practice in physician performance measurement in the commercial sector, funded by the Robert Wood Johnson Foundation.  Prior to joining that group, and prior to completing her doctoral program, she was a physical therapist council member of the State of California Industrial Medical Council, as well as a health care consultant and a practicing physical therapist.

She received her Bachelor’s degree in physical therapy from UC San Francisco, and received an MPH in health policy and administration, and a PhD in health services and policy analysis, with an emphasis on health economics — both from UC Berkeley.

And it is our pleasure to hear from her today during this cyber-seminar.  Thank you.

Doctor Sinnott:  Good morning – 

Moderator:  Patsy – 

Doctor Sinnott:  I’m not ready – 

Moderator:  (Inaudible) turn it over?  Okay, I’ll give you a moment.

Doctor Sinnott:  (Inaudible).  I have — I had a little meltdown on my laptop here, so it’ll be a second or two, but — I wanted to say good morning, and thank you very much for joining us.  I’m going totalk a little bit about all the different things you can do in implementation research, and the complications that implementation research bring you — I’m sorry, I can’t talk and type at the same time.

Moderator:  Not a problem.  I’ll remind our attendees that just joined us that today’s session, your mics and telephone lines have been muted, so you will need to type in all questions and comments using the Q&A function of your GoTo Webinar dashboard.  That’s located on the right-hand side of your screen, under “Questions.” Just type it in and press Send.  You can’t have more than one pending at a time, and I will respond when I can get to it, and we will be fielding all Q&A at the end of today’s presentation.  

Also, if you are having difficulty hearing at all, feel free to plug in a headset for clearer audio, or you can always call in using the toll line under the audio section as well.  

Doctor Sinnott:  Okay.  My screen is up, but I don’t seem to have your stuff up.  Is that okay?

Moderator:  You should be getting a popup right now.  

Doctor Sinnott:  All set.

Moderator:  Did you receive that?

Doctor Sinnott:  Yup.

Moderator:  All right.  And there we are.

Doctor Sinnott:  Great.  Good morning, again.  And thanks for joining us.  We’re going totalk about implementation research and the interesting (inaudible), and really, what we want to know is, what is the question you’re trying to answer by doing the cost analysis.

I’m having a little trouble with my – 

Moderator:  No problem.  Just click on the slide, and then you should be able to press Return or arrow down.

Doctor Sinnott:  Ah.  Okay.  Here we are.  Does the intervention work?  And how much does the intervention cost?  Those are key questions.  But also, how much does it cost to implement?  Will the intervention change over time?  Will the cost of the intervention change over time?  Can the implementation be made more efficient?  And can the intervention be made more efficient?

And then, other questions:  What programs can we achieve — what outcomes can we achieve with this program?  That means, are we looking at cost outcomes?  Are we looking at health outcomes?  How much does this program cost compared to others?  And what’s the value of this program compared to others?

All of these are questions that you have to think about when you start evaluating what kind of cost analysis and economic analysis that you’re going to be doing.

And then, who is — which programs should we choose, and who is the audience for this analysis?  Is it us?  You?  The payers?  Providers?  Patients?  If that’s the case, then you’re going to do something called cost identification, which is really to detail out the costs of the intervention.  

But the cost of the intervention is not the only thing of interest to policymakers and payers.  You might want to do a cost-effectiveness analysis when you’re looking for the incremental cost-effectiveness ratio, or the (inaudible) difference in value of two interventions.  Or you might be looking to do a budget impact analysis, where you’re really talking to the payers and policymakers about what it would cost to implement a new program.  And so that means not only the cost of the intervention, but also the implementation.

So, here’s some of the costs that might be included in your cost estimation and trying to figure out where to put these costs, or which ones should be included in the analysis, is part of your explaining process or your project development.  What are the costs?  Where do they go?  And, again, what costs where?  It’s up to you to set yourself up with an audience (inaudible) analysis and what cost, you can plot out what kind of a — what you need to plan for and what you need to do.

If your audience is patients, providers, payers, maybe what you want to do is the cost identification.  And the costs of including that are the costs of the intervention.

If what you’re looking for is really to explain to payers and policymakers how much it’s going to cost to implement a new program, because you found it to be effective and — in a clinic, then you’re probably going todo a budget impact.  But you’re not going todo a budget impact until after you’ve figured out whether or not the intervention is effective.  And so, if you’re going todo a budget impact, you’re going to look at the intervention costs plus the implementation costs and VA utilization if utilization costs are going to change, or you believe that the intervention is going to change VA utilization in the short run.

And then, if you’re going todo a cost-effectiveness analysis, because policymakers and payers need to compare the benefits of this program — the cost and value of this program — to other programs they might choose.  You’re going toinclude all the above, and the non-VA utilization, travel costs, and caregiver costs; and your indirects or facility costs.  In this lecture, we’re just going totalk about intervention costs and implementation costs to highlight what we’re talking about today.

So — the cost of the intervention:  If you — my experience working with clinicians — if I (inaudible) at that point — is that there a lot of things that we do, when we take care of patients or we define an intervention, that are natural or instinctive, and we don’t know that they represent a cost either to the system or to even develop the program.  

So, you really have to think about what is the intervention.  There are obviously going tobe personnel costs — and that includes clinician time and (inaudible); but there are also supplies; there’s durable medical equipment; their IT effects consequences— there’s equipment; there’s software; there’s space.  And then there’s maintenance of the software.  And there may be other things that you need to include in the estimation of the intervention.

And there’s three ways that you could estimate these costs, either by direct measurement or you observe or track activities.  A pseudo-bill — and you would use this if there were billing codes for the service that you were estimating the costs for, and you could assign costs to those billing codes.  Or you could do a cost regression, when the costs of care are known, or the accounting costs are fairly well matured, and you’re looking to look at the marginal difference in costs between two arms of the study of two interventions.

Another issue that you have to deal with in estimating costs is what data is available.  Obviously, if there are mature costs, you might use a cost regression.  But also, what methods are feasible; what’s going togive you the most flexibility; and which method will give you the greatest precision.  And the precision will be particularly important when you [guess] who’s looking at (inaudible).

So, in direct measurement, what you’re going todo is identify all the elements of the intervention, and track or observe these activities.  You’ll summarize the time or materials and supplies, and you see labor costs — estimate the value of the time, or other labor costs — possibly Bureau of Labor Statistics or other local costs and [part-time] the dollar per hour will give you the estimate of the intervention costs.

So, again, I just want to emphasize:  What is the intervention?  And what in the room, what in the space, what in the activities constitutes intervention, and how is it different from what is normally used to treat this patient, or what is considered usual care.

So, here’s some examples of data collection instruments that we’ve used in studies where we’re doing direct measurement.  So, in this case, this is a pilot document that we used to evaluate or set or test our data collection on time to do a CMP exam.  And you’ll see here that in this first draft, we tried to identify all of the other people who are involved in a single activity — doing a CMP exam, face to face, one provider with one patient.

So, you’ll see, the question number 1: We wanted to know if someone other than the examiner — who in this case is primarily a psychologist — was doing any chart extraction to pull history or records or pertinent information to the examiner; or would the — possibly the examiner is doing all of this.  And if somebody else helps, what is the license or credential of that person?  Is that license or credential or staff level going to predict their hourly rate — sorry — wage?  And then, again, in 1.2, you see there is a total amount of time that that person spent on the exam.  

And we’re creating a system where we’re defining the labor class of the person who is assisting, and then identifying the amount of time that that person is spending assisting with the exam.

And then we go into questions about — that the examiner completed:  How much time was spent in total in the exam?  And then, question number 3 is, “Did you perform the following activities?”  Actually, this question was changed into a time question:  “How much time did you spend on chart review?”  “How much time meeting face to face?”  “How much time preparing a report?”  Because the investigators were interested ultimately in finding out if there were differences in the intervention arm and the control arm, and how much time they spent in these various activities, and whether they were predictive, ultimately, of a more complete and accurate exam.

So, again, this is the first draft.  We pilot-tested it — I’m sorry, I’m going backwards — so, we pilot-tested it, and then we revised it prior to the initiation of the study.  And, just as an aside, we designed it to be used with Datafax, so that there was no manual download — or data entry.  Datafax is a form that’s set up so that it’s faxed and then scanned, and all the data is put directly into a faxed dataset.

So, this was the second page of that first exam example.  Again, trying to gather all of the information that we needed for estimating the time needed and the personnel needed to perform the exam.  

Here’s another exam — data collection — contact form — and just some highlights.  You’ll see that this is a single contact between a provider and a patient, and it includes all the attempts, which are at the bottom — how much time it took to be with the client; travel time; any expenses.  All of these things contribute to estimating the costs of the intervention, particularly if patients are being seen in their homes.  Imagine — let’s say that it was a telephone intervention — how many times you might need to contact the patient; whether that constitutes intervention time or not; just to highlight that it’s time consuming to track all of these activities.

You need to create forms with your participants.  You need to pilot-test them.  And then, in this particular case, the outreach workers’ manager reviewed these forms every week.  One of the issues you need to be aware of is that they — if you don’t document your time immediately, you will lose accuracy with every hour that passes.  So you need to create a system where this information is collected regularly, and in a timely manner.

And here’s a third example.  The — this is a study where (inaudible) vocational rehabilitation counselor is providing one-on-one services to a veteran.  You can see — at the top, you can see where the patient is — the voc rehab counselor is explaining who they’re talking to; how they contacted this person; how long a contact was; if it was a phone call or email; and then all the activities below that the voc rehab counselor might provide directly to the veteran.

And again, there’s always a way to summarize the time.  You need to create checks for the provider, so that they calculate the time, and that they make sure that the time in subgroups that they enter adds up to the time they think they totally spent.  And you need to take into consideration how the contact was made, and what you’re going todo about phone and email interactions.  

This is a CTRS template that was designed for this study, and it was implemented at all study centers.

So, now that you have all the time estimated in both arms of the study, how do you translate that into labor costs?  So, you summarize the time and you value the time by using some hourly rate that’s appropriate.  You can use VA labor estimates from this [FAQ] in a technical report on the HERC Web site.  This provides you an hourly rate currently through 2008.  And maybe there haven’t been any raises since 2008, so that might be helpful.

If these labor estimates do include benefits — which you do need to include when you’re estimating the cost of the intervention — you can use [federal] salary data and you can use BLS data — Bureau of Labor Statistics; or something else that is appropriate.

So then — the supplies, the equipment, the DME — you can use the NPPD, which is the National Prosthetics Patient Database.  This includes all of the, what are deemed prosthetics.  It’s almost all medical items for internal and external use.  It’s not just prosthetic arms.  You know, prostheses — prosthetics is defined as anything that helps improve patient function.  It could be stents; it could be hearing aids, as well as glasses.  So, all of these things are in this database.  And we have a guidebook on the HERC Web site on what these are.  And your local AMMS purchasing officer might be able to help you also.

So, then, a pseudo-bill you would use if the CPT coding is specified and has a [fake] validity inconsistently employed.  And I say that because we have had experiences where CPT coding which might define a fifteen-minute interval or a thirty-minute interval of time is at one facility always used as a fifteen-minute increment, while others use it consistently as a fifteen-minute.  So you have to watch out for that.

You can use Medicare or other payment schedules to estimate, using a utilization code.  So, for example, a patient might receive PT services for back pain.  The physical medicine CPT section that’s used — and I just want to highlight this as a physical medicine, not physical therapy code section, which means that many people use — many types of providers use this code section — not just physical therapists.  

So, for forty-five minutes’ worth of treatment, you might have three code standard use; but because this  physical medicine section is used by many providers — physicians, DOs, chiropractors, and kinesiotherapists and occupational therapists — you don’t know just from the code who is the provider or where it was provided.  So, I’m just highlighting this so that you know.  It sounds simple, but it may not be.

So, here are some caveats that we’ve learned.  You want to watch for double counting.  For example, even though clinicians may be completing the survey tools or the instruments that you’re providing them to microcost the intervention, they may also be required to enter CPT codes, as well as completing the CPT codes for their time.  Which means that, if you end up pulling utilization data, the CPT codes with their time may also end up in the utilization data or the administrative data, and might cause a problem.  

And also, watch for the inconsistent utilization of the codes across VA medical centers — and again, the thirty- to fifteen-minute issue.  Just for an example, on the voc rehab, we — this is a summary extraction that was used to get the weekly work from the VRC CPRS template for data entry purposes.  So, this form, 14, provided a weekly detail of the services provided by the VRC.  But if you look over here, you’ll see there are also procedure codes, and we discovered that the procedure codes — CPT codes — were also being entered into the data.  So, if we did not extract the CPT codes out of the utilization data when we estimated the total cost for the budget impact, we would’ve been double-counting the costs of the intervention.

So, the third way in these methods is to use a cost regression.  But it’s difficult to do this when the data exists for a current (inaudible), when you’re estimating a new intervention, you really need to get the detail of the new intervention, because the accounting cost is not going tobe mature enough to really understand the cost of the intervention.

So, now, we’re here (inaudible) the idea of the cost of the implementation.  So, if everybody’s really clear, what is the intervention cost — but again, what is the question?  Does the intervention work?  How much does it cost?  Here we are at how much will it cost to implement.  Will the implementation costs change over time?  Can the implementation be more efficient?  And how generalizable are the study costs, the implementation costs, across your system?  And again — what data is available?  Which is feasible?  You want analytic (inaudible) flexibility in the greatest (inaudible).

And, again, from previously, you can do direct measurement; you can do pseudo-bill; you can do the cost regression.  But remember — pseudo-bill you also want to do if there are no codes for the implementation, with (inaudible) validity and appropriate payment system for converting activities into costs.  

And you can only use a cost regression when the labor time cost is already known, and you’re looking to identify the marginal [in-treatment] cost.  The last two items are not likely to be useful when looking at implementation costs.

So, the question is, in implementation — how do we get from here to there?  How do we complete the arch from the idea to the balanced intervention?  What are the elements of adoption?  And how do we sustain the adoption of the intervention?

So, our [premium] implementation goes right alongside the planned use study action format for developing the implementation of a study.  You want to create the plan; define what your leadership engagement is; [physician] participation is; who are your support personnel; and what the technical experts are; how much training is there involved; and is it just for clinicians, or are there support and administrative staff training that needs to be done.  And how are you going tocollect this information?  Are you going touse diaries?  Can you put it into CPRS?  Would you use recall surveys, or someone observing what the implementation is?  Can you extract data from (inaudible)?  Are there other methods that you can use?

And then, how frequently are you going to gather this information on the implementation?  (Inaudible) you want to know whether the times (inaudible) — is the implementation going to be constant at every site, or will it change from site to site?  Will the cost and implementation become more efficient?  You could think through all of these questions as you develop your implementation and cost estimation for the implementation.

So, then the question is — and this is an issue that I’ve had with several investigators — the question being, is it an intervention or an implementation?  Can you tell where one starts and the other begins?  And then, finally — I’m just highlighting some of the issues for the implementation costs — where they belong — with physician training and intervention?  [For] part of the intervention, will every clinician need to be trained?  Will there be coaching, and how will the coaching occur?  Is this implementation, or is this the intervention?

How do you detail the time of the coach?  And then, how will you estimate the costs of ongoing leadership involvement, and those people who are going to be coaching or contacting the management to help keep this going and sustain the activity.  

And then, finally, you know, is it implementation, or intervention, or research?  And I just highlight this to emphasize that you need to be able to think through each of these questions and separate them, because of the implications for how much you’re going to gather the cost; are you going to bundle the cost; and what kind of analysis you’re going to be doing.

Again:  Will the costs change over time?  And then, a quick summary.  We have been through the intervention costs and the implementation costs.  This gets us to the edge of doing a budget impact analysis, but not completely.  How are you going to maintain the implementation?  

And I just have some references here.  The first reference is a detail from the implementation study on the TIDE project, which gives you a very clear and detailed explanation of how they estimated the costs for the intervention.  

And then the last two papers give you the framework for estimating the cost of the intervention, and in particular, Todd and Mary Goldstein’s paper on behavioral interventions and the conceptual models we need for estimating the cost of the intervention.

So, with that, I’m open to questions.

Moderator:  Thank you very much.  It sounds like we do (inaudible) — I’m sorry.  If anybody has their computer speakers unmuted, please mute them now.  I’m getting a little bit of an echo.  For anybody who joined class [at the top of the] hour, please just open up the Q&A function on your GoTo Webinar dashboard to submit any questions you have.  And we do have three pending ones.  Brian, would you like to read those aloud?

Brian:  Actually, I’m not able to see them.  (Inaudible).

Moderator:  (Laughs).  Let me email those over.  Or — I heard the email’s down in LA.  Should we just — should I just read them aloud?   

Brian:  Actually — give me one second, and I can go ahead and — okay.  I’m set.  So, Patsy, again thank you — and I want to go back to something that you mentioned briefly, and we talked about in the past, that has been raised by participants — and that is the existing HERC guidelines for economic (inaudible) implementation research.  From a number of years ago, through discussions, HERC had developed semiformal guidelines for conducting (inaudible) implementation research.  The question is whether those are still current and still available.

Doctor Sinnott:  They’re still available.  They’re still current, and they’re due for a review this year.  So, I think they’re available through Query and through the HERC Web site.

Brian:  And when those are updated, there’ll be an announcement presumably if this comes around?

Doctor Sinnott:  Yes.  Of course.

Brian:  Okay.  And then, on a similar note, there was also an article that was published in the Query series in the Journal of Implementation Science that covered many of these issues.  So, the question is the same — is that material still current?  Is that set of guides consistent with the presentation you’ve made today, and some of the newer ideas that are emerging, in terms of how to go about examining cost issues (inaudible) implementation science.

Doctor Sinnott:  Well, you know, yes — that article that was Mark and Paul’s article on economics and implementation science.  Yes, it’s still current.  You know, I think the biggest movement right now is in what a budget impact analysis really means, and can you — now I’m going todrop into kind of economics lingo — but, the question is, can you really estimate the production function of an intervention, and then can you — are there elements of the production that are modifiable?  In other words, can you do a budget impact analysis and identify areas where you can make it more efficient — make the intervention more efficient, or make the implementation more efficient.  So, that’s really where the movement is, to my mind.  I don't know if there are any other folks — I think Tod’s on the phone — who want to comment on that.

But the guidelines from the implementation science article are still relevant.  There are guidelines for budget impact analysis from the journal — the International Society of Pharmacoeconomics and Outcomes Research has guidelines for budget impact analysis.  And then, these three articles also are appropriate guidelines for doing the cost estimation.  

Brian:  All right.  Okay.  Next question that actually relates to some broader confusion within the field of implementation science.  The question is, when we study the effectiveness of a specific implementation or QI strategy or program, or a QI intervention, can we view that as any other type of health services intervention, in terms of how we conceptualize and measure costs?  And the confusion here that I alluded to is the fact that we are implementing a health services intervention or a clinical intervention.  

You know, the way that you presented the material today, it distinguishes between what we do to implement that intervention and the intervention itself.  But we often see research that is focused on evaluating a QI or an implementation intervention or implementation program.  And the researchers will view that as an intervention, and evaluate it in much the same way that they would a clinical intervention.

So, rather than implementing an implementation intervention or QI intervention, you know it’s one and the same.  I’m not sure if I’m clarifying or further confusing (inaudible).

Doctor Sinnott:  Well — here’s how I conceptualize it — you need some evidence that the intervention is effective.  And the big question is, if it’s effective in Clinic A — and let’s say it’s effective in Clinic A and B — is it effective in Clinics C, D, E, and F?  And — so that’s really the question about the intervention and intervention effectiveness.

I see the implementation part of it as a separate health services research question.  And, therefore, I look to cost it separately.  And it’s my impression that that is the — focus of implementation is that it is a separate thing.  Am I correct?

Brian:  Yes.  And again, it does get back to this broader confusion as to how to label better what we are studying and evaluating, and if it’s implementation intervention, then it might combine the implementation process into the intervention itself.  So, I think that we should leave this and go on to the next question, just to get recognized.  

Doctor Sinnott:  Okay.

Brian:  Because there’s some broader confusion, in that ultimately it’ll need to be handled on a case-by-case basis.

Doctor Sinnott:  Okay.

Brian:  So — the next questioner points out that some of the costs — for example, labor cost estimates — would’ve been incurred anyway, under usual care.  How does one account for that, so that what is captured is the marginal cost of the new intervention?

Doctor Sinnott:  Well, because you’re presumably comparing the cost between the usual care and the intervention arm.  And you will, of course, get the marginal increase in cost.  You’re — the questioner is correct.  You would not be trying to estimate the cost, in isolation, of the usual care.

Brian:  So, it’s a matter of correctly assigning each of those to one of those two categories.

Doctor Sinnott:  Right.  And — well, and that you would be looking at the same clinician’s activities in the usual care arm.

Brian:  Okay.

Doctor Sinnott:  Does that make — 

Brian:  So, our next question — pardon me?

Doctor Sinnott:  I just wondered if that answered the question.

Brian:  Yeah.  I think so, and again, I think it’s ultimately — as with many of these — will need to be handled on a case-by-case or project-by-project basis, to think through each of the distinct categories of cost, to try to determine whether they are likely to be present in the usual care comparison or pre-intervention, versus those that are marginal and directly attributable to the new intervention.

Doctor Sinnott:  Right.

Brian:  Next question:  What do you foresee in regard to payment sources to cover implementation costs, given the current challenges of paying for interventions themselves?

Doctor Sinnott:  Would you say that again?  

Brian:  Yes.  And this is not a question that you necessarily would use economic expertise and research expertise to answer.  The question is, What do you foresee in regard to payment sources to cover implementation costs, given the current challenges of paying for interventions themselves?

Doctor Sinnott:  I think, Brian, you’re a better respondent to that question.

Brian:  Yeah, I think each of us would say we’re not sure, although I suppose one answer that I would offer is that, you know, to the extent that this type of research does demonstrate the value or the cost effectiveness [and so on] of the new interventions, and that the costing of those interventions takes account of implementation costs.  That’s the kind of data that leaders would need in order to justify paying for these.

Doctor Sinnott:  Right.

Brian:  But I think — 

Doctor Sinnott:  — These are policy questions about adoption — new care models, not reimbursement models.

Brian:  Sure.

Doctor Sinnott:  Or service.

Brian:  Right.  Okay.  Next question:  “I am not inside the VA system, so I am not familiar with some of the IT systems.  However, one of the measurement tools appear to be on line, and I am curious — is this a simple MS patched as data entry form, or is it prepared in a proprietary application?” 

Doctor Sinnott:  In the examples that I gave today, there are pieces of paper.  There are — and a CPRS or, you know, a patient charting template.  I’m also involved in a study right now where the activities are all entered into an Access database.  So, the answer is, all of the above.

Brian:  Okay.  One comment, and then I’ll read another question.  The comment is, “Thank you for the important distinction between costing interventions and costing implementations separately.” Next question:  How or where to find out more about Datafax as a data-gathering tool.

Doctor Sinnott:  You have to search on line.  From my perspective, it’s a wonderful system to eliminate the potential problems of manual data entry.  And it was used for a long time in the VA in the clinical sciences program — the cooperative studies program.  But, unfortunately, you’ll just have to look for it on line.  I don’t have a reference.

Brian:  Okay.  Thank you.  (Inaudible) that I’ve seen as far as submitted questions.  

**[End of Recording]**

Cambridge Transcriptionswww.ctran.com

Cambridge Transcriptionswww.ctran.com