**Transcript of Cyberseminar**

**HERC Cost Effectiveness Analysis Course**

**Estimating the Cost of Intervention**

**Presenter: Todd Wagner**

**September 26, 2012**

**Estimating the Cost of Intervention**

**September 26, 2012**

Todd Wagner:So welcome everybody to today’s class in the HERC Cyber Seminar Series. What I am going to be talking about today is something that I know a lot of people are interested in which is estimating intervention cost. You have developed this beautiful new intervention that you believe works and you are trying to figure out how much value this thing adds. And so part of that is the cost question, what is the cost of implementing this beautiful new intervention that you have designed. So at the – I always struggle to figure out how to move these. Okay so at the end of the class, and I can believe I can go into – there we go.

Heidi Schleuter:You know Todd, if you go into pledge someone right there – perfect.

Todd Wagner:Then you are not actually seeing all the other windows that I have open, right. So the objective, at the end of the class, you should, I am hoping, understand what micro costing means. We will be using that term throughout the class, it is a method. You would be familiar with different micro costing methods, they are actually different ways to get there. And then, understand that the method you use will affect your future analyses and what you can do with them. And I will explain that in a behavioral trial that we have done. 

So the perspective of the course, and it’s going to continue with this class, is really focused on an estimating cost for cost effective analysis using a societal perspective. So if you are out there and you are an implementation researcher, keep in mind that you might have to do slightly different things for estimating cost. You might be only interested for example, in the variable cost, not necessarily the overhead costs. We have a separate lecture downstream that Patsy is going to give that is going to be much more on implementation research. So keep in mind, and I will try to highlight as we go through, how these things would differ if you were doing implementation research versus cost effectiveness. But for most of the talk it is going to be about a societal perspective using cost effectiveness analysis. 

So here is the outline of where I am hoping to take you today. So I am going to give you a brief introduction, that is where we are right now, I am going to go into the micro-costing methods. I want to highlight two in there, one is direct measurement and the other one is a cost regression approach, and I will explain how they are different. And then I am talk about what I see as an important assumption which is this idea of efficient production in economies of scale. And I will talk to you about why I think that is an important assumption. And what it is really going to highlight is how things differ for researchers from how they are actually used in real life. And I am going to walk you through – actually I have two examples today, one is a behavioral trial and that’s going to be an estimated labor cost. And another one is a robot trial where we are actually estimating capital costs of buying this new robot. So you sort of get to see the two different methods.    

The focusing question, what  is the cost of a new health care intervention? And think of it in sort of general terms from economics, what does it cost to use outreach workers to improve cancer screening, and in this case, if you are using outreach workers, the main input that you are defining for your intervention is labor and so you want to get a good handle on those labor costs. And the other one that I will talk about is perhaps you have this new robot that you have developed that helps people who have had a stroke improve their arm functioning and arm movement. And you want to know the question, what does it cost to buy one of these robots and actually get that robot implemented in a facility? So I will be talking about both of those. 

Outreach workers, a local county hospital routinely performs pap smears in the emergency department. We actually did this study. The problem is we saw low rates of follow-up among abnormal pap smears, approximately 30% follow-up. So here we know that there is an abnormality in the pap smear, they have been sent a letter telling them that they should get follow-up for further exam and only about a third of them are coming in. The question is, you have developed this new intervention, what is the cost of using an outreach worker to improve follow-up. You know that that outreach worker is going to be more expensive than usual care which is just the mailer to their home. The question is how much more does it cost and what is the benefit you get for that, what is the value?

The robot question, if you are interested in the robot question is engineers have developed robotic devices and the ones that we actually tested in a large trial was an MIT device that I will show you later on in the talk. Robots offer very precise repetitive actions to help the patients with upper extremity impairments, they can work on directions, speed, control for example. So really what is the cost of this robotic enhanced rehab. But if you have your own sort of pet study that you are  working on or intervention, and you can think about whether it is a labor or capital intervention and how you want to process. So hopefully you have got in your head these ideas of what these interventions might be. 

So how do we find the answer? To answer these questions, we can use micro costing methods. There is no – you cannot just go out and find, okay so what is the cost of an outreach worker, there is no website. I know the internet has all sorts of useful information but you are probably are not going to find that on the internet. And so you are going to have to do some research on your own and develop the answer here.

So I am going to walk you through, here is – that is the outline, hopefully if you did not see the introduction of what you were interested in, you probably do not want to stick around for the rest of the talk. If you did, I am going to walk you through two different methods and we will then go on from there. 

So micro costing, this term refers to a set of methods that researchers use to estimate the cost. Typically, what we use here at HERC is we are using these methods to estimate the intervention costs, we’re involved in a lot of interventional trials, whether those are randomized trials or observational trials, natural experiments. And we often want to understand the cost of that intervention. And methods are needed because costs are not readily observed and what I mean by that is that you do not have a competitive market there for health care and so you cannot really see what we mean by sort of the economic cost of this. We have some great accounting datasets in VA but often, even in those accounting datasets, we do not necessarily see the costs of this new intervention.

There are many ways to get here. Direct measure is one method and I will show you an example of the direct measure. When you are actually going to measure the activities, activities typically involve labor. And you are going to assign prices to them, so you are going to have to figure out what are the right prices to assign to them. You could also think of a pseudo bill, you could say well my intervention is largely providing services, but it is a different bundle. Let us say you are interested in integrated mental health. You could say well we are doing things slightly different but we are still doing services using CPT codes. And you can say well let us capture the major CPT codes that we are using. And we are going to assign costs to those billing codes. 

The third approach is you are going to use a statistical technique to identify the marginal cost and you might say well we have this really interesting data set and we are interested in the additional cost of this new method for telephone substance use follow-up. And we know that DSS has these great telephone data out there. Maybe we will just estimate the marginal cost based on the DSS data. You can create this cost regression and estimate the marginal cost per unit telephone call. 

Selecting a method- so clearly you are going to have to think about the data availability. Most of the time in many studies that you are doing, you do not actually, there is no billing data, there is no cost data relative already available, and so you are going to have to do the direct measurements. You are going to have to think a little bit about the method feasibility, and what is going to work for your study, and for your funder, and so forth. There is going to be obviously, some appropriate assumptions that are built into those. I will get to some of those assumptions today, and then precision and accuracy is one that is just going to haunt you, just because you are always trying to be accurate and precise, and that can be very expensive, especially the precision. If you want really tight precision around these cost estimates, you can spend a lot of time trying to estimate those costs. So I want you to be very up front about what we mean by that. 

So for direct measurement, I want you to think about four steps. The first is, think about this as a production process. Even if you are doing outreach workers, think about it as a production process and specify that process in your head or even draw it out. You are going to enumerate the inputs of each process. So let us say you are going to have to hire people. Okay, you can hire people, where are those people going to be, and you can start figuring out that process. You can identify the price of those inputs. You can say well the person we hired, here is their wage. For their inputs they needed office space, here is the number of square feet they use, here is what we believe is the right estimate for their square footage and so forth. You are then just going to sum up the quantity or the prices times your quantity across all your inputs to estimate your cost. You can do this at a very gross level or you can do it at a very precise level, and the level of precision is critical, depends, it really effects the analysis you can do downstream. 

So let us diverge away from health care for a second to coffee. Many people who know me know that I love, besides my work and my family, is that I love coffee. I am actually a coffee roaster and I have a roaster in my garage or a couple of them. So here is, on the far left, is the beans, the growing depends on where you are and the cultivar that you might get them. They are getting sorted. Luckily coffee, there is a market for coffee and a lot of these things sort out in different places. So you get to see for example, auction lots of coffee and the quality matters here, and you can see the pricing of them. They then get stored, you need to figure in the prices of the storage. There is value added in how it is roasted if you believe in sort of roast characteristics versus where the bean was grown. And those characteristics, you can think about having value added there. There is your final bean. Then if you really, you might just purchase the beans from the store or you might want to just purchase a cup of coffee. So keep in mind sort of the quality throughout this whole process, quality is critical and it is going to affect the price in the end. 

The scale of production is also incredibly important here. If you are a small roaster, let us say you are Stump Town in Portland, a very well known sort of micro roaster. You might say well they are going to be sourcing very specific beans and they are going to make it in a specific way and so you would expect that that scale of production is going to have an effect on the cost. Now luckily the cost of a cup of coffee is observable. The other nice thing is that it is, you can interpret your own quality, and you can go through this process and figure these things out. But if you had to micro cost this whole process, if it was not observable, this is the kind of processing that I would expect you to do, so it is the cost of growing the beans, the cost of sorting them, and hulling them and processing them. The cost of distributing them, remember they come from Equatorial countries around the world, the cost of roasting them so you need – all of a sudden you have to have inputs for utilities and then the costs of actually producing, the labor of producing. So and then hopefully, you are enjoying it. 

Now, I apologize if you are not a coffee lover out there and you are just a tea lover, but you can think about this another way. I am actually drinking a cup of tea as I give this lecture, but I have already had my pot of coffee today. 

Precision, you could – there are two ways here that I want to sort of walk you through and explain why precision is important. Let us say you have developed an intervention that uses two full time equivalents, so two outreach or workers that are working full time for you, and they are delivering services to 1,000 participants. You could just say that each outreach worker is $50,000 with benefits, the total labor costs is $100,000 for that year. And then you could say that you could divide the $100,000 by the 1,000, you could say it is the labor per participant for a year is $100. That is not really a precise method, that cost would then be attributed to everybody in your study. So there is no variance across it, it is just a very sort of very loose accounting method of doing it. 

A more precise method is to say, well we really want to track the intervention time. These outreach workers, they are really doing a really good job. They are spending a lot of time with some women, and other women are not as interested so they are spending a lot less time. Some women are really hard to track down, they just need more time. So we really want to track that intervention time per participant. And then use those time estimates as a way to distribute the labor cost. 

Precision is expensive, so this is actually a study that we did and this is the client contact form. We actually developed this client contact form with the outreach workers because we felt like they needed to be invested in what the form looked like, otherwise the data would not be very useful. We also then had the manager review the forms weekly for accuracy and to make sure they were completing it. What we did not want is sort of this recall. Oh yeah, I have got this huge stack of forms and I have got to recall what I did  over the past month or past year, what I really wanted them to do is to have done it more quickly as they were actually providing the services. So here is the client contact form that we actually created. We tracked – they were supposed to track for the client’s name, the total amount of time with each client, the travel time if they are providing it, and the expenses, the reasons for the calls and the visits so we could track what was actually happening. You can actually see that their attempts to contact as well. We did not want them to do a new form for each person because many of these women were hard to reach. Remember, this was a county hospital and they have abnormal pap smears. They were getting screened in the emergency department, so very highly mobile population that we were trying to track down.

So I also want to make sure people understand when I say precision, and I also want to differentiate it from accuracy. So the center of the target reflects perfect accuracy and we are always trying to be perfectly accurate. A and B in some sense are equally accurate, neither one is dead center, they are both sort of off center, but A is more precise. There is less variance around the point of A than the point of B. So if you think in terms of your standard errors, A is going to have a much tighter standard error than B will. So you are going to have ability to say more about A than you are to B statistically. 

So accuracy, you can think of other ways of developing these forms to improve your accuracy. I showed you a form that we used in a county health department. One study that Patsy Sinnott’s done here, is with the spinal cord injury vocational improvement program, if they are trying to get people with spinal cord injuries back to work, vocational [inaudible] right. They actually developed an app in CPRS. So that every time the person provided information or services to the veteran, it was tracked right in CPRS. So it was a very nice because they pulled up the patient’s medical record, they could do what they needed to do for record keeping in terms of VA regulations, and they could also track their time spent. And this improved data accuracy for that because it is built right into what they had to do already. 

The precision payoff really comes down to subgroup analyses, and it versus your expense and effort. If you took that very gross estimate and said it is just $1,000 or $100 per person, and everybody gets assigned that same cost, you are not going to be able to look at any subgroups because everybody has the same cost. There is no precision so to speak at that. But keep in mind that if you are going to go out there and collect all of this data on how much time each of your outreach workers are spending, that is a lot of expense and effort to do that. So you have to keep those things in your mind and balance them. 

I will show you an example at the end of the lecture about the subgroup analyses and why this can be important. And it typically has to do with risk. If you think the effect of your intervention really differs by the patient risk, then you might want to actually to collect that detail. Obviously the other question there is how much money does your funder have and do you actually get funded to do it adequately. 

So direct measurement, one of the things I do not want you to think about, especially when we are doing labor is personnel activities. Research staff can produce several products. Think of your own daily activities. There are several things that you do in your research, but I want you to think about, for an intervention is what we are really trying to do is what would this cost if this were implemented wide scale. So you want to exclude development cost, because that would not be included if this was done wide scale. You are going to exclude research related costs so if they are tracking certain things just for your research endeavors, you want to exclude those costs. 

The other real challenge here, let us say you have a part time outreach worker, What you really need to know is how much time each week do they spend on your outreach, because if they are doing other things that are research related, it is really hard. You do not want to just say that they spent 50% of their time doing outreach, you really want to have more precision than that. You want to exclude other research related costs. And then you should measure when the program is fully implemented unless you are really interested, this gets back to this budget impact analysis. If you are really interested in implementation, you might want to figure out how long does it take them to get implemented and what is that shape of that curve. You expect them to be very expensive at the start of it, they are figuring things out, there are things that go wrong, there are things that have to get redone. If you are doing it from a cost effectiveness analysis, or whatever, we are primarily interested in when the program is fully implemented, because that is really what we are focused on. 

Personnel Cost – typically what I have done in research is outreach workers are hired, they are given benefits, but please keep in mind that the way you pay your outreach workers or you pay your labor, can affect the quality and the quantity of services. Sales, if you are doing sales, you probably are not hiring people on sort of fixed pay. They are probably getting a different type of benefit structure, and maybe the pay is dependent on the number of patients or people they talk to. So you have to think about they attract very different types of people that affect the quantity and quality of the services that you are providing in your outreach intervention. 

If you believe that this is going to be developed so that it includes benefits when it is fully implemented, you need to include those benefits in that cost when appropriate. You also need to include indirect time. So if they are spending all these times working with the women and then they are also spending some time documenting things for record keeping for their own quality assurance, or in meetings and training, you want to document those trainings. Because that has a direct effect on the quality of their outreach, so you want to include those costs as well.

The other question that was an assumption here which is changing personnel pricing will not affect the quality or effectiveness of the intervention, and that is a huge assumption. So try to think about how you are doing things in your intervention such that that is actually going to look like it in real life. And here is one where it is – the VA has its own sort of ways of being. And so you have seen one VA you have seen one VA, I am sure you have all heard that. My sense is, in most VA’s if you are doing this and you are hiring labor in VA’s, it is going to be within the hiring regulations of VA’s. And then when you get your folks on board, if you ever get your folks on board, you would then pay them with benefits. You can actually, if you are interested in VA labor, we actually have spreadsheets on VA labor costs. So let us say you have hired an occupational therapist for your intervention, you want to say what is the hourly wage for an occupational therapist in VA, we have developed those spreadsheets so you can do that. 

So that is sort of a direct measurement method of doing things. You are going out there, you are identifying all these inputs, you are identifying all the prices of those inputs. The Bureau of Labor statistics is another great place to get labor cost. And then you are summing them up and you are trying to get the total cost or the total unit cost if you will.

Another method is a cost regression, and this method really uses, it is only feasible when you have existing data. So you are going to use a regression model. Hopefully people out there are familiar with multivariate regressions, you will use regression model to understand the marginal effect of an intervention. So like I said, it only works when there are existing cost data, you are not going to be able to run a regression if there are no cost data. And it is not a good method for a new technology, especially when the data really are not existing or when you believe that the cost data, if they are out there, are just not really accurate or precise. So if you are really interested in sort of pushing the new technologies or looking at that frontier, you are probably going to be interested in direct measurements.

So the cost of telephone care, so we conducted, and this is just an example of what I mean by a cost  regression. We conducted a randomized trial to examine whether telephone case monitoring improves substance use care relative to usual care. So the intervention was trying to use telephone care and DSS tracks telephone care. So the intervention averaged 9.1 phone calls and the control averaged 1.9 phone calls, and a difference of 7.2 phone calls. That is just the average and you see that it was statistically different. Now DSS tracks the telephone care costs in clinic stops 543, 544, and 545. You have to make the assumption that you believe that those cost data are accurate. If you make that assumption, you can summarize the cost data per person, and you can run a statistic model to say what is the average cost per additional telephone visit, so to speak. And here is that actual regression so we said, here is the number of phone calls on this DSS cost data, and we controlled for gender, we controlled this a two-site trial, we controlled for the differences in sites, we controlled for age, and just a linear way here. And you can see that the average cost per phone call was an additional $10.50, $10.53.

So this is what the cost regression is going to look like. You would then say if there was – let us say a patient at eight phone calls, you would take the eight times at $10.53 and you would get sort of the additional costs of that person. You really have to believe that the cost data are accurate before you want to head down this way. Like I said, you want the workload actually captured, the accuracy could vary by location, and then keep in mind if you believe that there is error in here, you are going to be biasing your cost towards zero if you believe you are not capturing or accurately capturing your data. 

The other thing, if you are using DSS, keep in mind that the costs are local and so those are not necessarily national costs. In our two site trials, both sites were in the Midwest, and so you want to then say well what would that look like if – how does wages vary if it is national versus local? 

The other thing that you have to keep in mind is boy now you are in a regression framework. There are benefits and there are costs. The cost of the regression framework is there are a lot of ways to analyze cost data. In our econometrics course, Paul Barnett goes through and there are actually archives out there that the CIDER folks have archived about how to do cost regression. The cost data are skewed and the questions of what is the best model to run, you could do ordinary least squares, you could do semi-log models, you could use GLM. So you sort of end up in this whole world of regression framework and you have to sort of be comfortable there. But I will sort leave it at that and if you want more information on the cost regressions, you are welcome to contact HERC. 

So it is a feasible approach when you have the data, depends on your regression models and you might end up with two different models that say very different things and you need to figure out for example, how are the goodness of fits of the models and so forth. 

I am actually going to stop here to make sure that there is no overriding questions. Angela is supporting me today, Angela, are there any questions out there?

Angela:No, there is no questions right now.

Todd Wagner:And I did actually receive, after my coffee example, I did actually receive that Stump Town which I touted from Portland was started by two VA employees who used to work on the impatient unit there. And I guess they liked coffee so much and they needed it to get through the VA’s day, that they decided to quit their jobs and make and sell good coffee. So it is kind of fun to get that feedback. So, continue to innovate and maybe you will have a job outside of VA. And I think they are a highly happy group up there at Stump Town. 

So I am going to go back to the outline here. What I am going to talk to you about is this assumption, efficient production in economies of scale. This is a critical thing. And before I get into it, what you will often hear in economies of scale, is this idea of – you probably read about it in the newspapers, two technology companies, we hear about it all the time in Silicon Valley, two technology companies merge, or one buys another one on the belief that it enhances economies of scale. And that you can reduce your administrative overload. So there is something here about the efficiency when you get bigger is really what is an issue here. 

And I see, oh so there is a question that has come up about local versus national DSS costs. And let me just address that because it is important and then we will move on. One of the questions with local costs is, each facility has local costs. Think about where you are located. There is a cost of living adjustment for VA pay. Well that exists there too for nursing labor and physician labor. If you are in the middle of the country, nurses do not make as much there than they would in San Francisco or Boston, two high wage areas. So if you are doing your study on labor in the Bay area, you have to keep in mind that if you are using the labor costs, what it costs to hire those outreach workers, that would not be the same as if it were in somewhere else in the country. So the idea is to come up with what – is there a national average wage such that it can be sort of think about in the national. And then of course if you are doing implementation research, you might be very focused on the local wages. But in general, cost effect analysis we are interested in national wages. I hope I addressed that question. 

So let me move onto this issue of economies of scale. So it has to do with what is the unit cost and how that effects the scales as you increase your quantity of output. Now that I have talked about Stump Town, maybe this is actually a great – you know the comparison to Stump Town would be Starbucks. Starbucks has figured out economies of scale for coffee, that they have become very large, they have figured out the distribution, roasting in huge, enormous quantities, and they have figured out how to sort of maximize the – minimize their unit cost of production to maximize their profits. 

Let me come back to healthcare and give you the example here. When we created this health guide for a randomized trial, we paid $14 per guide, for 1,000 guides. The entire trial was less than 1,000 people but we wanted some extra guides, so it was expensive for us to develop this guide for this few people. A company was actually contracted to do it, if we ordered more, when we talked to the company, we said well what if we were the county of or the health department for Los Angeles. If we ordered more, what is the cost of the guide? And they said well eventually, if you are ordering over 30,000 we would charge you $3 per unit, not $14 per unit. So which estimate should you use for the cost effectiveness analysis? And so, I do not know Heidi, if you were able to get the poll pulled up here, the question really is, which should you use, number one or number two? So the question is, should you use $14 for your CEA or number two for your CEA, number one is $14, number two is $3. And think of it from a cost effectiveness analysis, not the implementation study. 

Heidi:And responses are coming in, we will give it a few more seconds. 

Todd Wagner:Perfect, a good time for me to take a sip of tea. I should say coffee, but you know, if I had so much more coffee I would be jittery and people would not understand what I am saying. Alright, this question really confuses a lot of people, so you can see it is really split here. So in our research study, because it is very small, we paid a lot per guide, and so the question is if you are trying to do a cost effectiveness analysis, what you are really interested in is the national cost. What happens if this is widely implemented, so the correct answer here from a cost effectiveness analysis is actually answer two. 

Am I back to the slides now Heidi? Is that right, can you see my slide?

Heidi:Not quite yet. Okay, you should have the popup to show yours, did you get it?

Todd Wagner:Not yet.

Heidi:I will make it happen again.

Todd Wagner:Alright, let me know – oh there we go, show my screen, thank you.

Heidi:There we go.

Todd Wagner:Perfect. So you should be using, if you are doing cost effectiveness analysis, you should be using $3 per unit, not the $14 per unit. The $14 per unit is artificially high if you will, because we were doing this very small, randomized trial. And it was really just a function of our size. Now if you were doing an implementation study, you might say well, you would really want the cost to reflect the size of the population at that facility. Well you might say you have got some very small facilities and some very large facilities, and the cost might actually then vary, depending on the size of that facility. But here is the idea of economies of scale. As the quantity of the services that you produce, you have increasing returns to scale. Your unit cost, which is dollars over quantity, is going down. At some point, there is really no additional gain there, and it is a constant return to scale. The gold book, which was from 1996, so 15 years ago, talks about that is really the point at which you want to estimate your unit cost for your study for cost effectiveness analysis. 

There is also theoretically, although it is hard to find empirical data on this, I theoretically at this point, economies of scale, you get so large that the administrative part of it just gets overwhelming, and that curve starts to bend back up. But it’s really hard to find it. Typically you see these things where it is – which is decreasing rapidly, so we were doing a study for example on the cost of the VA Central IRB, and we saw this over time we looked at the first three years. And because it was new and it was starting up, they were seeing huge returns to scale over time. 

So hopefully at this point you have understood why you want to think about efficient production, economies of scale and how you would estimate that unit cost for your study. Let me walk you through some examples, and hopefully by going through these two examples, you will get a better handle on how to actually do this in real life.   

So here is the estimate in the labor cost by direct measurement and the two papers that came out of it and how we thought about these things. So again, it is that same form that I showed you earlier, it was the outreach workers, and it is the pap smears in the local emergency department of the county hospital. Like I said, this is a group of women who are highly transient, they are diverse, they are coming into the hospital county ED for all sorts of reasons, and the hospital routinely performs pap smears on them when there are pelvic problems. The problem is that they notice that there was considerable numbers of abnormals, but that very few women were coming back. So they were having a third of the women coming back for follow-up. So for them it was very concerning that they were getting so few women for follow-up, as well as the belief in the obstetrics and gynecology and among the cancer researchers, the oncology group there is that it was leading to downstream cervical cancer that was coming in later than it should have been. So sort of the idea is, that there were downstream costs associated with that. 

So they wanted to develop a new system, and at the time we were developing this entire study, the idea of a rapid test technology was not out yet. So they thought of outreach workers. The idea was to have outreach workers go to these women’s homes, make follow-up, the standard of care was to send them a letter and if it was a high-grade lesion, then they would also get a phone call. The high-grade lesion being pre-cancerous. And so they were saying, why don’t you send the outreach workers, and we are really interested in following the outreach worker. With this population, they also believed that one of the big barriers for follow-up was knowledge and such that they felt like if they were using outreach workers, and the outreach workers were actually people from the community that were then trained, is they thought it would made a big difference for the more severe cases. 

And so because we felt like we wanted to have that information, very precise level of information, we got them to direct measurement. So we evaluated the cost effectiveness and I should be very careful, we were just interested in sort of the cost per woman’s follow-up, using a tailored outreach intervention. At this point, we did not do a modeling to quality adjusted life years. And then we were also interested in did the cost effectiveness vary by disease risk. It was a randomized controlled trial, like I said, the usual care, they were notified by telephone if it was a high grade lesions but typically by mail, depending on the degree of the abnormality. And then, we provided the intervention after six months to the control group. We believed that there was sufficient benefit to this, and I will show you the data, that it was actually, that was how we got through the IRB so there was no consent here for this study. And the idea is that at some point everybody would be getting this enhanced higher quality care. So everybody got usual care, and the intervention got usual care plus these outreach workers. We trained them, we hired them, they were paid salary with benefits, and they were giving tailored individual counseling to the women, so we’re estimating the cost using direct measurement.

We summed up all the interventions. The two methods, if you wanted to get back to the two methods, one would be sum up all the intervention costs and divide by the number of participants, that would have been the easy way out of this. But it would not have allowed us to go downstream and say, did the cost vary by abnormality, because all the women would have had the same cost. And the belief is, maybe you spend more time with the high-grade people, the pre-cancerous people, with higher benefits. Or, the other option would be to estimate the cost of the intervention for each patient, really track the labor involved with each patient, or I should say participant to be more accurate there. But that is the idea. That is the hard approach, and that is the approach that we attempted to undertake for this. 

So if you want to ask what was the intervention more cost effective with subgroups, if you want to do subgroup analysis, you have to use method two. Let me just back up a second here if I may. That second paper, the Wagner and Goldstein paper, goes through and explains why that is the case, why you need to do, if you want to do cost effective analysis among subgroups, why you need to do that. And that also talks a little bit about behavior analysis, how do you stage a change of another measure of doing it. 

So the randomized controlled trial, direct measurements, we use intervention, I spoke through that, here are the two methods, here is unit cost, let me just jump into the results. So here is – it is a small study for this study, so outreach workers, $142 per woman is the average for the intervention. Obviously there was none to the usual care. You get the travel cost, you get the office supplies we tracked, the outreach worker quality assurance are meeting with the manager weekly to make sure they are doing it. Everybody got usual care, that telephone call, or typically the mailer, and that is that dollar, which is what we estimated based on data from the county hospital. And so you come up with this idea of what is the subtotal for the outreach workers, it is $47, the patient travel cost for follow-up, so what is the total unit cost that you have for the societal perspective, it is $214 versus $10.90. So what is the additional cost, it is really $194 per woman to do this outreach worker. Hopefully that is – and there is variance around it, there is a range around it but that is the average.

Was it effective – so the blue line that you are seeing here is the control line. So this is just months since initial pap, and these are for abnormals. These are all women with abnormals. You get to see the lines diverge up to six months. Remember, at six months, the women in the blue group, were given the outreach worker condition. That also complicated life tremendously, because now we had all of these women getting work by the outreach worker, which we then had to subtract out of the outreach workers wages. But it made life really hard for us, but you can see that the lines then start to come back together. They never quite meet at the end so there is a benefit in doing this early. But you get a sense that the intervention was highly effective. 

So what is the cost per follow-up? So this, in many ways, is a typical table for a cost effectiveness paper. And let me see if I can get my highlighter here. I am going to do a pen. So what you are going to start with is your cost on the first column and your rows, you are going to have one group, it is the control versus intervention. So this is the overall, you are going to have your $77 here, your $350 – these are averages. If you take the difference here, that gives you what we call the incremental cost. The probability of follow-up, you are going to see that exact same pattern here, gives you the incremental follow-up. The reason that this is standard is, what you are really interested in is the incremental cost to incremental follow-up. But people want to see what sort of, the basis, the absolutes as well, so it is the $77, $355, that is all probability follow-up. That allows you to, in this case, the ICE or the incremental cost per follow-up or incremental cost effectiveness ratio is $959. And we use bootstrapping to capture our confidence intervals. That is the confidence interval, and that is $959 per follow-up. 

As a segue, this is one of the reasons why people do not like incremental cost effectiveness ratios that are not based on QALYs. Because you are thinking to yourself is that good, is that bad, how would I compare that to incontinence or cardiology, and you cannot. Because it really is only defined by pap smear follow-up. So that is one of the reasons for using quality adjusted life years instead of this sort of naturalistic follow-up unit. If you ignore that small limitation, if you look at by severity you will see the same format here you can walk through for – this is the lower grade, the ASCUS/AGUS. Low grade, high grade, so high grade is pre-cancerous, you walk through your average cost. You know, there is not a huge incremental cost difference as you go down in severity. What is more shocking is the size of the effectiveness really varies by severity. Such that we did a much better job with the pre-cancerous lesions than with the lower abnormalities. 

So that was actually very encouraging, and so you can think of the second set of studies following on from this. Like maybe we really need to target these outreach workers for the high-grade lesions and you can think about also modeling this for quality adjusted life years and what would happen in a bigger environment or a bigger health care system. So that is how we did that one example. 

Are there questions Angela for this one, I see a whole bunch of questions popping in.

Angela:Yes, do you want me to go back to the first question or...

Todd Wagner:If you – to the best of your ability if you think they apply, and I might hold them off, I see some of these are quite long, so...

Angela:Great, do you want to do it at the end then?

Todd Wagner:That is probably a good idea because there might be some other questions that come up with the robot that are related and it might be helpful just to walk through the robot one and then we can answer the questions at the end, thanks.

Angela:Okay, there is a quick question I guess on this one, what are you bootstrapping on?

Todd Wagner:Okay so statistics, let me just back up a slide or two if I can. So the statistics, you could easily calculate a P value on the incremental cost. Because remember, it is an average for this and an average for this, and there is the standard error built around that $355 and there is a standard error built around that $77, so statistics give us an idea of your key test is $278 with a P value of X, because it uses those standard errors. You can do the same thing with your probability follow-up, the difference being is that this is a logit model in some sense because the follow-up is a 01. Did they get follow-up, did they not get follow-up. So what I am struggling with here statistically is that I have got these two models. But I want to put this one over the top of this one. I can easily do that analytically by saying what is $278 divided by .29 and come out with this. But I cannot do it in a way that is easy, that takes in account the simultaneous variance or covariance in the numerator and denominator. Bootstrapping, what it means is that you are taking, you are replicating the data 1,000 times, pulling out samples from imagine a bag of ping-pong balls and you are pulling out samples from that. And it allows you to build an estimate of ICERS and that allows me to come up with this confidence interval around it. 

So if you have more questions on bootstrapping, I am happy to address them offline. But you have to that if you want to do this and put a confidence region around your incremental cost effectiveness ratio. Good question. 

Let us get into robots. So when I started this trial, these are the robots that came up to my head, you had Robby, you had the Jetson Robots, and you obviously had this one, the Iconic robots of C3P0 and R2D2 from Star Wars, that was my youth, maybe I am dating myself too much. So, we are going to do a study on robots, and we always see these robots like walking, Honda has one that walks down the street and so forth, with stroke patients. This is like so cool. This is what the robot looks like, it is not anything like that. So the folks at MIT had developed this MIT Manus Robot, can assist patients so what is really thought of is robot assisted stroke rehab. So imagine a patient has a stroke, they have an upper extremity impairment that is a result of the stroke. And the robot can guide them to make movements that are very functionally related. 

So there are different types of upper extremity disorders, you might have – be able to move your arm very precisely but very slowly, some people are moving it very spastically and have very little control. And so the robot can help you work to try to improve where your limits are. The thought being is that these would be highly effective, much better than a physical therapist or an occupational therapist. And as the patient gains movement, the robot provides less assistance and continually challenges the patient. It can also provide feedback to the patient about how well the patient is doing. Everybody loves instantaneous feedback unless it is an SAT exam or something like that. But you typically like that and so you get to see how you are improving over time. And so the idea was that this would be a much bigger improvement over sort of our usual care.  

Robots are expensive, so and this gets even more challenging, and I will explain what I mean by expensive. So when MIT develops this robot, the purchase price right now for purchasing this robot, it is about $230,000, for that, at the time we were finishing this study. Well I do not know about you, I know the VA has a lot of money but I do not know about you. So imagine you are purchasing this robot. I am assuming most of you out there do not have $230,750 in cash. And even if you did, you would be pulling that money out of investments to purchase this robot, so you need to include financing, and the VA needs to include financing for the exact same reason. You could use that money elsewhere to make money for you in interest, or you are going to have to borrow that money if you are the Department of the Treasury, pay back in Treasury Bills, to people who are supporting the purchase here. So there is a cost of this purchase price, in which case we included the financing. Well it turns out this robot does not just plunk down in the middle of any PT room, it needs its own overhead, it needs a room, it needs a separate circuit. There are very specific, he needs to be able to move around it so the room size has specific needs here. There is maintenance, and there are different ways to handle maintenance, one way that we just heard of handled with it, is what is the cost of the maintenance contract with the company. Now you can say let us not have any maintenance costs and we will just hope that the thing does not break in five years. These tend to need maintenance, so there is a question about how to fairly estimate that cost. There is a depreciation of this capital investment. At some point, it is not going to be worth anything anymore. 

Let us take the I-Phone, everybody has their I-Phone 5 I am sure, on this call, even though I am not an Apple person, but at some point, this I-Phone is not going to be worth anything anymore. So there is a lifespan associated with this technology. Now you can continually update the guts of the technology and many of the technologies that we have, the scanning equipment, that is what happens. But there is a depreciation, this becomes less valuable over time. So you have to think about what is the time span and at what point do we need to replace this thing at the end of its lifespan. So it gets – we did the calculations and these things are not perfect, but our estimates just showed us that we said, wow over five years, we believe this is going to be a chunk of junk in five years, that there is going to be a new technology that exists that is going to replace it. And so this is going to have very little value in five years. The net present cost of this is $422,000, so it is not simply a $230,000 purchase, right now it looks much more than that, you are committing a lot more resources than just that, $422,000 in resources. 

Wow, so you are saying well a lot of people can use this thing and again, this gets back to the economies of scale, in our study, we had a physical therapist work with the patient on each encounter, but the robot is a little bit like a piece of gym equipment, and is actually called a piece of gym equipment. Because they want you to think about that the robot is not working with one patient at a time, it can actually work with multiple patients simultaneously. So it is a 60-minute robot session, so we said that each session lasts 75 minutes, that allows for cleanup and sort of transition between patients. We said two patients can be using the robot gym per session, they are using different components of it. Now we also made – there are therapists involved to oversee this, but the therapists cost is less because they are doing other things that are productive. They are spending in the 75 minutes of time, roughly 15 minutes with each patient. 

So we figure that over the five-year life of this robot, there are going to be 21,500 slots. Now here is another segue into implementation research. You have got to figure out whether your facility has enough stroke patients to fill 21,500 slots. My guess is, that not every facility does and it is mostly the large facilities that have enough of these. And it is the stroke patients that survive, had an upper extremity disorder. So you have to sort of think about it in those terms. But we just assumed, for this cost effectiveness analysis that it did have the sort of the full use of that 21,500. The robot cost per session in this case, the unit cost is only $20. That $422,000 net present cost, if you are fully able to utilize it, is not that expensive. And in fact, and I will show you in a sec, the therapist cost is a little bit cheaper, so the total cost per robot session is $140. 

So here we are, I am back to my pen, so here is that robot cost sheet per session is $140, $20 of it, or $19 and change is the robot, the rest is the therapist. The therapists are actually expensive, they are highly trained specialists, they have benefits and so forth. The average cost of the, and I should be careful here, I did not explain this already, we have two usual care groups. One is usual, usual care, everybody in our study was getting usual care, and then there is what is called intensive comparison therapy. We realize that there is a fair amount of variability in usual care, and we are going to be biasing ourselves towards not finding a difference because of that heterogeneity. And so we came up with, and we know that the robot is doing two things, it is doing frequent movement and it is doing all these other things that just the robot is doing. And so we want an intensive, conventional therapy to replicate the frequency of the high use of the robot but not doing everything else like the feedback that the robot was doing. 

So that was expensive, that was more expensive if you are looking here, it is $218 usual care, because everybody is getting at the added cost of usual care is zero. You get to see the number of sessions people completed, 32, this is a 12-week course here. You get the travel cost because of people having to travel. You know again, think about it, from – if you are a small facility and you wanted to buy one of these robots, would people actually travel three times a week to use this robot for 12 weeks for 36 sessions? So you have to be probably in a metro area to make this thing fly, to make this thing work, to have enough stroke patients. And here is your average intervention cost, $5,152 versus $7,000 so the ICT is actually more expensive than the robot. The robot is more expensive than usual care but less expensive than this intentional comparison therapy.

So I think I have the paper, on the bottom of the screen, it is actually the citation stroke with the appendix for calculating all of these and we walk through it and we do the full cost assessment analysis, we track the utilization downstream, sort of for the timing, they get questions here. I am going to end it there, but if you are interested in that, I am happy to walk you through it. We have a lot of resources on our website about how to convert time into money. You will always hear that average, time is money. And so here is the resources there, converting travel distances into money, caregiver costs, Louise Russell’s has to pay for medical care in 2009 but she has also given some Cyber seminars in the past. Just fabulous stuff that she does. 

Keep in mind the resource, the estimate, the cost of labor. Hopefully you have your handouts so you can just sort of store these so that you can find these in the future. If not, I am going to move on and you can sort of think about contacting us in the future if you have any questions. So this is probably the great segue for questions. 

Angela:Okay, so I will start with the first one. Would one adjust for the scaling up of a program with covariates in a multi-variate regression framework? If so, what are some good indicators of an economy which is scaling.

Todd Wagner:Yeah, you might have the empirical question of whether there are economies of scale, and you might ask the question that I feel like this – the unit cost is changing as the quantity goes up. And so you would have to have data about whether it is cross sectional or longitudinal. So let us say it is cross sectional and you have got a large number of facilities out there, they differ in their scale, the quantity of services they are providing. And you can use DSS data to say what is the cost of this production function if you will and does it not change, does the average unit cost change if the quantity increases. You could empirically come up and say aha, it is less than .05, it does, and then you could say well what is – do we believe that there is a point at which it bottoms out. And then you would say well let us take the minimum average cost of production and run with that. And you might want to test that, and make sure that there are no aberrations that you are minimum, but you could say at certain size, it is probably – you will see that slope decreasing. You have to play around with your functional form if you will. You might say it is not linear, it might be curvilinear. You could use spline, there is different statistical techniques. I did a paper on medical care on IRB, in which case we actually sketch out that curve using a cubic spline. So if you are interested in that, we could actually work you through it. Hopefully I answered the question.

Angela:Okay, the next question, okay the next question is a three-part question so I will read the first part first. When running a cost regression analysis, I got a very wide confidence interval. Why does this happen, especially if the data cells did not have N less than ten and what is the solution to this problem, especially if this is an important model variable. So that is the first part.

Todd Wagner:So, standard error is your standard deviation over the square root of your N. So if you have a small sample, implicitly you are going to have less precision. It is just or – you are just going to have – your standard errors are going to be bigger. So the easiest answer is to get a bigger sample size. You will notice that often when we work with tens of thousands or hundreds of thousands of units or cases or observations, that everything is significant. It is because the standard errors are so tiny at that point. But in this case, one of your easy answers is to increase your sample. That, is probably not the answer that you wanted to hear. The other answer is to figure out, okay so is there heterogeneity, is there something causing this high variance that you could then control for in your model. And maybe there is and maybe there is not, so there is sort of an art of statistics here that allows you to figure out why am I getting such variance, and maybe it is related to you picked four sites, and those four sites vary in wage. So maybe you picked Saint Louis, San Francisco and Togus, Maine. That is hard to imagine three sites with different wages any more than that, so you can imagine, wow I have a control for the wage differential and we want to adjust for the wage differential here. So think about other ways that could contribute to that difference there. And if you want to come back to us on the first part of that question, we can easily address that offline. 

Angela:Okay, and the second part of the question is, should propensity scores be used on cost data when looking at treatment costs?

Todd Wagner:There is a whole – in our econometric series, we actually talk about propensity scores, there’s a whole lecture I give on propensity scores. They are another way of adjusting for observables  - I sort of want to avoid the normative question that you are asking. Should they, they can be, are they useful, sometimes. I guess that is probably – without being more specific, I have my personal opinions on propensity scores, but I will try to remain bias free in this politicized age, especially as we approach the elections. But contact me if you want more on that.

Angela:And the third part is – okay and then the third part is – cost data is not normally distributed, which multi-variate regression analyses comparing three drugs are recommended, especially with observational data where patients move in and out due to eligibility issues. And then in parenthesis, right and left censored. 

Todd Wagner:Amen, cost data are some of the weirdest data that we handle out there. And Paul Barnett gives two lectures in the Cyber series in the econometrics about how to analyze cost data. So in particular, especially if you are analyzing a population, and some people are zero costs that creates incredible tension in your models, especially with some very high cost, right tail outliers. I guess at this point, I would be happy to work with people on sort of the econometrics and refer them to those talks. But you are right, getting into those models is challenging to say the very least. And you have to think about sort of the goodness of fit of the models. On that robot study, we actually had different results depending on the model that we ran for our cost regressions. I know there are a bunch of other questions so we will probably have to leave it at that.

Angela:Okay and then I see a question but it is on the current question so – in addition to the standard error sample bias may also occur even with sampling multi-factor factorial adjustment as needed for variability of the factor level. So I guess that was more of a comment. 

Todd Wagner:True, absolutely, yeah you have to be very careful. It certainly goes without saying, however you are sampling, and even if you wanted to go back and say we tracked the labor cost for all of our outreach workers, you might say well that is tedious, maybe we can track it with a sample or at a particular time once the outreach workers are up and running. And those are perfectly good, it does get you into this question of what is the appropriate way to sample that and how do you measure that. And I would – I am hemming and hawing, because it is a challenge as well, so good point.

Angela:Okay, so how do you know the point in which the unit cost levels out, the actual point that should be used for the estimation of the unit cost?

Todd Wagner:Hopefully you have a highly paid economist on staff who can look at that and make an answer. If you do not, and I am joking with you there, it is sort of the art of doing this as much as the science. At some point you have to make a judgment about where you sense this curve is, this economies of scale curve and what it looks like. So one of the reasons for using the cubic spline that we did in the medical care, was actually to visually graph it out and figure out okay, what does it look like, and is there economies of scale as IRB’s get bigger, and where do we think that that is – the curve is changing, where are the kinks. But you are right, that is a tough thing to look at. I would say visually is the easiest way to do it. But even still you might not have enough data points at  the full spectrum of size to make a good estimate. And you are probably just going to have to, at some point, choose a unit cost and then bury it in sensitivity analysis. I know Ciaran Phibbs talked at length about sensitivity analysis and how you would walk through the model and say well what if we use a different unit cost, is it sort of qualitatively change the results. Many times small variance and unit cost does not change the results much. 

Angela:So can you mention a good source for converting SF 12V to QALYs in STATA.

Todd Wagner:So I know we are over the time and we have 136 people who are so patiently waiting here, so I know people have to go. I am not sure we are going to have time to answer them all so I see there are many more questions, so at this point, let me just say, we will do our best to answer and respond to you by email if we cannot get to your question here and I apologize profusely. In terms of a good source of converting SF 12V to QALYs in STATA, I think I have actually have a stata a do program. Libby it looks like your name is on that, if you are still on, please email me at Todd.Wagner@VA.gov, I think I have that, if not, some similar code I think is what I have for that in STATA. I am just tickled pink that you are using STATA, I love to see people using STATA out there. I know people just love to do SAS, I just think STATA makes it easier to do the right thing. So congrats on using STATA, and I will see if I have any codes to help you get SF 12V to the QALYs. 

So Heidi, I think at this point, we should probably end the Cyber Seminar, otherwise it will go on forever here.

Heidi:That is fine, I totally understand and I have all the questions saved so I will send those over to you as soon as we wrap up here.

Todd Wagner:Okay, I will – I appreciate everybody’s patience in hanging on, so thank you so much.

Heidi:Great, thank you so much Todd. And for our audience, our next session in the Cost Effectiveness course is next Wednesday, unfortunately my computer just locked up on me here. 

Todd Wagner:I think I showed it.

Heidi:You did show it, I know Jean Yoon is presenting inpatient and outpatient costs from DSS. Everyone should have received an email today with the registration link for it. If you have not registered yet, go ahead and look for that email, I sent that out late this morning, and go ahead and register for it, and we would love to see you at next week’s session.

Todd Wagner:Awesome, thank you Heidi, and thank you Angela and thanks for all of our attendees today. 

Heidi:Thank you so much and we will see everyone at a future HSR&D Cyber Seminar, thank you.

Todd Wagner:Perfect. 

PAGE  

Page 19 of 19