**Transcript of Cyberseminar**

**HERC Cost Effectiveness Analysis Course**

**Estimating the Cost of an Intervention**

**Presenter: Todd Wagner, PhD**

**May 30, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact herc@va.gov.*

Todd Wagner:  This is Todd Wagner. I am one of the health economists here at HERC. And today I’m going to be talking about estimated costs of an intervention. I have 48 slides, hopefully that’s not too many; so what I’d like to do and I know that Christine, she is helping me out here with the questions and answers and my back up. I would encourage her to chime in if there are clarification questions; otherwise I might hold some of the bigger questions that you might have until the end to make sure that I can cover the material. And then if we can’t cover questions or material and you’re really interested in it, feel free to contact us and we can always address those questions off line. 

So what I hope to have for the objective of this class -- at the end of the class you should understand what micro-costing means and hopefully be familiar with some of the different methods for micro-costing. And then one of the key understanding points that I want to make is that you will -- your choice will affect your future analysis and what you can do. They’re sort of easy ways to do micro-costing and there’s much more complicated ways and you get more with the more complicated ways and you get more with the more complicated ways as I’ll explain.

So for the perspective of this class given that we’re part of a cost effectiveness course the talk is really about focused on estimating the cost of an intervention that is in a cost analysis using societal perspective. So if you’re a implementation researcher, particularly interested in let’s say the cost of -- to the provider you may have to vary these methods a little bit. 

Here is the sort of outline of the course or the class today. I’m going to go through a brief introduction. I’m going to get into some micro-costing methods and specifically talk more about direct measurements and cost regression. The third part I want to hit on is the important assumption when we’re doing these things which is this idea of efficient production on economies of scale. And then I’m going to finish it with an example so hopefully it makes sense to you all. 

So the focusing question hopefully in your mind you’re thinking about what is the cost of a new health care intervention and you might have different types of interventions. One might be an outreach worker to improve cancer screening primarily the outreach workers are labor. You might also be interested in using a robot for stroke rehabilitation and we think of the robot as much more of a capital intervention. 

So just to give you much more specifics about this in case you’re working on this and we’ve had a lot of interactions with researchers over the time and frequently they are labor interventions. So for example I worked on a study in which we use -- there’s a local hospital that was routinely performing PAP smears in the emergency department when clinically indicated; they’re problems they face is that they had really low rates of follow up among abnormal PAP smears and even among highly abnormal PAP smears. They were getting about 30% follow up and so they were talking about potential solutions and one of the ones they came up with and they did a study on this was to hire outreach workers and the outreach worker was then designed to go around and contact these women and work with them to come in for follow up. So the question that -- sort of the guiding question of that study is what is the added cost of using an outreach worker to improve follow up?

Another study that I worked on involved an MIT robot and this robot was designed to help people who had a stroke and you can imagine you survive your stroke but you might have upper arm extremity impairments and this robot was designed to help  you in the rehabilitation of your arms. Now the nice thing about robots is they offer precise repetitive actions, you can work on direction, you can work on speed, you can work on control. But as you might expect these robots are expensive so what is the cost of robotic enhanced rehab and sort of the unit cost per robotic session and so it’s a very different type of question. So hopefully at this point in your head you have an idea of what you’re trying to estimate the cost of and I’m going to walk you through some different methods for how you might do that as well the assumption. 

So for most of these things, to answer these questions we need to use micro-costing methods. So here’s sort of moving on the outline and I’m going to talk a little of direct measurements which I’ve used frequently as well as cost regression. 

So micro-costing, this term refers to a set of methods that researchers used to estimate costs and we have to do this because we can’t go out on the market and see what a cost to purchase a robot. The robots that we were using for that, robot studies were made at MIT and even if you were to purchase the robot, what you’re really purchasing is the robot, not the unit cost of it. It’s not like saying “I want a gym membership and what’s the cost for going to the gym per time”? We really don’t have competitive markets in health care and information that you might expect.

There are three commonly used methods for direct or for micro-costing. The first is direct measurements. And you can think of this as almost a time in motion study where you’re going to measure the activities and then assign prices to them and you can be very, very precise here and detailed if you want to; it gets a little overwhelming depending on how much data you want to collect. But in the case of the outreach workers I’ll show you all the forms that we use to collect the time that they were spending trying to track down women and working with women to get them in. 

Now let’s say you come up with two other methods are pseudo-bill and cost regression. So let’s say you’ve come up with an organizational change and the organization is using services in just a new way, but these services have billing codes and things like CPT codes associated with them. So in this case you might figure out if you have a pseudo-billing or trying to figure out what are the common billing codes and then what would the reimbursement be for those billing codes or vice verse. You might say “Well we have cost data and you know we’re trying to figure out does this reorganization effect the marginal cost” and you could come up with a cost regression. 

So I’ll give you some brief examples of these methods. Often we get questions about how do you choose among a method? Most people stick with direct measurement when they just -- there’s no other billing data and there’s just no knowledge about what it might cost to do this and so in that case I think of the data availability. There’s just no other information on what these services might cost and so you have to actually go out there and do the direct measurement. It may -- there may be questions of feasibility and you might have questions about the assumptions underlying what the data are being accurately captured and then there’s questions of precision and accuracy. What I would say is that it’s not always easy to select a method and often people come to us and say “How do I choose which method” and I think it’s fine to come to her -- with those types of questions if it’s not immediately apparent to you which method to use. 

So for direct measurement I think of four steps. And as an economist I often think of what we do in health care is a production. And if you’re not familiar with sort of production processes I’ll give you an example in a second of coffee being a big coffee aficionado. But think of this production process and you’re going to write the inputs for each step of that process and then you’re going to identify prices for each of those process steps if you will and then sum it all up.  So clearly if this is a very complicated process, let’s say you were redesigning surgical -- in patient surgical care. Well that’s a hugely complicated process and if you were to go out there and directly measure each step you know four minutes of surgeon time, three sutures, you know some drugs that would be a very complicated direct measurement. So you can see where it’s just not feasible in some cases. In the case of the outreach workers what you’re really trying to do is track down the outreach workers time that they’re spending for each woman in the study. And then the idea of the level of precision is critical in all of this. 

Like I said I love coffee. For those of you who have done this cyber seminar before you heard me talk about coffee. I’m crazy enough out there that I actually roast my own; that’s how off the deep end I am. But here’s -- imagine if you didn’t have a market for coffee and you didn’t know what a cup of coffee is. Hopefully -- for those of you who drink coffee or tea used to think “Well if I go to  Starbuck’s it’s about $2.00 a cup for a small”. But you could imagine a situation where you’re trying to micro-cost this and coffee would actually be a very complicated process. You have the growing and all of the techniques that you would say about growing coffee in equatorial parts of the world. And that includes the labor of the growing, any fertilizers that you need in that part and so forth and then you’ve got this processing of the green beans and that’s that second picture there and you’ve got the shipping and storage of the green beans, you’ve got the roasting of the coffee and then eventually you’re going to have to grind and sell the coffee and hopefully enjoy your coffee. So there’s a huge process here. 

If you were trying to do this on a very small scale you could imagine where the quality might be high but the cost would be very high as well; so we have to keep these ideas in mind about what’s the scale of production places like Starbuck’s are doing it on a huge scale and they enjoy some efficiencies from that scale. Luckily for researchers we don’t have to estimate the cup of -- what’s the cost of a cup of coffee, we just go out and you can then choose based on prices whether you’re going to pay $2.00 or $2.30 or something like that and then the sellers just compete on price and quality. Hopefully that example gives an idea of what we would be doing for micro-costing. 

Moderator:  Todd can I insert a question here?

Todd Wagner:  Yeah. 

Moderator:  Why is this method called micro-costing?

Todd Wagner:  I guess because it was the best available alternative and it’s probably better than macro-costing. In some sense what you’re trying to do is use a microscope or just examine what exactly is going on here at the micro level that would generate a cost. I know that is not entirely satisfying but that’s just sort of the vernacular that’s sprung up over time. 

Moderator:  Okay thank you.

Todd Wagner:  Sure. So let me tell you a little bit precision here and this goes back to our example on the outreach workers. So let’s assume that the intervention used to FTE. Now FTE stands for full time equivalent outreach workers; so that actually might be four people working at half time each, but we try to think about them in terms of FTE’s. And they’re working for 1,000 participants. Now you could just say “Well I’m interesting in understanding the cost per participant” and you see the total labor cost of the $100,000.00 per year and you would divide it by the $1,000.00 and that would give you your estimate of $100.00 per woman in the group. I think of that as sort of a less precise method -- there’s no variance for each of those, the woman were receiving the intervention.

Now some women might be very hard to track down and you can imagine outreach workers spending a huge amount of time with certain women. And so a more precise method would be to track the intervention time per participant and then you’re -- they have to somehow figure out how to track the time, which is a huge issue and I’ll show you a form for that. But then you use those time estimates in some sense is a relative value to portion your labor costs. 

There are huge downstream benefits to doing the latter but it’s expensive so here is a form that we developed for that PAP smear example that I talked to you about where we’re using outreach workers. The outreach workers they were hired, they were brought on board, we told them what we were wanting to do which was track the amount of time that they spent talking to each woman or on the woman’s behalf. Sometimes they’re just trying to understand the system, the hospital system itself so they’re not actually talking to the women; they’re trying to figure out scheduling of appointments and so forth. We wanted to track all that time so eventually got them involved with helping us design the form. And then we had a manager who had to review the form weekly because you can imagine it’s no one -- if the average worker feels like no one is actually reviewing the stuff they do it at the end of the day or they do it at the end of the week and it’s not very accurate information. So we were making sure that they were -- new that this was really important data that they were collecting, but they take the time to fill these forms out. 

I just want to differentiate the distinction here between precision and accuracy. Accuracy is we’re all trying to be accurate in our data and that’s hitting the bull’s-eye; that’s that red dot. But we’re also trying to be precise and precise matters a lot when we’re talking about statistics because we often will get no results if we’re very imprecise and we have a lot of noise. So A and B I think of being equally accurate but A is much more precise. Those dots are much more closer together than B. So there are times when precision matters but it’s expensive. So collecting all this data about where you’re spending the worker time with each woman is extensive.

In terms of accuracy another program, project that I worked on with Patsty Zinoff (ph) she’s a health economist here was a spinal cord injury employment program. And they were actually able to in VA CPRS is the VISTA system or medical records system actually developed an app so that every time that the clinician was recording services they were also recording the amount of time they were spending with the patient. So it was just timely, it was built in the system and we believed that improve data accuracy so I just want to show you there are many ways to actually collect this level of data and that can lead to hopefully more accurate information and more precise information.

So here’s what I think of the precision pay off. If I went back to that slide where you said you’re outreach workers cost $1,000.00 or $100,000.00 and you -- 1,000 participants and you just said, “Well let’s just give the same cost to each participant” that’s very quick but you can’t do subgroup analysis because there’s no variation among the women. Every woman has the same cost and so ideally would like to know if you want to do sub groups perhaps there’s a very high risk group of women who need follow up. Perhaps your intervention was particularly cost effective for that group well you need to then have that level of precise detail so that’s the payoff; hopefully that makes sense. I’ll give you an example of the end of the lecture about the sub group analysis and what I mean here. 

So direct measurement; so personal --  personnel activities. The one thing that you have to be careful of and I’ve talked a little bit about it when we talked about this form that we created for this outreach workers. They wouldn’t have collected all of that information if it wasn’t for us researchers telling them that they had to. So often the research staff are producing several products, one is they’re producing the service, you know the outreach worker service for the women and then the other one is providing us with the data that we need to analyze the study. Those are two different products. You’re supposed to exclude research related costs; so all of that time that they’re spent tracking the form so we actually timed them in completing these forms; we tried to remove that from their total labor costs if you will. We also didn’t include developing the form as part of their labor costs, that’s the development cost. 

Another one that is going to be really tricky here is you should measure the cost when the program is fully implemented. I guess little bit back to that question that I talked about you know, the economy scale and efficiency. We’re not interested in sort of the learning curve at the beginning of the intervention that is the women are learning again how to fill out this form. What ideally you want to know is what’s happening when the program is running smoothly. 

Now you might be, for example, interested in implementation research and you might be very interested in you know the early costs and how you know, that learning curve so to speak. The typical cost analysis or not. So personnel costs, what I would say here is almost all the studies that I’ve worked on people get hired on sort of a salary basis with benefits. And I just want to put out there that there are many ways of hiring and paying people and that could affect both the quantity and quality of the outputs. So if you said to the outreach workers we’re going to hire you and pay you a salary per month that you work for bi-weekly or whatever it might be it might attract very different people then if I said “We’re going to pay you with bonuses for your performance for how many women you contact”. So you have to be thinking a little bit about how this program wants to be implemented downstream. 

Now the -- you also need to include benefits when appropriate. Benefits being things like if you’re hiring -- if you wanted to hire full time outreach workers, add in item no ending date and they’re full time you would then need to include things like at least in VA benefits, health insurance and that can assess. And so you also need to consider not only just their direct productive time so the time they’re studying with their women, but also their indirect, non productive time so we put meeting times. But that you know, meeting with their managers, talking about what they’re struggling with you need to include those times as well. 

And what I would say is these decisions especially pay you can’t just sort of make easy assumptions about “Well maybe if we started off and we just said it’s easier for researchers and getting through our IRB to hire them as staff” but ideally what we like to know is if we gave them bonuses would they be -- would that be more cost effective? You just won’t have any data on that so you need to think a lot about how you’re going to pay your employees up front. And then there’s -- in terms of labor costs Bureau of Labor Statistics has a lot of information on labor costs and if you’re in VA we’ve identified labor costs and we have facts on how to do it and so forth. We’ve used payroll information and FMS data to identify average costs per different labor types per hour. 

So hopefully people have an idea of this direct measurement, at least the gist of it that you’re going to go tracking down the amount of time that these outreach workers spend or these workers spend and then you have to identify their labor costs associated with it, include their benefits and so forth and multiply the two together. 

Another method might be let’s say you have existing data that can say a lot about the costs of an intervention. And let’s say we’re going to use a cost regression; so here you’re going to use a statistical model to understand sort of the marginal effect of an intervention. And the chief caveat here is that often we just don’t have the data; so that’s the first caveat. Often -- this only works on their existing data. It’s really not a great method for a brand new technology; for example that robot technology that I was telling you about earlier for the rehab. There’s just no data on what are the costs to run a robot in VA. It’s a -- the data just aren’t there.

You could also have a situation where perhaps in VA you’re interested in secure messaging and you might argue that well the -- we’re tracking those things, hopefully DSS or the managerial cost accounting system is tracking those things but you might also be concerned that it’s underdeveloped in terms of all of the work loads capture and so you might say “I don’t think it’s going to be accurate”. I’m going to give you an example of a case of using regression. So we did a randomized trial to examine whether telephone case monitoring improved substance use care relative to usual care.

So it’s telephone calls; now VA does use a fair number of telephone calls and it tracks it in the DSS data. And I should be a little bit careful here because DSS recently changed its name; so if you’re in VA  you’re familiar with DSS. People are also talking now about that managerial cost accounting system. Now the intervention averaged 9.1 phone calls and the control group averaged 1.9 phone calls; so clearly in this randomized trial even if you’re in the control group you could still get a hone call. So the difference just subtracting the two is 7.2 and that was significant. So the intervention group measured or had more phone contacts than the control. 

And so then the question is what’s the cost of those sort of added phone calls? Now the DSS tracks substance used telephone care in different clinic stops. So assuming that those data are accurate you could take a regression approach; so we summarize the cost data per person, we ran a regression so this was just a linear regression where you can see the co-variants and you can see number of phone calls, female -- there was a bunch of other co-variants that I just excluded here for the sake of brevity. But that data coefficient on the number of phone calls tells us the added cost for -- the average added cost for each additional phone call, so $10.53. So that would be another way of using cost regression to come up with sort of added costs of your intervention. And again this only works if you have the data to make it work.

I will also say that keep in mind and talking about the assumptions here but the cost and the work load data has to be accurately captured and the accuracy can vary by location. So sometimes this just requires a little bit of sleuthing if you’re doing a multi-site trial or you’re working with national data to figure out “Okay do we think that all of the sites are accurately capturing the right data” and you can do stratified analyses by site to try to figure that out. If a site isn’t collecting the right data you’re -- all of your cost estimates are bias towards zero because they’re just undercounting. 

The other thing that is challenging that cost regression and we have cyber seminars on it is there’s -- and I showed you a regression that it was just a linear regression; there is a huge literature on how to analyze cost data. There are many different statistical models that are available to you that you might want to use and you might say “Well it’s -- I’ve got some very skewed data” and maybe the straight linear regression or ordinary square it, you know, maybe the data just violate many of the assumptions underlying that. So you can get into log OLS models, GLM models and I don’t want to go into here but if you’re really interested in that I would say we have some stuff on our website and then there’s some reading below that can help you on that. 

So hopefully at this point you have an idea “Okay so perhaps I want to do cost regression or direct measurement; I have a sense of what I’m doing here or at least the gist of it”. Now I want to talk a little bit about this idea of efficient production and economies of scale and then I’m going to get into the example. Hopefully the example will bring it all together. 

So going back to this idea of using outreach workers. We created a health guide for this randomized trial so that these outreach workers, you got this health guide. Now we paid, we went out on the market and we found a small company and what the company was using was our medical record data to create a tailored health guide; so it would tell information about the age of the women, the women’s history, how often women came to the healthcare system and try to address what we saw as the major issues for this woman to overcome in getting follow up care. 

So we paid on the market $14.00 per guide for those 1,000 guides. And we asked them, we said, “We’re doing a small study here. If we were the San Francisco Public Health Department or the Los Angeles Health Department how much would we pay for these things” and they said “Well you’d be ordering -- how much would you be ordering”? And we said, “You know 100,000 - 200,000” and they said “Well eventually at that point probably our lowest price would be at that level about $3.00 per guide”. So one question is “$14.00 is very different than $3.00”. I’m going to ask you a question, which cost estimate should you use for your cost effectiveness analysis? 

And so poll question for you and I’m hoping Heidi can help me with the poll here. Which method should you use or which cost assessment should you use? $14.00 per guide or $3.00 per guide or somewhere in between? So hopefully you see a little thing popped up and it’s -- you can put in -- keep in mind this is completely anonymous; I have no idea what you’re putting in here and I can broadcast the results, can’t I? If I broadcast them do they stay -- Heidi or --

Moderator:  Yes everyone sees the results; yep.

Todd Wagner:  Okay so very few people think and I -- is the -- so the numbers in parentheses and the percentages is what’s changing on the left. So 17% or thereabouts think that you should use $14.00 per guide. The actual cost to us researchers; a small -- another fifth of the people thing $3.00 per guide and the majority of the people think somewhere in between. I sort of want to ask the people out there if you’re somewhere in between where are you in between and that’s an interesting question. So I’m going to thank you so much for voting. 

There is an answer that we talked about in the literature which is $3.00 was what we used for our paper and let me talk to you a little bit about that. So economy is a scale. So as you grow the ideas that you become more efficient in production. And often this is an empirical question so such that if you’re doing a very small number of guides it’s expensive; if you increase it the costs go down so you have increasing returns to scale. To a point where the price really doesn’t change very much; you have constant returns to scale. Now many of you have already hear the term economies to scale; often in what we do in America and reading the newspaper you hear about companies merging. Years ago it was HP bought Compac when all this consolidation and the personal computer world. And the argument at that point was that it was because they could then cut down on the administrative costs and economies of scale. So often it’s this idea that things become cheaper as they get bigger. 

Now we have to be a little bit careful here because sometimes the quality can change, but let’s assume that we’re holding quality constant. You don’t see quality in this but we just have to assume we’re holding quality constant. The cost effectiveness analysis is making it a final perspective is making the assumption that we’re using prices where there is constant returns to scale. So that’s where we use the $3.00; now you might in a sensitivity analysis and I know Risha Gidwani talked at length about sensitivity analysis. You might think about “Well maybe we want to say a little -- maybe it’s a smaller health department’ what would that face and you could then say “Different -- perhaps the results change if you’re a smaller quantity”. 

So I just wanted -- I pointed that out because I wanted people to keep in mind as they’re doing these cost estimates specifically things where you’re developing something you might often face a development cost and I encourage you to think about what would be the cost for a large health department or a large system? Those are often not things that are captured in but it’s important to try to capture those. 

So let me just walk through examples or an example and hopefully at this point it’ll make it much more clear. So here’s the paper that I’ve been telling you a lot about which is labor cost, it’s an example of using labor costs by direct measurements and we’re going to be talking about using these outreach interventions for low income with abnormal PAP smears. And this may be think a lot about how to estimate cost analysis so behavioral intervention on them so we did a follow up paper on that as well.

You know I go back on that and in 2007, 2004 oh boy, a lot has changed. So as you get older hopefully some of these are still good studies and hopefully they make sense to you all? Outreach workers and I talked a little bit about this so let me just be fair in case you joined late; so let’s assume that this instance it was a local county hospital here in Oakland, California. It handles majority of high risk patients, patients without insurance and they perform routinely PAP smears in the emergency department for a lot of the women. And the problem that they’re facing as I mentioned about 30% of the women were coming back for follow up when there was an abnormality. And so they wanted to develop this outreach worker intervention to improve follow up. So we evaluated a cost -- usual care, which is just a mail postal reminder with a tailored outreach intervention but we wanted to say is one -- this new technology that we’re developing, this outreach intervention, cost effective compared to usual care. But we also wanted to say we knew that there are different types of abnormality and we wanted to say something about does the cost effectiveness vary by disease risks? There are some abnormalities that are just sort of -- dysplasia on the cells but it’s not a huge problem and then there’s other ones where it’s very serious. 

So the study overview; it’s a small randomized control trial. The usual care that folks were notified by telephone or mail and most cases it was mail depending on the degree of abnormality and then the provided the intervention after six months. So I just want to be clear here the -- IRB and working with all the different people involved said “Well let’s just start this off and we’ll run this intervention for six months. We think that the intervention is going to be effective and so we want to then provide it to the control group as well”. So the intervention they’ve got usual care plus they got this outreach intervention and individual tailored counseling. So we wanted to estimate the costs of this outreach intervention using direct measurement so that form that I showed you earlier was the exact form that we use in the study. 

So we summed all the intervention costs and we could have used method one and you know divided by the participants and that would have been the easy method. We didn’t do that; we actually did method two which was to estimate the cost of intervention for each patient and that was very hard. But we wanted to ask that question was the intervention particularly cost effective for certain sub groups? So we had to use method two.

So here’s what we came up with; so here’s actual data in 2002 dollars. So the outreach worker cost. You see the usual care, they weren’t assigned it in the first six months; they didn’t get any and they cost $142.00 for the intervention per person. There are some travel costs -- the travel costs were skewed in many cases the outreach worker just met the women at the hospital. In some cases they went to their home and helped them get to the hospital. We had an office for these outreach workers; there were supplies that they used so we estimated the cost of those. We had the manager who oversaw the outreach workers addressed weekly questions; they had a weekly meeting with the outreach worker to address sort of case issues that were coming up; so we included those costs. Usual care it’s the same; when we estimated the costs of mailing out the letter and it’s a system generated letter, that was $1.00 so both groups, because both the intervention and the usual care group got that usual care. And so then they came up with sub totals. We included, because we’re interested in the societal perspective, we were interested in the patient costs themselves and so estimated at $19.00 and so you get to see what the total cost is from a societal perspective per woman was $214.00 for the intervention and $10.00 for the usual care arm. That’s a big difference.

Now here’s the effectiveness data. So on your X access is months since initial PAP smear. So think month one as randomization date; so the blue group, this blue line is the control group. So you see over a period of time at six months about 33% of those people have follow up. And these are abnormal PAP follow ups. Well with the outreach worker intervention much higher rate of follow up; we’re approaching two-thirds, almost double the rate of follow up among abnormals for people in the intervention arm. Now remember at that point, right at six months the control group was then given the intervention; so that’s why the two lines start to come back together is because between month six and 12 now the control group is starting to get these outreach workers as well. And you can see that the curve for the red group largely you know flattens out; it ends up about 80% but there’s not a huge gain between month seven and months 12. 

Now sorry, my table got a little crunch there. You’ll see overall this is a very common way of presenting cost effectiveness data. So you’ll have a row for the control, a row for the intervention. These are your average costs for each group and then you have in column two; let me see if I can get my drawing. Pull up a highlighter here. There we go. Then you see this 278, hopefully that comes out. That’s the 355 minus the 77 and you just get to see the incremental cost; it’s commonly done this way just so you can walk people through what’s the incremental cost. Then we have the probability of follow up and you know here’s the 32 versus the 61% and so you -- 29 percentage point increase in follow up. And in this case at the time we did what’s the incremental cost per follow up. Now ideally you want to then extend this to a larger societal cost effectiveness ratio where you’re talking about quality life years. We just didn’t have the funding in this study to do that. But you’ll get a sense on here’s the incremental cost per follow up, which is the incremental ratio of 278 divided by that .29. And then we use bootstrapping to get the confidence interval.

Now what -- it got a little crazy here so I apologize; that’s by severity. So this group, the high grade lesions are the most severe group. And then you get the abnormalities and the low grade lesions. So what we found particularly interesting here is that let me just move on to effectiveness data. This .44; so the intervention was actually incredibly effective for the people at most risk for cancer, for cervical cancer. So the cost went down; we would never have been able to do this sub group analysis had we not taken the time to collect that actual data per women. So if you just said, “Well let’s just take the average cost” you would not something about the incremental follow up but you would never -- the cost would look the same per woman. But by collecting this level of detailed cost information we’re able to do this analysis here. So hopefully that makes sense.

So this was -- I’m not a cancer prevention people but people on our study were thrilled about this and they gave us some additional discussions about maybe you just stick with if there’s a low grade you know, this ASCUS which is what’s here, maybe for that you just send them the letter and maybe you’re outreach intervention just focuses on the people with the high grade lesions where with most effective as well. 

Let me -- if there’s any questions maybe I’ll just sort of hold the questions and try to get the second example; does that work for you Christine?

Moderator:  Yes that works.

Todd Wagner:  Okay so here’s the robot and this is not the robot that many people expect to see; many people think of R2D2 or some sort of science fiction robot. Here’s the MIT McManus robot and you know as an example of a patient on their website using this robot to try to do tasks and they’re trying to move something on the screen with their arm that’s been effected by the stroke. So the robot can guide the movements, the robot can be entirely just with a -- sort of a zero gravity movement so it depends on how you set it up. As the patient gains movement the robot can provide less assistance and challenge the patient .The nice thing about the robot that everybody talked about was this quantifiable immediate feedback on progress and performance. So the thought being is that “Wow it can really target what you need to target for your particular stroke issue” and it gives you immediate feedback so you might be much more motivated to continue with robot rehab.

But robots are expensive; they’re not cheap. So what’s the cost of a robot? Well and so let me get into this, the actual numbers here and then let me tell you a little bit more about the challenges for this. One is so the purchase price of that robot is about $230,000.00. Now if you’re going to spend $230,000.00 whether it’s a house or robot most of us are not going to just be able to pay cash and even if you could pay cash you then couldn’t put -- you could not put that cash into the stock market or wherever else you might want to put it. So most of us have to do financing; so one way of sort of thinking about the opportunity price of that purchase is to think about the financing; so we included financing. Now this robot needs room. And it’s very specific room and you know, set up it needs a dedicated electrical circuit of a certain ampage. It needs certain size room, so we had to calculate all this. Then the one question is what about maintenance? You bought the robot; if it breaks down you’re either having to pay every time it breaks down -- we didn’t have good data on how often this thing would break down so at this point we had to make an assumption about -- well let’s just assume that the hospital bought the maintenance contract. You get the first year free. And then we had to make another assumption which is that in talking to the people and the company we said, “How long is this robot going to live”? And then when it dies does it have value when it dies?

Now think about your car, the depreciation of your car. You bought this car, in a year you put miles on it, you can’t obviously sell it for the same purchase price it’s depreciated. But at the end of five years your hope is that you can sell that car, it still has value. So we had to make those types of assumptions here. So when you make those assumptions you can come up with what’s a net present cost for the five years of purchasing this robot and it’s much more than the $230,000.00 -- it’s $422,000.00.  Now you can say in that five years how many patients can you put through this thing? And we talked all the different sites and they said in the optimal situation a site could offer seven sessions per robot, each session would last about 75 minutes, two patients per session you could have -- there’s different components on each robot. And they talked to people who own these or create these robots, say they’re much more like gym equipment than they are a robot. And so I think that there’s a marketing issue here but robot sound cool. So you can use different components of these things and then you have to clean the components before the patient move and calculate the number of slots over five years and we calculated them to be about 21,500 possible rehab slots. Now if you divide that 422,000 by 21,000 you’re realized that the robot cost per session is about $19.00 -- $20.00 per session. Now you just don’t get one session; you get a course of therapy. And so there’s a course of therapy, you also need some therapist over-- supervision to make sure that these run but you could say per session is about $140.00 and then you have to figure out your whole therapy session; maybe it’s 10 sessions or 12 sessions that you do this.

So here’s -- let me see if the similar table as before. So per session you’ll see that the robot average -- and I apologize my numbers the E fell down there. The robot average $140.00 per session, $20.00 of which was the robot itself. Much more cost was actually the therapist and keeping the therapist on these robots and having them work with the patients. You can see that here is actual clinical trial data on the number of completed sessions; let me highlight that. So in the robot group they completed 32 session. I didn’t tell you a little bit about our control groups here; so let me tell you a little bit about the control groups. There’s the ICT control group; this is just a therapist control group but it’s meant to replicate the robot in frequency. The robot had you do a huge number of repetitions, much more so than the typical usual care rehabilitation. So we wanted to see whether the robot itself or was it just the number of the quantity of movement. So we had the ICT group.

And then we had a usual -- a smaller usual care group. So you get to eventually work out the travel cost of the patients and that bottom row shows you your average cost for the robot, the ICT group and I didn’t highlight the usual care but that’s zero across. It’s -- there’s P value there also that shows the difference. Oops I think that’s my last slide on that example. So hopefully you’ll see that you can work a little bit more complicated -- and there’s a couple of assumptions here and I wasn’t sure I was going to have time but I do have the time. So one of the things we talked about in our paper is the robots, the company that owns the robots is a subsidiary of this professor who works at MIT. They’re built in the US. I imagine over time that this robot was incredibly effective, maybe GE purchases it or all the production moves off seas, off shore to the Far East. You can imagine that dramatically affecting the cost of the robot. But still what we’re showing here -- is what you’re seeing there’s still a therapist cost per session. Yeah maybe the robot cost per session drops to $2.00 if you’re able to make much more inexpensive robots of the same quality. And then you have to figure out a way to do the therapist cost per session. 

The other thing that’s implicit in these models is that you’re fully utilizing all of those slots. Now that might make sense if you’re in New York City and you have a large population, a large population of patients who are having stroke with upper extremity problems. But maybe it doesn’t make sense for you to purchase this robot if you’re in a small town. So quite frankly you could think about well how do you deploy this new technology if it’s highly effective? That raises very different questions than is it cost effective? But I think also very interesting question you can easily come up with data from VA or from assistance to say how do we regionalize this system? The challenge of course is that the full course of therapy is 12-16 weeks so it’s not purely just a one time you know, visit once. You regionalize these how do the patients travel? How does that work?

So hopefully that makes sense. Resources. So if you’re just interested in converting time into money we have some guidebooks on our website. This is the idea if you’re doing some direct measurements and you’ve tracked time, how to change that into money. Converting travel distance into money, you’ll notice in a lot of those tables we talked about travel costs. A lot of those are based on travel distance. And then the last one there I just want to highlight PSSG, so this is a VA group, operational group, planning systems and support group has an actual data set called VAST on travel times; so if you’re part of VA and have access to this you can actually see travel times. Keep in mind caregiver costs; specifically if you think the benefit of the robot rehab is that it offsets some care giving; the patient gets better over time but you’d have to include the caregiver costs. So here’s some sources on Louise Russell who’s done some cyber seminars for us; she’s a professor at Rutgers and looks specifically at that, probably has some of the most advanced knowledge on caregiver costs and the US Bureau of Labor Statistics provides estimates on cost.

When we estimate the cost of labor we’ll just keep in mind that you have to add the employee benefit and often that’s just 30%; you also have to include overhead which is if you’re in VA think of the overhead, the organization that supports you to the HR, the IT, the back office components. You need to include those as well. Now including the overhead costs there’s not great data on what’s the true overhead costs. There was an older report that Arthur Anderson did -- said it was about a third in addition to the benefits. But we have some other reports on that as well; so one of the challenges is trying to figure out if you have direct costs, you then have benefits and then you have to think about the overhead. 

Questions? 

Moderator:  Okay Todd we have a few questions for you and I’m going to try to order them in kind of what part of your presentation they refer to. The first thing I had a few questions about your poll question about whether to use the $3.00 or $14.00 value for the cost of the guide and the discussion on the economy to scale. The first -- the question and comment but in the -- the question is in order to use the $3.00 value would you need to assume that you would be purchasing the number of guides that would correspond to the $3.00 price?

Todd Wagner:  Yeah, that’s a great question and so in many cases what we’re trying to understand is the -- if the system were to purchase this. If it was a societal benefit whether it was Medicare or large systems were to purchase this what would be the cost at that level? So you’re right we’re often talking about sort of system perspective and purchasing. 

Moderator:  And it seems like you would need to know the number of guides you would actually purchase, right? So if you’re doing this from a systems perspective versus maybe a local facility perspective maybe you could vary the costs?

Todd Wagner:  If you’re interested in implementation research and let’s say you had a small facility you might say well maybe it’s not $14.00 because that’s just if you’re doing a small study, maybe it’s closer to $7.00. The key in all of these things is being very open with your readers about how you estimated those costs. So and saying in our study we have $14.00 but when talking to the developers of these guides if we were much larger we’d be $3.00 but we use the $3.00 -- and there’s a citation it’s the 1996 gold book on cost effectiveness analysis that talk about ideally you’d be using it at that constant return to scale so the $3.00 amount.

Moderator:  Okay and then a related question could you use both the $3.00 and the $14.00 values in your analysis?

Todd Wagner:  Well you have to pick one as your primary and then you could easily pick one as a secondary or a sensitivity analysis. You can’t use both simultaneously or if you said they were both valid you’re probably just going to confuse people; so I would say pick one, be honest about which one you pick and then if you wanted to do another one in sensitivity analyses or a supplemental analyses you could easily do that.

Moderator:  Okay. Thank you and then I will move it on to your example with the follow up for ED PAP smears. The question there is would you say that the benefit of the intervention is estimated if incremental follow up is at or near 50%?

Todd Wagner: You’re going to have to repeat that one for me.

Moderator:  I’m not completely -- since the person is -- the person who sent in the question would clarify, please write in. but the question is would you say that the benefit of the intervention is estimate if incremental follow up is at or near 50%?

Todd Wagner:  Let me see if I can go back to that slide and maybe the person can also help clarify. So we had these -- so here’s the -- I guess I don’t quite understand the question -- has the person responded with more details?

Moderator:  No, maybe we can follow up afterwards then. 

Todd Wagner:  I think I’m going to have to just because I’m having a little bit of a hard time understanding the question and I don’t want to say the wrong thing.

Moderator:  Okay. Oh she says go to the next slide. 

Todd Wagner:  This slide.

Moderator:  I think so. 

Todd Wagner:  Okay. The incremental --

Moderator:  Is it because in that column incremental follow up you only have values when the probability of the follow up is greater than 50? So if you read across the rows --

Todd Wagner:  Yep, oh you mean here?

Moderator:  Mm-hmm. 

Todd Wagner:  And the differentials so you could imagine a case where the control group, perhaps this 32% I mean if it’s -- it looks like the control group in most cases is relatively consistent 30 -- around 32%. It was a little bit better for the high grade; so even in the control group here the control understood the letter or the phone call and they said, “High grade, I better get in” so they were slightly better than the average. But you could imagine a situation perhaps where that first group, that .32 for the ASCUS, this one here could have been terrible. It could have been 0% or .05% that got follow up. But you’re right there is a consistent and the intervention group did better across all three of these uniformly but the marginal difference was largest for the high grade lesions. So hopefully that -- the challenge here is that ideally you’d want to say something about quality just in life years, not just follow up. So follow up is important in part because the thought is that you’re able to detect the cancer earlier; you’re able to get the person treated if there is cancer and prevent the cancer. 

So there’s a bunch of assumptions there that play out and you would ideally need a large mark off model where you’re following these people or a cohort of a similar women through this sort of simulation through life. What was the quality just like your games --

Moderator:  Okay hopefully that answers the question; if not we can definitely follow up off line.

Todd Wagner: Absolutely; yeah. 

Moderator:  Next question is the concept of micro-costing applicable for cost effectiveness analysis including comparison of two drugs and if yes, if you could provide some examples.

Todd Wagner:  So you can -- there are times when you can think of drugs -- there are components of it where there’s micro-costing. So for those people who are not familiar with drugs let’s just pick on two that are made by Genentech so Avastin versus Lucentis so these are drugs Avastin is a cancer drug, Lucentis is a macular degeneration drug. They’re both used for macular degeneration Avastin is considerably cheaper; it’s according to Genentech it’s a large molecule and Lucentis is a small molecule that is batched and dosed according to macular degeneration. They give Avastin for macular degeneration; you’re using an off label and you’ve got to allocate it. So you’ve got to break the dose into smaller dosage. Now there is a cost in doing that and so you might want to say “Well the cost of the drugs you -- is just not the purchase price of the drugs and you know the dispensing of the drug you also have to include the breaking that dosage up into smaller dosages. So in that case you can direct measure it. Drugs in general pose a much harder question about how do you estimate the actual cost of the drug. So typically what we see is payment for a drug, a reimbursement for a drug, a wholesale price for a drug. There are typically differences between a branded drug and a generic drug; so it gets very complicated. 

Now depending on who you believe in these stories these prices are very large because they have to -- the companies, the developers need to recoup their development costs and only one in every 1,000 drugs that is started in that pipeline ever finishes and gets approved by the FDA. So I would say it’s not easily amenable to direct measurement and trying to figure out. The production level you say well it’s these chemicals. What is the cost to make and process these chemicals and that probably doesn’t make a whole lot of sense; so most people for drugs use available data after the drug has been approved. Great question.

Moderator:  The next question is would Medicare reimbursement rates for a specific CPT code be a good way to estimate costs from a Medicare perspective?

Todd Wagner:  Frequently we even use them for a societal perspective just because if you -- at least in the US Medicare represents the largest payer of goods and implicitly there’s not built into it you know systemic issues. So if you use one provider’s perspective -- let’s just say you’re a hospital in San Francisco and you say “Let’s just use the cost data from this hospital” but you have to keep in mind that labor costs in San Francisco are 60% higher. Maybe there’s other idiosyncrasies in San Francisco that make it unusual that may not apply generally; so people tend to like to use the Medicare reimbursement rate. That’s very true even if you’re doing VA studies; often people, reviewers will ask you “Well you’re using VA costs, how does that relate to Medicare reimbursement rates”? So yes, that’s very common.

Moderator:  Okay and then there was a comment that actually I think this is on your resources page, the link for caregiver costs doesn’t work. I wonder if there’s a way we can update that and resend out --

Todd Wagner:  Yeah I apologize; it’s one of those Department of Health and Human Services never checks with us and our cyber seminars about when they update their links -- I’m being sarcastic in saying that but you’re right we’ll have to get you some new links on that. I apologize.

Moderator:  Okay. And then another question what’s the difference between macro-costing testing and cost reduction? I wonder if the person meant micro-costing. 

Todd Wagner:  When we do these terms the gold book from 1996 used the terms gross costing instead of macro-costing or average costing we have a web -- a page on our website that has a spectrum and it shows the different levels for precision. So instead of getting hung up on the terminology whether it’s gross, average, micro what I would encourage you to think about is how precise is your cost estimate and if you’re doing direct measurements and you’re tracking each cost that’s going out to each participant in your study that’s a very precise cost estimate and often the people will use the term micro-costing. If you’re using a broader -- even the cost regressions some people might argue well that’s much more of an average costing and I would encourage you not to get caught up in those sort of debates on what’s the right term. Just be honest about it “Here’s the method we use”. 

Moderator:  Okay so Todd there are still a few additional questions; should we go through them?

Todd Wagner:  If people want to hang on we have a few more minutes and then we should probably address those off line if Heidi kicks us off here. 

Moderator:  Okay -- 

Moderator:  Going to kick us off before I do; so we have a little bit of time.

Todd Wagner:  All right.

Moderator:  Okay should I just go for it?

Todd Wagner:  Yeah. 

Moderator:  What distinguishes the societal and the provider’s perspective?

Todd Wagner:  So the societal perspective includes the costs that are encumbered by not only the patient, the payer and the provider. When you talk about a providers cost you’re not interested in the patients cost at all, so the distinctions would be what does the patient pay versus a form by the patient versus what are other people encumbered.

Moderator:  Okay and then are there any resources on how to collect cost data alongside a clinical trial or intervention? These would be costs of not just personnel and equipment but also medications, blood tests, fliers and things like that.

Todd Wagner:  It really depends on the exact type of study that you’re working on and so that’s typically the kind of question that people bring to us and say can you help us out on this, thinking about it and we talk to them about it. So there are resources but it really depends on the study.

Moderator:  Okay and then is the lowest cost always the point at which you would see constant returns to scale?

Todd Wagner:  Boy that’s a theoretical question; not always do we see constant returns to scale. It’s often a point where we assume that there’s some sort of efficiency in production here. And in theory if you get too big you’re average cost can start going back up because there’s additional challenges of trying to administer a huge -- let’s say your company is now 200,000 employees that starts to go back up. There’s also the distinction here between short and long run which I don’t want to get into. I -- any ideas how to answer that quickly Christine?

Moderator:  I think --

Todd Wagner:  Let’s just say use the lowest cost.

Moderator:  Yes. Okay and then when planning an intervention what steps can we take to set the stage for future cost analyses?

Todd Wagner:  That is a great question and what I often tell people especially in this world where funding is limited, what you want to do is do the direct measurements. In future studies if you have precise information on your costs of your intervention you can do a lot. You can go out and get administrative data later on that’s the cost of the healthcare utilization. The other reason for thinking of it in stages like that is let’s say you’re testing out this really novel idea and you just want to estimate the cost of the intervention but you don’t know if it’s even going to work. Well at the end of the study you’ll figure out is it effective or not. If it’s not effective you’ll have a hard time justifying a cost effective analysis. If it’s wildly effective you’ve got great data to say now let’s do the cost effectiveness data and in fact we understand the cost of the intervention already and what we really need is a way to go back to collect the administrative data on these people healthcare utilization. So trying to capture the costs of the intervention importance and if there’s further questions just let us know. 

Moderator:  Okay should we try for one more?

Todd Wagner:  Sure.

Moderator:  Is there a relationship between the billed costs from a hospital or clinic and the reimbursement rate?

Todd Wagner:  So payments costs and bills and charges. So there’s chargers are known to be wildly inflated. There was a great paper, an inquiry, and I’m going to date myself it was in the 1990’s by Schwartz looking at billable and what’s a billed cost and if you’ve ever had a loved one that’s gone into the hospital for heart bypass we think of the cost as heart bypass for that 10 day stay somewhere around $40,000.00, the bill that you’re going to receive is like $140,000.00 or more. They’re wildly inflated. Now there are ways that people have come up with which are trying to convert those bills back to cost and if you’re working with VA data we actually don’t even have billable, all we have is cost. So the keys here are looking often at Medicare payments. People have a lot more faith in the actual payments rather than the billable. If you’re stuck just with billable most people use what’s called a cost to charge ratio for the hospital to deflate that billable. Often you’ll see cost to charge ratios around .25. 

Moderator:  Okay Todd actually I lied, I have one final question; I think this is a great one to end on and then we can follow up with the others off line.

Todd Wagner:  Yes.

Moderator:  How do I contact you if I need specific help with a cost effectiveness analysis for implementation study?

Todd Wagner:  I would say the easiest way is to email HERC@VA.gov. We have a rotating help line and it sets triage down to the different health economists here and we try to answer your questions. I will note that we are usually over committed but we do our best to help you if it’s finding local resources or addressing questions and handling all those that we can. We just don’t typically have the time to do the analysis for you. How’s that? Did we make it?

Moderator:  We -- someone asked could we -- actually we have a few remaining questions I think we can follow up with off line;  they’re a little bit more --

Todd Wagner:  Yeah I think we’ll have to follow up off line. I know we’re going to get kicked off here in a second; so I appreciate people hanging on so long. 

Moderator:  I just put up the slide for the next session in the series scheduled one week from today, May 7, VA Cost HERC versus DSS, Jean Yoon and Todd Wagner will be presenting for that session. There is a link right on the screen, register here. Just click on that and it will open up the registration page for you. If you guys could hold on for just another minute I’m just about to put a feedback form up and we would  love to get your feedback. Right here if you could just take a few moments and fill that out; we really do read through all of the feedback that we receive. There is no submit button, we do receive it as soon as you click in there. Todd, Christine thank you so much for taking the time to present and assist for today’s session; we very much appreciate it. For the audience thank you everyone for joining us for today’s cyber seminar and we do hope to see you at a future session. Thank you.

Todd Wagner:  Absolutely; thank you so much everybody.

Page 1 of 19