**Department of Veterans Affairs**

**Topics of Timely Interest**

**Evaluation of Improvement Capability Grants:  Lessons from the Field:  Part 1**

**Presenters: Carol VanDeusen Lukas, EdD; Martin P. Charns, DBA**

**February 7, 2013**

**Evaluation of Improvement Capability Grants:  Lessons from the Field:  Part 1**

**February 7, 2013**

Molly: And we’re at the top of the hour, so at this time I would like to introduce our presenters, we have Dr. Martin Charns, director of the Center for Organization, Leadership and Management Research, also known as COLMR.  And that is an HSR&D Center of Excellence located at the VA Boston Healthcare System.  He is also professor of health policy and management and director for the program on healthcare organization studies at Boston University School of Public Health.  His co-presenter was going to be Dr. Carol VanDeusen Lukas, also from COLMR and Boston University School of Public Health, but she does have a substitute speaker for her today.  She does have some throat issues and so stepping in for her is Sally Holmes who is the co investigator for this improvement capability grant evaluation that they will be speaking on and she is also based at COLMR.  So at this time I would like to turn it over to our presenters.

Martin P. Charns:  So, this is Marty Charns, and we’re glad to be with you today and doing this presentation.  We just a little bit more explanation.  Sally has been involved with the project since its inception.  Part of the design and central to the work and because we have to use a sports analogy, a deep bench, Sally is stepping in for Carol who is struggling to speak today.  Her voice—she has lost her voice.  So we’re going to move through the material fairly rapidly in the beginning.  I’m going to start with some background information.  And then we’re going to—I’m going to turn it over to Sally.  And we’ll—we’re shooting for about a forty five minute presentation and time for questions at the end.  So first I just want to acknowledge our team.  We have a large team who has worked on this project.  You see some of the names here.  There are others that we didn’t have room to place on the slide.  And we’ve worked very closely with the sponsor of the project that we evaluated, the office of systems redesign.  

So the first question first topic we want to address is, why do we care about improvement capability?  And there are many reasons, and here are a few.  First off, the Under Secretary for Health had a goal for VHA of continuous improvement.  VHA strives to continue to move to higher levels of performance in terms of being veteran centered, data driven and team based.  And to accomplish this we really need to have the capacity and the will to continuously change and improve.  The priority of building this capability to continuously improve VHA is reflected in a number of things.  One of which is the fiscal 13 senior executive performance plan.  And you see under the critical element of leading people the culture of continuous improvement and learning.  The senior executive demonstrates strong commitment to ongoing real time learning to ensure the sustainability of quality improvement and patient driven model engineering and redesign efforts.

So we see it here in the evaluation, it is a goal for all senior executives in VHA—the organization takes it quite seriously and it is critical that we build this capability.  What is needed is systems thinking and improvement knowledge skills and experience to be able to do this continuous improvement work.  Yet, when the project was started, back in fiscal 2008, few staff in our medical centers had this expertise and very few centers had a very strong improvement culture.  Just to get a sense of the participant’s places of work at your medical centers, and your own involvement, we have a little poll question here and would like you to respond to this question.  In the past year, have you personally been involved in a team to improve work processes or outcomes?

Molly:  Excellent.  Thank you.  We do have answers streaming in from the audience.  Simply click the circle next to your answer.  It is automatically submitted.  We’ve already had about 71% answers the question, so we will leave it open for just a few more moments.

Great.  Thank you to those respondents.  We have an 80% response rate so I’m going to go ahead and close the poll now and share the results.  Dr. Charns you should be able to see those.

Martin P. Charns:  Well, I’m trying to see those, but unfortunately I don’t.  Here we go.  Okay.  So 84% of our respondents have said oh—84% said they have been involved in improvement work.  This just to put it in a little bit of reference is a bit higher than the average in the last survey with a similar question which medical centers range from about 31% to 62% involvement.  And that is much higher than was happening then we saw in 2008.  So we have another quick poll question here.  In what role were you usually involved?  If you’d please respond to these one of these four response choices.

Molly:  Thank you.  We have had about half of our audience respond so far.  So again, we’ll leave it open for just a little bit longer.  Quite a varied response for this one.  We’d like to thank those of you that are taking the time to respond.  It does help the presenters to gear the talk towards the audience’s experience.  Again we’ve had about an 80% response rate, so at this time I’m going to close the poll and share the results.

And Dr. Charns, do you see those?

Martin P. Charns:  No.  I do not.  So it must—

Molly:  Sorry.  I’ll be happy to talk through them.  We have 16% responded not involved in this type of team, 31% said they’re a team member.  18% say they have been a team leader and 35% say that they have been an improvement advisor, facilitator or coach.

Martin P. Charns:  A much higher percent of people with 35% who had been an advisor or coach role.  Okay.  Well let us move on knowing a bit about the audience then.  Okay.  There we go.  So I’m now going to talk about the improvement capability grant program which was designed in 2008 to address these gaps.  It was developed by the VHA systems redesign office and they funded innovative creative and practical approaches to creating improvement capability that engages leadership and front line staff together in activities that improve day to day function of every aspect of VA care.  One thing that I do want to emphasize here is the word capability.  So not only was the office interested in successfully accomplishing improvements, but also to build the capability within our medical centers and VISNs to do improvement work.  

The program was set up so that the office of system redesign competitively awarded thirty grants.  This followed a request for proposals and they had broad guidelines that encouraged local strategies so that medical centers and VISNs could design their programs to meet their specific situations.  In 2009 ten grants were awarded in this program.  Seven of them went to medical center grantees and three to VISN grantees.  Then in fiscal 2010 an additional twenty grants were awarded to eighteen medical centers and 2 VISNs.  There was a little bit of overlap so that there was some medical center grantees could be in VISNs who were also grantees.  They were all three year grants with the total possible funding of $450,000-$500,000 per year.  And at this point all grants funding has now been completed.

The grantees did develop local approaches to building their improvement capabilities.  And we classified them into four different categories.  One was grants that focused primarily on training in improvement methods and nine of the thirty total grantees were focused there.  Nine others focused on targeted clinical improvement projects.  Eight in their proposals presented strategy of combining training and improvement projects and the remaining four focused on building infrastructures such as registries or various kinds of resource centers that would support improvement work.

So I’m now going to turn the speaker over to Sally Holmes who will describe the next section.

Sally Holmes:  Good afternoon.  So the next slide, Marty, thank you.  This is the part that researchers like to talk about the most which is what we did.  This is the methodology for our grant.  To talk about these bullets for a second I want to reflect back on what you saw as slide number eight which talks about the objective of the grant funding in the first place.  What Marty said on slide eight was that the objectives were to fund innovative, creative and practical approaches to doing system redesign work with the goal of creating improvement capability.  So our evaluation focused on these three things.  First of all, did the grants.  Did the sites that got grants in fact were they able to carry out the initiates that they described in their proposals and meet their short term objectives.  Second, by doing that, by doing the work that they had outlined in their grants, did they achieve the longer term goal of building improvement capability in their medical center or in their visit?  And did that work contribute to the development of a learning organization.  And then as we went to each site and talked to them, we also emphasized, because it was important for us to not only know were people able to accomplish these first two objectives but what were the things that contributed to their successes and what were the things that served as challenges or barriers to lack of progress when they, in fact, experience that.

So how we did it.  Again, there are thirty four sites here and a little explanation regarding that is that when we had a site—when we had a VISN that had a grant, we visited the VISN network offices as well as one or two sites that were part of the VISN initiative so the number 34 reflects both the site visits that we did to grant two sites that held individual grants as well as our visits to multiple sites when a VISN grant was involved.  So for each of these site visits, there was a two person team from our larger team who either physically visited the site or interviewed people at the site by telephone at six month intervals.  We are—we have done a number of rounds of this.  The 2009 sites finished their grants about a year ago, so we are currently doing one year follow-up visits with them.  The 2010 sites just finished their grants and so we are doing end of grant visits with those sites at this time.  

The good news is that the majority of the sites we are visiting and evaluating have shown very good progress in being able to build towards improved capability.  Based on ratings of three to four meaning mostly or fully met their objectives on a zero to four scales.  Again going back to our objectives in terms of did they say what they were going to do in the grant?  Did they spread those activities beyond the original pilot or clinical focus area and were they able to sustain their grant objectives.  Seventy-four percent actually did—were able to do what they said they would do in their grants.  Sixty-five percent spread the grant activities beyond the original area targeted for that activity.  And sixty-five percent said that they had planned and were able to sustain grant activities after the funding has ended.

We were able in this research and evaluation, actually to identify a number of key factors that we believe are associated with facilitating the ability to achieve the grant objectives and build capability in the organization.  The three you see on the slide that are highlighted in yellow are the three we are going to spend time on today.  The remaining four in white are we will mention but we will spend more time on in the second session of this engagement.  So we are going to be talking today about the importance of training being linked with improvement projects, about data and skills existing in the organization and the importance of having a substantial improvement infrastructure.  At this point, we have a couple of more questions to you, so first of all, and I will read the question for those of you who may not be on a screen at the moment.  Roughly how many staff in your facility have been trained in systematic improvement methods beyond employee orientation?

Molly:  Thank you.  We do have the answers coming in and it looks like about half of our audience has voted so far.  We will give people some more time to get their answers in.

We’ve had about two thirds of the audience vote so far, but the answers are still streaming in so we’ll give people a few more seconds to select their response.  Thank you.  It looks like responses have stopped at about 70%.  I’m going to go ahead and close the poll.  And share the results.  

Sally Holmes:  So Molly can you read those to us.

Molly:  Not a problem.  It looks like 15% say more than 100. 11% say 50-99.  13% say 10-49.  10% say 0-9 and 52% say I don’t know.

Sally Holmes:  Okay.  Thank you and then the next question is, is there an explicit expectation in your organization that people train will participate in improvement projects when they return from training?

Molly:  Thank you.  I do have some answers coming in.  About a third of our audience has voted thus far.  And we’re at just over half of our audience response thus far, so we’ll give people a few more seconds.  And it looks like our response rate has maxed out at about 66% and the answers have stopped streaming in and so I’m going to go ahead and close the poll and share the results.  27% of our respondents say that yes for all trainees.  25% say yes, for some trainees, 14% said no and 35% said don’t know.  So thank you.

Sally Holmes:  Thank you, Molly.  I’m going to hand the phone now to Dr. Charns.  Hang on.

Martin P. Charns:  Okay.  So very briefly then, the results were on the first question, fifteen percent of the people on the call said 100.  Note that 52% did not know.  For the second question, 27% responded yes for all.  25, yes for some.  14 said no.  And 35% said they didn’t know.  So one of the things that I think is important that we found in the study is that neither training nor conducting projects alone is sufficient.  And the high—just doing a high volume of projects itself does not build the capability unless this is coupled with skill building.  We can see that in several ways.  One related point here is some of the sites had a very heavy reliance on improvement experts either inside the organization or from outside VA and without sustaining—without engaging staff in the projects that were being sometimes done by those experts and sometimes facilitated by those experts, but without engaging the staff, they had the projects had little impact on building the skills.  Similarly, or conversely when people were given training, often in formal sessions, and did not have the opportunity to apply the skills that they just learned, they did not retain those skills.  In fact one of the things that we observed is that people would come back from training and it was not until they really tried to apply the skills that they didn’t know what they didn’t know.  And further, the reinforcement of applying the skills to real situations was very important for honing the skills and retaining them.

What we did find was that those sites that combined their training and their project foci had a substantially greater success and in those cases, the trainees would work on projects either during the training or immediately after. For example the trainees might bring a project to the trainee session and focus on it as they were learning the skills.  You know?  Or there would be the consultation following up on the training, really helping them typically through improvement advisors to help them learn the skills.  We don’t want to preclude the opportunity here to training while doing projects.  So it doesn’t have to be formal, but what really did stand out was the importance of combining the training and the application to improvement projects.

The other piece that you can see in the last bullet is the importance of ongoing support of the new skills by supervisors and by the improvement advisors.  And you can imagine the situation when a trainee comes back to the work situation and at worst gets a luke warm reception by his or her supervisors for the application of the skills but important is to have that support by the supervisor.  Now all of this is not easy. 

 And you know as one of our earlier bullets highlighted the next area that we wanted to address is the importance of the data and the skills to analyze the data.  We all know that information is critical for improvement work.  Systematic data on processes and performance are critical in diagnosing the problem, identifying potential solutions and monitoring progress.  

Application in four areas we highlight.  Selecting and directing projects, tracking progress in meeting the goals, sharing and discussing at different levels of the organization and then monitoring usually after the improvements have been put in place to detect changes in performance.  Typically declines in performance indicating that the changes were not being sustained.  So of course measurement is critically important for all of these areas.

We have another quick poll question here asking how deep is the improvement expertise in your medical center.  So if you would quickly respond to this.

Molly:  Thank you we do have people responding to the question although at a little bit of a slower rate.  We’ve had about half of the attendees respond so far.  So we’ll get people a little bit more time.  All right.  We’ve had about two thirds of our attendees respond thus far.  And it looks like the Reponses have stopped streaming in.  in the sense of time I’ll go ahead and read through the results myself.

So we have 46% responding that a few people in an improvement/quality related office.  25% people across the facility with expertise to coach teams.  And 31% don’t know. 

Sally Holmes:  So this is Sally and I find those results very interesting because when we asked the question earlier about the extent to which people had perhaps been involved in improvement activities the proportions of people who were involved in improvement activities were fairly substantial and in this question when we start to ask about the depth of the improvement expertise in your medical center, the majority of responses here say that a few people have deep improvement expertise in the medical center and I would suggest because we’re heading towards the open microphone section of this meeting that that’s one of the things that would be very interesting to have a dialog about.

So Marty was talking a minute ago about the importance of moving from projects to being able to apply those projects or apply the knowledge that you gain in training to projects when you came back from training and perhaps sometimes even an iterative process where you go and get trainings, you come back and do projects.  You come back to training to talk about your experiences to fine tune your expertise.  I think one of the messages, and Marty can we move to the next slide, please.  The one of the messages about the importance of doing both training and projects is that it is not easy.  That what we just talked about is not an easy task for an organization and in fact I think there are some real dangers in just sort of attempting to do both training and projects without thinking strategically about how to do that and how to support that in the first place.  Infrastructure, therefore, going to the slide is very important because it provides both the structure and the capacity to support people when they’re doing projects after they return from training and in order for your medical center or your organization to be able to support people once they’ve been trained, there needs to be both the structure for doing that and the functional expertise in the organization in order to do that.

Some of the challenges we’ve seen as we’ve talked with sites who are working to do this is that the staff in our organizations whose job it is to support others who are trying to do performance improvements—the redesign work, are at different levels of expertise.  That that some of them may be newcomers to the whole science of system redesign and others may be black belts. That’s one thing to think about. The second thing we hear a fair amount of the time, is that there is a tension between being able to assign people to be facilitators, coaches and mentors and allow them to have the time to do that versus asking people to do that as part of what’s typically called collateral duty here in the veteran’s administration.  So if you have someone doing this only on a collateral duty basis you can imagine that person gets pulled in many directions and may not be available to provide the support and direction and coaching that’s needed and we also observed in some sites we talked with that are in fact there is someone who’s been designated and has the skills to do that.  If that person is working alone in the organization then the demands of system redesign or the demands of project in the organizations quickly can become overwhelming for the person.

Next slide.  Thank you.  So based on what we’re hearing, we believe that with regard to infrastructure, solid capacity is needed in these five key areas.  First of all, it’s perhaps obvious, but you need to have staff who have the expertise to lead and guide and mentor other staff in doing improvement work in the organization and especially people who have the experience to be able to oversee the improvement work that’s being done.  And we talked about staff here so its people but it’s also structure.  It’s structure so that as you know what improvement work is going on in your organization and you have the ability to know whether you are in touch with people who are doing this work and whether it is going well or not.

Clearly that requires sufficient numbers of staff in terms of system redesign, resources.  There are different jobs here.  There are management leadership visionary kinds of jobs.  There are coaches, mentors, facilitator kind of jobs.  There are people who need to know data in order to do this work and if you’re going to have an infrastructure you need to have both the right staff and the right number of staff to be able to support this function.  You need a clear plan in terms of what is going on and what the expectations are for people engaged in performance improvement system redesign.  And what the expectation are for support at all levels.  And I want to pause for a moment and emphasize all levels.  So it really in terms of support at all levels it ranges between the front line in an organization where support might play out in terms of staff having the time and the leeway to be part of multidisciplinary teams, working on improvement.  It requires people who are process owners in the organization to be involved in this work and to be available to do the work.  It requires support in terms of—I said several times already the expertise to guide the work as it goes along and frequently the support is important at the senior leaders in the organization because it’s the senior people in the organization who can help resolve problems and guide work and tie the work to strategic objectives and we’ll spend more time talking about that next week, but this concept of support at all levels is one thing we have seen that is critical.

There need to be clear linkages between training and the projects people have available to work on when they return from training.  So it’s not enough to send someone off to get a yellow belt and then say you need to do a project as part of this yellow belt training.  Good luck.  There needs to be someone making sure there are projects, that these projects are chartered and blessed by the organization and there is a way to match the staff who are going to be doing this work with the—with what needs to be done and the scope of expertise required to do the work and then as we said and implied earlier, it’s really important that this is iterative.  It’s not enough to send a person off to what some people sometimes call performance improvement or system redesign 101 to teach them a model or to give them tools for quality improvement and to set them loose in the organization.  There needs to be oversight and there needs to be refresher training because as I believe Marty said earlier, you don’t know what you don’t know until you’ve had some experience trying to do this.

So this is sort of a preview for next time but building in the way that we’ve been talking about today in terms of sort of linking back to having the projects; having the training; having the infrastructure begins to move an organization in our experience as we’ve watched these grant sites try to do this.  Begins to build an organization where a culture of improvement is developing.  So in our experience with this project and others an improvement culture consists first of all of engaged staff.  And engaged staff are not only people who are trained to do the  work and participate in improvement activities but also people who are excited and sort of optimistic about being asked, told of the work and help improve things in an organization.  The second aspect of an improvement culture is what I just talked about in terms of having expertise.  In terms of skills and structure to support the improvement work.  And we will talk more about structure next week, but structure is important in a number of ways.  It’s important in terms of sort of where you place the improvement people and skills in your organization and who’s part of that group and what they bring to the table and it’s important in terms of where they report, who’s overseeing their work and how they report.  How do they take the work that they are overseeing the organization and make sure that others in the organization especially people above them in the organization know what’s going on and know what they’re accomplishing.  

And where do they bring the problems?  Where do they bring requests for resources and where do people bring issues that need to be coordinated or resolved.  Skills, infrastructure and structure are another component of culture and as is implied by what I just said it’s also important that the senior leaders in the organization are engaged in the improvement work, in the process improvement.  What do we mean by engaged?  I can go back to how I defined engaged a minute ago in terms of being excited and enthusiastic about it.  Generally what we are seeing when the site has engaged senior leaders is that there is evidence of personal commitment to what’s going on in the organization with regards to improvement that—so this is not something that the organizations has to do with something that the organization and senior leaders really feel is important and are behind the effort and then because it’s a priority, because it’s important, part of being a leader is aligning your resources around the things that are important in organization and making sure that those resources are in place.  It’s really putting your dollars where the spoken word is in an organization.

We’re going to move onto this issue of culture where we’ll spend more time next week.  I would like to raise this poll question for people.  How would you rate the improvement culture in your facility?

Molly:  Thank you.  We have launched the question and about a third of our audience has answered so far.  And we’re up to 50%.  Responses are still streaming in so we’ll give people more time.  All right, it looks like answers are starting to slow down.  We’ve had about two thirds of our audience response and at this point in time I’m going to go ahead and close the poll and share the results.

So for the poll question, how would you rate the improvement culture in your facility, 17% report minimal, 67% report developing, 10% answered mature and 6% said don’t know.

Sally Holmes:  Thank you very much.  Those are very interesting results and I’m particularly interested in the fact that the overwhelming majority of people here said developing.  So I think we have one more slide in this section.  And then we are going to open it up to questions and discussion. So to recap what we’ve talked about and to preview what we’ll talk about in part two, first of all, we’ve talked about the importance of doing both training and doing, having improvement projects available to the people who are trained so that people who are trained and being trained have access—have the ability to apply their learning and their skills.  Second of all, we talked about the importance of having data to support these activities and having the skills available to analyze and manipulate data and to support improvement work.  And third of all, we’ve—I just talked about the importance and the characteristics of strong improvement infrastructure.  I’m going to hand the phone back in a second, but the white things in the slide from my engagement, management engagement, senior leadership engagement and strategic alignment within the organization of improvement priorities are the things that we will spend more time on in our next session.  So one moment—hold on.

Martin P. Charns:  So we’re now open for questions.   And we’re going to—we need to coordinate this with Molly because I believe a number of questions have probably come through so far in the session.

Molly:  Yes.  We have had a number of questions come in.  Roughly about ten thus far so those of you who joined us after the top of the hour, simply type your question or comment into the question section of the go to webinar dashboard on the right hand side of your screen and we’ll be able to ask the presenters in the order in which it was received.

Regarding a preview of part two, I just want to make note that you will be receiving a registration e-mail for that session in just about an hour, so please keep your eyes peeled for that.

With that the first question are you at liberty to provide more details about how the grantees were?

Martin P. Charns:  We can do that.  It’s public knowledge certainly within VA.  Since there is a long list, thirty, we can’t do it off the—well we do know them all, but instead of going through all thirty we’ll make the information available.  We could—we will add it to our slides next week and that way it will be archived and you can look it up.

Molly:  Excellent.  Thank you.  The next question, you indicated that 68% of sites perceived that they have sustained gains past the funding cycle.  Is this consistent with what you are learning in your one year follow up?

Martin P. Charns:  So in many cases not only are we seeing the commitment for funding the positions that are now coming out of the medical center budgets rather than the project—the grant budget.  But we’re also seeing monitoring in the data you know that those improvements are being sustained.  The question if you look at the literature on sustaining improvement or sustaining evidence based practice, there is a question of how long and so it a recent review article suggested at least two years before we’re convinced that an improvement or new practice has been sustained.  We’re not there yet with any of the grantees, two years post grant.  So we’ll have to keep watching but we were encouraged by many sites of being able to sustain.

Sally Holmes:  Most healthcare organizations today are resource constrained and one would expect to see shifts and changes not only over time as capability becomes baked in, but also as these extra resources go away the organization in the study have had to really work hard to figure out how to keep this going in the absence of those special resources.

Post grants, I agree with what Marty said.  We are seeing that the activities are being sustained.  They may look different or be structured differently than they were doing the grant and a lot of the sites that I’m personally familiar with have been quite creative at figuring out ways such as engagements with their academic affiliates to keep the work going that don’t require additional resources.

Martin P. Charns:  So we’re going to try to—Carol is going to try to say something to contribute to the discussion.  You may not recognize her voice.

Carol VanDeusen Lukas:  I think the other thing, since the improvement capabilities grants started in 2009, the office of system redesign and other parts of VA have done a lot more to deal with improvement capability so that’s another thing that we’re going to talk about next week is the variety of other initiatives that are going on to support system redesign and more broadly improvement.

Martin P. Charns:  And maybe just one final comment on this question is we also have to take into account the scope of the improvement effort and you know look at consider the responses that we got you know to the question on culture of improvement and most people said it’s developing.  While we have good evidence around some specific improvements being sustained, we also do not have the level of spread or culture change that we’re hoping that you know will ultimately be achieved.  A small number of sites have achieved a pretty good level of broad culture of improvement but it’s only a small number.

Shall we go to another question?

Molly:  Yes.  Thank you.  We do have fifteen pending questions.

Martin P. Charns:  Oh wow.

Molly:  Moving right along.

Martin P. Charns:  Shorter answers.

Molly:  That was my subtle cue for shorter answers.  Okay next question.  Was there a particular model of organizational change that guided the evaluation?

Sally Holmes:  We should have mentioned that actually.  So the model that was underneath some of our assumptions was the organizational transformational model which was ... Marty do you want to...

Martin P. Charns: We developed this model out of an earlier project we did in both the project sector and a project that we did in VA.  Carol, Ellie and I plus others were involved in it.  There—we called it the organizational transformation model and we can provide references to it.  Very quickly, five drivers of change.  Improvement initiatives, impetus for change.  Alignment, which we’ll talk about next week.  Integration and of course, leadership.  We’ll cover at least two or three of those in next week’s presentation.

Molly:  Thank you for that reply.  The next question have you considered conducting a preview of what the folks will be trained on for the supervisors?

Martin P. Charns:  I don’t understand the question.  I—a preview of what the folks will be trained on for this—supervisors?  In other words that supervisors should learn what their staff are being trained on—I don’t know if we can--?

Molly:  They want clear expectations for the supervisors as to their role and responsibility that might be helpful.

Martin P. Charns:  The best we have seen is when the supervisors have both the training and the experience so that they can be then supporting their subordinates.

Sally Holmes:  One of the things we didn’t have time for today is to give you sort of real life examples but there is a site that I know that is doing the model of front line improvement work that some of you may have heard you associated with beta care and that organization feels strongly that if you are working in it, you’re going to begin doing huddles which is part of the front line improvement model.  You have to be trained in that model as a unit supervisor before you’re basically going to be encouraged to start that work.  I think that’s the thinking behind that question is really the important thing.

Molly:  Thank you both for those replies.  And the next question, did you find that since the ICG sites had financial support, that leadership was more supportive as well or do you think this is just the product of an ongoing leadership support?

Martin P. Charns:  So we did not find the level of leadership support in all of the sites.  So we had a mixed situation.  In some cases getting the grant might have gotten leaderships attention.  In other cases it was the leadership that was encouraging going after the grant.  So that desire was there first.  And in those grantees and this evaluation was limited to the thirty grantees where the leadership support was strong on the very top, the quadrant down through each level to the front line supervisors is where we saw the most progress.  I’ll stop there.

Molly:  Thank you for that reply.  It seems to me that an intervention at the organizational level might be needed; leader’s supervisors and peers might need a “pretraining” progress to support and prioritize new practices.  What kind of interventions are available for this level?

Sally Holmes:  We’ll talk more about this next week but I think there are a variety of interventions with all of the VISNs in the last two years have gone through leading organizational improvement workshops that I think address some of those issues.  There are pilots now about lean deployment that I think address those issues.  There are things underway.

Molly:  Thank you.  Next question.  Are you able to discuss quality of the improvement programs, work that has been done in these organizations surveyed?

Sally Holmes:  Is that question about whether we can discuss the types of projects that people undertook and the outcomes that they had?

Carol VanDeusen Lukas:  Or how well they did?

Sally Holmes:  Or how well they did.  I’m not very clear?

Molly:  I will ask that that question submitter write in with further clarification—oh.  He said exactly—so—

Carol VanDeusen Lukas:  Let me see if I can rephrase the question.  The question is are we able to talk about specific projects and outcomes?  Um.  We certainly have that information on a site specific basis.  I’m wondering if what’s behind the question is the—

Molly:  Essentially he’d like to know if these programs are effective.

Martin P. Charns:  How.

Carol VanDeusen Lukas:  The problem is they’re so varied that there isn’t a single standard of outcome.  So the project—the grants that were project focused that were taking on a particular clinical issue we have information about whether—they usually were but in a few cases they were—had limited success  in meeting their objectives.  When we’re trying to evaluate was the training successful, the sites have information about the typical question of whether people were satisfied with the training when they came back.  They—we are also trying to work with them to have data on—so when people came back did they use those skills in our improvement project and there the information is more mixed.  There’s less of [inaudible].

Martin P. Charns:  So the [inaudible] sites where there was a better combination of training and improvement projects directly you know both the projects were more likely to be successful because people had the skills and their learning stuff better—or they understood better because they applied it but we have examples of sites where projects were successful, projects were not successful.  Training did or did not work.  And all combinations.

Sally Holmes:  So we have—just very quickly we have examples of sites where there are definitive improvements in either work processes or outcomes.  We have many sites or several sites that are able to demonstrate the financial impact of the improvements that they made to the organization and I think we as researchers learned as much from some of the failures as we learned from the successes.

Molly:  Thank you for those replies.  The next question, I know at least one awarded site had interim leadership are you able to connect this within their rate of progress.  It seemed like they really remained complacent for the first year?

Martin P. Charns:  Yes.  One of the things that we’ve reported back to the office of system redesign and various committees in the VA and the sites is that the difficulty when the leadership changes and vacancies and mostly we’re looking at that at the quadrad but also in other key positions including in the system redesign coordinators.  So any time that change takes place or there’s a vacancy, you can imagine that’s a negative impact on the program.  Of course there are situations where a new person comes in who is more committed and/or more skilled around improvement or situations where the converse occurs and the staff are then quite let down especially when they’ve made a big investment up to that time.

Molly:  Thank you.  We do have nine pending questions, are you all able to stay on and continuing answering.

Martin P. Charns:  Keep going.

Molly:  Do you happen to have any statistics that examine computer learning versus hands on conference venues, if so can you share the reference?

Martin P. Charns:  No.  No means I don’t think we have that systematically at all.

Sally Holmes:  I [inaudible] that question to—I think we should put that question aside because I think there may be resources in the larger system, but we the research team don’t have that.

Molly:  Thank you.  Did you identify any best practices in tracking return on investment and sustainability of improvement efforts?

Martin P. Charns:  Let’s start with sustainability.  Obviously when things are built in to the ongoing work process the likelihood of sustainability is much higher than when we’re just asking people to work harder.  I’ve given extremes but I think you understand he practice there and certainly there were sites that were just trying to get people to either focus differently or work harder without guiding it into the system and others that built it in more effectively.  The other half of the question is—

Sally Holmes:  I would suggest, among other things that there is in the leading organizational improvement workshop material a template for thinking about RLI.  That is available through the office of system redesign.  However, I got more excited about things that weren’t quite as CFOE as that formal ROI implies. I had talked to a site where they did a cleaning up your supply room exercise, realized that they had a fair number of supplies that they hadn’t used in a long time.  Were able to return those supplies and got something like $60,000 in savings, just from that.  One short, rapid cycle improvement.  So there is sort of ROI on capital and the smaller issues of you know are the savings inherent in this.

Molly:  Thank you for that reply.  I’m going to ask that you refresh the screen real quickly

Martin P. Charns:  Sure.

Molly:  The next question, where these grants awarded to projects changing clinical care or did it also include other things, changing R&D environments?

Carol VanDeusen Lukas:  The focus of the projects was not clinical care, so the emphasis of the RFA if you do a clinical project or an administrative project it’s to be in the service of creating improvement capabilities.  So where they did these projects, most of them were clinical, I think for those where the focus was on training and people were selecting projects, as part of their training experience, they did include administrative projects.

Martin P. Charns:  I thought the question also asked if any of the projects were R&D.  our colleagues in research in VA.  Unless I heard Molly wrong and I can’t think of any that were designed around that.

Molly:  Thank you for those replies.  Next question, where can I access the study that Dr. Charns referenced?

Martin P. Charns:  The transformation article.

Carol VanDeusen Lukas:  We’ll send it to Molly and she can put it up or we can include the reference in the slides next week.

Molly:  Excellent.  If you want to send me any supplemental materials and want me to put them in with the archive.  What does a facility that has an organizational structure for data validation look like?  If data is critical what structures are you seeing in your evaluation and what recommendations can you share?

Martin P. Charns:  Wow.  So fortunately I can think of a number of sites that did not use data as effectively as they needed to, but others where they either had terrific relationships with various people with data analysis skills and you know could either be on the IT side of the house—I know that may be hard to believe for some places or researchers who could be helpful there—I could think of one site where it was a [inaudible] that was the research entity or an entity that has research as part of its activities and you know especially one individuals who had just terrific knowledge of the data systems and analytical skills and you know was able to you know be the central person for measurement for their improvement work and it played a critical role in the transformation of that project.

Sally Holmes:  And in other sites they explicitly had [inaudible] out of their clinical structure their improvement applications something.  Well that doesn’t speak directly to validating the data but it does signal an orientation to having access to solid data in the medical center.

Carol VanDeusen Lukas:  And I would add in our discussion of data today that we perhaps didn’t go deeply enough.  We were trying to manage the time, but I think that you know I think that some of the sites that I have worked directly with have spent a significant amount of time with what data is out there, how is the data sources different, how can I bring this data together.  How can I be sure we’re measuring what we’re trying to measure and what is the validity and reliably, and how can we use that data to either inform a project or more importantly to sort of move the successes of that project to a dashboard environment where the organization can see that data and understand what’s happening as a result of performance improvements.

Molly:  Thank you for those responses.  We do just have five pending questions.  Any suggestions for staff developing an infrastructure for a new program, expertise for the program and assisting coworkers with the change process?

Martin P. Charns:  That’s a really big question but one thing that—one model that we saw work pretty well was a relatively small but highly expert group of let’s call them centralized improvement advisors and then a more diffuse but not necessarily as expert advisor or you know—folks with yellow belt capability diffused throughout the medical center so that you have both immediately available resources as well as that higher level of expertise that was needed for some projects or for coaching you know of around individual projects that went beyond what the “local” resource could provide, but the other piece that we want to keep emphasizing because it’s so important is the leadership involvement with this and the leadership support for both.

Carol VanDeusen Lukas:  But also paired with getting the front line staff directly involved in designing that new initiative.

MHP:  Right.

Carol VanDeusen Lukas:  And not coming up with solutions to the problems they’re trying to face.

Sally Holmes:  So one model I see more and more not only in this study but other studies like this that I’m involved in is that in the beginning the infrastructure is centralized.  You need to build a solid core and you need to build consistent expertise and over time that expertise often becomes decentralized in the organization which leads to what Carol and Marty are talking about in terms of diffusion and really sort of engagement throughout and at all levels of the organization.

Martin P. Charns:  And just one of the things because you asked about the change process.  One thing that might seem a little counter intuitive but is if you train a lot of people in the beginning and spread them they come from many different parts of the organization, the organization then faces the challenge of providing the support for all those people and too many of them I we’ve seen you know don’t have the opportunity to use the skills that they’ve been trained in and we have the even negative effects that we’ve talked about earlier.

Molly: Thank you all for those replies.  Next question it’s a long one.  System improvements shown to bring about efficiencies and therefore cost savings yet cost savings often push organizational resources and focus for improvement.  What are some strategies to keep the cart behind the horse and client and fiscal and staffing constraints?

Martin P. Charns: Well one of the things we had one bullet point on is or maybe two is that it does take resources to do this work and one of the advantages of having a grant is it could jump start and free up resources that then could be focused on specific projects if successful, of course, then you have a sequence of greater efficiencies on all projects, but some projects and those resources then being available you know for reinvestment.  I don’t and I because this is question and answer live here, I don’t have a better solution for kick start or for the beginning of making some resources available other than the commitment of someone who controls the resources to say this is so much of a priority for our organization that we’re going to do it, and you know we’ve seen that happen that they just—it is a priority and I will you know prioritize the resources to align with this.  You know but so—just an observations.

Sally Holmes:  The people I’ve met who are passionate about building improvement capability in their organizations believe that cost follows quality so first of all the initial emphasis is on quality with the belief that cost follows.  Some people I’ve seen, although I’m not sure if I’ve seen this in the VA, some of these passionate champions for improvement have actually made a bet with their organization that they can cover their salary or cover their department’s salary with the savings from the work they do.

Martin P. Charns:  And we have seen that happen.

Sally Holmes:  Uh huh.

Martin P. Charns:  A number of times.  And in our sites as well as in this project as well as outside.  One thing that Sally mentioned here which wasn’t directly asked but often we’ve seen situations where the focus is specifically on an efficiency rather than on patient centeredness or quality of care and it becomes very difficult especially in the beginning of a program to build improvement capability to get a lot of people engaged around that because it’s just not a high enough priority.  Think about the commitment that people in VA have to our mission of caring for veterans and in most healthcare organizations the commitment you know for providing care—efficiency is important but that itself is rarely the selling point.  

Molly:  Thank you for that reply.  I have just a few things I want to mention before we move on to the last question.  Number one is that someone did write in and share a SharePoint site which hosts all the grant proposals, so people can either follow up with Dr. Charns and his team or e-mail cyberseminar@va.gov and I can share that link with you.  Obviously that’s for VA employees only.  Also several people have written in saying thank you this was an excellent presentation and they’re excited to join next week.  And the final question we have were the grants awarded at the facility level or the service level or the team level?

Martin P. Charns:  The grants were all awarded at either the VISN level for five or the facility level for twenty five however, a number of the grants particular service line or a particular disease if the facility wasn’t organized in a service line way, so in fact, you know implied in the question is that the focus of the grants in some cases was at a smaller entity than the full medical center.

Sally Holmes:  I would say if you were to look at the RFA and the emphasis on the RFA is building organizational capability which is why the grants, I think were awarded at the medical center.  However, the initiative was often quite specific and quite local.

Martin P. Charns: And that then does create the challenge of how can you spread improvement capability beyond what was the focus of the grant initiative.

Molly:  Great.  Thank you for those responses.  That was the final pending question.  So once again I want to remind our attendees that you will shortly this afternoon receive the registration e-mail for part two of this presentation and with that I’d like to give our presenters a chance to say any concluding comments.

Martin P. Charns:  We appreciate your interest in the topic.  It’s a shared interest that our team has.  We believe many of the things that we’re seeing apply not only to improvement work but also to implementation of evidence based practices so those of you that have interest there thinking about how the work that has been done here around improvement and improvement capability may generalize and we will make the materials available that we spoke of and we look forward to next week.

Sally Holmes:  And I would add to that very quickly for those of you who are listening to this who were the subjects of our evaluation, we want to acknowledge that this has been a privilege for us to be with you on your journey.  We appreciate that.

Martin P. Charns:  We’ve learned from you as we’ve been conducting this work.

Molly:  Great. Thank you very much to our presenters and for our attendees for joining us and I’d like to encourage our attendees once you exit the session, if you wait just a moment a survey will populate on your screen.  Please take just a moment to give us some feedback.  Your opinions are what [inaudible] the topics we present on.  Thank you once again to everyone and enjoy the rest of the afternoon and if you’re in the New England area, be safe this weekend.  Thank you.

**[End of Recording]**

Page 1 of 16

Page 2 of 16