**Transcript of Cyberseminar**

**VIReC Clinical Informatics Seminar**

**Creation of a Mental Health Information System to Support VA Office of Mental Health Operations Quality**

**Presenters: Jodie Trafton, Ph.D.; Jeanne Schaefer, R.N., Ph.D.**

**March 19, 2013**

**Creation of a Mental Health Information System to Support VA Office of Mental Health Operations Quality**

**March 19, 2013**

Moderator:At this time I would like to introduce our speakers for today; Jodie Trafton, Ph.D., and Jeanne Schaefer, R.N., Ph.D. Dr. Trafton is Director of the VA Program Evaluation and Resource Center for the Office of Mental Health Operations, and a research health science specialist in the Palo Alto HSR&D Center of Excellence known as the Center for Healthcare Evaluation. Dr. Schaefer is a research health science specialist in both the VA Program Evaluation and Resource Center and the Palo Alto HSR&D Center of Excellence. Without further ado, may I present both Doctors Trafton and Schaefer? Thank you very much.

Jodie Trafton:Hi, thank you. This is Jodie Trafton. 

Jeanne Schaefer:This is Jeanne Schaefer.

Jodie Trafton:We are excited to tell you a little bit about an information system that we developed as part of the Office of Mental Health Operations. I am going to give you a little bit of background on it. Because I think it is a little bit unique in that – in that we were started as a brand new office and tasked with creating a system to help support the office activities. That is the system we are going to tell you about. 

I also want to mention that while both Jeanne and I are from the Program Evaluation and Resource Center, we worked with the – our two other evaluation centers.  In the Office of Mental Health Operations. That is the SMITREC, the serious mental illness center, which is located in the Ann Arbor and NEPEC, the Northeast Program Evaluation Center in West Haven. This was definitely a very large group effort. We also worked in conjunction with the rest of the mental health operations office, including our technical assistance program. We are going to talk a bit about that as well. That program is led by Lisa Carney. 

To start just a quick overview of what we are going to try to cover today. I am going to tell you a little bit about the Office of Mental Health Operations since we are a brand new office. I will tell you about what we were tasked with doing and what our mission was. We will then tell you how we developed the mental health information system. Why we designed the features the way we did. 

Then we are going to talk about how that mental health information system has been used as the core to facilitate and evaluate our offices nationwide quality improvement program. That program includes site visits and technical – a site visit and technical assistance program, an action planning system; and a best practice dissemination program. All designed to help the field better implement mental health policies and improve the quality of care. 

Also, we help with specific initiatives that are started in VACO to help make sure that those are successfully and – successfully implemented and well supported by Central Office. To start – so, our office was created a couple of years ago now with a reorganization of the VA Central Office. We were basically broken off of the former policy branch of patient care services and put in a new office under 10N, under operations as part of a group of offices created to support clinical operations. 

The main tasks of the office; or the reason the office was created is that we were supposed to help make sure that policies created in patient care services were effectively disseminated to the field and implemented in the field. We were supposed to interact with the field to find out what sorts of problems they were struggling with. What sorts of barriers they had in order to implement policy and ensure good access and quality of care for mental health. We were supposed to help try to reduce variation across the system so that mental health care received in one facility would be of similar acceptability, and qualities, and content as services delivered at any other VA. 

Our office’s first task, the main things that we were asked to do initially was to try – was to focus on the facilitating and ensuring the implementation of what is called the Uniform Mental Health Services Handbook. This is very comprehensive policy document created in 2008, which covers all of the services that were to be delivered in VA’s mental health programs. All of the requirements and also specific on how they were meant to be delivered. For example, a Veteran should be able to pick the gender of their provider, for example. 

We were also tasked when reducing variability as I said in mental health treatment access and quality across the 841 healthcare systems. Those were the big tasks we were asked to follow. We were told we needed to create a – to develop an information system that would help guide – help guide those efforts. The system that we tried to create was focused on helping both VISNs and facilities implement both sets, or all of the policies in that handbook. Also guide their quality improvement efforts. Fix up places where they were having specific access problems or difficulty delivering services in alignment with requirements. 

We were also, our office was tasked with creating a national mental health site visit program. The information system was also designed to help guide that site visit program. The site visit teams would go in with knowledge of – with a background knowledge of the current state of the system. What potential concerns or strengths that the program might need. It was – the system was also to – also support – to support a national technical assistance program. 

This program was designed. We have a team that works with the VISNs and the facilities to try to come up with specific actionable plans to address concerns at the facilities. That can be both; we have parts of this program that are directed by us that we will work with a facility directly to try to get them to address concerns that we have identified together. Or they can contact us for information or help with anything that they need. 

Lastly, we know that there is lots of fabulous local innovation that happens across our facilities. We also wanted this system to help us identify those best practices. Get them share and disseminated across all of the VA. 

Okay, look, there we go. The first goal of the information system that we were just to develop was to try to assess the level of implementation of all of the key elements in that mental health services handbook. This is a very large document for anyone who has actually had to try to implement it. You will know that this is – this is not a two page policy. It is many pages. Yes, it is the steps. The handbook is divided into domains. 

Each of those domains describes the key principals or requirements for either how to treat specific patient populations, how to run or design certain specialty programs, or what sorts of processes and care need to be in place.  The handbook has been up to the development of the system. It had been evaluated by a self-report survey that asked facilities to let us know whether they had done any of – anything closely resembling that requirement in the past. For example, if the handbook said that all patients with opiate dependency needed to be able to have access to methadone or buprenorphine maintenance treatments. If the facility had treated one patient, they would pass on that survey. 

We wanted to create something that gave us a little more information about the quantity and quality of implementation that was happening at those sites. We also wanted to detect and decrease variability between facilities and VISNs. We really focused on having the system help us identify positive and negative outliers. Facilities that were doing either far worse or far better than the norm across all of the VA. Then we wanted to make sure that we could track implementation of these over time, so that facilities as they started making attempts to approve or implement, they could tell whether or not those actions were having the desired impact. 

To start, we were told we could actually get some information from the field. We wanted to know if you – how many people on this call have actually ever used the MHIS in the past.

Okay, I have to do something here. I must close the poll to enable it to do sharing. Here we go. 

Moderator:Your responses are coming in. We will give it just a couple of more seconds here. Okay, there you go.

Jodie Trafton:Okay, that looks good.

[Crosstalk] 

Jeanne Schaefer:It looks like about nearly three quarters of you have not had not any. But there is, about a quarter have. That is – 

Jodie Trafton:I am impressed.

Jeanne Schaefer:Good.

Jodie Trafton:Can we get our slides back?  Okay, can everyone see the slides again?  No, not yet. Here we go. Okay, so in order to develop the dashboard; because we have a guiding document that – which we were really trying to match specifically. We put together a process to try to make sure that we were accurately and comprehensively working across this entire document to assess implementation. The first thing we did to start was we took the handbook and we tried to extract all of the unique requirements that the handbook listed so that we could try to come up with measures for each of those requirements. We then, once we had those requirements, we tried to break those requirements down into specific concepts that we would need to operationalize in order to make an actual metric out of them.  

I will give you an example of sort of what that looks like. But the idea being if I wanted to look at patients with opiate dependence as I described from that last example. I have to be able to define what a patient with opiate dependence is in a way that I can find that information in the data. 

We then took that each requirement. To the extent that there was data available to do so, we created metrics that matched those handbook requirements in terms of language and structure so that the requirements – so that what we were measuring matched the wording, the specific wording of the policies as much as possible. Just to give you an example. This is a piece of text taken directly from the – from the handbook. It goes through the fact that all facilities have to make medically supervised withdrawal management available as needed based on the systematic assessment of symptoms and risks of serious adverse consequences related to the withdrawal process from alcohol, sedatives, hypnotics or opiates. Although withdrawal management can often be accomplished on an ambulatory basis, facilities must make inpatient withdrawal management available for those whose require it. Services can be provided at the facilities by referral to another VA facility or by sharing arrangement, and contract, or non VA fee-basis arrangement to be accepted if the Veterans are eligible with a community based facility. 

Taking that piece of policy; so, what we did. [aside comment] Okay. Looking at that example, the types of context that we looked at.  

First, that one contact that is in there is that both inpatient and outpatient services are required. You can do most of this by outpatient, but you at least have to be able to deliver this with inpatient services as well. That requirement told us we are going to have to look in files and data sources that provide information on services delivered both inpatient and outpatient. How we are going to have to be able to find withdrawal services?  Whether they – or regardless of which setting they are delivered in. 

Another concept from this is that this is – these services are to be available for patients with alcohol, sedative, or opiate withdrawal. That told us which diagnosis to look for. What source of diagnostic codes we should be looking for to define the patient population. It also told us which sorts of medications, for example, that we should look for as signs of withdrawal management. Right, so we could look for specific treatment for opiate withdrawal in terms – in the patient record. 

It also mentions that the facility is responsible for ensuring that the patient receives withdrawal management. But exclusively says that they do not have to actually do it themselves. This was a common theme throughout most of the handbook language. To respond to that, we developed what we call our home facility methodology. The idea here was we assigned patients to whatever facility they received the majority of their care at. 

Assuming that they would go to that facility preferentially, we said, okay, well, if they went to that facility, that facility was able to arrange for them to get care elsewhere, then the facility that they – that they normally go to should get credit for delivery of that service. We gave credit for all services that the patient received to the facility that they received the majority of their care – so, their favorite facility. That way, sometimes we get questions about our metrics; like a facility can have a substantial amount of inpatient services delivered even if they do not have inpatient services at their facility themselves. 

That is on purpose. We know that they do not have an inpatient facility. But, the handbook does not want everyone to create an inpatient facility. They want everybody to have access to one. That was the sorts of things that we were trying to measure. This gives you an example of how we designed each individual measure. We then used those sorts of concepts. Those sorts of – that sort of logic to develop initial metrics specifications for each of the requirements that we can – that we pulled from the handbook. 

I will say we have currently close to 200 different measures on the mental health information system. So as we said, it is a pretty dense document. We tried to cover it as much, as comprehensively as possible. There is a lot of information on the mental health information system. We then wanted to make sure that we did a reasonable job at actually creating these measures, so all of the specific specifications in terms of what data elements and the definitions for the data elements to be used and the actual construction of the metrics were reviewed by a larger group of clinical experts in the specific area covered by that measure. As well as the policy leaders on the patient care services side – so, in mental health services. 

They provided feedback. We modified the measures as needed to make sure that they fit the intent of the measure as much as possible. Other things that sort of had to be addressed in the development. We wanted to make sure that we standardized concepts across different measure developers because there were numerous people doing that. 

We wanted to make sure we find patient the facilities consistently. Use similar time frames across all metrics. Then made – we also wanted to help people filter because there were 200 measures. We wanted it, sites to be able to quickly find the places where they were having problems. We created thresholds that would highlight items of particular concern. 

Those thresholds for highlighting, we based on two different types of logic. Some of the measures, they were very specific policy based goals. For example, there was a requirement that facilities implement at least 95 percent of all of the elements to some extent. In that case, because it was already set by policy, we just used the policy based threshold. If there was not a policy based program goals, then we basically tried to highlight outliers. 

We tried to find sites that were doing substantially worse or significantly worse than the average site. We came up with very --typically very low thresholds, which would flag only the lowest performing facilities across the system so that they – if you got highlighted based on one of those measures. This was probably something you should be paying attention to. Again, with 200 measures, it could become fairly overwhelming pretty quickly if we – if we were not so – without those source of limited threat and thresholds. 

The distribution based thresholds, again, we reviewed with our policy leads and content experts to makes sure that they agreed that those thresholds were reasonable. These are all of the different domains that are in the dashboard. They are taken. They are based on the handbook structure for them as part. You can see they cover a wide range of types of services. Measures about specific types of care like inpatient, residential, emergency programs, emergency services, specialty PTSD and SUD programs. 

There are also measures around special populations like services for older adults or women. Other special types of service delivery like evidence based psychotherapy. It covers a lot of – a lot of different areas; again, in alignment with the handbook. Within each of those domains we created a variable number of specific measures to match each of those individual requirements. This is just an example of – in the substance abuse disorder domain there are 14 measures. 

Some of them are basic like what percent of the facility population was actually diagnosed with a substance abuse disorder?  It has to help give them a sense of whether they are doing a reasonable job with K findings. Then other measures like what proportion of diagnosed patients received treatment?  If you get into treatment, what is the length of the average length that a patient stays in treatment?  We mentioned the withdrawal metrics. We have measures of both; whether or not the patient got inpatient. What proportion of diagnosed patients got medically managed withdrawal including inpatient and outpatient? If they did, what was the likelihood that they were followed up in outpatient care afterwards?  We had pharmacotherapy measures, and so on, and so forth.  

But as you can tell, kind of by looking through these, they really try to cover a broad range of both access to services. To what extent is this facility able to treat a broad proportion of the patients to also sort of process measures?  If you get somebody in treatment, are you able to keep them in treatment?  Hand – and do a reasonable care transitions and things like that. 

This is just the link, direct link to the MHIS. I am going to show you how to get there really quick. This is mostly for your records. But if you go to the main page, on VSSC. That is VSSC dot med dot VA dot gov. There is a little box that says clinical care. Under mental health, if you click under mental health, you will get a big list of dashboards. 

Our dashboard is this one labeled Office of Mental Health Operations, and Mental Health Evaluation, and Center of Information Systems. If you open that up, this is what it looks like. The dashboard itself was designed to allow you to pick the fiscal years and the quarters that you wanted to look across. You can decide the time range that you want to see data. You can pick the location. 

We have national, VISN, and all the 141 facilities as options. This is just showing you those filters. You can pick whatever you want. Once you have made your selections, you click on view report. You will first get a large domain level view of the boards. What we did was because this was 200 measures. It was a little overwhelming, we collapsed all of the measures into those domains. Gave people a summary score, which was the percentage of items in that domain that were above the thresholds that was set.  

If you then click on the plus box on the side of each of those domains, it will drill down to give you all of the individual measures. The program goals for those measures so the thresholds that we set for highlighting. Then it will give you whatever locations you have chosen and your data for the time period that you have selected. We tend to recommend to people that they look both at their specific facility that they are interested in. 

Also, the national data for a comparator of – because we know it is because this program goals are typically defined low, low outliers; comparing yourself to the program goal is – does not tell you much about your actual performance. We highly recommend comparisons to either the national level or to your VISN. All employees have access to the MHIS. This is not a restricted site because it is program level data. It is just showing we get our data from a wide variety of places. But it is all VA administrative data. Some of it now comes from CDW as well. 

Jeanne Schaefer:Jodie, do you want me to take over for a little bit?

Jodie Trafton:That is what I was thinking. Jeanne has been hugely involved in taking this data system and actually converting it into a poly improvement program. She is going to talk now about how that data system was used to guide our site visit program. Just I do not know if we mentioned it in detail. But we were tasked with site visiting 141 facilities in nine months. This was a bit of a monumental task. But we have successfully completed it. We are very proud of ourselves. Okay.

Jeanne Schaefer:These are comprehensive mental health site visits. The aim of the site visit was to do a baseline assessment of the facilities implementation of the Uniform Mental Health Services handbook. As Jodie said, there are site visits conducted in fiscal year ’12 to all 140 VA health systems. The aim was to identify areas where the facilities needed to grow in terms of implementing the handbook. 

Areas where facilities had really developed some best practices and were doing quite well. You should know that moving forward now, after we have done the 140 in just facilities will now be on a three year cycle. This year we will be doing 45. The next year we will do another 45. Continue to follow up with the facilities to kind of monitor how they are doing. Get more data on their implementations. The site visits occurred over a course of two days. 

During that time a team of four site visitors met with healthcare system and mental health leadership. The front line mental health staff, the people in the trenches, Veterans and their families; as well as community stakeholders, the partners that the facilities had out there. Okay, and so when the site visitors get at the site visit, they had an excel workbook that we developed here as part of our effort to support this process. The workbook was split into two sections. There was a pre-work section and a site visit meeting section. 

The pre-work section included data on the mental health services and staffing at the facility. These data came from the MHIS. They were filled out by evaluation center staff and the TAs. For each of the main content areas that would be covered during the site visit, for example, coverage for PTSD care. We would look at the mismeasures related to PTSD. Then enter that data into the workbook so it would be available for the site visitors. 

The idea behind this also was to identify strengths and concerns based on that MHIS data as well as other reports. I will be going over that in a minute. The site visit meeting section, then contained questions to be asked during the site visits that were derived from concerns or strengths raised by the MHIS data, but also a series of other questions that we just wanted to get information on.

During the meeting, the site visits asked about the mental health services domains that were the main domains within the handbook. For example, inpatient residential treatment, and general ambulatory care; services for specific populations. Whether it was PTSD, SUDs, MSP; and as I said, the concerns and strength for identified areas. Then were specific questions that were incorporated into each meeting. Just to emphasize the importance of having this document. The – because we were trying to meet with so many different stakeholders over such a short period of time. Cover just a huge number of domains, these site visits were a bit of a whirlwind. In order to be able to really get detailed information about the areas of concern and the barriers, we had to be able to get to the point pretty quickly – 

Jodie Trafton:Right. 

Jeanne Schaefer:– With facilities while still living – leaving them room to add in. This basic data, having this data going in really helps guide the questioning and guide the discussions so that we got really good focused information. The template also was crucial in terms of helping to record that information in a very – 

Jodie Trafton:– Systematic way.

Jeanne Schaefer:Systematic way because without that it – we did not have this for the first two site visits. The data came back and on napkins and wrapping paper. All sorts of other things.

Jeanne Schaefer:Yes. It just allowed us to capture the data in a consistent way. To be able then to go back and in terms of writing report also to have a consistent set of data on each of these facilities. Again, this is just kind of the pre-work process. The focus was on identity – using the MHIS to identify concerns and strengths. We flagged areas, identified those so that the site visitors would know where the facility really was not doing very well at all. We also identified non-flagged areas where they were perhaps fairly weak. As well as strong areas and trends over time. We will show you; now, this is just sort of a screenshot of what the workbook looked like. 

We don’t expect you to be able to read this. But on the far left, you will see us doing the gray buttons. Those the site visitors could click on and it would take them to a specific section. If it was on population coverage. You are not going to be able to read that. But I am going to show in the next slide just a close up. This is what the pre-worked looked like. This is a section where you will see in the box that we are focusing in on this site, metrics and the MHIS. If you look at that box. You go to the right, you will see whether or not the measure was flagged. There is a column for whether it was flagged or not. 

Then the column to the right of that is whether, even though it might not have been flagged, if the comparison to the national average. Seeing how well the facility does. Whether there some more better, markedly worse, or average. Then to the far right is the trend column. Where we looked over the past four to eight quarters. We indicate whether the facilities’ scores have been stable. Whether they are getting better over time, or whether they are getting worse over time. 

If you go off to the top, you will see how the program evaluation centers would enter, indicate strengths. In this case, the percent of opioid agonist treatment is markedly better at this facility. It has got 40, nearly 47 percent and the national average is only 27 percent. There is an indicator that it is trending stable over time. On the other hand there is some concern with the facility in terms of its delivery of sub specialty treatment. It is doing markedly worse. It has scored only 14 percent with the national average being nearly 30 percent. The cutoff; I mean, they are not flagged because the cutoff was 12.5 percent. But they are not doing well. 

This is the screenshot from the pre-work. You can see in the box on the top right-hand corner. There are two gray buttons. This is data is used in the mental health staff meetings. It is used in the leadership meeting. If the site visitors click on those buttons it would take them. This data would be prepopulated into that meeting. Then they could ask the questions. 

Okay. There were 22 site visits, meetings during this two day period. As I said, it covered mental health services, domains, all of them. It has pre-worked strengths and concerns that they were covered for each domain. Then specific questions for each domain. Again, this is one of these shots where you are not going to be able to really read it. But this is what the site visitors would see. This is the question section that they would be asking during the meeting. There is the general introduction that strengths and concerns were then outlined with the pre-work that is pre-populated from the other section. Then for those sites that are now being followed for a second time. There as, the site visit report is included along with an action plan that the facility had developed. We can follow and ask about how they are doing post site visit. 

Okay, so this is a close up of again what you would see. Areas of strengths and concerns. You will see that we have a column that asks for whether the discussion agrees with the pre-work strength or the pre-work concerns. That is important. Because sometimes there is a discrepancy between the data in the MHIS and what the facility is telling us. That can be attributed to a number of things. Perhaps they are not coding correctly. Their workload is not being captured. They are going to be looking worse in the MHIS than they actually are, or that they think they are. 

One of the things that we do here is we explore with them. We ask them to go back and maybe look at their coding. If they – with regard to the concerns, we ask them whether they have got a plan that they have been implemented to improve their practice and what that plan is. From the prior site visit report down there at the bottom, we ask them the extent they have been addressing the issues that have been raised during the prior site visits. 

This is a screenshot of the meeting questions. Each section usually begins with questions one and two, which asks what are the three key areas of strength?  What are the key challenges of the facility it is facing?  Then you can see here, this is a section where we are asking about them to describe their site treatment services. Asking them to walk us through the referral assessment and treatment process. To the right you will see there are places. There are text boxes where the site visitors would type in their notes. There are also some questions where there are check boxes that they could quickly check. In this case, it is looking at which evidence based practices that they have available for SUD. 

This is also the section where there would be following questions, which would refer them to the MHIS concerns that have been raised by the MHIS.

Jodie Trafton:We will note just that the questions for each different types also depend on the population being interviewed at that time.

[Crosstalk] 

Jeanne Schaefer:Right. 

Jodie Trafton:There are separate questions for leadership, field staff, and patients.

Jeanne Schaefer:Right. Then at the end of two of site visit, the site visit team meets in an exit meeting with the facility leadership. They have prepared an exit summary. What the site visitors do is they identify five key overall areas of concern. The five key overall areas of strength. Then those are listed in this template that we have created. This information is presented to the facility and submitted to the Office of Mental Health Operations. They send this exit summary, the facility for them to begin to start thinking about a plan to address the main issues that have been identified.

Jodie Trafton:One thing to emphasize is this exit summary is actually delivered within an hour or so of having completed all of the interviews. It is generated incredibly quickly based on the data that was collected. Part of why that is really important, I think is because it gives the site visit team while they are there, the opportunity to bring some of this to the attention of the facility leadership so that there is direct communication started sort of face to face at the end of the interview. The fact that we can pull this together that quickly, I think adds strength, or adds power to the site visits in terms of getting things done. 

Jeanne Schaefer:Then there is the final report page. At the end of the site visit, the site visit team. There is a technical assistance associated with the team. They write a final report based on all the data that has been captured in the excel workbook. It reviews each of the main domains that were covered. It reviews the strengths and concerns. It incorporates data from the interviews as well as data from the MHIS and other reports, or that were included. 

The final report is then reviewed in the Office of Mental Health Operations and shared with the facility. Then the facility is asked to develop an action plan to address to report recommendations. The strategic action planning process that then is triggered by this. It is – I would say it is designed to guide the facilities quality improvement process and in response to the recommendations. There is action planning process involved; the VISN now, health leadership, facility leadership, facility mental health leadership in general. They work to create a plan to address the recommendations with input from the Office of Mental Health Operations. 

The Office of Mental Health Operations will offer them suggestions for ways, action steps that might be taken. Milestones that might be used to make sure that project is on track in terms of meeting whatever deliverables that are expected. The facilities are – submit quarterly reports. This is their chance to indicate progress on the plan. Then the Office of Mental Health Operations looks at progress. If the facilities are stuck and they are not making progress. Or they are needing help, then we offer additional technical support to help them make progress in implementing.

Jodie Trafton:One of places I can just say that I think our office can help the most with this is some sites struggle. They know that they are having a problem. But they do not know what to do about it. Because we have information across so many sites, we can make suggestions that seem to have worked at other places. Give them specific information from research studies, and so on, and so forth that might help them. We can also help them figure out how to make a concrete action plan. Because sometimes they will know roughly what they need to do. They just do not know how to get there. 

Jeanne Schaefer:Right. Or, we can offer them. We can say to them, you need to make an improvement on getting access to care. Then we can suggest which of the MHIS metrics might be something that could measure their progress in doing better at that particular thing.

This is just quickly the strategic action plan has action steps, milestones, deliverables, targets, and measures. Then there is as, I said, the MHIS data can be used as a way to measure progress in meeting those milestones. Those will be tracked in the quarterly reports.

Jodie Trafton:Lastly, the other – the one last thing that we used the MHIS for in conjunction with these site visits is our strong practices program. The idea here is we, during those site visits identified punitive areas of strength that are brought out in the site visits. Things that are potentially innovative or just practices that work incredibly well for a facility in terms of improving care or care delivery. We try to explore those during the site visits. Then get back in touch with facilities afterwards to find out more about what those private practices actually are. 

We evaluate those programs based on their impact on care delivery using the MHIS data and other data sources. We can see whether they actually worked, or whether they just sound cool when described. Then those practices that are found to have good data and these seems to be excellent are shared on a strong practice website. Facilities can go in and look up to find out what things other groups are doing in areas that they might be struggling with more. 

This is just a summary. We wanted to emphasize the fact that we use...  This Mental Health Information System is not just a dashboard. It is not just numbers that we post. We know there are a  lot of numbers that get posted. Probably, most of the – power of the numbers depends on how they are used. We have taken a lot of effort into trying to get this data actually used in a cyclical process so that it is driving quality improvement. We use the data to identify facilities, strengths, and concerns. Those strengths and concerns inform our site visit interviews and questions; also, our technical assistance and action planning process. 

If you – we then use the MHIS to assess progress in meeting of goals that are – or goals that are recommended – identified in the site visits and in other processes. That data provides feedback to the facilities about whether their quality improvement efforts are actually effective so that they can revise action plans provides their efforts as needed in order to make real change. If you have questions, comments, suggestions, we are happy to take them now. We also just wanted to, in terms of this actual development of the dashboard, myself or Alex Harris are good for people to contact  at PERC. Rani Hoff and Greg Greenberg at  NEPEC, our leads there. Fred Blow or John F. McCarthy  at SMITREC can provide lots of information that is there. If you have questions about the site visit and action plan reporting tools. The developments and how those relate into the site visit, Jeanne Schaefer and Sara Tavakoli who are the leads in designing those systems. We would be happy to take questions now.

Moderator:Okay, thank you very much. That was excellent. What a phenomenal job you all did. There are a couple of questions at the moment. I am sure more will come in. The first one – can you describe the sequence of grants you were awarded to perform the initial metric analysis and implementation of the dashboard?  How is the project currently supported?

Jodie Trafton:We are a VACO program office, we are not research supported. We are part of – we are a part of Central Office. We get funding from Central Office for staff. This whole system is continually funded in that method.

Moderator:Okay. What steps do you implement to validate the accuracy of your performance metrics?

Jodie Trafton:We have actually partnered with research to help validate some of these measures. We also do some internal validation ourselves. A lot of those concepts we designed, we have gone back and done chart review based validation to make sure that we are actually picking up what we think we are picking up. That is something we do internally. We also get a lot of feedback from the field when the measures do not pick up what they are supposed to. We do a lot of field validation of our numbers. Try to make changes or adjust our coding definitions as needed. Lastly, the substance abuse disorder query and the mental health query work with us very closely on a lot of this stuff. 

They are, for example, funded – there is a funded HSR&D project right now that is doing validation of the substance abuse disorder metrics that we – that we posted here. That is comparing performance on those measures to outcomes in various data from various trials and other sources that were available. There are also some other – some other QUERI projects and HSR&D projects that are helping us with some data validation to make sure that performance on these measures actually matters for Veterans access, quality of care, and patient outcome.

Moderator:Great. Okay. Our next question – if a facility is initiating a new mental health quality improvement project, what is the best way to interface with your organization?

Jodie Trafton:If they are – if they are starting a quality improvement project, and they would like our help. Or help with evaluating or help with design, they can either contact basically anybody on this program evaluation list. Or their VISN mental health liaison will be in very close contact with our technical assistance program. Also, leader of mental health leadership, if at their site. You can make requests through basically any of these people, or your locally assigned technical assistant who will help make sure you get what the...  Get the content expertise you need out of the office.

Moderator:Okay. Do you see the MHIS measures of being a stable source that could be used for research purposes?  Specifically in the area of evaluating different implementation strategies of evidenced-based care in mental health?

Jodie Trafton:Yes. In fact, I know we have talked with QUERI. They are very interested in, for example, borrowing all of our code. We are very open to that. We are not...  Because we are stably funded and working to try to get things changed. We are happy to share whatever we have. I think these measures, they have. They have been validated to a greater or lesser extent. We have a good amount of historical data. Their plans are to keep calculating them. Going on into the future. They can be used for research. That being said, you cannot just pull the numbers off of the system, like all of the VSSC data. You need to have a data use agreement with our office before you can use the actual numbers generated in a research project. Again, if you contact anybody on this list, we can help you get a data use agreement to use our numbers or our code in your research.

Moderator:Okay. What were the key steps or contact person with your group to elicit support from Central Office?

Jodie Trafton:We are Central Office. You mean, how do you get support from us?  Or, have...?

Moderator:That is probably  a research sort of question. How do researchers work with Central Office?  But, as you say you are Central Office.

Jodie Trafton:Yes. We will not have any trouble getting help from Central Office. There is a lot of... but, if you would like to partner with Central Office on a research project, the things that I would suggest is as you are starting to develop that research project. Not when you have your grant like 99 percent done and are within two days of trying to submit it. Please get in touch with somebody in the – in the program office. Again, we have daily calls. If you hit the wrong person, it does not matter. It will get to the right person. You just need to get in touch with somebody in the office. You can go through the office lead, if you want. Our office is Mary Shone. You could go through any of these evaluation center groups. We will be happy to work with you to both make sure that your project is aligned with the current initiatives. That you are aware of that we have a lot of people who contact us for help finding pilot sites or support. For help designing data systems; or evaluation pieces. Or to find ways of actually tag teaming a research project along with current initiatives that are rolling out. Sometimes we can provide substantial – not money, typically but support in terms of resources, and expertise, and needs to help facilitate those projects. But it is...  We can be a lot more helpful if you get in touch with us early rather than late. We get a lot of requests from people, well, they do not want to bother us. They wait until two days before. They send us a letter with a proposal that may or may not actually align with the way that the system works. Those are hard for us to deal with because we do not have enough time to help people get them right; or to support them appropriately. Contact anyone in the office as early as you can. We are happy to help. 

Moderator:Great, next question. Have you also drawn on the 1163 VHA handbooks in the development revisions of the measures?

Jodie Trafton:Which handbook?

Moderator:It is – it is called...  It is a numerical handbook, 1163 VHA handbooks.

Jodie Trafton:Okay. I do not know what specific handbook that one is. But we... Our – this whole system has been based on the Uniform Mental Health Services Handbook, which is a specific policy document. That document is being currently revised this year. We are involved in that revision process. We will be revising our measures as needed to keep them aligned with the policy. We also use other mental health policy documents to – which sometimes we will add measures that go with other specific policies or with other specific initiatives. But – so, it is definitely a policy driven system. But, I am not sure what exactly that handbook is off the top of my head. 

Moderator:Okay. I thought maybe the person would type it in, but I do not see it. Here is another question. Was it difficult serving in somewhat of a go between role between policy, policymakers, and the field in terms of wonderful policies that may be very difficult – ?

[Crosstalk]

Moderator:– Scholarly of the field?

Jodie Trafton:Yeah, I mean, I think – I think most of the people in our office really enjoy that role in that we do get to be the people. We get to be the helpers rather than the enforcers, so to speak. There is definitely a dance that you have to play between. But our goal is never to like shove things down the throats of the field. 

Jeanne Schaefer:Yes.

Jodie Trafton:It is – we are here to really...  We can get policy to revise policy, if they have suggested things that do not work. We can collect feedback from the field. We can deliver it back up to leadership. We can facilitate communication, I think, in ways that are really helpful. We are – I mean, we are really happy. We have very close communication with the VISN mental health leads, for example. They provide us all sorts of like incredibly valuable information about how things are going. Where people are struggling, which help guide both revision of policy, new initiatives, changes in resource allocation; and, as we get special funding we use that in ways based on what we have learned from the field and things like that. 

I think it is a very useful place to be in. I know that we have based this all; because I know that there are a lot of researchers on this call. We have worked with mental health query. This is our technical assistance, it is all based on an external facilitation model to try to be aligned with what works for – in terms of you actually getting programs implemented from an outside perspective. 

Jeanne Schaefer:I would just like to add that there was a really strong emphasis with regard to the site visits. That this would be a collaborative process. That we would not come in there and just kind of okay, this is a report card; a, b, c, d, f in terms of your ability to implement this policy in the handbook. It was this – the notion that we are here to provide technical support. We are here to provide the resources for the facilities. That we want to understand what the issues are for you in terms of how are you struggling to implement this?  If you are struggling, what can we do to help you?  

Jodie Trafton:Yes. We really love the role. Because we know people at the facilities are working really hard. Anything we are able to do to make it easier to deliver better care, it feels good for us. It is really good for the system. 

Moderator:Great. Probably one last question. Well, it is not really a question. It is the information about the handbooks, 1163 series. Maybe it is a single handbook – addresses specific mental health programs such as psychosocial rehabilitation and recovery services. Addresses peer support, therapeutic supported employment; and psychosocial family services. 

Jodie Trafton:Yes. Those reports also guided development of the MHIS. I note that the – what the specific ones you mentioned also have their own specific evaluations that are driven – that are conducted by the Northeast Program Evaluation Center. In addition to this sort of broad national program, we also have specific evaluation programs that work with different types of specialty programs individually to help make sure that the specific policies are implemented. That they are getting the support that they need. Yes, those policies were considered in the development of this. But they are much more strongly represented in some of those – some of our smaller programs, specific evaluation programs. 

Moderator:Great. Well, it is the top of the hour. The questions have all been answered. I want to thank you both for a really excellent presentation. I want to let our audience know about our next session scheduled for Tuesday, April 16th. Dr. David Ganz, who is both an M.D., and a Ph.D. will be presenting on the redesign of an electronic clinical reminder to prevent falls in older adults. Thank you all. Please answer the survey when you sign out of GoToWebinar. Thank you, all, good-bye.

Jodie Trafton:Thanks.

[END OF RECORDING] 

Page 1 of 15

Page 15 of 15