**Transcript of Cyberseminar**

**Patient Aligned Care Teams (PACT) Demonstration Labs**

**PACT: part 1. Assessing Reactions to PACT Implementation: The Team and Individual Role Perceptions Survey (TIRPS), part 2. Burnout and Working to Top of Competency in VA Primary Care**

**Presenters: Christian D. Helfrich, PhD; Greg L Stewart, PhD; Stacy Lolkus, BS**

**April 16, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact Christian.Helfrich@va.gov or John.Messina@va.gov.*

Moderator:I would like to introduce our speakers for today. Speaking first we have Dr. Christian Helfrich. He is Chair of the Organizational Function Working Group at the PACT Demonstration Lab Coordinating Center in the VA Puget Sound Healthcare System.

For part two of the presentation we have Dr. Greg Stewart from the VISN 23 Patient Aligned Care Team Demonstration Lab in Iowa City VA Medical Center and joining him today is Stacy Lolkus who is also from the VISN 23 PACT Demo Lab also located in Iowa City VA Medical Center. 

We want to thank them for providing their expertise today and Christian, I am on your first slide.

*Dr. Helfrich:*I think we had a couple of poll questions we wanted to start off with to find out a little more about our audience. I see, I think, that first one on basic role in the VA so folks can fill that in.

*Moderator:*Thank you. The question is...What is your primary role in VA? Are you a student, trainee or fellow, a clinician, a researcher, manager or policy maker or other? We do understand that a lot of your wear many different hats in your roles so please, what is your primary role, and at the end during the feedback survey you will have the option for a more detailed list if you would like to select your role off that as well. 

It looks like the answers have stopped streaming in so I am going to go ahead and share the results. It looks like we have roughly 6% that identify as student, trainee or fellow, about 17% clinician, 28% researchers, roughly 19% manager or policy makers and 26% say other. Thank you for those responses. 

We do have one more poll that we are going to do before we get to the presentation and that is...How many of you have a role with PACT? Are you a member of the PACT teamlet, member of broader PACT team working with multiple teamlets, providing other support to PACT for example training, PACT primary care administrator or studying PACT? 

It looks like we have got a nice cross section of attendees with us. About 16% are a member of a PACT teamlet, 1 person identifies as a member of the broader PACT team, 24% provide other support to PACT, about 19% are PACT or primary care administrators and about 40% are studying PACT. Thank you once again to all of our respondents. It is helpful to know who is in our group. With that we will get back to the slides.

*Dr. Helfrich:* Great. If you could go to the first of the background slides of Patient Aligned Care Team initiative background that would be great. 

Again, thanks everyone for the opportunity to talk today. My name is Christian and I work with the PACT Demonstration Lab Coordinating Center which I will talk about in a moment. I want to talk with you today about some findings we have on changes in primary care employees working to top of competency and changes in burnout from 2012 to 2013...a couple of what we feel are sort of important assessments that we are tracking for the PACT initiative.

For those that don’t know, just very briefly, the Patient Aligned Care Team or PACT initiative is VHA’s patient-centered medical home model. It was formally launched in April 2010 and it comprises multiple components or changes. Perhaps the biggest one is an emphasis on team based care and having a Primary Care Provider and an MD, Nurse Practitioner, Physician Assistant paired with a team that includes a Nurse Care Manager, a Clinical Associate and that is an LPN, Medical Technician, LVN and a Clerical Assistant or Administrative Clerk. These four individuals together work as a team and share responsibility for caring for a defined panel of patients. The PACT initiative also involved a number of other changes including innovations in scheduling and alternatives to in person visits. It includes use of secure messaging and telephone visits and a major emphasis of the model is more active support management of patients, particularly with the use of Nurse Care Managers and the additional health promotion support that is available in the clinic via Social Workers, Primary Care based Mental Health Providers and others such as Clinical Pharmacy Specialists. There was the idea of providing more systematic support of patient behavior and self-management.

If you could go ahead and advance the slide. The PACT initiative involved a number of resources to provide support for PACT. This included funding to the networks and hospitals to support the expanded staffing model to insure that the teamlets are made up of the requisite individuals and also training, most notably regional learning collaboratives that teams for every medical center travel to and these learning collaboratives were supported and sponsored by VA Systems redesigning the traveling teams and then went back to their clinics and shared what they learned. There was also the formation of five PACT Demonstration Laboratories. These are five centers located around the country that evaluate and study different aspects of the PACT model and are able to do so in more detail and Greg and Stacy are from one of those Demonstration Laboratories are going to be talking about some of their demo lab work. Then there was a Demonstration Lab Coordinating Center and that is who I work with. It is based in Seattle. We have got members in other centers and are collaborating with members of all of the Demonstration Labs. In addition to serving a coordinating function across the Demonstrating Labs, the Demo Lab Coordinating Center was also tasked with doing an overall evaluation of the extent of implementation of PACT and the effects of PACT. My particular role in this has been assessing the implementation of PACT in terms of the team structure and organization and also the effects of PACT on the primary care employees. 

Go ahead and advance the slide if you can. A couple of the key goals that relate to the effect of PACT on employees was to increase the time that primary care employees spend working, as we say, to the top of competency and to reduce their burnout. The idea is that with forming these teamlets and pairing Primary Care Providers with the same Nurse Care Manager, the same Clinical Associate, the same Administrate Clerk, that they could develop efficiencies in delegating activities and coordinating tasks within the team so that each one of them is doing work that really they are best suited to doing and reduce the amount of time they spend doing things they are really not best suited to do or that somebody else could do and that by doing this they would increase the proportion of time that they spend working to the top of their competency and reduce the amount of time that they feel that they are doing work that someone with less training could do. One of the hopes was that this would help reduce burnout. 

Perhaps one of the most encouraging pilot program results in the patient centered medical home literature came out of group health cooperative which is the centers based here in Seattle. They did a pilot program on a medical home model at one of their clinics that had very high levels of burnout. They implemented a medical home model there and over the course of a year saw their burnout rate among providers drop from 30% to 10% and this was while the burnout rate was remaining roughly as high as the baseline burnout rate of the clinic it remained that high at the other group health clinics. It was seen as a really pretty spectacular success and there was some hope that we might see a similar effect in the VA. My talk today is to present some findings about changes that we have seen in working at top of competency and especially burnout will be the major focus of my talk.

Advance the slide if you can. The findings that I am going to share in just a moment are part of a series of analyses that we have been doing for the past several years. So far, findings on the effects of PACT on Primary Care Employees are mixed. We did an analysis last year, a cross sectional survey analysis, looking at specific elements of PACT team based care and whether they were associated with lower odds of burnout and higher job satisfaction. We looked at specific elements of PACT team based care such as whether or not respondents reported that they were assigned to a teamlet that was staffed to the recommended ratio so three full time equivalents were matched with every full time provider, whether or not there was a delegation of clinical responsibilities within the team, whether or not there was participatory decision making within the team, the quality of communication with the team and a number of other elements both structural and process elements. We did find several of the elements of PACT team based care particularly participatory decision making but also delegation of clinical responsibilities was associated with significantly lower odds of burnout and higher job satisfaction. We also found that clinics that overall have a higher level of PACT implementation, that is when we make a global assessment of the extent to which these clinics seem to have implemented PACT and look at the clinics that are in the highest category of PACT implementation versus those in the lowest category we see those in the highest category have significantly lower average burnout rates. 

At the same time, we have done analyses of existing survey data. We will be talking about one of these survey data sources- the All Employee Survey. We looked at annual survey data of employees where we can define a cohort of Primary Care respondents and looked at trends in job satisfaction, workplace climate and self-reported intent to quit. We see a steady trend in declining job satisfaction and increasing intent to quit among Primary Care Employees from 2009-2012. That was what my colleague David Moore in Boston did. We also see a significant increase in Primary Care Provider turnover since the launch of PACT. Looking at turnover rates for Primary Care Providers before PACT and after, we see an increase in the rate of turnover since implementation of PACT and that turnover rate is greater for older and more experienced PCP’s. That is work that my colleagues Philip Sylling  and Chuan Lu and Paul Hebert have done. Overall, we have mixed findings so far on the effects of PACT on our Primary Care workforce. 

Advance the slide please. What I wanted to do is share some analyses that we are currently working on, looking at changes in working to top of competency and burnout measures that were collected in three surveys and actually represent four different survey cohorts. The three surveys were conducted in 2012 and 2013. The first is the PACT Primary Care Personnel Survey. This was fielded in 2012. It was a comprehensive survey that our Demonstration Lab put together and fielded with the help of the VA Health Analysis and Information Group. The sampling frame was all employees that worked in Primary Care and the survey methodology was to send a survey link, a web based survey link to Primary Care leadership in central office who disseminated it to the networks and Primary Care clinical leadership in the field and they were asked to then distribute the survey link to all of their employees who worked in Primary Care. 

We also have data from the 2012 All Employee Survey. Again, this is a survey that is conducted annually in the VA. The sampling frame is all employees of the VA including WOC’s or without competency appointments. The All Employee Survey starting in 2008 had a self-report type of service question that included Primary Care as an option. It asks...What is the main type of service you provide? Again, one of the options out of approximately 19, I believe, is Primary Care.

In 2013, instead of fielding an independent PACT survey, we developed a shorter PACT module and fielded that as part of the All Employee Survey so again the All Employee Survey goes out to All Employees in the VA. The way that the All Employee Survey is field is it is largely a web based survey but there are specific All Employee Survey representatives at every facility who define specific work groups that are surveyed and determines which employees belong to that work group. The surveys are then fielded within the facilities and tracked at the facility level. 

In the 2013 survey, we have both the employees who self-report that the main type of service they provide is Primary Care and we also included then this PACT module and all employees were asked if they were a member of a PACT teamlet. We defined the teamlet for them and if they indicated that yes, they were a member of the PACT teamlet, they received a PACT module. These are the survey data that I will be presenting. All of the surveys were anonymous so we are not able to link observations across years. Essentially all of our analyses are serial cross sectional analyses and we are not linking them.

Advance the slide please. There are two measures that I am going to focus on today- one measuring top of competency and the other burnout. Top of competency we measured with self-authored item- one that we made in consultation with researches and Primary Care leaders. It asked respondents to assess the percentage of time spent on work well suited to their training. There were four response categories reflecting quartiles of time: less than 25%, 25% to 49%, 50% to 74% and greater than or equal to 75% of their time. This survey item was only available in the 2012 PACT survey so it was part of our 2012 PACT survey and in the 2013 PACT module- the All Employee Survey respondents, unless they indicated they were on PACT in 2013, they did not get that question. 

The burnout we measured with a single item measure that was used in the Physician Worklife Study. It is scored on a 5 point scale and respondents are asked to define for themselves- based on your definition of burnout how would you rate your level of burnout, and then there are five response categories that are all descriptive. The middle response category is, “I am definitely burning out and have one or more symptoms of burnout such as physical and emotional exhaustion.” That middle response category is generally used to define burnout so response of greater than or equal to three, so if they indicate that response category or the higher ones we define that as burnout. This measure has previously been validated against Maslach Burnout Inventory which is a longer measure, a 22-item measure in its full scale that is generally considered the industry standard and we have previous validated against MBI.

Advance the slide please. I am going to largely present descriptive statistics and then for the change in burnout present logistic regression. This is where we are testing for a difference in the overall burnout nationally in the VA from 2012 to 2013. The regression analysis I will present is cluster adjusted by facility also adjusting for responded demographics and some workload and staffing figures. The surveys were anonymous so these are serial cross sectional analyses; they are not adjusted for within respondent correlations. For the 2013 All Employee Primary Care cohort, so these are individuals who indicated that their main type of service provider was Primary Care, unless they also indicated they were on a PACT and some did not, we did not have the single item measure for burnout. We did have a measure of burnout from the Maslach Burnout Inventory and so we used that measure of burnout from the Maslach Burnout Inventory along with respondent characteristics to impute a value for the single item measure.

Advance two slides please. This is a table of respondent demographics across these four cohorts. You have got the 2012 All Employee Primary Care cohort so their main service type is Primary Care respondents to the 2012 All Employee survey. Next is the 2013 All Employee survey in the Primary Care cohort. These are individuals who self-report, their main service type is Primary Care. The third column is the 2012 PACT survey. This is our comprehensive PACT survey. The last column is the 2013 All Employee survey. These are respondents who indicated they were on a PACT- member of a PACT teamlet. 

I will point out that the demographics we are comparing here are demographics that are independently associated with burnout so age, tenure with the VA and supervisory level. The demographics are very similar across all of these survey groups. The most important thing to note is that for 2012 PACT Primary Care survey we only had an estimated response rate of 25% whereas for the All Employee survey cohorts the response rates exceeded 50%.

Advance the slide please. In terms of working to top of competency, we have this broken down by PACT occupation. This is the percentage of respondents who reported the highest category of time as greater than or equal to 75% of their time spent on work well matched to their training. Over half of the providers reported spending the vast majority of their time on work well suited to their training. Fewer than half of nurse care managers did. The percentage of Primary Care Providers did differ between the PACT 2012 survey where it was 51% to the respondents to the PACT module in 2013 where it was 57%. Conversely, clinical associates also changed and declined from 67% of clinical associates in the 2012 survey reporting that they spent 3⁄4 of their time on work well suited to their training to 61% who reported that in 2013. 

Advance to the next slide please. These are the unadjusted burnout rates by occupation from 2012 to 2013 among the PACT Primary Care survey respondents. That is the comprehensive PACT survey in 2012 and the PACT module respondents in 2013. The purple bar is the 2012 survey and the green bar is the 2013 PACT module. Overall, the highest burnouts for Providers 45% and 43% respectively for 2012 and 2013, next highest for Nurse Care Managers 39.9% and 33.3% and lowest for Clinical Associates 30.7% in 2012 and 28% in 2013. Overall, for those three groups burnout did decline between the PACT survey or comprehensive 2012 survey and the 2013 PACT module.

Advance to the next slide. These are the same burnout figures but for the All Employee survey respondents in 2012 and 2013 who indicated that primary care is the main type of service they provide. The burnout rates are similar at 40% and 46.7% for Providers, 36.1% for Nurse Care Managers and 25.1% to 30% for Clinical Associates. However, in this case we see an increase in burnout for Providers and Clinical Associates. Nurse Care Managers were not identified as a distinct occupation group in the 2012 All Employee Survey so we were not able to assess their burnout rates. Burnout rates also appeared to increase. Again, these are unadjusted, for Administrative Associates between 2012 and 2013.

Advance the slide please. This table is the regression output. Here we are looking at the last group. These are the respondents to the All Employee survey who identified their main service type as Primary care. We are looking at the change from 2012 to 2013 but we are adjusting for occupation. We are also adjusting for tenure and supervisory level which we know are associated with burnout and also adjusting for some measures of workload and staffing. The most critical thing is that top line that is bolded- 2013 vs 2012. This is the odds ratio for someone in 2013 relative to 2012 of being burned out. The important thing is that it is not significantly different. We do not see a significant difference or change in burnout from 2012 to 2013. We do see that Clinical Associates and Nurse Care Managers have significantly lower odds of burnout than Physicians, however, Physicians as we saw had very high rates of burnout. Like I noted before, the longer tenured employees have higher odds of burnout. That is something that we have long observed.

Advance the next slide please. Some important limitations of this work. These are serial cross sectional data precluding our ability to track individuals over time. Working to top of competency was assessed with a self-report item. It was not observed or independently verified. Differences in the survey sample and response rate should lead us to have some real caution about inferring too much from these findings. It could be that differences in the working to top competency are due to differences in sample and response rate.

Advance to the next slide please. That being said, there are some important conclusions I think we can draw. First and foremost, burnout remains high. Burnout rate is in excess of 30% for most occupation categories over most years. Burnout does not appear to have changed significantly in 2013. There we have a little more confidence and I think that is an encouraging sign. We had seen a declining trend in job satisfaction and increasing trend in intent to quit. There may be changes in working to top of competency with improvements for PCP’s but declines for nurses but there we have got much less confidence and more work needs to be done. We have got opportunities to continue to understand the effects of PACT on employees with a great deal of work happening in the Demo Labs and you will hear from Greg and Stacy in just a moment and also additional survey data becoming available and we are continuing to analyze the survey data we have and are looking forward to sharing more findings with you all in the near future.

With that, thank you very much and I will hand it over to Greg and Stacy.

Dr. Stewart:Thanks, this is Greg. We are excited to be here with you today. We will follow up a little bit on some of what Christian has talked about and build on some of the issues that he has been discussing. Again, we are one of the Demo Labs that sits under the Coordinating Center. You will notice that a lot of what we will talk about will fit together and be complimentary on what it shares about this implementation of PACT.

Let me process what we want to talk about today. We are going to talk about a specific survey called the Team and Individual Role Perception survey. We have been fortunate enough to capture this data locally for about three years now. We were able to go in the summer of 2010 just as we were beginning PACT implementation and get pretty much a baseline level of some measures and we have been able to follow that up one, two and three years after that. We are also planning to continue this into the future. What this does is allow us to longitudinally track changes and perceptions of these individuals who are working within teams.

We specifically have surveyed on two things. One is the team characteristics. How well is the group working together and getting along? These are team based measures in essence. The other one is how do individuals within each of these teams or teamlets experience their work? We will talk a little bit more about definitions of each of these and we will share some of the data around these. I will talk mostly about the team characteristics and then Stacy will share some of our research about the individual role perceptions.  You will notice from the dates there that we have been studying the individual role a little bit longer and have some more longitudinal data on that. You will see there on the slide that we have been measuring this from Providers, RN Care Managers that Clinic Associate role that Christian talked about and from the Clerical Role. Over this entire four year period we have pretty small samples here. One of the things that we have done differently than Christian where he has the large samples international we have much more of a localized sample. We are also able to match this with some of our qualitative data and we will share some of those findings. 

The other thing that we have been able to do over the three or four years that we have been tracking this data now is expand this somewhat. In 2013, it doesn’t show on this role here, but we were able to get survey responses from close to 500 people in different teams. One of the advantages that we have been able to do is take some of the measures that we have collected and feed those back to the specific teams. 

I am going to spend a few minutes talking about the team characteristics part of it and focus specifically on what we did in 2013 when we were able to gather data from approximately 500 people and we were able to give individualized or team level feedback to about 54 teams that were groupings we had that we were able to get enough survey response to get them a localized, customized report that we were able to feed back a picture of how they were doing in terms of their team development. Once I have talked about that, Stacy will share with us some of the changes that we have seen from 2010 to 2013 that will mirror and hopefully build on what Christian has done a little bit. One of the things that we find with these individual role perceptions is there have been some declines although we do see some swing and things getting a little bit better over the last year or so which probably isn’t very unexpected any time you do large scale change initiatives like the implementation of PACT has been. 

Let me go ahead and talk about this team portion of it. What you have on your screen there is just a sample report. What we were able to do in 2013 is we were able to develop these specialized reports on a much more localized level for a number of teams and again we had to group them and find ways where we got enough of a response that we could do this in a way that was confidential and yet the teams could get a picture that would let them see how well they were doing along team implementation lines. 

We were able to create four page reports that were fed back to the members of each of these teams. This is just kind of the first page that you will see here that provides them with an overall idea of what the survey is and then we have got a little thermometer there and you can see that we are able to compare their team score to a comparison group. In this case, the comparison group was VISN 23 which is where we are housed and where this survey was collected. We were able to survey all members of PACT teams within VISN 23 and then to take that down and essentially compare the different teams within VISN 23 so that each team could get a sense of how they were doing compared to other teams. 

What we will talk about here is VISN 23 and that is where our Demonstration Lab primarily works with but I might mention that we are more than open with working in other areas so if there are people who are listening that might be interested in doing some of this in the future, please contact us. We are by no means trying to be exclusive with VISN 23. It is where we are housed and we work but our goal is to broaden this and to accelerate implementation throughout the VA so if there are people listening that are interested in following up and doing some of what we are talking about please contact us. We would love to work with you and help you capture similar snapshots. That is the first page that we did and we provided these reports.

The next page that will come up would be the team characteristics page. What you will notice again is that each team was able to get feedback on five characteristics of their teams and then aggregated feedback on individual perceptions around five characteristics. Once again we have used thermometers here so they could see how they are in comparison to the rest of the VISN. I will talk about each of these five dimensions in just a minute but if you look at the top one there you can see that we measured performance monitoring which is just how well is the team tracking and monitoring its progress in terms of achieving its objectives. This particular team did a little bit below average for the VISN. What this does is identify this kind of weakness of what they might need to work on and help them get some feedback about areas they can get some development and we are continuing to help within VISN 23 these teams develop and get better in areas where they were perceived as maybe being below average.

If you look at the bottom of this page you will see psychological safety. Again, I will describe each of these in just a minute but they were better than average there and might show a relative strength. Also included in this survey was a summary page that showed each team how it did relative to the VISN and overall in terms of its team characteristics and how well they did in terms of individual role perceptions. You can see their overall level how they did on a five point scale and you can also see how they did relative to others in the VISN to get a snapshot of their strengths and weaknesses.

The other thing that we are able to do with this is we are able to provide...I am going to come back to that slide in a minute...we are able to provide a summary for the VISN. You will see here that we have got each of the five team performance majors or team characteristic majors and I will talk about those in just a second but I have got these in terms of what we call a box and whiskers plot. Let me go back a slide and help set the reading of these. You will notice that there is a box and right in the middle of the box there is a line right in the middle of the box that would be the average for the 54 teams that we did. At the very top of the line would be the values for the highest scoring team and at the very bottom would be the score for the very lowest scoring team. Around the box, at the top of the box here, above that line would be where 25% of the teams scored and below that line would be where the bottom 25% scored. Looking at these quickly you can see the average but you can see the range and see how the teams fell into broadly relative to others. 

Let’s go and I will overview with you some of the findings that we had in our 2013 survey. This, again, would be for our VISN to look at some things that have worked well with the team characteristics and maybe some areas where we continue to need to improve. The lowest overall score was in this area of performance monitoring starting from the left of this graft. This is a team dimension that gets at how well does the team track its progress and its goals. You will notice that our average is just a little bit below 3 on a 5 point scale and ranges from about 2 to 3.7 or 3.8. That gives us a little bit of a sense of how well we are doing and shows and demonstrates an area where we need to improve in terms of how well these teams are measuring their progress and their goals and tracking that.

If you look at the next one moving right, we measure team member back up. This gets at how well the team members back each other up, support each other, help when the work load becomes too large. Again, we get an average of about 3. You will notice that the box is a little tighter here suggesting that they don’t spread out as much. The bottom line is suggestive of the bottom scoring team about 2.5. In other words, there are no some that are really low like we might find in the performance monitoring. The two areas of relative strength in terms of team characteristics were first team coordination which we are finding an average of about 3.5 and this is how well did the team works together- a measure of teamwork and cooperation. You will notice that some teams are doing very well up in the 4.5 range- the top performing teams and overall we do pretty well there. 

Conflict management gets at how well the team members deal with their conflict and their differences of option. Again we will see a measure of about 3.3 or 4 spread out a little bit more than the others which would suggest there is more variation in how well it is doing. Some teams are doing better than others particularly. The last team based measure we have is psychological safety. You can think about psychological safety as kind of measure of how free people feel to share their concerns and their opinions. Do they support each other and do they feel safety in raising concerns about the safety of patients and the way things are going. Overall we get a measure of about 3.2. We see quite a bit of range here. This is another area we are interested in improving and given its importance for medical work we are consistently looking for ways to improve that.

That is an overview of the team characteristics and the portion that we measure that has to do with the team characteristics. I am going to turn it over to Stacy now and she is going to talk a little bit more about what these individual role perceptions are and how we measured those in some of our findings.

Stacy Lolkus:Thank you Greg. We looked at the 2013 Individual Role Perceptions pretty similarly to how we looked at the team characteristics. This was in a summary that we provided to the VISN 23. As Greg mentioned, the individual role perceptions portion of the survey really measures the five different perceptions held by team members about their individual roles within the team. We look specifically at role capacity, role harmony, role clarity, skill variety and personal empowerment. I will give you a brief description of the five different role perceptions before we get into the results.

Role capacity is the perception of having the time and resources necessary to complete work tasks. People generally responded positive to this when they felt like the amount of work they were being asked to do was reasonable, that their task could be completed in the necessary time frame that was given. This could almost be seen as a precursor to burnout that Christian was discussing earlier.

If we go to the second portion to role harmony this really looks at and measures how the work is coordinated within the team. It is generally rated highly when individuals have the tools and resources necessary to complete their tasks and they are able to complete the most critical tasks as needed. Respondents generally rate this a little bit lower when they work under incompatible policies and guidelines, when they receive conflicting requests from multiple people or if they have to work around these policies and guidelines in order to complete their tasks.

Next we looked at role clarity which knows what teammates and others expect of the individual. These individuals are really clear about the responsibilities, they understand what is expected of them and they generally rated role clarity as lower when they felt like they were being pulled in many different directions and had a lot of uncertainty about how their performance would be measured. 

Skill variety looks at the extent to which individuals engaged in numerous different types of tasks and work to the top of their capabilities. People who rate this highly feel like their work utilizes a number of their different skills. 

Finally, the personal empowerment is an extent to which people feel that their work is meaningful and important. Also, people who are high in personal empowerment are confident in their capabilities and feel like they have mastered the skills necessary to be successful. 

Similar to the teach characteristics, these perceptions were measured on a scale of 1-5 with 5 being high in these attributes. As you can see, skill variety and personal empowerment were generally pretty high. Team members were generally working to the top of their capabilities and felt that their work was meaningful and important. As you can see, skill variety was actually the highest among the TIRPS measures with an average of 4.2 out of 5 and personal empowerment had an average of 3.9 out of 5. 

On the left hand side, if we go back to role capacity, role harmony and role clarity part...the role capacity was generally one of the lowest measures among the TIRPS. People generally felt like they were being asked to do much with their time and resources available to them so their average there was 2.7 out of 5. This also had a relatively wide range of scores as you can see. There is some potential that some teams are better at helping others feel like they have the necessary time and resources and so there is some potential to learn from each other for that. Also, team members were not really sure what was expected of them. You can see the role harmony and the role clarity was a little bit lower here as well with an average for the role harmony of 3.1 and role clarity was 3.6. 

Like Greg said, we were fortunate enough to be able to distribute questionnaires and have them answer the surveys from 2010 to 2013 so we were able to get a look at how these role perceptions changed throughout the implementation. We looked at this based on the different roles. We have it sectioned out so Provider, you can see, is the blue line there, the RN Care Managers are the yellow lines, the Clinical Associates are the purple and then the Clerks are the green lines on these graphs.

Basically for role capacity, which is having the time and resources to get tests done, basically you can see that all roles showed an initial decrease in role capacity after the first survey but the Providers, RN Care Managers and Clerks all seemed to improve after 2011 but then the Clinical Associates seemed to dip a little bit longer. It seems like some of the Nurse Care Managers and the clerks were pushing some of their tasks more toward the Clinical Associates so that seemed to lag a little bit behind. As you can see, and as Greg mentioned, it does seem that a lot of roles tended to refer back to their baseline levels as they became more used to the implementation.

If we go to overall harmony, this is basically just to remind you is when the work is properly coordinated. Once again, Provider role harmony dropped a little bit initially but stayed relatively the same throughout the four years of the implementation. Once again, the Clinical Associates, Clerks and RN Care Manager roles dropped initially but the RN and Clerk perceptions increased in 2012 while the Clinical Associates tended to lag behind again so those negative effects were felt through 2012. Once again, they did seem to return to baseline as the implementation progressed. 

With role clarity and knowing what is expected of the individuals, the Providers did relatively stable throughout the implementation and improved slightly towards the end. The Nurse Care Managers had small initial drops but then seemed to improve but the LPN and the Clerk had the largest drops in role clarity and as you can see those lagged a little bit behind the Nurse Care Managers between 2011 and 2012 and then increased back to baseline levels.

As we mentioned before, skill variety was generally pretty high among all of the roles. It didn’t change a whole lot for many of the roles overall but it is pretty interesting. You can see that the clerks dropped in skill variety from 2010 to 2011 and then just started to improve after that. Those were all able to also go back to baseline levels.

Finally, with the personal empowerment and how meaningful and important people felt their jobs were, those stayed generally pretty stable throughout the implementation however the RN Care Managers saw a significant drop between 2010 and 2011 which continued through 2013 and never actually recovered back to their baseline levels. We thought that maybe this had to do with spending more time on the phones, maybe not having as much contact with the actual patient and that might be one of the explanations why this decreased after the PACT implementation. 

In summary, the TIRPS was used to provide team level feedback to 54 groups within the VISN, within VISN 23 included information about team characteristics and also individual perceptions and feelings about their roles in the team. In general, as Greg mentioned, team coordination was pretty high and teams need to work on improving the tracking of their progress towards their goals. Individual team members did feel a sense that they were being asked to do too many tasks in too little time but empowerment and skill variety were generally pretty high for all of the individuals. 

Thank you very much for letting us talk today. We are looking forward to your questions and your feedback.

Moderator:Excellent. Thank you to all of you. We do have a lot of great pending questions. For those of you that joined us after the top of the hour, if you want to submit a question or a comment just use that Q&A box located in the upper right hand corner of your screen. I am just going to go ahead and alternate between questions for each of the different presentations. 

The first question is for Dr. Helfrich. How did your burnout item compare to the Maslach? I know you validated against it but I am asking what the outcome was.

Dr. Helfrich:We did a number of things including just sensitivity, specificity...let me see if I can pull those up. We did sensitivity and specificity. We looked at the area under the receiver operator curve and also just looked at correlations. It is certainly not a perfect representation but the sensitivity and specificity were in the high 80’s and combined with responding demographics which tenure, age, supervisor level are independently correlated with burnout it gives us a reasonably good ability to calculate burnout. It is also worth noting that the Maslach Burnout Inventory, even though it is widely considered the industry standard and also is a self-report measure, these things vary over time. There actually is a clinical diagnosis of occupational burnout and some of the original Maslach Burnout Inventory work was done comparing it to a clinical diagnosis and that is not something we have done with a single item measure. I will see if I can pull up the specific figures. Let’s go into the next question and then I can come back to the specific figures.

Moderator:That sounds good. The second half of that question was....Also, what do you think you aren’t capturing with your single item that you could capture if you had the luxury of a few more items? Not necessarily all 22.

Dr. Helfrich:It is interesting. The Maslach Burnout Inventory includes three sub-scales. One is emotional exhaustion which is what I think we are getting at with the single item measure and that is what we typically think of as burnout. It is like...you feel like you are just emotionally exhausted, you don’t have the same motivation. The Maslach Burnout Inventory also assesses professional self-efficacy...whether or not you think that you are capable of doing your job and also cynicism...that feeling of sort of losing interest and empathy with the people that you are trying to serve. Those are things that we miss with the single item measure. We are really getting at whether or not someone feels like they just can’t go on and we are missing the...do they feel like their connection with the patients has been degraded and do they feel like their abilities as clinicians and caregivers is not there. We are missing that.

Moderator:Thank you for that reply. The next question is for Greg and Stacy. Were the team characteristics clustered by teams? In other words, if you had multiple members of the same PACT respond, how as that handled? I guess I am asking what the unit of analysis was- PACT or individual team members?

Dr. Stewart:That is a great question and obviously from your question we realize that is not necessarily an easy issue. What we did is we went back and looked at the membership and we found the lowest level unit that we could that was often a team that included two or three teamlets. Now, there were instances where we could find a teamlet that consisted of the four individuals that were assigned to that teamlet only. We could give feedback there. Now, in that case, we probably wouldn’t because you would only have four members and it is hard to keep the confidentiality of it, but we essentially in some of our other work that we will present at another time is going back and mapping these team memberships and what we are trying to do is find the most logical unit, the lowest we can where the people are shared. What would happen most often is those 55 we talked about would be teams with three or four teamlets in them where they would share members across those and that was the feedback that we were given there. I might say in some of our other research what we are finding is that the work is more effective and we are able to do better when teamlets don’t share their members as much. Obviously, that is a different issue but we do find that in terms of this analysis we went back and tried to group them as logically as we could and in most cases those 55, what we call teams, would include two or three teamlets under them.

Moderator:Thank you for that reply. Going back to the first half of the presentation...Do you know if when respondents didn’t feel their work was well matched to their training, was it because the work was beneath the highest level of their competency or above it? 

Dr. Helfrich:That is a great question. For 2013, we don’t know but we had a more nuanced measure at working at top of competency in 2012 which asked exactly that. It was much more frequent that respondents said they were being asked to do work beneath their level of training rather than exceeding their training. Perhaps, as interesting or more important, that had a higher association with burnout. Being asked, in that smaller proportion of respondents who said they were being asked that exceeded their training that was not associated with burnout as strongly as folks who said they were being asked to do a high proportion of their work that was beneath their training essentially.

Moderator:Thank you for that reply. This is going back to the TIRPS survey...perhaps it is; maybe it is more generalized. Is there a centralized place where findings from PACT Demo Labs are being archived?

Dr. Stewart:Within our Demonstration Lab we do have a point that demonstrates those and has those reports and other things. That is our Demo Lab that is also coordinated through the national center and maybe Christian...

Dr. Helfrich:That is a very timely question because actually there is an effort right now...like Greg said...I think that the Demo Labs are keeping summaries of their findings. We also have this PACT cyber seminar series which documents many of the key findings both nationally and with the Demo Labs. There is also currently an effort underway to synthesize all of the evaluation findings and that is something...there are currently meetings with representatives from the Demonstration Labs from VA Central Office and from the Coordinating Center as well as other researchers that have worked on PACT to try to just document in one place the comprehensive report what we found and that work is underway right now.

Moderator:Thank you for that reply. We are going to continue on with the rest of our questions but I just want to let the attendees know that I am about to put our feedback survey and we do very much appreciate you providing us feedback as it is your opinions that help guide which presentations we are going to support in our program. Go ahead and fill out these questions while we continue with the Q&A. When you submit your answer you are not going to see a confirmation that we received it but don’t worry; I assure we have. 

The next question for Dr. Stewart...The number of respondents within each role over the four years is very small. What percent response rate do these numbers represent?

Dr. Stewart:The sample that we started with in 2010 was a sample of 22 pilot teams and so in those 22 pilot teams we only had about 100 participants and so that is a pretty representative or pretty good response rate for that small group. What we have done over time is we have grown that. We started with the 22 pilot teams and the trends that Stacy shows you today are consistent across those 22 teams and the people that we started tracking early on. Fortunately, over time, we have been able to grow this and broaden our sample. We talked about in 2013, we are able to get about 500 which is a response rate of about 35% or 40% of all the people within the VISN, which isn’t as high as we want but probably fits with most of our research. When we take the pattern in 2013, for instance, and we look at the results for those initial small sample and those 22 pilot teams and we compare that to the sample from the entire VISN, those responses aren’t very different. That gives us some hope that maybe we have tracked this and had we measured this across the VISN back in 2010 we might have seen similar patterns. Obviously, we don’t know that but what we do know is in the later years as we have grown the sample our initial pilot teams and the people we track more specifically aren’t really different than the rest of the VISN. We kind of make an assumption perhaps that they have tracked along similarly but what we have presented today is just defined in those 22 teams where we had a pretty good response rate because we worked closely with them early on.

Moderator:Thank you. The next question goes back to physician burnout. Given that we did not see a significant change over time in physician burnout, what other factors could be accounting for the high burnout rates besides PACT?

Dr. Helfrich: This is something that I didn’t have time to go into but I think we have good reason to believe that burnout is an issue in medicine overall. Last year, Tait Shanafelt who is a physician researcher who has done a lot of work on burnout and professional development in physicians published a national survey. They did a cross sectional survey but national representative survey, and for physicians in general in internal medicine the burnout rate- this was assessed with Maslach Burnout Inventory, exceeded 50%. Tait had some data on a small group but still a reasonably large group of 180 I think, physicians from the VA in his sample, and the burnout rate was not statistically different from general internal medicine overall in that sample and the VA sub-sample. I think this is a problem generally. I think the demands and expectations of what primary care is going to be like and the amount of time spent with patients...I think those things are changing dramatically and I think many of the changes have result in making clinicians, both physicians, nurses and other support staff feel like maybe they don’t have the same prerogative and control over their jobs and in many cases they feel more and more like it is a job instead of a vocation. That is just my perspective on this.

Moderator:Thank you for that reply. We do just have one more pending question and Christian, it is also for you. Could slight decrease in burnout shown in some of your data be a result of burned out providers and staff leaving?

Dr. Helfrich:Absolutely and that is a very good point and something that we are trying to understand and it is something that we are looking into. If you have somebody that is really burned out, a couple of things. They may be less likely to respond to the survey so even with the high response rates to the All Employee survey we still are having 45% in 2013 who didn’t respond and those folks are much more likely to be burned out. Of course, you could have them actually leave the VA and as we have noted in the background we have noted an increase in burnout. The baseline turnover rate for providers is about 30 providers per 1000 providers and since PACT that has increased to 34, I believe, per 1000 providers. Those folks are not going to be responding to our survey. Presumably they are going to be, as non-respondents, not showing up as burned out. We are going to be looking at burnout and turnover. That is something we have on the docket for future analyses. At the same time, I do think that we have seen this consistent trend in burnout in job satisfaction, declining job satisfaction, so I think that whatever the turnover rate is it has not masked a trend in declining job satisfaction.

Moderator:Thank you for that reply. That was the final question. I can’t thank you three enough for lending your expertise to the field. I do want to give each of you an opportunity to make any concluding comments you would like. Christian, do you have anything you would like to wrap up with?

Dr. Helfrich:Yes, two things. If anybody would like more information including, for example, information on the psychometrics of the single item measure please contact me. I am happy to share a paper that is under review right now. Overall, I would like to say I think that this is a critical time in primary care nationally- not just in the VA. This is a big initiative and like Greg said, big initiatives pose a lot of challenges and I think it is not surprising that we have seen a high level of stress and dissatisfaction. I think one of the critical things for the VA and for us as a research community is going to be understanding where it is working and what is driving that and that is something that our Demo Labs are working hard on and the Coordinating Center is going to be working hard on.

Moderator:Thank you kindly. Greg and Stacy, do you have any concluding comments?

Dr. Stewart:We just echo Christian in thanks for everybody’s participation and your thoughts that have been shared through your questions. Once again, you can contact us- we do have several things that are documented on share points and reports and if anybody would like more information please let us know. Our goal is to share that broadly and echo again Christian’s idea that this is a very complex roll out. I think it may very well be one of the largest change initiatives ever as we think about changing medicine in somewhere as complex as the VA. I think our results suggest that it is very difficult and some of our Demo Lab work that we do find evidence maybe we have turned a little bit and things have improved over the last year. We look forward to continuing to get that and to assess as it goes on. We are optimistic about many of the things that we are seeing in terms of eventually improving work life as they work through some of these pressures and improving patient outcomes particularly as Christian talked about places where we get this right and learning how to find out what they do and share that broadly is an important facet. We are encouraged by what we are finding.

Moderator:Excellent. Thank you so much. Once again, thanks to our presenters and thank you also to our attendees for joining us. Remember that we do have PACT sessions every third Wednesday of the month at noon Eastern and you can always go to our online registration catalog to sign up for those. Just so our attendees know, I will be leaving up this feedback survey for the next 15 minutes or so...so take your time submitting your responses and any notes that you would like us to take a look at. Once again, thank you everybody and that does conclude today’s HSRND cyber seminar. 

 [End of audio]

Page 1 of 16