Timely Topics of Interest

Identifying Best Practices within the MOVE! Weight Management Program for Veterans: An Application of Qualitative Comparative Analysis

12/15/11 

Event ID: 1876888

Event Started: 12/15/2011 11:53:51 AM ET

I would like to provide introductions real quick. And we will get going. 

Dr. Leila Kawadi is the Deputy Chief Consultant for preventative medicine within the office of patient care services VA Central Office. She is involved in developing policy related to MOVE!, the weight-management program for veterans and has led to program evaluation since 2006 She is board certified in both of family medicine and general preventative medicine, public health. Before joining the VHA in 2004, she was a national research service award primary care research fellow at the University of North Carolina in Chapel Hill. 

Joining are today is Dr. Heather Kane, she is a health services analyst at RTI international, specializing in qualitative research methods and evaluation she has over five years experience conducting mixed method evaluations for varied programs including workforce development programs and obesity nutrition programs. Before joining RTI, she was an agency for health care research and quality national research service award post-doctoral research fellow at the Cecil G. Shepp Center for health services research at University of North Carolina, Chapel Hill. 

Finally, presenting today also is Dr. Megan Lewis, a senior research scientist in RTI international health communication program. She is a team leader for the program healthcare communication team and she has over 20 years experience conducting health related research evaluation funded by the national Institute of health, the CDC and foundations.  

We are very thankful to have all of you presenting for us today and now I will 

turn it over to you. 

Thank you good afternoon everyone. On behalf of my co-presenters Dr. Kane and Dr. Lewis, I want to thank you for this opportunity to share our program evaluation work. We will each be presenting portions of today's talk and we will get started with just a quick poll to gauge you familiarity with qualitative comparative analysis methods. The poll questions should be coming up on your screen in just a moment. The question is, have you heard of qualitative comparative analysis methods before? Yes, no, or not sure? 

Thank you...we do have alot of answers coming in we will give people just a few more seconds to respond to that. about 3/4 of people have voted thus far. ok we've had an 80% response rate so I'm going to go ahead and close down the polls and share the results with everybody, so as you can see 45% of people have answered yes, 40% of people answered no and 15% of people answered don't know so thank you to those respondants.

Ok so thank you all that is helpful, this next slide is just a brief overview of todays presentation we'll spend just a few minutes on background and then spend some time on defining some QCA terms and definitions, we'll spend the majority of the presentation discussing the actual steps for conducting a QCA and we'll specifically be referencing our application of it with the MOVE! best practice evalutations and then finally we will wrap it up by discussing the advantages and challenges of this method. 

So, little bit of background.  First The MOVE! weight management program for veterans is a national program. It was initially developed in 2002, piloted and then rolled out nationally in 2006. It is based on NIH and VA DOD clinical practice guidelines, US Preventive Services task force recommendations and other influencial literature like the diabetes prevention program. There is a VHA handbook concerning MOVE! policy and facility implementation is supported via tools, and training for staff and Internet and intranet website, national monthly conference calls for technical assistance,data -- outcomes report. 

while group care is the predominant mode of MOVE! care, it can be provided to individuals via a face-to-face clinic visits, or telephone, as well as a Telehealth option that just became available within the last year. My office, the national Center for health medicine and disease prevention which is a component of the office of patient care serives central office is responsible for MOVE! policy and oversight and about three years into implementation, we could already tell anecdotedly that there was tremendous variation with respect to how MOVE! was being implemented at different facilities. We figured that was also probably going to be the case for patient outcomes in terms of variablity and more importantly field staff are asking us for what are the MOVE! best practices, how should we be organizing our programs? So our office contracted with RTI international to identify those facility structure processes, policies and clinical organizational features that were linked with better facility patient weight loss outcome.  And so We refer to this as the   MOVE! Best Practices Evaluation and today's presentation is the result of this wonderful collaboration with RTI. I'm going to toss the next to my colleague from RTI, Dr. Heather Kane to get us started. 

Thank you, Leila. I wanted to spend a few moments kinda going over some characteristics 

and definitions of QCA since as we saw there are a number of folks who are kinda brand-new to the 

method. So, QCA is an analitic technique that bridges qualitative and quantitative methods. You have the qualitative component insofar as you are drawing on in depth and deep knowledge of particular cases. You have the qualitative -- quantitative component that in the -- algebra is sort of the underlying mathematical logic to the method. 

This method is based on a set theory. So it examines set theoretic relationships, if X, then Y. Now set theoretic relationships is just Kinda a fancy way of saying you're looking at sets and subsets of things. It is in fact important because most of the hypothesizing that we do is that theoretic in nature. For example, if you have a physician champion, you will have a successful intervention. That is a set theoretic statement or hypothesis and what your hypothesizing that the cases with the condition of having a physician champion or having X exhibit the outcome y of success. 

Another feature of QCA is that it is useful for small or intermediate sample sizes. Whenever you are doing an evaluation and you realize that probabilistic or statistical methods are really not going to be useful or admittable to your circumstances, this method offers a way to systematize your analysis using smaller number of cases. 

Another characteristics of QCA which is something of a limitation as well, is that you have to focus on a limited number of conditions and conditions are just another way of saying something like a variables for those who are more regression oriented. Now, that means that you cannot use a kitchen sink approach to how you design your model. You cannot toss in every variable. 

In this analysis, you're analyzing combinations of the condition or the variable that you are putting in. As you add a condition, you are increasing the possible number of combinations exponentially. For example, if you have four conditions, you have two to the fourth power of possible combinations, so you have 16 potential combinations. As you add those conditions, you will need more cases in order to minimize the problem of limited diversity, and I won't go into that today, by I will point 

you to Reagan on that. 

Another feature of QCA is that it examines multiple pathways or multiple solutions that can lead to success and so this principle is known as equafinalities-- that you can have multiple recipes and  this is not uncommon in an evaluation where you find that there are different program models that you find are all equally successful. 

Finally, this is a flexible method that supports theory building as well as theory 

testing, so Working from an existing model. And then, as I hope you'll sees through our presentation today, it can provide actionable policy information. 

So, the findings from QCA are reported differently from outcomes than you might see in a quantitative or qualitative study. Now when you are using traditional quantitative methods, you're looking at the additive models or metafacts. Holding all of the variables constant X has this effect on Y. In conventional qualitative methods like case studies or textual analysis, you're getting sort of --------descriptions of cases of or drop case analysis. So QCA looks at something in little bit different. It presents necessary and sufficient conditions as part of its findings. 

Briefly, and I'll do next ended example in a moment, necessary conditions are those that must be present for an outcome to occur. Sufficient conditions are those that guarantee an outcome will occur. Our outcome for this evaluation was the facility aggregated patient weight loss outcomes at six months. 

We will spend a few moments to think through necessary and sufficient conditions. They are little tricky and hard to get your head around but I think this is a really fun example that makes it very clear. So, in the example of the left, you have the necessary condition. Buying a lottery ticket is a necessary condition for winning the lottery. When you hear the slogan, you have to play to win, that is right. Now buying a lottery ticket is not guaranteed that you will win the lottery. But to even have a chance to win, you have to have a ticket. In this case, necessary conditions are cases with the outcome or Y which is a subset of the cases with the condition X. You see that in the drawing. In the example on the right, you see some samples of sufficientt condition. This is a 

guarantee of the outcome. In this example you can see that there are several sufficient conditions and a combination of sufficient conditions. You have X1, matching all of the numbers, you get the jackpot. It guarantees you a lottery win. You have X2, where you matched five of the six numbers, you're a lottery winner, but you may not get a bigger payout. And then you have a combination of X1 and X3, were not only get all of the numbers, get a secret bonus number of the day, you get like a double jackpot. So in all of these cases these are sufficient conditions and you can see that the cases with X or combinations of X are subsets of the outcome Y. So with that I'm going to hand over the mic to Megan Lewis. 

Thank you, Heather. In the next part of the presentation, we're going to walk through a series of 

steps that we have outlined to help describe what we did in our evaluation. In on this slide, you will see a set of 10 steps I will go over briefly and then will go into more detail about each one. 

In any research or evaluation you start with and identified the research question and develop a conceptual model and that is where we started as well. In our second step, we selected theorectically meaningful cases, we'll talk about that more later. In step three, we chose a variables or conditions that were the focus of the study. As Heather said in QCA, variables are called conditions. In step four, we identified or developed data sources that were used in the evaluation. We have multiple data sources that we use help triangulate information about the conditions that we were interested in in this study. 

In step five, we calibrated the conditions which basically means that we operationalized our variables. This involves setting decision rules about group membership and we'll talk more about that as well. Finally, once you have all of your data we did the same thing that everybody 

does, we managed the data, coded the cases and then we analyzed the data using QCA. 

On this slide, it shows the conceptual model that we started with as part of our project. We use the conceptual model to guide the implementation evaluation of move that I developed with my colleagues, Brian Weiner and lore -- Leman. In it we describe the process by which the move would be most effective taking into consideration the variety of factors about move and about the implementation contact. 

Do to some factors that we will discuss later, we primarily focus on conditions that related to the boxes and concepts resource ability and implementation policies and practices. In addition to this model, we also develop a flow diagram that described the various clinical program features that we would expect in the successful weight management program and how this related to the process of care for move. 

Now I will turn it back over to Leila. 

The next up is to identify cases. In QCA cases are selected into one of two groups based on the presence or absence of the outcome of interest. As Heather mentioned in our case, we used patient weight loss outcomes aggregated at the facility level to guide site selection. We considered all facilities including in community-based outpatient clinics who had seen at least a 30 patients with move and calculated the facility's mean body weight change for MOVE treated patients at six months of also the percent of patients losing 5% or more body weight. We then ranked facilities from highest to lowest on both of those measures and created a combined ranking from which to select facilities. 

We selected a total of 22 sites and this number was based partly on the resources that we had available to conduct the evaluation but also based on a QCA requirement which is the need to maintain imperical intimicy which is something that is typical of case oriented research but not so much for variable oriented research, as most of you know. 

We selected 11 sites from among the top of the combined rankings and 11 sites 

from the bottom of the rankings taking into consideration facility size, location and complexity to ensure diverse sites. So we didn't just take the top 11 or the bottom 11 in straight order. 

The most important thing I think to mentioned about the case selection, is that in our case, it was performed by the VHA part of our team and the RTI team was blinded to which group each site had been selected into. They were blinded throughout the data collection and analysis phase. I think that is a strength of our application here.

Thank you Leila. I'm going into a little more detail about the choice of our condition. So as I mentioned previously, we prioritize resource ability and implementation policies and practices from the implementation effectiveness model focusing on clinical program features that should be part of any evidence-based weight management program. As Leila mentioned, we do this for a variety of reasons. Some were practical, we had a finite budget to work within like any research study. We also knew from VHA staff, in Leila is office and the reports from the facilities that these two areas that were sometimes important or challenging for MOVE! implementation. We also knew from starting to identify and develop data sources that these might be areas where we might be able to either develop or identify multiple data sources that we could use to help in our calibration. So for each prioritized concept, we developed an exhaustive list of conditions that might be a potential indicator or measure. 

Our challenge really was kind of narrowing the list of concepts from the models and factors that we knew were important because in QCA, as Leila mentioned, you want to have a narrow set of conditions, four to five is generally thought to be best. We ended in our particular study with a list of 17 conditions that are shown on the next slide. 

On this slide, we've also grouped the conditions by the concept so you can see how we classified  them. I'm not going to go through each of these, but you will see that for example, an interface between screening and treatment, the first one there, under implementation of policies and practices, we looked at is did they use an orientation or some type of class that would link a patient once they had been screened on their weight to treatment. There may be some kind of facilitation or process there. And so, we looked at how we might collect that information because we thought it would be important. And so we went through Want to this process for each of the conditions that are shown on this slide. 

On the next slide, there's a little more detail about how we identify and develop the data sources that we used. For each condition, we identified preexisting data sources that might be available to us through reports or EMR, or other things like that or developed a data collection tool ourselves. For example, we developed a program summary form in which we asked sites to describe the patient flow through the program. We also collected organizational charts that showed the supervisory hierarchies and functional relationships between the different lines of care that focused on move and information about their staffing and level of effort that was available for move. 

We also conducted interviews with move staff that usually included the move coordinator and usually one, two or three other people involved in the care. Then we had a random sample of 50 patients from each site in which we extracted information from EMR's. In addition, we had sites review a summary form that we provided that was a summary of the information from the interviews, and the organizational charts, their patient flow diagram to make sure that we had a very accurate representation of their program. They provided feedback to us and clarification on important points. 

Then we dove into calibrating the condition. On this slide you will see the formal definition of calibration which is the degree to which cases satisfied member criteria in a condition which in turn are usually imperically determined and not inductively derived. Basically that means that we developed decision rules about membership in the group, or another way of thinking about it who have not used QCA, we operationalized our variables. For example for the position active champion, we looked at the data and give up with a definition that we thought represented a physician champion that would be active. For example, the active physician champion supported other medical personnel and program staff. They ensure that clinical reminders were met and emphasized move to patients, determined if patients were medically fit to participate, counseled patients and actively sought out resources for move. If a site had a physician champion and showed these types of characteristics, a physician champion was designed as the active at that site. If the champion with was one in, then they were classified as having had no active champion at this site. 

A little more detail on calibration and QCA. We used crisp set calibration. QCA also uses fuzzy set calibration which we decided not to use. Crisp set is basically like developing a dichotomy. A dichotomous variable whereas fuzzy set is more like a continuous variable. In crisp set each condition is evaluated and rules are established about membership. The three sub bullets on the slide, describing absolute or relative membership in what we call QCA jargon are all the same way of describing if a condition met group membership. 

We drew on expert knowledge to develop calibration decision rules. There were long meetings between the RTI, VHA staff to decide on these rules. Like any research in which measures are defined, it is a somewhat interitive process in which we establish rules and divise them to pilot testing with two sites. Just to mention, we chose crisp set for the study because we thought it would be easier to interpret and make policy recommendations about. A physician champion is either present or absent, but what does it mean to have half a physician champion. The crisp set made more sense in terms of policy recommendation. The disadvantage is that you lose some granularity in describing group membership. 

Next slide. Our next step was really to manage and code the cases. For each facility, we compiled extensive information for each condition by data source and then make a judgment about whether that condition was present or absent. For example shown on this slide, and the table for position champion involvement, our spreadsheet in real life would not have shown the detailed information and text from interviews and numbers and things like that, instead of XYZ, but basically 

for each facility we compiled the different data sources for each of the conditions and look across the multiple data sources to triangulate and then come up with a decision rule. Would the physician be classified as being present and fully in the set or absent and not in the set. We do this for each facility for each condition in our study. 

We'll see step seven. Just like in any QCA analysis, you first have to create a truth table to summarize your results. The truth table links possible combinations of conditions with the number of cases exhibiting the outcome for each combination. Because we had so many conditions, we did a modified version and in doing so, we determined that no two cases, no two facilities were identical on their combination of conditions. I will show you sample version of a truth table in a moment. We used data to create this truth table because with seventeen conditions, that is two to the 17th power there were 131,072 possible combinations. This somewhat emphasizes the importance of limiting conditions when you can. Also, with a smaller number of conditions, you can even create your truth table manually in Excel or in word. Then, by using STATA, we look for single conditions that were necessary or single conditions that were sufficient. We found that two conditions were individually 

necessary and we did not find any single condition that was sufficient. With our 17 conditions, we ended up needing to use a bottom-up approach to QCA. Just as with any research study, we reviewed our findings, re-examined our data and we considered models iteratively until we accounted for the unusual findings. 

This is a sample truth table with some results of calibration. For each site, you can see that we're looking at conditions X1, X2, and X#. And look whether it is present or absent. For you see a +, it is present, and where you see a minus sign, it is absent. Once you've laid this out, that's when you examine the truth table for patterns and the configuration of the condition. In this simple site example, we can see that all sites with the outcome present have condition X1 present, so sites in blues, sites B, E, H, and J. You also notice sites C and I have condition X1 as present, but the outcome is absent. So, with that analysis, we call X1, a necessary condition. That is, in order to have a chance of achieving the outcome, having the outcome present, the 

condition is X1 necessary, but it alone does not guarantee that the outcome will be achieved. 

Now, conversely, condition X2 and condition X3 are not necessary because we can see patterns where the outcome is present and X2 and/or X3 is absent. So X3 is not present in site B or site J, and X2 is not present in site E. For this analysis, we can say condition X1 is necessary, that is to say it must be present. So, the next up we look at conditions that are sufficient for achieving the outcome. For our analysis, we then only focus on the sites were condition X1 is present because sited without X1 did not even have a chance at exhibiting the outcome of success so we're 

looking at the blue and red sites in this phase of the analysis. What you will notice about the remaining sites, is that either condition X2 or condition X3 is required in combination with X1 in order for the outcome to be present. Thus, the combination of X1 and X2 or the combination of X1 and X.3 are sufficient combinations. That is, these combinations guarantee the outcome. 

The necessary conditions for MOVE!, we found two that were individually necessary. The use of a standard curriculum and the delivery of care via a group-based care. These two conditions were present at all 11 sites with the larger patient weight loss outcomes as well as five sites with the smaller patient weight-loss outcome. Therefore, the absence of these two conditions guarantees that a site will have a smaller weight-loss outcome. As I mentioned, we tested single conditions for sufficiency and no single condition was sufficient. We did find sufficient combinations that we identified. We had a high program complexity in combination with high staff involvement. Active physician champion in combination with low facility accountability. Use of a mixed delivery format, both a combination of group and individual, in combination with low facility accountability. And use of quality improvement strategies in combination with no waiting list. I would like to point out that most sites had more than one sufficient combination of conditions present, hence why you see n=5, n=5, n=5 and then n=3. These findings might be somewhat unusual or even counterintuitive, so this demonstrates why the imperical intimacy that Leila mentioned earlier is essential. The low facility in combination with the active physician champion and the low facility with the mixed group format required more investigation. So Went back into the quality of data and look at these high-performing sites that exhibited in these recipes. What we found, is that these sites, even though they did not use a formal reporting mechanisms as much, they had a great deal of informal reporting mechanisms in place. So, dealing with problems as they emerge rather than relying on formal reports  up the chain of command. Similarly, the use of QI strategy in combination with no waiting list seemed a little unusual to us. We investigated those facilities that had that recipe. What we found, were those sites were using quality improvement to monitor patient volume as well as waiting times and then they adjusted their program to minimize the waiting time. 

So in the next slides, is a nice visual summary of our findings. It also shows how cases could have more than one combination of sufficient conditions present. If you look at the 12 o'clock, at the center circle, the active champion and low program accountability, you can see that sites one, six, 11, 21, 19 have that recipe present. If you go o'clock wise to the high program complexity and high staff involvement, you can see that sites one, four, 11, 12, 20 have that recipe. In that case you can see that site one and 11 are repeated in both recipes. With that I'll pass it back onto Leila for the limitations of the evaluation. 

Thank you, Heather. The major limitation of our particular QCA evaluation involved issues related to case selection. When we first set out to identify the cases to be in the analysis, we used crude patient outcomes without any risk adjustment or relative comparative to untreated overweight obese patients. We also used six-month patient weight loss outcomes from the year prior to the QCA data collections, but mitigated this issue by ensuring that interviews and program summary forms clearly distinguished MOVE practices in place at the time of the data collection versus what may have been in practice or in place the year prior. Overall, we found that MOVE practices and features were generally stable between years at all of the sites. 

Lastly, outcomes that we used to select cases were determined based on vital data from the VA corporate data warehouse which is an electronic extract of VISTA vital sign package and unfortunately weights are not always consistently entered into the vital sign package, they often gets reported in text progress notes which aren't as easily extractable and there's also data entry errors by front line clinical staff that sometimes results in data that is difficult to interpret. The second limitation is the particular QCA calibration method we selected, crisp set, as Megan and Heather mentioned before, forces a dichotmy for coding sites. Either the site has the condition or does not. For some conditions this may make sense, but it may not be appropriate or as useful for all types of conditions that we study. Consideration to be using fuzzy set calibration in the future would allow for more graded calibration of conditions, that it also has downsides and that might be more difficult to interpret. 

The real benefit of this overall evaluation to our program office was the ability to immediately take the findings and translate them back into policy and practice for move. We have worked with the field staff over this past year to not only disseminate the findings but to task them with making sure that their local programs have the necessary conditions that were identified through this evaluation. So site that did not have a group component for their care services have been asked to implement the 12 session group curriculum that has already been available at the move site for some years.We have defined more explicit expectations for the physician champions and have 

supported local VISN efforts to reevaluate existing physician champions that are in place and consider whether or not the right person is in the role and meeting expectations. We've asked sites to reevaluate their current structure and build in additional complexity through the addition of maintenance components, staff from other disciplines or orientation sessions to create a smoother transition between primary care screening and actually getting started with the program. We have asked move coordinators to become a little bit more engaged in their local quality improvement and our system design efforts so that not only can they learn those strategies but also help create relationships with those local experts in order to be able to apply those kinds of principles to the move program. Lastly, facilities are now asked to report on the exntent that some of these as 

practices have been implemented on the annual end of year report that has to be submitted by each facility. This put some accountability on the sites but it also brings these best practices to the attention of senior facility leadership who have to sign off at the end of the year report that get submitted. 

We are just going to end by talking about some of the general challenges and advantages of QCA. On the left side of the slide are some challenges. It is not suitable for very small case studies. We had actually quite a large number of cases in our particular evaluation, but 3 to 5 cases might be pushing it. It also involved intensive data collection and iterative analysis which can take a lot of time and resources. And the idea that we talked about before, the limited number of conditions are allowed. You cannot just have the kitchen sink approach, there has to be well 

defined concepts, models and variables to guide the selection of conditions that are included. It also may be less useful when programs are not comparable. In our case, move, although there was a variable implementation, had a pretty similar programmatic structure throughout the facilities. 

Also, for some evaluations or studies, it can be challenging to have a collaboration 

between the program implementers, in our case Leila and her staff and evaluators. We had a great collaborative working relationship, but sometimes that can be challenging and in some disciplines it is actually seen as a negative to have a close collabortation. I want to say one more point about this collaboration, to kind of guard against any kind of methological slippage, we had safeguards that we have built-in. So for example, Leila described the RTI staff being blind to the outcomes at each of the facilities. One thing I do not mention earlier, is that the VHA staff were blinded and were not involved at all in the assignment of the calibration for each of the varaibles at each of the sites. So even though they were involved in the definition of the rule, as experts and having expert knowledge, they were not involved in the coding of the data. We kept those kind of fences around the data and around the site selection that helped ensure that there would not be some bias. 

Some advantages of QCA that we have found is that it offers a useful analytic tool for studying organizational processes and programs. I think most important, it allows for testing the equifinality prinicple that's so important in organizational studies. There are multiple pathways to success and you can test that specifically in QCA which you cannot do another kinds of methods and you can quantify it. It can be applied in healthcare settings with actionable results as Leila just described and it supports theory building and theory testing, both of which are important research activities. 

We're going to put up our second poll. The poll question is, now that you have learned more about QCA, do you think it would be a helpful approach to use in your research? The choices are very helpful, somewhat helpful, possibly helpful, not helpful at all. 

So, as you can see, 25% find it very helpful, or 33% somewhat helpful, 41% possibly helpful and 1% says not helpful at all. 

Just to close up, we're going to leave you with a few citations. The first one is the result of our move the best practices evaluation that was just published this past November in the American Journal of preventive medicine. In addition to the publication, there is a fairly detailed technical appendix that is available through the journal website that provides a little bit more 

detail than what we had time to go into on today's talk if you're interested in specifically how we approached the case selection or calibration of conditions which always seemw to be what generates the most questions. Then, at the bottom of the slide are some other citations, having to do with QCA or the model that we'd used. The last citation on the page is actually one of the relatively few applications of QCA in the health services literature that we could find when we started out this project. The one at the bottom is the one that we used since we took on this project there have been a handful of others that we have seen pop in the literature but for another health application of the method, we refer you to that last citation by Dy in health services research. 

Finally, is our contact information. We would be happy to take any follow-up questions after the seminar ends. I think we will turn it back to Molly for questions at this point. 

Thank you. For those of you are looking to see how to submit a question or comment, go to your gotowebinar panel on the right of your screen and click the + next to questions and then you can submit your question or comment in that bottom box and press send. The first question we have, do you happen to know whether data collected at site level was based on primarily male patients or male and female patients or has anyone looked at how the move program outcomes may vary by sex or gender of veterans and by site characteristics? I'm asking because of the sex differences in weight management. 

That is a great question. This is Leila. We do have some outcomes by gender for move that are available at the move website. There's actually a --- outcome reporting service that is available that does allow you to look at outcomes by gender. Overall, we find that the weight to differences are minimal. Men do seem to do a little bit better than the women, at least the data that was from a year ago, but they're not large differences. Overall, the number of women who participated in move is about 13% to 14% of the move treated population which is still quite low but it is higher than the percent of women generally seen in the VA nationally. So there is a little bit of a skew to the genders that participate in move. I hope that answers the question. 

Thank you for that response. We do have somebody writing and wondering if you 

have the URL for the outcomes reporting that you just mentioned? 

I do not have a right at my fingertips but, I'm not sure what the best way to pass on other than, it is currently at that --- and I can have people, if you want to e-mail me, I can send you the link to it. It is too long and complicated to just read over the phone. 

That sounds good. Also, we have a request for you to move back one slide to the reference. Another question, were those women only move groups or mixed gender groups that you just referred to? 

For the most part, they are mixed gender. Most sites have had almost no success holding women only groups just because of difficulty finding time in days of the week that meet the needs of most participants. For the most part, they are mixed gender groups and predominantly, generally more men than women. 

Thank you for that response. That is the last question that we have in the queue at this time. If anybody has any remaining questions, feel free to write in now. If any of you ladies have any concluding comments you would like to make, I invite you to do so. 

This is a Leila, I think somebody had just popped up the --- website but you will not be able to get to the outcomes report from the main BCC website because it is a brand-new report and has not found a permanent home a yet on the --. It is live in production, you will not be able to find it unless you have the specific link. Do not spend a lot of time going to the the VSCC website and 

find it directly from there.

The next question that has come in, would it be appropriate to use QCA to determine effective components of an individual level intervention compared to an organizational level intervention? 

That is a great question. Heather, Megan to have any thoughts on that? 

Absolutely, instead of organizations, your cases are individuals. Again, that is another situation where you'd want to balance what we mentioned as imperical intimacies which is having enough knowledge about the individual cases or your individuals with the number of conditions. That said, we kept noting that you could use this primarily for smaller and intermediate studies. There are also works under way where people are using it for large sample sizes. That is when you lose your imperic intimacy but what is a really lovely use of QCA in that purpose, it complements your probablistic methods. So if you have some unusual findings it is a little hard to figure out what is distinct about this cluster of individuals that yields those findings but using QCA hand-in-hand with regression or other probablistic methods is fantastic because they both answer different questions and can be quite revelatory. 

I would add to that, in our process of learning about QCA for this project, is that you could use it as Heather said, you can apply it to any unit that you are looking at. There's been examples of QCA looking at country data level, example looking at couples, individuals, so it's something that you could combine as Heather has said with multiple different types of units. 

As a side note, there is an article coming out in health services research that does a nice combination of QCA with regression analysis where they have individuals and also bring in some organizational data. That should be out, I'm not sure exactly, but the lead author is Chaung so keep an eye out for that if you'd like to see another application of QCA. 

Thank you. This is a related question which may have already cover, but have case management programs been assessed using QCA? 

This is Leila, case management, I am assuming patient case management programs. That is not an application that I have seen. I don't know, Heather, Megan if you've seen that. 

I do not recall any. 

I have not seen any does far, but that said as Leila mentioned earlier, this method is just starting to make inroads in health services research so the applications are quite rare. I have not encountered anything related to examinations of case management programs. That might be something that you could do. 

Thank you. There is one more question regarding QCA. How do you validate the 

results of QCA?

So, we did not go into too much into this of this presentation, but there are analysis, did we have our additional slides? There are measures to do that validation. Those are called consistency and 

coverage. Coverage is saying how many of those cases have that outcome. You could go back to the lottery example, if you're outcome was keeping people out of poverty, Y, the solution was winning the lottery , that is a very rare solution. It would not provide a very actionable information because it doesn't happen much. Coverage looks at how imperically relavent these solutions are. We can see what we did that analysis here. There is also a measure of consistency. That is, how many of those cases exhibit the outcome. And I'll try to think of a nice example. If you had some cases that, some of the cases that had the outcome, other cases do not have the outcome, yet it still seems to be a dominant solution to your problem then you would say, it has high consistency but not all of the cases showed whatever outcome. Those are some of the validation measures that we use for QCA. I would point you to the Reagan book for a very thorough discussion about consistency and coverage. 

Thank you for that response. The next question we have, how many resources do you deed to do the study? 

This is Leila, on the VHA side, we had a team of about three people, obviously not working full-time on the project, but who consulted throughout the length of the project. It was a project that's spanned the course of about two years from beginning to end. Megan, on the RTI side, did you remember about how may people? 

I think we had a five or six. Again they were not full-time. It would be hard to quantify the level of efforts because we all had specialized roles that we played. It again was over the same period of time. 

I think, if I recall, the interview, key informant interviews were probably what took the most efforts. That and developing the interview guide and figuring out what we were going to do and scheduling and following- up with sites- that took the most effort. 

The next question, can you explain in a bit more detail related to your finding that low accountability was a partial/sufficient component of successful sites? 

This is Leila, I think that is where having the imperical intimacies really helped us 

understand that finding because when we saw that finding initially, it was sort of counterintuitive, particularly within the VA culture. So when we looked back to the sites that were covered by the solutions- let me just go back to the vendiagram slide- so the sites that were covered by that solution were basically, there were five sites that were covered by that solution. Really, only two sites were only covered, uniquely covered by that solution. The rest of the sites that included that solution also were covered by other solutions so we went back specifically 

to look at the site that were only uniquely covered by the solution and they were, when we went back to look, they were quite unique sites in terms of their move a program. One of the sites, particularly during the interview reported about how internally motivated of their move team was, how they set goals for themselves, they worked together really well as a team. We thought that the fact that they had what sounded like a very high functioning team may have preconcluded the need for the facility to impose any kind of ownerous accountability onto them. That was what made sense at that one site. I do not remember the specific of the other site, but there was something similar, something unique about that site that made sense when we went back and looked at that specific case. Megan, Heather I do not know if you recall any of the other specifics on the other sites? 

I don't. I think though that one of the points that was an advantage of the approach that we use, highlighting, is that although we have multiple data sources and triangulated information, and then quantified it using QCA, we went back to the qualitative dated so we never went forward and never looked at the qualitative dated. That qualitative data provided a very important contextual 

understanding of what was happening. Some of it is described in the American Journal medicine article. 

One of the hypotheses that we had was that, particularly for site six, which was covered by the solution and included the active physician champion is that perhaps having an active physician champion preconcludes the need for the facility to have a heavy hand in terms of accountability so that would be one hypothesis. As a program office responsible for move, we were really challenged with how to convey this finding because we were not going to come out with a policy that said, facilities should not hold their move programs accountable. That just would not be politically doable. It was a real challenge to understand those solutions. In the end, this solution call high program complexity and high staff involvement has the most imperical coverage and that is the solution that we've focused on because it was covered at the most number of sites and uniquely covered at three sites. That is what we sort of think of as most relevant. 

Thank you for those responses. Somewhat related question, what is low program accountability? 

That was defined, that condition was defined as the requirement for the service or the staff responsible for move to report to leadership on a regular basis in a formal way either by reporting that at one of the Quad leadership meetings or a service line meeting, or to have the support submit things in writing or to explain on a regular basis, performance on measures or requests to provide clinical outcomes. The more involved the facility leadership was in asking the move program for outcomes, not just patient weight-loss outcomes but waiting lists and how are we doing on screening, how are we doing on participation, it was how accountable the move program was held to reporting to facility leadership was what defined high program accountability. One of the other conditions we also looked at was accountability to VISN leadership because what we had seen anecdotally through implementation was that there was some VISNs that were more actively involved at the VISN level with move implementation and so we looked at accountability both at the facility 

local level but also at the VISN level, but the VISN one did not turn up in any of the rest of these. 

The next question we have, in your discussion of methods and steps, you mentioned a diagram of patient flow through the program. Did you have each site describe this process to you or did you create a flowchart for them to modify? 

This is Megan. We had a sample that we provided for some sites, like this is what we 

are looking for. And then each site develop their own model based on what their program utilized. 

What was the range of weight loss? What was the experience of the participants/patients. What was the minimum number of sessions needed to get the weight-loss? 

The range of weight-loss at the highest performing site, I am just pulling it up here in the paper, the highest ranked facility had 54% of its move treating patients achieving a 5% or more body weight loss at six months. Whereas the lowest site had only 4.3% of its patients achieving that threshold. The highest ranked facility had a mean percentage body weight change of -7.1% at six 

months and the lowest one had a mean body weight change of a weight gain of .5%. So there was a pretty big difference in weight-loss outcomes between the facilities that were selected into the larger patient weight loss outcome group versus these facilities that were selected into the smaller patient weight loss outcomes group. With respect to the number of sessions etc., that we cannot answer because that is different. That is one of the differences within implementation. 

We already know from weight loss literature that more intense treatment, more vistis, more contact results in more weight loss. So that is already part of move policy is to provide a more intense treatment. That was not a specific condition that we were looking at because that was, we already knew that was something of that was supported by the literature. 

The high program complexity condition captures a little bit of that 'number of times visit', so places that- facilities that- had more complex programs had more interface with the patients. They would have longer sessions, have maintenance treatments, they would have follow-up. Whereas a facility with the with less complex programs might have shorter program, like a four-week program rather than an eight or 12 week program. So it's embedded a little bit in our analysis. 

We do have a few more questions to get through, I know we're at the top of the hour, but we have three questions left if you can stick around. 

This is Leila, unfortunately I have to go be on another call right at one o'clock. I will have to sign off. Again, if there are any questions that come up, please feel free to e-mail me after the seminar. 

Thank you Leila. Heather and Megan, are you able to stick around and finish answering questions? 

We can stick around for a few more minutes, but some of the more VA specific questions I would recommend be directed to Leila. 

No problem. We have two left. The next question, were the attrition rates reported or analyzed? 

That is one that we will have to direct to Leila. She worked with that data. 

No problem. We do have a comment followed up by a question. The best practice for one site might not be applied to the other sites due to other environmental factors. How do we make sure that the best practice can be applied universally? 

Well, that also seems kind of like a VA office answer but what I guess I would say, from our perspective is that the best practices that we discovered in the process here were really informed by multiple data sources, by expert knowledge, by making sure it made sense, linked with the literature. An then those become recommendations that the Leila's office can help facilities put into place in their local context. So, do not know that is getting at what's the questioner is asking, but it is true that there was variability in the context for each of the sites and each of the facilities that would look that. These findings kind of help raise, for example, program complexity. It is important to have an intensive program and we did not see that in all of the 

sites. To the extent then that sites can learn from other sites, what best practices are working in terms of weight loss, I think that would be helpful tool for facilities.

We saw that some of the conditions were necessary. They did not guarantee the outcome, rather they just, they need to be in place for them to have a chance. Some of the necessary conditions were not required dramatic changes to the program. Integrating what the handbook that they already have in place and making some minor adjustments. So those are really tangible policy changes that ultimately aren't damaging or harmful, I think it is not a bad step in terms of a policy recommendation. 

Thank you both very much. The remaining questions are VA related so what I'm going to do for everybody, I'm going to send those to Leila off-line. Once I receive a written responses, I will go ahead and post them with the archives. You will get a direct link to that archive and inside the handouts there will be the written responses for the remaining questions. 

I very much want to thank Megan and Heather for joining us today. Do either of your ladies have any concluding comments you'd like to make?  

Thank you so much for having us. It was a wonderful working with Leila and her office it was a great experience. We're pleased to be here to be able to share this work with you.