**Transcript of Cyberseminar**

**Realist Evaluation:**

**What Works, for Whom,**

**in What Circumstances**

**Shirley M. Moore, RN, PhD, FAAN**

**July 10, 2012**

*[Poor audio quality on the phone lines made accurate transcription of this session impossible. Please contact the presenters if you need clarification of any parts of it.]*

IR:Good afternoon, ladies and gentlemen. This is Molly Kessner from CIDER. I would like to welcome you to today’s HSR&D Cyberseminar. I do apologize for the delay in getting started and as we are at the top of the hour I would like to go ahead and begin the introduction for our speaker today. Today presenting for us we have Dr. Shirley More. She is an Edward J. and Louise Mellon professor of nursing and associate dean for research at the Francis Payne Bolton School of Nursing in Case Western Reserve University, Cleveland Ohio. And at this time I would like to turn it over. Are you prepared to share your screen, Dr. Moore? 

Dr. Moore:Yes. Are you going to rid of your screen? 

Molly:Yes. You should see a pop-up now and accept that and we should be able to see your slides. 

Dr. Moore:Okay. Well, thank you, everyone. It is great to be here with you today. 

Molly:Sorry to interrupt. We are not actually seeing your slides yet. Let me try to make you a presenter again. You should see a pop-up and then you can go ahead and accept it. It should be popping up on your screen now. 

Dr. Moore:It will show on my screen? And when do I accept? 

Molly:Yes. Just click that. 

Dr. Moore:Okay. 

Molly:There you go. Thank you. 

Dr. Moore:Good to be here, everyone. Today we are going to talk about Realist evaluation. And just to give you some context; I went a few years ago to London to be trained by Pawson Tilley in this approach. I have used it a few times for evaluation purposes and have recently been writing a NIH grant using it. The goal for today will be to talk about what is realist evaluation, what are the methods associated with it and then a little about – I am going to provide some examples by using it by way of highlighting what it is about. If I had you here in person I would do a quick poll so we have a quick poll question. I just want to get a feeling for my audience in terms of who has used realist evaluation for research or program evaluation. 

Molly:Thank you. The poll is now up. I will ask our attendees to go ahead and select one of the options. And we have about a 70% or 60% response rate thus far. We will leave it open for just another few seconds. And we have had about two thirds of the audience reply. It looks like the responses have just about slowed down. I will close the poll in about five seconds and I will start the results well, actually they are pretty short and sweet I will go through them for you. Okay. It looks like we have about 15% response rate saying yes and about 85% response rate saying no. 

Dr. Moore:Okay. Only a few people have really used this in the past and had much experience with it. So I’ve made it kind of a level that would move from beginners through people I think have some experience. So the level of today should be fine. What might differ mostly is the type of questions that occur. 

Molly:So moving right ahead the context for today in using realist evaluation for implementation research and so I think those of us in implementation research have been saying we are really working at this issue of what works for whom in what circumstances. Actually this was used by Pawson and Tilley as much as 15 years ago that this statement we have seen more commonly now in the last few years but I don’t want to say they were the original people, but certainly they are people that have devised and defined a message to approach that  STATIC  implementation research. 

So, a goal today is a goal of realistic evaluation I just want to talk about in general is this idea that what it is all about is a close examination of how variables are connected in terms of what influences what in what way. As we think of implementation research in some ways we are always evaluating some intervention that has occurred in that research. As we know in intervention it is only as strong as the weakest link in its implementation and often or explanatory models we are trying to do is to provide the evidence to strengthen every link in this implementation change. And we are going to be talking a lot today about the chain of action in implementation research and how they are evaluating realist evaluation. 

So realist evaluation it actually draws from the natural sciences in terms of using case comparison. If you go back and kind of think of the Darwin era, people in England and France and Germany in particular a number of scientists in the natural world and how they did constant comparisons among cases. Primarily how they ended up deciding on how things were associated if you will, randomized control trials really weren’t used at that time and if you think about implementation science and natural events unfolding around us we want to go in and find out what is happening in that natural event, what is occurring, whether that be a hospital unit and their attempt for clinical practice guidelines for example this approach we are going to go in and study what is happening. We may have put an intervention in place that we want to study then naturally what has occurred as a result of that. So it is an attempt to control all factors over one event. It is looking at all the factors that are happening. I will talk about why this is relevant. 

It uses both qualitative and quantitative data that is important to say. The realist in realist evaluation comes from critical realist thinking about understanding reality that is going on around us. So it is looking at reality. Probably the best thing I like to think here is that when we are trying to implement something let’s say clinical practice guidelines and then we say, but they are just not doing it. There is some underlying mechanism there that represents the real world or the real reality going on. We thought we put in the best intervention but people aren’t doing it. What is the reality? What is the true mechanisms that are happening. What is the underlying mechanism of what the situation really is. The result of realist evaluation is kind of a middle range theory of how something works. So if we think of the simplest definition of any theory is that it is a statement of how something works in the world and in some ways realist evaluation is testing our best guesses about how something is working and developing some empirical data about that. 

So some key assumptions in realist evaluation, one key assumption is that every intervention that you put in place or every program that is going on are actually series incarnate. In other words they are these series in action about how something is unfolding in the real world. And so the aim of these interventions are to change problematic conditions or behaviors. They hope the interventions will introduce an aim or achieve an aim. And that usually interventions incorporate ideas about how they will achieve their aim. However, most of these ideas about how aims are achieved in reality are not uncovered very well. 

So to continue with these assumptions about realist evaluation and how interventions work is that these theories that are unfolding or the mechanisms are unrecognized, are poorly articulated. These mechanisms are apt to change as programs over time or the interventions unfold. So what we thought we were putting in place didn’t work a little bit. The intervention or the program works differently among different subgroups or different context and I think those of us in implementation research context is an important variable for us in understanding how things differ among context. 

And then the interventions are designed to work through the ideas and attention of those who should be implementing them and implicated them. Meaning, essentially there that the people who carry them out are the possible influence on the actual cause and effect or affect that has occurred. 

A focus of realist evaluation is to identify the mechanisms of actions to generate the change in an intervention. The contexts that are most conducive to positive outcomes are studied. So a big hallmark of realist evaluation is that the contextual factors are not considered compounders to be controlled, but rather they are considered variables to being manipulated by us in some way to be able to understand how the intervention interacts in different contexts, so I’ll mention how we do that. 

Now it usually says STATIC say that they developed originally realist evaluation to be used when we are trying to understand complex interventions and I do think a lot of healthcare STATIC what we call complex interventions. There is a wide range of variability’s and possible outcomes in them. Again, think of trying to implement a new system for IP or a new system for guidelines adherence in a hospital STATIC or new discharge system possible wide range of possible outcomes STATIC . Usually in complex intervention there is difficulty STATIC of it. I can talk to you briefly, a moment ago talking about the delivery and receipt is different from person to person who is putting it in place or from context to context. 

There can be variability in target population. Physicians may react differently to nurses, to others in the environment. There is sensitivity to features of local context. In other words, again, each unit or microsystem may have its own culture on that. There are? flexibility or handling of the intervention are permitted in complex intervention. Usually we don’t see them as standard or fixed as we might find it in an RCT. 

And then often in complex interventions and trying to understand them we are looking at long, causal change that leads interventions outcome. It is not just one quick cause and effect that we find in a more natural or physiological phenomenon. 

Give some examples of some complex interventions; from a research application are always evaluating the effect of an intervention that might be complex such as self-management program for heart failure patients, internet intervention for home recovery following hip replacement, weight management program for persons in nursing homes. We think that there is often more on the continuing end of our research application. But when you are putting in a whole program where it is teaching self-management, in a healthcare system it becomes what we call complex. 

Program evaluation applications I just have a few here. Professional education training programs, professional education programs a lot of us are involved in those now. I’ve mentioned institution of the clinical guidelines. Which can be whenever you are trying to assess whatever system you are trying to gain an understanding of how the outcomes are actually produced. 

I think when we think of implementation evaluation again, and my understanding of why realist evaluation might be useful when its complex. the complexity is added in implementation side because often we are doing kind of evaluation with multi objectives, multi-agency, multi sites. We are looking at evaluating things when we are seeing wholesale transformation systems or seeking to shift institutional philosophy as well as practice. So our standard RCT approaches is to getting at and understanding of how these things are done have certainly been wanting and the realist evaluation approach offers an alternative that I think has a lot of promise. 

So in a realist evaluation the programs or interventions are essentially inspected and we are going to talk about how this is done for their series of change or their mechanisms of action or of your propositions on how they work. I call that looking for the active ingredient of what produced the change of what you are hypothesizing. The evaluation has the adequacy your proposition to put forth how you think it is going to work in different context. And the end product of this evaluation this inquiry is a better understanding of which ideas work for whom in what circumstances and what respect and why work. I think getting at this mechanism is a really important aspect. 

So to move forward to talk about the method; there is three key terms that we associate with realist evaluation if you are going to talk about it with someone intelligently you need to be able to talk about mechanisms, context and outcome pattern. So again, the mechanism is what it is about the intervention that brings about the effects. So active ingredients or the particular programmatic features that you built in because you decided that those would produce the outcome. 

Context, this is a very traditional definition of context. Those features are the conditions in which the programs and inventions are introduced and are relevant to the particular mechanism. And then the outcome pattern. The intended or unintended consequences of them for intervention. 

So this is kind of the standard way that a research team would start to think then about using realist evaluation is that this is a tool that calls a CMO matrix a context mechanism outcome matrix. And it is a table in which you line up, if you were evaluating a program let’s say the program you wanted to evaluate was institution of political practice guidelines in a particular area. And we might have a number of outcomes associated with that institution that we had hoped to achieve and it might be that there was increased use of the guidelines. There might be that you had a set of patient outcomes resulting from increased use. There could be a whole set of outcomes that you don’t want to look at, air rates or whatever. 

And in for each outcome you put together a mechanism for what you want to look at to see if indeed your proposition this mechanism indeed produced this outcome. So again, let’s just take an example here. If I say that this first outcome is increased use of them I might say that my mechanism is to have a reminder system that came up electronically. And I want to see if – so I would look to find out if people – if reminder systems were available that I might look also if they used them were they were more likely a reminder system people who had access to it were they more likely to be using the clinical practice guidelines. And then there is a set of patient outcomes related to those and some mechanisms there all which might be, again, to the extent to which they use them or – I might have an outcome in terms of documenting the use of clinical practice guidelines. So I might look at the presence of electronic health records that solves the problem for that documentation, actually increase that. 

I have an outcome. I have a mechanism that I proposed for that. It could be that I then have a context that I am going to evaluate that in. These are actually written propositions you write out the mechanism you are going to look at, the outcome you want to achieve and the context. In this case it is three different ones. In contrast, on this matrix what I have done here is I have held the outcome, one outcome let’s say increased use of clinical guidelines. Constant. I hold my mechanism constant. And I change the context. I am going to move it from pediatrics to cardiology to outpatient unit for orthopedics. 

In this matrix I’ve held context and outcome constant that I am going to hypothesize different mechanisms or different features of my program that I built that indeed might affect the outcome. Now you can see how you would do very complicated matrixes where you change context, change mechanisms, test each of them under different conditions. 

So this is an example of one broadly it’s a broad example. This actually turned out in the long run we created in much more detail. This one, this was a situation in which the National VA Quality Scholars program was evaluated by Mark Twain and a team of us and what we wanted to look at was the – in this particular case was the effect of added nurses to the program and it had been all physicians for years we wanted to say so did we achieve some outcomes. So in this case I have used the same outcome down my matrix and the outcome was better improvement and patient outcomes resulting from QI project. So that was a pretty lofty goal that we had. So we said that we thought that you know what, I am going to change, and this is not the one on nurses. This is the overall VA Quality Scholars program. So we said that one mechanism is – is that if the scholars in the program had increased system thinking from the QI team that they would do better projects and have a better outcome. If the projects collected more focused data they would have better outcomes. If the scholars on this project and if the projects of the scholars were done more quickly and things happened faster patients would have better improvement from those making change factor and thus improving outcome. 

We could have used the same mechanism here and also just changed context. We could have tested any mechanism across the sites, there were six sites in the country we could have tested the mechanism across types of learners predocs or post docs. We could have tested that against the background of the faculty at each site. In other words the faculty background as a context makes a difference in terms of our causal path we would have for any of these. 

So what does one actually do after the matrix is built, okay – the matrix becomes a set of working hypothesis. So you illicit and form a theory on which you think the program works. You collect data and I am going to talk a moment about data collection and analysis, which there is a whole reflection stage on that to find out which theories were upheld, which ones are not and then essentially you reformulate, you revise your understanding of the context, matrix outcome configuration. It is a process for one person with a working hypothesis for this set of propositions, the CMO configurations or matrix, collect data. Collecting data is actually done – I will get to that in more detail in a moment – through a set of case comparisons where you choose a place to go in and look at very purposively. We will talk about that in a moment. 

So as I said, the basic methods are to formulate context mechanism patterns. And these are really a lot of fun. It is a lot of head work. You sit down, how we really think this is working. How do we – how did we sometimes you built a feature into a program based on a literature or based on a best guess. Sometimes you work on a working hypothesis of how it works based on hints you are getting from the field on what the real underlying mechanism is that is producing or not producing an outcome if you will. You can produce in such a way to uncover what is really happening there so the outcome may or may not be being achieved but at least you can understand the processes going into that. 

We suggest you look at case studies first where you are achieving your outcome and then you look at being able to uncover the mechanism behind that. Then you would compare it with cases in which they are not achieving the outcome to see if indeed that mechanism is not being carried out in the same way which would reinforce the idea that perhaps that was the cobble path because it wasn’t achieving the outcome where it wasn’t being enacted but it was achieving the outcome. 

Formulating the ____[0:26:54] matrices is a large, large part of the work. I don’t know what percentage you actually assign it, but one wouldn’t want to skimp on that stage. A lot of energy there. 

Devising methods of testing the hypothesis. This is where you would choose your sampling for your cases to go in and collect data from and then you would actually compare the data you collect with your matrix, your hypothesis. 

So the goal is to draw kind of lasting lessons about the change or hypothesis of particular propositions across different context on the basis of the theory of basically limited pilot studies is the word used by Pawson Tilley they used ____[0:27:53] their whole case studies are just small approaches to just trying to build the larger case. So if there is no one big omnibus approach, one big omnibus study but it is a series of constant comparison across cases. Again, I think of Darwin and his bringing in feathers from all over the world and doing constant comparisons to end up with his classifications and of course, he abstracted it eventually to a large, large theory. We can talk in a moment about the levels of abstraction to a particular instance and how that relates to the particular philosophical approach to the research. 

So the focus is on uncovering an underlying mechanism for the purpose of eventually having transferability, general ___-[0:29:00] and large external validity from the case study. 

Case study comparison, what I like to think about it is after I build a CMO matrix, a content mechanism outcome matrix. You think of the first set of propositions I want to study and then I look for a case to be able to study that in. I will choose a context this is the case here. I am going to look at a particular mechanism to prove an outcome. I would choose a second case that I might want to look at or excuse me a second context and again, I might look at the same one in a second context or I could look at a new mechanism and outcome in a second context or first context. Normally what I would do is we would start building these ____[0:30:01] the propositional statements in the matrix the CMO matrix we say how can I best take this proposition and explore whether it holds up or not. 

So what are the data sources for that exploration? The data sources are observation. And those observations can be direct observations. They can be indirect observations that are done, they can be stories like people, they can be people who sit with complex time looking at how people use their time and energy. They can be extremely quantitative or more qualitative to stories. Interviews, they can be in person interviews, they can be self-report interviews through more structured surveys that are done. Pawson Tilley like personal interviews and they actually have an interview method that they suggest where you listen for the very subtle little comments people would make in an interview about how something works. And that interview would be driven by your proposition statement from the matrix you put together, one of the lines in the matrix. You would have a set of questions about how you think it works for them. You listen for them to suggest that is not quite it is really this or that. Or that is what they think we really know it works like this. And it is when they hear it _____[0:31:50] matrix where you uncover the goal is to uncover ___[0:31:57] mechanisms. We start with our hypothesized ones and through iterative comparisons and learning from the system we keep putting forth kind of revised propositional statements about how the system works until we think we are pretty close to what it is really reflecting reality if you will. 

Document analysis is another source of data looking at records and outcomes. Everything from patient data to information about how the organization is working, cross data, personnel time, etc. could be a lot of different sources to do that. 

How do you analyze this data? One of the things we pretty much - ____[0:32:55] how something works by thinking about patterns in it. So you have a working proposition and you are finding truth both empirical, quantitative data from your data sources or qualitative data and usually we start by making at able of that in terms of did the proposition you have hold up or not. It could be as simple as yes or no or it holds up in some way. Depending on the data you could get more – I am going to give you a reference list. Some of the articles will talk about how some of them do scatter thought. Where they have their outcome on one axis and then the mechanism on the other and then they actually plot the extent to which across different contexts or even with over time they found that in different instances they were looking at they found indeed the data matched or didn’t match the mechanism and the outcome. So I am seeing scattered thoughts done, data matrices done. Essentially you are going after pattern making. The more you can add qualitative data with the quantitative data it is probably – Pawson Tilley make a big deal that a lot of the qualitative should be used by you to also go back to revise and refine your proposition to even finer as you uncover slowly or peel back getting closer and closer to reality of how something works. 

So the aim is to draw big conclusions from a small number of studies. I’ve actually found that it isn’t necessarily a small number of studies. It is more like a number of small studies to actually do it. There is an analytic generalization that is made at the end. And this generalization I just want you to know you can do a lot of countless extractions in other words moving further and further on the level of extraction on how the system works. This is head work. This is thinking when you are kind of creating generalization. And a proposition and a theory is a generalization of some sort. You can also move down to pretty particular instances. How this works among any particular group or even individuals your experts what is the mechanism for those experts that make it work. That is one thing I like about this is moving through large generalizations we can make about how something works and also being able to speak to some very particular instances of expertise or of excellence. gain

The methodology is built on fields where case study analysis is used. So just comparative historical methodology of there is a lot of fields I mention natural science is one. So usually as an analysis using a small n, these case studies you don’t need a large end for them. We talked about being comparative. Propulsive sampling is very, very important. If I want to say where is my best example of where the outcomes are being achieved and I have to look for that best example to go study it. Alright? Or if I want to look for an example of how this unfolds in a particular context I need to propulsively sample for that context. I just mentioned here also typological models are used, configurational analysis. This is when you use a ___-[0:37:08] to space or distance in actually being able to kind of map out the use of time in your if time is a factor or other geographical features that might be important to whatever you are studying. 

Molly:Dr. Moore, I apologize for interrupting. Can you please speak up a little bit. 

Dr. Moore:Oh, okay. 

Molly:Thank you. Also, I wanted to – this slide is simply showing what I call the idea that I brought up from simple intervention to complex. And in some ways the RTCs are best when there is a very discreet preferably short toggle chain and the toggle mechanism is pretty well worked out in order to be able to develop predictive knowledge from that. So the simpler the intervention is actually what the RCT is best for I’m suggesting it is best for. I am suggesting it is a complex intervention, realist evaluation can shed a lot of light about what reality is there. So I kind of devised this into what I call predictive knowledge and exploratory knowledge saying that in some ways CQI projects are a way to explore making a change and getting a feel on how we can move forward when we want to go to more predictive stuff we might use something like realist evaluation. 

So a couple of ground clearing issues on this. There is usually no statistical probability sample. Either perspective is important. The systems that we study are often changing as we study them. So that has to be visited by the team and appreciated about the context as one of the contextual features, I think. So again, just to review the process; developing the theories are done by surfacing and articulating and prioritizing the series incarnates that we had in mind when we implemented a program or that we are supposed to create this set of theoretical statements to be tested and how they were developed. 

I have an example here of some theories of change. This is when we were putting in a theory chart for institutional change to institute self-management care practices with patients. The series of change here was that procedures could be changed in the organizations, the settings could be changed. So these were some of the large theories of change on how the institution could change towards the more self-management approach to care. And then from these they were able to put together more finite hypothesis to actually study to what extent did user friendliness – the procedures that were user friendly would actually change the use of self-management by practices or self-management encouragement by practitioners to help patients and then by use of those by patients themselves. Again, this is an example of actually thinking it through and how much work that is. 

I talked a little bit about choosing the cases, having ___-[0:40:52] cases to explore the relative success of mechanisms as it played out in different context. Similarities and differences can be used in choosing your mechanisms and your context. Particularly context. Discrepancies in the series are observed it can be yes or no it works a lot or they needed a more finer scale than yes or no. 

The choice of each successive case to be studied is you propulsively decide what do you want to put that next case in and you say what comes next in this case study. So what is the chain of cases I want to put together or I let unfold depending on what I learned in the last step of it. And then again the last phase is assembling the evidence or process tracking here this is again where we put in intervention into producing outcomes and we tested mechanism one, two, three and collected data about them. The actual processes are tracked over time using both quantitative and qualitative data. This study just says again where I might move to different context for analysis for different outcomes. 

I put these slides in as it can help a group think through. So I had one last example I wanted to give. This is the example where we added interprofessional learning into the VA quality scholars program. So we wanted to know what are the mechanisms through which interprofessional program elements produce their impact. First we had to know what impact we were looking for and then how that happens and what was the interprofessional part of it that happens. Then what characteristics of the local environment contributed to better interprofessional learning. That sounds like a tough thing to do, to go after. On the other hand if we could shed light on what makes interprofessionalism work and it would tell us about the components that would help put in better teaching programs together, approach people, etc. 

So we put together some program outcomes here that we had we have to have outcomes. We said by making a program interprofessional we thought that we would have better patient outcome. In this case it was from quality improvement projects. We would have faster cycles of improvement if it was interdisciplinary. More research publications, etc. I won’t read them all, but increased ____[0:43:50] active members for the goal because of the literature showing that opt ins are not members of teams. 

We put together a set of mechanisms. Increased sense of application of increased knowledge of health by different members of the QI team. Large numbers of tools, a larger set of tools if only one discipline did it. It would be increased systems thinking by having people who had different views of the system present because of their discipline. And then lastly, increased of contextual variables would actually be taken into account when they did do a project. Again, thinking that people with different disciplines would think of different things to take into account. They wouldn’t ____[0:44:37]. 

Then we did a group of context. Does our professional learning differ by site, by the type of learner, by a mentor’s background. Okay? ____[0:44:52] advisors at the site. An example of our matrix. This is where we put together some of our mechanisms. And again, in this case it is all the same outcome. This is the one I showed you before. These are some possible contexts that we would use. 

I am not going to show you the results of that. We actually did that evaluation and a summary of that can be obtained but that is a pretty long summary to do. I want to end on just talking about some challenges in using realist evaluation. My guess is these are some of your questions you may be calling in. I just wanted to say we have found in the literature of people who have been using it talk a lot about the difference in distinguishing the difference between context and mechanism. Because sometimes the same feature could be either one. So it depends on the proposition at the moment. For instance, the presence of electronic health records sometimes could be the context whether they have one or doesn’t. Sometimes the use of an electronic health record can be a mechanism. So that is just an example. You know, I usually when a team just has trouble doing that I said well, let’s just study it both ways. First we will propose electronic health records as a health mechanism and another and then we will look at it as a context and see how that might change our data, our conclusions at the end. 

Also, I think number two here on challenges in knowing what the end point is or when to close. You can keep finding cases and keep doing more propositions. So when does a group come in and say I think I learned a lot to put forth our generalization or our if you will our summary of what we think we have learned, our enhanced understanding here. But I think it is tough for a team to do that. You kind of get into this with case comparison sort of thing. 

It can be time consuming. Building your proposition statements for the CMO matrix should be a group effort and it takes some time deciding on what are the best case comparisons to make and then analyzing the tables and the matrices you end up with in terms of if the data supported the proposition or not. 

There is some criticism in the literature in number four here, in realist evaluation studies being replicated. So Pawson Tilley would say in complex interventions they are pretty hard when context is so different and things ____[0:47:52] and time changes things and people involved would make it very, very hard to replicate. On the other hand, I guess with my background I believe that – I hope that I come up with generalizations are such that at least the results can be useful if not the exact study replicated. 

And then lastly, how does realist evaluation differ from traditional mediator/moderator analysis? That was where my head was because I am a regression queen. And actually, it is different. Part of it is propulsive sampling. Part of it is starting with the exemplar cases where your hypothesis can be tested in terms of uncovering the real world. And quite frankly in most of our mediator/moderator analysis we prior have decided exactly which ones are going to be in the study and if a new one arose along the way we can’t do anything about it. Okay? We have ____[0:48:55] we are collecting. And realist evaluation you can add a new proposition and study something for its mechanism, if you will, or its context. 

I want to end just suggesting that it is possible that a ___-[0:49:15] standard appropriate for the evaluation of complex intervention could be realist evaluation. Thank you. 

Molly:Thank you very much, Dr. Moore. We do have a number of questions that have come in and before we get into the audience questions I just want to make a brief announcement. There have been several requests for the references to be flagged and I mentioned that I will be receiving them and posting them with the archives recording in the next few days. 

Dr. Moore:I just wanted to say on the reference list I have everything from message papers to several where they had used the method and some great examples in some of those papers. 

Molly:Thank you. The first question that came in; does this always involve case studies when intervention was deployed or can one take a neutral experiment approach where the mechanisms are dictated by the site? 

Dr. Moore:Well there has to be some – it has to be clear that you are looking for essentially some toggle effect. Something was introduced or is happening where you are going to be able to put together the CMO matrix, okay? So that you can actually say these are the mechanisms we propose. And, of course, one would want a series of propositions about how it works. I do want to say that there have been, people use this method more recently, Pawson Tilley did not start with it all in mind. They are using it for the reviews of literature so they say we hypothesize that something works and we want to use the literature as our cases, if you will, to be able to see if the literature upheld that relationship. The difficulty with that is the literature is usually based on random case trials when they do this. It is only going to cover the variables that were in that study. And so if you will, some real underlying mechanisms may not be uncovered at all. It is one way to see if the literature ____[0:51:32]. 

To answer the question a group could put forth something it needs to be in the context of trying to uncover a causal chain. 

Molly:Alright. Thank you for that response. We do have about eight more pending questions. The next one; do you consider the process used by Charles Darwin to formulate the theory of evolution a prime example of realist evaluation? 

Dr. Moore:You know, I don’t know if anyone has ever written about that that way. I know I am going to get real philosophical here. I do think that depending on what came in, from my understanding I read several books about Darwin and from Darwin of that science at that time and the idea of they would get a new piece of information and they would then formulate a new hypothesis. And then the next thing that came in is whether it was consistent with that or it was something that was so inconsistent they would have to try a new hypothesis. So there is a lot of iterative stuff. It is not exactly it is not I wouldn’t call it a perfect example of it. I would kind of say the methods of philosophy behind it are very consistent. 

Molly:Thank you for that reply. The next question is referring to the querying issues slide and the question is please explain by “usually no statistical or probability sampling” on the ground clearing slide issue. 

Dr. Moore:That is really an important feature because probability sampling would be essentially that you are not – you are not going after the example of what the exemplars are usually to where we start. So probability sampling you would also have to decide how many you want to do. You would have to decide a priority that you already knew exactly what was there and this was or wasn’t a good example to even get it in your pool. Propulsive sampling. We say for this purpose and to explore this particular proposition we want to compare three of the best places that we see this and it doesn’t matter that it is a probability sample. You would do it based on feasibility, ease on doing it. The emphasis is not there at all when you read the book on it. There are times when you could but it is usually a hallmark to think about propulsive. 

Molly:Great. Thank you for that reply. The next question we have it would be helpful – sorry. Also a request for the VAQS interprofessional evaluation summary to that query I did respond that they could contact you directly. Is that okay? 

Dr. Moore:Yea. I have to talk to my colleague to explain about that. I am pretty sure we don’t mind sharing what we have learned from that. And by the way, that program was evaluated using more than just this method. There were you know, we triangulated method. Realist evaluation was just one approach. 

Molly:Dually noted. Thank you. The next question we have is can you provide more detail on how your approach focuses on generability, general ___-[0:55:39] ability and external validity. 

Dr. Moore:Oh how the focus is you said? I’m sorry, I missed a word in your question. 

Molly:No problem. Can you provide more detail on how your approach focuses on transferability, general liability and external validity? 

Dr. Moore:Well the general livability would be and external validity is just the idea of how well something matches the real world and is useful to them. So the more we can drop the idea of controlling for things because they are messy the more we can put forward a number of hypothesis in the goal of continually getting closer and closer to a real mechanism. To me the idea is to try and match reality as close to which we can. And I would just say that most of the time we are not very close and complex intervention evaluations we really don’t understand those ___-[0:56:56] paths. And so external validity would be the extent to which it matches reality and of course transferability if we attested on a lot of context. We are more likely to understand where it can be transferred and where it can’t. 

Molly:Thank you for that response. We do have two more pending questions. And the first one, how do you recommend dealing with levels in a really realist evaluation of a complex multi-level intervention. Different levels might include individual, project team, clinical micro disturbance, facility, region, etc. In particular, how do you account for the interaction of context mechanism outcome patterns at one level with context mechanism outcome patterns at another level? 

Dr. Moore:That would be fun to do. Certainly you could create propositional statements or hypothesis to be tested at any of those I would suggest a team might say let’s deal at this level and come up with what we think is happening here. And then we could do that at each level. To go across levels I don’t know. I am trying to think of whether you could say does this propositional statement hold out in the context of another one that – an uphill hypothesis at another level. I would make what is happening at that level be a context might be one. I don’t know. That would be really interesting. 

Molly:Thank you. 

Dr. Moore:I’m not sure that’s it. 

Molly:We do have a quick follow-up to a question asked a few moments ago. Can you please define exemplar? 

Dr. Moore:Well, for me one of the ideas for example, are you receiving the outcome that you expected and I think – and I said this is a starting point. I think sometimes you will choose a place where it is not working at all to understand the mechanisms there and how it might differ from those in the exemplar. To me the exemplar is just those places you think you are achieving the outcomes you want would be my first definition of it you could probably put other attach other constraints to that. I want to achieve a particular – I might want to combine a couple of outcomes. Am I getting the patient outcome and the outcome of staff being happy. That is my real exemplar perhaps, where I achieve a couple of things. Again, that would be contextual at the time of what the group wants to do. A lot of discussion about what you are about. ___-[1:00:04] everybody writes their own proposition statements and then we put them together and just be ___[1:00:10] or don’t’ as a way to start generating the discussion. 

And sometimes ___[1:00:17] had overt outcomes when they started and they had some overt features put in that are natural, natural ____[1:00:28] for a propositional statement. 

Molly:Thank you. We do have just about two minutes left so we have one comment that came in and then I would like you to have an opportunity to make concluding comments. The comment from ___[1:00:44] is implementation science may be a misleading term and that performance improvement is more of an art than a science. I think that realist evaluation is an important step forward in improving the quality of evidence to support the option of innovations. However, it will never achieve conclusion with the general livability and certainty of Newtonian physics. I hope it doesn’t lead to an elitist view of initiating organizational change that is contrary to most management theories about how to do it and can demoralize people with operational experience on the front lines. 

Dr. Moore:My comment on that is that I would think it is the opposite. If there are a set of managerial theories in place and we are able to use realist evaluation to show that indeed those theories are being upheld I would just suggest that I ___-[1:01:43] don’t have this management thing all that down or how to produce change or have the outcomes we want. I don’t know, I see it completely as complimentary to most other methods. In fact, I think in some cases it could be a superior method to use over at RCT or in observational studies that isn’t as strongly rooted in a priority thinking about uncovering the mechanism. That is that. 

Molly:Thank you. I appreciate it. That is why we have the interaction with the audience. Several people did ask for a copy of the slides. Please note, it is in the reminder email that you received. I sent each a personal link and you will receive and email to follow-up the session with a link to the handouts. With that I would like to allow Dr. Moore to make any concluding comments. 

Dr. Moore:I would just suggest that you start small. If you are interested in this particular approach look at the readings we have you will start to get a feel of who is using this in the country. You can start with a very small program which you want to evaluate and start getting more comfortable with the methods involved. 

Molly:Thank you, excellent. I want to thank you very much for taking the time to present your expertise for us. It was very helpful. And our audience did stay very engaged throughout. And our audience members lead discussion please wait a moment or two and a survey will pop up on our web browser. Just a few short questions and we do take all of your feedback into account. So please relay your opinions back to us. Thank you again, Dr. Moore. And thank you to our audience for joining us. This will formally conclude today’s HSR&D presentation. 

Dr. Moore:Thank you. 

Page 1 of 16