**Transcript of Cyberseminar**

**Enhancing Implementation Science**

**Advanced Program Session 3- Measurement Issues in Implementation Science**

**Presenters: Cara C. Lewis, PhD, and Kate A. Comtois, PhD, MPH**

**November 1, 2012**

**Advanced Program Session 3- Measurement Issues in Implementation Science**

**November 1, 2012**

Moderator:We are just about at the top of hour. So I would like to provide the introduction for our speakers. Today we have Dr. Cara Lewis who is a Clinical Assistant Professor in the Department of Psychology and Brain Sciences at Indiana University. And joining her is Dr. Kate Comtois, Associate Professor in the Department of Psychiatry and Behavioral Sciences and an Adjunct Associate Professor at University of Washington Medical Center. So I would like to thank them for joining us today. And, Cara, are you ready to share your screen? Sorry about that. I will turn it over to you now. You should see a popup that says “show my screen.” Just click on that. Great, thank you.

Dr. Cara Lewis:Thank you. It is a pleasure to be here today to talk to you about Measurement Issues and Implementation Science. Before jumping in I wanted to acknowledge my collaborators, Ruben Martinez and Kate Comtois, who are both on the line today.

I should note that there are minor changes to this slide, just a condensing to be sure that there is ample time for question and answers. 

So for the agenda today I would like to highlight the importance of measurement and dissemination in Implementation Science. And I will be talking about ten key measurement issues implicated in D&I that we should consider carefully over the course of the hour today. I would also like to present preliminary results of the Seattle Implementation Research Conference Series systematic review of instruments. And then, as I mentioned, have time for questions and answers. 

So in 1964 Siegel said, “Science is measurement.” And I think metrics clarification in 1988 was an important one that we will address carefully today and that was that measurement is not necessarily science. We can probably all agree that it should be so, at least in the case of D&I because measurement secures the outcomes of our research such that they reflect and enhance understanding of the individual and clinical change process. So in essence strong measurements with good psychometrics allows us to confidently interpret our findings as valid.

And I like this quote by Achenbach, who in reference to the evidence-based treatment movement said, “Without evidence-based assessment, evidence-based treatment may be like a magnificent house with no foundation.” And I think that Implementation Science needs to prioritize the importance of evidence-based assessment in the face of this praiseworthy rush to get evidence-based treatments implemented to avoid building our field on a shifty foundation.

So though it is beyond the scope of the talk today to talk in detail of the role of design, it is important to acknowledge that design that maximizes internal validity is critical to confidently interpret the data that we generate. So careful design will allow us to understand mediators as implementation and rule out their variable problems and things like that. In addition, measurement allows for evaluation of our D&I efforts and sets the stage for comparative effectiveness. This is an important measurement for enabling cross-study comparisons that will facilitate building a D&I knowledge base. And, in particular, measurement allows for the understanding of mechanisms and approaches responsible for change to ultimately improve public health impact.

I think we all know that successful D&I involves a complex package of interventions to achieve clinical practice change. So measurement will hopefully allow us to gain understanding of the core effective components and perhaps even incremental utility of specific components over others.

Moderator:Cara, I am sorry to interrupt. Can I ask you to speak up just a little bit?

Dr. Cara Lewis:Yes.

Moderator:Thanks.

Dr. Cara Lewis:So unfortunately, at least as of 2006, Grimshaw and colleagues identified that D&I measurement issues seem to be getting in the way of us effectively evaluating our efforts. And so, in particular, ten key measurement issues in D&I will be covered today, not necessarily listed in their order of importance. These issues include: discussing the role of psychometric properties, talking about what seems to be a bit of an overwhelming literature on frameworks, construct identification, and differences in definitions. And then we will consider together what to measure when and at what level, followed by prioritizing the need for communication and instrumentation with respect to specific constructs, and acknowledging that this praiseworthy rush to implement evidence-based treatments has challenged teams to create what we are calling “homegrown” instruments, followed by discussing the role of instrument specificity and how adaptation affects our findings. And then as the seventh key issue, we will be talking about shared method bias and the pitfalls of relying on self-reports. The eighth issue, talking about the role of mixed methods and how to leverage that, followed by a discussion of practicality versus burdensomeness, which is really relevant for our field, and then just the need for decision-making tools and what is emerging from the literature and the researchers in that regard.

So I wanted to point out at this time that there is a handout, I think, made available to you before we started. This includes a listing of each of the issues that I have identified here along with a careful breakdown of potential solutions or directions across four domains: so theoretical, empirical, practical, and as well in terms of psychometrics. Hopefully this will help you follow along today and leave you with a concise summary to review. 

This is just a brief aside that I would like to make to provide you with some context for some of the data that I will be discussing today. So as co-director alongside Dr. Kate Comtois, who is on the line with us, of the Seattle Implementation Research Conference Series, an NIMH-funded conference series, we have as our mission to promote rigorous implementation research methodology. Our core team identified measurement issues very early on as one of the top priorities for our team to try to tackle. And so we have embarked on a systematic review of dissemination and implementation implicated instruments. We have identified approximately 450 to date. And, as I mentioned, this is just some context for you to understand some of the preliminary results that I will present to you today that have emerged from this work.

So in terms of a first issue, and arguably perhaps one of the most important ones, has to do with psychometric properties. So thinking about reliability and validity as the foundation for useful and accurate measurement – that these components, these properties, are necessary in our instruments to move the field of D&I forward as they both relate to the capacity to interpret scores. And I think we know that reliability is necessary but not a sufficient component to achieve validity or valid results. And we can think of reliability together as the reproducibility or consistency of scores from one assessment to another. In terms of validity, just to make sure we are all on the same page, it is being that which addresses whether interpretations are well-grounded and meaningful. Construct validity in particular, having to do with the degree to which the instrument measures what it is purported to measure.

And so there are five sources, metrics proposes, that provide us with evidence to support this kind of validity. And we will actually talk in detail I think today about several of these. Content, for instance, has to do with construct definitions, the instrument’s intended purpose, the process for developing and selecting items, the specific wording of items, qualifications of item reviewers and writers. These are all important things that we consider an appropriate test in instrument development. Response process: we might look for things such as the methods for scoring and understanding how raters approach ratings and even having respondents think aloud, to understand how they are making sense of particular questions. We also look at internal structure, not only with respect to reliability, but also factor analyses so that if a construct is thought to be [inaudible] in a uni-dimensional way by the instrument, we can actually employ factor analysis to assess whether or not that is the case. We also will look toward relation to other variables with respect to construct validity, looking at correlations, or lack thereof, that should support how theory makes sense of this underlying construct. As well as considering consequences or unexpected biases with particular populations in terms of how the instrument is working.

And also just briefly to touch on criterion-related validity, which is the degree to which an instrument correlates with other instruments’ scores, typically gold standard ones that exist in the field. Unfortunately, and this is specific to D&I here, the gold standards are not necessarily available for a large majority of the constructs that we seek to measure. And I think in particular predictive validity is something we are very interested in, which is the degree to which instrument scores correlate with scores on established instruments administered at some point in the future. And predictive validity will help us understand which constructs in each stage of an implementation predict constructs or outcomes in subsequent or later stages.

So to summarize this first issue, psychometric properties can perhaps be thought of as the foundation of our measurements and they arguably will be the most important issue that we discuss today. But that does not make these other issues less complicated. In terms of issue number two, this has to do with the plethora of frameworks, constructs, and the definitions that exist. Fortunately, we can all benefit from the excellent review that was recently published by Tabak and colleagues who report that 61 model series and frameworks exist. And so in their review, they rated each of the models according to their “D” versus “I”-ness – so whether they were dissemination-focused versus implementation-focused, as well as according to the socioeconomic framework level that each of the models considered in their framework.

The evidence synthesis results, just in brief – this is a highlight here – suggests that broad models are perhaps the most common and they tend to be “D” focused or dissemination focused. And they define broad as containing constructs that are more loosely outlined or defined, allowing researchers greater flexibility to apply to a wide range of D&I activities and that the majority – 25 of the models – were categorized as a 3 on this dimension of 1 to 5 – 1 being broad, 5 being operational.

And just to note how I am sort of making sense of that finding, though this allows you a greater flexibility and application across studies, it does challenge us a little bit in terms of understanding the validity, which hinges on the constructs requiring clear definition as a first step. So it is something we really need to think about as we move forward.

Another one of their findings – their results from this synthesis – indicate that the majority of the models are doing a nice job acknowledging the role of community and organization. However, only eight of them are addressing the policy level, which we know plays an important role in our D&I work and we will be talking at a more in-depth level about the level of analysis in a few slides.

I think it is worth mentioning that one of the models that are available to D&I researchers – the consolidated framework for implementation research or the CFIR. So what you see here are the domains and constructs that comprise the CFIR, which is a meta-theoretical framework that was generated to address the lack of uniformity in the D&I theory landscape that minimizes overlaps and redundancies in available frameworks, separates ideas that have been formerly seen as inextricable, and strives to create a uniform language for the domains and constructs in D&I. 

To achieve these goals the CFIR was empirically derived via a snowball sampling technique whereby Laura Damschroder and colleagues attempted to identify all known implementation frameworks and models at the time, culminating in these five domains: the Characteristics of Individuals, the Inner Setting, Intervention Characteristics, Outer Setting, as well as Process. Across these five domains, 26 possible measurable factors emerged.

And despite the fairly comprehensive nature of the CFIR, clearly defined outcomes for D&I seem to be missing. And to address this limitation, D&I researchers might simultaneously consider a second framework. But according to Proctor and colleagues, delineating implementation outcomes to specifically the isolation and concrete operationalization of implementation outcomes, separate from service and client outcomes was a unique and important addition to the literature. And this isolation may be critical in future research seeking to understand the temporal relations between constructs. And looping back to the project that we have been doing research, these are the two frameworks that we merged to try to conduct our review.

So even with these helpful frameworks, what we are seeing in the field is inconsistent and sometimes even lack of defining constructs. And some of the leaders in the field are suggesting a need for consensual common language. And, again, coming back to this issue, that clear definitions are required for our constructs to insure the validity. And I think a nice effort toward resolution here is the Consolidated Framework for Implementation Research Wiki, which has as its goal to provide online collaborative space to refine and establish terms and definitions related to D&I. Also to promote consistent use of these terms and definitions and to provide a foundation on which a knowledge-base of findings related to implementation can be developed.

So even with shared definitions though we see inconsistent identification and evaluation of constructs which tends to limit our cross-study comparison. I think here is a nice example that I have on this slide where synonyms might be what these terms are but they also might be distinct constructs. And until we carefully assess this, it is difficult to determine which is which. So appropriateness being one of the implementation outcomes, we have seen folks use the term Perceived Fit, Fitness, Relevance, Compatibility, Suitability, Usefulness, Practicality, Applicability sometimes, as I mentioned, as a synonym. So if we were to measure these constructs together in a study at the same stage and they changed together in the same stage, then they really might be the same thing. But if they do not, then we might think of them as independent constructs where they have their own evaluation. But I think it is really important for us to start to make these determinations because instruments that use different language are responded to differently by the test taker and these important nuances are things that we should attend to hopefully with future work. 

If nothing else, I think that the field would stand to benefit by following a recommendation that Proctor and Brownson make in their D&I book chapter on measurement issues. Specifically, it was identified as important to present a model as the relation between constructs that you intend to evaluate in a D&I project. So this is an example here of a program that may be only mildly acceptable to stakeholders because it is seen as too costly to sustain. So implementation success would be low as a function of effectiveness being high, acceptability being moderate, and cost high, and sustainability low. 

Now we come to this issue of when do we measure these potentially important constructs? The current state of this field suggests that these are not terribly consistent in terms of timeframe as to when constructs are being measured. And others are not reporting during what stage of the implementation instruments are being administered. Our team has come to think about CFIR constructs as predictors, moderators, and mediators of an implementation. And it perhaps might seem more obvious that implementation outcomes would serve as dependent variables. So important to consider that they are implicated at several stages throughout the implementation and not just at an end, for instance.

Another commonly used framework – the RE-AIM constructs have been thought of as mediators. And more recently Greg Aarons and colleagues, as well as Abe Wandersman and colleagues have presented models that help us to know what to measure when. So these are going to be important frameworks for us to look to for guidance as we are planning our implementation. 

And we can take that question one step further and ask what do we measure when and at what level? So many of the conceptual frameworks have intentionally laid out the levels of analysis at which D&I constructs are relevant ranging from the individual, through the organization, to the community, the system, and policy. And what I would like to do is to try to bring all of these issues together -- the what to measure, when, and at what level. So what is on your screen now is not meant to be the 60-second model for D&I, rather this is something that our team has come up with as a way to communicate with you today about how to think through these different issues that I just raised captured in that question of what do I measure, when, and at what level?

So this model here depicts a fairly parsimonious stage model that comes from Greg Aarons and colleagues conceptual model that I just referenced. So what you see is exploration as the first stage, adoption, decision, preparation, followed by active implementation and sustainment. The circular nature of the way we have depicted the implementation stages here is to remind us that D&I projects rarely are linear in nature. In addition what is captured by this graphic is an abbreviated version of the levels that Tabak and colleagues views in rating their models, separating out the individual, the organization, and community, so the individual being the innermost ring outside of that, which is the organization, which lives inside the community. The policy level has been left off just for the sake of parsimony here. 

So I am hoping that initial model was not too complicated because it’s about to get a little bit more. What you are looking at here is the addition of a single construct. So we have acceptability being reflected over early in the adoption decision preparation stage and this was a suggestion that Proctor and Brownson made in their chapter that acceptability is important here at this stage of the implementation. But in addition, also thought to be implicated important here during the active implementation or penetration stage, as well as here in the sustainment stage of an implementation. And so in brief what this is meant to capture is that it is absolutely critical to remember that constructs vary in their importance and salience across stages.

Now one more issue seems to be quite important here and it is that, unfortunately, stakeholders do not necessarily neatly align in terms of their stage of the D&I process. So this is a slightly altered figure that is a hypothetical representation of maybe like a grassroots implementation effort whereby the community has engaged in exploration first. After which we see an individual engaging in exploration, followed maybe by the organization. This is a way to show how we need to be thinking not only about at which stage our construct is salient, but at which stage are our levels of stakeholders at within the implementation process. So, hopefully, this figure which for us is merely a communication tool, can help us think through what should be measured when and at what level.

So I should say in looking at this slide – this is also some preliminary data from the Seattle Implementation Research Conference Series or the SIRC Instrument Review. One of our core functions of SIRC is to promote communication. I bring this up here because we have seen some pretty frequent, unnecessarily redundant efforts in instrumentation. So over on the right-hand side of your screen, things like culture and climate have several instruments that are used to tap into those constructs. At the same time we see that over 50% of the constructs included in our review – again, this is the frameworks we looked at were the CFIR and implementation outcomes – the majority fall into this high priority category where fewer than five instruments exist for these particular constructs. We as a team have prioritized collecting “in development” instruments so that we might connect teams looking to measure similar constructs so that they actually might benefit from one another’s work and maybe even work together to establish psychometric properties on instruments that are newer. 

And we do this because, as I mentioned at the start of today, there seems to be an influx in homegrown instruments. And we define homegrown loosely as those developed in haste without systematically using theory, not engaging in necessary steps of appropriate instrument development, and notably without conducting tests of psychometrics. And indeed actually – I am not sure if this is surprising or not – but when we looked at the implementation outcomes and we included six in our review, we found 92 instruments so far, and 41% of those would fit within this definition of homegrown. 

And so by nature of this definition this means that these instruments have not gone through, been vetted through, the perhaps appropriately characterized stages of instrument development. So I will briefly walk through what we mean by that. And this is indeed a very laborious process. It is not surprising. I am not placing blame here. It is really challenging to engage with this, but this is what we might expect through strong, systematic instrument development – that the initial item generation phases folks might borrow from related instruments with the psychometrics, in addition to reviewing literature relevant to constructs, discussing what is in a working group, subjecting the items to expert review, and establishing a rating scheme, followed by careful piloting of the instrument where suggestions would be solicited for modification, the item pool itself would be refined and narrowed, and a second expert review might occur.

And finally, but perhaps most importantly, establishing the psychometrics is key. So researchers might wish to conduct an exploratory factor analysis on random halves of a large sample or doing the EFA followed by a confirmatory factor analysis to assess structural validity as well as examine internal consistency, test/retest reliability, and also have the opportunity to evaluate the instrument with respect to already established instruments that assess convergent, divergent, and concurrent validity.

Related to this, moving on to measurement issue number six, we found that several constructs are necessarily specific to intervention or population or setting. So just as some examples here: an intervention-specific instrument seems to be necessary when looking at fidelity, that we see a lot of the client outcome instruments need to be population-specific, and that understanding patient needs and resources that those instruments tend to be very setting-specific. When we get really specific like this, it is important because it allows us to be more sensitive to what we are trying to understand and test. Unfortunately, the flipside of that is it makes it challenging to compare across studies when we get more and more specific with our instruments.

And actually one of the things that we have noticed most when we find these specific instruments is that research teams are adapting them and it is having two negative kinds of outcomes on the field. That they are adapting them in ways that affect their psychometrics. So it is unclear whether they, you know, remain valid and reliable. But also we are seeing very frequently actually that folks are adapting instruments and not reporting how it is being adapted.

So it is things like psychometrics such as structural validity cannot be assessed because of sample size issues. We are sort of recommending that if nothing else it would just be really important to report how those instruments are being modified.

And issue number seven related to Shared Method Bias and Pitfalls of Relying on Self-Reports – I am not sure – this might be an issue that flies a bit under the radar for some people. But I wanted to discuss it alongside the Pitfalls of Relying on Self-Reports just so that we can keep it in mind and figure out good ways of addressing this as a field. Shared Method Bias results from providing common cues to an individual that influence what’s retrieved from memory and thereby influences the correlation between measures of the predictor and the criterion variables. So in general just being very thoughtful and planful from whom which information should be gathered, thinking carefully about prioritizing the predictor or the criterion is being measured with maybe a different method or [inaudible]. And this would just encourage us to capitalize on opportunities for direct observation or independent assessor as a way of measuring something, as opposed to relying simply on self reports. And a by-product of doing things like that would mean that we might minimize the burden of assessments on certain individuals that are involved in our implementation process.

So it is important to remember that Self-Report tends to be less accurate for particular D&I constructs such as adherence. And then in addition Self-Report provides a range – a restricted range, I should say – of content. As other limitations, Self-Report can be influenced by intentional false reporting or presentation bias. It is also subject to inattentive responding as well as cognitive or memory limits. And unfortunately it might also receive differential responding due to unintentional item ambiguity. These are just important things to remember, and actually sets the stage for us considering what is the role of Mixed Methods in D&I? 

As we think about this, Mixed Methods is often thought of as a way to provide better understanding than either approach alone, so qualitative analysis might be really helpful in exploration and gaining a deeper understanding of issues as related to D*&*I, whereas quantitative analysis might help with testing our processes. There are a range of Mixed Method designs from simple to complex that might be more or less useful dependent on the stage of implementation. And this comes from a nice article by Palinkas and colleagues. And I will briefly summarize some of the other things that they said about Mixed Methods. They have indicated five reasons for Mixed Method use and it might be to understand process, to simultaneously conduct exploratory and confirmatory research, to examine context in addition to content, to incorporate consumer perspective, and to compensate for each kind of method and achieve balance. So I will point you to their article if you are curious about thinking more carefully about involving Mixed Methods in a planful way. 

I think we can all appreciate in our field in particular the challenge of balancing practicality versus burdensomeness. Our work is happening in the community and is meant to stay in the community and there are real measurement issues for us to consider. So cost – some of the really strong instruments that are available are available at a cost that is outside of the parameters of certain projects. Accessibility of instruments is also an important thing for us to think about. Are these in the public domain or not? We are contacting all of the authors of instruments we have located in hopes that there will be a willingness to make these available, but that is not always the case or possible. Also in terms of practicality, thinking about the length of an instrument which tends to increase the burden on our stakeholders who are often just very busy and taxed by their primary responsibilities. We also want to think about how user friendly the different instruments are. Are they at a reading level that is accessible and that kind of thing? How applicable they are to individuals involved and is the scoring scheme something that is accessible and doable? 

And these are all practicality issues that a parallel project, the Grid-Enabled Measures Project, is prioritizing, and I’ll talk more in just a moment about what they are up to.

I think one of the things that stands out from all that I have discussed so far is that there is a real need for some decision-making tools for the field of D&I and for stakeholders to engage in measurement systematically. Some of the sort of obvious general things that come out include considering theory and previous research and psychometrics and the practicality as we try to make decisions about which instrument to use. And this is really important. As I mentioned earlier, there are some constructs, such as culture that has in particular 21 at least instruments available to assess it. Three of which actually have the same name by different research groups. So it is really tough but complicated area, I think, for stakeholders to figure out what to use when. 

So I mentioned on the previous slide that the GEM project – and I have a link on the previous slide as well. This is a team – a project that was initiated and co-developed by the Cancer Research Network, Cancer Communications Center at Kaiser Permanente Colorado, and the National Cancer Institute’s Division of Cancer Control and Population Services. And they looked to identify the outcomes and associated measures of the evidence-base to inform D&I research and practice. Its purpose is to create a growing and evolving resource for standardized vetted D&I measures that can lead to comparable data sets and facilitate collaboration and comparison across disciplines, projects, content areas, and regions. So a lot of the issues we discussed today, this particular group is really invested in helping us figure out. 

And as a parallel effort, I wanted to share some of the preliminary results from the Seattle Implementation Research Conference Systematic Review of Instruments in the remaining minutes that we have. And that should leave lots of time for questions and answers. 

So in brief, the goals of the SIRC Instrument Review include conducting a systematic review of the CFIR and implementation outcome framework constructs to identify instruments that have these constructs. And once the instruments themselves are identified, we aim to systematically review each instrument to identify associated literature that has information on its psychometric properties and usability.

The second goal of the SIRC Instrument Review was to create an evidence-based assessment rating criteria specific to our dissemination implementation projects that would be applied to each instrument by a task force of volunteer raters.

And then finally one by-product of this effort might be that a consensus battery for the field emerges, that the top-rated instruments might be thought of as both go-to’s for different implementation projects. And that the product – the instrument repository – would be made freely available to SIRC members on our website, which I will walk you through in just a moment.

I should say that it was quickly evident early on in our review that we needed to build capacity if we were going to finish this project in my lifetime. So as such we welcomed excellent partnerships. These are core research teams at the Universities of Montana, North Carolina, Indiana, and Washington, as well as a new site developing at the Toronto Hospital for Sick Children in Canada. We also welcome volunteers to serve on our task force to rate the quality of the instruments and we have approximately 60 task force members currently around the world who have agreed to rate at a minimum five instruments.

And in line with the top priority measurement issue that we talked about today, we have generated an evidence-based assessment criteria drawing on the work of Hunsley and Mash, as well as Terwee and colleagues, who have done really nice work in this area. This evidence-based assessment criteria was developed iteratively by our group over the course of one and a half years, including eliciting expert feedback from both test developers and D&I-focused researchers. We had input from 60-odd people with that respect. We have done highlighting of the rating system, revising it, assessing its reliability, and so on. And have now established – it seems to be fairly easy to use – rating system for assessing instrument quality with very specific anchors that maximize reliability of ratings through limiting subjectivity. And we have used a two-phased review process, which seemed to be optimally practical for our very busy volunteer task force numbers.

The core criteria that we have included in this evidence-based assessment rating system are reliability and validity because we think that without either of these properties that it would not be in our best interest to try to make sense of what the instrument is actually telling us. We have also included usability in our core criteria. This is a simple measure for us. We are just thinking about the number of items – thinking about test links in that way. We have included this as a core dimension to rate instruments on so that we can empirically examine the balance between reliability, validity, and usability across 450 instruments that we have to rate.

So if an instrument scores at least good, which is a score of three on a four-point scale, on the core criteria, then the additional criteria are applied at which time we look at things like norms and responsiveness. 

I thought I would just walk you through a glimpse at our website now since we have a few minutes left. I am going to pull over a screen. And what you are looking at here is a construct-specific page. All of the construct pages will be laid out similarly. This is the acceptability construct. So at the top of the page you see the definition. And we pull our definitions from the respective frameworks from which the construct was generated. So in this case the definition comes from Proctor and colleagues’ paper. And then you have the instruments listed in alphabetical order along with the authors of the original paper, the year the instrument was developed, and a single sentence summary of the instrument, including what it purportedly measures, the level of stakeholder it is meant for, and the number of items. And then by clicking on the hyperlink to an instrument you are taken to a specific instrument page. At the top of the page is the same summary as on the previous page. And now you have a hyperlink to the main article abstract, followed by some sample items, and a graph that summarizes the EBA criteria. And next you have the reference list of all the literature that we have been able to identify through the process of doing that review. And then at the bottom of the page is where you see a head-to-head comparison of the evidence-based assessment criteria ratings for instruments purportedly tapping that same construct. 

So each bar on this graph represents a unique instrument identified by its acronym which you will find at the top of each instrument page spelling that out. And then each bar has the scores from the ratings. So, recall there are four core criteria: internal consistency, structural validity, predictive validity, and then we are curious about usability. And they are rated on a zero to four scale with anchors listed below from zero to none or not yet available, all the way to four which is thought of as excellent. And there are qualitative anchors that are domain-specific that we use in our ratings, but they are listed on the EBA page.

So just to finish up here, I think it is helpful if we look at the AARP, which did quite well on the psychometric properties as well as usability, but does not yet have predictive validity established, so it does not go past the base rating. And then the EBPAS or the evidence-based practice attitude scale that Greg Aarons developed is one that is scoring quite highly and makes it past the base rating and tells us a lot more about the psychometric properties there. 

We just hope this kind of summary data and graph would be helpful as a decision-making tool for D&I stakeholders as they move forward in the work that they are doing. 

So this is a summary table slide that we – I think was made available to you at the start of today. So it lists the ten issues that we have discussed and then tries to break things down into theoretical, empirical, psychometric, and practical considerations that you might make as you move through your D&I efforts. And before I wrap up, I just wanted to acknowledge my excellent collaborators and contributors to how I am thinking about measurement issues and D&I. So I have an excellent SIRC core team, which Ruben Martinez and Kate Comtois who are on the line are part of. As well as [inaudible] at the University of Montana who is the co-PI of the instrument review and has a really hardworking team of individuals where she is. And Dr. Brian Weiner has been instrumental in terms of making sense of the evidence-based rating and how to do that and is leading a team of raters over at UNC. And then I have an awesome team here myself – training research and implementation in psychology lab, with whom I could not do without. 

So I will stop there and open it up for questions as promised.

Moderator:Great, thank you very much. Before we get into the questions, I just want to make a quick announcement. Many people are asking about the handouts and references. Those are included in the slides that were sent out for today. You can access those by going to the reminder email you used to enter the session and there is a direct hyperlink to those handouts. 

So the first question that came in is – sorry, one second: Can you please explain again what the point was from revolving the framework a bit instead of keeping it aligned in the three levels: community, individual, etc.?

Dr. Cara Lewis:Okay. I think if I understand it correctly why make it into a circle as opposed to keep it linear? And I do not think I am going to get feedback here if I got the question right. Is that correct?

Moderator:We can have the person write in further information.

Dr. Cara Lewis:I will presume that I am getting it and then if they write in and help me understand it differently, I will respond to that as well. For making it circular in this way as opposed to linear, it is not doing the best job perhaps as it could if it were to have bi-directional arrows. But just trying to reflect this idea that implementation – that we might hope they would move from exploration to adoption and active implementation, the same engine – we would be done with it. That process – we might see adaptation happening in order for sustainment to occur and further exploration of other kinds of D&I efforts. So it is just meant to capture that idea. 

Moderator:Great. The submitter did write in and say, yes, that is exactly what I was asking.

Dr. Cara Lewis:Great.

Moderator:Okay, the next question we have: How do you manage/measure explicit opposition to the implementation of an alleged improvement or change in a healthcare process? Assume that a policy decision has been made at some level, but, as is the case for many reforms, there is some but not overwhelming evidence for the change and much invested in the process it is meant to replace.

Dr. Cara Lewis:So for that one: how do we measure explicit opposition, especially if the implementation is perhaps come in a top-down sort of fashion? If we had policy as the outer ring here, that it would be moving down to the inner circles with perhaps less flexibility in terms of what is going to happen next. I think that is a really important question. There are some nice instruments that even though we see an impression management with the use of self-reports, there seem to be some that are worded in such a way about change that folks are able to respond perhaps in an approximation, to be honest. But this also might be an opportunity to leverage methods to better understand what components of change are most challenging. Because if you select a specific instrument thinking you might understand where the resistance lies, you might miss it, given that we talked about how self-reports allow you to only understand a restricted range of data. So I have had success with focus groups where the upper management are not present to elicit a better understanding of what that resistance might look like and then might provide a window for additional D&I strategies to engender some commitment to change.

Moderator:Thank you very much. The next question is actually a request asking: Please do give us the article speaking of Mixed Methods and implementing them in a thoughtful way. Number eight, Mixed Methods slide.

Dr. Cara Lewis:Yes, absolutely. And that is included in the reference list that we submitted today.

Moderator:Alright, thank you. What is viewed as an excellent/good number of items for usability?

Dr. Cara Lewis:Oh, that is a great question. Sort of how are we defining this, is what I am hearing being asked. And so per our criteria, we have indicated that fewer than ten items would be termed excellent. 

Moderator:Thank you. And that is the final question that has come in at this time. So if you or Kate would like to give any concluding comments, feel free.

Dr. Cara Lewis:At this point, Kate, I am not sure if there is anything you would like to add. I hope that this was a fruitful consideration of issues that we are all kind of up against in this field. And, as I mentioned, it is not an exhaustive list of things that we need to be thinking about, but, hopefully, it gets people to be really planful about instrumentation and its role in implementation.

Dr. Kate Comtois:And I encourage people to contact Cara if they are interested in participating in our project and also to be looking to the May 2013 when we will be kind of launching the EBA criteria and the rating process for a large number of the instruments up on our website. So that hopefully some of the work of – as Cara described – the decision tool – some of the work of deciding which measure should I use where will be kind of made easier for you.

Moderator:Thank you both. We do have several people writing in saying thank you for the great presentation. And we even did have one person joining us from Australia today. 

Dr. Cara Lewis:Great.

Moderator: So it is nice to know we have a far reach.

Dr. Cara Lewis:Yeah.

Moderator:Okay, well, that is – those are the last comments that have come in, so I am going to ask – thank our attendees for joining us and also thank our presenters for providing their expertise. And to our attendees, as you exit the session, a survey will pop up on your screen. Just give it about a minute or so to pop up and please fill out the feedback survey. It does help us to provide the sessions that you have interest in. So thank you very much to everyone and have a great day.

Dr. Kate Comtois:Thank you.

Dr. Cara Lewis:Thank you.

Page **13** of **13**