5/16/22

ACL Quality Measures

>> Okay.  Hi everybody, this is Shawn Terrell the Office of Policy analysis and development at the Administration for Community Living.  Thank you for joining this webinar.  We're going to be learning about a bunch of new measures we've been funding over the last several years and also encourage anybody who's interested in participating in testing these measures, we'll talk about that as well.  We have a really great panel, fill Beaty the research Director of The Institute on Disability, Independent Living and Rehabilitation Research, known as NIDILRR.

>> We have Brian Abry, Ph.D. a principal investigator and co-director of The Rehabilitation Research and Training Center on Home and Community Based Services Outcome Measurement and the National Center for College Students with Disabilities.  And the Global Disability Rights and Inclusion Program at the University of Minnesota Institute on Community Integration, ICI, as well as, adjunct faculty at the special education program.  He's been the principal investigator on numerous projects designed to have self-determination, social inclusion and employment for youth with disabilities.  And he understands the impact of home and community-based services on the outcomes experienced by home and community-based services recipients.  And we also have from the same organization, Renata Ticha, is the co-director of The ROTC on HCBS outcome measures at ACL.  And, is also co-director of The national -- project of national significance focus on the transition of youth adults with intellectual disabilities to employment, post-secondary education, interdependence and an adjunct faculty at the special ed organization is University the Dr. Has co-authored of application on assessment, the coefficacy and the Research Training Center on Community Living and HCBS outcome measurement.  We have Matthew Roberts, he serves as the project coordinator for the ROTC on HCBS outcome measurement, here at the Institute for Community Integration, a parent of a child with Autism and has an experience of experience working with healthcare professionals and educators and believes strongly in the mission of making the community more inclusive for people with disabilities.  So we're going to go over a couple of things, I want to first point out the ACL invested in this idea of developing measures of really sort of very practical and usable at all levels of the system not just the state level, but also, health plans, which is increasing, and has been around for a long time in some populations for mental health, it's for providers, we can learn to compare provider to each other and just individuals how well are individuals progressing how are person-centered preferences and goals being by all the services that are implemented in that plan.  It's very person-centered oriented.  We also believe that measures need to be able to get down into a level of the system that can really see and address racial, ethnic and LGBTQ, and economic inequities which is a major focus of this administration.  And so, what we're going to be talking today about is, what Brian and Renata are going to be talking about is, a set of measures that we think allows that to happen in a much highly efficient way.  And that is much more flexible than a lot of what we have now.  We also believe that measures should be across populations.  The same measures should apply everybody to every degree possible, a and that they again should be a low level of burdensome we're very excited about this work.  We think this really presents a turning point in how we measure HCBS and really will help us move forward in a way that we can be much more confident about the data that we have, and, much more -- the data is much more actionable at all levels of the system.  So I'll hand it over to Phil Beaty who will talk about NIDILRR's commitment.

>> Phil Beaty:  Hi, Shawn, thank you for that important overview of ACL's approach to this important policy area.  And, g everyone, thank you for joining us, and I should say Good morning to those of you on the West Coast, perhaps further west.  But thank you all for being here.  I do want to take just a few short minutes to give you some background, addition tall background to the work that our ACL and NIDILRR grantees have been doing in this space since around 2015.  So back then, NIDILRR was in its first year as a center within the Administration for Community Living.  And this grant opportunity that you're going to hear more about now in this grant that you're going to hear more about was one of our first that we made as an ACL center, then as now, this opportunity, this effort was very much a team effort among our ACL colleagues.  So, in our funding opportunity back in 2015 we broadly summarized the aim of this new research grant this way.  We said, the grantee will engage in research, development, and testing of measures to assess the quality of home and community-based services in terms of the person-centered outcomes achieved by people with disabilities who use services, who use the services in the home, and community settings.  In short, we said that the measure of the quality of HCBS programs is -- and the measure of quality that we're interested in is the assessment of outcomes achieved by service recipients.  We went on to say in our funding opportunity that HCBS measures developed by the grantee must be nonmedical, and must focus on the end user's experience of community living, independent living, social integration, community participation, and other similar outcomes.  As a result of this opportunity, we made a research and training center grant to the University of Minnesota, Brian Abery and Renata Ticha are here with us today as you've heard to tell us about their ongoing measurement -- measure development efforts.  And to share with you potential opportunities to be engaged with them in their work.  Please note that this work of building measurement infrastructure is an ongoing effort at NIDILRR.  In addition to the work that Dr. Abery and Dr. Ticha will tell you about from their home at the University of Minnesota, we also have other grantees working in this important area as well.  Our NIDILRR grantees at the Shirley Ryan AbilityLab are also doing HCBS measurement development work, and we'll look forward in the future to the opportunity to host them for a related webinar in this space.  For now though, I'm just really happy to turn the floor over to Brian Abery who will begin walking you through the University of Minnesota's ongoing, just really thoughtful two, in this area.  Dr. Abery the floor is yours.

>> Thank you, Phil.  Thank you everyone for coming.  What we are going to be doing today is talking to you about some of the work we're doing, some of the theory and thought that lies behind it, and give you some updates as to where we are today and where we hope to go in the future.  I am going to now attempt to share my screen.  Have I been successful with that?

>> Yes.

>> Okay.  Thank you.  Okay.  So, we're going to start with something that is truly basic and you know, from our perspective what HCBS outcome measurement is all about.  You know, as Phil and Shawn said, you know, we want to really do two things with the outcome measures that we're developing.  You know, number one we want to make sure that we're taking a look at the quality of supports that are received by HCBS beneficiaries, including individuals who are aging and individuals who have disabilities.  We want to make sure that those supports are of high quality, that they're timely, that they're well-coordinated, respectful of the rights of beneficiaries and they support a high quality of life, a quality of life which is what the individual themselves wants.  Now, take a look at the quality of supports is one piece of the puzzle.  The second piece is taking a look at those personal outcomes that are experienced by individuals who are the recipients of supports through home and community-based services.  We really made a decision at the beginning of our work that we were going to take the National Quality Forum's HCBS Outcome Measurement Framework as a starting point.  Now, as we think about you know, measurement and why it should be important to those of you who are providers, there are a number of different reasons.  At the kind of top-level there's the notion that you know, the measurement will allow you to collect data that shows that you're compliant with both state and federal legislation and policy, you know, concerning home and community-based services.  But we feel that that really is only the tip of the iceberg.  For us, you know, the real importance of quality measurement is that it allows us to really take part in improvement efforts that are driven by data, drive been I the information that we're receiving from people with disabilities, their family members when necessary, and providers so that we can make sure that individuals are getting the best services and supports to allow them to achieve outcomes that are personally important to them.  You know, we believe that outcome measurement should be important to you because, you know, it's going to provide evidence that the innovations that you institute in your programs are actually achieving the desired outcome, and not having any negative unintended impact on the individuals who you serve, and then the end-point, you know, it certainly should provide you with information that allows you to better market your program to potential users and for users to be able to select home and community-based service programs that meet their specific needs.  One of the things that Shawn talked about a little were, some of the principles underlie HCBS Outcome Measurement, and as we went about the process that we've engaged in to develop these 13 new measures, you know, we really were driven by the notion that there were some underlying principles that we needed to make sure that all of our measures are attested to.  The first one of those is I think probably the most important.  We wanted our measures to be person-centered.  We wanted our measures not to consist of frequency counts of the number of times that people have accessed this service or that, or been out into the community.  We wanted to make sure that the measures were actually getting at what was important to people to the degree that people wanted to engage in those activities, to have that level of control over their life.  So, number one, measures need to be person-centered.  The second piece that we thought really needed to be incorporated into our measure development process was, the opportunity for measures to be used longitudinally over time.  If you're a provider agency, and you are implementing an innovative program, you want measures to use at the beginning and kind of once the program has been instituted.  So, you can have information tell you whether you're innovation has been successful and has led to the outcomes that you desire, your recipients of HCBS to experience.  As Shawn said, we want to make sure that measures are feasible to administrators, that they don't put an excessive burden on either the respondents people who were asking questionsof , or to providers.  As I said earlier, we really wanted to start with the National Quality Forum framework for HCBS Outcome Measurement, and some revisions that we made to it.  Because we thought that that really should be one of the driving forces for measurement in this area over the next decade.  And then finally, for those of you who are service providers, we wanted to make sure that you know, the measures that we developed had utility at your level.  We have a number of different approaches to measurement that are available now that are great to allow states to show that they're in compliance with the final settings rule and other legislation surrounding HCBS, but we really felt that we needed to get to a deeper level to allow providers to really be able to demonstrate that their service and supports were leading to intended outcomes.  A lot of this goes back to this notion of person-centered measurement.  And it's certainly driven the work that we've done at our institute on community integration, at HCBS Outcome Measurement from the beginning.  And that is the notion that measurement needs to be based upon the experiences, the needs and the goals, and the Vales that people with disabilities themselves express.  We wanted that to inform the decisions that we made about what we were going to measure, how we were going to measure it, you know, how we were going to evaluate the performance of community-based supports.  Now person-centered measurement is not the same thing as assessing person-centered supports and practices that should of course be person-centered but we want person-centeredness to underlie all measurements in this area.  And that is a type of measurement that balances what's important for the person, with what's important to the person.  And that's something that we don't think measurement in the past has done a good enough job at.  Now, I know this is a little bit overwhelming for people, but, this just giving you an idea of the complexity of the types of things that you know, we believe we need to be assessing at the consumer level the provider level the system level to understand the outcomes that people with disabilities and people who are aging experienced when they receive home and community-based supports and the quality of services and supports that are being provided.  You know, and, basically, in outcomes, we're taking a look at the degree of community inclusion that people, the choice of control they have over the services and supporting and their daily lives, the degree to which they as consumers of supports have input into system development, system evaluation efforts, the degree to which there's equity in the services and supports that are provided to HCBS recipients.  And a number of other areas.  We believe that all of these are important because, we are really trying to reach across kind of our aging population, older adults, plus individuals with disabilities.  We selected on the basis of a lot of nationally-based research that we did in the beginning of our center, 13 specific areas that we have focused on.  Now as we think about measurement, there are different levels of measurement.  And as we've developed our measures, we tried to keep this in mind.  Kind of at the bottom of that pyramid there, are compliance and kind of quality assurance measures at the state level.  Next up, we is have quality of service delivery which is the agency level.  We have measurement that leads to kind of individual outcomes that affects the lives of real people on a real-time basis.  We have tried to be sensitive to the fact that as we move up this pyramid, we need to increase the level of person-centeredness, maximize it, and we need to also increase the level of precision in measurement.  It's one of the things which in-home and community-based services we really felt the need to put more emphasis on so that when we have measures and when measures are released, the individuals who are users can be assured that they are psychometrically sound and they can have faith in them.

So we went through a number of different phases in our measure development process.  Starting out with taking that National Quality Forum framework, which has 11 different domains and somewhere between 3 and 7 sub-domains within each domain, we took it out to people, people with disabilities, people who are aging, family members of those individuals, providers like yourselves, policymakers and researchers.  And we asked them to provide input into that framework, to tell us if they thought everything was missing, and to provide us with importance weighting so we could figure out which domains and subdomains, which outcomes, and which kind of aspects of service quality were most important to measure.  We then took that information and we compared it to the current measures that we had available.  We did what we refer to as a gap analysis.  And we have developed a database that is available on our research and training center site that allows users to look at I won't say all existing HCBS outcomes measure MMHSCEAP, but a large number of them.  And we broke over 130 of these measures down on an item-by-item basis and assessed the degree to which they met those quality indices that we have established, starting out with being person-centered.  We then took the input from that gap analysis and our work with stakeholder groups and that led us to kind of select 13 different National Quality Forum domains and subdomains on which to focus our work.  Our next phase, we kind of took those initial domains and subdomains to a number of technical expert panels, and these were individuals with disabilities, family members, and experts in these content areas.  And we had them provide us with weightings of their importance, the utility, the usefulness of having measures in these areas and the feasibility with which they thought measurement data could be collected.  As a result S we basically have you know, focused our work on developing measures in 13 different areas that focus on the left-hand side on 8 different domain, or.  Different domains within the National Quality Forums outcome measurement model, and, within each of those domains, anywhere from 1 to3domains, so you can see that we have focused our measures of developing measures of choice and control, we have three different measures of choice and control, one of which looks at personal choices and goals, one of which looks at choice with respect to supports and service, a third one which focuses on individuals who are part of self-directed support programs.  We looked at community inclusion and developed two measures in that area, employment human and legal rights, and transportation, a very important piece of the National Quality Forum model that was missing until we got input from stakeholders and people with disabilities people who are part of our aging population told us without good transportation it doesn't matter how many resources are available in their area, they can't access them until they can get to them.  Consumer leadership and system development and person-centered planning and coordination were also other areas in which we engaged in measure development efforts.  As I said earlier our initial measurement development efforts started out with domains and subdomains from the National Quality Forum Outcome Measurement framework, we then did something which we don't think has been done enough in our field before, we did an intensive review of current research related to each of the measurement constructs that we were developing.  We really wanted to know what the current research was saying about what are the components of social connectedness, what really needs to be present for individuals to feel socially connected to significant others, and socially connected to their community, what were the important components of an individual feeling that they had choice and control, or self-determination, over their lives?  After that intensive review of the current research and literature we set up guiding questions and claims statements, those are questions that we'll be able to measure for providers and for individuals who are recipients of home and community-based services.  We then took a look at NIDILRR's charge to develop measures that basically allowed us to cut across different groups of HCBS participants and the groups that our measures focus on are four groups with disabilities, in addition to older adults, we don't necessarily have a disability, but who receive HCBS services and supports.  So our measures have been developed with these groups in mind, and have been tested with these groups, individuals with intellectual and developmental disabilities, people with physical disabilities, psychiatric disabilities or mental health challenges, and traumatic or required brain injury.  Of course older adults who receive home and community-based services and supports.  We believe that the characteristics of a measurement system need to fit the needs of the people who are going to be using it.  And while we currently have a number of instruments that have been developed, that can be used to assess all aspects of the home and community-based services, we thought we really needed to take a different approach.  That is develop our measures in a modular format some we haven't developed an instrument.  But 13 different measures that providers or states can pick and choose from, each measure can be used independently or in conjunction with other measures.  We thought that was important because, in getting these measures down to a level where they could be used by providers, we understood the fact that if a provider is doing an innovative program to enhance the social inclusion of HCBS recipients, they don't want to necessarily be measuring things that have to do with choice and control or consume are leadership or system development and design.  We wanted our measure to be able to be used longitudinally so that they could be used in quality improvement efforts.  And in order to make our measures feasible.  We develop our measures so they have two tiers.  In the first TIRR there are three to five general items that are intended to provide use her with kind of an overview of an individual's outcomes within a specific domain or subdomain.  Across those 13 measures, you're thinking of somewhere in the neighborhood of 39 to 40 items, which would allow you to get a general picture of how as a provider agency for example, you're doing in providing supports that leads to these person-centered outcomes.  At the second tier, we have additional items.  And those that teirs, and those items are really met for a provider to be able to do a deeper dig down into kind of the outcomes that their HCBS recipients are experiencing and really get what we refer to as "actionable data" data which will give you information about if you need to make changes, if you need to make some kind of innovative practices in order to achieve the outcomes that you and your recipients want to achieve, it should provide you with that type of information.  Now, while most of the people who we have worked with can respond to these questions on their own, Americans with disabilities and older adults, we do realize that individuals who have the most significant disabilities and the most intensive support needs might not be able to take part in an interview be that Zoom meeting interview, an in-person interview, and respond to those question, in addition to kind of our general measures, we also have developed measures that can be responded to by proxy, they're an abbreviate set of measure, and are intended to be answered by a person who knows the individual who is the HCBS recipient well.  I am now going to turn things over to Renata, she is going to talk a bit about the measure development trajectory, which we have followed which is based upon the CMS measure development blueprint, and the results of our pilot study, and what we are currently doing at a national level in order to collect validity and reliability data.  Renata?

>> Renata:  .  Thank you, Brian.  There was before I start, there were some notes in the chat about whether we can make the text a little bit larger.  You can click on the A and enlarge the slide text?  I just made it smaller, oops, I don't want to waste time, Brian has already presented a lot of information, I know it's a lot to digest.  But just wanted to talk a little bit about sort of the visual represents the various phases we went through in order to make sure that the measures are reliable and have good validity, good accuracy and are going to be usable.  At the end of the day, we are currently are in the last phase, it's been a long journey, we're in phase five, we have gone through all of these different pieces including cognitive testing, highlighting further revision, now we are testing them on scale.  Next slide.  So before cognitive testing, we have gathered several technical expert panels to review the approach we were taking to measure development as well as the content for each of the sets of measures.  For example, participation and activity to make sure that we are covering the latest concepts, the latest research, in the field, and also that, people who have disabilities themselves, family members very supportive of those ideas, be we conducted cognitive testing.  Meaning that we had to make sure that the way we worded our questions and constructed our response options was understandable to as many people as possible, even though it goes baaing to people with disabilities.  And, we have conduct add pilot study, I'll describe that on -- we have conduct add pilot study, I'll describe that on the next slide.  We hood a sandal of over 100 participants on our study.  The purpose of the study was to really start testing the initial accuracy, reliability and validity of the measures.  The pilot was conducted in Minnesota and Pennsylvania.  And we tried to get as a representative touch of a sample as we could.  You can see some of the breakdown on the slide.  Next slide.  And it was because, it was at a time of COVID was just starting.  So we did manage to conduct 85 in-person interviews before the pandemic really begun, then we finished online.  And we were able to test the measures with those two modes and including interrater observations, so meaning, using two people who administrator the measures.  And also testing, and retesting the measures within a two-week period.  So, I'm not going to develop these numbers, the numbers are important to us, because they indicate the level of different types of reliability that these measures demonstrated and you can see the range is there between 0 and 1.  And the higher the number, the better the measure is, and you can see that most of the measures really displayed high levels in terms of this consistency and the other two types of reliability.  So we were very, very pleased with the pilot results.  As I indicated in the table, the measures overall demonstrated initial really good properties.  There were two measures, freedom from abuse and neglect, and barrier to employment that have slightly lower levels of reliability.  At least in one of the categories, so we substantially revised those measures based on the pilot study.  Other things that we learned through the initial testing is that, as shown indicated at the very beginning, the measures are designed to be used across people with different types of disabilities, the measures, and the realities really indicate that there were no significant differences in the levels of accuracy, reliability these measures displayed which was a really good finding.  It means that we can administer the same measure to an individual with an intellectual developmental disability and to someone who is who has a traumatic brain injury.  The piloting also was evidence even if it was indirect to validate the framework that we -- the NQF framework that we used to build our measures from.  We also found that people with all types of disabilities are able to respond to the questions and the response options produced reliable data.  The other thing that our measures do, they have two levels.  They have Tier 1 questions two to three questions that get at a global concept of that measure, for example, social connectedness.  Then there is additional questions that follow, and that structure has worked well in the initial administration, in the study we are conducting now which is the large national field study, we will be further examining the relationship teen those two questions so we could use the Tiers separately.  Of course we are going to be conducting other analyses including factor analysis to investigate the subscales of these measures and look at the relationship between the measures themselves.  So now where we are at in the last phase of testing these measures before they will be available to the general public, the goal is to test the measures on our population of a thousand people with different types of disabilities.  And that's as you can imagine, a very large undertaking but is very, very important to us that we do that so that we know the measures have good quality.  We are recruiting in various space and one of the reasons why we have this webinar today is that we are hoping that some providers, and service providers they'll be interested in helping us out to finish testing these measures.  So, we have those measures that Brian already presented.  There's about 12 or 13 of them.  We do the data collection, either online using Zoom, or in person.  We are trying to test the measures of using three data points, so three times.  And then we also have a sub-sample of people without disabilities to make sure that we are really not treating people with disabilities differently, but have similar expectations, that the problems that the disability may present.  So, as I said we are very active in recruiting participants.  We are including participants from the United States at the provider level.  So, our goal is that, we would contact our we would have an interest from more providers, we already have very much but we need more.  And we would provide the service provider with more information.  Our project coordinator or center coordinator Matt Roberts will be able to step in and provide training, and technical assistance.  We have a really solid infrastructure how we collect this data.  We have all of the ethical review approvals.  You know, so this is a smooth process for us.  But we just need more providers to be interested in how they access these measures.  The individual participants they'll get a gift card for participation.  And we have a contact on this slide, if you are interested we will share the contact widely, it's also in our information as well.  If you can go to the next slide.  I'm just going to start showing why you would want to do this in addition to the fact that it's a good scientific practice is that, you will have a steak over our input to the validation process of these measures, the perspective of the people that you serve will be represented, we'll take into account anything that is not working, or not worded correctly, or the people are having trouble responding to.  And, it will provide actionable data that will give you an idea of what this data looks like if you wanted to adopt these measures in the future.  You will understand better what goes into making a measure a really good measure to be used widely.  And you will also then be one of the first people after this is finished to receive training how to use these measures if they so choose to be.  So, the final goal of this measure as they introduced it is really not to develop this whole measurement program, but really to generate measures that can be used at the provider level with actionable data, and to be able to use the information and the data for states for providers advocacy, self-advocacy organization to really advocate for better quality of life using the indicators, using the outcomes in which we have developed these measures, and, our measures, there will be no charge to use the measures.  But, there is going to be of course a requirement for training to use them appropriately.  So, this is our final slide.  Again, we would love if the service providers were interested in assisting this effort, in you are.  Please use any of our contacts, preferably, our center coordinator Matt Roberts that's the last email.  I'm sure we'll share this information as well in the chat in Q&A, and within the PowerPoint slides, thank you very much for your time.  We would be happy to answer questions.

>> Shawn, we'll turn it back over to you.

>> Great, okay.

>> Thanks for the excellent presentation.  A lot to chew on, I think for the average person who's not a researcher, which I think is really good, because it gives us all an opportunity to think about how we know when we see a measure what kind of questions we may want to ask about how well it was tested, et cetera.  So I guess I'll just go right into a couple of questions that came out through the CHAD and in the Q&A section.  The first one was that, will the presentation and the recording et cetera be available?  And the answer is, yes.  We're going to post them on the ACL website.  I put the link to that in both the chat and the Q&A.  So you should be able to see that.  It's going to take a little bit of -- I won't be like today, it will be a little bit of time to get it up there, make sure back in a week or two et cetera.  And we'll eventually get it up there.  The other questions, a bunch of really good ones in here, first of all, I wanted -- a couple of people have already sort of provided us, expressed interest in participation.  So that's really great.  We really love to get these tested so we can get them out to people to use.

If you are interested, just please email, you see the email on the final slide, to either Brian or Renata.  They would be happy the get you connected.  Actually, a specific email here at the bottom to use.  Any one of those, even me, whoever you want to email, we'll make sure you get to the right place.  There's no wrong door here on getting to the testing.  And we don't -- I think there is a one-pager.  I guess someone just pulled that up.  There is a one-pager flyer that, I believe they have.  But let me let Brian or Renata answer that one.

>> Yes, we do have a one-page flyer for recruitment purposes.  We also have videos that providers who are interested can use.  They're very short, one and a half minute two minute videos that you can show to those individuals who you provide supports and services to encourage them to take part in this process.  And again, that's something that we can get to you very easily in addition to the flyer.  One of the things I do want to make sure people understand, essentially we're asking you to help us recruit individuals who are willing to take part in this.  You know, we will do the interviewing.  We have interviewers probably in 8 or 9 different states now, so we have people in all time zones.  We can do interviews via Zoom meeting which is how most of them are being conducted.  But we can also do face-to-face interviews if that is necessary.  We'll jump to a couple of other questions, we'll get into some sort of meaty issues here.  The first one is, --

>> Oh, Shawn?  I'm sorry to interrupt you, Brian can you stop sharing your screen real quick, so that when you guys answer questions, I can bring everyone on the screen.

>> Great, thanks.  So I think the contact info is now in the chat if you're interested in participation and testing.  So, another question is, did you consider including a workforce-related measure which is obviously a major issue these days.  Is Brian or someone wants to take a stab at that?

>> I can take a stab at it.  We have our colleagues here at the institute on community integration who are doing work in that area.  Because of the input that we got from people with disabilities, families and providers, our notion was to start with a focus on these personal outcomes for the most part.  Then I believe that some of our colleagues at UAL are also -- and Northwestern are focusing on that with the new RTC and outcome-based measurability.

>> I would also add, like one of the things this is really important.  So when you -- there's a measure looking at community inclusion, you know, the degree to which you feel part of your community, et cetera.  And you know, when you get responses that say well, no, I don't feel a part of the community, I'm not really feeling included.  Some of the causes of that may be, maybe workforce-related.  Once you start digging into the answers, wow we all know there are a lot of workforce issues, particularly around COVID around providing community-based supports.  So I think, what do you think of that Brian, that approach?

>> Certainly, I think that's true.  You know, our colleagues here at the institute will be I think within the next two or three weeks releasing a workforce primarily focused on direct support providers survey to take a look at what is going on in the workforce at this point.  They've done this three previous times both pre-COVID and during COVID.  And that hopefully will provide us with a lot of good information on that area.  It's a sample of I think the first iteration had over 10,000 direct support providers who actually responded to them.

>> Great, thanks, let's move on.  A couple more questions.  One is, sort of kind of go in order here I'm bouncing back and forth between the Q&A and chat.  From the Q&A there was a question or thought about, my experience is that people with IDD often pre tend to give an answer that they feel is expected of them.  When people who are leg usually impaired are interviewed did you consider subtle coercion by the staff who are present in the testing?

>> We have gone to great lengths do make sure that individuals are in a private setting when they are being interviewed just because of that question.  We've also tried to design our items because we know that sometimes people with intellectual disabilities have kind of an acquiescence response set and are likely to say yes with avoiding to asking for satisfaction items.  Because as our former colleague and former Director of NIDILRR Charlie Lakin used to say, 85% are 85% are satisfied with 85% of the services they get 85 percent of the time, so we've avoided those types of questions primarily to get to questions that give us better evidence of the quality of services they're getting and the outcomes they're receiving.

>> Great, there's another question that's related to sort of the -- participant in this case.  So would this work for people who are verbal and nonverbal?  And is there a certain level sort of, I guess, you know, I don't know what level?  But, -- the cognitive level that people need in order to participate?

>> That is a really good question.  As far as for the participant responding themselves, we have a scale that we use to determine whether the person's responses can be really valid or reliable some we go through that when we do our consent process.  But if the person is not really able to understand the questions that we're asking, we then will ask a proxy respondent.  For example, it can be a family member, it can be a support staff, it can be a friend to respond.  With we a version of these measures for proxy respondents so that we don't really prevent anyone from testing.

>> The other thing that we have expected as we have been going through this process is the use of visuals to help individuals understand the consents that we are asking them questions about and to provide them with an approach to responding that doesn't require a verbal response.

>> Okay.  Great.  I'll see if we can get through a couple more here for the Q&A.  After testing, will measures be able to be collected using different modes as in telephone, online, and/or mail.  Snail mail?

>> Right.  So right now our two main modes are in-person and online.  We have had several participants who are not able to use online Zoom.  So we did a phone interview, but our preference is Zoom, online or in person.

>> Okay, great.

>> I mean it's very difficult for a lot of -- this is a challenge for the field in general to get to some really strong oh protocols about how to do sort of, ask questions around measures among different groups of people.  I think hopefully we we can get to one of the projects from a policy perspective some of the protocols that are really effective that other measure developments can use to help us develop those.  There are others as well.  So, let's see.  Are the measures only admirable to Medicad HCBS or might they be used by providers in a private pay launch of care insurance world with good validity?  Are there limits on the program side, these measures would apply to.

>> I would say that the only real limits is that the questions and the items are designed for individuals who are receiving some form of home and community-based services, or, the equivalent of home and community-based services, we know there are some states that provide support to individuals with mental health challenges, not under HCBS but under other programs.  They were applicable to those individuals.  I would say that they weren't intended to be applicable to individuals who are in institutional settings.

>> So how long does it take to do -- so this is a really important question.  You kind of mentioned it.  But I think we'll make sure people are clear on how the time requirement is around one or more of these measures.

>> You know, we basically look at the amount of time it takes to administrator measures as opposed to the whole group of all 13 measures we've developed, given the fact that they are not intended to necessarily be administered as an instrument.  You know, I would say that if you're talking about administering both Tier 1 and Tier 2 items, it takes anywhere from 10-15 minutes to work through a measure.  Okay.  Again, we developed these measures in a modular format because we didn't necessarily foresee their actual use after validation being as an instrument.  You know, certainly, as we're going through our field testing, would like to see as many of these measures administered as possible with each individual.

>> Okay, great.  So, somebody's asking for an example of one of the measures, you had a slide that kind of played out at least a topical area, I don't know if she's asking specifically about all of the questions that go on the measure.  Maybe pick a measure and describe a little bit more about it.  That might help.

>> Okay.  Renata maybe you can help me with this, and Matt feels free to chime in.  One of the measures that we developed has to do with social connectedness.  One of the problems that we saw with existing members of social connectedness, they really just ask people, how many times, how frequently they got to see family, how frequently they got to see friends.  Our measure in that area focuses on a number of different parts of the social connectedness construct.  You know, not only -- connectedness construct.  Not only do you get to see your friends, you get to see people who you want to be to be part of your social network.  Do you get to see them enough to meet your needs in do you see them in the environments that you get to see them?  Another important point that we found missing from the measures of social connectedness, social connectedness has been found research found by the person is able to receive social support and to provide support supports from others.  We have questions that look at various aspects of social supports, so we can see whether that individual's social network is providing them with social-emotional support, informational support, or the other types of support which will help them feel more included in their community.  Thank you.  I mean, I don't know, I understand that you would be willing to share all the research results.  You have really extensive measure blueprints on each of these measures, which is great.  It's like a literature review in and of itself on everything that's out there in these topics.  That might be useful to other people when we get these sort of out to the world.

>> There are individuals who are -- interested in participating in the field testing, we're more than willing to share those blueprints.  It's a lot of good bedtime reading.  Each of the blueprints is about 30 pages in single space.  Kind of goes there you the process that we went through including the review of the literature, the whole kind of item development process, certainly when these measures are able to be released, each measure we hope will get eventual you NQF endorsement will come with a measure blueprint to you can see how they were developed.

>> If you want a little more digestible format, we do have on our website several briefs that talk about these topics in a shorter way.  We also have modules that we have developed, we call them training modules, they are really adjustable information modules of ATS outcome measures that include information,

>> There's a follow up -- there's a follow up -- on the time question which is that, okay, we have one more question on this, we'll see, 15 minutes to work through a measure.  There is whatever number of people.  I'm sorry, whatever a number of Tier 1, Tier 2 measures, whatever, it's like 14.  So, equals how much time to set aside for an interview.  I think the question is assuming all of the measures if you did all of them at once.  Including both Tier 1 and Tier 2 there would be two numbers.

>> If you were to administer all of those 13 measures, you know, at once, it would probably take I would say anywhere from 60 to 90 minutes for a full interview.  You know, again, what we foresee is these very views of these items initially being administered Tier is and providers looking at on the base of the Tier 1 responses, what area do we need more in-depth data on?  Then delves into those areas with Tier the questions.

>> So, yeah.  I think also, part of the question that we have on this is, since these are modules, you can -- ask, you can sit down with somebody in the context of say a case management visit, or, sort of conceptually assuming the train is there, ask during the context of a visit you're going to make anyway, ask some questions, roll out a measure or two in that visit.  Then the next one you could ask other questions if you want.  And that becomes -- it's flexible at that level.  And if there's a way to think about this, I think this is for everybody who might be joining the testing, a way to think about this in the context of how to incorporate these sort of flexible measures into existing workflow across your system would be, we would all be very interested in that, in how that might work and be a more efficient way of doing measures than sitting down and doing a long interview with a whole bunch of measure as the same time.  So that's kind of the hope that these modular measures offer that kind of support.  I know we're on time.  And I think we've got to most of the questions.  I probably didn't -- somebody asked me for my title it a health insurance specialist, I leave it off because it's so bureaucratic nobody knows what it means, I'm in the Office of Policy and analysis.  We can wrap up.

>> I would like to say we are very interested this working with any providers out there who would be interested in being part of this process, we are working right now on about 15 different states, we are working with a couple of MCOs as part of this process because we do realize that home and community-based services are increasingly being provided through the managed care approach.  You know, I guess what I would say kind of in leaving is that if you have any further questions or need additional information, you know, prior to making a decision as to wanting to participate or you just have questions about what we're doing here at our research and training center, you know, please feel free to contact either Renata, myself, or send our cord needer Matt Roberts.  Because as part of the responsibilities of our center, you know, we also provide free technical assistance when people have measurement questions, and want to make sure that we support kind of increasing the quality of measurement regardless of whether you're using the measures that we're developing or other measures.  Renata, anything you would like to add to that?

>> It really doesn't matter to us whether you're a small or large service viewed, we will work on a provider any size in part,

>> Well, with that, I guess that will wrap up our discussion today.  I really -- we're all very excited about this, we're really happy that we had so many people on, excellent question, and we have additional questions you could address me, you can address Brian, Renata, whoever you'd like to.  My name is -- my email's in the ACL announcement.  So you can click on that.  I'll make sure we get them responded to on an ongoing basis.  Okay.  Well, that's it, I guess we'll say goodbye then.

>> Thank you, everybody, we appreciated your attention.