NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 1

August 16, 2017

2:00 pm EST

Coordinator: Welcome and thank you for standing by. At this time all participants are in a 

listen-only mode until the question-and-answer session. For questions from 

the phone line, you may press Star 1. Today’s conference is being recorded. 

If you have any objections, you may disconnect at this time and now I’ll turn 

today’s meeting over to Meredith Raymond. Thank you, you may begin.

Meredith Raymond: Good afternoon, everyone. Thank you for joining us for the third event in 

our HCBS quality Webinar series, quality measure development. This series 

consists of informational Webinars occurring on a bimonthly basis to build 

awareness of ACL’s commitment to and development of HCBS quality 

measures and to provide a platform among internal and external stakeholders 

to share developments and collaborate on efforts concerning HCBS quality. 

In June we hosted our second Webinar, HCBS framework development which 

provided a deep dive into the significant role of framework development in 

HCBS quality. Organizing frameworks play an integral role in today’s 

Webinar topic measure development.

ACL strongly supports any progress toward nationally-validated outcome 

measures for long-term services and support and has invested in a number of 

initiatives that are working towards this goal.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 2

Due to Human Services Research Institute, ACL supports the development 

and implementation of a national core indicators for people with intellectual 

and developmental disabilities and the national core indicators for aging and 

disability.

These two related survey instruments continue to inform the field on state 

system performance. ACL and NIDILRR fund a rehabilitation, research and 

training center on home and community-based services outcome 

measurements.

A result of the research conducted through the center will be a set of 

recommended measures and procedures to ensure that they support the quality 

of life outcomes for people with disabilities and older adults.

We are fortunate to have representatives from the Administration for 

Community Living, the Human Services Research Institute and the 

Rehabilitation Research and Training Center on Home and Community-Based 

Services Outcomes to discuss the importance of quality measurement, how 

measures are developed and the progress made through these initiatives.

The Webinar agenda is as follows. Dr. Heather Menne, Social Sciences 

Analyst at ACL will provide a brief overview of the basics of quality 

measurement. Alexandra or Alex Bonardi, Project Director at the Human 

Services Research Institute will discuss the progress made in developing 

HCBS quality measures from the national core indicators and national core 

indicators for aging and physical disabilities survey instrument.

Dr. Brian Abery, Co-Director and Dr. Renáta Tichá, Research Associate at the 

University of Minnesota rehabilitation, research and training center on home 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 3

and community-based services outcome measurement will discuss tools and 

progress resulting from measure development research.

The presentations will be followed by a 15-minute question-and-answer 

session. At the conclusion of Dr. Abery’s presentation, the operator will 

provide directions as to how to enter your question into the Q&A queue.

To begin I’d like to introduce Dr. Heather Menne, a Social Science Analyst at 

the Administration for Community Living. On behalf of ACL she oversees 

the nutrition services program, outcome evaluation, the outcomes evaluation 

for the national family caregiver support program and the annual National 

Survey of Older Adults Act participants.

Dr. Menne earned her doctorate in sociology from Case Western University 

and a Masters in (gerontological) studies from Miami University. Thank you, 

Heather.

Heather Menne: Good afternoon. As Meredith mentioned, I’m Heather Menne and I am in 

ACL’s Office of Performance and Evaluation. I have been working in the 

field of aging specifically gathering information about how programs operate 

and whether they are providing quality services to individuals and to 

communities and I’ve been doing this for over 10 years.

Program evaluation is a systematic method for collecting, analyzing and using 

information to answer questions about projects, policies and programs, 

particularly about their effectiveness and efficiency. 

It is important to remember that when we think about measuring how our 

programs or services are performing, we have to look at quality and quality 

measurement as a type of evaluation with regard to home and community-
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 4

based services and ACL’s mission to maximize the independence, well-being 

and health of older adults, people with disabilities and their families and 

caregivers.

We can expand the Institute of Medicine’s definition of quality from clinical 

services to include supportive services and the degree to which those services 

impact the well-being of those we serve.

Items that we need to consider when thinking of program or service quality 

commonly include six dimensions. One, effectiveness which relates to 

providing services that achieve the desired outcome, in this case of helping 

people to live independently with maximum health and well-being.

Two, efficiency. Are we doing the most that we can with the resources we 

have? Three, equity. This relates to providing services of equal quality to 

those who may differ in personal characteristics. The fourth dimension is 

person-centeredness. This relates to meeting consumers’ needs and 

preferences.

Fifth is safety relating to actual or potential bodily harm and sixth is 

timeliness. This relates to obtaining needed services as quickly as possible. 

The move to quality is not new even if it was and I always talked about using 

that term. GPRA or G-P-R-A stands for Government Performance and 

Results Act from 1993.

The GPRA Modernization Act serves as a foundation for helping agencies to 

focus on their highest priorities in creating a culture where data and empirical 

evidence plays a greater role in policy, budget and management decisions, in 

other words, using data such as that about program and service quality to 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 5

make programmatic and policy decisions that improve the programs and 

services we offer. 

In 2013 Memo M1317 from the Office of Management and Budget discusses 

approaches for strengthening agencies’ abilities to continually improve 

program performance by applying existing evidence about what works, 

generating new knowledge and using experimentation and innovation to test 

new approaches to program delivery.

In the HCBS context this means finding-out which policies, programs and 

services are best at helping individuals of communities make their own 

choices and participate fully in society.

In the recent 2018 budget blueprint, the current administration also discusses 

use of evidence and measurement of program quality. The budget blueprint 

describes the programs proposed to be cut as lacking evidence and those 

proposed for expansion as having it.

If we do not examine our programs objectively, we cannot really know 

whether they are improving people’s lives either directly through service 

provision or indirectly through improved efficiency. Measurement of 

program quality can help us to maintain high-quality programs and improve 

lower-performing programs.

Give individuals and stakeholders information about program quality to help 

them choose the services and approaches that are best for them and ensure that 

we are getting our money’s worth by making sure that we are getting the level 

of quality we expect. It’s important to remember that measurement is a 

quality improvement tool, not an end in and of itself.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 6

We should use quality measurement to tell us where we need to look more 

closely as we work to serve individuals and communities in the best, most 

effective ways possible. In reality we cannot and should not measure 

everything.

Even though I’m an evaluator, I do realize measurement should not cost more 

than a program. Measurement should complement the program and gather the 

most important information needed for decision-making and service 

improvement.

When we think about home and community-bases services, there are several 

program dimensions that it makes the most sense to measure because these are 

areas that we have some control over.

These include the quality of how care is structured, for example is it holistic 

or is it multidisciplinary? Second, how the process of providing care affects 

consumers and their health outcomes. Third, what are the consumer outcomes 

in terms of health and well-being?

We can also look at the level of resources used and compare the resources per 

positive outcome. I look forward to hearing from our colleagues at the HSRI 

and the University of Minnesota RRTC as they discuss their work on HCBS 

quality measure development. Thank you all for your attention and thank you, 

Meredith.

Meredith Raymond: Thank you, Heather. Now I’d like to introduce Alexandra Bonardi. Alex 

Bonardi is currently the Director of National Core Indicators at the Human 

Services Research Institute. 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 7

She has worked to support people with intellectual and developmental 

disabilities throughout her career in a variety of capacities including clinically 

as an occupational therapist, as a researcher focused on interventions to reduce 

risks such as (false) prevention and on data collection efforts to enhance 

policy-based decision-making and to improve quality and equity in the 

delivery of support. One moment, please. 

Alexandra Bonardi: Thank you, Meredith. This is Alex Bonardi and I think now I’ve got 

control of the slides so than you for the opportunity. This afternoon I plan to 

be talking about two tools that exist for measuring and improving quality of 

supports for people with disabilities and our project’s work to progress 

towards nationally-validated measures in the area of home and communitybased services.

As Meredith mentioned we are talking about two quality monitoring tools, the 

National Core Indicators and the National Core Indicators for Aging and 

Disability. While these are two distinct quality monitoring tools, they do have 

common protocols implementation and some commonality as related to the 

development and measurement domains which we’ll be getting into in a 

minute.

I wanted to emphasize that these are state-level quality monitoring tools and 

these were really developed in response to states identifying a need for 

effective tools to monitor program quality. HSRI Human Services Research 

Institute has partners and has partnered from the inception of both of these 

tools for the development, administration and the use of quality data.

In the case of national core indicators, it is a partnership with the National 

Association of State Directors of Developmental and Disability Services and 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 8

in case of national core indicators for aging and disabilities in the partnership 

with the National Association of States United for Aging and Disability. 

We do have contact information at the end of our little set of slides so a little 

bit more detail about national core indicators and national core indicators for 

aging and disabilities. These were developed to measure service outcomes 

from people who are receiving services.

There’s information that is collected from both administrative and service 

coordination data but primarily what our emphasis is on these tools being inperson interviews by trained surveyors, face-to-face meetings to develop what 

we will be talking about later as patient-reported outcome measures.

I will also reference in a little bit that the national core indicators project 

which has been around for a bit longer has additional tools which are not faceto-face surveys but primarily we’re talking about the face-to-face surveys in 

the context of patient-reported outcome measures.

We do have protocols for both tools allowing for proxy response if the person 

is unable or unwilling to respond for themselves and as I’ve mentioned this 

being something that states really focus on, states may opt to add questions to 

the core set of questions that exist in the tool and then they would be 

implemented alongside the core set of questions.

There are standardized training materials and implementation protocols and 

the survey questions and the structure were developed and tested through 

multiple pilot phases. This is really demonstrating the two tools side-by-side 

or I should say the two projects side-by-side. 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 9

As the national core indicators and the national core indicators for aging and 

disabilities, one really key thing to focus-in on is that the focus population 

differs. The focus population for national core indicators is adults 18 and over

who are receiving services from their state developmental and disability 

support agency. 

And the focus population for national core indicators with aging and 

disabilities is older adults and adults with typical and other disabilities 

accessing publicly-funded services and there’s a list on the slide including 

Older Americans Act programs, Medicaid waivers, Medicaid state plan, statefunded programs, PACE, MLTSS programs and skilled nursing facilities.

So there are a number of funding streams that are then being monitored in 

terms of this quality measurement. As I’ve mentioned we have collaborators 

on both these projects and then one other thing that I do want to highlight 

right now is that the national core indicators was launched in 1997. This 

current survey cycle is the 20th year of data collection for in-person surveys. 

The national core indicators for aging and disabilities began implementation 

in 2015 so it’s relatively new in terms of its implementation but in a minute I 

will just get into talking about how far it has gotten in terms of reach. One 

other area that I want to highlight is the national core indicators staff ability 

survey which is relatively new. 

This is again focusing-in on the disability and quality of the direct support 

professional workforce and when we talk in a little bit about the areas where 

it’s important to monitor quality in terms of assessing quality of services, this 

was an area that was determined to be a gap area in terms of understanding the 

ability of staff.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 10

So this program was developed, it’s a tool that actually collects data 

specifically from provider agencies so it has a different kind of methodology. 

All of these reports that I’m referencing are available and for example I’ll just 

say in 2016 the dataset for the staff’s ability survey included data from 21 

states.

So here’s the reach of national core indicators and national core indicators for 

aging and disabilities. The states that are colored-in in blue are those states 

that are participating in national core indicators only. 

The states that are colored-in in that weird brown color it looks like on my 

screen are states that are participating in both national core indicators and 

national core indicators for aging and disabilities.

So demonstrating here that actually in the span of a couple of years national 

core indicators for aging and disabilities have been taken-up by quite a 

number of states. This actually needs to be updated. 

Utah has recently been added as participating in national core indicators for 

aging and disabilities so that’s another state that we’ll need to change in the 

color, the color representing participating in both surveys. National core 

indicators and national core indicators for aging and disabilities have similar 

lists of domains. 

We show them here on this slide and I’m not going to spend a great deal of 

time except to say that Human Services Research Institute projects to have a 

question-by-question crosswalk indicating where there are areas of overlap 

between national core indicators and national core indicators for aging and 

disabilities both in domains and in particular questions.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 11

And the current reporting and data use of national core indicators I will say 

that national core indicators and national core indicators for aging and 

disabilities have publicly-available reports.

This is one of the areas where we feel like this is important and the states that 

participate feel it’s important to have the data and the results publicly 

available so these are posted on the Website. There are reports posted on the 

Website.

I just want to reference that the reports do include state sampling details and 

quite a lot of detail about the methodology as well so that would be a good 

place to go and reference that. I also want to say that states may choose to use 

larger samples when they do the survey, the implementation of either national 

core indicators or national core indicators for aging and disabilities. 

For the purpose of being able to understand substate entities for example 

managed-care organizations if they sample so that they can look at outcomes, 

services and some characteristics of services of people who are receiving 

services based on at the level of some substate entities.

So this does this is possible if states work on establishing and appropriate 

sample size and appropriate methods for doing that. States may choose to 

share those reports but that’s up to the states. 

And generally states are using the data to benchmark to other states and for 

data-based quality monitoring and planning related to outcomes and for 

reports to a number of stakeholders.

One other piece of context that I wanted to add to what was presented earlier 

by Heather as well is another piece of another report that came-out about a 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 12

year ago based on a process looking at quality and home and communitybased services.

This was the result of a multi-stakeholder committee that was convened by the 

National Quality Forum and the plan the goal was to develop a shared 

understanding and approach to measuring quality in home and communitybased services.

This included an environmental scan and some details about the 

characteristics of high-quality home and community-based services and these 

characteristics include things such as a person-driven system, social 

connectedness, balance of personal safety and dignity of risk, (asset) supplies 

and support and a system that supplies and supports an appropriately skilled 

workforce that is stable.

And that is why I referenced this stability report as one of the various were 

that can be used to monitor this as well as outcome-oriented data to all 

stakeholders. 

So in the last few minutes that I have I do want to talk about I want to 

reference specifically the projects that we are doing now funded through the 

Administration on Community Living to develop home and community-based 

service quality measures from national core indicators and national core 

indicators for aging and disabilities.

These tools have been developed with support from ACL along the way and 

this project really has some key areas that we’re focusing-in on to advance the 

development effort, to enhance home and community-based service quality 

measurements.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 13

So the project activities briefly are to focus-in and I should say this is listed as 

HSRI human service research project activities. These are all projects - all 

these project activities - are being done in concert and collaboration with our 

partners the National Association for State DB Directors as well as (nasua).

So the project activities are to take a little bit of a look back to synthesize and 

publish evidence and protocols to revise existing and/or existing measures or 

develop new measures of person-centered planning and this is something I 

will get into in a little bit. 

The third main focus is to submit at least 20 measures for National Quality 

Forum endorsement and then the fourth is to continue the work and to support 

the expansion of our work to support states to use national core indicators and 

national core indicators for aging and disabilities in terms of their quality 

monitoring and quality processes so the first activity as I’ve mentioned is to 

synthesize and publish evidence and protocols. 

What I’d like to focus-in on the second bullet we have existing evidence of 

good psychometric properties in the development of these tools we have done 

extensive testing and this component of the work is to focus-in on the 

synthesizing the testing that has been done to date and developing a plan for 

additional studies that we would like to complete.

The additional studies will focus-in on these three main areas in terms of a 

round of confirmatory cognitive testing of questions that are being used, 

analysis of understanding those who respond and those who opt not to 

respond. This is a random sample of service recipients but people of course 

are opting-in.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 14

It is a voluntary participation in the survey and so we would like to do some 

work to understand those who choose not to respond and we have process 

underway to do that and then we want to do some additional inter-rater 

reliability testing in several settings.

The additional work we’ll be doing is to again synthesize and update training 

procedures, requirements and protocols and that process is underway as well 

as to clarify with some of our states the sampling procedures and requirements 

across states.

The second activity that we are working on currently involves focusing-in on 

person-centered planning, the questions and the modules - questions and 

potentially a module - that mixes with both surveys.

This is in response to a demonstrated and recognized need that we have heard 

from a number of states that we have worked with that are interested and keen 

to be able to have an effective way to understand the extent to which their 

services are both person-centered services and that the planning is done in a 

person-centered way so this is work that we have underway.

We have done a number of consultation focus groups with state partners and 

our intent is that we will be piloting questions with states in the current survey 

cycle. The third activity that we’re working on will be to submit to the 

National Quality Forum a number of measures for endorsement.

Some of the foundation (unintelligible) that I have mentioned already in the 

(unintelligible) of activities are leading to this as well as work that we’re 

doing to align with staff areas that exist. 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 15

We are working quite closely with our colleagues at the RTC on outcomes 

measurement who you’ll be hearing from shortly to make sure that as we are 

working to develop measures that this is done in a synergistic way because 

this is and learning from the work that they have been doing as well as sharing 

the work that we have underway to contribute to their work.

Our targeted timeline for submission of our first round is late 2018 and briefly 

I just wanted to mention to those who are interested and participating, there’s 

a great deal of information available on the National Quality Forum Website 

laying-out what the criteria the general criteria are for endorsement of 

measures.

But primarily there are these larger (insets) that we’re focusing-in on in terms 

of demonstrating this. In terms of the feasibility for the tools the national core 

indicators and the national core indicators for aging and disabilities, the last 

bullet. Well, we’ve got good we’re able to demonstrate feasibility in that the 

tools are exist in the field.

We have a great deal of resources that we’ve pulled together for both to 

demonstrate the importance as well as the scientific accessibility and the 

usability so these are all cases that we’re pulling together. Some of it is a 

question of demonstrating in the context and the language that aligns with 

what is being sought by the National Quality Forum.

Finally the other piece of work that we’re doing as a portion of this ACLfunded work includes as I have mentioned technical assistance as this involves 

enhanced training for state quality staff on understanding and using data that 

is available, how to integrate not just as a once-a-year review of a report but 

rather into developmental plan for quality improvement.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 16

We are also generating both data briefs and what we’re calling data nuggets 

which are short summaries which can be developed and shared and some 

states have adopted this kind of format in terms of their general reporting.

We’re also working on working with states to help them focus-in on sampling 

procedures to ensure comparability and allow states to think about how they 

would sample if they want to look at a particular issue within their state and 

looking at how to use these results for system improvement.

Finally we are looking to support the recruitment of additional states. There 

are four states remaining that are not participants in national core indicators 

and there are additional states that we’re looking to work with for national 

core indicators for aging and disabilities as well.

There’s plenty of information available on the national core indicators 

Website and the national core indicators for aging and disabilities Website and 

with that I’d like to say thanks to the Administration on Community Living 

for supporting this work to improve measurement and for the quality of 

support for people with disabilities living in the community.

And also thank you to our partners, (nasuad) and (nasdes) and to the people in 

the multiple states that we’ve worked with for their ongoing commitment to 

improving quality and with that I will pass the green ball back to Brian.

Meredith Raymond: Thank you, Alex, very much appreciate your help with this presentation. 

Now I’ll introduce Dr. Brian Abery and Dr. Renáta Tichá. Dr. Brian Abery is 

the Co-Director of the University of Minnesota’s rehabilitation and research 

and training center on HCBS outcome measurement and the educational 

assessment and intervention program at the University’s Institute on 

Community Integration.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 17

He is an adjunct faculty member within the Institute on Child Development 

and special education programs at the University of Minnesota and serves as 

the principal investigator or co-investigator on a number of (nydler) and U.S. 

State Department projects focused on enhancing the inclusion of people with 

disabilities.

Dr. Renáta Tichá is a Research Associate at the University of Minnesota’s 

Institute on Community Integration with extensive experience in services to 

people with disabilities as a program evaluator, measurement expert and 

researcher. She holds a doctorate in special education and direct projects 

involving survey assessment and intervention research for people with a 

variety of disabilities.

Dr. Tichá has authored a number of publications on the national core 

indicators project we just heard about as well as serving as a principal 

investigator or co-investigator on multiple projects being undertaken as part of 

the UMN’s RTC on community and the RTC on HCBS outcomes 

measurement. Thank you Brian and Renáta.

Brian Abery: Thank you for the opportunity to share the work we’re doing through our 

research and training center on HCBS outcome measurement. Renáta and I 

very much appreciate this opportunity.

We are a five-year research and training center funded by the National 

Institute for Disability, Individual Living and Rehabilitation Research. We 

are a consortium in the sense that we have a number of primary and secondary 

partners to support us with this work.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 18

Our primary partners are the RTC at the University of California San 

Francisco led by Steve Kaye, our colleagues at Temple University including 

(Mok Selver), (Beth) and (Retrin), our colleagues at Ohio State University 

(John Corrigan) with his expertise in TBI and Joe Caldwell from the National 

Council on Aging.

As Alex Bonardi indicated, we also have a close relationship with HSRI and 

will be working with them so that we can collaborate with respect to testing 

new and refined measures. 

Now our purpose is a little bit different than those of our colleagues at HSRI 

in the sense that while their development of the NCI set of instruments is 

really focused on state-level monitoring tools, ours really is more focused on 

kind of multi-level monitoring. 

So to summarize we really are trying to improve the way we measure quality 

of home and community-based services and the subsequent personal outcomes 

that people with disabilities experience as a result of the support that they 

receive. Our starting point is with the National Quality Forum HCBS 

outcome measurement framework. 

We thought that given this framework is new and is probably the first 

comprehensive attempt to take a look at what is most important to measure, 

the first thing that we wanted to do is to do some content and social validation 

of that framework through a national study with four different stakeholders 

and those included large groups of individuals with disabilities, family 

members, providers and program administrators.

Our next step focused on doing a gap analysis, taking a look at the recentlyreleased NQF framework and existing measures with respect to their content, 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 19

the extent to which the constructs that were attempting to be measures were 

well-saturated, the person-centeredness of existing measures and measure 

concepts and their relevance to multiple disability populations.

As a third phase or step in our process, we have been working on kind of 

prioritizing measures that either need refinement or new measures that need to 

be developed taking a look at the gaps between prioritized NQF domains and 

subdomains and the existing measures that we have identified focusing on 

importance, feasibility and usability.

We’re actually now in the stage of measure development which is an iterative 

process in which we are taking a look at the most promising measure concepts 

that are out there where there are gaps developing new measures and then 

we’ll be pilot-testing including significant cognitive testing of these measure 

concepts with all disability populations to determine both the psychometric 

and other properties.

So we’ll be focusing on reliability, validity, sensitivity to change and 

applicability across disability groups. Now our approach to measure 

development is to look at a couple of other things in addition to the quality of 

the measures. 

Number 1 has to do with the administrative factors so we are working to really 

take a look at best practices in training protocols for interviewers, 

administration approaches, sampling and data handling approaches so that 

when we have good measures we can be assured that they are going to be 

implemented in a high-fidelity fashion.

We will also be looking at risk adjusters or what in the educational field we 

typically refer to as covariates. As part of our work, we are identifying 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 20

potential risk adjusters at the individual level and the systems or 

environmental level and then we’re going be testing promising risk adjusters 

in a national sample.

So again we started our work with an understanding of the National Quality 

Forum’s framework for HCBS outcome measurement and as you all know, 

that has 11 domains and within each of those domains there are two to seven 

subdomains.

What we wanted to note is whether stakeholder groups of individual with 

disability, family members, program administrators and providers generally 

agreed with the domains and subdomains outlines by the National Quality 

Forum as most important and to figure-out whether they differed in how they 

prioritized the domains and subdomains.

So we went-out using a process that we refer to as participatory planning and 

decision-making and with each stakeholder group which consisted of 

anywhere from four to 10 individuals we took a very close look or deep dive 

into the NQF framework.

Having the group provide us with information regarding the operational 

definitions used, whether they were gaps, missing domains or subdomains and 

to have them really evaluate whether the subdomains accurately reflected 

what we’re measuring at each domain level.

We had people provide importance weighting as part of the work that we’re 

doing in this area from zero not important at all to measure to 100 most 

important to measure and as you can see from this slide, we attempted to get a 

national sample or as close as we could get to that sampling individuals from 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 21

all areas of the country and from all of those stakeholder groups that I have 

previously discussed.

And this is the type of information that we were provided by the groups with 

respect to kind of the quantitative aspect of our groups. Taking a look here 

you can see across the NQF domain that there were some domains that across 

all stakeholder groups there was high degree of consensus that these are the 

most important to measure.

Others that were considered to be somewhat less important to measure but I 

think one of the things that we did learn from this exercise was that overall the 

National Quality Forum did come-up with a framework that pretty much 

indicated what the different stakeholder groups are thinking of as the most 

critical aspects of home and community-based services to measure both with 

respect to quality assurance and the personal outcomes that people experience.

Now although these was consensus, you know, there were some differences 

with respect to what individuals thought were most important to measure. As 

you can see on this slide, person-center service planning for nation, service 

delivery and effectiveness, choice and control and human and legal rights 

were above average with respect to the ratings that they received.

We have four kind of domains which were in the average range, workforce, 

equity, holistic functioning and community inclusion and three subdomains of 

the National Quality Forum framework which were a bit below average. 

Again, all of the weightings that were assigned by the groups indicated that all 

of these were important to measure. What we were focusing-on is which did 

they feel were most important to measure.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 22

One of the things that we did find-out as we looked at the various domains 

and subdomains is that the subdomain level there were certain aspects of each 

of these domains that stakeholders agreed were more important to measure 

than others so for example with respect to system performance and 

accountability, you know, financing and service delivery structures were 

considered to be most important.

The use of evidence-based practice, somewhat less important. Looking at 

equity, you can see that there were two subdomains that our participants saw 

as more important to measure than the others. Again I want to reemphasize 

that all of these were considered as important to measure, some just more 

important than others.

One of the things that we found was that when we took a look at the personal 

outcomes experienced by persons with disabilities, we tended to get much less 

variation across subdomains. 

Here’s just an example of choice and control that all the aspects included in 

the National Quality Forum’s outcome measurement framework which were 

identified by that select stakeholder group reviewed by a much broader group 

of 350 some-odd individuals from all the stakeholder groups as important to 

measure - as equally important to measure - I should say.

The same with respect to community inclusion. I know I’m going through 

these slides quickly. We will be actually having a Webinar this fall which 

will in much greater detail go over the results of this first content and social 

validation study focusing on the National Quality Forum’s framework.

We also found that among the stakeholder groups at least in some areas there 

were specific stakeholder groups that rated some areas as either more or less 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 23

important than others. As you can see here in this slide, choice and control 

was rated above average by policymakers, individuals and providers, 

somewhat below average by family members.

We have a similar type of response to our questions about the human legal 

rights piece and again what people were actually weighting was the 

importance of each of these aspects of the National Quality Forum’s 

framework to measure.

So the main takeaway from our stakeholder input is that we do have initial 

evidence of the social and content validity of the NQF framework where the 

stakeholder groups were represented. We have some additional suggestions 

that the groups recommended with respect to domains and subdomains.

Our groups for example felt that employment was not highlighted enough for 

most of those groups in the National Quality Forum’s framework that we 

needed to take a closer look at workforce turnover and at transportation. Also 

the differences in importance weightings suggests that the framework may 

apply somewhat differently to the various disability groups.

Now these results were basically intended to drive our measure development 

process and the improvement of measures so that we could really look at those 

measures which were deemed of greatest importance and as I said we’ll have 

more information specifically about this aspect of our development process in 

a Webinar that we will make sure all of you on hear about which will be 

taking place this fall.

I’m now going to turn things over to my colleague Renáta Tichá who’s going 

to talk about the work that we have been doing with respect to gap analysis in 

this area.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 24

Renáta Tichá: So one of our charges was to look at existing instruments in the field of 

disability services (unintelligible) different types of disabilities and match 

them against the domains and subdomains of the NQF framework so with the 

help of our team here at the University of Minnesota we were able to review 

132 instruments across five target populations with disabilities.

And we were able to code these instruments based on the categories that you 

have listed on the slides, for example respondent types, as to what extent they 

are person-centered, domain and subdomain and then available also their 

properties. Now, one of our other sort of focuses here was to be sure to 

review instruments that are widely used, like NCI and NCIAD, but also 

instruments that are not as widely used but had more extensive psychometric 

studies that were done on them, for example, assessment tools that were 

developed as part of NIDILRR initiatives at universities. 

So this chart just gives you a sample of some of our findings of the GAP 

analysis. So this shows, for example, what percentage of the instruments 

coded covers certain content areas of NQF. There were instruments that --

about 23% of instruments cover the domain of community inclusion, for 

example, as opposed to 9% of those cover human and inalienable rights, and 

the other domains that you can see on the chart.

Now, this just really shows the coverage by the number of questions that were 

coded within the domains and sub-domains. It doesn't really speak to quality 

of those items, which is our next step in this particular process. This is 

another visual that might help to see what we have been working on. In the 

left hand corner are the domains that were covered by the tools, by the 

assessment tools. And then you have got the assessment tools' names at the Y 

axis.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 25

And then you can see sort of the saturation or the percentage of coverage of 

the particular domain and sub-domain by the different instrument. And again, 

this doesn't really take into account the quality of the coverage. It's just the 

sort of frequency account of the NQF coverage. There is another sort of map 

that shows you the coverage by type of disability and I just want to highlight 

some main findings of the GAP analysis. You can see based on the chart that 

I showed a few minutes ago that there is different level of coverage by domain 

and sub-domain. That shows that equity participation in system development 

by people with disabilities is covered the least as far as we could tell. 

Then there are also many items that can be classified as person centered but 

there are also many others that we coded as not fitting that particular 

description. There are many domains and sub-domains that don't include 

sufficient number of questions or questions of sufficient quality and that's an 

area we are currently working with Temple University to cover some of those 

gaps. And there were very few measures that have good psychometric 

properties or even existing psychometric properties, NCI being one of the 

exceptions and there were a few others that had at least psychometric 

properties under development. So that's an area that our project is working on 

for the next two years.

Brian Abery: I'm going to go through the next several stages of our project quite quickly so 

that there's time for questions. I do want to let you know, as I said earlier, that 

we really need to do a better job understanding measure administration to 

identify amongst existing outcome measurement programs what are the 

practices that are being implemented to actually collect this information, 

whether it be interview, whether it be interviews which are done on the basis 

of conversations or of a more structured type, to really understand how to best 

train individuals and what groups of individuals are best to collect this data. 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 26

Some states are using processes where an individual disability is the data 

collector or works together with a family member to collect the data.

So what we're going to be doing is conducting a number of case studies with 

the help of organizations such as HSRI and CQL to take a look at the 

methodological components needed to assure high quality data measured 

administration, to identify the strengths and challenges of outcome 

measurement programs and their impact on measure administration fidelity, 

and finally to identify factors that facilitate or are barriers to effective 

measurement implementation.

At this point in our process, we are kind of prioritizing our measures for 

development or the measure concepts we are going to be focusing on. So we 

basically have looked at all the domains and sub-domains of the NQF 

framework, taken a look at the feedback from study one, from stakeholders, 

and the gap analysis that took place in study two. And we are working with 

HSRI so we can minimize redundancy with respect to the work that they are 

doing through ACL to develop specific aspects of the NCI set of instruments. 

So as part of this process, what we have done is to identify within the 

domains, those domains which, based on the gap analysis and stakeholder 

input, our stakeholders felt were most important to measure. And taking a 

look within those domains at the sub-domains, which either (unintelligible) 

gaps with or our stakeholders said, we need to do a better job of measuring 

these. 

And these are the domains which were prioritized by both our leadership 

group at the RTCOM and our national advisory group, and they will be the 

measure concepts that we are going to initially be working on over the next 

three years of our research and training center. The measure development 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 27

process we are using is an iterative process where we are developing and 

revising items based upon the results of our initial work. We are, as part of 

this process, doing extensive review of existing contextual frameworks for the 

measure concepts to be developed,

One of the things that we found in doing our gap analysis is that a lot of the 

tools that are out there really haven't taken into consideration existing theory, 

existing conceptual frameworks in these areas. For example, personal choice 

and control, self-determination, social inclusion. And as a result, while we 

have items, while we have measure concepts out there, they don't really do a 

good job of saturating what most people, including people with disabilities 

mean by the construct in question, such as social inclusion.

As part of this process, we are developing operational definitions for the key 

components of the measure concepts based upon NAC's existing frameworks 

when they exist. And then as part of this iterative validation process, we are 

refining existing measures that are promising, putting them through kind of a 

content expert review process, and through a pilot study that will include 

individuals with all of the five different disability groups that we're working 

with, doing extensive cognitive testing with each of these.

So this just gives you an idea of kind of the measure lifecycle that we are 

following. We are at the point now where we have gotten through kind of the 

conceptualization and we are at the measure specification phase. And we will 

quickly be moving into our initial pilot testing within the next six months. I'm 

now going to turn things over to my colleague, who is going to talk about the 

next step, or future step in our measure development process.

Renáta Tichá: Yes, just for the sake of time, very briefly, after we have piloted, done 

cognitive testing, and revised some of the items or questions within the 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 28

measure concepts, we then will be working very closely with many of our 

partners across the country, including our partner on this call, to validate those 

measures with hopefully 1,000 individuals across the five categories of 

disability and looking at the main psychometric properties that are listed on 

the slide that we already discussed.

And lastly, one of our focuses is not only looking at outcomes and predicting 

variables for the measures that we are studying, but also risk adjustors. So we 

are currently at the stage where we have done a comprehensive review of the 

literature of risk adjustors across the populations and outcomes, working on 

putting it into a manuscript form to inform the field about the risk adjustors 

for various situations. So we'll have that available sometime in the fall.

Brian Abery: Thank you. I'm now going to turn things back over to Meredith and hopefully 

we have a few minutes for questions.

Meredith Raymond: Yes, thank you so much (Brian) and (Renata). Operator, just to be 

conscious of time, if participants have to leave the webinar, please feel free. 

We are going to at least have a time for two questions. So operator, could you 

please provide directions on participants can enter the Q&A queue?

Coordinator: Thank you. For questions from the phone, press star one. Please unmute your 

line and record your name to be introduced. Again, for questions or 

comments from the phone you may press star one please. 

And currently, we're showing no questions at this time from the phone. 

Meredith Raymond: Okay, I'll give it one more minute and then we have two questions here if 

we don't have any questions.
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 29

Coordinator: Thank you. Again, as a quick reminder for questions from the phone, 

questions or comments, press star one please. Thank you. We're showing no 

questions from the phone.

Meredith Raymond: Thank you. We'll start with our question here. (Alex), (Brian), (Renata), 

how can local and state level organizations and providers be involved in 

HGBS quality measure development.

Brian Abery: One of the things that's going to be essential for us to have quality measures is 

to do extensive testing on the measures to make sure that they're feasible, 

they're usable and they possess high quality psychometric properties. At any 

level, whether it's a state level, whether it's a provider level, being willing to 

be involved in this process will make a significant contribution to the field. 

Meredith Raymond: Thank you, (Brian). For our next question, in the absence of endorsed 

measures covering all areas of HGBS quality, what measures should local and 

state level or organizations use?

Alexandra Bonardi: This is (Alex) and I would say that we've been talking about the 

frameworks and the concepts around measures that are in development. I 

think that local and state organizations can certainly be adopting measures that 

are in development and actually following the process that NQF is going 

through in terms of reviewing and endorsing measures would actually help 

providers understand I think where things are being measured, where there are 

gaps and where there's work that they have to continue, that we all have to 

continue to do.

Brian Abery: And to add to what Alex is saying, I think we here at the University of 

Minnesota want to make sure that people educate themselves about the 

measures that they're being asked to use so that they know as each measure 
NWX-HHS-AOA-1

Moderator: Meredith Raymond

8-16-2017/12:30 pm CT

Confirmation # 5110059

Page 30

has its strengths and its limitations what those strengths and limitations are, 

why was the measure developed, for what is its intended purpose. What 

questions was it designed to answer so they don't end up using measures for 

one purpose which they really weren't developed for.

Meredith Raymond: All right, thank you very much for the answers to those questions. That 

was extremely helpful and thank you for your presentations. Now that we're 

five minutes over I'm going to go ahead and close our call today. Thank you 

everyone for joining. A PowerPoint will be sent out to everyone who 

RSVPed for the webinar and it will also -- the webinar materials will also be 

posted to our website early next week. So thank you again everyone for your 

participation and please stay tuned for future webinars and discussion on 

HGBS quality. Thank you.

Coordinator: Thank you for your participation. That does conclude today's conference. 

You may disconnect at this time. 

END