The Webinar Will Begin Shortly. 

Please remember to mute your phone lines and 

computers. 

Please use this link for closed captioning: 

http://www.fedrcc.us//Enter.aspx?EventID=3347319&CustomerID=321

This link is also included in the invite. 

If you have any questions or concerns please let us know VIA chat. 
HCBS Quality Measure Development 

Webinar 

August 16, 2017 2pm EST
Webinar Agenda 

I. Introduction 

II. Presentations:

I. Heather Menne, The Administration for Community Living 

II. Alexandra (Alixe) Bonardi, Human Services Research Institute 

III. Brian Abery, University of Minnesota Research Rehabilitation 

Training Center on Home and Community-Based Services 

Outcomes Measurement

III.Question and Answer Session

IV.Closing

3
The Basics of Quality Measurement 

Social Science Analyst, 

Office of Performance and Evaluation, ACL 

Heather Menne, PhD

4
Why is Quality Measurement Important? 

The original mission of program evaluation and 

performance management was to assist in improving the 

quality of programs. 
GPRA-Modernization Act of 2010

●The GPRA Modernization Act of 2010 

(GPRAMA) aims to ensure that agencies 

use performance information in decision 

making and holds them accountable for 

achieving results and improving 

government performance. 
Office of Management and Budget

●In Memorandum M-13-17 the Office of Management and 

Budget provides guidance to Federal agencies about 

“harnessing evidence and evaluation”

● It discussed delivering “a smarter, more innovative, and 

more accountable government for citizens. 

● An important component of that effort is strengthening 

agencies' abilities to continually improve program 

performance by applying existing evidence about what works, 

generating new knowledge, and using experimentation and 

innovation to test new approaches to program delivery.” 
FY 2018 Budget Blueprint

• The Administration will take an evidence-based approach to 

improving programs and services—using real, hard data to 

identify poorly performing organizations and programs. We 

will hold program managers accountable for improving 

performance and delivering high-quality and timely services to 

the American people and businesses...”
Why Measure? 

For many reasons, including: 

• Measures drive improvement. 

• Measures inform consumers and other stakeholders. 

• Measures influence payment. 

Measurement is a quality improvement tool, not an end in 

and of itself 

• A performance measure is a way to calculate whether and how 

often the system does what it should. 
What to Measure?

Not everything that counts can be counted, and not everything 

that can be counted counts

But...

You can’t improve what you don’t measure
Areas for Measurement

1. Quality 

i. Structures of care 

ii. Processes of care 

iii. Outcomes 

i. Intermediate clinical outcomes 

ii. Health outcomes (mortality, complications, etc.) 

iii. Patient-reported outcomes (experience, functional status, engagement, 

quality of life, etc.) 

2. Resource use/cost 

3. Efficiency (combination of quality and 

resource use) 
How to Create Measures?

In order to be most effective, objectives should be 

clear and leave no room for interpretation. S-M-A-R-T 

is a helpful acronym for developing objectives that 

are 

●specific, 

●measurable, 

●achievable, 

●relevant, and 

●time-bound

● Source: http://www.cdc.gov/healthyyouth/evaluation/pdf/brief3b.pdf 
Developing HCBS Quality Measures 

from

National Core Indicators (NCI)

and

National Core Indicators for Aging 

and Disabilities (NCI-AD)

ACL Quality webinar

August 16th, 2017 
NCI and NCI-AD

• Two distinct quality monitoring tools with 

common protocols, implementation 

methodology, measurement domains

• State-level quality monitoring. Some states opt to 

extend quality monitoring to sub-state entities.

• HSRI partners for development, administration, 

and use of quality data 

– NCI a partnership between NASDDDS-HSRI-States

– NCI-AD a partnership between NASUAD-HSRI-States

14
NCI and NCI-AD

• Measure service outcomes from individuals receiving 

services

 Includes some administrative and service coordination/case 

management data

 In-person interviews by trained surveyors (face to face) 

(PROMs)

 Protocols allow for proxy response if person unable or 

unwilling to respond themselves

States may opt to add questions to core set

Survey questions and structure developed and tested 

through multiple pilot phases 

Standardized training materials & implementation protocols

15
National Core Indicators (NCI)

• Focus population: Adults (18+) receiving 

services from state DD support agency. 

• HCBS waiver plus state-only funded 

programs. 

• Collaboration between NASDDDS and 

HSRI

• Multiple tools

 Adult In-Person Survey (IPS)

 Family Surveys 

 Staff Stability survey (detail on next 

slide)

• Launched in 1997, 2017-2018 is 20th year 

of data collection (for in-person survey).

• Current participation: 46 states plus DC, 

and 21 regional centers.

National Core Indicators for 

Aging and Disability (NCI-AD)

• Focus population: Older adults and adults 

with physical and other disabilities 

accessing publicly funded services in: 

– Older Americans Act Programs

– Medicaid Waivers

– Medicaid State Plan

– State Funded Programs

– PACE

– MLTSS

– Skilled Nursing Facilities

• Began development in 2012; began 

implementation in 2015

• Collaboration between NASUAD and HSRI

• Adult in-person survey only

• 2017-2018: Third year of data collection

• Current participation: ~20 states

16
NCI Staff Stability Survey

• Tool: stability and quality of the DSP workforce 

providing supports to adults with ID/DD

Piloted in 2014, rolled out in 2015. 

Data can be used at state level: inform 

policy/programs, develop workforce initiatives, 

compare with other states, provide context for 

consumer outcomes

Info. collected: types of supports provided, 

turnover rates, vacancy rates, wages, benefits, 

recruitment and retention strategies

• 2016 dataset includes data from 21 states

17
18

RI

AK

HI

OR

NV

UT

AZ

SD

NE

KS

AR

LA

WI

IN

KY

TN

GA

SC

VA

ME

MS AL

WV

CA

ID

MT

WY

NM

TX

ND

OK

MN

IA

MI

IL

MO

OH

FL 

PA

NY

CO

NC

WA

VT

NH

MA

MD

DE

NJ

DC

RI

CT

Participating in both NCI and NCI-AD

Participating in NCI Only

Not Participating

States participating in NCI and NCI-AD
NCI and NCI-AD Domains

NCI

 Community Inclusion

 Choice and Decision Making

 Relationships

 Satisfaction

 Service Coordination

 Work

 Self-Determination

 Access 

 Health

 Medications

 Wellness

 Respect and Rights

 Safety

NCI-AD

 Community Participation

 Choice and Decision Making

 Relationships

 Satisfaction

 Service Coordination

 Care Coordination

 Work

 Self-Direction

 Access

 Health Care

 Medications

 Wellness

 Rights and Respect

 Safety

 Everyday Living

 Affordability

 Future Planning

 Control
Current reporting and data use

• National and state reports publicly available 

www.nci-ad.org www.nationalcoreindicators.org

• Reports include state sampling details

• Sub-state entity reports may be shared by states

• Benchmarking to other state performance, databased quality monitoring of support system, 

reporting outcomes to legislature, stakeholders.

20
In context: Quality In Home and 

Community Bases Services

• Multi-stakeholder committee 

convened by NQF

• Developed shared 

understanding and approach

• Environmental scan, and 

detailed characteristics of High 

Quality HCBS

• Report available at this link: 

http://www.qualityforum.org/Publications/2016/09/Quality_in_Home_and_CommunityBased_Services_to_Support_Community_Living__Addressing_Gaps_in_Performance_Measurem

ent.aspx

21
NCI and NCI-AD 

Measure Development

• ACL funded project (2016 – 2021):

“Developing HCBS Quality Measures from NCI 

and NCI-AD”

Advance development efforts to enhance HCBS 

quality measurement.

Emphasis on developing PRO-PMs, expansion of 

use of tools, TA to states in use of data. 

22
HSRI Project activities

1) Synthesize and publish evidence and protocols

2) Revise existing and/or develop new measures of 

person-centered planning process to adapt to 

current expectations

3) Submit at least 20 measures for NQF 

endorsement from NCI-AD and NCI surveys

4) Technical Assistance to States to expand use of 

NCI and NCI-AD

National Core Indicators (NCI) 
Activity #1 Synthesize/publish 

evidence and protocols

– Existing evidence of good psychometric properties

• Extensive testing conducted to date (reliability, validity)

– Additional testing and analysis planned

• Confirmatory cognitive testing

• Analysis of responders vs. non-responders

• Inter-rater reliability testing

– Training procedures, requirements and protocols

– Sampling procedures and requirements 

– Interviewing protocols

– Implementation consistency & validity across states

National Core Indicators (NCI) 
Activity #2: Person Centered 

Planning questions/ module

• Developed in consultation with state 

partners

• NCI and NCI-AD modules

• Piloting with states in 2017-2018 cycles

National Core Indicators (NCI) 
Activity #3 Submit to NQF for 

Endorsement

• Identify NCI and NCI-AD measures (20) to advance for 

submission (PRO-PMs)

• Develop measures for submission

• Alignment with other HCBS measure development 

efforts underway – UMN RRTC/OM 

– Planning for synergistic measure testing through measure 

prioritization, measure pilot work, data collection. 

• Targeted timeline – submission first round late 2018. 

National Core Indicators (NCI) 
NQF Endorsement is based on 4 

criteria

• Important to measure and report (focus on priority areas with 

good evidence)

• Scientifically acceptable (consistent (reliable) and credible (valid) 

measurement of quality). 

• Useable and relevant (intended users understand the results of the 

measure and useful for quality improvement / decision making.

• Feasible to collect.

https://www.qualityforum.org/Measuring_Performance/ABCs/What_NQF_Endorsement_Mea

ns.aspx

National Core Indicators (NCI) 
Activity #4 Expand Use of NCI 

and NCI/AD by States

• Technical Assistance

• Enhanced training for state quality staff.

• Data briefs and ‘data nuggets’

• Sampling procedures

• Implementing surveys and analyzing results

• How to use results for system improvement

• Support recruitment of additional NCI and NCI-AD 

states

National Core Indicators (NCI) 
For further information

NCI

www.nationalcoreindicators.org

Alixe Bonardi – NCI project director (HSRI) abonardi@hsri.org

Mary Lou Bourne – NCI project director (NASDDDS) 

mlbourne@nasddds.org

NCI-AD 

www.nci-ad.org

Julie Bershadsky – NCI-AD project director (HSRI) jbershadsky@hsri.org

April Young – NCI-AD project director (NASUAD) ayoung@nasuad.org

29
on home and community based services outcome measurement

on home and community 

based services outcome 

measurement
on home and community based services outcome measurement

RRTC/OM partners and funding

 Primary Partners

 University of Minnesota – Institute on Community Integration

 University of California–San Francisco 

 Temple University

 The Ohio State University

 National Council on Aging

 Additional Partners

 HSRI

 Funded by:

 National Institute on Disability, Independent Living and 

Rehabilitation Research (NIDILRR)
on home and community based services outcome measurement

RRTCOM: Driving Purpose

To improve the way we measure the 

quality of home and community based 

services & subsequent personal 

outcomes experienced by adults with 

disabilities 
on home and community based services outcome measurement

Our Approach to Measure Development

• Starting Point: NQF HCBS Outcome Measurement Framework 

– Social Validation of NQF framework through national study with 4 

stakeholder groups

• Individuals with disability

• Family members

• Providers

• Program administrators

– Refinements recommended by stakeholder groups made to framework 

• Gap Analysis: NQF framework and existing measures:

– Content (domain & subdomain)

– Saturation of constructs: Do existing measures truly assess all critical 

aspects of the construct in question

– Person-centeredness of existing measures/measure concepts

– Relevance to multiple disability populations
on home and community based services outcome measurement

Our Approach to Measure Development

• Prioritization: Gaps between prioritized NQF domains & subdomains and

existing measures identified.

– Prioritized based on Stakeholder Input, Gap Analysis plus RRTC/OM & NAG 

process ratings of:

• Importance

• Feasibility

• Usability

• Measure Development: Iterative process

– Measures revised (when necessary)

– New measure concepts developed to fill domain & subdomain gaps

– Pilot testing including cognitive testing with all disability populations 

• Determine psychometric and other properties of measures to ensure:

– Reliability

– Validity

– Sensitivity to change

– Applicability across disability groups
on home and community based services outcome measurement

Our Approach to Measure Development

• Administrative Factors: Identification of administration factors 

that influence data quality including: 

– Training protocols

– Administration approaches

– Sampling

– Data handling

• Risk Adjustment:

– Identify potential risk adjusters:

• Individual/personal

• Systems/environmental

– Test use of promising risk adjusters on national sample 
on home and community based services outcome measurement

NQF FRAMEWORK FOR HOME & 

COMMUNITY BASED SERVICES

OUTCOME MEASUREMENT

11 Domains

2-7 Subdomains

Choice and 

Control Human and Legal 

Rights

Community 

Inclusion

Holistic Health 

and Functioning

Workforce

Caregiver Support

Person-Centered 

Service Planning 

and Coordination

Service Delivery & 

Effectiveness

Equity

System 

Performance & 

Accountability 

Consumer 

Leadership in 

System 

Development 

National Quality Forum Framework
on home and community based services outcome measurement

Questions

• Do stakeholder groups generally agree with the domains and 

subdomains outlined by the NQF?

• Do stakeholder groups or disability populations differ in how 

they prioritize NQF domains and subdomains?

• Stakeholder feedback re: domains and subdomains present in NQF 

framework? 

– Operational Definitions

– Gaps/missing domains/subdomains

– Do subdomains accurate reflect what we are measuring at domain level 

(concept saturation)

• How important is to measure each given element of the framework to truly 

capture the quality of your HCBS services? What is most important to 

measure?

• How do these elements of service quality impact the disability community?

• Importance weightings: 0-100 Scale
on home and community based services outcome measurement

Participants: Study 1
on home and community based services outcome measurement

Note: n = 277
on home and community based services outcome measurement

PPDM Priority Ratings for NQF Domains

Note: n = 277

Domain M SE

Person-Centered Service Planning and Coordination 94.9 0.62

Service Delivery and Effectiveness 94.9 0.60

Choice and Control 94.9 0.59

Human and Legal Rights 94.5 0.56

Workforce 92.8 0.89

Equity 92.6 0.70

Holistic Health and Functioning 91.9 0.67

Community Inclusion 91.5 0.69

System Performance and Accountability 89.8 0.98

Consumer Leadership in System Development 89.3 0.87

Caregiver Support 89.0 0.92

Above 

Average

Average

Below 

Average
on home and community based services outcome measurement

System Performance & Accountability
on home and community based services outcome measurement

Equity
on home and community based services outcome measurement

Choice and Control
on home and community based services outcome measurement

Community Inclusion
on home and community based services outcome measurement

Choice and Control by Stakeholder Type

• Families rated 

as average.

• All other groups 

rated as above 

average.
on home and community based services outcome measurement

Human and Legal Rights by Stakeholder Type

• Families rated 

as average.

• All other 

groups rated 

as above 

average.
on home and community based services outcome measurement

Main Takeaway: Stakeholder Input

• Provides initial evidence of social validity of the NQF framework 

with stakeholder groups represented

– Some additions at domain and subdomain level recommended 

for inclusion groups e.g.,

• Employment

• Workforce turnover;

• Transportation

– Differences in importance weightings suggests that the 

framework may apply differently to various disability populations

• Results meant to drive measure development and improvement of 

measures deemed of greatest importance

• Webinars under development
on home and community based services outcome measurement

Gap Analysis

• Determining coverage of NQF framework domains & subdomains by 

existing measures

• Deconstructed 132 assessment instruments across the 5 target 

populations

• 7,893 items coded across all surveys

– Items coded by two researchers on basis of: 

• NQF domain/subdomain

• Response options provided

• Respondent

• Person-centeredness

• Psychometric properties (when available)

• Development of interactive web data-base
on home and community based services outcome measurement

Community Inclusion

23%

Holistic Health and 

Functioning

16%

Service Delivery and 

Effectiveness

11%

Choice and Control

18%

Person-Centered 

Planning and 

Coordination

8%

Workforce

10%

Human and Legal Rights

9%

Caregiver Support

3%

Equity

1% Consumer Leadership in 

System Development

0%

System Performance and 

Accountability

1%

Note: Numbers represented percent of total items coded (n = 6673)
on home and community based services outcome measurement

Instrument Heat Map

Instrument NCI AD NCI ACS PEONIES PLQ PES-HCBS TUCPM PES-MRDD MFP

Choice and Control 26 26 32 38 14 26 21 16

Human and Legal Rights 22 18 33 6 16 0 9 14

Community Inclusion 15 56 34 71 8 78 11 9

Holistic Health and Functioning 56 29 33 6 5 0 2 14

Workforce 18 9 2 0 32 0 21 12

Caregiver Support 0 0 0 0 0 0 0 0

Person-Centered Planning and 

Coordination 42 4 10 11 13 0 13 9

Service Delivery and Effectiveness 56 10 10 0 21 0 14 13

Equity 11 11 2 0 2 0 0 0

System Performance and 

Accountability 0 0 0 0 0 0 0 0

Consumer Leadership in System 

Development 0 0 0 0 0 0 0 0

Total Items 246 163 156 132 111 104 91 87
on home and community based services outcome measurement

Population Map (16 instruments)

Aging IDD MH PD TBI General

Caregiver Support 21 118 0 61 0 0

Choice and Control 130 158 124 83 0 1

Community Inclusion 184 286 278 131 41 87

Consumer Leadership in 

System Development 0 0 0 0 0 0

Equity 39 49 28 13 0 1

Holistic Health and 

Functioning 117 138 63 149 1 71

Human and Legal Rights 192 197 161 78 0 1

Person-Centered Planning 

and Coordination 142 139 100 79 0 6

Service Delivery and 

Effectiveness 164 185 108 112 0 0

System Performance and 

Accountability 0 0 0 0 0 0

Workforce 36 68 13 56 0 0
on home and community based services outcome measurement

Main Takeaways

1) Existing items frequently address issues that span multiple domains 

and/or subdomains of NQF framework

2) Few measures that target:

a) System performance and accountability.

b) Caregivers and caregiver support

c) HCBS recipient leadership in system development

d) Equity

4) Many items related to outcomes that are clearly “personal” are 

not framed in a person-centered fashion (one size fits all).

5) Some domains and subdomains are inadequately “saturated” 

(i.e. they reflected only limited aspects of the constructs they 

purport to measure)

6) Few measures (as opposed to instruments) report adequate 

psychometrics
on home and community based services outcome measurement

Measure Database Functions

• Database will include items & measures coded by 

NQF domains and subdomains, psychometrics, and 

descriptions of how the items are used

– (e.g. respondent type, population, etc.)

• Designing web-based platform for Study 2 results

– Data dashboard

– Used with TA from RTC/OM
on home and community based services outcome measurement

Understanding Measure Administration

• Identify existing outcome measurement programs 

used in which identified HCBS outcome measures 

are being implemented. 

• Conduct case studies of varied measurement 

approaches and programs:

– Methodological components needed to ensure high fidelity 

measure administration

– Identification of strengths and challenges of selected 

existing outcome measurement programs and impact on 

measure administration fidelity?

– Identification of factors that facilitate or detract from 

effective measurement implementation
on home and community based services outcome measurement

Measure Prioritization Process

• All domains/subdomains based on NQF framework

• New subdomains based on feedback from Study 1

• Rated on three criteria by:

– RRTC/OM Leadership Group

– National Advisory Group

 Feasibility

 Usability

 Importance

• Other considerations

– Minimizing redundancy with work of other measure developers, 

partners (HSRI)

– Domain & Subdomain coverage

– System-level vs. Individual-level measures

– Person-centeredness
on home and community based services outcome measurement

Combined Stakeholder Input and Gap Analysis

Domain PPDM Rating # Items

Person-Centered Service Planning and Coordination 94.9 485

Service Delivery and Effectiveness 94.9 653

Choice and Control*T 94.9 1088

Human and Legal Rights*PT 94.5 521

Workforce 92.8 602

Equity 92.6 85

Holistic Health and Functioning*T 91.9 949

Community Inclusion*P 91.5 1415

System Performance and Accountability 89.8 40

Consumer Leadership in System Development 89.3 31

Caregiver Support 89 208
on home and community based services outcome measurement

Prioritized NQF Subdomains for Measure Development

Personal choices and goals

Transportation

Choice of services and supports

Meaningful activity

Person's needs met and goals realized

Self-direction

Social connectedness and relationships

Freedom from abuse and neglect

Employment

Workforce/Direct Care Staff Turnover

Person-centered planning

Access to resources

Note: bold type indicates a new subdomain provided by stakeholders in study one qualitative data
on home and community based services outcome measurement

Measure Development Process

• Iterative process to develop or revise items addressing 

gaps in items/measures identified in studies 1 and 2.

– Items prioritized based on input of stakeholders in 

study 1 & 2.

– Extensive review of existing conceptual 

frameworks for measure concepts to be developed

– Development of operational definitions for key 

components of measure concepts based on existing 

frameworks
on home and community based services outcome measurement

Measure Development Process

• Items from Study #2 mapped onto the construct 

definitions

• Staff with content expertise draft measure 

specifications and revise/develop items

• Iterative validation process of item and response 

format

– Content expert review 

– Cognitive testing w/ all disability groups

– Pilot study N = 100
on home and community based services outcome measurement

Measure Lifecycle
on home and community based services outcome measurement

Ascertaining Psychometric Quality of Measure 

Constructs

• Multi-site investigation of psychometric properties of 

prioritized HCBS measure concepts based on 

previous RRTC/OM studies including:

– Reliability (inter-rater, test-retest, inter-source, internal 

consistency)

– Validity (concurrent, predictive, discriminant, content, 

construct, inter-source)

– Measure discrimination

– Sensitivity to change

• Stratified random sample of 1,000 individuals (16+ 

years) receiving HCBS drawn from the target 

populations with PD, IDD, TBI, MH challenges, and 

ARD
on home and community based services outcome measurement

Identification & Testing of Promising Risk 

Adjusters

• Study focus is on identification and evaluation of 

risk adjusters used in research with populations of 

interest:

– Phase 1: Initial identification and analysis of 

risk adjusters used with HCBS recipient groups 

through systematic literature review 

– Phase 2: Prioritization of a set of promising risk 

adjusters to be used in RTC/OM data collection

– Phase 3: Development of risk adjusted models 

to predict specific HCBS outcomes to increase 

validity of the measure estimates. 
Q&A SESSION 

63