Directions in Measuring Disability: 

Use and Evaluation of the Washington Group 

Short Set on Functioning

2023 FCSM Research & Policy Conference

October 25, 2023

Jennifer Madans

Discussant
Disability as a Case Study:

How and When to Change Survey Content 

• Maintaining trend lines is a MAJOR consideration for the federal statistical system.

• Changes should not be made without good reason:

• Changes in how a topic is conceptualized

• Improvement in methods 

• Changes should not be made without understanding the impact of the change:

• The objective is not perfection, but improvement

• This should not be confused with finding a way to maintain the series

• The impact of the change should be disseminated

• The proposal to change the disability questions on the ACS presents a useful case study 

of how such change can and should be conducted.
Development of the Current ACS-6

• Developed by an interagency committee with expertise in the evolving methodologies 

related to obtaining information on disability.

• Occurred at approximately the same time as work was being done internationally under 

the aegis of the UN Statistical Commission through the Washington Group (WG)

• Some of the committee members were involved in both efforts

• Both the ACS and international activities shared the same conceptual framework

• WG-Short Set (WG-SS) adopted as the standard measure of disability by UNSC and the UN 

Economic Commission for Europe for censuses and population surveys, as well as the IAEGSDGs as the measure of disability for disaggregating the SDGs

• The constraints of the ACS paper form did not allow for the adoption of the international 

question set

• Modifications were made in wording to address the inability to have graded responses

• Graded responses were endorsed as the better option, but the change would need to be made 

at a future date when space restrictions were not an issue

• Both the ACS version and the WG-SS underwent extensive testing 
The Interval Between the Adoption of the

ACS-6 and the Proposal to Make a Change

• Extensive analyses of the ACS-6 and the WG-SS

• The ACS-6 and the WG-SS were both included in the NHIS for the purpose of 

understanding the differences between the two approaches

• Both included in NHIS: ACS 2009-2017 / WG-SS 2010-current

• Both administered to the same respondent (subset of the sample) 2011-2012

• Analyses were conducted comparing the results from the two question sets

• The findings were consistent with the cognitive testing conducted during 

development of both question sets

• Analytic results have been widely disseminated
Testing of the Proposed Change

• The vast majority of the testing of the WG-SS was for interviewer administered surveys 

although they have been used in self-response data collections.

• An interagency group participated in the evaluation of the proposed questions.

• A key test objective was to evaluate the mode differences.

• Two rounds of cognitive tests were conducted.

• A large field test was conducted with results confirming what was already known about the 

question sets.

• The proposed questions are not perfect, but the questions in use are also not perfect.

• The tests demonstrated that the advantages of the proposed questions in terms of 

conceptual clarity and the scope of the resulting data (continuum versus yes/no).

• The ACS mode did not adversely affect data quality compared to the questions in use.

• The results of the test will be made public, which along with the already published test results, 

will inform users of the impact of the change.
Impact of the Recommended Change

• There will definitely be a break in the trend.

• Given the nature of the change, there is no way to crosswalk between the two 

question sets as can be done when major categories are split into component parts 

or there is a change in coding criteria.
Deciding if the Change Should be Made

• There will be a break in the trend if the disability questions are changed.

• The nature of the break was anticipated prior to the current round of testing.

• The current round of testing reinforced what has been found in past testing:

• The new questions are conceptually clearer.

• The graded response provides more information related to functional status.

• There is no way to cross walk between the two sets of answer categories.
Should the Change Be Made?

• Change is necessitated by improved methods and the need for additional information on the 

topic, with minor increases in burden.

• Extensive evaluations fully describe the need for the change, the nature of the change, and the 

benefits of the change.

• Targeted testing in the ACS format was consistent with prior findings and did not uncover any 

new quality issues with the questions.

• The questions under consideration are probably the most studied and evaluated of any other 

questions being used on any survey.

• The benefits outweigh the costs, which were know prior to the most recent round of testing.

• The process used to evaluate the proposed questions supports the recommendation to change 

the questions.
Next Steps

• The process used to evaluate the proposed change in the ACS disability 

questions presents a good case study for an appropriate evaluation.

• It is crucial that information on the change and its impact be shared with users 

as widely and clearly as possible. 

• However, it is not clear how further research will add much to what is already 

known about the differences in the two question sets. 

• If it is known that a change will result in a break in trend and that the proposed 

questions are based on improved methodology, is it reasonable to continue to 

use a break in trend as an evaluation criteria?
Discussion