DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Census Bureau Evaluation of Proposed 

Changes to the ACS Disability Questions

Federal Committee on Statistical Methodology 

2023 Research and Policy Conference

October 25th, 2023

Natalie Young

Social, Economic, and Housing Statistics Division

U.S. Census Bureau

1

The U.S. Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and approved the 

disclosure avoidance practices applied to this release. [DRB # CBDRB-FY24-ACSO003-B0002]

This presentation is released to inform interested parties of ongoing research and to encourage discussion. Any views expressed are 

those of the author[s] and not those of the U.S. Census Bureau.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Motivation and Background

• The American Community Survey (ACS) is our largest household survey and a 

critical source of national and sub-national disability estimates in the U.S.

• Current ACS Disability measure: ACS-6.

• Standardized 6-item question set.

• Developed by the ACS Subcommittee on Disability Measurement.

• Used since 2008.

• Proposed Change: Washington Group Short Set on Functioning (WG-SS).

• Developed by the Washington Group (WG) on Disability Statistics.

• The Washington Group has invested considerable time and resources into evaluating the 

validity of the WG-SS in the U.S. and internationally (Altman, 2016).

• By WG tallies, used in over 90 national surveys or censuses worldwide, including key health 

surveys in the U.S.

• Recommended by the National Center for Health Statistics (NCHS) and the United Nations.

2
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Side-by-Side Comparison: 

ACS-6 and WG-SS

3

• 6-item question set.

• Based on WHO’s ICF.

• Measures 6 functional 

domains including:

• Vision.

• Hearing.

• Ambulation.

• Cognition.

• Self-Care.

• Graded response 

categories: 

• No difficulty. 

• Some difficulty. 

• A lot of difficulty. 

• Cannot do at all.

• 6th domain measured:

• Communication.

• Dichotomous response 

categories:

• Yes.

• No.

• 6th domain measured:

• Independent living.

ACS-6 WG-SS
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002) 4

Question Wording Differences

American Community Survey (ACS-6) Washington Group - Short Set (WG-SS)

Hearing Is this person deaf or does he/she have serious difficulty 

hearing?

Does this person have difficulty hearing, even if using a 

hearing aid?

Seeing Is this person blind or does he/she have serious

difficulty seeing even when wearing glasses?

Does this person have difficulty seeing, even if wearing 

glasses?

Cognition Because of a physical, mental, or emotional condition, 

does this person have serious difficulty concentrating, 

remembering, or making decisions?

Does this person have difficulty remembering or 

concentrating?

Ambulation Does this person have serious difficulty walking or 

climbing stairs?

Does this person have difficulty walking or climbing 

steps?

Self-Care Does this person have difficulty dressing or bathing? Does this person have difficulty with self-care, such as 

washing all over or dressing?
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Census Bureau Evaluation of the WG-SS

• In 2020-2022, the Census Bureau tested proposed changes to the 

American Community Survey.

• This included testing a disability measure based on the Washington 

Group Short Set on Functioning (WG-SS).

• Advantages of WG-SS: internationally comparable; graded response 

categories.

• Structure and purpose of the evaluation of the WG-SS: 

• Part 1: Cognitive Interviews – to examine validity of WG-SS.

• Part 2: Field Test – to evaluate performance of WG-SS in ACS environment.

5
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Testing: 

Research Goals and Methods

• Cognitive interviews were conducted in 2020-2021:

• Three rounds of cognitive interviewing (language: English and Spanish).

• Due to COVID-19 pandemic, all interviews were conducted virtually.

• Research goals:

1. Evaluate wording differences between WG-SS and ACS-6 and whether 

questions understood as intended.

2. Assess how respondents decide between WG-SS response categories.

• A total of 419 interviews were conducted (over 3 rounds).

6
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Interviews: Construct Validity 

- Self Care and Ambulation

Does this person have difficulty 

with self-care, such as washing all 

over or dressing?

Washing all over: Phrase seemed 

strange to some, but generally 

understood as intended.

7

Vast majority later described difficulties consistent with intended concepts.

Does this person have difficulty 

walking or climbing steps?

Difficulty climbing steps: Less 

consistently understood than “stairs”, 

but generally worked as intended.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Interviews: Condition Capture 

- Cognition and Communication

Removed concept of “difficulty making decisions.” 

Interview probe: Can you give me some examples of health conditions that make 

it difficult to remember or concentrate?

8

Using his or her usual language, 

does this person have difficulty 

communicating, for example 

understanding or being understood?

New concept for ACS.

Interview probe: Can you give me some 

examples of conditions that may make it 

difficult to communicate?

Spanish language revisions required.

Does this person have difficulty 

remembering or concentrating?
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Interviews: Construct Validity

- Hearing and Seeing

Removed terms “deaf” and “serious [difficulty]”; Added “even if” clause.

9

Does this person have difficulty 

seeing, even if wearing glasses?

Removed terms “blind” and “serious [difficulty].”

Some reports of difficulty due to outdated 

prescriptions.

Does this person have difficulty 

hearing, even if using a hearing aid?

For both questions: “Even if...” clause sometimes overlooked.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Interviews: 

Response Category Selection

10

Ambulation:

“Some Difficulty”

Ambulation:

“A Lot of Difficulty” or “Cannot Do at All”

• Respondent who currently has difficulty 

walking or climbing steps due to recent foot 

surgery, but anticipates a full recovery 

without further issues.

• Individual with a pinched nerve that doesn’t 

bother him to the same degree every day.

• Respondent over age 60 who “can’t get 

upstairs in 3 minutes but...can in 6 minutes”.

• Respondent who, after suffering a stroke, has 

use of only one limb.

• Respondent who uses a walking stick and is 

not allowed to climb steps on her own.

• Nursing home resident who used to be able 

to climb steps with “a lot of difficulty” but 

now “cannot do [it] at all” because she is 

wheelchair-bound.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Cognitive Interviews: Key Take-Aways

• Provided additional evidence of the validity of the WG-SS.

• Although alternative wording was tested, the interviews indicated 

that the WG-SS wording worked sufficiently well.

• Next step: Further evaluated a version of the WG-SS in the 2022 ACS 

Content Field Test.

11
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Field Test: 

Methods

• Nationally representative sample of 120,000 housing unit addresses, excluding 

Puerto Rico, Alaska, and Hawaii (divided into 3 treatments).

• Split-ballot design: households received either control or test version of ACS.

• Control treatment: same as production ACS.

• Test treatment: included proposed change to disability question set.

• Modes of administration:

• Self-response (internet and paper).

• Nonresponse follow-up (Computer-Assisted Personal Interview [CAPI]).

• Content Follow-Up (CFU) interview.

• Response rates:

12

Response Rates Control Treatment Test Treatment

Original interview 83.6% (SE 0.4) 83.3% (SE 0.4)

CFU re-interview 32.9% (SE 0.5) 34.2% (SE 0.5)
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002) 13

Field Test: Disability in Control vs. Test

Control Test

Yes/No Likert Scale

18a. Hearing 18a. Seeing

18b. Seeing 18b. Hearing 

19a. Cognitive 19a. Ambulatory

19b. Ambulatory 19b. Cognitive

19c. Self-Care 19c. Self-Care

20. Independent Living 19d. Communication

20. Independent Living

Yes/No Graded Response
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Field Test: 

Goals

• Primary: To evaluate performance in the ACS environment.

• Key elements of the ACS environment include...

• Multiple modes (CAPI; paper; internet).

• Within-household proxy reporting.

• Decision criteria:

• Response reliability (Gross Difference Rate [GDR] and Index of Inconsistency [IOI]).

• Item missing data rates.

• Secondary: To explore the impact on ACS disability estimates.

• Informational only- Not a decision criteria.

• Two definitions tested: 

• Definition 1: ‘a lot of difficulty’ or ‘cannot do at all’ (recommended by WG).

• Definition 2: ‘some difficulty’, ‘a lot of difficulty’ or ‘cannot do at all’.

14
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

13.9

4.0

2.6

5.2

7.4

2.5

5.9

8.1

1.7 1.7 2.3

4.3

1.4

3.4

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Control

Test (Def. 1)

Percentage-Point Difference

Field Test: Percentage with a Disability 

by Treatment and Type

15

When using Definition 1, the estimated percentage of the U.S. population with a disability was 

significantly lower in test than in control (two-tailed t-tests at the α=0.1 level).

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types. Minor additive 

discrepancies are due to rounding.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

13.9

4.0

2.6

5.2

7.4

2.5

5.9

8.1

1.7 1.7 2.3

4.3

1.4

3.4

-5.8

-2.2 -0.9

-2.9 -3.1

-1.1 -2.4

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Control

Test (Def. 1)

Percentage-Point Difference

(Test - Control)

Field Test: Percentage with a Disability 

by Treatment and Type

16

When using Definition 1, the estimated percentage of the U.S. population with a disability was 

significantly lower in test than in control (two-tailed t-tests at the α=0.1 level).

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types. Minor additive 

discrepancies are due to rounding.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

-5.8

-2.2

-0.9

-2.9 -3.1

-1.1

-2.4

-7.0

-6.0

-5.0

-4.0

-3.0

-2.0

-1.0

0.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Percentage Point

Percentage-Point Difference

(Test - Control)

-41.7%

-57.5%

-34.6%

-55.8%

-41.9% -44.0% -42.4%

-70.0%

-60.0%

-50.0%

-40.0%

-30.0%

-20.0%

-10.0%

0.0%

Percent

Percent Difference

(Test - Control) / Control

Field Test: Difference Between Test and 

Control Disability Estimates

17

The magnitude of the difference between test (Def. 1) and control may vary by disability type.

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

6.3

8.7 8.3

12.4

31.2

3 3.6 3.8

7.4

19.6

-20

-10

0

10

20

30

40

5 - 14 15 - 24 25 - 39 40 - 64 65+

Control

Test (Def. 1)

Percentage-Point Difference

Field Test: Percentage with a Disability 

by Age

18

When using Definition 1, the estimated percentage of the U.S. population with a disability was 

lower in test than in control across all age groups (except 0-4).

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between age groups.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

6.3

8.7 8.3

12.4

31.2

3 3.6 3.8

7.4

19.6

-3.3 -5.1 -4.5 -5.0

-11.6

-20

-10

0

10

20

30

40

5 - 14 15 - 24 25 - 39 40 - 64 65+

Control

Test (Def. 1)

Percentage-Point Difference

(Test - Control)

Field Test: Percentage with a Disability 

by Age

19

When using Definition 1, the estimated percentage of the U.S. population with a disability was 

lower in test than in control across all age groups (except 0-4).

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between age groups.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

-3.3

-5.1 -4.5 -5.0

-13.0 -11.6

-11.0

-9.0

-7.0

-5.0

-3.0

-1.0

5 - 14 15 - 24 25 - 39 40 - 64 65+

Age Group

Percentage Point

Percentage-Point Difference

(Test - Control)

-52.4% -58.6% -54.2%

-40.3% -37.2%

-80.0%

-60.0%

-40.0%

-20.0%

0.0%

Percent

Percent Difference

(Test - Control) / Control

Field Test: Difference Between Test and Control 

Disability Estimates by Age

20

The magnitude of the difference between test (Def. 1) and control may vary by age.

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent and percentage point comparisons are nominal. We did not statistically test differences in percentages between age groups.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Field Test:

Decision Criteria Results

• Response reliability (Decision Criterion #1).

• Test had lower GDR than control, with only very limited exceptions (where no 

difference was detected between test and control).

• Test had higher IOI than control for overall disability when all modes were combined, 

but not across all disability types and modes (sometimes, no difference detected).

• Item missing data rates (Decision Criterion #2).

• Test had higher item missing data rates for overall disability when all modes 

combined, but results by disability type and mode varied.

• When item missing data rates were higher in test, only by ~1 percentage point.

• Conclusion: Test treatment meets the decision criteria.

• The proposed question set performs adequately in the ACS environment based on 

these criteria, including in self-response modes.

21
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Conclusion and Recommendations

• The disability subcommittee is recommending the proposed change 

to the disability question set.

• If the proposed question set is adopted, definition 1 of disability is 

recommended as the primary Census Bureau measure.

22
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Implementation

• The revised disability measure will not be comparable to the current ACS 

measure.

• The proposed disability measure is expected to capture a smaller population and a 

population with a higher level of functional difficulty.

• Next steps:

• Conduct additional research to further examine how the population captured by the 

revised measure differs from that captured by the ACS-6.

• Engage key stakeholders (e.g., members of disability subcommittee) to help 

communicate with data users about the change.

• Other agencies will need to address the implications of this change for their own 

surveys and programs.

23

Test ~8% 2022 ACS ~13%1

1 U.S. Census Bureau. (2022). Disability Characteristics. American Community Survey, ACS 1-Year Estimates Subject Tables, 

Table S1810. Retrieved October 11, 2023, from https://data.census.gov/table/ACSST1Y2022.S1810?q=s1810.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Thank You

• Questions/comments?

• This project was made possible by the efforts of many, including:

• Census Bureau staff in the American Community Survey Office, Application 

Development and Services Division, Center for Behavioral Science Methods, 

Decennial Information Technology Division, Decennial Statistical Studies Division, 

Field Division, National Processing Center, Population Division, and Social, Economic, 

and Housing Statistics Division.

• Representatives from other agencies in the federal statistical system serving on 

OMB’s Interagency Working Group for the ACS and the topical subcommittees 

formed by the Working Group for each topic tested in the 2022 ACS Content Test.

• Staff in OMB’s Statistical and Science Policy Office.

• For additional comments or questions, please contact:

• Sharon Stern (Assistant Division Chief for Employment Characteristics, Social, 

Economic, and Housing Statistics Division, U.S. Census Bureau) at 

Sharon.M.Stern@census.gov.

24
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

References

• Altman, B. M. (Ed.). (2016). International Measurement of Disability: Purpose, Method, and Application. New York: 

Springer.

• RTI International (2022a). 2022 ACS Content Test: Round 1 and Round 2 Cognitive Testing. Retrieved July 19, 2022, 

from https://www.census.gov/content/dam/Census/library/working-papers/2022/acs/2022_Wilson_01.pdf

• RTI International (2022b). Cognitive Testing for the 2022 ACS Content Test: Round 3 Briefing. Retrieved August 29, 

2022 from https://www.census.gov/library/working-papers/2022/acs/2022_Wilson_02.html

• Spiers, S., Baumgardner, S., and Mojica, A. (Forthcoming). 2022 American Community Survey Content Test Evaluation 

Report: Unit-level Response. U.S. Census Bureau. American Community Survey Memorandum Series #ACS23-RER-03. 

• Steinweg, A., Young, N., Stern, S., Contard, L., and Spiers, S. (Forthcoming). 2022 American Community Survey 

Content Test Evaluation Report: Disability. U.S. Census Bureau. American Community Survey Memorandum Series 

#ACS23-RER-12. 

• U.S. Census Bureau. (2022). Disability Characteristics. American Community Survey, ACS 1-Year Estimates Subject 

Tables, Table S1810. Retrieved October 11, 2023, from https://data.census.gov/table/ACSST1Y2022.S1810?q=s1810.

25
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Appendix

26
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002) 27

Wording Differences:

Hearing and Seeing

Differences in Test Version

• Switched order of ‘Hearing’ and 

‘Seeing’ questions.

• Used graded scale answer choices.

• Removed word “serious”, and other 

wording adjustments to pick up a 

range of difficulties.

• Added the words “even if using a 

hearing aid” to the hearing question.

Control Version Test Version
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002) 28

Wording Differences: Cognition, Ambulation, Self-Care, and 

Communication

Control Version Test Version

Differences in Test Version

• Switched order of ‘Cognition’ and 

‘Ambulation’ questions.

• Used graded scale answer choices.

• Added “Communication” question.

• Removed word “serious” to pick up a 

range of difficulties.

• Cognition question: Removed 

“preamble” and reference to 

“making decisions”.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002) 29

Wording Differences: Independent Living

Control Version

Differences in Test Version

• Used graded scale answer choices.

Test Version
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

13.9

4.0

2.6

5.2

7.4

2.5

5.9

31.7

11.7

16.3 15.3 16.0

5.7

9.6

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Control

Test (Def. 2)

Percentage-Point Difference

Field Test: Definition 2

Disability Estimates by Disability Type

30

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types. Minor 

additive discrepancies are due to rounding.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

13.9

4.0

2.6

5.2

7.4

2.5

5.9

31.7

11.7

16.3 15.3 16.0

5.7

9.6

17.8

7.8

13.7

10.1

8.6

3.1 3.7

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Control

Test (Def. 2)

Percentage-Point Difference

(Test - Control)

Field Test: Definition 2

Disability Estimates by Disability Type

31

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types. Minor 

additive discrepancies are due to rounding.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

17.8

7.8

13.7

10.1 8.6

3.1 3.7

0.0

4.0

8.0

12.0

16.0

20.0

Hearing Seeing Cognition Ambulation Self-Care Ind. Living

ANY Disability Disability Types

Percentage Point

Percentage-Point Difference

(Test - Control)

128.1%

192.5%

526.9%

194.2%

116.2% 128.0%

62.7%

0.0%

100.0%

200.0%

300.0%

400.0%

500.0%

600.0%

Percent

Percent Difference

(Test - Control) / Control

Field Test: Definition 2

Differences by Disability Type

32

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between disability types.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

Field Test: Definition 2

Disability Estimates by Age

33

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between age groups.

0.6

6.3

8.7 8.3

12.4

31.2

3.9

13.5

18.3

21.4

34.4

59.5

0

10

20

30

40

50

60

70

0 - 4 5 - 14 15 - 24 25 - 39 40 - 64 65+

Age Group

Control

Test (Def. 2)

Percentage-Point Difference
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

0.6

6.3

8.7 8.3

12.4

31.2

3.9

13.5

18.3

21.4

34.4

59.5

3.3

7.2

9.6

13.1

22.0

28.3

0

10

20

30

40

50

60

70

0 - 4 5 - 14 15 - 24 25 - 39 40 - 64 65+

Age Group

Control

Test (Def. 2)

Percentage-Point Difference

(Test - Control)

Field Test: Definition 2

Disability Estimates by Age

34

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between age groups.
DRB Clearance Number: (CBDRB-FY24-ACSO003-B0002)

3.3

7.2 9.6

13.1

22.0

28.3

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 - 4 5 - 14 15 - 24 25 - 39 40 - 64 65+

Age Group

Percentage Point

Percentage-Point Difference

(Test - Control)

550.0%

114.3% 110.3%

157.8% 177.4%

90.7%

0.0%

100.0%

200.0%

300.0%

400.0%

500.0%

600.0%

Percent

Percent Difference

(Test - Control) / Control

Field Test: Definition 2

Differences by Age Group

35

Source: U.S. Census Bureau, 2022 American Community Survey Content Test.

Note: All percent comparisons are nominal. We did not statistically test differences in percentages between age groups.