2020 CENSUS

Additional Steps 

Needed to Finalize 

Readiness for Peak 

Field Operations

Report to Congressional Requesters

December 2018

GAO-19-140

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-19-140, a report to 

congressional requesters

December 2018

2020 CENSUS

Additional Steps Needed to Finalize Readiness for 

Peak Field Operations

What GAO Found

In preparation for the 2020 census, the Census Bureau (Bureau) set out to 

enumerate over 140,000 housing units during the 2018 Census Test at a site in 

Providence County, Rhode Island. The 2018 Census Test marked the Bureau’s 

last chance to test enumeration procedures for peak field operations under 

census-like conditions before 2020. Implementation of this test identified the 

following concerns:

• The Bureau experienced operational issues during implementation of the 

Non-Response Follow-Up (NRFU) as part of the 2018 Census Test. For 

example, the Bureau had not finalized procedures for data collection during 

late phases of NRFU (e.g., after multiple attempts to interview had been 

made) until after the work had already started. As a result, enumerators and 

their supervisors did not have standardized procedures during the test, which 

made it difficult to evaluate the effectiveness of the test procedures. GAO 

also observed a range of other NRFU implementation issues during the test, 

such as the Bureau’s use of progress reporting that overstates the number of 

NRFU cases not needing any additional fieldwork and the Bureau having 

fewer of its enumerators work Saturdays, which can be among the most 

productive interview days. The Bureau is taking steps to assess and mitigate

these and other issues that GAO identified.

• The Bureau’s field workforce was not fully prepared to face all of the

enumeration challenges that arose during the test. For instance, the Bureau 

expects census field supervisors to provide front-line coaching to 

enumerators but did not screen these employees to ensure they had the 

needed skills. Moreover, it did not provide them with the authorities and 

information that would have helped them serve that role. As a result, we 

believe that supervisors did not have the casework expertise, information, or 

authority to help enumerators with procedural questions, and higher-level 

census field managers ended up providing direct support to enumerators. 

• While the Bureau provided extensive online and in-person training to 

enumerators prior to NRFU fieldwork for the 2018 Census Test, the Bureau 

lacked any standardized form of mid-operation training or guidance as new 

procedures were implemented. GAO observed that during the test some 

enumerators continued to have questions and were uncertain about 

procedures. Developing targeted, location-specific training could help ensure 

that, in 2020, enumerators receive the guidance they need to collect census 

data consistently and in accordance with NRFU procedures.

The Bureau has made progress addressing prior test implementation issues but 

still faces challenges. For example, the Bureau improved its collection of 

enumerator case notes, which reflect real-time knowledge gained during 

enumeration. However, enumerators did not always report cases using flags built 

in to their interviewing device that would benefit from supervisory review, such as 

for language barriers. Moreover, supervisors were not systematically analyzing

case notes to identify cases not flagged properly. As a result, critical data on 

fieldwork challenges were not being communicated effectively to those who could

analyze and use them.

View GAO-19-140. For more information, 

contact Robert Goldenkoff at (202) 512-2757 

or goldenkoffr@gao.gov.

Why GAO Did This Study

The cost of the decennial census has 

steadily increased over the past 

several decades, with self-response 

rates declining over the same period. 

The largest and costliest operation that 

the Bureau undertakes, NRFU is the 

Bureau’s attempt to enumerate 

households not initially self-responding

to the census. 

GAO was asked to review NRFU 

implementation during the 2018 

Census Test as well as the Bureau’s 

overall readiness for peak field 

operations, which cover the actual 

enumeration of residents. This report 

examines (1) how peak field 

operations, including NRFU, were 

implemented during the test; and (2) 

the extent to which prior test 

implementation issues have been 

addressed. GAO reviewed test 

planning and training documentation, 

as well as production and payroll data. 

At the test site, GAO observed and 

interviewed enumerators, field 

supervisors, and managers conducting 

peak operations.

What GAO Recommends

GAO recommends that the Bureau (1)

determine procedures for late-NRFU 

data collection; (2) align census field 

supervisor screening, authorities, and 

information flows; (3) prepare for 

targeted mid-operation training or 

guidance as needed; and (4) improve 

training on reporting cases that need 

supervisory attention and alternative 

ways to communicate these cases.

The Department of Commerce agreed 

with GAO’s findings and 

recommendations, and the Bureau 

provided technical comments that were 

incorporated as appropriate. 
Page i GAO-19-140 2020 Census

Letter 1

Background 3

The Bureau Experienced Operational Planning, Workforce 

Management, and Other Issues during Its 2018 Test of Peak 

Operations 7

The Bureau Has Made Progress but Continues to Face 

Challenges Addressing Implementation Issues Identified in 

Prior Tests 18

Conclusions 26

Recommendations for Executive Action 27

Agency Comments and Our Evaluation 27

Appendix I Objectives Scope & Methodology 30

Appendix II Comments from the Department of Commerce 32

Appendix III GAO Contact and Staff Acknowledgments 33

Tables

Table 1: Time Frames for Operational Activities Observed during 

2018 Census Test 5

Table 2: Census Bureau Established Procedures for Late NonResponse Follow-Up (NRFU) after Test Implementation 

Had Begun 8

Figures

Figure 1: Non-Response Follow-Up Was Implemented in a MultiTiered Structure at the Test Site 4

Figure 2: The Bureau’s Regular Reporting on Non-Response 

Follow-up Progress during the 2018 Census Test 

Overstated the Number of Cases Not Needing Additional 

Fieldwork 15

Contents
Page ii GAO-19-140 2020 Census

Abbreviations

Bureau Census Bureau

IT Information Technology

NRFU Non-Response Follow-Up

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-19-140 2020 Census

441 G St. N.W.

Washington, DC 20548

December 10, 2018

Congressional Requesters

The cost of the decennial census has steadily increased over the past 

several decades, with enumeration costs rising from about $16 per 

household in 1970 to around $92 in 2010 (all in constant 2020 dollars). 

The Census Bureau (Bureau) estimates that overall decennial costs will 

increase by over $3 billion from the 2010 Census to $15.6 billion in 2020 

(in current decennial time frame costs).1 During this period of increasing 

costs, the percentage of households self-responding to mailed census 

questionnaires has declined from 78 percent in 1970 to 63 percent in 

2010. The Bureau anticipates that the self-response rate will further 

decline to roughly 60 percent in 2020, in part because, as the Bureau has 

noted, the population is overloaded with requests for information and has 

become increasingly concerned about sharing information.

When a household does not initially respond to the census, the Bureau 

attempts to enumerate the residents through Non-Response Follow-Up 

(NRFU), an operation where enumerators personally visit to count the 

household. NRFU is labor intensive and is the largest and costliest 

operation that the Bureau undertakes. Another enumeration operation 

happening at about the same time is Group Quarters, when the Bureau 

counts residents of group facilities (such as skilled nursing facilities and 

correctional facilities).

The Bureau planned its 2018 Census Test in Providence County, Rhode 

Island, to rehearse most of the operations, systems, and procedures that 

it will implement during the 2020 Census.2 Previously, the Bureau 

conducted operational tests from 2013 through 2017, as well as multiple 

small-scale tests designed to demonstrate specific functionalities (such as 

submitting census data over the Internet). The 2018 test is the last 

 1

According to October 2017 Commerce documents, the reported figures are “inflated to 

the current 2020 Census time frame (fiscal years 2012 to 2023);” the Bureau had cited 

constant 2020 dollars for prior figures.

2

Address canvassing, another decennial field operation, was also tested in Pierce County, 

Washington, and the Bluefield-Beckley-Oak Hill area of West Virginia earlier during the 

test. 

Letter
Page 2 GAO-19-140 2020 Census

opportunity for the Bureau to demonstrate readiness for its major 

operations in 2020 and to apply lessons learned from prior tests.3

You asked us to review implementation of NRFU testing during the 2018 

Census Test as well as the Bureau’s overall readiness for peak 

operations, which cover the actual enumeration of residents. This report 

examines (1) the implementation of peak operations during the 2018 

Census Test at the Providence County, Rhode Island, site; and (2) the 

extent to which implementation issues raised in prior 2020 Census tests 

have been addressed and what actions the Bureau could take to address 

these issues.4

To address both of these research objectives, we visited the test site in 

Rhode Island to observe implementation of the peak operations being 

tested between May and August 2018. These visits included 

nongeneralizable observations of door-to-door field enumeration and 

office clerical work, as well as interviews with local managers. We also 

observed debrief sessions held with multiple levels of the census field 

workforce after the operations. From each of these visits, we documented 

observations and provided feedback to Bureau managers in near real 

time so that the Bureau could mitigate and adapt to issues raised by the 

test’s implementation in a timely manner. Implementation issues are a 

natural part of the testing environment and are what testing is intended to 

uncover. We also discussed any mitigation or evaluation strategies 

developed in response to our observations with the cognizant Bureau 

headquarters officials.

In addition to our fieldwork, we collected real-time production data on the 

tested operations. These data included tallies of case outcomes, 

transactional case activity by enumerators, and hours worked by Bureau 

employees. After testing the case tallies and distributions and interviewing 

cognizant officials, we determined that these data were sufficiently 

 3

In November 2018 Bureau officials told us they plan to conduct a randomized control test 

for the 2020 Census using the American Community Survey infrastructure in the summer 

of 2019 for the purposes of informing NRFU and the Integrated Partnership and 

Communications Campaign for the 2020 Census. The test will ask the 2020 questions, 

including a question on citizenship in one panel of the test and excluding it from the test’s 

second panel.

4

Peak operations include self-response and other enumeration activity. Due to the timing 

of our work and the schedule of the 2018 Census Test, we specifically reviewed the NRFU 

and Group Quarters operations. 
Page 3 GAO-19-140 2020 Census

reliable for our reporting purposes. We also received daily progress 

reports from the Bureau throughout the test, and we reviewed Bureau 

test-planning documentation and our work from prior tests to examine 

how, if at all, the Bureau planned to address prior implementation issues.

We conducted this performance audit from April 2018 to December 2018 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

NRFU is a field-based operation that the Bureau administers following the 

self-response period so that it can (1) determine the occupancy status of 

individual non-responsive housing units and (2) enumerate them. In most 

instances, the Bureau typically allows up to six enumeration attempts for 

each nonresponsive housing unit, or case. If the Bureau is unable to 

enumerate the housing unit in the field, it may have to impute attributes of 

the household based on the demographic characteristics of surrounding 

housing units as well as administrative records.

Within the test site in Providence County, Rhode Island, the Bureau set 

up an area census office to administer field operations. Figure 1 provides 

an overview of the managerial hierarchy of the area census office. The 

area census office manager oversees day-to-day operations within the 

office and acts as a liaison with the Bureau’s New York Regional Census 

Center, which is a Bureau regional office with jurisdiction over the 

Providence area census office. Census field managers are to monitor 

operational progress and performance indicators to understand any areas 

of concern and shift resources as needed within the test site. Census 

field supervisors are to act as front-line supervisors for individual 

performance and payroll processes and receive procedural questions 

from enumerators, who conduct the count.

Background
Page 4 GAO-19-140 2020 Census

Figure 1: Non-Response Follow-Up Was Implemented in a Multi-Tiered Structure at 

the Test Site

The Bureau has another operation—Group Quarters—to enumerate 

those living or staying in a group facility that provides housing or services. 

Such facilities can include skilled nursing facilities, college and university 

student housing, and correctional facilities. Within the Group Quarters 

enumeration, the Bureau also enumerates places such as soup kitchens, 

homeless shelters, and other service-based enumeration facilities.

Prior to Group Quarters enumeration in the field, the Bureau attempts to 

establish the facilities’ approximate population count and preferred 

enumeration method through the Advance Contact operation. These 

facilities can choose among methods including paper listing, where the 

facility provides a roster of residents as of census day to the Bureau, and 

in-person enumeration, where a team of enumerators count residents. 

For the 2020 cycle, the Bureau is also adding an “eResponse” option, 

tested on a small scale in 2016, whereby facility administrators can 

electronically submit enumeration data at a date of their choosing within 

operational time frames.

The Bureau’s testing of the peak operations that we observed during the 

2018 Census Test was intended to test collection of census data from 
Page 5 GAO-19-140 2020 Census

those either not responding themselves via paper, telephone, or over the 

Internet or those living in group quarters. Prior to the start of NRFU during 

the 2018 test, roughly 45 percent of anticipated housing units in the test 

area of Providence County, Rhode Island, self-responded, leaving more 

than 140,000 remaining housing units to be attempted by NRFU itself, 

which took place between May 9 and July 31, 2018. The Bureau 

conducted a test of its Group Quarters enumeration from July 25 through 

August 24, 2018, including service-based enumeration. Both portions of 

Group Quarters fieldwork were preceded by the Advance Contact activity. 

Dates for the peak operations we observed during the 2018 test are listed 

in table 1.

Table 1: Time Frames for Operational Activities Observed during 2018 Census Test 

Operation Testing period

Non-Response Follow-Up May 9 through July 31, 2018

Group Quarters Advance Contact June 18 through July 10, 2018

Group Quarters July 25 through August 24, 2018

Service-based Enumeration July 25 through July 27, 2018

Source: Census Bureau planning documentation. | GAO-19-140.

The Bureau’s operational plans for this phase of the fieldwork for the 

2018 test incorporated two innovation areas that the Bureau hopes will 

produce savings for 2020.

• Reengineered field operations. For most of NRFU during the 2018 

test, the Bureau relied on automated data collection methods, 

including a system-based, automated process for assigning work to 

enumerators, a smartphone-based application for collecting 

enumeration data in the field, and system-generated supervisory 

alerts.5

• Use of administrative records. To help reduce costly NRFU visits 

during the 2018 test, the Bureau reviewed and, where appropriate, 

applied administrative records—information already provided to the 

 5

Consistent with the 2016 Census Test, the Bureau had designed traditional NRFU, which 

began in May 2018, to involve a maximum of six enumeration attempts per nonresponding housing unit. In most cases during this phase, the Bureau used its automated 

process for case assignments, whereby the system was to assign and route cases by 

geographic clusters based on multiple factors including: enumerator work availability; the 

likeliest time of day to successfully make contact with a resident; and spatial relationship 

with other open cases. 
Page 6 GAO-19-140 2020 Census

government as it administers other programs, such as Social Security, 

the Selective Service, or the Special Supplemental Nutrition Program 

for Women, Infants, and Children—to determine the occupancy status 

of housing units and thus remove vacant housing units from the 

NRFU workload, as well as to provide population counts of 

households not responding.6

The Bureau also tested multiple operational features for the first time this 

decennial cycle under full census-like production conditions in 2018:

• NRFU Closeout. During the 2018 test, the Bureau tested how best to 

relax certain business rules and enumeration procedures late in the 

NRFU operation so that it can enumerate persistently non-responsive 

housing units. Examples of procedural modifications include 

increasing the maximum allowable number of enumeration attempts 

for each housing unit and manually assigning cases to the highestperforming enumerators.

• Office-Based Group Quarters Advance Contact. In the 2010 

Census, the Bureau sent enumerator crews in person to each facility 

in advance of enumeration to establish the facility’s preferred method 

of enumeration and to obtain an approximate population count. In the 

2018 Census Test, the Bureau implemented a new method for 2020 

that instead involved clerical staff contacting facilities by telephone 

and updating the group quarters address list and enumeration 

information remotely to reduce expenses associated with field visits 

for its enumerator crews.

 6

For more information on the Bureau’s use of Administrative Records, see: GAO, 2020 

Census: Bureau Is Taking Steps to Address Limitations of Administrative Records, 

GAO-17-664 (Washington, D.C.: July 26, 2017). 
Page 7 GAO-19-140 2020 Census

The Bureau began the last phase of NRFU data collection in the 2018 

test without having yet determined the procedures it would use for that 

critical phase.7 Bureau planning documentation from February 2018 

described a late-operation “closeout” phase of NRFU that would attempt 

to resolve cases that had not yet responded. However, we found that the 

Bureau had not determined the procedural modifications this phase would 

involve, either in terms of rules enumerators followed or business rules for 

how cases were to be assigned. By late May, nearly 3 weeks into the 

operation, the Bureau issued a set of closeout procedures to census 

areas where most cases had either been completed or where at least four 

of the six allowable enumeration attempt day assignments had been 

made. The Bureau also placed a priority on having high-performing 

enumerators—in terms of their ability to complete cases—available to 

work these cases during this phase of the NRFU testing.

Table 2 summarizes the chronology for when the Bureau implemented 

and documented procedural changes governing the transition from early 

to late-NRFU data collection, as well as the nature of those changes.

 7

Every decennial, some housing units do not respond to the Bureau’s multiple attempts to 

count them. In the 2018 test, NRFU included separately what it referred to as “closeout”

and “final attempt” phases to maximize the participation rate of these housing units.

The Bureau 

Experienced 

Operational Planning, 

Workforce 

Management, and 

Other Issues during 

Its 2018 Test of Peak 

Operations

The Bureau Did Not 

Determine Procedures for 

Late-NRFU Data 

Collection until after It 

Started the Work
Page 8 GAO-19-140 2020 Census

Table 2: Census Bureau Established Procedures for Late Non-Response Follow-Up (NRFU) after Test Implementation Had 

Begun

Operational Phase Initial data collection Final attempt

Initial scope of workload for the 

phase of data collection

Entire NRFU workload NRFU areas with >3% of cases visited maximum 

number of times without getting a population count

Date that test data collection began 

for the phase

May 9, 2018 June 23, 2018

Date that test data collection 

procedures were documented for 

the phase

January 4, 2018 July 13, 2018

Method of assigning cases Automated assignment Experimented with mix of manual and automated 

assignments

Notable changes to enumerator 

procedures relative to initial data 

collection

— Unlimited number of attempts per day during time 

remaining; enumerators could sequence cases on 

their own

Legend: — refers to not applicable.

Source: GAO analysis of Bureau documentation. | GAO-19-140

In late June 2018, the Bureau began testing the third phase of NRFU data 

collection, what it referred to as the “final attempt” phase, with officials 

citing a high incidence of non-interviews during prior phases as the 

reason. However, the Bureau had also begun this phase’s data collection 

before it had established the procedural modifications it would be using. 

The modifications were intended to further increase the chances of 

enumerators completing cases in the field, such as by removing the limit 

on the number of attempts enumerators could make at each remaining 

case before NRFU ended.

Standards for Internal Control in the Federal Government states that 

agencies should implement control activities by, for example, 

documenting policies.8 However, the Bureau did not determine 

procedures for the final attempt phase until after testing for this phase of 

NRFU had begun. Enumerators and census field supervisors thus began 

working closeout and final attempt cases without a standardized set of 

test procedures.

Without determining the procedural changes the Bureau would be 

testing—or the business rules guiding when to make those changes—the 

Bureau was not well positioned to collect data to assess the alternatives it 

 8

GAO, Standards for Internal Control in the Federal Government, GAO-14-704G

(Washington, D.C.: Sept. 10, 2014). 
Page 9 GAO-19-140 2020 Census

used during the test to inform planning for 2020. Bureau officials shared 

with us that they believed their automated case assignment approach is 

most effective during initial data collection but that it is less effective at 

targeting the toughest cases to resolve late in NRFU data collection. Yet, 

in part because the Bureau had not established when the transition from 

automated to manual case management would occur—or the business 

rules for determining when—some of the highest-performing enumerators 

were unavailable to receive assignments when the Bureau needed to 

begin the final attempt phase, according to the area census office 

manager. By not establishing the scope and timing of procedural changes 

for late-NRFU data collection in 2020, the Bureau may not be in a position 

to efficiently shift from its automated assignment approach to a manual 

one at the right time and position its most effective enumerators to 

receive assignments when needed.

In November 2018, the Bureau provided a draft contact strategy for 

NRFU in 2020 that included an outline of a multi-phase strategy for lateNRFU data collection. By including multiple phases of (1) shifting away 

from a fully-automated case assignment process and (2) relaxing 

management controls to complete as much casework as possible in 

areas with continued high non-response rates, this strategy appears to 

follow what the Bureau ultimately implemented during the 2018 test. It will 

be important, however, for the Bureau to determine the business rules for 

procedural changes and their timing in advance so that it can maximize 

the value of NRFU in reducing the number of housing units that have to 

be imputed for the 2020 Census.

Census field supervisors were not integrated into casework 

management. As described in the Bureau’s training and operational 

planning documents, census field supervisors were to be the primary 

points of contact in fielding and addressing enumerator questions. 

Census field managers—the next step above census field supervisors—

were to focus their efforts on monitoring progress in completing the 

caseload, reviewing cases flagged by enumerators as problematic in one 

of a small number of pre-defined ways (e.g., dangerous addresses), and 

resolving significant performance issues. Among field supervisors’ key 

responsibilities, according to the Bureau’s plan for NRFU, were providing 

guidance to help enumerators understand procedural matters and to offer 

coaching and problem-solving support to enumerators who may need it. 

They also led enumerator training prior to the beginning of NRFU and 

generally were to train their specific team of enumerators.

The Bureau Did Not Fully 

Ready Its Field Workforce 

for Enumeration 

Challenges
Page 10 GAO-19-140 2020 Census

However, census field managers and enumerators indicated that census 

field supervisors were often not the primary actors involved in fielding and 

addressing enumerator questions. Instead, enumerators and census field 

managers reported having direct contact with each other over procedural 

questions. Moreover, seven of the nine enumerators participating in the 

Bureau’s operational debrief focus group who responded said they 

thought finding someone who could answer their questions was either 

difficult or very difficult.

We found that census field supervisors went underutilized in part because 

the Bureau did not recruit and position them to assume front-line 

supervising and coaching responsibilities. As outlined in training 

documentation, the Bureau vested supervisory review authority (for 

special cases, such as resident refusals and language barrier issues) 

within census field managers, the area census office manager, clerks, 

and office operations supervisors instead of census field supervisors. 

Additionally, as part of the Bureau’s reengineered field operations for 

2020, census field supervisors are given automated tools to monitor 

enumerators, and enumerators we observed told us that they generally 

did not interact in person with their supervisors apart from training. We 

believe that the combination of these factors resulted in census field 

supervisors having limited exposure to NRFU casework and any 

problematic situations enumerators might encounter. Officials also told us 

that the Bureau did not screen census field supervisors for their 

supervisory or coaching skillsets, though officials noted that this has been 

the practice in prior censuses, too. Rather, they hired census field 

supervisors based on their scores on the online enumerator training and 

because they reported an interest in supervising.

Additionally, census field supervisors lacked access to certain data 

streams from the test that could have helped them answer or troubleshoot 

enumerator questions. According to two census field managers, the 

Bureau did not regularly share consolidated records of procedural 

changes with census field supervisors. Information technology (IT) and 

census field managers also noted that the Bureau did not share or 

compare observations between the census field supervisor hotline and 

the decennial IT hotline, even though enumerators could potentially call 

either or both with technical or procedural questions. As a result, without 

sharing how best to respond to similar questions across support lines, 

enumerators could receive different answers for related questions 

depending on which hotline they contacted.
Page 11 GAO-19-140 2020 Census

Standards for Internal Control in the Federal Government states that 

agency management should demonstrate a commitment to recruit, 

develop, and retain competent individuals.9 Management should establish 

expectations for competence in key roles and should consider the level of 

assigned responsibility and delegated authority when establishing 

expectations. Yet, the role the Bureau envisioned census field 

supervisors having was not aligned with the authority supervisors were 

given, the skills for which the Bureau hired them, or the access to 

information that they had for the 2018 test.

When we raised this issue related to using census field supervisors, 

Bureau officials agreed and cited feedback they had received that census 

field managers felt inundated with the combination of the volume of 

supervisory review cases that flowed to them and with troubleshooting 

day-to-day enumerator questions. In October 2018, the Bureau provided 

documentation to us proposing a set of questions that they could use in 

screening applicants for the census field supervisor position to identify 

supervisory skills. Officials also said they were still evaluating options for 

granting census field supervisors more supervisory review authority. As 

the Bureau continues to learn from the 2018 test as part of its planning for 

2020, it will be important to align census field supervisor roles with their 

authorities, skills, and information flows so that the Bureau does not 

underutilize a key portion of its field management chain. Doing so could 

also lessen the operational burden on higher-level census field managers.

Enumerators did not receive training to address mid-operation 

issues. Prior to the start of 2018 NRFU testing, the Bureau trained 

enumerators with a series of online training modules and assessments 

and one full day of in-person training facilitated by census field 

supervisors. The training included modules on data stewardship 

requirements, payroll responsibilities, and procedural directions for 

conducting respondent interviews. However, officials acknowledged that 

when the Bureau implemented its closeout and final attempt phases of 

NRFU, it did not provide standardized training to enumerators on the 

rollout of procedural changes. Five enumerators we observed during 

these stages said they relied on informal communications from their 

census field supervisors or census field managers for guidance. The 

initial practice had been for enumerators to receive daily assignments and 

follow pre-specified case sequencing and routing based on the Bureau’s 

 9

GAO-14-704G.
Page 12 GAO-19-140 2020 Census

automated system. During the final attempt phase, enumerators were 

given discretion over the sequencing, routing, and number of attempts to 

make for cases that could be manually assigned, yet they were not given 

standardized training on how to handle this shift.

During our field observations, some enumerators we spoke with said they 

were uncertain about core procedures. For example, enumerators were 

not consistently aware that they had some discretion in large multi-unit 

settings to deviate from the assigned sequence of their cases provided by 

the automated system. Enumerators we observed and spoke to were also 

not always clear on how to flag within their field enumeration application 

the commonly occurring cases with confusing address markings and 

numberings. For example, enumerators had the option of selecting a case 

outcome of “missing unit designation,” but they were not always sure 

whether this selection would capture the nuances of what they were 

seeing on the ground or how it differed from other selection options.10

Standards for Control in the Federal Government states that agencies 

should demonstrate a commitment to competence by, for example, 

tailoring training based on employee needs and helping personnel adapt 

to an evolving environment.11 Targeted informational training would help 

the Bureau ensure that staff understand mid-operation procedural 

changes, and the training could be an opportunity for the Bureau to 

address commonly-observed and persistent implementation issues that 

may be arising. By developing brief, targeted mid-operation training, 

either as formal modules, guidance, or other standardized job aids, such 

as “frequently asked questions” worksheets, the Bureau could better 

position itself to react nimbly to enumerator feedback.

We have previously reported challenges the Bureau faces with its field 

work in other locations, such as connecting to the Internet during testing 

of address canvassing in rural West Virginia in 2017 and dealing with 

language barriers and other circumstances in unincorporated 

communities in southern Texas or with migrant and seasonal farmworkers 

 10“Missing unit designation” is one of several case outcomes that refer to multi-unit 

address frame discrepancies. Other examples include “does not exist,” “uninhabitable,”

and “nonresidential.” Bureau officials indicated that they are reviewing the categorization 

of these outcomes in preparation for 2020. 

11GAO-14-704G.
Page 13 GAO-19-140 2020 Census

in southern California during the 2000 Census.12,13 All challenges are not 

universal to all locations. Given that some of the enumeration challenges 

enumerators encountered in 2018 NRFU testing might not occur 

everywhere, and that some other areas of the country will have their own 

types of challenges, locally- or regionally-specific training or guidance 

may better address some needs. By relying solely on pre-NRFU training, 

the Bureau risks having little opportunity to course-correct with 

enumerators who may not have absorbed all of the training and are 

experiencing difficulty completing interviews or not collecting quality data.

We observed and discussed with Bureau officials in real time several 

other implementation issues that occurred during the 2018 test. Bureau 

officials acknowledged these issues and, as of September 2018, were 

assessing them and developing mitigation strategies as part of their test 

evaluation process. These issues include:

Training certification. Census field managers estimated that roughly 

100 enumerators were unable to transmit their final test scores because 

the Bureau’s online learning management system had an erroneous 

setting. According to Bureau officials, this problem delayed the start of 

unsupervised work for these otherwise-qualified enumerators by an 

average of 2 days per enumerator and resulted in the attrition of some 

who were able to quickly find other work. Bureau officials told us they 

have fixed the system setting and are considering an alternative means to 

certify training, such as by having the option of trainees taking and 

verifying their final assessment as part of their final capstone day of 

classroom training. According to Bureau officials, development of this 

backup strategy will begin in December 2018.

Assigning cases manually in batches. During the 2018 test, the 

Bureau’s automated case management system was not configured for 

non-Headquarters staff to manually assign multiple cases to an 

enumerator at once. Rather, according to officials, census field managers 

were faced with having to manually assign thousands of cases 

individually during latter stages of NRFU. According to field management, 

 12GAO, 2020 Census: Actions Needed to Improve In-Field Address Canvassing, 

GAO-18-414 (Washington, D.C.: June 14, 2018).

13GAO, Decennial Census: Lessons Learned for Locating and Counting Migrant and 

Seasonal Farm Workers, GAO-03-605 (Washington, D.C.: July 3, 2003). 

The Bureau Is Assessing 

Other Implementation 

Issues That Arose during 

the 2018 Test
Page 14 GAO-19-140 2020 Census

this problem presented an unexpected burden on them, delayed 

assignments of the hardest-to-count cases, and contributed to highperforming enumerators not receiving work timely and in some cases for 

days in a row. Officials told us that, as a work-around, the Bureau shifted 

responsibility for assigning cases to a headquarters official with access 

rights in the system to assign large numbers of cases at once. Bureau 

officials acknowledge the unsustainability of this work-around if needed at 

a national level and the importance of resolving this before the 2020 

Census. As of October 2018, Bureau officials showed us system 

screenshots of how census field managers would be able to manually 

assign batches of cases and indicated that this functionality would be 

ready for the 2020 Census.

Monitoring operational progress. The Bureau’s reporting on its 

progress in completing the NRFU casework for the 2018 test emphasized 

a process-oriented measure that overstated the extent to which the NRFU 

efforts were resulting in completed workload. In planning documentation, 

the Bureau listed the outcomes of interview attempts that it considered 

complete and thus not in need of further enumeration assignments. 

These outcomes—such as a full interview of the household or 

confirmation of a housing unit being vacant or nonexistent—would also 

result directly in reduction in the number of incomplete cases needing to 

have some of their missing data imputed by the Bureau later.

Yet the daily Bureau progress report and “dashboard” the Bureau 

provided us for the 2018 test, which decennial leadership also identified 

as their primary monitoring report, did not reflect these pre-planned 

definitions of completed workload. Rather, as officials acknowledged, it 

included cases that the Bureau had unsuccessfully attempted to 

enumerate the maximum number of allowable times for the initial phase 

of NRFU being tested, even though those cases could still—and did—

receive additional attempts during later phases of NRFU. Officials noted 

that the measure reported could be helpful during early stages of the 

operation in determining whether enough employees had been hired, or 

whether case assignments were being worked quickly enough.

Figure 2 demonstrates the gap that arose during 2018 NRFU test 

implementation between the reported progress measure and the number 

of cases actually being completed. The totals reflected in the Bureau’s 

reported measure include those that either have to be re-worked in the 

field during the final attempt phase as discussed or have their data 

imputed after fieldwork had ended. By contrast, an outcome-based 

measure of operational progress, like the one the Bureau designed, 
Page 15 GAO-19-140 2020 Census

would capture only those cases where the Bureau had completed 

enumeration of the nonresponding housing units and thus be a more 

accurate representation of the operation’s status.14

Figure 2: The Bureau’s Regular Reporting on Non-Response Follow-up Progress during the 2018 Census Test Overstated the 

Number of Cases Not Needing Additional Fieldwork

Bureau officials acknowledged the need to maintain measures that focus 

on process as well as outcomes—such as avoiding having to impute data 

for cases after field work—when measuring progress completing NRFU. 

They said that managers in the field and in Bureau headquarters had 

access to alternative measures and reports that more closely identified 

outcomes. The officials noted that the reporting mechanism expected to 

be used in 2020 was not fully available in time for the beginning of NRFU 

 14The Bureau-designed, outcome-oriented measure that we report also includes cases 

that, while not enumerated in the field, were successfully enumerated using the Bureau’s 

sources of administrative records. 
Page 16 GAO-19-140 2020 Census

testing in 2018, so the reporting format and measures will likely differ. In 

addition, in October 2018, the Bureau provided a draft dashboard for 

2020 that included greater detail on the number of cases that could still 

require work to enumerate. Such detail could help assist with determining 

when to transition to the final-attempt phase of the operation to address 

cases without sufficient information yet collected.

Integrating key systems settings. At the beginning of NRFU test 

implementation, the Bureau’s case assignment and case sequencing 

systems were operating as if they were on time zones 4 hours apart. 

Bureau officials said that this resulted in enumerators receiving 

mismatched case assignment times, which hampered early NRFU 

production, and census field supervisors having to process erroneous 

“work not started” supervisory alerts. Bureau officials said they addressed 

the problem within the first week of the operation and that they would 

ensure that future updates and key settings would be coordinated across 

systems.

Tracking employees and equipment. The Bureau used two different 

sets of employee identification numbers to track their payroll status and 

use of Bureau-issued equipment (e.g., smart phones), respectively, 

without cross-walking them. According to census field managers, this 

resulted in extra work when trying to monitor changes in enumerators’

employment statuses and whether enumerators had returned their 

equipment to the Bureau. The managers noted that office staff had to 

spend extra time comparing different lists of staff, while one manager 

developed a spreadsheet listing all staff by their two different identification 

numbers. Bureau officials said they considered this a priority issue to be 

resolved during final systems development for 2020 and had already 

developed a fix within their case management system so that the crosswalk between the two systems would be integrated within their 

management system. This would eliminate the need for manually 

reconciling the differing identification numbers.

Having more enumerators work weekends. Until the latter stages of 

2018 NRFU testing, the Bureau assigned cases to enumerators based on 

the alignment of the Bureau’s estimated probability of finding respondents 

at home at certain times and enumerator-reported work availability. 

Bureau officials told us that Saturdays are generally one of the best days 

to find a household member home to respond to the census. However, 

during the test, our analysis showed that Saturdays had the secondfewest number of enumerators assigned to cases of any day of the week. 

Bureau officials said that they would review whether the incentive 
Page 17 GAO-19-140 2020 Census

structure for working on Saturdays should be altered and that they would 

examine ways to ensure that more enumerators are working on those 

days. This includes exploring the feasibility of hiring and assigning work to 

applicants who may only want to work weekends and being clearer with 

enumerators about what the expected peak enumeration hours are.

Increasing electronic completion rates for Group Quarters. The 

Bureau hopes to reduce field costs for Group Quarters (such as skilled 

nursing homes, college and university student housing, and correctional 

facilities) by, for the first time, encouraging facilities to self-enumerate 

electronically, when possible. As previously discussed, clerical staff first 

establish facility enumeration preferences during Group Quarters 

Advance Contact, and enumeration (either in person or otherwise) takes 

place afterward. During the 2018 test, the Bureau reported that only 25 of 

the 75 facilities that selected the “eResponse” enumeration option during 

Advance Contact submitted responses by the enumeration deadline. As 

of September 2018, Bureau officials said they were still evaluating 

potential causes of the low response rate by Group Quarters facilities but 

noted that issues with the required format for the submission of response 

files may have prevented some submissions. Bureau officials also 

acknowledged the need to conduct more active follow-up with these 

facilities during the eResponse period to ensure a full and accurate count 

of Group Quarters facilities.
Page 18 GAO-19-140 2020 Census

We reported to the Bureau during the 2015 Census Test that information 

enumerators were typing into their case notes did not appear to be 

systematically used by their managers.15 We also reported that, during 

the 2016 Census Test, the Bureau was not reviewing case notes written 

by enumerators providing respondent information on better times-of-day 

for future NRFU visits to their housing unit, and enumerators did not 

always have this note-taking feature available.16 During the 2018 test, 

enumerators were trained to take notes and, when appropriate, identify 

special cases that would later require supervisory review. We also 

observed that enumerators appeared to be consulting case notes from 

prior enumerator visits when planning NRFU visits to the same housing 

unit. Enumerators can use markings within their automated interview 

instrument to describe certain types of cases (e.g., hearing barriers and 

dangerous situations), which the automated system would then route to 

receive supervisory review. Enumerators could select other case 

outcomes that the automated system would apply predetermined 

business rules to either reassign the cases (e.g., refusal, no one home) or 

treat them as completed (verified vacant or not a housing unit).

However, we identified multiple scenarios in which enumerators had 

described cases in their case notes but for which the enumerators had 

not selected the corresponding case flag for the situation that would have 

resulted automatically in a supervisory review. One census field manager 

 15GAO, 2020 Census: Additional Actions Would Help the Bureau Realize Potential 

Administrative Records Cost Savings, GAO-16-48 (Washington, D.C.: Oct. 20, 2015).

16GAO, 2020 Census: Additional Actions Could Strengthen Field Data Collection Efforts, 

GAO-17-191 (Washington, D.C.: Jan. 26, 2017).

The Bureau Has 

Made Progress but 

Continues to Face 

Challenges 

Addressing 

Implementation 

Issues Identified in 

Prior Tests

The Bureau Has Improved 

Its Collection of Case 

Notes, but Enumerators 

Remain Unclear on How 

to Report Fieldwork Issues
Page 19 GAO-19-140 2020 Census

described discovering several dozen cases that had been inactive where 

enumerators had written case notes describing language barriers 

encountered but had not specifically marked the flag within the device for 

“language barrier.” Because these cases thus were not triggered for 

supervisory review, they were eligible to be reassigned by the Bureau’s 

standard automated system. As a result, the Bureau was not controlling 

for the requisite language skills in assigning the cases for subsequent 

enumeration attempts.

The Bureau’s use of automated systems to apply business rules to 

efficiently manage field casework for 2020—including identifying which 

cases receive supervisory review—relies critically on field staff 

understanding how to reflect what they are seeing on the ground within 

the choices provided to them with which to flag cases in their interview 

device. The Bureau’s use of remote management as part of reengineered 

field operations also relies on enumerators knowing when and how to 

report issues to their supervisors.

We observed multiple field scenarios that called these conditions into 

question, however. For example, enumerators we observed during NRFU 

told us that they indicated address listing issues in their case notes, such 

as if the unit designation was missing or incorrectly marked. Yet, these 

enumerators did not know how to flag such cases in their interview 

instrument to trigger supervisory review. According to Bureau officials, 

this type of address listing issue turned out to be a broadly experienced 

challenge within the test area. Additionally, during Group Quarters, an 

enumerator we observed received supplemental information about the 

number of residents at a neighboring facility after that facility had been 

enumerated. The enumerator made note of this discrepancy and included 

the original facility’s identifying information but was uncertain about how, 

if at all, to alert the supervisor about the discrepancy.

Moreover, we saw little evidence that census field supervisors or 

managers were systematically reviewing case notes for the purpose of 

identifying either cases not being marked properly or for which the 

selected flags may not have been fully describing the case 

characteristics. For example, a census field manager confirmed that case 

notes recorded at Group Quarters facilities that were enumerated would 

not be reviewed during clerical processing, leaving the possibility—such 

as we observed—that if enumerators relied on case notes to 

communicate information about the accuracy of data collected, it would 

not be acted on. Bureau officials told us that reviewing all case notes 

could require more staff time than budgeted for, and changing the 
Page 20 GAO-19-140 2020 Census

automation process to selectively present unflagged cases for 

supervisory review could necessitate requirements changes to systems 

whose development is already pressed for time.

Standards for Internal Control in the Federal Government states that 

agencies should use quality information by, for example, processing 

reliable information to help it make informed decisions.17 Bureau 

enumerators can record useful local knowledge about their cases with 

their choice of case type flags and within their descriptive case notes. 

While the Bureau has anticipated a broad range of types of cases for 

enumerators to select from when documenting their casework, 

enumerators we observed did not uniformly understand those options, 

and the descriptors did not fully anticipate what enumerators were 

encountering. Improving training and guidance to field staff on the 

intended use of case notes and on alternative ways to communicate their 

concerns about cases, such as flags for different types of cases, can help 

ensure the Bureau has reliable data on the cases during its field 

operations relying on automated interviewing instruments.

During the 2015 Census Test, we reported that certain technical and 

procedural problems that enumerators were encountering in the field 

were going unreported and that enumerators did not always know who to 

contact for assistance. We further noted that the Bureau was not 

systematically assessing or tracking the extent of these issues during 

testing, and we recommended that it enable such capture of information 

by training enumerators on where to record issues and whom to 

contact.18 The Bureau agreed with our recommendation.

During the 2018 test, the Bureau had both an information technology (IT) 

hotline and a census-field-supervisor hotline established for technical and 

procedural questions, respectively. Yet, enumerators we observed told us 

they did not always report to their support lines the technical issues that 

they were easily able to resolve by, for example, turning their devices on 

and off to reset. This lack of reporting kept the Bureau from getting 

information on commonly-occurring challenges that might be useful realtime feedback in the testing environment. Moreover, a Bureau IT 

manager noted that the Bureau does not formally review and share 

 17GAO-14-704G.

18GAO-16-48.

Leveraging Information on 

Enumerator-Reported 

Technical and Procedural 

Issues Remains a 

Challenge
Page 21 GAO-19-140 2020 Census

observations and troubleshooting notes from IT hotline and census field 

supervisor hotline calls. Because enumerators may call either or both 

hotlines when having difficulty operating their Bureau-issued smart 

phone, operators of these hotlines could be unaware of the prevalence of 

or solutions to a given problem if the Bureau does not monitor 

troubleshooting information across the two operational silos. For the 

Bureau to be informed on any additional training needs or other 

operational decisions for 2020, it will need to continue to expand its 

efforts in collecting information on enumerator-reported problems per our 

2015 recommendation.

Depending on the size of a Group Quarters facility (e.g., skilled nursing 

facility, college and university student housing), the Bureau can use 

varying sizes of enumerator crews to conduct an onsite count. During the 

2010 Census, we observed overstaffing during the service-based 

enumeration (e.g., homeless shelters and soup kitchens) portion of Group 

Quarters. While determining staffing levels at these facilities can be 

challenging, such overstaffing can lead to poor productivity and

unnecessarily high labor costs. We recommended that the Bureau 

determine and address the factors that led to this overstaffing prior to 

2020.19 The Bureau agreed with our recommendation.

However, the Bureau has faced challenges determining the right staffing 

ratios in light of complications with the Advance Contact phase of Group 

Quarters. As previously noted, the Bureau used this phase to establish 

facilities’ enumeration method preferences. For the 2018 test, most Group 

Quarters facilities selected the facility-provided paper listing and the 

eResponse enumeration options. Therefore, the Bureau allocated a large 

share of its enumerator and census field supervisor workforce in the test 

area to the 44 known service-based enumeration facilities, which were 

restricted in terms of the enumeration options they could select and 

tended to select in-person enumeration. However, only 11 of these 

facilities responded to initial inquiries, so the Bureau had less work than 

anticipated for its enumerator crews. At multiple sites we observed in the 

test area, enumerators appeared either idle or underutilized. Moreover, 

several of the Group Quarters facilities we observed had changed their 

initial choice of enumeration method on the day of enumeration. 

 19GAO, 2010 Census: Key Efforts to Include Hard-to-Count Populations Went Generally 

As Planned; Improvements Could Make the Efforts More Effective for Next Census, 

GAO-11-45 (Washington, D.C.: Dec. 14, 2010).

The Bureau Continues to 

Assess How to Ensure 

Efficient Staffing Levels for 

Group Quarters
Page 22 GAO-19-140 2020 Census

Enumerator crews thus ran the risk of either being overstaffed (in the 

case of switching to a facility-provided paper listing) or understaffed (in 

the case of switching to an in-person enumeration).

The Bureau’s Advance Contact activities have a potential benefit—if the 

Bureau can get accurate information on the method of enumeration and 

approximate population within a facility ahead of time, Bureau managers 

and enumerator crews can more proactively allocate resources and 

prepare for the count. Bureau officials said they are still assessing 

outcomes of Advance Contact to see if these gains were realized and 

may have completed the assessment by as early as January 2019. Doing 

so will help the Bureau determine appropriate staffing sizes and thus 

address our prior recommendation.

According to preliminary data from the 2018 Census Test, the Bureau 

experienced similarly high rates of cases coded as non-interviews as it 

did during its last major field test of NRFU in 2016. Non-interviews are 

cases where enumerators collect no data or insufficient data from 

households either because enumerators made the maximum number of 

visits without a successful interview, or because of special circumstances 

like language barriers or dangerous situations.20 When this happens, the 

Bureau may have to impute the census data for the case, such as 

whether the housing unit is vacant or not, the population counts of the 

households, or demographic characteristics of their residents. In January 

2017, we reported that, during the 2016 Census Test, the Bureau 

incurred what it considered high non-interview rates (31 and 22 percent 

across the two test sites, respectively, as the Bureau preliminarily 

reported at the time), and we recommended that the Bureau determine 

the causes of these rates.21

Using the same method to calculate the rate of non-interviews for the test 

as in 2016, the 2018 Census Test had similarly high non-interview rates—

33 percent of all NRFU cases. Bureau officials said they are still 

examining causes of these elevated non-interview rates and whether final 

attempts helped to mitigate the non-interview rate and will report out on 

what they learn as part of their comprehensive assessment of the test, 

 20According to the Bureau, it needs to collect a combination of data fields during field 

interviews in order to consider the interview complete. 

21GAO-17-191. 

The Bureau Is Working to 

Determine the Causes of 

Elevated Non-Interview 

Rates
Page 23 GAO-19-140 2020 Census

planned for December 2018. A draft of the Bureau’s revised contact 

strategy for NRFU, provided in November 2018, indicates that as part of 

enumerator training in 2020 the Bureau will need to incorporate 

messaging that emphasizes the importance of obtaining sufficient data 

from interview attempts. Officials noted that any interim lessons learned 

from this assessment process would inform updates to the field 

enumeration contact strategies for 2020.

Enumerators are directed to try and complete a NRFU case by 

interviewing a proxy for a household respondent, like a neighbor, after 

multiple failed attempts have been made to contact someone in the 

household for that case. We previously observed in the 2016 Census 

Test that enumerators did not seem to understand the procedures for 

conducting these interviews and, as a result, underutilized the 

interviewing method. In our January 2017 report, we therefore 

recommended that, as part of determining the causes of its non-interview 

rate, the Bureau revise and test any needed changes to proxy procedures 

and associated training.22 The Bureau agreed with our recommendation 

and subsequently developed automated supervisory performance alerts 

for census field supervisors and census field managers that would inform 

them when an enumerator was not following prompts to conduct proxy 

interviews for eligible cases.

However, in implementing proxy interview procedures for the 2018 

Census Test, the Bureau experienced a technical glitch resulting in some 

confusion among some enumerators and their supervisors about related 

procedures. Early in NRFU data collection for the test, a programming 

error within the field enumeration application was prompting enumerators 

to make more than the allowable three attempts to interview a proxy 

respondent.23 The Bureau reported promptly implementing a technical fix 

to this issue; yet, enumerators we observed reported receiving varying 

guidance from their supervisors on whether to abide by the erroneous 

prompts. While some of these enumerators appeared to understand the 

importance of attempting proxy interviews, some did not appear to 

understand Bureau guidance that enumerators should make no more 

 22GAO-17-191.

23The 2020 NRFU operational plan indicates that enumerators should, in most cases, only 

attempt three proxy interviews per visit. These interviews can be conducted with 

neighbors, relatives, landlords, or other sources with connections to the respondent or 

housing unit. 

The Bureau Experienced a 

Programming Error While 

Implementing Procedures 

for Proxy Interviews
Page 24 GAO-19-140 2020 Census

than three attempts to interview a proxy respondent, and some appeared 

conditioned to follow the erroneous prompts.

Proxy interviews can be a substantial portion of completed interviews 

during the census. In 2018 NRFU testing, interviews of proxy respondents 

accounted for 27 percent of all successful interview-based enumerations 

of occupied housing units—compared to 24 percent during the 2010 

Census and 9 percent during the 2016 Census Test. Given the role that 

proxy interviews play in completing census data collection, it will be 

important for the Bureau to fully implement our recommendation so that 

enumerators are properly pursuing and conducting these interviews.

Initial visits to property managers of multi-unit residences can help the 

Bureau identify vacant and occupied housing units before sending 

enumerators to individual units within the facilities. We have previously 

reported that property managers can also be a helpful source of 

information on respondents who are not at home, thereby making 

subsequent follow-up visits to individual units more productive.24 During 

the 2016 Census Test, we observed that enumerators were uncertain of 

how to handle individual cases within a multi-unit once they were 

unsuccessful in contacting a property manager initially. As a result, we 

recommended in January 2017 that the Bureau revise and test procedural 

and training modifications as needed to aid enumerators and their 

supervisors in these cases.25 The Bureau agreed with this 

recommendation and indicated that the evaluations of the 2018 test would 

inform its strategies for 2020.

However, we observed a similar issue during the 2018 Census Test in 

that enumerators were unclear on what, if any, proxies to attempt if they

were unsuccessful in finding the listed property manager. Additionally, we 

observed multiple enumerators leave voicemails with their contact 

information—not a central number—for the listed property manager, but it 

was unclear how these voicemails would produce a successful interview 

because, later, the automated system could reassign other enumerators 

to visit the manager. When we raised this concern with Bureau officials, 

they acknowledged that they need to continue to refine procedures for 

handling initial property manager visits for 2020.

 24GAO-17-191.

25GAO-17-191.

Enumerators We 

Observed Were Uncertain 

of How to Proceed When 

Property Managers Were 

Unavailable
Page 25 GAO-19-140 2020 Census

Previously, during the 2016 Census Test, we observed that enumerators 

were unable to re-open closed non-interview cases even if they happened 

upon the respondent in question soon after and nearby. We noted this 

inefficiency, since these cases would get re-assigned later, and in 

January 2017, we recommended that the Bureau revise and test 

procedures that would grant flexibility to enumerators to access cases in 

these circumstances.26 The Bureau agreed with our recommendation.

For the 2018 Census Test, the Bureau provided a list in the field 

enumeration application of the cases that had been worked by the 

enumerator that day but that had not been submitted for processing or 

reassignment. Training for enumerators described this enumeration 

option, and enumerators were authorized to access these cases when 

needed, but not all enumerators we observed were consistently aware of 

how to do so. Enumerators we spoke with cited uncertainty over how to 

access these cases and whether enumerators were allowed to do so as 

considerations. Continuing to review the procedures and guidance to 

enumerators on this flexibility for completing interviews, consistent with 

our prior recommendation, will help the Bureau make better use of it in 

2020.

As we reported in 2017, the Bureau previously modified how it would treat 

some of the households that did not respond to the 2020 Census and that 

the Bureau’s use of administrative records had determined to be not 

occupied. The Bureau’s earlier testing had determined that the Bureau 

should require two—instead of just one—notices from the United States 

Postal Service that mail could not be delivered to these households 

before removing their addresses from the NRFU workload.27 After we 

provided a draft of this report to the Department of Commerce to obtain 

agency comments, Bureau officials provided us with findings from an 

evaluation of the 2018 Census Test. In the evaluation, Bureau officials 

observed that there were households for which they had received multiple 

notices from the United States Postal Service that mail was undeliverable 

but that Bureau enumerators recorded as occupied. While Bureau 

officials believe, based on their follow-up research, that these addresses 

may likely be vacant or not housing units, they are concerned about 

possible undercounting from not enumerating people who may be at 

 26GAO-17-191.

27GAO-17-664.

The Bureau Provided 

Enumerators Access to 

Incomplete Closed Cases, 

but Enumerators Were Not 

Consistently Aware They 

Had Access

The Bureau May Add an 

Extra Check on NRFU 

Addresses Identified by 

Administrative Records as 

Vacant or Nonexistent
Page 26 GAO-19-140 2020 Census

these addresses. As of November 2018, the Bureau was considering 

adding one physical visit for each of these cases. Bureau officials said 

they are continuing to analyze these evaluation results and expect to 

document and include changes within its final operational plan for the 

2020 Census due in January 2019.

The 2018 Census Test offered the Bureau its last opportunity to test key 

procedures, management approaches, and systems under decennial-like 

conditions prior to the 2020 Census. As the Bureau studies the results of 

its NRFU and Group Quarters testing to inform 2020, it will be important 

that it address key program management issues that arose during 

implementation of the test. Namely, by not establishing the intended 

procedural changes for late-NRFU data collection ahead of time, the 

Bureau risked not getting the most out of NRFU to minimize the number 

of housing units having to have their information imputed by the Bureau 

later. Additionally, by not aligning the skills, responsibilities, and 

information flows for census field supervisors, the Bureau limited their role 

in support of enumerators within the reengineered field operation. The 

Bureau also lacks mid-operation training or guidance, which, if 

implemented in a targeted, localized manner, could further help 

enumerators navigate procedural modifications and any commonlyencountered problems when enumerating. Finally, without enumerators 

understanding how to use case notes and flags for various types of cases 

in their enumeration device and to report enumeration challenges to 

supervisors and managers, the Bureau may be unaware of field work 

issues that could affect the efficiency of its operations and the quality of 

its data.

We provided near real-time feedback to the Bureau across a range of test 

implementation issues. Some, such as those related to staffing ratios for 

the Group Quarters operation, build on long-standing implementation 

issues that, if addressed, can contribute to the efficiency and 

effectiveness of 2020 field operations. Others, like not having NRFU 

progress measures that provide true indications of completed workload, 

are issues specific to this test that the Bureau is assessing as part of its 

2018 Census Test evaluations. It will be important for the Bureau to 

prioritize its mitigation strategies for these implementation issues so that it 

can maximize readiness for the 2020 Census.

Conclusions
Page 27 GAO-19-140 2020 Census

We are making four recommendations to the Department of Commerce 

and the Census Bureau:

The Secretary of Commerce should ensure that the Director of the 

Census Bureau determines in advance of Non-Response Follow-Up what 

the procedural changes will be for the last phases of its data collection 

and what the business rules will be for determining when to begin those 

phases, which cases to assign, and how to assign them. 

(Recommendation 1)

The Secretary of Commerce should ensure that the Director of the 

Census Bureau identifies and implements changes to align census field 

supervisor screening, authorities, and information flows to allow greater 

use of the census field supervisor position to provide supervisory support 

to enumerators. (Recommendation 2)

The Secretary of Commerce should ensure that the Director of the 

Census Bureau enables area census offices to prepare targeted, midoperation training or guidance as needed to address procedural changes 

or implementation issues encountered locally during Non-Response 

Follow-Up. (Recommendation 3)

The Secretary of Commerce should ensure that the Director of the 

Census Bureau improves training and guidance to field staff on the 

intended use of case notes and flags, as well as on alternative ways to 

alert supervisors and managers when case characteristics are not readily 

captured by those flags. (Recommendation 4)

We provided a draft of this report to the Secretary of Commerce. In its 

written comments, reproduced in appendix II, the Department of 

Commerce agreed with our findings and recommendations and said it 

would develop an action plan to address them. The Census Bureau also 

provided technical comments and an update on their evaluation of the 

test, which we incorporated as appropriate.

We are sending copies of this report to the Secretary of Commerce, the 

Undersecretary of Economic Affairs, the Acting Director of the U.S. 

Census Bureau, and the appropriate congressional committees. In 

addition, the report is available at no charge on the GAO website at 

http://www.gao.gov.

Recommendations for 

Executive Action

Agency Comments 

and Our Evaluation
Page 28 GAO-19-140 2020 Census

If you or your staff have any questions about this report please contact 

me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made key contributions to this 

report are listed in appendix III.

Robert Goldenkoff

Director

Strategic Issues
Page 29 GAO-19-140 2020 Census

List of Congressional Requesters

The Honorable Trey Gowdy

Chairman

The Honorable Elijah E. Cummings

Ranking Member

Committee on Oversight and Government Reform

House of Representatives

The Honorable Mark Meadows

Chairman

The Honorable Gerald E. Connolly

Ranking Member

Subcommittee on Government Operations

Committee on Oversight and Government Reform 

House of Representatives
Appendix I: Objectives Scope & Methodology

Page 30 GAO-19-140 2020 Census

This report examines (1) how peak field operations were implemented 

during the 2018 Census Test; and (2) the extent to which implementation 

issues raised in prior 2020 Census tests have been addressed, and what 

actions the Census Bureau (Bureau) is taking to address them.

To address these objectives, we reviewed 2018 Census Test and 2020 

Census operational planning and training documentation. We also 

reviewed our prior reports and documentation on prior census testing 

operations. Non-Response Follow-Up (NRFU) operations took place from 

May 8 through July 31, 2018, while Group Quarters took place from July 

25, 2018 through August 24, 2018, with the service-based enumeration 

portion taking place July 25, 2018 through July 27, 2018.

To review the Bureau’s test implementation and mitigation strategies for 

previously-identified implementation issues for peak operations, we 

visited Providence, Rhode Island, multiple times between May and 

August 2018 to observe enumerators, census field supervisors, and 

management operations. NRFU visits took place between mid-May and 

late-July 2018, while we also conducted two iterations of visits of Group 

Quarters in late July and early-August 2018. These multiple iterations 

both across and within operations enabled us to see how, if at all, 

implementation of procedures varied over time. It also enabled us to get 

direct feedback from Bureau field managers on how various phases of 

test operations were proceeding. These visits consisted of nongeneralizable observations of field enumeration and office clerical work, 

as well as interviews with local managers. For each of these visits, we 

developed data collection instruments to structure our interviews and to 

cover topics that were pertinent to the given phase of the operation we 

were observing. We also observed debrief sessions with multiple levels of 

the Bureau’s field workforce following the field work.

To translate our observations into actionable feedback for the Bureau, we 

shared high-level observations in near real-time to Bureau headquarters 

management overseeing the operations so that the Bureau could mitigate 

and adapt to known issues in a timely manner. We also discussed any 

mitigation or evaluation strategies developed in response to our 

observations with the cognizant Bureau headquarters officials. For 

objective two specifically, we reviewed Bureau test planning 

documentation and our work from prior tests to examine how, if at all, the 

Bureau planned to address known implementation issues.

To gain insight into how implementation was proceeding when we were 

not directly observing test implementation, we received daily 

Appendix I: Objectives Scope & 

Methodology
Appendix I: Objectives Scope & Methodology

Page 31 GAO-19-140 2020 Census

management progress reports from the Bureau throughout the NRFU 

operation testing that included information on the total number of NRFU 

cases, the final outcomes of each case, and the number of cases that the 

Bureau reported as completed for each day of the NRFU operation. We 

also received Periodic Management Reports that summarized high level 

outcomes of both the NRFU and Group Quarters workload.

To fully understand the source of the Bureau’s daily progress reports, we 

requested and received all transactional data collected during NRFU 

production. We reconciled case totals and outcomes with the final 

numbers in the NRFU progress reports and then used these data to 

analyze the Bureau’s progress during NRFU production. We also 

received and analyzed Bureau payroll data on enumerator hours worked 

during NRFU operations. Specifically, we assessed the number of 

enumerators working each day, the number of enumerator’ hours paid 

each day, and the days of the week that were worked the most by 

enumerators. We found the Bureau’s transactional and payroll data 

sources to be sufficiently reliable for our reporting purposes.

We conducted this performance audit from April to December of 2018 in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Comments from the Department of Commerce

Page 32 GAO-19-140 2020 Census

Appendix II: Comments from the Department 

of Commerce
Appendix III: GAO Contact and Staff Acknowledgments

Page 33 GAO-19-140 2020 Census

Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov

In addition to the contact named above, Ty Mitchell (Assistant Director), 

Devin Braun, Karen Cassidy, Joseph Fread, Robert Gebhart, Krista 

Loose, Kathleen Padulchick, Lisa Pearson, Kayla Robinson, Robert 

Robinson, and Cynthia Saunders made significant contributions to this

 report.

Appendix III: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(102758)
The Government Accountability Office, the audit, evaluation, and investigative arm 

of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to https://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and

distribution and depends on the number of pages in the publication and whether

 the publication is printed in color or black and white. Pricing and ordering

 information is posted on GAO’s website, https://www.gao.gov/ordering.htm. 

 Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

 TDD (202) 512-2537.

 Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at https://www.gao.gov.

Contact:

Website: https://www.gao.gov/fraudnet/fraudnet.htm

Automated answering system: (800) 424-5454 or (202) 512-7700

Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.