2020 CENSUS: Additional Actions Could Strengthen Field Data Collection Efforts
Published: 2017-01-26T00:00:00

Fast Facts:
If you don't return your Census form, a Census taker will interview you in person using a mobile device.
To prepare for the 2020 Census, the Census Bureau tested how well this interview process works. Our review of their 2016 tests found many non-interviews—cases where no data or insufficient data were collected. The Bureau is researching potential causes.
We also found some cases where better procedures or more training may improve the process. For example, Census takers could not access recently closed incomplete cases on their mobile devices—and were therefore unable to interview a household when people returned home.

Highlights:
section_title: Why GAO Did This Study
paragraphs: With a life-cycle cost of about $12.3 billion, the 2010 Census was the most expensive enumeration in U.S. history. To help control costs and maintain accuracy, the 2020 Census design includes new procedures and technology that have not been used extensively in earlier decennials, if at all. While these innovations show promise for a more cost-effective head count, they also introduce risks. As a result, it will be important to thoroughly test the operations planned for 2020.
The objective of this report is to assess key NRFU operations performed during the 2016 Census Test to identify any lessons learned that could have a potential impact on pending design decisions for the 2020 Census. To assess NRFU operations GAO visited both test locations and observed enumerators conducting NRFU interviews, and reviewed relevant documents including the test plan and enumerator training manuals.
section_title: What GAO Found
paragraphs: The Census Bureau (Bureau) recently completed its 2016 Census Test in Los Angeles County, California, and Harris County, Texas. One primary focus of the test was to assess the methodology for non-response follow-up (NRFU), where enumerators personally visit households that do not self-respond to the census. GAO found that during the 2016 Census Test, NRFU generally proceeded according to the Bureau's operational plan. However, data at both test sites indicate that the Bureau experienced a large number of non-interviews. Non-interviews are cases where either no data or insufficient data are collected. Bureau officials are not certain why there were so many non-interviews for the 2016 Census Test and are researching potential causes. Going forward, it will be important for the Bureau to better understand the factors that contributed to the non-interview rate because of its relationship to the cost and quality of the census.
GAO also found that refining certain enumeration procedures and training enumerators better could produce additional efficiencies by enabling the Bureau to be more responsive to situations enumerators encounter on the ground. For example, enumerators, by design, were unable to access on the mobile device recently closed, incomplete cases. Bureau officials acknowledged that closing cases in this fashion represented a missed opportunity and plan to test greater flexibilities as part of the 2018 End-to-End Test. Programming some flexibility into the mobile device—if accompanied with adequate training on how and when to use it—should permit enumerators to complete some interviews and reduce the cost of follow-up attempts. Further, enumerators did not always understand procedures for visiting property managers in multi-unit buildings. Specifically, the 2016 Census Test demonstrated that vacant units could quickly be removed from the NRFU workload where a property manager was readily available to provide that information; however, in other cases the procedures confused enumerators and they did not understand how to proceed. Without the knowledge of which units were vacant, enumerators may have unnecessarily visited some vacant units and thereby increased the cost of NRFU.
During GAO's field visits, GAO encountered several instances where enumerators learned that returning at a specific time on a later date would improve their chance of obtaining an interview from either a household respondent or a property manager. However, the Bureau's 2016 Census Test and automated case management system did not have an efficient way to leverage that information. Attempting contact at non-responding households at times respondents are expected to be available increases the completion rate and reduces the need to return.
section_title: What GAO Recommends
paragraphs: GAO recommends the Secretary of Commerce direct the Bureau to: (1) determine causes for non-interviews, and revise and test what, if any, changes need to be made to operational procedures and training; (2) revise and test procedures and training on accessing closed cases, (3) revise and test procedures and training for initial property manager visits; and (4) revise and test procedures and training for how to use enumerator-collected data on the best time or day to conduct an interview. The Department of Commerce agreed with GAO's recommendations, and the Bureau provided technical comments that were incorporated, as appropriate.

section_title: Letter
paragraphs: The cost of the decennial census has steadily increased during the past  40 years. For example, at about $12.3 billion, the 2010 Census was 31  percent more costly than the $9.4 billion 2000 Census (in constant 2020  dollars). Given budgetary realities, that cost growth is difficult to sustain.  Beginning in 1990, we reported that rising costs and difficulties in  securing public participation, among other challenges, required a new  approach to taking the census—a view that was shared by the U.S.  Census Bureau (Bureau) and other stakeholders.
Earlier this year the Bureau conducted the 2016 Census Test in Los  Angeles (L.A.) County, California, and Harris County, Texas. A key  objective of the test was to assess the methodology for non-response  follow-up (NRFU), where enumerators personally visit households that do  not self-respond to the census. NRFU is the largest and costliest of all  census-taking activities because it is so labor intensive. The Bureau  selected Harris and L.A. counties as test sites for several reasons  including language diversity, demographic diversity, high vacancy rates,  and varying levels of Internet usage. There are around 225,000 housing  units in each test area. In November 2016, we testified on the progress of  the test as well as other key preparations, and described important  lessons learned from the 2010 Census that can apply to the 2020  Census.
You asked us to review how selected NRFU operations performed during  the 2016 Census Test. The objective of this review was to review the test  and identify any lessons learned that could potentially impact pending  design decisions for the 2020 Census. To address this objective, we  reviewed key documents including the 2016 Census Test plan that  discussed the goals and objectives of the 2016 Census Test, as well as  training manuals, respondent contact strategy documents and business rules for NRFU. We interviewed census staff at both test sites including  local supervisors of operations, enumerators, and office personnel. At the  test sites, we observed enumerators conducting NRFU interviews and we  used the training manuals to determine whether enumerators collected  information as prescribed by the Bureau. In total we conducted 31 in-field  observations of Bureau enumerators conducting NRFU. We also  interviewed Bureau officials in headquarters to obtain information on the  strategies and procedures for collecting data in the 2016 Census Test  and what changes to procedures were made based on lessons learned  from the 2015 Census Test. We reviewed NRFU interview data from the  Bureau’s workload case management system and looked for patterns in  the data such as the number of refusals and completed interviews. We  performed a data reliability assessment on NRFU interview data collected  from the workload case management system and found that the data was  reliable for the purpose of our reporting objective.
We conducted this performance audit from April 2016 to January 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: For the 2020 Census, the Bureau intends to limit its per-household cost to  not more than that of the 2010 Census, adjusted for inflation. To achieve  this goal, the Bureau is significantly changing how it conducts the census,  in part by re-engineering key census-taking methods and infrastructure.  The Bureau’s innovations include (1) using the Internet as a self-response  option; (2) verifying most housing unit addresses using “in-office”  procedures rather than costly field canvassing; (3) in certain instances,  replacing enumerator-collected data with administrative records  (information already provided to federal and state governments as they  administer other programs) and, (4) re-engineering field data collection  methods.
The Bureau’s various initiatives have the potential to make major  contributions toward limiting cost increases. In October 2015, the Bureau  estimated that with its new approach it can conduct the 2020 Census for  a life-cycle cost of $12.5 billion, $5.2 billion less than if it were to repeat  the design and methods of the 2010 Census (both in constant 2020  dollars). Table 1 below shows the Bureau’s estimated cost savings it  hopes to achieve in 4 innovation areas.
Sufficient testing, while important to the success of any census, is even  more critical for the Bureau’s preparations for 2020. To help control costs  and maintain accuracy, the 2020 Census design includes new procedures  and technology that have not been used extensively in earlier decennials,  if at all. While these innovations show promise for a more cost-effective  head count, they also introduce new risks. As we have noted in our prior  work, it will be important to thoroughly test the operations planned for  2020 to ensure they will (1) produce needed cost savings, (2) function in  concert with other census operations, and (3) work at the scale needed  for the national head count. The Bureau’s failure to fully test some key  operations prior to the 2010 Census was a key factor that led us to  designate that decennial as one of our high-risk areas.
The 2016 test was the latest major test of NRFU in the Bureau’s testing  program. In 2014, the Bureau tested new methods for conducting NRFU  in the Maryland and Washington, D.C., area. In 2015, the Bureau  assessed NRFU operations in Maricopa County, Arizona. In 2018, the  Bureau plans to conduct a final “End-to-End” Test which is essentially a  dress rehearsal for the actual decennial. The Bureau needs to finalize the  census design by the end of fiscal year 2017 so that key activities can be  included in the End-to-End Test.
The Bureau plans to conduct additional research and testing through  2018 in order to further refine the design of the 2020 Census but recently  decided to alter its approach. On October 18, 2016, the Bureau  announced plans to stop two field test operations planned for fiscal year  2017 to mitigate risks from funding uncertainty. The Bureau said it would  stop all planned field activity, including local outreach and hiring, at its test  sites in Puerto Rico, North and South Dakota, and Washington State. The  Bureau will not carry out planned field tests of its mail out strategy and  NRFU in Puerto Rico. The Bureau also cancelled plans to update its  address list in the Indians lands and surrounding areas in the three  states.
However, the Bureau said it will continue with other planned testing in  fiscal year 2017, such as that focusing on systems readiness and internet  response. Further, the Bureau said it would consider incorporating the  stopped field activity elements within the 2018 End-to-End Test. The  Bureau maintains that stopping the 2017 Field Test will help prioritize  readiness for the 2018 End-to-End Test, and mitigate risk. Nevertheless,  as we noted in our November 2016 testimony, it represents a lost  opportunity to test, refine, and integrate operations and systems, and puts  more pressure on the 2018 End-to-End Test to demonstrate that  enumeration activities will function as needed for 2020.
section_title: 2016 Census Test Highlighted Data Collection Challenges
subsections: section_title: The Bureau Needs to Better Understand Factors Contributing to NRFU Non-interviews
paragraphs: NRFU generally proceeded according to the Bureau’s operational plans.  For example, the Bureau demonstrated procedures for quality assurance  and training. On the other hand, according to preliminary 2016 Census  Test data, there were 19,721 NRFU cases coded as non-interviews in  Harris County, Texas and 14,026 in L.A. County, California, or about 30  and 20 percent of the test workload respectively.
According to the Bureau, non-interviews are cases where no data or  insufficient data were collected, either because enumerators made six  attempted visits without success (the maximum number the Bureau  allowed), or visits were not completed due to, for example, language  barriers or dangerous situations. In such cases for the 2020 Census, the  Bureau may have to impute attributes of the household based on the  demographic characteristics of surrounding housing units as well as  administrative records. According to Bureau officials, they are not certain  why there were so many non-interviews for the 2016 Census Test and  are researching potential causes. Bureau officials told us that they expect  higher numbers of non-interviews during tests in part, because, compared  to the actual enumeration the Bureau conducts less outreach and  promotion. While the 2016 Census Test interview rate is not necessarily a  precursor to the 2020 non-interview rate, because of its relationship to the  cost and quality of the census, it will be important for the Bureau to better  understand the factors contributing to it.
Bureau officials hypothesized that another contributing factor could be  related to NRFU methods used in the 2016 Census Test compared to  earlier decennials. For the 2010 and prior censuses, enumerators  collected information during NRFU using pencil and paper. Enumerators  may have visited a housing unit more than the six maximum allowable  visits to obtain an interview but did not record all of their attempts, thus  enabling them to achieve a higher completion rate. For the 2020 Census,  and as tested in 2016, the Bureau plans to collect data using mobile  devices leased from a contractor, and an automated case management  system to manage each household visit (see figure 1). The Bureau  believes that this approach will provide a faster, more accurate, and more  secure means of data collection. Unlike previous censuses and one prior  test, enumerators in the 2016 Census Test did not have an assigned set  of cases that they alone would work until completion. Instead, the Bureau  relied on an enhanced operational control system that was designed to  provide daily assignments and street routing of NRFU cases to enumerators in the most optimal and efficient way. At the same time, the  mobile device and automated case management system did not allow an  enumerator to attempt to visit a housing unit more than once per day,  reopen a closed case, or exceed the maximum allowable six attempts.
One factor we observed that may have contributed to the non-interview  rate was that enumerators did not seem to uniformly understand or follow  procedures for completing interviews with proxy respondents (a proxy is  someone who is a non-household member, at least 15 years old, and  knowledgeable about the NRFU address). According to the 2016 Census  Test enumerator training manual, when an eligible respondent at the  address cannot be located, the automated case management system on  the mobile device will prompt the enumerator when to find a proxy to  interview, such as when no one is home or the housing unit appears  vacant. In such circumstances, enumerators are to find a neighbor or  landlord to interview. However, in the course of our site visits, we  observed that enumerators did not always follow these procedures. For  example, we observed that one enumerator, when prompted to find a  proxy, looked to the left and then right and, finding no one, closed the  case. Similarly, another enumerator ignored the prompt to find a proxy  and explained that neighbors are usually not responsive or willing to  provide information about the neighbor and did not seek to find a proxy.  Enumerators we interviewed did not seem to understand the importance  of obtaining a successful proxy interview and many appeared to have  received little encouragement during training to put in effort to find a  proxy.
Proxy data for occupied households are important to the success of the  census because the alternative is a non-interview. In 2010 about one- fourth of the NRFU interviews for occupied housing units were conducted  using proxy data. We shared our observations with Bureau officials who  told us that they are aware that enumerator training for proxies needs to  be revised to convey the importance of collecting proxy data when  necessary. Converting non-interviews by collecting respondent or proxy  data can improve interview completion rates, and ultimately the quality of  census data. The Bureau told us it will continue to refine procedures for  2020.
section_title: Enumerators Faced Challenges Implementing New Procedures
paragraphs: According to the Bureau, its plans to automate the assignment of NRFU  cases have the potential to deliver significant efficiency gains. At the  same time, improving certain enumeration procedures and  communicating better could produce additional efficiencies by enabling  the Bureau to be more responsive to situations enumerators encounter in  the course of their follow-up work.
Enumerators were unable to access recently closed incomplete  cases. Under current procedures, if an enumerator is unable to make  contact with a household member, the case management system closes  that case to be reattempted at a later date, perhaps by a different  enumerator; assuming fewer than six attempts have been made.  Decisions on when re-attempts will be made—and by whom—are  automated and not designed to be responsive to the immediate  circumstances on the ground. This is in contrast to earlier decennials  when enumerators, using paper-based data collection procedures, had  discretion and control over when to re-attempt cases in the area where  they were working. According to the Bureau, leaving cases open for re- attempts can undermine the efficiency gains of automation when  enumerators depart significantly from their optimized route, circling back  needlessly to previously attempted cases rather than progressing through  their scheduled workload.
During our test site observations, however, we found how this approach  could lead to inefficiencies in certain circumstances. For example, we  observed enumerators start their NRFU visits in the early afternoon as  scheduled, when many people are out working or are otherwise away. If  no one answered the door, those cases were closed for the day and  reassigned later. However, if a household member returned while the  enumerator was still around, the enumerator could not reopen the case  and attempt an interview. We saw this happen at both test site locations,  typically in apartment buildings or at apartment-style gated communities,  where enumerators had clear visibility of a large number of housing units  and could easily see people arriving home.
Bureau officials acknowledged that closing cases in this fashion  represented a missed opportunity and plan to test greater flexibilities as  part of the 2018 End-to-End Test. Programming some flexibility into the  mobile device—if accompanied with adequate training on how and when  to use it—should permit some interviews to be completed without having  to deploy staff to the same case on subsequent days. This in turn could  reduce the cost of follow-up attempts and improve interview completion  rates.
Enumerators did not understand procedures for visits to property  managers. Property managers are a key source of information on non- respondents when enumerators cannot find people at home. They can  also facilitate access to locked buildings. Further, developing a rapport  with property managers has helped the NRFU process, such as when  repeated access to a secured building or residential complex is needed  on subsequent days by different enumerators.
In response to problems observed during the Bureau’s 2014 and 2015  Census tests and to complaints from property managers about multiple  uncoordinated visits by enumerators, the Bureau’s 2016 Census Test  introduced specific procedures to conduct initial visits to property  managers in large multi-unit apartment buildings. The procedures sought  to identify up front which, if any, units needing follow-up were vacant,  eliminating the need for enumerators to collect this information from  property managers with subsequent visits on a case-by-case basis.  According to Bureau officials, the automated case management system  was designed to allow for an enumerator to make up to three visits to  property managers to remove vacant units.
According to the Bureau, the 2016 Census Test demonstrated that vacant  units could quickly be removed from the NRFU workload using these  procedures in cases where a property manager was readily available;  however, in other cases the procedures caused confusion. For example,  whenever an initial visit was unsuccessful, all of the cases at that  location—up until then collated into only one summary row of the  enumerator’s on-screen case list—would suddenly expand and appear as  individual cases to be worked, sometimes adding several screens and  dozens of cases to the length of the list, which the enumerators we spoke  with found confusing. Furthermore, without the knowledge of which units  were vacant, enumerators may have unnecessarily visited these units  and increased the cost and the time required to complete NRFU.
During debriefing sessions the Bureau held, enumerators and their  supervisors identified training in these procedures as an area they felt  needed greater attention in the future. Bureau officials said that they are  pleased that the test demonstrated their progress in automating case  management at multi-unit locations, but at the same time, they recognize  the need to better refine the initial property manager contact procedures  and integrate multi-unit procedures into the training.
section_title: The Bureau Can Better Leverage Enumerator- Collected Information on Respondent Availability
paragraphs: During our field visits, we encountered several instances where  enumerators had been told by a respondent or otherwise learned that  returning at a specific time on a later date would improve their chance of  obtaining an interview from either a household respondent or a property  manager. According to the Bureau, while there was a mechanism for  capturing and using this information, it was not uniformly available to the  enumerators, nor did the enumerators always use the mechanism when  appropriate. As a result, the Bureau’s 2016 Census Test and automated  case management system did not have an efficient way to leverage that  information. Attempting to contact non-responding households at times  respondents are expected to be available can increase the completion  rate and reduce the need to return at a later date or rely on proxy  interviews as a source of information.
The Bureau’s automated case management system used estimated hour- by-hour probabilities for the best time to contact people when making  enumerator assignments. The estimation relied on various administrative  records, information from other Bureau surveys that had successful  contacts in the past, as well as area characteristics. The 2016 Census  Test did not have a way to change or update these estimates when cases  were subsequently reassigned. The assigned time windows were  intended to result in more productive visits and reduce costs.
When enumerators identified potentially better times to attempt a contact,  they were instructed to key this information into their mobile devices. For  example, one enumerator keyed in a mother’s request to come back  Thursday afternoon when her kids were in camp, while others keyed in  information like office hours and telephone contact numbers obtained  from signs on the property they had seen for property managers.  However, according to the Bureau this updated information went unused,  and we met enumerators who had been assigned to enumerate  addresses at the same unproductive time after they had written notes  documenting other better times to visit. Another enumerator reported  visiting a property manager who complained that the enumerator was not  honoring the manager’s earlier request made during a prior enumeration  attempt that an enumerator return during a specified time window. Such  repeat visits can waste enumerator time (and miles driven), and  contribute to respondent burden or reduced data quality when  respondents become annoyed and may become less cooperative.
We discussed our preliminary observation with Bureau managers at the  test sites, who expressed frustration that the automated case  management system did not allow them to use the locally-obtained data  on when to contact people whom they found in enumerator notes in a way  to affect future case assignment. Headquarters staff told us that while  they have not fully evaluated this yet, they are concerned that providing  local managers with too much flexibility to override the results of  optimized case and time assignments would undermine the efficiency  gains achievable by the automation. They also explained that  enumerators were provided the capability to record what day or what time  of day for follow-up. This information could have been used by the  automated case management system to better target the timing of future  assignments. However, they acknowledged that this procedure may not  have been explained during enumerator training. Reviewing the  enumerator training manual, we confirmed that there were no procedures  to allow enumerators to systematically record what day or what time of  day to follow-up at a housing unit. Bureau officials have said that this is  another area they are looking into and plan to address.
section_title: Conclusions
paragraphs: The key innovations the Bureau plans for 2020 show promise for  controlling costs and maintaining accuracy, although there are significant  risks involved. The Bureau is aware of these risks, and robust testing can  help manage them by assessing the feasibility of key activities, their  capacity to deliver desired outcomes, and their ability to work in concert  with one another under operational conditions.
Going forward, to help ensure a cost-effective enumeration, it will be  important for the Bureau to improve its NRFU procedures by addressing  the challenges identified during the 2016 Test, updating related training  materials as needed, and completing these efforts in time to be included  in the Bureau’s End-to-End Test scheduled for 2018. The challenges we  observed include (1) reducing high non-interview rates, (2) difficulty  accessing recently closed, incomplete cases, (3) the need to improve  coordination with managers of multi-unit properties, and (4) the need to  better leverage operational information collected by enumerators.  Resolving these issues should help the Bureau improve its ability to  collect quality data and reduce the cost of unnecessary follow-up visits  during NRFU.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce and Under Secretary for  Economic Affairs direct the Census Bureau to take the following actions:  1.  Determine the cause(s) for non-interviews experienced during the  non-response follow-up operation and revise and test what, if any,  changes need to be made to operational procedures, training, or both,  including making contact with proxy respondents.  2.  Revise and test operational procedures for accessing incomplete  closed cases and revise and test training material to reflect when this  flexibility to access incomplete closed cases should be used by the  enumerator.  3.  Revise and test operational procedures and relevant training materials  for initial property manager visits to ensure procedures and training  material are communicated to and understood by enumerators and  their supervisors.  4.  Revise and test procedures on how to better leverage enumerator- collected information on the best time or day to conduct interviews,  and ensure enumerators are properly trained on these procedures.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of the Department of  Commerce for comment. In its written comments, reproduced in appendix  I, the Department of Commerce agreed with our findings and  recommendations. The Census Bureau also provided technical  comments that we incorporated, as appropriate.
We are sending copies of this report to the Secretary of Commerce, the  Counselor to the Secretary with Delegated Duties of the Undersecretary  of Commerce for Economic Affairs, the Director of the U.S. Census  Bureau, and interested congressional committees. The report also will be  available at no charge on our website at http://www.gao.gov.
If you have any questions about this report please contact me at (202)  512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. The GAO staff that made major contributions to this report  are listed in appendix II.
section_title: Appendix I: Comments from the Department of Commerce
section_title: Appendix II: GAO Contact and Staff Acknowledgements
subsections: section_title: GAO Contact
paragraphs: Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov.
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Lisa Pearson, Assistant Director;  Mark Abraham, Shea Bader, Richard Hung, Donna Miller, Ty Mitchell,  Cynthia Saunders; A.J. Stephens, and Timothy Wexler made significant  contributions to this report.