2020 Census: Actions Needed to Improve In-Field Address Canvassing Operation
Released: 2018-07-16T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The success of the decennial census depends in large part on the Bureau's ability to locate every household in the United States. To accomplish this monumental task, the Bureau must maintain accurate address and map information for every location where a person could reside. For the 2018 End-to-End Test, census workers known as listers went door-to-door to verify and update address lists and associated maps in selected areas of three test sites—Bluefield-Beckley-Oak Hill, West Virginia; Pierce County, Washington; and Providence County, Rhode Island.

GAO was asked to review in-field address canvassing during the End-to-End Test. This report determines whether key address listing activities functioned as planned during the End-to-End Test and identifies any lessons learned that could inform pending decisions for the 2020 Census. To address these objectives, GAO reviewed key documents including test plans and training manuals, as well as workload, productivity and hiring data. At the three test sites, GAO observed listers conducting address canvassing.

section_title: What GAO Found
paragraphs: The Census Bureau (Bureau) recently completed in-field address canvassing for the 2018 End-to-End Test. GAO found that field staff known as listers generally followed procedures when identifying and updating the address file; however, some address blocks were worked twice by different listers because the Bureau did not have procedures for reassigning work from one lister to another while listers work offline. Bureau officials told GAO that they plan to develop procedures to avoid duplication but these procedures have not been finalized. Duplicating work decreases efficiency and increases costs.

GAO also found differences between actual and projected data for workload, lister productivity, and hiring.

For the 2020 Census, the Bureau estimates it will have to verify 30 percent of addresses in the field. However, at the test sites, the actual workload ranged from 37 to 76 percent of addresses. Bureau officials told GAO the 30 percent was a nationwide average and not site specific; however, the Bureau could not provide documentation to support the 30 percent workload estimate.

At all three test sites listers were significantly more productive than expected possibly because a design change provided better quality address and map data in the field, according to the Bureau.

Hiring, however, lagged behind Bureau goals. For example, at the West Virginia site hiring was only at 60 percent of its goal. Bureau officials attributed the shortfall to a late start and low unemployment rates.

Workload and productivity affect the cost of address canvassing. The Bureau has taken some steps to evaluate factors affecting its estimates, but continuing to so would help the Bureau refine its assumptions to better manage the operation's cost and hiring.

Listers used laptops to connect to the Internet and download assignments. They worked offline and went door-to-door to update the address file, then reconnected to the Internet to transmit their completed assignments. Bureau officials told GAO that during the test 11 out of 330 laptops did not properly transmit address and map data collected for 25 blocks. Data were deleted on 7 laptops. Because the Bureau had known there was a problem with software used to transmit address data, it created an alert report to notify the Bureau staff if data were not properly transmitted. However, Bureau officials said that either responsible staff did not follow procedures to look at the alert reports or the reports were not triggered. The Bureau is working to fix the software problem and develop new alert reports, but has not yet determined and addressed why these procedures were not followed.

The Bureau's data management reporting system did not always provide accurate information because of a software issue. The system was supposed to pull data from several systems to create a set of real-time cost and progress reports for managers to use. Because the data were not accurate, Bureau staff had to rely on multiple systems to manage address canvassing. The Bureau agreed that not only is inaccurate data problematic, but that creating workarounds is inefficient. The Bureau is developing new requirements to ensure data are accurate but these requirements have not been finalized.

section_title: What GAO Recommends
paragraphs: GAO is making seven recommendations to the Department of Commerce and Bureau including to: (1) finalize procedures for reassigning work, (2) continue to evaluate workload and productivity data, (3) fix software problem, or determine and address why procedures were not followed, and (4) finalize report requirements to ensure data are accurate. The Department of Commerce agreed with GAO's recommendations, and the Bureau provided technical comments that were incorporated, as appropriate.

section_title: Letter
paragraphs: The federal government is constitutionally mandated to undertake the  decennial census, a complex and costly activity—estimated at $15.6  billion (dollars inflated to the current 2020 Census time frame fiscal years  2012-2023) for the 2020 Census. The data that the census produces are  used to apportion the seats of the U.S. House of Representatives; realign  the boundaries of the legislative districts of each state; allocate hundreds  of billions of dollars in federal financial assistance; and provide a social,  demographic, and economic profile of the nation’s people to guide policy  decisions at each level of government.

The success of the census depends largely on the ability of the Census  Bureau (Bureau) to locate every person residing in the United States. To  accomplish this monumental task, the Bureau must maintain accurate  address and map information for every person’s residence. If the  Bureau’s address list and maps are inaccurate, people can be missed,  counted more than once, or included in the wrong location. In an effort to  help control costs, the Bureau is using new procedures to build its  address list for 2020. As these procedures have not been used in prior  decennials, the Bureau has conducted several tests in the last few years  to help ensure the new approach will function as planned and produce a  complete and accurate address database. The 2018 End-to-End Test is  the last opportunity to demonstrate census technology and procedures— including new methods for building the address list—across a range of  geographic locations, housing types, and demographic groups under  census-like conditions before the 2020 Census.

On August 28, 2017, the Bureau began what it calls the “in-field” address  canvassing operation for the End-to-End Test where temporary census  employees known as listers walked the streets of designated census  blocks. In three test sites—Bluefield-Beckley-Oak Hill, West Virginia;  Pierce County, Washington; and Providence County, Rhode Island— listers knocked on doors and, using laptops connected to the internet,  verified the address and geographic location of assigned housing units  and identified any additions, deletions, and any other changes that need  to be made to the address list. For example, they would add converted  basements, attics, and other “hidden” housing units to the list.

You asked us to review how the address canvassing operation performed  as part of the 2018 End-to-End Test. This report (1) determines the extent  to which key “in-field” address listing activities functioned as planned and  (2) identifies any lessons learned that could potentially affect pending  decisions for the 2020 Census.

To address these objectives, we reviewed key documents including the  2018 End-to-End Test plan that discussed the goals and objectives for  the test, as well as training manuals and other related documents for  address canvassing. We interviewed Bureau staff at the three 2018  Census test sites including census field supervisors (CFS), address  listers, and office personnel to discuss what went well and what  challenges they faced during address canvassing. At each test site, the  Bureau selected Census field staff for us to interview and observe from  among those working on the days of our visits. At all three test sites, we  observed listers conduct address canvassing. In addition, we used the  training manuals to determine whether listers collected address  information as prescribed by the Bureau. In total we conducted 18 in-field  observations of listers and used a data collection instrument to document  our observations. These observations are not generalizable. We also  interviewed Bureau headquarters officials to discuss the use of  management reports for monitoring and overseeing the operation.

We reviewed workload estimates, address lister productivity rates, and  hiring information for each test site in order to report how many housing  units were included at each test site, how many addresses the Bureau  expected to canvass per hour, and how many people they needed to hire.  To assess the reliability of these data, we reviewed available  documentation and interviewed knowledgeable officials. We found the  data to be sufficiently reliable for the purposes of our reporting objectives.  We also met periodically with Bureau headquarters staff to discuss  progress of the operation.

We conducted this performance audit from July 2017 to June 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Bureau’s address canvassing operation updates its address list and  maps, which are the foundation of the decennial census. An accurate  address list both identifies all households that are to receive a notice by  mail requesting participation in the census (by Internet, phone, or mailed- in questionnaire) and serves as the control mechanism for following up  with households that fail to respond to the initial request. Precise maps  are critical for counting the population in the proper locations—the basis  of congressional apportionment and redistricting.

Our prior work has shown that developing an accurate address list is  challenging—in part because people can reside in unconventional  dwellings, such as converted garages, basements, and other forms of  hidden housing. For example, as shown in figure 1, what appears to be a  single-family house could contain an apartment, as suggested by its two  doorbells.

During address canvassing, the Bureau verifies that its master address  list and maps are accurate to ensure the tabulation for all housing units  and group quarters is correct. For the 2010 Census, the address  canvassing operation mobilized almost 150,000 field workers to canvass  almost every street in the United States and Puerto Rico to update the  Bureau’s address list and map data—and in 2012 reported the cost at  nearly $450 million. The cost of going door-to-door in 2010, along with the  emerging availability of imagery data, led the Bureau to explore an  approach for 2020 address canvassing that would allow for fewer boots  on the ground.

Traditionally, the Bureau went door-to-door to homes across the country  to verify addresses. This “in-field address canvassing” is a labor-intensive  and expensive operation. To achieve cost savings, in September 2014  the Bureau decided to use a reengineered approach for building its  address list for the 2020 Census and not go door-to-door (or “in-field”)  across the country, as it has in prior decennial censuses. Rather, some  areas (known as “blocks”) would only need a review of their address and  map information using computer imagery and third-party data sources— what the Bureau calls “in-office” address canvassing procedures.

According to the Bureau’s address canvassing operational plan, in-office  canvassing had two phases:

During the first phase, known as “Interactive Review,” Bureau  employees use current aerial imagery to determine if areas have  housing changes, such as new residential developments or  repurposed structures, or if the areas match what is in the Bureau’s  master address file. The Bureau assesses the extent to which the  number of housing units in the master address file is consistent with  the number of units visible in the current imagery. If the housing  shown in the imagery matches what is listed in the master address  file, then those areas are considered to be resolved or stable and  would not be canvassed in-field.

During the second phase, known as “Active Block Resolution,”  employees would try to resolve coverage concerns identified during  the first phase and verify every housing unit by virtually canvassing  the entire area. As part of this virtual canvass, the Bureau would  compare what is found in imagery to the master address file data and  other data sources in an attempt to resolve any discrepancies. If  Bureau employees still could not reconcile the discrepancies, such as  housing unit count or street locations with what is on the address list,  then they would refer these blocks to in-field address canvassing.

However, in March 2017, citing budget uncertainty the Bureau decided to  discontinue the second phase of in-office review for the 2020 Census.  According to the Bureau, in order to ensure that the operations  implemented in the 2018 End-to-End Test were consistent with  operations planned for the 2020 Census, the Bureau added the blocks  originally resolved during the second phase of in-office review back into  the in-field workload for the test. The cancellation of Active Block  Resolution is expected to increase the national workload of the in-field  canvassing workload by 5 percentage points (25 percent to 30 percent).

During in-field address canvassing, listers use laptop computers to  compare what they see on the ground to what is on the address list and  map. Listers confirm, add, delete, or move addresses to their correct map  positions. At each housing unit, listers are trained to speak with a  knowledgeable resident to confirm or update address data, ask about  hidden housing units, confirm the housing unit location on the map,  (known as the map spot) and collect a map spot using global positioning  systems (GPS). If no one is available, listers are to use house numbers  and street signs to verify the address data. The data are transmitted  electronically to the Bureau.

The Census Bureau expects that the End-to-End Test for address  canvassing will identify areas for improvement and changes that need to  be made for the 2020 Census. Our prior work has shown the importance  of robust testing. Rigorous testing is a critical risk mitigation strategy  because it provides information on the feasibility and performance of  individual census-taking activities, their potential for achieving desired  results, and the extent to which they are able to function together under  full operational conditions.

In February 2017, we added the 2020 Census to GAO’s High-Risk List  because operational and other issues are threatening the Bureau’s ability  to deliver a cost-effective enumeration. We reported on concerns about  the Bureau’s capacity to implement innovative census-taking methods,  uncertainties surrounding critical information technology systems, and the  quality of the Bureau’s cost-estimates. Underlying these issues are  challenges in such essential management functions as the Bureau’s  ability to:  collect and use real-time indicators of cost, performance, and  schedule;  follow leading practices for cost estimation; scheduling; risk  management; IT acquisition, development, testing, and security; and  cost-effectively deal with contingencies including, for example, fiscal  constraints, potential changes in design, and natural disasters.

section_title: The Listers Generally Followed Procedures, but the Bureau Experienced Some Issues Reassigning Work, Estimating Workload and Lister Productivity, and Managing to Staffing Goals
paragraphs: The Bureau completed in-field address canvassing as scheduled by  September 29, 2017, canvassing approximately 340,400 addresses. Most  of the listers we observed generally followed procedures. For example, 15  of 18 listers knocked on doors, and 16 of 18 looked for hidden housing  units, which is important for establishing that address lists and maps are  accurate and for identifying hard-to-count populations. Those procedures  include taking such steps as:  comparing the housing units they see on the “ground” to the housing  units on the address list,  knocking on all doors so they could speak with a resident to confirm  the address (even if the address is visible on the mailbox or house)  and to confirm that there are no other living quarters such as a  basement apartment,  looking for “hidden housing units”,  looking for group quarters such as group homes or dormitories, and  confirming the location of the housing unit on a map with GPS  coordinates collected on the doorstep.

To the extent procedures were not followed, it generally occurred when  listers did not go up to the door and speak with a resident or take a map  spot on the doorstep. Failure to follow procedures could adversely affect  a complete count, as addresses could be missed or a group quarter could  be misclassified as a residential address. After we alerted the Bureau to  our observations, the Bureau agreed moving forward, to emphasize the  importance of following procedures during training for in-field address  canvassing.
subsections: section_title: Some Listers Duplicated Each Other’s Work Due to a Lack of Operational Procedures for Reassigning Work
paragraphs: Address canvassing has tight time frames, so work needs to be assigned  efficiently. Sometimes this means the Bureau needs to reassign work  from one lister to another. During address canvassing, the Bureau  discovered that reassigned census blocks sometimes would appear in  both the new and the original listers’ work assignments. In some cases,  this led to blocks being worked more than once, which decreased  efficiency, increased costs, and could create confusion and credibility  issues when two different listers visit a house.

According to Bureau procedures, listers were instructed to connect to the  Bureau’s Mobile Case Management (MCM) system to download work  assignments (address blocks) and to transmit their completed work at the  beginning and end of the work day but not during the work day. Thus  during the work day, they were unaware when unworked blocks had been  reassigned to another lister. Bureau officials also told us that the Listing  and Mapping Application (LiMA) software used to update the address file  and maps was supposed to have the functionality to prevent blocks from  being worked more than once, but this functionality was not developed  because of budget cuts.

For 2020, Bureau officials told us they plan to create operational  procedures for reassigning work. According to Bureau officials, they plan  to require supervisors to contact the original lister when work is  reassigned. We have requested a copy of those procedures; however,  the Bureau has not finalized them. Standards for Internal Control in the  Federal Government (Standards for Internal Control) call for management  to design control activities, such as policies and procedures to achieve  objectives. Finalizing these procedures should help prevent blocks from  being canvassed more than once.

section_title: The Bureau Has Not Evaluated Workload, Productivity Rates, and Staffing Assumptions for Address Canvassing
paragraphs: The Bureau conducts tests under census-like conditions, in part, to verify  2020 Census planning assumptions, such as workload, how many  houses per hour a lister can verify (also known as a lister’s productivity  rate), and how many people the Bureau needs to hire for an operation.  Moreover, one of the objectives of the test is to validate that the  operations being tested are ready at the scale needed for the 2020  Census. For the 2018 End-to-End Test, the Bureau completed in-field  address canvassing on time at two sites and early at one site; despite  workload increases at all three test sites and hiring shortfalls at two sites.  The Bureau credits this success to better than expected productivity. As  the Bureau reviews the results of address canvassing, evaluating the  factors that affected workload, productivity rates, and staffing and making  adjustments to its estimates, if necessary, before the 2020 Census would  help the Bureau ensure that address canvassing has the appropriate  number of staff and equipment to complete the work in the required time  frame.
subsections: section_title: Workload
paragraphs: For the 2020 Census, the Bureau estimates it will have to send 30  percent of addresses to the field for listers to verify. However, at the three  test sites, the workload was higher than this estimate (see table 1). At one  test site, the percent of addresses verified through in-field address  canvassing was 76 percent or 46 percentage points more than the  Bureau’s expected 2020 Census in-field address canvassing workload  estimate of 30 percent.

Bureau officials told us that the 30 percent in-field workload estimate is a  national average and is not specific to any of the three test sites. Prior to  the test, officials said that the Bureau also knew that the West Virginia  site was assigning new addresses to some of the test site’s housing units  due to local government emergency 911 address conversion and that the  in-field workload would be greater in West Virginia when compared to the  other test sites.

We requested documentation for the Bureau’s original estimate that 30  percent of the 133.8 million expected addresses would be canvassed in- field for the 2020 Census. However, the Bureau was unable to provide us  with documentation to support how they arrived at the 30 percent  estimate. Instead, the Bureau provided us with a November 2017  methodology document that showed three in-field address canvassing  workload scenarios, whereby, between 41.9 and 45.1 percent of housing  units would need to go to the field for address canvassing. The three  scenarios consider a range of stability in the address file as well as  different workload estimates for in-field follow-up. At 30 percent the  Bureau would need to canvass about 40.2 million addresses; however, at  41.9 and 45.1 percent the Bureau would need to canvass between 56  million and 60.4 million addresses, respectively. According to Bureau  officials, they are continuing to assess whether changes to its in-office  address canvassing procedures would be able to reduce the in-field  address canvassing workload to 30 percent, while at the same time  maintaining address quality. However, Bureau officials did not provide us  with documentation to show how the in-field address canvassing  workload would be reduced because the proposed changes were still  being reviewed internally.

Workload for address canvassing directly affects cost – the greater the  workload the more people as well as laptop computers needed to carry  out the operation. We found that the 30 percent workload threshold is  what is reflected in the December 2017 updated 2020 Census cost  estimate that was used to support the fiscal year 2019 budget request.  Thus, if the 30 percent threshold is not achieved then the in-field  canvassing workload will likely increase for the 2020 Census and the  Bureau would be at risk of exceeding its proposed budget for the address  canvassing operation.

Standards for Internal Control call for organizations to use quality  information to achieve their objectives. Thus, continuing to evaluate and  finalize workload estimates for in-field address canvassing with the most  current information will help ensure the Bureau is well-positioned to  conduct addressing canvassing for the 2020 Census. For example,  according to Bureau officials, preliminary workload estimates will need to  be delivered by January 2019 for hiring purposes and the final in-field  workload numbers for address canvassing will need to be determined by  June 2019 for the start of address canvassing, which is set to begin in  August 2019. Moreover, by February 2019 the Bureau’s schedule calls for  it to determine how many laptops will be needed to conduct 2020 Census  address canvassing.

section_title: Lister Productivity
paragraphs: At the test sites, listers were substantially more productive than the  Bureau expected. The expected production rate is defined as the number  of addresses expected to be completed per hour, and it affects the cost of  the address canvassing operation. This rate includes time for actions  other than actually updating addresses, such as travel time. In the 2010  Census the rates reflected different geographic areas, and the country  was subdivided into three areas: urban/suburban, rural, and very rural.  According to Bureau officials, for the 2020 Census the Bureau will have  variable production rates based on geography, similar to the design used  in the 2010 Census. The Bureau told us they have not finalized the 2020  Census address canvassing production rates.

Table 2 shows the expected and actual productivity rates (addresses per  hour) for the in-field address canvassing operation at all three test sites.

To ensure address canvassing for the test was consistent with the 2020  Census, Bureau officials told us they included the blocks resolved during  the now discontinued second phase of in-office review, into the in-field  workload for the test. The Bureau attributed the greater productivity to this  discontinued second phase. Bureau officials told us that they believe that  listers spent less time updating those blocks because they had already  been resolved, and any necessary changes were already incorporated.  Moreover, while benefitting from the second phase of in-office address  canvassing may be one explanation for why listers were more productive.  Bureau officials told us that they are unable to evaluate the differences in  expected versus actual productivity for blocks added to the workload as a  result of the discontinued second phase because of limitations with the  data. However, there could be other reasons as well such as travel time  and geography. Standards for Internal Control require that organizations  use quality information to achieve their objectives. Therefore, continuing  to evaluate other factors from the 2018 End-to-End Test that may have  increased or could potentially decrease productivity will be important for  informing lister productivity rates for 2020, as productivity affects the  number of listers needed to carry out the operation, the number of staff  hours charged to the operation, and the number of laptops to be  procured.

section_title: Hiring
paragraphs: For the 2018 End-to-End Test address canvassing operation, the Bureau  hired fewer listers than it assumed it needed at two sites and hired more  at the other site. In West Virginia, 60 percent of the required field staff  was hired and in Washington, 74.5 percent of the required field staff was  hired. Nevertheless, the operation finished on schedule at both these  sites. In contrast in Rhode Island the Bureau hired 112 percent of the  required field staff and finished early.

According to Bureau officials, both the West Virginia and Washington  state test sites started hiring field staff later than expected because of  uncertainty surrounding whether the Bureau would have sufficient funding  to open all three test sites for the 2018 End-to-End Test. When a decision  was made to open all three sites for the address canvassing operation  only, that decision came late, and Bureau officials told us that once they  were behind in hiring and were never able to catch up because of low  unemployment rates and the short duration of the operation. According  to Bureau officials, their approach to hiring for the 2018 End-to-End Test  was similar to that used for the 2010 and 2000 Censuses. In both  censuses the Bureau’s goal was to recruit and hire more workers than it  needed because of immutable deadlines and attrition.

After the 2010 Census we reported that the Bureau had over recruited;  conversely, for the 2000 Census the Bureau had recruited in the midst of  one of the tightest labor markets in three decades. Thus we  recommended, and the Bureau agreed to evaluate current economic  factors that are associated with and predictive of employee interest in  census work, such as national and regional unemployment levels, and  use these available data to determine the potential temporary workforce  pool and adjust its recruiting approach. The Bureau implemented this  recommendation, and used unemployment and 2010 Census data to  determine a base recruiting goal at both the Los Angeles, California and  Houston, Texas 2016 census test sites. Specifically, the recruiting goal for  Los Angeles was reduced by 30 percent.

Bureau officials told us that it continues to gather staffing data from the  2018 End-to-End Test that will be important to consider looking forward to  2020. Although address canvassing generally finished on schedule even  while short staffed, Bureau officials told us they are carefully monitoring  recruiting and hiring data to ensure they have sufficient staff for the test’s  next census field operation non-response follow-up, when census  workers go door-to-door to follow up with housing units that have not  responded. Non-response follow-up is set to begin in May 2018.  According to test data as of March 2018, the Bureau is short of its  recruiting goal for this operation which is being conducted in Providence  County, Rhode Island. The Bureau’s goal is to recruit 5,300 census  workers and as of March 2018, the Bureau had only recruited 2,732  qualified applicants to fill 1,166 spots for training and deploy 1,049 census  workers to conduct non-response follow-up. Bureau officials told us they  believe that low unemployment is making it difficult to meet its recruiting  goals in Providence County, Rhode Island, but they are confident they will  be able to hire sufficient staff without having to increase pay rates.

Recruiting and retaining sufficient staff to carry out operations as labor- intensive as address canvassing and nonresponse follow-up for the 2020  Census is a huge undertaking with implications for cost and accuracy.  Therefore, striking the right staffing balance for the 2020 Census is  important for ensuring deadlines are met and costs are controlled.

section_title: Resolving Challenges from the Address Canvassing Test Will Better Position the Bureau for the 2020 Census
subsections: section_title: The Bureau Does Not Have Procedures to Ensure All Collected Address Canvassing Data Are Retained
paragraphs: Bureau officials told us that during the test 11 out of 330 laptop computers  did not properly transmit address and map data collected for 25 blocks.  The lister-collected address file and map data are supposed to be  electronically transmitted from the listers’ laptops to the Bureau’s data  processing center in Jeffersonville, Indiana. The data are encrypted and  remain on the laptop until the laptops are returned to the Bureau where  the encrypted data are deleted. Prior to learning that not all data had  properly transmitted off the laptops, data on seven of the laptops was  deleted. Data on the remaining four laptops were still available. In  Providence, Rhode Island, where the full test will take place, the Bureau  recanvassed blocks where data were lost to ensure that the address and  map information for nonresponse follow-up was correct. Recanvassing  blocks increases costs and can lead to credibility problems for the Bureau  when listers visit a home twice.

Going into address canvassing for the End-to-End Test, Bureau officials  said they knew there was a problem with the LiMA software used to  update the Bureau’s address lists and maps. Specifically, address and  map updates would not always transfer when a lister transmitted their  completed work assignments from the laptop to headquarters. Other  census surveys using LiMA had also encountered the same software  problem. Moreover, listers were not aware that data had not transmitted  because there was no system-generated warning. Bureau officials are  working to fix the LiMA software problem, but told us that the software  problem has been persistent across other census surveys that use LiMA  and they are not certain it will be fixed.

Bureau officials told us that prior to the start of address canvassing they  created an alert report to notify Bureau staff managing the operation at  headquarters if data were not properly transmitted. When transmission  problems were reported, staff was supposed to remotely retrieve the data  that were not transmitted. This workaround was designed to safeguard  the data but according to officials was not used. Bureau officials told us  that they do not know whether this was because the alert reports were not  viewed by responsible staff or whether the alert report to notify the  Bureau staff managing the operation was not triggered. Bureau officials  told us they recognize the importance of following procedures to monitor  alert reports, and acknowledge that the loss of data on seven of the  laptops may have been avoided had the procedures that alert reports get  triggered and monitored been followed; however, officials did not know  why the procedures were not followed.

For 2020, if the software problem is not resolved, then officials said the  Bureau plans to create two new alert reports to monitor the transmission  of data. One report would be triggered when the problem occurs and a  second report would capture a one-to-one match between data on the  laptop and data transmitted to the data center so that discrepancies  would be immediately obvious. While these new reports should help  ensure that Bureau staff are alerted when data has not properly  transmitted, the Bureau has not determined and addressed why the  procedures that required an alert report get triggered and then reviewed  by Bureau staff did not work as intended. Standards for Internal Control  require that organizations safeguard data and follow policies and  procedures to achieve their objectives. Thus, either fixing the LiMA  software problem, or if the software problem cannot be fixed, then  determining and addressing why procedures that alert reports get  triggered and monitored were not followed would position the Bureau to  help prevent future data losses.

section_title: More Useful and Accurate Monitoring Data for Field Supervisors Would Strengthen Management of Operations
paragraphs: To effectively manage address canvassing, the Bureau needs to be able  to monitor the operation’s progress in near real time. Operational issues  such as listers not working assigned hours or falling behind schedule  need to be resolved quickly because of the tight time frames of the  address canvassing and subsequent operations. During the address  canvassing test, the Bureau encountered several challenges that  hindered its efforts to efficiently monitor lister activities as well as the  progress of the address canvassing operation.
subsections: section_title: System Alerts Were Not Consistently Used by Supervisors
paragraphs: The Bureau provides data-driven tools for the census field supervisors to  manage listers, including system alerts that identify issues that require the  supervisor to follow-up with a lister. For the address canvassing  operation, the system could generate 14 action codes that covered a  variety of operational issues such as unusually high or low productivity  (which may be a sign of fraud or failure to follow procedures) and  administrative issues such as compliance with overtime and completion of  expense reports and time cards.

During the operation, over 8,250 alerts were sent to CFSs or about 13  alerts were sent per day per CFS. Each alert requires the CFS to take  action and then record how the alert was resolved. CFSs told us and the  Bureau during debriefing sessions that they believed many of the  administrative alerts were erroneous and they dismissed them. For  example, during our site visit one CFS showed us an alert that incorrectly  identified that a timecard had not been completed. The CFS then showed  us that the lister’s timecard had indeed been properly completed and  submitted. CFSs we spoke to said that they often dismissed alerts related  to expense reports and timecards and did not pay attention to them or  manage them. Bureau officials reported that one CFS was fired for not  using the alerts to properly manage the operation.

To assist supervisors, these alerts need to be reliable and properly used.  Bureau officials said that they examined alerts for errors after we told  them about our observation. They reported that they did not find any  errors in the alerts. They believe that CFSs may not fully understand that  the alerts stay active until they are marked as resolved by the CFS. For  example, if a CFS gets an alert that a lister has not completed a timecard  the alert will remain active until the CFS resolves the alert by stating the  time card was completed. The Bureau’s current CFS manual does not  address that by the time a CFS sees the alert a lister may have already  taken action to resolve it. Because this was a reoccurring situation, CFSs  told us they had a difficult time managing the alerts.

Standards for Internal Control call for an agency to use quality information  to achieve objectives. Bureau officials acknowledge that it is a problem  that some CFSs view the alerts as erroneous and told us they plan to  address the importance of alerts in training. We spoke to Bureau officials  about making the alerts more useful to CFSs, such as by differentiating  between critical and noncritical alerts and streamlining alerts by perhaps  combining some of them. Bureau officials told us they would monitor the  alerts during the 2018 End-to-End Test’s nonresponse follow-up  operation and make adjustments if appropriate. However, while the  Bureau told us it will monitor alerts for the non-response follow-up  operation, the Bureau does not have a plan for how it will examine and  make alerts more useful.

Ensuring alerts are properly followed up on is critical to the oversight and  management of an operation. If the CFSs view the alerts as unreliable,  they could be likely to miss key indicators of fraud such as unusually high  or low productivity or an unusually high or low number of miles driven.  Moreover, monitoring overtime alerts and the submission of daily time  cards and expense reports is also important to ensure that overtime is  appropriately approved before worked and that listers get paid on time.

section_title: The Bureau’s Management Dashboard Did Not Always Display Accurate Information
paragraphs: Another tool the Bureau uses to monitor operations is its Unified Tracking  System (UTS), a management dashboard that combines data from a  variety of Census systems, bringing the data to one place where the  users can run or create reports. It was designed to track metrics such as  the number and percentage of blocks assigned and blocks completed as  well as the actual expenditures of an operation compared to the budgeted  expenditures. However, information in UTS was not always accurate  during address canvassing. For example UTS did not always report the  correct number of addresses assigned and completed by site. As a result,  Bureau managers reported they did not rely on UTS and instead used  data from the source systems that fed into it. Bureau officials agreed that  inaccurate data is a problem and that this workaround was inefficient as  users had to take extra time to go to multiple systems to get the correct  data.

Bureau officials reported problems importing information from the feeder  systems into UTS because of data mismatches. They said that address  canvassing event codes were not processed sequentially, as they should  have been, which led to inaccurate reporting. Bureau officials told us  that they did not specify that the codes needed to be processed in  chronological order as part of the requirements for UTS. Bureau officials  said UTS passed the requisite readiness reviews and tests. However,  Bureau officials also acknowledged that some of these problems could  have been caught by exception testing which was not done prior to  production.

To resolve this issue for 2020, Bureau officials stated they are developing  new requirements for UTS to automatically consider the chronological  order of event codes. The Bureau told us they are working on these UTS  requirements and will provide us with documentation when they are  complete. They also said the Bureau plans to implement a process which  compares field management reports with UTS reports to help ensure that  the reports have the same definitions and are reporting accurate  information. Standards for Internal Control call for an organization’s data  be complete and accurate and processed into quality information to  achieve their objectives. Thus, finalizing UTS requirements for the  address canvassing reporting should help increase efficiency for the 2020  Census by avoiding time consuming workarounds.

section_title: The Bureau Does Not Have Documented Procedures to Address Broadband Internet Service Coverage Gaps
paragraphs: The Bureau has taken significant steps to use technology to reduce  census costs. These steps include using electronic systems to transmit  listers’ assignments and address and map data. However, during the  address canvassing test, several listers and CFSs at the three test sites  experienced problems with Internet connections primarily during training.  The West Virginia site, which was more rural than the other sites,  experienced the most problems with Internet connectivity. All six West  Virginia CFSs reported Internet connectivity problems during the  operation. As a work around, CFSs told us that a couple of their listers  transmitted their work assignments from libraries where they could  access the Internet.

Bureau officials stated that the laptops in the 2018 End-to-End Test only  used two broadband Internet service providers, which may have  contributed to some of the Internet access issues. Bureau officials added  that despite the reported Internet connectivity issues, the 2018 End-to- End Test for address canvassing finished on schedule and without any  major problems. While this might be true for the test, we have previously  reported that minor problems can become big challenges when the  census scales up to the entire nation. Therefore, it is important that  these issues get resolved before August 2019 when in-field address  canvassing for the 2020 Census is set to begin.

The Bureau is analyzing the cellular network coverage across all 2020  Census areas using coverage maps and other methods to determine  which carrier is appropriate (including a backup carrier) for geographic  areas where network coverage is limited. According to Bureau officials,  they anticipate identifying the cellular carriers for each of its 248 area  census offices by the summer of 2018. The officials said they are  considering both national and regional carriers to provide service in some  geographic areas because the best service provider in a certain  geographic area may not be one of the national providers, but a regional  provider. In those cases, listers and other staff in those areas will receive  devices with the regional carrier. According to Bureau officials, for the  2020 Census, the ability to access multiple carriers should provide field  staff with better connectivity around the country.

We also found that there was no guidance for listers and CFSs on what to  do if they experienced Internet connectivity problems and were unable to  access the Internet. Bureau officials told us that staff in the field can use  different methods to access the Internet, such as using home wireless  networks or mobile hotspots located at libraries, or coffee shops to  transmit data. However, the Bureau did not provide such instructions to  listers. In addition, the Bureau also does not define what constitutes a  secure Internet public connection. Ensuring data are safeguarded is  important because census data are confidential. Bureau officials told us  that the Bureau plans to provide instructions to field staff on what to do if  they are unable to access census systems and what constitutes a secure  Internet connection for the next 2018 End-to-End Test field operation,  non-response follow-up. However, the Bureau has not finalized or  documented these instructions. Standards for Internal Control call for  management to design control activities, such as providing instructions to  employees to achieve objectives. Finalizing these instructions to field staff  will help ensure listers have complete information on how to handle  problems with Internet connectivity and that data are securely transmitted.

section_title: The Bureau Has Not Identified Alternative Sites for Listers to Take Online Training When Access to the Internet is Unavailable
paragraphs: Some listers had difficulty accessing the Internet to take online training for  address canvassing. This is the first decennial census that the Bureau is  using online training, in previous decennials training was instructor-led in  a class room. According to the Bureau, in addition to the Bureau provided  laptop, listers also needed a personal home computer or laptop and  Internet access at their home in order to complete the training. However,  while the Bureau reported that listers had access to a personal computer  to complete the training, we found some listers did not have access to the  Internet at their home and were forced to find workarounds to access the  training.

According to American Community Survey data from 2015, among all  households, 77 percent had a broadband Internet subscription. Bureau  officials told us they are aware that not all households have access to the  Internet and that the Bureau’s field division is working on back-up plans  for accessing online training. Specifically, Bureau officials told us for 2020  they plan to identify areas of the country that could potentially have  connectivity issues and plan to identify alternative locations such as  libraries or community centers where Internet connections are available to  ensure all staff has access to training. However, they have not finalized  those plans to identify locations for training sites. Standards for Internal  Control call for management to design control activities, such as having  plans in place to achieve objectives. Finalizing these plans to identify  alternative training locations will help ensure listers have a place to  access training.

section_title: Conclusions
paragraphs: The Bureau’s re-engineered approach for address canvassing shows  promise for controlling costs and maintaining accuracy. However, the  address canvassing operation in the 2018 End-to-End test identified the  need to reexamine assumptions and make some procedural and  technological improvements. For example, at a time when plans for in- field address canvassing should be almost finalized, the Bureau is in the  process of evaluating workload and productivity assumptions to ensure  sufficient staff are hired and that enough laptop computers are procured.  Moreover, Bureau officials have not finalized (1) procedures for  reassigning work from one lister to another to prevent the unnecessary  duplication of work assignments, (2) instructions for using the Internet  when connectivity is a problem to ensure listers have access to training  and the secure transmission of data to and from the laptops, and (3)  plans for alternate training locations. To ensure address and map data  are not lost during transmission, Bureau officials will also need to either  (1) fix the problem with the LiMA software used to update the address  and map files or (2) determine and address why procedures that alert  reports be triggered and monitored were not followed.

Finally, the Bureau has made progress in using data driven technology to  manage address canvassing operations. However, ensuring data used by  supervisors to oversee and monitor operations are both useful and  accurate will help field supervisors take appropriate action to address  supervisor alerts and will help managers monitor the real-time progress of  the address canvassing operation. With little time remaining it will be  important to resolve these issues. Making these improvements will better  ensure address canvassing for the actual enumeration, beginning in  August 2019, fully functions as planned and achieves desired results.

section_title: Recommendations for Executive Action
paragraphs: We are making the following seven recommendations to the Department  of Commerce and the Census Bureau:

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau continues to evaluate and finalize workload estimates  for in-field address canvassing as well as evaluates the factors that  impacted productivity rates during the 2018 End-to-End Test and, if  necessary, make changes to workload and productivity assumptions  before the 2020 Census in-field address canvassing operation to help  ensure that assumptions that impact staffing and the number of  laptops to be procured are accurate. (Recommendation 1)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau finalizes procedures for reassigning blocks to prevent  the duplication of work. (Recommendation 2)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau finalizes backup instructions for the secure  transmission of data when the Bureau’s contracted mobile carriers are  unavailable. (Recommendation 3)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau finalizes plans for alternate training locations in areas  where Internet access is a barrier to completing training.  (Recommendation 4)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau takes action to either fix the software problem that  prevented the successful transmission of data, or if that cannot be  fixed, then determine and address why procedures that alert reports  be triggered and monitored were not followed. (Recommendation 5)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau develops a plan to examine how to make CFS alerts  more useful so that CFSs take appropriate action, including alerts a  CFS determines are no longer valid because of timing differences.  (Recommendation 6)

Secretary of Commerce should ensure the Director of the U.S.

Census Bureau finalizes UTS requirements for address canvassing  reporting to ensure that the data used by census managers who are  responsible for monitoring real-time progress of address canvassing  are accurate before the 2020 Census. (Recommendation 7)

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Commerce. In its  written comments, reproduced in appendix I the Department of  Commerce agreed with our recommendations. The Census Bureau also  provided technical comments that we incorporated, as appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we are sending copies of this report to the  Secretary of Commerce, the Under Secretary of Economic Affairs, the  Acting Director of the U.S. Census Bureau, and interested congressional  committees. The report also will be available at no charge on GAO’s  website at http://www.gao.gov.

If you have any questions about this report please contact me at (202)  512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff that made major contributions to this report are  listed in appendix II.

section_title: Appendix I: Comments from the Department of Commerce

section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Lisa Pearson, Assistant Director;  Kate Wulff, Analyst-in-Charge; Mark Abraham; Devin Braun; Karen  Cassidy; Robert Gebhart; Richard Hung; Kirsten Lauber; Krista Loose; Ty  Mitchell; Kayla Robinson; Kate Sharkey; Stewart Small; Jon Ticehurst;  and Timothy Wexler made key contributions to this report.

Fast facts: The success of the census depends in large part on the Census Bureau's ability to record every location where a person resides. An ever-changing housing inventory and "hidden" housing—such as a converted attic or shed that may not show up in records—make this huge task even more difficult.

As part of a test run, census workers went door-to-door to try to record each housing unit in selected areas in 3 states. We reviewed this effort and found the workers generally followed procedures but software problems sometimes kept them from transmitting data they collected.

We made 7 recommendations for the 2020 Census.

Question-summary annotations: questions: question: How long ago did the Census Bureau last complete in-field address canvassing?
answer: The Census Bureau (Bureau) recently completed in-field address canvassing for the 2018 End-to-End Test.

question: How was the canvassing process made redundant in some cases?
answer: GAO found that field staff known as listers generally followed procedures when identifying and updating the address file; however, some address blocks were worked twice by different listers because the Bureau did not have procedures for reassigning work from one lister to another while listers work offline.
child_questions: question: How does the Bureau plan to mitigate this redundancy?
answer: Bureau officials told GAO that they plan to develop procedures to avoid duplication but these procedures have not been finalized.

question: How does the redundancy affect the Bureau's work?
answer: Duplicating work decreases efficiency and increases costs.

questions: question: How many addresses does the Bureau expect to have to verify?
answer: For the 2020 Census, the Bureau estimates it will have to verify 30 percent of addresses in the field.
child_questions: question: To what extent is this estimate accurate?
answer: However, at the test sites, the actual workload ranged from 37 to 76 percent of addresses.

question: How was the Bureau estimate so far off from the actual workload?
answer: Bureau officials told GAO the 30 percent was a nationwide average and not site specific; however, the Bureau could not provide documentation to support the 30 percent workload estimate.

questions: question: How were address files updated?
answer: Listers used laptops to connect to the Internet and download assignments. They worked offline and went door-to-door to update the address file, then reconnected to the Internet to transmit their completed assignments.
child_questions: question: To what extent did this system preserve and properly transmit the required data?
answer: Bureau officials told GAO that during the test 11 out of 330 laptops did not properly transmit address and map data collected for 25 blocks. Data were deleted on 7 laptops.

question: How did the Bureau respond to the software issues?
answer: Because the Bureau had known there was a problem with software used to transmit address data, it created an alert report to notify the Bureau staff if data were not properly transmitted.
child_questions: question: To what extent did staff regard these alerts?
answer: However, Bureau officials said that either responsible staff did not follow procedures to look at the alert reports or the reports were not triggered.

question: How is the Bureau attempting to respond to the software and/or user issues?
answer: The Bureau is working to fix the software problem and develop new alert reports, but has not yet determined and addressed why these procedures were not followed.

questions: question: To what extent did the Bureau's data management system provide accurate information?
answer: The Bureau's data management reporting system did not always provide accurate information because of a software issue.

question: How was the system intended to be used?
answer: The system was supposed to pull data from several systems to create a set of real-time cost and progress reports for managers to use.
child_questions: question: How did the system's shortcomings affect how Bureau staff used it?
answer: Because the data were not accurate, Bureau staff had to rely on multiple systems to manage address canvassing.

question: How does the Bureau react to user workarounds?
answer: The Bureau agreed that not only is inaccurate data problematic, but that creating workarounds is inefficient.

question: How is the Bureau responding to the data issues?
answer: The Bureau is developing new requirements to ensure data are accurate but these requirements have not been finalized.

questions: question: How is the decennial census so successful and accurate?
answer: The success of the decennial census depends in large part on the Bureau's ability to locate every household in the United States.
child_questions: question: How does the Bureau keep track of these addresses?
answer: To accomplish this monumental task, the Bureau must maintain accurate address and map information for every location where a person could reside.

question: How is the address information collection process tested?
answer: For the 2018 End-to-End Test, census workers known as listers went door-to-door to verify and update address lists and associated maps in selected areas of three test sites—Bluefield-Beckley-Oak Hill, West Virginia; Pierce County, Washington; and Providence County, Rhode Island.

questions: question: How is the in-field address canvassing reviewed?
answer: GAO was asked to review in-field address canvassing during the End-to-End Test.

question: How does this report assess the in-field canvassing?
answer: This report determines whether key address listing activities functioned as planned during the End-to-End Test and identifies any lessons learned that could inform pending decisions for the 2020 Census.

question: How did GAO use external data to help further their assessment?
answer: To address these objectives, GAO reviewed key documents including test plans and training manuals, as well as workload, productivity and hiring data.
child_questions: question: To what extent did GAO observe these documents in action (ie. the testing of the procedures)?
answer: At the three test sites, GAO observed listers conducting address canvassing.