2020 Census: Bureau Needs to Better Leverage Information to Achieve Goals of Reengineered Address Canvassing
Published: 2017-07-20T00:00:00

Fast Facts:
For the 2020 Census, the Census Bureau originally planned to phase in technology and third-party data sources to build its address list—reducing the need to go door-to-door. The Bureau estimated it could update 75% of its address list this way, reducing costs by $900 million.
However, the Bureau has not evaluated the accuracy of its approach, which involves comparing aerial images to data in the master address list. It also suspended the operation's second phase—resolving mismatches—without detailing the cost, value, or accuracy of the new approach.
We made 3 recommendations to help the Bureau tackle these issues prior to the 2020 Census.

Highlights:
section_title: Why GAO Did This Study
paragraphs: With a lifecycle cost of $12.3 billion (in 2020 dollars), the 2010 Census was the most expensive in U.S. history. Reengineering how the Bureau builds its address list is one of the ways the Bureau intends to reduce the per–housing unit cost of the 2020 count.
GAO was asked to evaluate the Bureau's reengineered approach for 2020 address canvassing. This report (1) describes the Bureau's design for 2020 address canvassing, (2) evaluates the extent to which the Bureau assessed the cost and quality implications of its reengineered address canvassing approach, and (3) assesses the status of the Bureau's efforts to reduce the in-field address canvassing workload. GAO reviewed relevant design and testing documentation and interviewed cognizant Bureau officials. GAO also reviewed Bureau address canvassing production and payroll data in December 2016 and March 2017.
section_title: What GAO Found
paragraphs: To achieve cost savings, the Census Bureau (Bureau) decided in September 2014 to use a reengineered approach for building its address list for the 2020 Census without having to go door-to-door (or “in-field”) across the country, as it has in prior decennial censuses. Rather, some areas (known as “blocks”) might only need a review of their address and map information using computer imagery and third-party data sources – what the Bureau calls “in-office” procedures. The Bureau planned to use a two-phase, in-office address canvassing operation to update 75 percent of all housing units in its master address list, and initially estimated this approach could reduce 2020 Census costs by $900 million.
The Bureau has not completed evaluations of results from tests and activities it carried out in 2016 to assess the effectiveness of in-office address canvassing. Specifically, the Bureau has not completed its evaluations designed to evaluate the accuracy of in-office address canvassing. Similarly, citing budget constraints, the Bureau also cancelled 2017 fieldwork that could have provided additional data to evaluate the reengineered approach. Further, the Bureau has changed the design of the in-office address canvasing operation with limited information on the cost and quality implications of the changes. Citing budget uncertainty, in March 2017 the Bureau suspended its second phase of in-office address canvassing, which was designed to resolve address coverage issues identified in the first phase. Bureau officials indicated that this decision would increase the in-field address canvassing workload from 25 percent to at least 30 percent, but they could not provide details of the cost or quality tradeoffs of this decision. Using its remaining evaluations to determine cost and quality implications would better position the Bureau to justify future decisions for its reengineered approach.
GAO has previously reported on the need for the Bureau to rethink its approach to testing and evaluating the census. The Bureau agreed, and its early plans called for the use of incremental, small-scale testing throughout the decade. As the Bureau begins planning for future census operations, small-scale testing by the Bureau may be less susceptible to budgetary constraints and allow it to better demonstrate the effectiveness of its operational innovations.
The Bureau manages the day-to-day production of in-office address canvassing using measures that do not fully reflect the effectiveness of the approach in reducing fieldwork. For example, the Bureau regularly tracks the number of blocks for which it has conducted an initial round of review. However, the workload the Bureau identifies as completed includes blocks that have been put on hold and that await availability of better information before being reviewed again. Other assumptions about how quickly the workflow proceeds have not been met. In testing the Bureau's productivity assumptions using actual production data through March 2017, GAO found that even small differences in measurement can have significant impacts on estimates of the remaining workload and the time to complete it. Without measuring how much in-office work is truly completed and how much fieldwork has been reduced, the Bureau risks underestimating the time and resources needed to complete the in-office operation, and may require more in-field address canvassing—at greater costs—than the Bureau currently projects.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Bureau (1) use its remaining evaluations before 2020 to determine cost and quality implications and to justify future decisions for its reengineered approach; (2) plan and execute smaller, more flexible tests needed to support key address canvassing design decisions in future census operations; and (3) use productivity measures that track the progress of the in-office address canvassing and its effectiveness in reducing fieldwork. The Department of Commerce had no disagreements with GAO's findings and recommendations. In addition, the Secretary noted that the Department was conducting its own review.

section_title: Letter
paragraphs: With a life cycle cost of $12.3 billion, the 2010 Census was the most  expensive population count in U.S. history, costing over 30 percent more  than the $9.4 billion 2000 Census (in constant 2020 dollars). For the 2020  Census, the U.S. Census Bureau (Bureau) aims to reduce its per- household cost. Controlling costs is a priority because the cost of  enumerating each housing unit has increased from $16 per housing unit  in 1970 to $92 per housing unit in 2010, and the estimated number of  housing units has increased markedly from 2010 to 2020. To achieve  savings, the Bureau is extensively changing how it plans to conduct the  2020 Census.
Specifically, the Bureau is changing how it builds its address list, seeking  to improve self-response by encouraging the use of the Internet and  telephone, using administrative records to reduce field work, and  reengineering field operations using technology to reduce manual effort  and improve productivity, among other things. In October 2015, the  Bureau estimated that with these innovations it could conduct the 2020  Census for a life-cycle cost of $12.5 billion. This is in contrast to its  estimate of $17.8 billion to repeat the design and methods of the 2010  Census (both in constant 2020 dollars).
In building the address list for 2020, the Bureau aims to use a mix of  street and satellite imagery, as well as address and other administrative  records from state, local, and tribal governments and commercial  partners, to canvass most addresses virtually. By doing so, the Bureau  plans to reduce the need for more costly door-to-door or “in-field” address  canvassing.
You asked us to evaluate the Bureau’s reengineered approach to 2020  address canvassing. This report (1) describes the Bureau’s design for its  reengineered 2020 Census address canvassing, (2) evaluates the extent  to which the Bureau assessed the cost and quality implications of its  reengineered address canvassing approach, and (3) assesses the status  of the Bureau’s efforts to reduce the in-field address canvassing  workload.
For all three objectives, we reviewed documentation related to 2020  Address Canvassing, such as operational planning documents and test  plans, and interviewed cognizant Bureau officials. For the first objective  we reviewed the Bureau’s December 2015 Address Canvassing  Operational Plan to highlight design features and innovations affecting  cost and quality. For the second objective, we assessed how the Bureau  was implementing its stated plans related to testing and evaluating the  reengineered address canvasing approach, in particular how the Bureau  was assessing cost and quality implications. For the third objective, we  analyzed production and payroll data in December 2016 and March 2017  to report on the productivity of the Bureau’s reengineered approach to  address canvassing in light of the work required to complete the operation  and reduce fieldwork. We verified the reliability of these data sources by  interviewing officials responsible for the data, reviewing data dictionaries,  and carrying out various integrity checks on the data received. We found  the data to be sufficiently reliable for our purposes. More information on  our scope and methodology can be found in appendix I.
We conducted this performance audit from August 2016 to July 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Bureau’s address canvassing operation updates its address list and  maps, which are the foundation of the decennial census. An accurate  address list both identifies all households that are to receive a notice by  mail requesting participation in the census (by internet, phone, or mailed- in questionnaire) and serves as the control mechanism for following up  with households that fail to respond to the initial request. Precise maps  are critical for counting the population in the proper locations—the basis  of congressional reapportionment and redistricting. If the Bureau’s  address list and maps are inaccurate, people can be missed, counted  more than once, or counted at the wrong location.
However, our prior work has shown that developing an accurate address  list has been both labor-intensive and costly—in part because people can  reside in unconventional dwellings, such as converted garages,  basements, and other forms of hidden housing. For example, as shown in  figure 1, what appears to be a single-family house could contain an  apartment, as suggested by its two doorbells.
In order to account for the complexity of the nation’s housing stock, the  Bureau uses different procedures for canvassing and enumerating  different types of living arrangements. These different types of living  arrangements include housing units (e.g., multi-unit structures and single- family units) and group quarters (such as nursing facilities and college  residence halls). The Bureau also uses different procedures for  canvassing and enumerating different areas of the country, depending on  the predominant way that people receive their mail in those areas. The  Bureau generally uses methods that rely on individuals self-responding to  either notices or questionnaires in areas where mail is generally delivered  to the same individual locations where people live, and it calls these  areas “Self-Response” areas.
In other areas where mail is typically delivered at post office boxes or  along rural routes, the Bureau relies on door-to-door visits to canvass and  deliver notices and questionnaires or to conduct interviews. The Bureau  refers to these areas as Update/Enumerate areas, where the Bureau  canvasses the addresses in the areas while enumerating the residents.  The Bureau also uses similar procedures in sparsely settled and isolated  parts of Alaska, which often require special travel or other arrangements.
During address canvassing, the Bureau verifies that its master address  list and maps are accurate so the tabulation for all housing units and  group quarters is correct. For the 2010 Census, the address canvassing  operation mobilized almost 150,000 field workers to canvass almost every  street in the United States and Puerto Rico to update the Bureau’s  address list and map data—at a cost of nearly $450 million. The cost of  in-field address canvassing in 2010, along with the emerging availability  of imagery data, led the Bureau to explore an approach for 2020 Address  Canvassing that would not involve walking every street in the country.
section_title: Reengineered Address Canvassing Was Designed to Substantially Reduce In-Field Canvassing
paragraphs: To achieve cost savings, the Bureau decided in September 2014 that  while it would still verify all the addresses in the country, not all of this  canvassing would take place by going door-to-door (or “in-field”), as it had  in prior decennials. Rather, some areas might only need a review of their  address and map information using computers and what the Bureau  refers to as “in-office” procedures. The Bureau cited multiple factors in  support of this decision:
Advancements in technology allow the Bureau to continually update  address and spatial data throughout the decade using its partnerships  with state, local, and tribal governments.
The availability of up-to-date, high quality, high-resolution aerial and  street-level imagery now provides a viable and effective tool to help  reduce field work for many parts of the United States.
More efficient and effective uses of administrative records, such as  state, local, and tribal address lists as well as address information  from other federal agencies like the U.S. Postal Service, can provide a  substitute for field work, especially in areas that have been relatively  stable residentially.
The Bureau estimated in October 2015 that these changes to its address  canvassing operation could save $900 million for the 2020 Census.  These savings are part of an overall redesign that the Bureau estimated  could reduce costs by $5.2 billion compared to conducting another  census using methods from the 2010 Census. The Bureau’s 2015  estimates were the latest available when we concluded our audit work.
According to the Bureau, the potential for savings from the reengineered  approach comes from the fact that a vast majority of housing units are  eligible for a substantial reduction in in-field canvassing in 2020. Of the  over 144 million projected housing units for 2020, nearly 132 million (91  percent) are in Self-Response areas. Because housing units in these  areas are physically located where their mail is delivered, the Bureau can  eliminate the need for fieldwork if it is able to canvass addresses in these  areas in-office. The remaining housing units lie in Update/Enumerate  and related rural areas. Because people in these areas would not receive  a mailed request to participate in the census at their homes, the Bureau  will still require fieldwork to complete interviews for the census.
The potential for savings from reduced housing unit workload for in-field  canvassing is moderated by the fact that Update/Enumerate areas  account for a disproportionate share of land area throughout the country.
Figure 2 illustrates what the 2010 distribution (the most recent publicly  available) of types of enumeration areas across the country would be  using 2020 categories. As shown, Bureau employees would still have to  traverse large sections of the country door-to-door, even if in-office  address canvassing removed all of the Self-Response areas from  fieldwork.
The Bureau began its in-office address canvassing operation in  September 2015. In December 2016, the Bureau estimated that its new  approach would reduce the in-field canvassing workload by 75 percent— that is, about 25 percent of housing units would still require in-field  canvassing. Table 1 shows several elements that go into the overall  reengineered approach. In addition to the two forms of address  canvassing (in-office and in-field), the Bureau included multiple ways for  state, local, and tribal governments to contribute to the address list. The  Bureau also planned for an annual address list coverage measurement  study–in part to validate the ongoing work of in-office address  canvassing.
As described in the Bureau’s December 2015 Address Canvassing  Operational Plan, in-office canvassing had two phases to identify the  geographic areas that will not need to be canvassed in-field:
During the first phase, known as “Interactive Review,” Bureau  employees are to use current aerial imagery to determine if areas  have housing changes, such as new residential developments or  repurposed structures, or if the areas match what is in the Bureau’s  master address file. The Bureau would assess the extent to which the  number of housing units in the master address file is consistent with  the number of units visible in the current imagery. If the housing  shown in the imagery matches what is listed in the master address  file, those areas would be resolved and set aside as not needing in- field address canvassing.
During the second phase, known as “Active Block Resolution,”  employees would try to resolve coverage concerns identified during  the first phase and verify every housing unit by virtually canvassing  the entire area. As part of this virtual canvass, the Bureau would  compare what is found in imagery to the master address file data and  external data sources to attempt to resolve any discrepancies. If  Bureau employees could not reconcile housing unit count and  geographic feature (such as street location or highway median) with  what is in the address list, they would refer these blocks to in-field  address canvassing.
Figure 3 shows the intended relationships between these phases of  address canvassing, and the master address file. In both phases of in- office address canvassing, areas could be temporarily set aside if there is  insufficient imagery or other data to fully review the area. The Bureau  plans to further review these areas again as the necessary imagery and  data become available. Any geographic areas not resolved after  completing the two phases of in-office address canvassing will be  canvassed in-field.
section_title: The Bureau Lacks Information on Implications of Reengineered Design on Cost and Quality of Address Canvassing
subsections: section_title: The Bureau Has Not Completed Evaluations of 2016 Tests as Scheduled
paragraphs: The Bureau has not completed evaluations of results from tests and  activities it carried out in 2016 to assess the effectiveness of in-office  address canvassing. Information from these tests would help the Bureau:  assess the implications on cost and quality of the reengineered design,  and make more informed decisions about Census operations.  Specifically, the Bureau has not completed evaluations for its 2016  Address Canvassing Test or its 2016 Master Address File Coverage  Study.
The Bureau’s 2016 Address Canvassing Test measured the cost and  quality of address canvassing in two sites from October to December  2016—in Buncombe County, North Carolina, and St. Louis, Missouri.  Meanwhile, the 2016 Master Address File Coverage Study ran from  October 2015 to April 2016 gathering information to measure the quality  of in-office canvassing.
The Bureau’s November 2014 Address Canvassing Recommendation  Report, which summarized the proposed components of the reengineered  address canvassing approach and recommended implementing it, noted  that future research would be needed to refine the process and determine   impacts on cost and quality. Yet, citing budget uncertainty, in January  2017 the Bureau announced it was cancelling 2017 Master Address File  Coverage Study activities, removing another data point for evaluating in- office address canvassing. Bureau officials described this activity as a  lower priority than other activities that are more necessary to conduct the  census. In particular, Bureau officials pointed to the need to prioritize  preparations for the 2018 End-to-End Test, where all major decennial  operations are to undergo a dress rehearsal prior to 2020. Appendix II  depicts the Bureau’s reengineered address canvassing key activities,  from its 2014 decision to proceed through its recently delayed and  canceled evaluations.
According to its December 2015 Address Canvassing Operational Plan,  the Bureau was going to decide whether to refine several design  assumptions based, in part, on the results of these two evaluation efforts– including whether the Bureau would be able to meet its in-field  canvassing goal of 25 percent without sacrificing address list quality.  According to the address canvassing operational plan, these decisions  related to anticipated workload and the quality of the data collected were  to have occurred by January 2017. Because the 2016 Master Address  File Coverage Study and 2016 Address Canvassing Test evaluations  have not been completed, they have not supplied information to the  Bureau to help make this decision. access to large multi-unit structures;  reconciliation of conflicts between  household responses and visible  clues about unreported or hidden  housing units; and  allocation of assignments in an  efficient manner (i.e., clustering  housing units geographically).
Additionally, in October 2016 we visited the test sites of the Bureau’s  Address Canvassing Test. Given the Bureau’s timeline for making  planning and additional testing decisions, we reported to the Bureau in  near-real-time any observed implementation challenges experienced  during the test with respect to in-field canvassing (see sidebar). These  observations underscore the complexity of address canvassing and the  need to test design innovations. The Bureau was aware of these  challenges; however, structured evaluations of cost and quality are still  not complete. Bureau officials have stated that these planned evaluations  will be delayed until August 2017.
The Bureau agreed with these  observations and indicated that it would  examine them as part of its evaluation of  the 2016 Test.
Delaying evaluations and canceling the 2017 Master Address File  Coverage Study and the 2017 Census Test has meant missed  opportunities to assess and potentially improve the effectiveness of  reengineered address canvassing. It has also left the Bureau without  clear justification for making decisions related to its reengineered address  canvassing approach—such as suspending the second phase of in-office  address canvassing. Should the delayed evaluations of the 2016 Address  Canvassing Test and 2016 Master Address File Coverage Study show  that in-office address canvassing yields accurate and cost effective  results, it will be important to prioritize the Bureau’s investments in this  approach to achieve planned savings.
In addition to the cancellation of the 2017 Master Address File Coverage  Study, the cancellation of the field work components of the 2017 Census  Test, which the Bureau similarly attributed to budgetary constraints it  faced, represents a lost opportunity to collect data that could have been  used to evaluate the quality of in-office address canvassing. Specifically,  this fieldwork would have provided the Bureau with more in-field collected  address information that could have been used as a check on in-office  collected data collected in the same areas. Table 2 provides the status of  key address canvassing tests and their evaluations. After we raised the  issue of missed testing opportunities with the Bureau, Bureau officials  indicated in March 2017 that they would include within its 2018 End-to- End Test a sample of in-field canvassing at one of the three test sites  which would let it assess the quality of the in-office canvassing in those  areas.
section_title: Suspension of the Second-Phase of In-Office Canvassing Could Reduce Estimated Cost Savings
paragraphs: Citing budget uncertainty, the Bureau altered its design for reengineered  address canvassing in March 2017 by suspending the second phase of  in-office address canvassing. Without the second phase of in-office  address canvassing, blocks that are not resolved by phase one will have  a greater chance of requiring in-field canvassing. Bureau officials told us  that they anticipate that canceling the second phase of in-office address  canvassing altogether would increase their estimated in-field canvassing  workload by 5 percentage points, from 25 percent to 30 percent of  housing units—increasing costs. We could not independently weigh the  costs of continuing this phase, because current Bureau estimates do not  provide enough details on the specific costs of each phase. However,  based on cost assumptions underlying its October 2015 life-cycle cost  estimate we found that the potential addition of 5 percentage points to the  field workload alone could reduce the Bureau’s cost savings by $26.6  million.
The results of the second phase of in-office address canvassing  suggested that the operation potentially could reduce fieldwork by a  significant amount. We found that in phase two, prior to its suspension  (September 2016 to March 2017), the Bureau reviewed 38,170 blocks  that phase one had indicated needed additional review. We found that the  Bureau had resolved 25,777 of these geographic areas (68 percent),  meaning those geographic areas would not need to be worked in-field,  saving the costs associated with that additional workload.
The Bureau did not develop cost and quality information on address  canvassing projects, and detailed information on cost tradeoffs was not  readily available when we requested it. What information the Bureau did  have did not break out the estimated cost of the different phases of in- office address canvassing through 2020. However, the totaled estimated  cost for both phases one and two was approximately $22 million. Thus,  this suspension might save a portion of the $22 million, but it will  potentially increase the cost of the address canvassing operation  downstream.
For example, the Bureau cannot say what $1 saved on suspending the  second phase of in-office address canvassing today costs in increased in- field workload down the road, or what the marginal cost of canvassing X  percent more in-field is. Additionally, the lack of details in the cost  estimate means it is unclear how the potential 5 percentage point  increase in workload would impact the potential downstream savings that  were factored into the Bureau’s estimated $900 million savings. The  Bureau is continuing to weigh whether and when to resume the second  phase of in-office address canvassing. Without cost and quality  information, the Bureau is limited in its ability to determine what tradeoffs  to make as part of this and related future operational decisions.
By canceling the 2017 Master Address File Coverage Study, the Bureau  has limited itself to two evaluations— both of which are delayed— to  assess the workload, quality and cost of in-office address canvassing  (i.e., the 2016 Address Canvassing Test and 2016 Master Address File  Coverage Study). As a result, the Bureau has had limited information to  demonstrate the cost and quality of in-office address canvassing along  the way and has made major design decisions, such as the suspension of  the second phase of in-office address canvassing, without the benefit of  information from testing.
We have previously reported on the need for the Bureau to rethink its  approach to, among other things, testing and evaluating the census. The  Bureau agreed, and its early plans for the 2020 Census called for the use  of incremental, small-scale testing throughout the decade. Such an  approach would have been less susceptible to budgetary constraints and  better allowed the Bureau to demonstrate the effectiveness and accuracy  of the full in-office address canvassing operation to Congress and  stakeholders. Scarce time remains in this decennial cycle to alter testing  strategy. However, as the Bureau begins planning for future census operations in the coming years, it will be important to commit to an  approach of small-scale, more frequent testing that provides real-time,  actionable data to inform decisions.
section_title: The Bureau’s Measures of In-Office Address Canvassing Do Not Reflect the Effect on Reducing Fieldwork
subsections: section_title: Bureau’s Measures Do Not Reflect Total Completion of Workload
paragraphs: The Bureau includes work that has been put on hold in its measure  of completed workload. In measuring the day-to-day process of the in- office address canvassing operation, the Bureau regularly tracks the  number of blocks for which it has conducted an initial round of review.  The Bureau uses this high-level data to describe what it calls a first-pass  of canvassing of all 11 million blocks of the nation. However, the workload  the Bureau identifies as completed as part of this first pass includes  blocks that have been put on hold. A block may be put on hold because  the Bureau did not have sufficient information to identify all the housing  units within that block at the time of the review. For example, this may  occur if imagery was blocked by cloud cover. In these cases, the Bureau  will need to review the blocks again later to try to resolve them without  fieldwork. As of March 2017, the Bureau reported completing a first pass  of 89 percent of all blocks. However, after accounting for blocks initially  placed on hold, the Bureau had only completed 79 percent of the  workload, and 64 percent of all blocks had been marked as not needing  fieldwork.
The Bureau has planned additional rounds of review in order to complete  the blocks it has placed on hold. However, the Bureau is counting these  blocks as completed, according to the Bureau’s external progress  reporting and other documents used to determine staff needs. By  calculating completed workload as including the blocks that will require  further review to resolve in-office, the Bureau does not have a true  indication of how much progress it is making on the operation. A measure  of completed workload that focuses on the amount of in-office workload  that has been completed and resolved without need for fieldwork would  help the Bureau make more informed future staffing decisions.  Specifically, the Bureau may ultimately have greater fieldwork demands  that it will need to staff than its measure is currently showing.
The Bureau has additional work to update previously-reviewed  blocks. The Bureau also plans additional rounds of in-office review to try  to resolve blocks initially flagged for in-field canvassing, which it  anticipates will help it to minimize its fieldwork, which it initially estimated  would be only 25 percent of housing units. Yet the Bureau’s estimates do  not include projections for how much of the workload will undergo these  additional rounds of review. According to the Bureau, these additional  rounds of review are essential to reducing fieldwork and ensuring that  previously-reviewed blocks are current with housing changes during the  decade. These types of blocks by definition require additional review to  be resolved in-office, so by not including estimates for this workload, the  Bureau may be underestimating the time and effort needed to use in- office canvassing to maximize fieldwork reductions and keep the master  address file up to date.
The Bureau’s average review time measure does not capture the  total time it is taking to complete the workload. The Bureau calculates  the average time it takes to review blocks so it can estimate daily and  monthly production, as well as how it is performing against annual  production goals. The Bureau measures the average time each employee  spends reviewing each block onscreen at a given point (1.02 minutes per  block). The Bureau uses this measure to track the productivity of  individual employees and the day-to-day efficiency of the overall  operation. However, not every one of these reviews results in a block  being completed in-office or removed from the later in-field workload, and  some of these reviews are duplicate reviews by others as part of the  quality control process. So the actual average review time required to  remove blocks in this process from the operation’s workload— to  complete them— is longer, with significant implications for estimates of  how long it will take to complete the work. When we adjusted the  Bureau’s calculations for the additional reviews as of March 2017, we  found that the average time it takes to complete a block through in-office  address canvassing— including resolving blocks put on-hold and  conducting any additional quality control reviews— was about 20 percent  longer (1.22 minutes per block). As discussed below, this difference in  average review times may not seem like much, but multiplied over the  millions of block reviews that the Bureau has remaining, the implications  of this difference become significant.
Moreover, using a single measure of average review time obscures  significant variation across in-office address canvassing outcomes that  can affect estimates of future resources needed. It is likely that the  remaining workload disproportionately contains more blocks that will need  to be referred to fieldwork, because the Bureau prioritized its earlier work  based in part on the availability of quality imagery and data, which  facilitates resolving blocks in-office. Among blocks completed in-office, we  found that the average time per review for blocks referred to the field (2.8  minutes per review) was well over twice as long as the average time  spent on blocks removed from fieldwork (1.0 minutes per block). If the  Bureau has disproportionately more blocks in the former category yet to  review, the remainder of the operation will likely take longer than the  Bureau’s base measure would suggest, possibly requiring more  resources than planned.
section_title: Bureau Workflow Assumptions Are Not Being Met
paragraphs: In December 2016, the Bureau estimated that employees would spend  6.5 hours per day on in-office canvassing. The Bureau’s payroll data offer  the ability to include the effect that non-canvassing activities, such as  training, systems downtime, and supervisory tasks, have on the  productive workday. When taking these activities into account for  production through March 2017, for example, we found that the average  productive workday was less than 6.2 hours per day. At this rate, the  average employee would conduct 19 fewer block reviews per day than  what the Bureau had assumed.
The Bureau had also anticipated in December 2016 that 5 percent of  blocks would undergo quality assurance procedures, whereby blocks can  receive up to two additional reviews each. In our March 2017 review of  Bureau data, however, we found that nearly three times, or 14 percent, of  the reviews that the Bureau had conducted were part of the quality  assurance procedures, meaning that there was more canvassing activity  required to complete individual blocks. If carried forward, the result of  these discrepancies is that the Bureau would have more remaining in- office canvassing, and would get less done per day, than its estimates  show.
section_title: Revised Productivity Estimates Would Help Bureau Understand Resource Needs for Address Canvassing
paragraphs: At first glance, the above issues can each seem like small differences in  measurement, but as table 3 shows, even small differences can add up to  large cumulative effects on calculations of workload and estimates of the  time to complete it. In a scenario that updates the Bureau’s assumptions  and accounts for the total number of reviews needed to complete a first  pass of the operation, the Bureau would have nearly 3 million more  reviews and— at estimated production rates and staffing levels— nearly  220 days of production.
Moreover, as discussed, the Bureau has not indicated how many blocks it  estimates will undergo additional rounds of review to capture housing  changes during the remainder of the operation. The Bureau has indicated  that these additional rounds of review are key to resolving as much  workload as possible in-office while ensuring that the master address file  stays current with housing changes. Going forward, the Bureau will also  need to account for the fact that, as discussed above, remaining blocks  may be more difficult and time-consuming to complete given that the  Bureau has previously prioritized the assignment of blocks with more  readily available imagery.
Best practices for project planning state that organizations should  establish estimates that contain information consistent with project  requirements to determine effort, cost, and schedule. We have also  noted that when scheduling major initiatives, agencies should include all  activities necessary to accomplish a program’s objectives. Doing so is  critical to verifying that an agency is properly allocating resources and  accounting for scheduling risks.
With address canvassing, the risk is not that the Bureau will not be able to  produce a master address file in time for the 2020 Census, but rather that  the Bureau may have to devote more staff and fieldwork— at greater  cost— to complete the operation within needed timeframes. The Bureau  had initially planned on resolving 75 percent of housing units in-office, but  as of March 2017 the blocks that the Bureau had successfully removed  from fieldwork account for 51 percent of all housing units. Furthermore, it  is uncertain how much more workload the Bureau will be able to resolve  in-office during the remainder of the operation, particularly since officials  have indicated that the Bureau is not likely to resume the second-phase  of in-office address canvassing. If the Bureau does not adequately gauge  the amount of remaining in-office address canvassing workload within its  time frames, it risks having to complete more address canvassing in-field.  Given the uncertainty about the scope of the remainder of the operation,  then, it is important for the Bureau to incorporate measures of productivity  that reflect the progress and effectiveness of the operation to date.
section_title: Conclusions
paragraphs: Given the escalating costs of recent censuses, it is critical for the Bureau  to apply innovative approaches that cost-effectively steward resources  and deliver quality results throughout the decennial cycle. With its design  for reengineered address canvassing, the Bureau made important strides  in the types of data and stakeholders it intended to leverage in order to  produce an accurate, cost-effective address list.
However, recent management and testing decisions raise questions  about whether the Bureau can fully deliver on the Bureau’s initially  estimated gains of the reengineered approach. The Bureau suspended  the second phase of in-office address canvassing without detailed data  on the cost, value, or accuracy of the operation itself, citing budget  uncertainty. Given that the Bureau will need to make consequential  decisions about the scope and nature of its fieldwork for 2020, it will be  important for the Bureau to make use of information on the effectiveness  of its reengineered operation, in terms of both the cost and quality of  constructing the address list, so that it can justify these decisions and  achieve desired outcomes.
Our work has also found that the Bureau needs more and earlier testing  before implementing major design changes, yet the fact that the Bureau  has completed much of its work on in-office address canvassing without  finalizing the evaluations for any related testing leaves questions  unanswered about the cost and quality of the operation. A testing  program that as laid out early in the decennial cycle, includes smaller and  more flexible test opportunities, which could be less susceptible  individually to budget uncertainty, can better position the Bureau to  support key design decisions and modifications.
In order to achieve savings, the Bureau also needs to make more  effective use of productivity information from in-office address canvassing  and use estimates that reflect the actual time it takes to complete work. A  number of gaps in the Bureau’s production measures, while individually  small, appear to have a large cumulative effect on time and workload  calculations that could affect resource allocation decisions. With the start  of 2020 in-field address canvassing scheduled to begin in a little over 2  years, the Bureau has limited time to finalize its decisions about the  scope and nature of address canvassing. Using measures that reflect the  progress and effectiveness of the operation— in addition to the process- oriented measures that the Bureau already tracks— can help the Bureau  make more informed decisions.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce direct the Under  Secretary of the Economics and Statistics Administration and the Director  of the U.S. Census Bureau to take the following actions:  1.  Use the Bureau’s evaluations before 2020 to determine the  implications of in-office address canvassing on the cost and quality of  address canvassing, and use this information to justify decisions  related to its re-engineered address canvassing approach.  2.  Early in the next decennial cycle, plan and execute more flexible, and  perhaps smaller, address canvassing test and evaluation activity  needed to support key design decisions having significant effect on  the cost and quality of the census.  3.  Use productivity measures that track the progress in completing in- office address canvassing workload and the effectiveness of in-office  address canvassing in reducing fieldwork in order to make informed  decisions on allocating resources to current and future address  canvassing workload so that the operation is completed in a timely  and cost-effective manner.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Commerce. In its  written comments, reproduced in appendix III, the Department of  Commerce had no disagreements with GAO’s findings and  recommendations. In addition, the Secretary noted that the Department  was conducting its own review.
We are sending copies of this report to the Secretary of Commerce, the  Under Secretary of Economic Affairs, the Acting Director of the U.S.  Census Bureau, and interested congressional committees. The report  also will be available at no charge on GAO’s website at  http://www.gao.gov.
If you have any questions about this report please contact me at (202)  512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. The GAO staff that made major contributions to this report  are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: You asked us to evaluate the U.S. Census Bureau’s (Bureau)  reengineered approach to 2020 Address Canvassing. We (1) described  the Bureau’s design for its reengineered 2020 Census address  canvassing, (2) evaluated the extent to which the Bureau assessed the  cost and quality implications of its reengineered address canvassing  approach, and (3) assessed the status of the Bureau’s efforts to reduce  the in-field address canvassing workload.
For the first objective, we interviewed Bureau staff responsible for the  Bureau’s 2020 re-engineered address canvassing approach, and  reviewed relevant documentation, such as the Bureau’s December 2015  operational plan for address canvassing, to highlight design features and  innovations affecting cost and quality. We also observed the 2016  Address Canvassing Test in-field activities to verify our understanding of  how in-office canvassing interfaces with in-field canvassing. Given the  Bureau’s timeline for making planning and additional testing decisions, we  reported to the Bureau in near-real-time any observed implementation  challenges experienced during the test with respect to in-field or in-office  canvassing.
For the second objective we interviewed officials in the Bureau’s  Geography and Decennial Census Management Divisions and reviewed  relevant documentation to determine how the Bureau was managing  address canvassing testing, assessing cost and quality implications,  making decisions regarding the design of the reengineered address  canvassing approach, and managing related risks. In particular, we  followed up with officials regarding updates we received to the operational  design as well as project-level and program-level risk registers. We  assessed how the Bureau was implementing its stated plans for these  efforts. We also interviewed officials and reviewed documentation, such  as available breakdowns of cost estimates, specific to the Bureau’s  estimated cost savings from the reengineered address canvassing  approach.
For the third objective we interviewed officials in the Bureau’s Geography  Division to determine how they are measuring productivity of the in-office  address canvassing operation. We also visited and interviewed officials  managing and conducting the Bureau’s in-office address canvassing  production to better understand how Bureau employees receive and  complete workload assignments and how managers assure quality and  track time management data of the operation.
We collected and analyzed Bureau production data and payroll data for  in-office address canvassing from two as-of dates during the operation—  December 2016 and March 2017— to cover in-office address canvassing  since the start of production. We tested the data quality with both sets of  data by interviewing officials responsible for the data; reviewing manuals,  data dictionaries, and related explanatory documentation; and running  calculations to verify the record counts and check the integrity of the data.  We found the Bureau’s production data to be sufficiently reliable to  calculate estimates of time and workload, while we found the Bureau’s  payroll data to be sufficiently reliable to comment on aggregate time spent  on the in-office address canvassing operation as a whole (as opposed to  the specific phases of the operation).
Using the Bureau’s block-level production data and staffing estimate  documentation, we identified different ways to articulate the productivity of  the operation, and explored the sensitivity of estimated completion dates  and production rates to variation in key productivity assumptions and  measures. We also synthesized the Bureau’s payroll data for in-office  address canvassing to test the Bureau’s assumptions about staff time  allocation. Combined, these activities allowed us to illustrate the effects  on time and resource estimates of having accurate productivity measures  that focus on the progress and effectiveness of the in-office address  canvassing operation in reducing fieldwork. We did this evaluation in light  of best practices for scheduling major projects and outlining resource  needs.
We conducted this performance audit from August 2016 to July 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ty Mitchell, Assistant Director;  Devin Braun; Robert Gebhart; Richard Hung; Cynthia Saunders; Kayla  Robinson; Robert Robinson, and Timothy Wexler made key contributions  to this report.