2020 Census: Bureau Needs to Better Leverage Information to Achieve Goals of Reengineered Address Canvassing
Released: 2017-07-20T00:00:00

Letter

Background

The Bureau’s address canvassing operation updates its address list and maps, which are the foundation of the decennial census. An accurate address list both identifies all households that are to receive a notice by mail requesting participation in the census (by internet, phone, or mailed- in questionnaire) and serves as the control mechanism for following up with households that fail to respond to the initial request. Precise maps are critical for counting the population in the proper locations—the basis of congressional reapportionment and redistricting. If the Bureau’s address list and maps are inaccurate, people can be missed, counted more than once, or counted at the wrong location.

However, our prior work has shown that developing an accurate address list has been both labor-intensive and costly—in part because people can reside in unconventional dwellings, such as converted garages, basements, and other forms of hidden housing. For example, as shown in figure 1, what appears to be a single-family house could contain an apartment, as suggested by its two doorbells.

In order to account for the complexity of the nation’s housing stock, the Bureau uses different procedures for canvassing and enumerating different types of living arrangements. These different types of living arrangements include housing units (e.g., multi-unit structures and single- family units) and group quarters (such as nursing facilities and college residence halls). The Bureau also uses different procedures for canvassing and enumerating different areas of the country, depending on the predominant way that people receive their mail in those areas. The Bureau generally uses methods that rely on individuals self-responding to either notices or questionnaires in areas where mail is generally delivered to the same individual locations where people live, and it calls these areas “Self-Response” areas.

In other areas where mail is typically delivered at post office boxes or along rural routes, the Bureau relies on door-to-door visits to canvass and deliver notices and questionnaires or to conduct interviews. The Bureau refers to these areas as Update/Enumerate areas, where the Bureau canvasses the addresses in the areas while enumerating the residents. The Bureau also uses similar procedures in sparsely settled and isolated parts of Alaska, which often require special travel or other arrangements.

During address canvassing, the Bureau verifies that its master address list and maps are accurate so the tabulation for all housing units and group quarters is correct. For the 2010 Census, the address canvassing operation mobilized almost 150,000 field workers to canvass almost every street in the United States and Puerto Rico to update the Bureau’s address list and map data—at a cost of nearly $450 million. The cost of in-field address canvassing in 2010, along with the emerging availability of imagery data, led the Bureau to explore an approach for 2020 Address Canvassing that would not involve walking every street in the country.

Reengineered Address Canvassing Was Designed to Substantially Reduce In-Field Canvassing

To achieve cost savings, the Bureau decided in September 2014 that while it would still verify all the addresses in the country, not all of this canvassing would take place by going door-to-door (or “in-field”), as it had in prior decennials. Rather, some areas might only need a review of their address and map information using computers and what the Bureau refers to as “in-office” procedures. The Bureau cited multiple factors in support of this decision:

Advancements in technology allow the Bureau to continually update address and spatial data throughout the decade using its partnerships with state, local, and tribal governments.

The availability of up-to-date, high quality, high-resolution aerial and street-level imagery now provides a viable and effective tool to help reduce field work for many parts of the United States.

More efficient and effective uses of administrative records, such as state, local, and tribal address lists as well as address information from other federal agencies like the U.S. Postal Service, can provide a substitute for field work, especially in areas that have been relatively stable residentially.

The Bureau estimated in October 2015 that these changes to its address canvassing operation could save $900 million for the 2020 Census. These savings are part of an overall redesign that the Bureau estimated could reduce costs by $5.2 billion compared to conducting another census using methods from the 2010 Census. The Bureau’s 2015 estimates were the latest available when we concluded our audit work.

According to the Bureau, the potential for savings from the reengineered approach comes from the fact that a vast majority of housing units are eligible for a substantial reduction in in-field canvassing in 2020. Of the over 144 million projected housing units for 2020, nearly 132 million (91 percent) are in Self-Response areas. Because housing units in these areas are physically located where their mail is delivered, the Bureau can eliminate the need for fieldwork if it is able to canvass addresses in these areas in-office. The remaining housing units lie in Update/Enumerate and related rural areas. Because people in these areas would not receive a mailed request to participate in the census at their homes, the Bureau will still require fieldwork to complete interviews for the census.

The potential for savings from reduced housing unit workload for in-field canvassing is moderated by the fact that Update/Enumerate areas account for a disproportionate share of land area throughout the country.

Figure 2 illustrates what the 2010 distribution (the most recent publicly available) of types of enumeration areas across the country would be using 2020 categories. As shown, Bureau employees would still have to traverse large sections of the country door-to-door, even if in-office address canvassing removed all of the Self-Response areas from fieldwork.

The Bureau began its in-office address canvassing operation in September 2015. In December 2016, the Bureau estimated that its new approach would reduce the in-field canvassing workload by 75 percent— that is, about 25 percent of housing units would still require in-field canvassing. Table 1 shows several elements that go into the overall reengineered approach. In addition to the two forms of address canvassing (in-office and in-field), the Bureau included multiple ways for state, local, and tribal governments to contribute to the address list. The Bureau also planned for an annual address list coverage measurement study–in part to validate the ongoing work of in-office address canvassing.

As described in the Bureau’s December 2015 Address Canvassing Operational Plan, in-office canvassing had two phases to identify the geographic areas that will not need to be canvassed in-field:

During the first phase, known as “Interactive Review,” Bureau employees are to use current aerial imagery to determine if areas have housing changes, such as new residential developments or repurposed structures, or if the areas match what is in the Bureau’s master address file. The Bureau would assess the extent to which the number of housing units in the master address file is consistent with the number of units visible in the current imagery. If the housing shown in the imagery matches what is listed in the master address file, those areas would be resolved and set aside as not needing in- field address canvassing.

During the second phase, known as “Active Block Resolution,” employees would try to resolve coverage concerns identified during the first phase and verify every housing unit by virtually canvassing the entire area. As part of this virtual canvass, the Bureau would compare what is found in imagery to the master address file data and external data sources to attempt to resolve any discrepancies. If Bureau employees could not reconcile housing unit count and geographic feature (such as street location or highway median) with what is in the address list, they would refer these blocks to in-field address canvassing.

Figure 3 shows the intended relationships between these phases of address canvassing, and the master address file. In both phases of in- office address canvassing, areas could be temporarily set aside if there is insufficient imagery or other data to fully review the area. The Bureau plans to further review these areas again as the necessary imagery and data become available. Any geographic areas not resolved after completing the two phases of in-office address canvassing will be canvassed in-field.

The Bureau Lacks Information on Implications of Reengineered Design on Cost and Quality of Address Canvassing

The Bureau Has Not Completed Evaluations of 2016 Tests as Scheduled

The Bureau has not completed evaluations of results from tests and activities it carried out in 2016 to assess the effectiveness of in-office address canvassing. Information from these tests would help the Bureau: assess the implications on cost and quality of the reengineered design, and make more informed decisions about Census operations. Specifically, the Bureau has not completed evaluations for its 2016 Address Canvassing Test or its 2016 Master Address File Coverage Study.

The Bureau’s 2016 Address Canvassing Test measured the cost and quality of address canvassing in two sites from October to December 2016—in Buncombe County, North Carolina, and St. Louis, Missouri. Meanwhile, the 2016 Master Address File Coverage Study ran from October 2015 to April 2016 gathering information to measure the quality of in-office canvassing.

The Bureau’s November 2014 Address Canvassing Recommendation Report, which summarized the proposed components of the reengineered address canvassing approach and recommended implementing it, noted that future research would be needed to refine the process and determine impacts on cost and quality. Yet, citing budget uncertainty, in January 2017 the Bureau announced it was cancelling 2017 Master Address File Coverage Study activities, removing another data point for evaluating in- office address canvassing. Bureau officials described this activity as a lower priority than other activities that are more necessary to conduct the census. In particular, Bureau officials pointed to the need to prioritize preparations for the 2018 End-to-End Test, where all major decennial operations are to undergo a dress rehearsal prior to 2020. Appendix II depicts the Bureau’s reengineered address canvassing key activities, from its 2014 decision to proceed through its recently delayed and canceled evaluations.

According to its December 2015 Address Canvassing Operational Plan, the Bureau was going to decide whether to refine several design assumptions based, in part, on the results of these two evaluation efforts– including whether the Bureau would be able to meet its in-field canvassing goal of 25 percent without sacrificing address list quality. According to the address canvassing operational plan, these decisions related to anticipated workload and the quality of the data collected were to have occurred by January 2017. Because the 2016 Master Address File Coverage Study and 2016 Address Canvassing Test evaluations have not been completed, they have not supplied information to the Bureau to help make this decision. access to large multi-unit structures; reconciliation of conflicts between household responses and visible clues about unreported or hidden housing units; and allocation of assignments in an efficient manner (i.e., clustering housing units geographically).

Additionally, in October 2016 we visited the test sites of the Bureau’s Address Canvassing Test. Given the Bureau’s timeline for making planning and additional testing decisions, we reported to the Bureau in near-real-time any observed implementation challenges experienced during the test with respect to in-field canvassing (see sidebar). These observations underscore the complexity of address canvassing and the need to test design innovations. The Bureau was aware of these challenges; however, structured evaluations of cost and quality are still not complete. Bureau officials have stated that these planned evaluations will be delayed until August 2017.

The Bureau agreed with these observations and indicated that it would examine them as part of its evaluation of the 2016 Test.

Delaying evaluations and canceling the 2017 Master Address File Coverage Study and the 2017 Census Test has meant missed opportunities to assess and potentially improve the effectiveness of reengineered address canvassing. It has also left the Bureau without clear justification for making decisions related to its reengineered address canvassing approach—such as suspending the second phase of in-office address canvassing. Should the delayed evaluations of the 2016 Address Canvassing Test and 2016 Master Address File Coverage Study show that in-office address canvassing yields accurate and cost effective results, it will be important to prioritize the Bureau’s investments in this approach to achieve planned savings.

In addition to the cancellation of the 2017 Master Address File Coverage Study, the cancellation of the field work components of the 2017 Census Test, which the Bureau similarly attributed to budgetary constraints it faced, represents a lost opportunity to collect data that could have been used to evaluate the quality of in-office address canvassing. Specifically, this fieldwork would have provided the Bureau with more in-field collected address information that could have been used as a check on in-office collected data collected in the same areas. Table 2 provides the status of key address canvassing tests and their evaluations. After we raised the issue of missed testing opportunities with the Bureau, Bureau officials indicated in March 2017 that they would include within its 2018 End-to- End Test a sample of in-field canvassing at one of the three test sites which would let it assess the quality of the in-office canvassing in those areas.

Suspension of the Second-Phase of In-Office Canvassing Could Reduce Estimated Cost Savings

Citing budget uncertainty, the Bureau altered its design for reengineered address canvassing in March 2017 by suspending the second phase of in-office address canvassing. Without the second phase of in-office address canvassing, blocks that are not resolved by phase one will have a greater chance of requiring in-field canvassing. Bureau officials told us that they anticipate that canceling the second phase of in-office address canvassing altogether would increase their estimated in-field canvassing workload by 5 percentage points, from 25 percent to 30 percent of housing units—increasing costs. We could not independently weigh the costs of continuing this phase, because current Bureau estimates do not provide enough details on the specific costs of each phase. However, based on cost assumptions underlying its October 2015 life-cycle cost estimate we found that the potential addition of 5 percentage points to the field workload alone could reduce the Bureau’s cost savings by $26.6 million.

The results of the second phase of in-office address canvassing suggested that the operation potentially could reduce fieldwork by a significant amount. We found that in phase two, prior to its suspension (September 2016 to March 2017), the Bureau reviewed 38,170 blocks that phase one had indicated needed additional review. We found that the Bureau had resolved 25,777 of these geographic areas (68 percent), meaning those geographic areas would not need to be worked in-field, saving the costs associated with that additional workload.

The Bureau did not develop cost and quality information on address canvassing projects, and detailed information on cost tradeoffs was not readily available when we requested it. What information the Bureau did have did not break out the estimated cost of the different phases of in- office address canvassing through 2020. However, the totaled estimated cost for both phases one and two was approximately $22 million. Thus, this suspension might save a portion of the $22 million, but it will potentially increase the cost of the address canvassing operation downstream.

For example, the Bureau cannot say what $1 saved on suspending the second phase of in-office address canvassing today costs in increased in- field workload down the road, or what the marginal cost of canvassing X percent more in-field is. Additionally, the lack of details in the cost estimate means it is unclear how the potential 5 percentage point increase in workload would impact the potential downstream savings that were factored into the Bureau’s estimated $900 million savings. The Bureau is continuing to weigh whether and when to resume the second phase of in-office address canvassing. Without cost and quality information, the Bureau is limited in its ability to determine what tradeoffs to make as part of this and related future operational decisions.

By canceling the 2017 Master Address File Coverage Study, the Bureau has limited itself to two evaluations— both of which are delayed— to assess the workload, quality and cost of in-office address canvassing (i.e., the 2016 Address Canvassing Test and 2016 Master Address File Coverage Study). As a result, the Bureau has had limited information to demonstrate the cost and quality of in-office address canvassing along the way and has made major design decisions, such as the suspension of the second phase of in-office address canvassing, without the benefit of information from testing.

We have previously reported on the need for the Bureau to rethink its approach to, among other things, testing and evaluating the census. The Bureau agreed, and its early plans for the 2020 Census called for the use of incremental, small-scale testing throughout the decade. Such an approach would have been less susceptible to budgetary constraints and better allowed the Bureau to demonstrate the effectiveness and accuracy of the full in-office address canvassing operation to Congress and stakeholders. Scarce time remains in this decennial cycle to alter testing strategy. However, as the Bureau begins planning for future census operations in the coming years, it will be important to commit to an approach of small-scale, more frequent testing that provides real-time, actionable data to inform decisions.

The Bureau’s Measures of In-Office Address Canvassing Do Not Reflect the Effect on Reducing Fieldwork

Bureau’s Measures Do Not Reflect Total Completion of Workload

The Bureau includes work that has been put on hold in its measure of completed workload. In measuring the day-to-day process of the in- office address canvassing operation, the Bureau regularly tracks the number of blocks for which it has conducted an initial round of review. The Bureau uses this high-level data to describe what it calls a first-pass of canvassing of all 11 million blocks of the nation. However, the workload the Bureau identifies as completed as part of this first pass includes blocks that have been put on hold. A block may be put on hold because the Bureau did not have sufficient information to identify all the housing units within that block at the time of the review. For example, this may occur if imagery was blocked by cloud cover. In these cases, the Bureau will need to review the blocks again later to try to resolve them without fieldwork. As of March 2017, the Bureau reported completing a first pass of 89 percent of all blocks. However, after accounting for blocks initially placed on hold, the Bureau had only completed 79 percent of the workload, and 64 percent of all blocks had been marked as not needing fieldwork.

The Bureau has planned additional rounds of review in order to complete the blocks it has placed on hold. However, the Bureau is counting these blocks as completed, according to the Bureau’s external progress reporting and other documents used to determine staff needs. By calculating completed workload as including the blocks that will require further review to resolve in-office, the Bureau does not have a true indication of how much progress it is making on the operation. A measure of completed workload that focuses on the amount of in-office workload that has been completed and resolved without need for fieldwork would help the Bureau make more informed future staffing decisions. Specifically, the Bureau may ultimately have greater fieldwork demands that it will need to staff than its measure is currently showing.

The Bureau has additional work to update previously-reviewed blocks. The Bureau also plans additional rounds of in-office review to try to resolve blocks initially flagged for in-field canvassing, which it anticipates will help it to minimize its fieldwork, which it initially estimated would be only 25 percent of housing units. Yet the Bureau’s estimates do not include projections for how much of the workload will undergo these additional rounds of review. According to the Bureau, these additional rounds of review are essential to reducing fieldwork and ensuring that previously-reviewed blocks are current with housing changes during the decade. These types of blocks by definition require additional review to be resolved in-office, so by not including estimates for this workload, the Bureau may be underestimating the time and effort needed to use in- office canvassing to maximize fieldwork reductions and keep the master address file up to date.

The Bureau’s average review time measure does not capture the total time it is taking to complete the workload. The Bureau calculates the average time it takes to review blocks so it can estimate daily and monthly production, as well as how it is performing against annual production goals. The Bureau measures the average time each employee spends reviewing each block onscreen at a given point (1.02 minutes per block). The Bureau uses this measure to track the productivity of individual employees and the day-to-day efficiency of the overall operation. However, not every one of these reviews results in a block being completed in-office or removed from the later in-field workload, and some of these reviews are duplicate reviews by others as part of the quality control process. So the actual average review time required to remove blocks in this process from the operation’s workload— to complete them— is longer, with significant implications for estimates of how long it will take to complete the work. When we adjusted the Bureau’s calculations for the additional reviews as of March 2017, we found that the average time it takes to complete a block through in-office address canvassing— including resolving blocks put on-hold and conducting any additional quality control reviews— was about 20 percent longer (1.22 minutes per block). As discussed below, this difference in average review times may not seem like much, but multiplied over the millions of block reviews that the Bureau has remaining, the implications of this difference become significant.

Moreover, using a single measure of average review time obscures significant variation across in-office address canvassing outcomes that can affect estimates of future resources needed. It is likely that the remaining workload disproportionately contains more blocks that will need to be referred to fieldwork, because the Bureau prioritized its earlier work based in part on the availability of quality imagery and data, which facilitates resolving blocks in-office. Among blocks completed in-office, we found that the average time per review for blocks referred to the field (2.8 minutes per review) was well over twice as long as the average time spent on blocks removed from fieldwork (1.0 minutes per block). If the Bureau has disproportionately more blocks in the former category yet to review, the remainder of the operation will likely take longer than the Bureau’s base measure would suggest, possibly requiring more resources than planned.

Bureau Workflow Assumptions Are Not Being Met

In December 2016, the Bureau estimated that employees would spend 6.5 hours per day on in-office canvassing. The Bureau’s payroll data offer the ability to include the effect that non-canvassing activities, such as training, systems downtime, and supervisory tasks, have on the productive workday. When taking these activities into account for production through March 2017, for example, we found that the average productive workday was less than 6.2 hours per day. At this rate, the average employee would conduct 19 fewer block reviews per day than what the Bureau had assumed.

The Bureau had also anticipated in December 2016 that 5 percent of blocks would undergo quality assurance procedures, whereby blocks can receive up to two additional reviews each. In our March 2017 review of Bureau data, however, we found that nearly three times, or 14 percent, of the reviews that the Bureau had conducted were part of the quality assurance procedures, meaning that there was more canvassing activity required to complete individual blocks. If carried forward, the result of these discrepancies is that the Bureau would have more remaining in- office canvassing, and would get less done per day, than its estimates show.

Revised Productivity Estimates Would Help Bureau Understand Resource Needs for Address Canvassing

At first glance, the above issues can each seem like small differences in measurement, but as table 3 shows, even small differences can add up to large cumulative effects on calculations of workload and estimates of the time to complete it. In a scenario that updates the Bureau’s assumptions and accounts for the total number of reviews needed to complete a first pass of the operation, the Bureau would have nearly 3 million more reviews and— at estimated production rates and staffing levels— nearly 220 days of production.

Moreover, as discussed, the Bureau has not indicated how many blocks it estimates will undergo additional rounds of review to capture housing changes during the remainder of the operation. The Bureau has indicated that these additional rounds of review are key to resolving as much workload as possible in-office while ensuring that the master address file stays current with housing changes. Going forward, the Bureau will also need to account for the fact that, as discussed above, remaining blocks may be more difficult and time-consuming to complete given that the Bureau has previously prioritized the assignment of blocks with more readily available imagery.

Best practices for project planning state that organizations should establish estimates that contain information consistent with project requirements to determine effort, cost, and schedule. We have also noted that when scheduling major initiatives, agencies should include all activities necessary to accomplish a program’s objectives. Doing so is critical to verifying that an agency is properly allocating resources and accounting for scheduling risks.

With address canvassing, the risk is not that the Bureau will not be able to produce a master address file in time for the 2020 Census, but rather that the Bureau may have to devote more staff and fieldwork— at greater cost— to complete the operation within needed timeframes. The Bureau had initially planned on resolving 75 percent of housing units in-office, but as of March 2017 the blocks that the Bureau had successfully removed from fieldwork account for 51 percent of all housing units. Furthermore, it is uncertain how much more workload the Bureau will be able to resolve in-office during the remainder of the operation, particularly since officials have indicated that the Bureau is not likely to resume the second-phase of in-office address canvassing. If the Bureau does not adequately gauge the amount of remaining in-office address canvassing workload within its time frames, it risks having to complete more address canvassing in-field. Given the uncertainty about the scope of the remainder of the operation, then, it is important for the Bureau to incorporate measures of productivity that reflect the progress and effectiveness of the operation to date.

Conclusions

Given the escalating costs of recent censuses, it is critical for the Bureau to apply innovative approaches that cost-effectively steward resources and deliver quality results throughout the decennial cycle. With its design for reengineered address canvassing, the Bureau made important strides in the types of data and stakeholders it intended to leverage in order to produce an accurate, cost-effective address list.

However, recent management and testing decisions raise questions about whether the Bureau can fully deliver on the Bureau’s initially estimated gains of the reengineered approach. The Bureau suspended the second phase of in-office address canvassing without detailed data on the cost, value, or accuracy of the operation itself, citing budget uncertainty. Given that the Bureau will need to make consequential decisions about the scope and nature of its fieldwork for 2020, it will be important for the Bureau to make use of information on the effectiveness of its reengineered operation, in terms of both the cost and quality of constructing the address list, so that it can justify these decisions and achieve desired outcomes.

Our work has also found that the Bureau needs more and earlier testing before implementing major design changes, yet the fact that the Bureau has completed much of its work on in-office address canvassing without finalizing the evaluations for any related testing leaves questions unanswered about the cost and quality of the operation. A testing program that as laid out early in the decennial cycle, includes smaller and more flexible test opportunities, which could be less susceptible individually to budget uncertainty, can better position the Bureau to support key design decisions and modifications.

In order to achieve savings, the Bureau also needs to make more effective use of productivity information from in-office address canvassing and use estimates that reflect the actual time it takes to complete work. A number of gaps in the Bureau’s production measures, while individually small, appear to have a large cumulative effect on time and workload calculations that could affect resource allocation decisions. With the start of 2020 in-field address canvassing scheduled to begin in a little over 2 years, the Bureau has limited time to finalize its decisions about the scope and nature of address canvassing. Using measures that reflect the progress and effectiveness of the operation— in addition to the process- oriented measures that the Bureau already tracks— can help the Bureau make more informed decisions.

Recommendations for Executive Action

We recommend that the Secretary of Commerce direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to take the following actions: 1. Use the Bureau’s evaluations before 2020 to determine the implications of in-office address canvassing on the cost and quality of address canvassing, and use this information to justify decisions related to its re-engineered address canvassing approach. 2. Early in the next decennial cycle, plan and execute more flexible, and perhaps smaller, address canvassing test and evaluation activity needed to support key design decisions having significant effect on the cost and quality of the census. 3. Use productivity measures that track the progress in completing in- office address canvassing workload and the effectiveness of in-office address canvassing in reducing fieldwork in order to make informed decisions on allocating resources to current and future address canvassing workload so that the operation is completed in a timely and cost-effective manner.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Commerce. In its written comments, reproduced in appendix III, the Department of Commerce had no disagreements with GAO’s findings and recommendations. In addition, the Secretary noted that the Department was conducting its own review.

We are sending copies of this report to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U.S. Census Bureau, and interested congressional committees. The report also will be available at no charge on GAO’s website at http://www.gao.gov.

If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

You asked us to evaluate the U.S. Census Bureau’s (Bureau) reengineered approach to 2020 Address Canvassing. We (1) described the Bureau’s design for its reengineered 2020 Census address canvassing, (2) evaluated the extent to which the Bureau assessed the cost and quality implications of its reengineered address canvassing approach, and (3) assessed the status of the Bureau’s efforts to reduce the in-field address canvassing workload.

For the first objective, we interviewed Bureau staff responsible for the Bureau’s 2020 re-engineered address canvassing approach, and reviewed relevant documentation, such as the Bureau’s December 2015 operational plan for address canvassing, to highlight design features and innovations affecting cost and quality. We also observed the 2016 Address Canvassing Test in-field activities to verify our understanding of how in-office canvassing interfaces with in-field canvassing. Given the Bureau’s timeline for making planning and additional testing decisions, we reported to the Bureau in near-real-time any observed implementation challenges experienced during the test with respect to in-field or in-office canvassing.

For the second objective we interviewed officials in the Bureau’s Geography and Decennial Census Management Divisions and reviewed relevant documentation to determine how the Bureau was managing address canvassing testing, assessing cost and quality implications, making decisions regarding the design of the reengineered address canvassing approach, and managing related risks. In particular, we followed up with officials regarding updates we received to the operational design as well as project-level and program-level risk registers. We assessed how the Bureau was implementing its stated plans for these efforts. We also interviewed officials and reviewed documentation, such as available breakdowns of cost estimates, specific to the Bureau’s estimated cost savings from the reengineered address canvassing approach.

For the third objective we interviewed officials in the Bureau’s Geography Division to determine how they are measuring productivity of the in-office address canvassing operation. We also visited and interviewed officials managing and conducting the Bureau’s in-office address canvassing production to better understand how Bureau employees receive and complete workload assignments and how managers assure quality and track time management data of the operation.

We collected and analyzed Bureau production data and payroll data for in-office address canvassing from two as-of dates during the operation— December 2016 and March 2017— to cover in-office address canvassing since the start of production. We tested the data quality with both sets of data by interviewing officials responsible for the data; reviewing manuals, data dictionaries, and related explanatory documentation; and running calculations to verify the record counts and check the integrity of the data. We found the Bureau’s production data to be sufficiently reliable to calculate estimates of time and workload, while we found the Bureau’s payroll data to be sufficiently reliable to comment on aggregate time spent on the in-office address canvassing operation as a whole (as opposed to the specific phases of the operation).

Using the Bureau’s block-level production data and staffing estimate documentation, we identified different ways to articulate the productivity of the operation, and explored the sensitivity of estimated completion dates and production rates to variation in key productivity assumptions and measures. We also synthesized the Bureau’s payroll data for in-office address canvassing to test the Bureau’s assumptions about staff time allocation. Combined, these activities allowed us to illustrate the effects on time and resource estimates of having accurate productivity measures that focus on the progress and effectiveness of the in-office address canvassing operation in reducing fieldwork. We did this evaluation in light of best practices for scheduling major projects and outlining resource needs.

We conducted this performance audit from August 2016 to July 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ty Mitchell, Assistant Director; Devin Braun; Robert Gebhart; Richard Hung; Cynthia Saunders; Kayla Robinson; Robert Robinson, and Timothy Wexler made key contributions to this report.