2010 Census: Efforts to Build an Accurate Address List Are Making Progress, but Face Software and Other Challenges
Released: 2009-10-21T00:00:00

Letter

Background

As you know, Mr. Chairman, the decennial census is a constitutionally mandated enterprise critical to our nation. Census data are used to apportion seats and redraw congressional districts, and to help allocate over $400 billion in federal aid to state and local governments each year.

We added the 2010 Census to our list of high-risk areas in March 2008, because improvements were needed in the Bureau’s management of IT systems, the reliability of handheld computers (HHC) that were designed in part to collect data for address canvassing, and the quality of the Bureau’s cost estimates. Compounding the risk was that the Bureau canceled a full dress rehearsal of the census that was scheduled in 2008, in part, because of performance problems with the HHCs during the address canvassing portion of the dress rehearsal, which included freeze-ups and unreliable data transmissions. In response to our findings and recommendations, the Bureau has strengthened its risk management efforts, including the development of a high-risk improvement plan that described the Bureau’s strategy for managing risk and key actions to address our concerns. Overall, since March 2008, the Bureau has made commendable progress in getting the census back on track, but still faces a number of challenges moving forward.

One of the Bureau’s long-standing challenges has been building an accurate address file, especially locating unconventional and hidden housing units, such as converted basements and attics. For example, as shown in figure 1, what appears to be a single-family house could contain an apartment, as suggested by its two doorbells. The Bureau has trained address listers to look for extra mailboxes, utility meters, and other signs of hidden housing units, and has developed training guides for 2010 to help enumerators locate hidden housing. Nonetheless, decisions on what is a habitable dwelling are often difficult to make—what is habitable to one worker may seem uninhabitable to another.

If the address lister thought the house in figure 1 was a single family home, but a second family was living in the basement, the second family is at greater risk of being missed by the census. Conversely, if the lister thought a second family could be residing in the home, when in fact it was a single family house, two questionnaires would be mailed to the home and costly nonresponse follow-up visits could ensue in an effort to obtain a response from a phantom housing unit.

LUCA Submissions Generated a Small Percentage of Additions to the MAF

Under the LUCA program, the Bureau partners with state, local, and tribal governments, tapping into their knowledge of local populations and housing conditions in order to secure a more complete count. Between November 2007 and March 2008, over 8,000 state, local, and tribal governments provided approximately 42 million addresses for potential addition, deletion, or other actions. Of those submissions, approximately 36 million were processed as potential address additions to the MAF—or what the Bureau considers “adds.”

According to Bureau officials, one reason LUCA is important is because local government officials may be better positioned than the Bureau to identify unconventional and hidden housing units due to their knowledge of particular neighborhoods, or because of their access to administrative records in their jurisdictions. For example, local governments may have alternate sources of address information (such as utility bills, tax records, information from housing or zoning officials, or 911 emergency systems). In addition, according to Bureau officials, providing local governments with opportunities to actively participate in the development of the MAF can enhance local governments’ understanding of the census and encourage them to support subsequent operations.

The preliminary results of address canvassing show that the Bureau added relatively few of the address updates submitted for inclusion in the MAF through LUCA. Of approximately 36 million addresses submitted, about 27.7 million were already in the MAF. Around 8.3 million updates were not in the MAF and needed to be field-verified during address canvassing. Of these, about 5.5 million were not added to the MAF because they did not exist, were a duplicate address, or were nonresidential. Address canvassing confirmed the existence of around 2.4 million addresses submitted by LUCA participants that were not already in the MAF (or about 7 percent of the 36 million proposed additions).

Bureau officials have indicated that they began shipping out detailed feedback to eligible LUCA participants on October 8, 2009, that includes information on which addresses were accepted. On November 1, 2009, the Office of Management and Budget is scheduled to open the LUCA appeals office that will enable LUCA participants who disagree with the Bureau’s feedback to challenge the Bureau’s decisions. This appeals process allows governments to provide evidence of the existence of addresses that the Bureau missed. If the government’s appeal is sustained, then Bureau will include those addresses in later enumeration activities, and enumerate them if they are located in the field.

The LUCA program is labor intensive for both localities and the Bureau because it involves data reviews, on-site verification, quality control procedures, and other activities, but produced marginal returns. While these were unique additions to the MAF that may not have been identified in any other MAF-building operation, they were costly additions nonetheless. As a result, as the Bureau prepares for the 2020 Census, it will be important for it to explore options that help improve the efficiency of LUCA, especially by reducing the number of duplicate and nonexistent addresses submitted by localities.

The Bureau Generally Completed Address Canvassing Ahead of Schedule but Went Over Budget

The Bureau conducted address canvassing from March to July 2009. During that time, about 135,000 address listers went door to door across the country, comparing the housing units they saw on the ground to what was listed in the database of their HHCs. Depending on what they observed, listers could add, delete, or update the location of housing units.

Although the projected length of the field operation ranged from 9 to 14 weeks, most early opening local census offices completed the effort in less than 10 weeks. Moreover, the few areas that did not finish early were delayed by unusual circumstances such as access issues created by flooding. The testing and improvements the Bureau made to the reliability of the HHCs prior to the start of address canvassing, including a final field test that was added to the Bureau’s preparations in December 2008, played a key role in the pace of the operation; but other factors, once address canvassing was launched, were important as well, including the (1) prompt resolution of problems with the HHCs as they occurred and (2) lower than expected employee turnover.

With respect to the prompt resolution of problems, the December 2008 field test indicated that the more significant problems affecting the HHCs had been resolved. However, various glitches continued to affect the HHCs in the first month of address canvassing. For example, we were informed by listers or crew leaders in 14 early opening local census offices that they had encountered problems with transmissions, freeze-ups, and other problems. Moreover, in 10 early opening local census offices we visited, listers said they had problems using the Global Positioning System function on their HHCs to precisely locate housing units. When such problems occurred, listers called their crew leaders and/or the Bureau’s help desk to resolve the problems. When the issues were more systemic in nature, such as a software issue, the Bureau was able to quickly fix them using software patches.

Moreover, to obtain an early warning of trouble, the Bureau monitored key indicators of the performance of the HHCs, such as the number of successful and failed HHC transmissions. This approach proved useful as Bureau quality control field staff were alerted to the existence of a software problem when they noticed that the devices were taking a long time to close out completed assignment areas.

The Bureau also took steps to address procedural issues. For example, in the course of our field observations, we noticed that in several locations listers were not always adhering to training for identifying hidden housing units. Specifically, listers were instructed to knock on every door and ask, “Are there any additional places in this building where people live or could live?” However, we found that listers did not always ask this question. On April 28, 2009, we discussed this issue with senior Bureau officials. The Bureau, in turn, transmitted a message to its field staff emphasizing the importance of following training and querying residents if possible.

Lower than expected attrition rates and listers’ availability to work more hours than expected also contributed to the Bureau’s ability to complete the Address Canvassing operation ahead of schedule. For example, the Bureau had planned for 25 percent of new hires to quit before, during, or soon after training; however, the national average was 16 percent. Bureau officials said that not having to replace listers with inexperienced staff accelerated the pace of the operation. Additionally, the Bureau assumed that employees would be available 18.5 hours a week. Instead, they averaged 22.3 hours a week.

The Bureau’s address list at the start of address canvassing consisted of 141.8 million housing units. Listers added around 17 million addresses and marked about 21 million for deletion because, for example, the address did not exist. All told, listers identified about 4.5 million duplicate addresses, 1.2 million nonresidential addresses, and about 690,000 addresses that were uninhabitable structures. Importantly, these preliminary results represent actions taken during the production phase of address canvassing and do not reflect actual changes made to the Bureau’s master address list as the actions are first subject to a quality control check and then processed by the Bureau’s Geography Division.

The preliminary analysis of addresses flagged for add and delete shows that the results of the operation (prior to quality control) were generally consistent with the results of address canvassing for the 2008 dress rehearsal. Table 1 compares the add and delete actions for the two operations.

Address Canvassing Costs Exceeded Budget Because of Unanticipated Workload and Hiring

According to the Bureau’s preliminary analysis, the estimated cost for address canvassing field operations was $444 million, or $88 million (25 percent) more than its initial budget of $356 million. As shown in table 2, according to the Bureau, the cost overruns were because of several factors.

One such factor was that the address canvassing cost estimate was not comprehensive, which resulted in a cost increase of $41 million. The Bureau underestimated the initial address canvassing workload and the fiscal year 2009 budget by 11 million addresses. Further, the additional 11 million addresses increased the Bureau’s quality control workload, where the Bureau verifies certain actions taken to correct the address list. Specifically, the Bureau did not fully anticipate the impact these additional addresses would have on the quality control workload, and therefore did not revise its cost estimate accordingly. Moreover, under the Bureau’s procedures, addresses that failed quality control would need to be recanvassed, but the Bureau’s cost model did not account for the extra cost of recanvassing addresses. As a result, the Bureau underestimated its quality control workload by 26 million addresses which resulted in $34 million in additional costs, according to the Bureau.

Bringing aboard more staff than was needed also contributed to the cost overruns. For example, according to the Bureau’s preliminary analysis, training additional staff accounted for about $7 million in additional costs. Bureau officials attributed the additional training cost to inviting additional candidates to initial training due to past experience and anticipated no show and drop out rates, even though (1) the Bureau’s staffing plans already accounted for the possibility of high turnover and (2) the additional employees were not included in the cost estimate or budget.

The largest census field operation will be next summer’s nonresponse follow-up, when the Bureau is to go door to door in an effort to collect data from households that did not mail back their census questionnaire. Based on the expected mail response rate, the Bureau estimates that over 570,000 enumerators will need to be hired for that operation. To better manage the risk of staffing difficulties while simultaneously controlling costs, several potential lessons learned for 2010 can be drawn from the Bureau’s experience during address canvassing. For example, we found that the staffing authorization and guidance provided to some local census managers were unclear and did not specify that there was already a cushion in the hiring plans for local census offices to account for potential turnover. Also, basing the number of people invited to initial training on factors likely to affect worker hiring and retention, such as the local unemployment rate, could help the Bureau better manage costs.

According to Bureau officials, they are reviewing the results from address canvassing to determine whether they need to revisit the staffing strategy for nonresponse follow-up and have already made some changes. For example, in recruiting candidates, when a local census office reaches 90 percent of its qualified applicant goal, it is to stop blanket recruiting and instead focus its efforts on areas that need more help, such as tribal lands. However, in hiring candidates, the officials pointed out that they are cautious not to underestimate resource needs for nonresponse follow-up based on address canvassing results because they face different operational challenges in that operation than for address canvassing. For example, for nonresponse follow-up, the Bureau needs to hire enumerators who can work in the evenings when people are more likely to be at home and who can effectively deal with reluctant respondents, whereas with address canvassing, there was less interaction with households and the operation could be completed during the day.

Problems with accurately estimating the cost of address canvassing are indicative of long-standing weaknesses in the Bureau’s ability to develop credible and accurate cost estimates for the 2010 Census. Accurate cost estimates are essential to a successful census because they help ensure that the Bureau has adequate funds and that Congress, the administration, and the Bureau itself can have reliable information on which to base decisions. However, in our past work, we noted that the Bureau’s estimate lacked detailed documentation on data sources and significant assumptions, and was not comprehensive because it did not include all costs. Following best practices from our Cost Estimating and Assessment Guide, such as defining necessary resources and tasks, could have helped the Bureau recognize the need to update address canvassing workload and other operational assumptions, resulting in a more reliable cost estimate.

The Bureau Needs to Improve Its Policies for Fingerprinting Temporary Employees

To better screen its workforce of hundreds of thousands of temporary census workers, the Bureau plans to fingerprint its temporary workforce for the first time in the 2010 Census. In past censuses, temporary workers were subject to a name background check that was completed at the time of recruitment. The Federal Bureau of Investigation (FBI) will provide the results of a name background check when temporary workers are first recruited. At the end of the workers’ first day of training, Bureau employees who have received around 2 hours of fingerprinting instruction are to capture two sets of fingerprints on ink fingerprint cards from each temporary worker. The cards are then sent to the Bureau’s National Processing Center in Jeffersonville, Indiana, to be scanned and electronically submitted to the FBI. If the results show a criminal record that makes an employee unsuitable for employment, the Bureau is to either terminate the person immediately or place the individual in nonworking status until the matter is resolved. If the first set of prints are unclassifiable, the National Processing Center is to send the FBI the second set of prints.

Fingerprinting during address canvassing was problematic. Of the over 162,000 employees hired for the operation, 22 percent—or approximately 35,700 workers—had unclassifiable prints that the FBI could not process. The FBI determined that the unclassifiable prints were generally the result of errors that occurred when the prints were first made. Factors affecting the quality of the prints included difficulty in first learning how to effectively capture the prints and the adequacy of the Bureau’s training. Further, the workspace and environment for taking fingerprints was unpredictable, and factors such as the height of the workspace on which the prints were taken could affect the legibility of the prints.

Consistent with FBI guidance, the Bureau relied on the results of the name background check for the nearly 36,000 employees with unclassifiable prints. Of the prints that could be processed, fingerprint results identified approximately 1,800 temporary workers (1.1 percent of total hires) with criminal records that name check alone failed to identify. Of the 1,800 workers with criminal records, approximately 750 (42 percent) were terminated or were further reviewed because the Bureau determined their criminal records—which included crimes such as rape, manslaughter, and child abuse—disqualified them from census employment.

Projecting these percentages to the 35,700 temporary employees with unclassifiable prints, it is possible that more than 200 temporary census employees might have had criminal records that would have made them ineligible for census employment. Importantly, this is a projection, and the number of individuals with criminal backgrounds that were hired for address canvassing, if any, is not known.

Applying these same percentages to the approximately 600,000 people the Bureau plans to fingerprint for nonresponse follow-up, unless the problems with fingerprinting are addressed, approximately 785 employees with unclassifiable prints could have disqualifying criminal records but still end up working for the Bureau.

Aside from public safety concerns, there are cost issues as well. The FBI charged the Bureau $17.25 per person for each background check, whether or not the fingerprints were classifiable.

The Bureau has taken steps to improve image quality for fingerprints captured in future operations by refining instruction manuals and providing remediation training on proper procedures. In addition, the Bureau is considering activating a feature on the National Processing Center’s scanners that can check the legibility of the image and thus prevent poor quality prints from reaching the FBI. These are steps in the right direction. As a further contingency, it might also be important for the Bureau to develop a policy for refingerprinting employees to the extent that both cards cannot be read.

The Bureau Used Enhanced Training and Guidance for Canvassing Hurricane Affected Areas

The scale of the destruction in those areas affected by Hurricanes Katrina, Rita, and Ike made address canvassing in parts of Mississippi, Louisiana, and Texas especially challenging (see fig. 2). Hurricane Katrina alone destroyed or made uninhabitable an estimated 300,000 homes. Recognizing the difficulties associated with address canvassing in these areas because of shifting and hidden populations and changes to the housing stock, the Bureau, partly in response to recommendations made in our June 2007 report, developed supplemental training materials for natural disaster areas to help listers identify addresses where people are, or may be, living when census questionnaires are distributed. For example, the materials noted the various situations listers might encounter, such as people living in trailers, homes marked for demolition, converted buses and recreational vehicles, and nonresidential space such as storage areas above restaurants. The training material also described the clues that could alert listers to the presence of nontraditional places where people are living and provided a script they should follow when interviewing residents on the possible presence of hidden housing units.

Additional steps taken by the city of New Orleans also helped the Bureau overcome the challenge of canvassing neighborhoods devastated by Hurricane Katrina. As depicted in figure 3 below, city officials replaced the street signs even in abandoned neighborhoods. This assisted listers in locating the blocks they were assigned to canvass and expedited the canvassing process in these deserted blocks.

To further ensure a quality count in the hurricane-affected areas, the Bureau plans to hand-deliver an estimated 1.2 million questionnaires (and simultaneously update the address list) to housing units in much of southeast Louisiana and south Mississippi that appear inhabitable, even if they do not appear on the address list updated by listers during address canvassing. Finally, the Bureau stated that it must count people where they are living on Census Day and emphasized that if a housing unit gets rebuilt and people move back before Census Day, then that is where those people will be counted. However, if they are living someplace else, then they will be counted where they are living on Census Day.

Validating the Group Quarters Address List Is Important for Reducing Potential Duplicates and Other Errors

To help ensure group quarters are accurately included in the census, the Bureau is conducting an operation called Group Quarters Validation, an effort that is to run during September and October 2009, and has a workload of around 2 million addresses in both the United States and Puerto Rico. During this operation, census workers are to visit each group quarter and interview its manager or administrator using a short questionnaire. The goal is to determine the status of the address as a group quarter, housing unit, transitory location, nonresidential, vacant, or delete. If the dwelling is in fact a group quarter, it must then be determined what category it fits under (e.g., boarding school, correctional facility, health care facility, military quarters, residence hall or dormitory, etc.), and confirm its correct geographic location. The actual enumeration of group quarters is scheduled to begin April 1, 2010.

According to the 2005-2007 American Community Survey 3-year estimates, more than 8.1 million people, or approximately 2.7 percent of the population, live in group quarter facilities. Group quarters with the largest populations include college and university housing (2.3 million), adult correctional facilities (2.1 million), and nursing facilities (1.8 million). The Bureau drew from a number of sources to build its list of group quarters addresses including data from the 2000 Census, LUCA submissions, internet based research, and group quarters located during address canvassing.

During the 2000 Census, the Bureau did not always accurately enumerate group quarters. For example, in our prior work, we found that the population count of Morehead, Kentucky, increased by more than 1,600 when it was later found that a large number of students from Morehead State University’s dormitories were erroneously excluded from the city’s population when the Bureau incorrectly identified the dormitories as being outside city limits and in an unincorporated area of Rowan County. Similarly, North Carolina’s population count was reduced by 2,828 people, largely because the Bureau had to delete duplicate data on almost 2,700 students in 26 dormitories at the University of North Carolina at Chapel Hill. Precision is critical because, in some cases, small differences in population totals could potentially impact apportionment and/or redistricting decisions.

The Bureau developed and tested new group quarters procedures in 2004 and 2006 that were designed to address the difficulties the Bureau had in trying to identify and count this population during the 2000 Census. For example, the Bureau integrated its housing unit and group quarters address lists in an effort to reduce the potential for duplicate counting as group quarters would sometimes appear on both address lists. Moreover, the Bureau has refined its definition of the various types of group quarters to make it easier to accurately categorize them. The operation began on September 28, as planned, in all 151 early opening local census offices and was 95 percent complete as of October 16, 2009. We have begun observations and will report our findings at a later date.

It Will Be Important for the Bureau to Determine Return on Investment for Each MAF-Building Activity

With the cost of enumerating each housing unit continuing to grow, it will be important for the Bureau to determine which of its multiple MAF- building operations provide the best return on investment in terms of contributing to accuracy and coverage. According to the Bureau, it is planning to launch over 70 evaluations and assessments of critical 2010 Census operations and processes, many of which are focused on improving the quality of the MAF. For example, the Bureau plans to study options for targeted address canvassing as an alternative to canvassing every block in the country. The Bureau considered two major criteria for determining which studies to include in their evaluation program—the possibility for significant cost savings in 2020 and/or the possibility of significant quality gains in 2020. As the Bureau makes plans for the 2020 Census, these and other studies could prove useful in helping the Bureau streamline and consolidate operations, with an eye toward controlling costs and improving accuracy.

Completing Testing for MAF/TIGER System Will Be a Challenge

Automation and IT systems will play a critical role in the ability of MAF/TIGER to extract address lists, maps, and provide other geographic support services. In our prior work, however, we have called on the Bureau to strengthen its testing of the MAF/TIGER system. In March 2009, for example, we reported and testified that while the MAF/TIGER program had partially completed testing activities, test plans and schedules were incomplete and the program’s ability to track progress was unclear.

Specifically, while the Bureau had partially completed testing for certain MAF/TIGER products (e.g., database extracts) related to address canvassing, subsequent test plans and schedules did not cover all of the remaining products needed to support the 2010 Census. Further, Bureau officials stated that although they were estimating the number of products needed, the exact number would not be known until the requirements for all of the 2010 Census operations were determined. As such, without knowing the total number of products and when the products would be needed, the Bureau risked not being able to effectively measure the progress of MAF/TIGER testing activities. This in turn increased the risk that there may not be sufficient time and resources to adequately test the system and that the system may not perform as intended. At that time we recommended that the MAF/TIGER program establish the number of products required and establish testing plans and schedules for 2010 operations.

In response to our recommendations, the Bureau has taken several steps to improve its MAF/TIGER testing activities, but substantial work remains to be completed. For example, the MAF/TIGER program has established the number of products and when the products are needed for key operations. Furthermore, the program finalized five of eight test plans for 2010 operations, of which the testing activities for one test plan (address canvassing) have been completed; three are under way; and one has not yet started. Lastly, the program’s test metrics for MAF/TIGER have recently been revised; however, only two of five finalized test plans include detailed metrics. While these activities demonstrate progress made in testing the MAF/TIGER system, the lack of finalized test plans and metrics still presents a risk that there may not be sufficient time and resources to adequately test the system and that the system may not perform as intended.

Given the importance of MAF/TIGER to establishing where to count U.S. residents, it is critical that the Bureau ensure this system is thoroughly tested. Bureau officials have repeatedly stated that the limited amount of time remaining will make completing all testing activities challenging.

Concluding Observations

The Bureau recognizes the critical importance of an accurate address list and maps, and continues to put forth tremendous effort to help ensure MAF/TIGER is complete and accurate. That said, the nation’s housing inventory is large, complex, and diverse, with people residing in a range of different circumstances, both conventional and unconventional. The operations we included in this review generally have proceeded as planned, or are proceeding as planned. Nevertheless, accurately locating each and every dwelling in the nation is an inherently challenging endeavor, and the overall quality of the Bureau’s address list will not be known until the Bureau completes various assessments later in the census. Moreover, while the Bureau has improved its management of MAF/TIGER IT systems, we continue to be concerned about the lack of finalized test plans, incomplete metrics to gauge progress, and an aggressive testing and implementation schedule going forward. Given the importance of MAF/TIGER to an accurate census, it is critical that the Bureau ensure this system is thoroughly tested.

On October 15, 2009, we provided the Bureau with a statement of facts for our ongoing audit work pertaining to this testimony, and on October 16, 2009, the Bureau forwarded written comments. The Bureau made some suggestions where additional context or clarification was needed and, where appropriate, we made those changes.

Mr. Chairman and members of this Subcommittee, this concludes my statement. I would be happy to respond to any questions that you might have at this time.

If you have any questions on matters discussed in this statement, please contact Robert N. Goldenkoff at (202) 512-2757 or by e-mail at goldenkoffr@gao.gov. Other key contributors to this testimony include Assistant Director Signora May, Peter Beck, Steven Berke, Virginia Chanley, Benjamin Crawford, Jeffrey DeMarco, Dewi Djunaidy, Vijay D’Souza, Elizabeth Fan, Amy Higgins, Richard Hung, Kirsten Lauber, Andrea Levine, Naomi Mosser, Catharine Myrick, Lisa Pearson, David Reed, Jessica Thomsen, Jonathan Ticehurst, Kate Wulff, and Timothy Wexler.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.