2020 Census: Initial Research Milestones Generally Met but Plans Needed to Mitigate Highest Risks
Released: 2012-12-07T00:00:00

Letter

Background

The Bureau’s 2020 Census early research agenda is focused on conducting the 2020 Census at a lower cost than the 2010 Census while maintaining high quality results. In order to do this the Bureau has developed a range of design alternatives for the 2020 Census. The Bureau’s 2020 design alternatives have potential for containing costs but also have the potential to increase risks. The design alternatives focus on options to use the Internet and other social media to increase response rate, target address canvassing, and expand the use of administrative records. Figure 1 shows the current range of six 2020 Census design alternatives currently being considered and how the three key efforts fit into them. According to the Bureau, the final 2020 design is likely to incorporate both existing approaches as well as design alternatives that have never been used in the decennial census. Greater changes to the overall design will result in greater potential for cost savings. However, greater design changes also could result in greater risk, and testing will be needed to identify the risks, costs, and benefits of any new approaches. According to the Bureau, alternative one has the lowest risk because it most closely mirrors the 2010 Census design (mailout of census forms, and in-person follow-up of non respondents) and is not dependent on implementing innovations such as increased use of administrative records and targeted address canvassing.

The Bureau formally kicked off initial planning for the 2020 Census on October 3, 2008, with the 2020 Census Planning Summit. At that meeting, decennial-census managers and subject matter experts discussed challenges, strengths, and weaknesses that would need to be considered during the 2020 Census strategic decision-making process. From 2009 to 2011, during the options analysis phase of 2020 Census planning, Bureau officials went through a research topic winnowing process where they: identified cost and quality drivers, conducted brainstorming sessions, received expert input (e.g., National Academy of Sciences), cataloged over 600 ideas, clustered those ideas into over 100 projects with research questions, and then prioritized them against the for the 2020 Census. The Bureau identified Bureau’s Guiding Principles26 research projects to inform preliminary design decisions with the goal of conducting the 2020 Census at a lower cost than the 2010 Census, while maintaining high quality.

Using the 26 research projects, Bureau officials plan to validate the feasibility of the various strategies for reducing costs between fiscal years 2012 and 2014. Bureau officials stated that they are planning to narrow down the design alternatives by September 2014, and that research would continue into the next phase. Bureau officials believe that conducting research earlier in the decade will better position them to make more effective changes to the 2020 Census design. Figure 2 shows the five phases of the life cycle for the 2020 Census.

The Bureau is now in the “Early Research and Testing” phase and is currently refining and developing its 26 project plans. These plans include, among other requirements, project goals, objectives, scope, limitations, and skills required, and had a set of required iterative research and testing deliverables due 30, 60, 90, and 120 days after project kickoff. According to Bureau documentation, this process encouraged the development and proper consideration of ideas. The Bureau selected 17 of the 26 projects to start in fiscal year 2012. The remaining projects are scheduled to begin in fiscal years 2013 and 2014. Of the 17 research projects the Bureau is conducting in 2012, 14 projects are to inform design decisions related to the Internet, administrative records, and targeted address canvassing. See figure 3 for a list of these 14 research projects and their objectives. According to the Bureau’s timeline, project teams are to complete their work on these 14 projects by September 2014.

Move mouse over each activity to see more information afterwards CLICK to close the open pop-up information display panel.

To print text version of this graphic, go to appendix II.

Key Efforts Present the Bureau with Both Opportunities and Risks

Research on new methods likely to result in a more cost-effective 2020 Census must be accomplished early enough in the decade to confirm their likely impact on both cost and quality and to inform timely design decisions. The use of the Internet as a response option, a potential move towards targeted address canvassing, and the possible use of administrative records to replace data collected during census field operations present the Bureau with opportunities to reduce costs while maintaining a high quality census. As part of the 2020 Census planning process the Bureau has identified both program-level and project-level risks associated with these options. According to the Bureau, program- level risks span the entire 2020 research and testing phase and if not managed properly could jeopardize the phase’s goals and objectives. In contrast, project-level risks pertain to the successful completion of specific projects. However, the Bureau has not developed mitigation and contingency plans for the project-level risks. Also, the Bureau has not developed cost estimates for any of its 2020 research and testing projects as required in guidance provided to project teams. Finally, some of the project plans we reviewed were incomplete. For example, not all project plans identified the necessary staff resources, while other project plans failed to provide performance metrics for measuring progress and for determining the project’s final outcome.

All Three Key Efforts Present the Bureau with Cost Saving Opportunities

Internet

During the 2000 Census, the Bureau piloted an Internet response option that had a limited number of respondents. They considered building on this 2000 experience for the 2010 Census, but the Bureau decided in July 2006 not to include the Internet option in the design for the 2010 Census because test results indicated that the Internet option did not increase the overall response rate, including response among hard-to-count population groups, and they had underestimated the costs of the contract that included developing the Internet option.

However, more recent tests conducted in 2011 showed that adding an Internet response option increased the overall response rate. The 2011 test results, coupled with the increased prevalence and accessibility of the Internet, led Bureau officials to commit to providing an Internet response option for the 2020 Census. With regard to the cost for the 2020 Census, if this option can help achieve an overall increase in the response rate it can save money, since Bureau field staff would need to visit fewer households during nonresponse follow-up (NRFU), which is the largest and most costly census field operation. Furthermore, testing has shown that the cost of an Internet survey is low compared to a mail survey, which incurs printing and postage costs. Moreover, web survey responses are generally available more quickly and are of better quality than responses from a mail survey because there is no lag time as the responses are captured in real time and there are reminders to prompt the respondent if a question is unanswered. Quicker and complete responses can also help reduce the amount of time and money spent on following up on late or incomplete census forms.

In April and November 2011, the Bureau tested an Internet response option in its American Community Survey (ACS), and beginning in January 2013, the Bureau plans to offer it as a response option. Bureau officials have stated that they intend to build on this existing information technology infrastructure in order to reduce the cost of implementing an Internet option for the 2020 Census.

Given the Bureau’s decision to move forward with an Internet design option, six research and testing projects are focused on how best to implement an Internet response option in concert with other self-response options. The project team with primary responsibility for coordinating the design of the Internet response option is tasked with answering the following research question:

How does the Bureau leverage technology and new response modes (including the Internet) to increase self-response, improve nonresponse follow-up data collection strategies, while maintaining overall quality?

One objective of this project is to determine what contact strategy should be used to inform the public that census responses can be filed via the Internet in order to increase self-response (i.e., Bureau use of email, social networks or phone for initial contact or for reminders). Contact strategy testing will examine the impact of various Bureau instructions to respondents, as well as the timing of the delivery of the mailing pieces, on such factors as response rates, return rates, percentage of Internet returns, and speed of returns. This project will also determine what mode (paper, Internet, or phone) is best for each demographic, geographic, and language group, as well as determine the effect a lack of access to technology has on each demographic group. Additionally this project will refine and test security features. The other five projects will cover such areas as types of languages that will be offered by response mode, public perception and potential concerns about an Internet response, and management of the workload for multiple modes of response.

In the 2010 and earlier censuses, the Bureau mounted a full address canvassing operation, where field staff verified every housing unit in the nation to update the Master Address File (MAF) and the associated mapping system called TIGER® (Topologically Integrated Geographic Encoding and Referencing). This labor-intensive effort was one of the more expensive components of the 2010 Census: the 2010 Address Canvassing Operation required 140,000 temporary workers to verify 145 million addresses (by going door-to-door) at a cost of $444 million, or 3 percent of the $13 billion total cost of the 2010 Census. One of the reasons the Bureau conducted a full address canvassing for the 2010 Census was because it wanted to capture hidden and hard-to-capture addresses, such as housing units in converted garages and large storage sheds. For the 2020 Census the Bureau would like to reduce the amount of field work and cost of address and map updates by targeting the address canvassing operation. According to Bureau planning documents, through early research and testing the Bureau plans to attempt to determine the quality and stability of its address file in terms of the number of changes (adds, deletes, duplicates, and corrected addresses) that took place to each census block in 2010. Depending on that outcome, the Bureau would target for address canvassing only those blocks where the quality of the addresses was determined to be inadequate.

To determine whether to target address canvassing for the 2020 Census, the Bureau has two ongoing efforts to improve the Bureau’s map and address databases. First, the Bureau’s Geography Division is working with the United States Postal Service (USPS) and other federal agencies on a new program called the Geographic Support Systems (GSS) Initiative, which allows government agencies at all levels to regularly share and continuously update their address lists with the Bureau. According to current plans, USPS will continue providing the Bureau with address updates. Likewise, tribal, state, and local governments, which maintain address lists for purposes such as emergency response and property assessment, will also have the opportunity to share addresses with the Bureau throughout the decade, rather than solely 2 years prior to the decennial, as had been the case in prior decennial censuses. Additionally, the GSS Initiative is working to develop, and in some cases improve, methods for identifying and capturing hidden and hard-to- capture addresses.

As part of this first effort, the Bureau is investigating the role and possible contributions the private sector can make in improving its address and mapping databases. For the 2010 Census the Bureau relied heavily on the private sector to update addresses and maps. Specifically, in June 2002, the Bureau awarded an 8-year contract of about $200 million to improve the accuracy of its mapping database and investigate options and methods of updating its address database. The contract, among other tasks, corrected in TIGER® the location of streets, boundaries, and other map features so that coordinates would be aligned with their true geographic locations. According to Bureau officials, continued reliance on the private sector is necessary in order to maintain the major upgrades that were made to its address and mapping databases last decennial. The Geography Division has nine contracts in place, totaling about $90 million dollars, to help it manage its mapping and address database (see app. III for a list of those contracts). Some of the work being done by the private sector includes a service contract to provide system upgrades and maintenance to the MAF/TIGER® database and a data entry services contract to update the MAF/TIGER® database with address information provided by state and local governments.

Moreover, to ensure the most current address and mapping data are available for the 2020 Census, the Bureau, in April 2012, began conducting market research with three private-sector firms to assess how those firms might assist the Bureau with its address and mapping needs. Possibilities to be explored include how to best maintain address and map data, understand private-sector innovation and technology, and options for address and map data-sharing methods. According to Bureau officials, negotiations concerning the scope, tasks, and activities to be completed with the first contractor have taken place. However, negotiations with the remaining two firms have not begun. Use of the private sector for mapping is an area we have previously reported needs to be explored. Specifically, in December 2010, we listed a number of reexamination areas which have particular implications for controlling costs and improving accuracy and raised the question: To what extent can private-sector and other sources of information such as maps, address lists, and geographic databases be employed to help support the census?

The second effort to improve the Bureau’s address and mapping database is centered on the 2020 Research and Testing Program. There are three research projects designed to inform the Bureau’s decision about the extent to which it might be able to conduct a targeted address canvassing operation. Two projects will use modeling to predict where coverage errors (e.g., missed or duplicate addresses) occur in the MAF. These models will be used to assess the quality of data sources, types of errors, and what information, such as a missing map location for an address, is not in the MAF. If the Bureau decides to conduct targeted address canvassing, this information will be used to determine which areas meet acceptable quality standards, as well as where targeted address canvassing would be effective. According to the Bureau, collectively the two projects are to answer the following question:

As the related GSS Initiative proceeds, how will the Bureau determine the required level of quality needed in the address frame to conduct an accurate census and then measure the quality of the continually updated MAF for that purpose?

The third research project in this effort, the Local Update of Census Addresses (LUCA) Program Improvement project, will examine how to modify the Bureau’s existing partnership program in consultation with the GSS Initiative. The LUCA program is a partnership program that provides an opportunity for local and tribal governments to review and update individual address information or block-by-block address counts from the MAF and associated geographic information in the TIGER® database just prior to the decennial. The goal of LUCA is to improve the completeness and accuracy of address information. This research project’s objective is to determine the best way to modify and improve LUCA procedures to ensure compatibility with continual updating of address information throughout the decade. The project will also determine whether it is necessary for local and tribal governments to actively participate in LUCA, a once-a-decade operation, if they have been submitting addresses and mapping data continuously throughout the decade as part of the GSS Initiative.

Administrative records are a growing source of information on individuals and households. For purposes of the decennial census, the Bureau is considering administrative records from government agencies, including tax data and Medicare records, as well as commercial sources to identify persons associated with a particular household address. During the 2010 Census, the Bureau made limited use of administrative records. For example, the Bureau used USPS files to update its address list, and federal agency records (such as those from the Department of Defense) were used to count military and federal civilian employees stationed outside of the United States.

The amount of and quality of administrative records the Bureau is able to collect will impact the amount of cost savings the Bureau is able to realize. could allow the Bureau to reduce the scale of its NRFU operation and reduce the need for field office space and staff. The 2012 research and testing projects are also focused on identifying the best available administrative records to use for address frame building. Administrative data from other sources would be combined with USPS data, thus allowing the Bureau to continuously update its MAF throughout the decade. Finally, administrative records could reduce the cost of quality control, according to Bureau officials. In order to ensure the accuracy of the field work conducted during census field operations, such as NRFU, the Bureau sends quality control staff into the field to verify the work completed by a random sample of enumerators. Bureau officials are determining for 2020 whether administrative records can be used as a quality control check, rather than sending staff into the field.

There are nine research and testing projects that will inform the Bureau’s decision making on the extent to which the Bureau can expand the use of administrative records in order to reduce costs. These research and testing efforts are focused on determining whether the quality of administrative records is sufficient to be used for this purpose. Initial testing is to be completed and decisions on these matters are to be made by September 2014. Of the nine, there are two primary administrative records research projects. They are tasked with answering the following research questions:

How can the Bureau best develop and maintain an independent collection of administrative records and assess the quality of those records (best sources and methods)?

How can the Bureau leverage administrative records (including commercial files) to significantly reduce decennial census cost while maintaining quality?

The objective of one of these projects is to acquire, process, and analyze administrative records from federal, state, and commercial sources to assess their utility for the 2020 Census. The other project will research and test methods to enhance NRFU operations with administrative records, such as replacing phone or in-person collected data with administrative data.

The Bureau Has Identified 2020 Census Program- Level and Project-Level Risks but Has Not Developed Project-Level Mitigation and Contingency Plans

Bureau officials are aware that the changes they want to make to the decennial census come with many risks. The Bureau has identified and prioritized from high to low both program-level and project-level risks. According to the Bureau, program-level risks span the entire 2020 research and testing phase and if not managed properly could jeopardize the phase’s goals and objectives. In contrast, project-level risks pertain to the successful completion of projects. In accordance with best risk management practices, the Bureau has identified and drafted mitigation plans for all of the 14 program-level risks it has identified. In May 2012 the Bureau designated two of the 14 program-level risks as high risk:

Timely research and testing results: The 2020 research and testing strategy involves “many small field tests to research design alternatives with an accelerated, agile, and informed decision-making process for incorporating changes to the 2020 Census design.” According to the Bureau, if the research and testing results are late, then decisions will not be made on time and the program may not be ready to move out of the research and testing phase on schedule. According to the Bureau’s 2020 Census Risk Management Mitigation Plans, officials have begun implementing a mitigation strategy for this risk, including documenting clear roles and responsibilities and communicating them to staff; establishing a schedule that contains decision milestones and tracking interdependencies of research results and testing projects; and reviewing ongoing project status including updates on dependencies, risks, and metrics.

Significant budget cuts: The Bureau reported that if its funding for 2020 Census planning is significantly reduced for fiscal year 2013, key projects will have to be delayed and as a result the Bureau could have major technical and operational difficulties making preliminary design decisions. According to Bureau documents, to mitigate this risk, Bureau officials have briefed staff from House of Representatives Subcommittee for Commerce, Justice, Science and Related Agencies, Committee on Appropriations, and the Office of Management and Budget on the implications of reduced funding and plan to brief oversight committee staff. In addition, Bureau officials reported they have analyzed budgetary needs for fiscal year 2013 and prioritized projects to manage limited resources.

Related to these risks, a June 2012 congressional hearing held by the Joint Economic Committee discussed the impact of eliminating or reducing funding to the ACS. According to Bureau officials, such action would have a significant impact on the 2020 Census research and testing program because ACS is one of its testing platforms. This risk is not listed as a program-level risk, or part of the current risk mitigation plans, but Bureau managers told us they are strategizing on what they will do if ACS is not funded and will update the risk mitigation plan accordingly.

The Bureau has identified lack of support from external stakeholders as a medium program-level risk. Bureau officials are concerned that if external stakeholders, such as Congress, the Office of Management and Budget, and the National Academy of Sciences, do not support research on alternative design options, then the research and testing agenda will have to be redesigned. To manage this risk, Bureau officials have communicated the alternative design options to a wide range of external stakeholders, but are concerned that support may diminish as time passes and stakeholders change. To mitigate this risk, the Bureau is engaging external stakeholders in communications regarding 2020 research and testing efforts and seeking feedback on Bureau plans. According to Bureau documents, the Bureau has met with external stakeholders, including congressional staff on its House of Representatives and Senate oversight committees, to discuss 2020 Census planning, and is working on a strategy for keeping stakeholders informed as plans progress.

Unlike the program-level risks, risk mitigation and contingency plans have not been drafted to address the project-specific risks. Specifically, each of that the Bureau’s research and testing projects has its own risk registeridentifies and prioritizes the risks associated with that project, but no risk mitigation or contingency plans have been developed for these project- level risks. Several risks appear in multiple project team risk registers, including tight time frames and accurate cost information. For example, one Internet-related project listed that there is a risk that if tests are not timed properly, and if time frames are too tight then they may not be able to apply what they have learned from one test to the next. Similarly, one of the administrative records project teams was concerned that one year to integrate data, build a model, and conduct adequate analysis of administrative records was “not realistic.” In another example concerning cost data, two teams stated that without accurate cost data at the project level, research results will not be sufficient to inform the design of the 2020 Census.

Project teams also identified project-level risks directly related to the use of administrative records in place of data collected by enumerators during field visits. Among these risks is access to administrative records to build a high quality compilation of administrative records for each household across the country. To accomplish this task, the Bureau needs timely access to data collected by the federal government and, as previously mentioned, is currently compiling administrative records from various federal agencies including the Departments of Housing and Urban Development and Health and Human Services. However, not all agencies are authorized to provide their data to the Bureau thus limiting, and in some cases preventing, the Bureau’s use of that data. Bureau officials provided us with two examples of agency administrative record sources that they are not authorized to access: records maintained under the Family Educational Rights and Privacy Act and the National Directory of New Hires. Bureau officials said they would like to have access to these and other data sources for statistical purposes in order to further increase the number and improve the quality of available administrative records. According to Bureau officials, reviewing and determining whether they have access to administrative records with every agency is a time consuming process and the Bureau would like to have the ability to fully access and use agency administrative data, especially since the confidentiality of census data are protected by Title 13.forward, according to a senior Bureau official, legislation might need to be enacted that would compel federal agencies to provide the Bureau with access to administrative data for the decennial census. However, the To move this enactment of such legislation is likely to be a time consuming process that raises serious policy concerns including whether the Bureau has made a sufficiently strong case that it needs access to all federally collected data and the resulting impact on personal privacy protections.

According to Bureau officials, the project teams did not submit risk mitigation and contingency plans for the project-level risks as required at the 90-day checkpoint because after reviewing all of the project-level risk registers the project leaders first wanted to unduplicate similar risks across projects in order to prevent redundancies. Bureau officials were not able to provide a date for when risk mitigation and contingency plans would be available. However, they stated they are committed to managing risk in order to avoid the missteps encountered in 2010 when the Bureau was on our high-risk list due to weaknesses in managing IT, operational planning, and cost estimating. In our prior work, we reported that risk mitigation involves identifying, analyzing, prioritizing, and documenting risks, and ideally more than one alternative should be assessed. The Department of Commerce’s Inspector General has recommended that risk management activities begin from the outset of the current decennial census life cycle, rather than just before field operations. Without mitigation plans the 2020 Census planning team may not be able to fully manage risks associated with these projects.

Project-Level Cost Estimates Have Not Been Developed

The Bureau did not develop cost estimates for any of its 2020 research and testing projects by the 90-day checkpoint as required in guidance provided to project teams. The importance of reliable cost estimates is underscored by the Bureau’s experience leading up to the 2010 Census, where we found that the Bureau’s cost estimates lacked detailed documentation on data sources and significant assumptions and was not comprehensive because it did not include all costs. Partly as a result, some operations had substantial variances between their initial cost estimates and their actual costs. For example, the 2010 address canvassing operation cost $88 million more than the original estimate of $356 million, an overrun of about 25 percent; and the Bureau’s 2010 NRFU operation cost $1.6 billion, about $660 million, or 29 percent, less than the Bureau initially estimated.

According to 2020 Census planning documents, at the 90-day checkpoint project teams were to: (1) prepare and submit cost estimate requests needed for the project; (2) document estimates for completing the work for the project; (3) identify and document funds needed for additional resources and requirements, and (4) prepare and submit requests for additional funds. According to a senior Bureau official, cost estimates have not yet been completed because more work needs to be done to determine project costs and they are in the process of formulating cost estimates for each project.

Research and Testing Project Plans Were Delivered on Schedule, but Not All Project-Level Resources and Performance Metrics Have Been Identified

The Bureau’s early research and testing is currently refining the project plans associated with the14 projects related to the Internet, administrative records, and targeted address canvassing. Each of the research and testing projects has a set of required deliverables due 30, 60, 90, and 120 days after project kickoff. We determined that each project team had provided the 120-day required deliverables on schedule to Bureau managers in June 2012. The research and testing documents required by the 2020 Census planning team and delivered by the 14 project teams describe management and technical plans for each project, and include a team charter, project plan, resources, preliminary schedule, and field tests. These documents are important because they serve as the baseline for each of the research and testing projects.

However, not all resource requirements were documented. For example, some project teams did not fully document the types of skills needed, or perform a skills gap assessment to determine the resources needed to carry out their respective projects as required in the Bureau’s planning template (the template lays out in detail what is to be included in the deliverables). Bureau officials plan to use this skills information to allocate staff to ensure teams have the skills necessary to complete their projects. While 11 project teams provided all the required information, 2 project teams conducting research on issues that will inform decisions on targeted address canvassing and expanded use of administrative records, respectively, did not provide documentation that a skills gap analysis was conducted and another project researching expanded use of administrative records did not provide the necessary skill sets. According to Bureau officials, they are working to document these skills sets, as well as any gaps. Completing this analysis is important to ensuring sufficient resources are available for conducting the research and testing projects.

We also found incomplete performance metric documentation for several projects. According to Bureau guidance, project teams were to provide performance metrics (including the methodology for monitoring and evaluating project performance), as well as exit criteria for determining each project’s final outcome. However, the “Enhancing Demographic Analysis” team that is researching the use of administrative records to assess the accuracy of census counts at the national level did not provide any of the required performance metrics, while six teams failed to include performance metrics that could be used to monitor research and testing progress. According to Bureau officials, not all teams followed the project templates, and those teams have been asked to adhere to the requirements and provide all necessary information. Project performance metrics are a key element of effective project planning. As we have noted in prior work on project planning, measurable performance goals should be identified and performance data should be gathered to determine progress and whether desired outcomes have been achieved. Absent such metrics, the Bureau does not have the assurance that it will be able to avoid potential slips in the research and testing schedule, or certainty as to whether project outcomes will be adequate for making decisions.

Conclusion

Just over 7 years remain until Census Day 2020. While this might seem like an ample amount of time to shore up the Bureau’s planning process and take steps to control costs, past experience has shown that the chain of interrelated preparations that need to occur at specific times and in the right sequence leave little room for delay or missteps. According to the Bureau, if 2010 operations are repeated in 2020, continued growth in population size and complexity will likely lead to an unsustainable increase in census cost. Further, traditional enumeration methods may no longer effectively produce a high quality census. To contain costs and maintain quality, bold innovations in both planning for and conducting the 2020 Census will be required.

The early research and testing phase represents a critical stage in preparing for the 2020 Census. At this time Bureau management is shaping the next decennial census as it determines what new operations will be a part of the 2020 Census design, which operations need to be revised, what risks remain, and how those risks will be mitigated. We have highlighted some of the cost savings that can be attained through three new operational changes being considered—the use of the Internet as a self-response option, replacing enumerator collected data with administrative records, and targeting only certain addresses for field verification.

However, these innovations also come with risks that need to be managed. Specifically, while the Bureau has identified project-level risks for the 14 projects we reviewed, it has not developed risk mitigation plans for them. Given the number of changes the Bureau would like to make for 2020 it is imperative that risks are sufficiently managed and that mitigation plans be in place. For example, the Bureau needs to manage the risk for accessing federally collected administrative data that it will use for the 2020 Census. While it currently has access to some federally collected data, the Bureau would like access to all federally collected data, which according to the Bureau would require a statutory change. However, such a change raises serious policy concerns including the impact on personal privacy protections, and most likely would be a time consuming process. Therefore, it is imperative that risk mitigation and contingency plans be in place if the Bureau is unable to successfully gain access to all federally collected data. Other areas that need attention include ensuring that all research and testing projects have reliable cost estimates, so as to avoid research and funding shortfalls, that performance metrics are identified to ensure project goals are monitored and met, and that skills are identified to ensure the selection and assignment of appropriate staff to each project.

Recommendations for Executive Action

We recommend that the Acting Secretary of Commerce require the Under Secretary for Economic Affairs who oversees the Economics and Statistics Administration, as well as the Acting Director of the U.S. Census Bureau, to take the following three actions to improve the Bureau’s Research and Testing for the 2020 Census, and thus better position the Bureau to carry out a cost-effective decennial census:

Develop risk mitigation and contingency plans for all projects to ensure that risks are adequately managed to minimize their effect on the project.

Develop cost estimates for each project.

Ensure documentation for projects are complete, including specifying the performance metrics that will be the basis for determining that each of the projects has completed its work and identifying skills needed to inform the selection and assignment of appropriate staff to each research project.

Agency Comments and Our Evaluation

The Acting Secretary of Commerce provided written comments on a draft of this report on October 22, 2012. The comments are reprinted in appendix IV. The Department of Commerce agreed with the assessments and recommendations of the report. In addition, the Acting Secretary of Commerce provided a technical comment and suggestion where additional context might be needed, and we revised the report to reflect this comment.

The Bureau in its comments stated that our report accurately represented the extensive work that has been completed during the early research and testing phase for the 2020 Census. To address our concerns, the Bureau noted that it is now focusing on managing risks at the project level and will begin obtaining more specific cost estimation details for each project. The Bureau also agrees with our recommendations to ensure that performance metrics and skill sets are identified for all projects and teams. Finally, the Bureau stated that these efforts are either already underway or are planned as a major focus during fiscal year 2013.

As arranged with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days after the date of this report. At that time, we will send copies of this report to the Acting Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on the GAO website at http://www.gao.gov.

If you have any questions on matters discussed in this report, please contact me at (202) 512-2757 or by e-mail at goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are located in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report evaluates the U.S. Census Bureau’s (Bureau) efforts to improve the cost-effectiveness of the enumeration, paying particular attention to the following three key efforts that past Bureau research has shown could result in substantial cost savings: (1) leveraging the Internet to increase self-response; (2) improving how the Bureau builds its address and mapping databases, including a possible move to targeted address canvassing, and use of private-sector geographic data; and (3) using administrative records, such as tax data, to reduce nonresponse follow-up costs. To meet our objectives of identifying what opportunities and risks, if any, the Bureau might need to consider for these efforts going forward, and examining to what extent these three efforts are on track with respect to scheduling, resources, and other performance metrics, we reviewed Bureau documents pertaining to the early planning of the 2020 Census. These generally consisted of the Bureau’s operational plans, strategic framework documents, strategies and planning memorandums, timelines, and benchmarks. Many of these documents were considered draft, but Bureau officials said they were sufficiently developed for purposes of our review. To describe the Bureau’s efforts to use private-sector firms to assist with its address and mapping needs, we spoke to agency officials and reviewed the Bureau’s current mapping and address contracts and its request for information to solicit input from private-sector firms. We interviewed Bureau officials responsible for the early planning of the 2020 Census and inquired about their process for designing their research and testing program, as well as opportunities and risks presented by these three key efforts.

We reviewed our prior reports on and other reports on leading practices to identify gaps in the Bureau’s research and testing efforts as it relates to schedule, resources, and performance metrics. We also reviewed reports from and spoke to knowledgeable stakeholders, such as officials at the National Academy of Sciences to determine opportunities and risks related to the Internet, administrative records, and the address and mapping databases. In addition, we spoke with Bureau officials and reviewed our prior work to identify obstacles the Bureau may need to address moving forward; for example, legal barriers to the Bureau’s access to administrative records.

To assess the Bureau’s progress in meeting its stated goals for the research and testing program, we evaluated whether each of the 14 research projects the Bureau initiated in 2012 had provided the required deliverables: the 120 day project plan, the project schedule, the project charter, and the project risk register. We also performed a content analysis of Bureau planning documents. For this analysis we examined the deliverables for each of the 14 research projects. We developed a data collection instrument to record the presence or absence of 11 pieces of information, or elements, in the deliverables for each project. The 11 elements were identified through a review of the Bureau’s project plan template which outlines information that each project was expected to report by the end of the 120-day initiation phase. A coding dictionary was created to provide a uniform set of coding criteria for each element coded. Each element could be coded as present, partially present, or absent. During the coding process, two analysts independently coded documents for each project using the coding dictionary as a guide. In the next phase, the two coders met to reconcile any discrepancies between their coding decisions and then made a final determination for that element. In cases of disagreement, the engagement methodologist was consulted as a tiebreaker. Following the coding process, the team analyzed the number of present and absent elements to identify patterns for each element across the 14 projects, as well as the completeness of documentation for each project.

This engagement did not examine the Bureau’s information technology policies, procedures, and information security in depth because our information technology team is working on two engagements that cover these issues.

Appendix II: Fourteen Research Projects Will Inform Design Decisions on Key Efforts (Text for Interactive Figure 3)

Appendix II: Fourteen Research Projects Will Inform Design Decisions on Key Efforts (Text for Interactive Figure 3)

Objectives in Internet development Develop requirements for the Internet response option and coordinate the relationship between different response modes (i.e., Internet, paper, telephone)

Appendix III: Census Mapping and Address Contracts

Appendix III: Census Mapping and Address Contracts Description of services Develop, implement, and support a system which allows the Census Bureau (Bureau) to update its Master Address File (MAF) with TIGER® (Topologically Integrated Geographic Encoding and Referencing) data, the U.S. mapping data, as the data is received.

Analyze mapping data as it is provided by state and local governments and other agencies, and upload changes into TIGER®.

Customize and maintain software so that over 40,000 state, local, and tribal governments can update and verify the Bureau’s address and mapping databases.

Supply digital map files and digital aerial imagery to maintain and update the MAF/ TIGER® database.

Customize, install, and provide technical support for software to process and edit files to outline geographic areas, and support mapping applications, including the use of handheld computers to collect and edit data.

Provide software to match addresses from surveys and other agencies to the MAF/ TIGER® database.

Develop and maintain software which automatically places text on Bureau maps so all map features are clearly identified.

Update and consolidate the infrastructure, servers and database, of the MAF/ TIGER® system.

Total

Provide services for the modernization of the Bureau’s MAF/TIGER® system, including expansion, design, programming, operation and maintenance of the infrastructure supporting the MAF/ TIGER® system.

Cost of the most recent purchase.

Appendix IV: Comments from the Department of Commerce

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Other key contributors to this report include Lisa Pearson, Assistant Director; David Bobruff; Robert Gebhart; Richard Hung; Kirsten B. Lauber; Andrea Levine; and Timothy Wexler.

Related GAO Products

2020 Census: Sustaining Current Reform Efforts Will Be Key to a More Cost-Effective Enumeration. GAO-12-905T. Washington, D.C.: July 18, 2012. 2020 Census: Additional Steps Are Needed to Build on Early Planning. GAO-12-626. Washington, D.C.: May 17, 2012.

Decennial Census: Additional Actions Could Improve the Census Bureau’s Ability to Control Costs for the 2020 Census. GAO-12-80. Washington, D.C.: January 24, 2012. 2010 Census: Preliminary Lessons Learned Highlight the Need for Fundamental Reforms. GAO-11-496T. Washington, D.C.: April 6, 2011. 2010 Census: Data Collection Operations Were Generally Completed as Planned, but Long-standing Challenges Suggest Need for Fundamental Reforms. GAO-11-193. Washington, D.C.: December 14, 2010.

GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs. GAO-09-3SP. Washington D.C.: March 2, 2009. 2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census. GAO-08-554. Washington, D.C.: June 16, 2008.

Information Technology: Significant Problems of Critical Automation Program Contribute to Risks Facing 2010 Census. GAO-08-550T. Washington, D.C.: March 5, 2008. 2010 Census: Cost and Design Issues Need to Be Addressed Soon. GAO-04-37. Washington, D.C.: January 15, 2004.

High-Risk Series: Quick Reference Guide. GAO/HR-97-2. Washington, D.C.: February 1997.