2020 Census: Key Challenges Need to Be Addressed to Successfully Enable Internet Response
Released: 2015-03-09T00:00:00

Letter

Background

The Bureau’s mission is to collect and provide comprehensive data about the nation’s people and economy. Its core activities include conducting decennial, economic, and government censuses; conducting demographic and economic surveys; managing international demographic and socioeconomic databases; providing technical advisory services to foreign governments; and performing other activities such as producing official population estimates and projections.

The Bureau is part of the Department of Commerce and is in the department’s Economics and Statistics Administration, led by the Under Secretary for Economic Affairs. The Bureau is headed by a Director and is organized into directorates corresponding to key programmatic and administrative functions, as depicted in figure 1. Two of these directorates are responsible for the projects that support the Internet response option for the 2020 census: (1) Associate Director for 2020 Census and (2) Associate Director for Information Technology and Chief Information Officer.

2020 Decennial Census Research and Testing

Bureau officials have established a goal of conducting a high-quality 2020 census at a lower cost than the 2010 census. In order to achieve cost savings and quality targets for the 2020 Decennial Census, the Bureau must make fundamental changes to the design, implementation, and management of the decennial census. Accordingly, the Bureau identified four key design areas that are intended to enable the Bureau to achieve its goal: (1) reengineering address canvassing to eliminate a nationwide field address canvassing effort in 2019, (2) utilizing existing administrative records to reduce non-response follow-up workload,field operations to more efficiently and effectively manage the 2020 census fieldwork, and (4) optimizing self-response to generate the largest possible self-response rate, thus eliminating the need to follow-up with those households. The optimizing self-response design area includes examining contact strategies and self-response modes to identify which are best for each demographic, geographic, and language-based group. This includes giving households the option of responding to the census through an Internet-based survey—referred to as the Internet response option. Other self-response modes include mailing paper questionnaires to households and asking them to complete and mail back the forms, as well as providing a phone number for households to call and provide their responses via telephone.

In fiscal year 2012, the Bureau began research and testing alternatives in the design areas. Key research and testing is expected to continue until the end of fiscal year 2015 (the end of September 2015), which is when the Bureau plans to decide how it will design the operations for the 2020 Decennial Census—referred to as the preliminary design decision—and will produce an updated total life-cycle cost estimate. The Bureau completed a major field test in 2014, and as of October 2014, was in the process of finalizing the design of an Optimizing Self-Response test that is planned for the spring of 2015, and was beginning to design a National Content Test, which is planned for the fall of 2015. These tests are intended, in part, to test components of the Internet response option. Following the preliminary design decision, the Bureau plans to conduct additional research and testing and further refine its design decisions through 2018. By September 2018, the Bureau plans to have fully implemented the 2020 design so that it can begin operational readiness testing. Figure 2 provides the timeline for planned 2020 Decennial Census research and testing.

The Bureau is planning two additional field tests in 2015 that are not related to the Internet response option—Address Validation Test and 2015 Census Test.

Bureau’s Past Efforts to Adopt an Internet Response Option

The 2020 census is intended to be the first time that the Bureau implements an Internet response option on a wide scale for the decennial census. During the 2000 Decennial Census, the Bureau tested the use of an Internet response option. The Internet option had few respondents (approximately 63,000 households representing about 169,000 persons), in part because the Bureau did not advertise this response option. However, the test demonstrated that the Internet response option worked operationally.

The Bureau considered building on the 2000 experience for the 2010 Decennial Census by including the Internet response option in the scope of the Decennial Response Integration System contract that was awarded in October 2005. This contract included requirements to provide functionality for the public to respond to the 2010 census via paper, telephone, and the Internet. In July 2006, the Bureau decided not to include the Internet response option in the design for the 2010 census and eliminated it from the scope of the contract. This was because testing indicated that the Internet response option did not increase the overall response rate enough to justify the costs of building and securing this option, which the Bureau had underestimated. The Bureau also had continued concerns about the ability to sufficiently secure respondents’ data.

Additionally, in January 2013, the Bureau implemented an Internet response option for its American Community Survey (ACS), which is another household survey conducted by the 2020 Census Directorate on a smaller scale than the decennial census. The ACS continuously collects data on social, demographic, economic, and housing characteristics that help determine how federal funds are allocated to states and localities and provide information to communities to aid in planning investments and services. The Bureau collects ACS data on a monthly sample of households and aggregates the results into 1-, 3-, and 5-year estimates, depending on the population size of the area. Bureau officials stated that they intend to build on the IT infrastructure and lessons learned from the ACS in order to implement the Internet response option for the 2020 census.

Bureau’s Plans for an Internet Response Option for the 2020 Census

The Bureau determined that the Internet response option offers several benefits for the 2020 census, including the added convenience for households in an increasingly Internet-enabled population to respond to the survey; better quality data, which could reduce the amount of follow- up that is needed for surveys with incomplete or inconsistent data; and less printing, postage, and processing of paper questionnaires. Bureau officials have also stated that the Internet response option would provide opportunities to administer the survey in multiple languages more easily than with paper questionnaires.

The Bureau’s efforts to deliver an Internet response option for the 2020 census include several key components: Internet response application: Design and develop an online survey instrument that allows respondents to enter and submit their information to the Bureau. To enhance Internet response participation, the Bureau is researching an option for respondents to submit their information without having the traditional Bureau-issued ID number (referred to as “non-ID processing”). To achieve this, the Bureau will need to develop the capability to validate respondent-provided addresses either automatically against its master address file in real- time or in batches offline.

IT infrastructure: Develop and acquire the IT infrastructure (e.g., servers, hardware, and network capacity) needed to support the data processing, storage, and transactions from Internet responses. The Bureau has stated that it plans to use cloud computing solutions, which is a means for establishing on-demand access to shared and scalable pools of computing resources, to help support the large volume of data processing, storage, and transactions needed for Internet responses.

Communication and outreach: Planning for how the Bureau can make use of partnership efforts, advertising, and outreach methods like social media to maximize the use of an Internet response option and motivate households to self-respond via Internet.

Additionally, the Bureau is exploring other potential design features related to Internet response during the research and testing phase, which may or may not be included in the final design of the 2020 census. For example, the Bureau is testing whether there is value in asking households to pre-register via a separate online portal prior to the census with a preference on how they would like to receive reminders (e.g., e- mail or text message) from the Bureau on completing the survey. The Bureau is also testing whether there is value in using e-mail addresses purchased from commercial sources to contact respondents and ask them to complete the survey. According to the Bureau, it will also need to conduct research to help address privacy and information security concerns the public may have regarding its use of the Internet to contact respondents and collect their personal information, as well as to determine which non-English languages will be offered with the Internet response option.

The Bureau currently has 16 projects, planned or under way, that are related to the 2020 census Internet response option.being managed by two different directorates—the 2020 Census These projects are Directorate is responsible for many of the research and testing projects, while the IT Directorate is responsible for developing and acquiring the IT systems and infrastructure needed to support the 2020 census. The IT Directorate’s projects fall under an enterprise-wide program initiated in fiscal year 2015 called the Census Enterprise Data Collection and Processing (CEDCAP) program. CEDCAP is intended to integrate disparate, program-specific survey data collection and processing systems that the Bureau uses to conduct its many surveys. Table 1 summarizes the Bureau’s projects related to the Internet response option.

Enumerating Historically Hard-to-Count Populations

While the Bureau plans to offer multiple self-response modes in the 2020 census, including the paper and telephone options that have been offered in prior censuses, the effect of an Internet response on population groups that are already typically undercounted or missed during the census is unclear. The Bureau has identified segments of the population that are more difficult to enumerate based on prior censuses, such as minorities, renters, children, low-income households, and low-education households. To help identify hard-to-count populations, the Bureau segmented the population into eight unique groups based on census demographic, socioeconomic, housing, and mail response data. Each group contained housing units with similar characteristics such as housing vacancy, tenure, marital status, education, poverty, and unemployment level. The groups that targeted the hard-to-count population included single unattached mobile renter, economically disadvantaged homeowner, economically disadvantaged renter, ethnic enclave homeowner, and ethnic enclave renter.

To help reduce the undercount for the 2010 census, the Bureau embarked on a number of outreach and enumeration activities aimed at getting the hard-to-count populations to participate in the census. For example, the Bureau used actual participation data from the 2000 census as well as market and attitudinal research that identified different mindsets people have about the census, such as those who are less likely to participate because they doubt the census provides tangible benefits or are concerned that the census is an invasion of privacy and that the information collected will be misused. The Bureau used this information to tailor its paid media efforts, such as buying additional media in areas with low participation rates. with a population that is growing larger, As we have previously reported,more diverse, and more reluctant to participate, a complete and accurate census has become an increasingly daunting task. While the Bureau invested more resources in reaching out to and enumerating the hard-to- count population groups in the 2010 census, it achieved the same overall participation rate as in the 2000 census. This trend is likely to continue as the nation’s population gets larger, more diverse, and more difficult to count.

Internet Use in the United States

According to data from the Pew Research Center, Internet use has increased significantly since 2000, from 50 percent to 87 percent of adults in the United States (see fig. 3).

Also according to data from the Pew Research Center, as of 2014, Internet use varied among different demographic groups, such as race/ethnicity, age group, education level, and household income (see table 2).

Another population group to consider when introducing the Internet response option is adults age 65 or older. Although they have historically been a high self-response group using the paper-based method, Internet use among this group was only 57 percent, compared to 97 percent among adults under age 30.

In addition, ownership of mobile computing devices such as smartphones and tablets has increased significantly in the past several years, as has the use of cell phones to access the Internet, send and receive e-mails, download software applications, and send and receive text messages (see figs. 4 and 5).

Prior Related GAO Reports

Our prior work has identified the importance of having sound management processes in place to help the Bureau as it manages the multimillion dollar investments needed for its decennial census. For the last decennial, we issued multiple reports and testimonies from 2005 through 2010 on weaknesses in the Bureau’s acquisition, management, and testing of key 2010 census IT systems. Since the 2010 census, we have issued additional reports and testimonies on weaknesses in the Bureau’s efforts to institutionalize IT and program management controls for the 2020 census. Relevant reports include the following: In June 2008, we reported that the 2010 census life-cycle cost estimate was not reliable and the Bureau had insufficient policies and procedures and inadequately trained staff for high-quality cost estimation.was not reliable, annual budget requests based on that estimate were not fully informed. We recommended that the Bureau, among other things, thoroughly document and update the estimate and for future estimates, establish policies and procedures for cost estimation. The Bureau has partially implemented these recommendations.

We also stated that because the life-cycle cost estimate In January 2012, we reported that the Bureau was taking steps to strengthen its life-cycle cost estimates but had not yet established We also guidance for developing the 2020 life-cycle cost estimate. reported that the Bureau had not identified decision points at which executives would review progress and decide whether the Bureau is prepared to move from one project phase to another. We recommended that the Bureau, among other things, identify decision points and finalize guidance for the 2020 life-cycle cost estimate. The Bureau has not yet implemented these recommendations.

In May 2012, we reported that the Bureau was taking steps consistent with leading practices for long-term project planning for the 2020 census, such as creating a high-level schedule of program management activities. However, the Bureau’s schedule did not include milestones or deadlines for key decisions needed to support transition between planning phases, which could result in later planning activity not being based on evidence from early research and testing. We also reported that the Bureau was taking steps in strategic workforce planning, but it had not yet identified the goals that should guide workforce planning or how to monitor, report, and evaluate its progress toward achieving them. Accordingly, we made several recommendations aimed at addressing these issues. The Bureau has taken steps to implement these recommendations, but has not fully implemented them.

In September 2012, we reported that the Bureau had drafted a new investment management plan, system development methodology, and requirements development and management processes to improve its ability to manage IT investments and systems development.However, additional work was needed to ensure that these processes were effective and successfully implemented across the Bureau, such as finalizing plans for implementing its new investment management and systems development processes across the Bureau. We also reported that the Bureau had not fully put in place key practices for effective IT workforce planning, including conducting an IT skills assessment and gap analysis and establishing a process for directorates to coordinate on IT workforce planning. To address these weaknesses, we made a number of recommendations to the Bureau. The Bureau has taken steps to address the recommendations, such as finalizing its investment management process, conducting an enterprise-wide IT competency assessment and gap analysis, and developing action plans to address the identified gaps.

In November 2012, we evaluated the Bureau’s efforts to improve the cost-effectiveness of enumeration in the 2020 census, paying particular attention to three key efforts, one of which included leveraging the Internet to increase self-response. We reported weaknesses in developing mitigation or contingency plans for several project risks, including those related to tight time frames and accurate cost information; weaknesses in developing cost estimates for research and testing projects; incomplete project plans; and incomplete performance metric documentation. We made a number of recommendations to address these weaknesses, and the Bureau has partially implemented them.

In January 2013, we reported on the Bureau’s implementation of information security controls to protect the confidentiality, integrity, and availability of the information and systems that support its mission. We concluded that the Bureau had a number of weaknesses in controls intended to limit access to its systems and information, as well as those related to managing system configurations and unplanned events. We attributed these weaknesses to the fact that the Bureau had not fully implemented a comprehensive information security program, and made 13 public recommendations and over 100 other recommendations that were for limited distribution to address these deficiencies. The Bureau has partially implemented the recommendations.

In September 2013, we testified on progress the Bureau had made in its efforts to contain enumeration costs, including its efforts to strengthen IT management and security practices. We noted that the Bureau was exploring technology options for 2020 census operations that collectively represent a dramatic leap from 2010, including the “bring your own device” model for field data collection and the Internet response option. We stressed the importance of the Bureau strengthening its ability to manage its IT investments as well as its practices for securing the information it collects and disseminates.

In November 2013, we reported that the Bureau was not producing reliable schedules for two efforts related to the 2020 census: (1) building a master address file and (2) 2020 census research and testing. We reported, for example, that the Bureau did not include all activities and required resources in its schedules, or logically link a number of the activities in a sequence. We recommended that the Bureau take actions to improve the reliability of its schedules, including ensuring that all relevant activities are included in the schedules, complete scheduling logic is in place, and a quantitative risk assessment is conducted. We also recommended that the Bureau undertake a robust workforce planning effort to identify and address gaps in scheduling skills for staff that work on schedules. The Bureau has taken steps to implement these recommendations, but has not fully implemented them.

In April 2014, we reported on the Bureau’s IT-related efforts for the 2020 census. We found that several of the IT-related projects lacked schedules and plans, and that it was uncertain whether the work would be completed in time to inform the operational design decision for the 2020 census, planned in September 2015. We also reported that the Bureau had not prioritized its projects to determine which were the most important to complete before the decision. We recommended that the Bureau prioritize the research and testing that it needed to complete in order to support the operational design decision, and ensure that project plans and schedules were developed consistent with the new prioritized approach. The Bureau has made significant progress toward addressing the recommendations. For example, it developed a document to guide the Bureau’s path to making the preliminary design decision in September 2015, and Bureau officials identified projects that could be deferred to after the preliminary design decision.

Census Bureau Has Taken Preliminary Steps to Examine the Impact of the Internet Option on Hard-to- Count Populations and Has Further Evaluation Plans

The Bureau has taken preliminary steps to identify demographic groups likely to use the Internet response option in the 2020 census and how they compare to historically hard-to-count populations by examining existing ACS studies in this area and applying the lessons learned to the decennial census. For example, the Bureau issued a report in May 2014 on the effects of adding an Internet response option to the ACS among different segments of the population. The study found that the total self- response rate was statistically significantly higher after introducing the Internet response option and the Internet response rate was about 55 percent. The study also found that, while none of the groups had dramatically low rates of Internet participation, the addition of an Internet response option had a positive effect on certain groups (e.g., advantaged homeowner, single unattached mobile) and a negative effect on others (e.g., ethnic enclave homeowner, economically disadvantaged homeowner). The study suggested that, by pushing households to respond by Internet (i.e., mailing an Internet response invitation and only later mailing a paper questionnaire if the household did not first respond by Internet), the Bureau may be discouraging some households from self- responding at all and that this may be happening in certain hard-to-count groups. The 2020 Census Directorate officials told us that they are incorporating the results of the ACS studies into future decennial census field tests. For example, officials said that they plan to include both the Internet response and paper questionnaire option in the first mailing for hard-to-count population groups in the 2015 National Content Test.

In addition to using the ACS studies, the Bureau plans to conduct future tests and research specific to the 2020 Decennial Census, which is expected to produce more information in this area. For example, the Bureau is planning two tests in 2015 that are expected to provide data on Internet response rates among different demographic groups, including historically hard-to-count populations. Specifically, the April 2015 Optimizing Self-Response Test site location was selected based on, among other things, lower than average 2010 Decennial Census response rates and ACS Internet response rates, Internet penetration at least as high as the national average, and the ability to segment by hard- to-count populations. Additionally, 2020 Census Directorate officials stated that the Bureau plans to begin coverage improvement research with the September 2015 National Content Test, which is expected to provide a nationally representative demographic sampling and national Internet self-response rates. Bureau officials stated that additional research in this area in 2016 and beyond is yet to be determined.

Preliminary 2020 Census Internet Response Option Cost Estimate Was Not Reliable

GAO’s Cost Estimating and Assessment Guide identifies a number of best practices that are the basis of effective program cost estimating and should result in reliable and valid cost estimates that management can use for making informed decisions. Specifically, a reliable cost estimate should be comprehensive (costs are neither omitted nor double counted); well-documented (the estimate is thoroughly documented, including source data and significance, clearly detailed calculations and results, and explanations for choosing a particular method or reference); accurate (the estimate is unbiased, not overly conservative or overly optimistic, and based on an assessment of most likely costs); and credible (the estimate discusses any limitations of the analysis from uncertainty or biases surrounding data or assumptions).

A rough-order-of-magnitude estimate is a less-rigorous cost estimate developed from limited data and in a short time. of the estimate in 2014, with a resulting total of approximately $12.7 billion. This included, among other things, about $73 million for the Internet response option. However, the estimate for the Internet response option did not meet the characteristics of a reliable estimate. Specifically:

The Internet response option cost estimate was not comprehensive. This is because the Internet response option cost estimate included costs from 2010 to 2020 and provided a subset of assumptions for researching, testing, and deploying an Internet response option. While the estimate was structured around these high-level cost elements, these elements were not defined, and therefore it is not clear whether all costs associated with the Internet response option were included. Bureau officials stated that the estimate was not developed based on a work breakdown structure with defined elements because the 2020 census program was not mature enough to have such a structure at the time the initial estimate was developed. They stated that the estimate will be updated to reflect the program’s work breakdown structure once the preliminary design decision is made in September 2015. However, a work breakdown structure should have been initially set up when the program was established and successively updated with more detail over time as more information became known about the program.

The Internet response estimate was not well-documented. We found that the documentation included selected source data, assumptions, and calculations, but it did not provide a clear and traceable view of the estimated costs and all the assumptions that were included in the Internet response option cost estimate. Additionally, the Internet response option cost estimate was developed in a separate file and rolled up into a single line item labeled “Internet” within the 2020 census life-cycle cost estimate, without including source information to show where the estimate originally came from or details of what the estimate was based on. This was problematic because Bureau officials were initially unable to locate the relevant files and were unable to provide us with an explanation for how the Internet response option estimates were developed. After several weeks, officials finally located the separate file that contained the high-level Internet response cost elements.

The estimate was not accurate. Specifically, while accurate estimates are to be based on an assessment of most likely costs and historical data for similar programs, the Internet response option cost estimate was based on subject matter expert opinion and analogous data from the ACS, which is similar in function to the decennial census but not in scale. The scale of the decennial census is significantly larger than ACS, and will require IT systems and infrastructure to be sized accordingly. However, it is not clear in the documentation how the Bureau applied the ACS cost data in estimating 2020 census costs, nor could Bureau officials explain this. For example, 2020 census IT infrastructure costs were estimated to be higher than ACS IT infrastructure costs, but Bureau officials could not explain what assumptions and methodology were used to account for the much larger scale of IT infrastructure needed for the 2020 census compared to the ACS. Without insight into how historical data were used to estimate future costs, we cannot determine whether the cost models are accurate.

Additionally, in 2014, the Bureau revised selected components of the 2020 census life-cycle cost estimate to inform the fiscal year 2015 budget request, but did not revise the Internet response option cost estimate as part of this update even though there had been significant changes to the program since 2011. For example, projects related to the Internet response option have evolved, and thus not all projects are represented in the 2011 $73 million estimate, such as non-ID processing. Also, the Bureau determined in 2013 that its existing IT infrastructure was not sufficient to support the scale-up needed for Internet response data processing and storage in the 2020 census, and is planning to address this as part of the enterprisewide IT data collection and processing program, known as CEDCAP. According to Bureau officials, they are in the process of developing estimated costs for relevant CEDCAP projects, and these will be incorporated into the Internet response option cost estimate. Additionally, the Internet response option cost estimate was not updated with more current data from the analogous ACS program, as a result of its implementation of an Internet response option in January 2013.

The estimate was not credible. Sensitivity and risk and uncertainty analyses were conducted on the 2020 census life-cycle cost estimate, but they were not completed properly. For example, a risk and uncertainty analysis was only applied to fiscal years 2018 to 2020 rather than on the total cost estimate. According to Bureau officials, the Bureau focused on fiscal years 2018 to 2020 because 80 percent of all costs for the decennial census are expected to be incurred during this period. However, accounting for all risks throughout the life of the program is necessary to adequately capture the uncertainty associated with a program’s estimate. The Bureau also did not perform risk and uncertainty analyses on the Internet response option cost estimate to reflect the uncertainty introduced by the significant program changes discussed previously, such as the uncertainty associated with the initial estimate for IT infrastructure costs ($11.4 million).

The 2020 Census Directorate officials have recognized that the estimate for the Internet response option had weaknesses and stated that the preliminary 2020 census rough-order-of-magnitude life-cycle cost estimate was not developed to be an official cost estimate, but rather a “top-down” approach to estimating costs and potential savings focused around the major design categories. Nevertheless, the Bureau considered the estimate to be a budget-quality estimate and used it to inform the fiscal year 2015 budget request. Bureau officials stated that the preliminary 2020 census life-cycle cost estimate (including the Internet response option estimate) will be updated once the preliminary design decision is made in September 2015.

Additionally, Bureau officials identified several ongoing efforts that the Bureau has planned to improve its institutional cost estimating practices. Specifically, Bureau officials stated that the Bureau established a new centralized cost estimating office in August 2013 that is expected to, among other things, issue guidance and policies for developing reliable cost estimates and establish a standardized work breakdown structure for censuses and surveys by the third quarter of fiscal year 2015, create a cost estimating certificate program by the fourth quarter of fiscal year 2015, and conduct independent cost estimates, as needed. Bureau officials also stated that they have certified four staff in cost estimation techniques, moved the 2020 census estimate to a more robust cost estimating tool, and planned actions to address competency gaps in cost estimating. While these are important steps to institutionalizing good cost estimating practices, until the Bureau updates the Internet response option cost estimate to ensure that it conforms with best practices, the estimate will continue to be unreliable.

In addition to the $12.7 billion 2020 census rough-order-of-magnitude life- cycle cost estimate, the Bureau estimated potential cost savings for the major design options, such as optimizing self-response, which includes the Internet response option. The Bureau estimated that implementing all design options could result in potential savings of up to $5 billion. Potential cost savings were estimated by adjusting inputs for different 2020 census design scenarios and calculating the difference between those estimated 2020 census costs and the cost of repeating the 2010 census design. For example, to estimate potential cost savings for the optimizing self-response design option, the Bureau reduced the amount of printing, postage, and infrastructure needed for processing paper questionnaires based on the percent of responses expected via Internet. The Bureau estimated that the optimizing self-response option could result in potential savings of about $550 million to $1 billion. However, the unreliability of estimated costs for the Internet response option discussed above casts doubt on the reliability of estimated cost savings associated with the optimizing self-response design option.

Census Bureau Faces Scheduling, Task, and Capability Challenges in Planning for an Internet Response Option

While efforts are under way to deliver an Internet response option for the 2020 census, significant challenges remain. We identified four major challenges the Bureau faces in implementing an Internet response option for the 2020 census: project schedules associated with the Internet response option are not fully integrated; key questions may not be answered in time for the preliminary design decision; high-level time frames for cloud computing decisions and implementation for the 2020 census have not been determined; and gaps in IT skill sets continue to exist. According to 2020 Census Directorate and IT Directorate officials, the Bureau has recently begun to develop methodologies for answering key research questions, using a contractor to assist in assessing cloud computing technologies for the 2020 census, and implementing actions to close critical competency gaps. However, without established methodologies and time frames for completing these efforts, the Bureau will be limited in its ability to make informed design decisions, which could impact implementation of the 2020 Census.

Internet Response Option Project Schedules Have Not Been Fully Integrated

According to best practices identified in GAO’s Schedule Assessment Guide, a detailed schedule should be horizontally traceable, meaning that it should link products and outcomes associated with other sequenced activities. These links are commonly referred to as “hand-offs” and serve to verify that activities are arranged in the right order for achieving aggregated products or outcomes. Such mapping or alignment of levels enables different groups to work to the same master schedule.

As previously mentioned, the Bureau has several different projects related to the Internet response option, planned or currently under way (see table 1 for the complete list of projects). Certain projects, such as IT Infrastructure 2020 Decennial Scale-Up and E-correspondence, were initiated in October 2014 and thus did not yet have detailed schedules. For selected ongoing projects, such as the Optimizing Self Response, Non-ID Processing, and 2014 Census Test projects, the Bureau had integrated the schedules into the 2020 census program’s integrated master schedule. However, for other projects, which have been under way for 5 months, the Bureau has not yet integrated their schedules with the 2020 census program’s integrated master schedule, so the dependencies among the different projects (e.g., hand-offs between teams) are not linked together. For example:

The Bureau established a schedule for developing the Centurion Internet response option application for the 2015 tests, which included activities starting as early as July 2014. However, as of November 2014, this schedule was not yet linked to the overarching 2020 census program schedule. According to Bureau officials, the Centurion schedule was recently developed and is eventually to be integrated with the 2020 census program schedule. However, Bureau officials did not know when this would be complete—despite ongoing activities with the Centurion project and the 2020 census program.

Activities for the 2015 Optimizing Self-Response Test and 2015 National Content Test were not included in the 2020 census program schedule. According to 2020 Census Directorate officials, detailed schedules for the 2015 tests were still under development and will be incorporated into the 2020 census program schedule once they are complete. However, as of November 2014, Bureau officials did not have a time frame for when this would be done, and preparations for the 2015 Optimizing Self-Response Test are well under way, with the test planned to begin in less than 5 months.

As stated earlier, we have previously recommended that the Bureau take actions to improve the reliability of its schedules, including steps to ensure that all relevant activities and associated resources are included in the schedules and complete scheduling logic is in place. Bureau officials have recognized the need to fully integrate project schedules with the 2020 census program’s integrated master schedule, but stated that they are currently working to identify required resources for activities in the 2020 census program integrated master schedule. They stated that, in the meantime, the project teams are conducting work according to their project schedules, which are maintained separately. However, until the Bureau fully implements our recommendation to integrate dependent activities in the program schedule, the Bureau is unable to generate an accurate “critical path”—the sequence of steps needed to achieve the end goal that, if they slip, could negatively affect the overall project completion date. This limits the Bureau’s ability to use the 2020 census program schedule to estimate reliable dates, determine the impact of any schedule slippages on major milestones, and identify opportunities for increased efficiency.

Key Questions May Not Be Answered In Time for Preliminary Design Decision

In September 2014, the Bureau released a planning document titled The Path to the 2020 Census Design Decision, which identified inputs, such as research questions, design components, and testing, needed to inform the preliminary design decision planned for September 2015. This included the following key research questions related to the Internet response option that were determined to be critical inputs into the preliminary design decision:

What are the best methods for communicating the importance of responding to the 2020 census, including methods for promoting the use of Internet response?

What percentage of the population has access to the Internet?

What is the estimated Internet self-response rate?

What IT infrastructure for security and scalability is necessary to support the Internet as the primary mechanism for self-response?

Is there value in asking households to pre-register online for the 2020 census?

Is it necessary to provide households with an identification code to respond via the Internet?

While the Bureau has documented in relevant project plans its methods for answering the questions related to communication strategies, online pre-registration, and processing household responses without identification codes, it has not yet established how it will determine answers to the questions related to self-response rate and IT infrastructure for the preliminary design decision. As stated in 2020 census program management guidance, detailed project plans and specific research questions, as well as study plans describing the management and technical approach should be established for research and testing projects.

According to Bureau officials in November 2014, they began to establish a new project team that was intended to be responsible for estimating the Internet self-response rate. However, Bureau officials did not have a time frame for when a project plan or study plan would be developed that would document the methodology for how the Internet self-response rate will be estimated to inform the preliminary design decision.

Additionally, Bureau officials told us that they had established a new project—the 2020 Census Architecture and IT Roadmap—in June 2014 that is intended to help determine the IT infrastructure needed to support the Internet response option. This project is to deliver initial 2020 census architecture and IT roadmap documents by preliminary design. However, the Bureau had not yet established a project plan or study plan, and had not developed the methodology—as part of this project or another—for determining scalability and security infrastructure needs for the Internet response by September 2015.

As previously stated, the Bureau is committed to producing the preliminary design decision and developing the life-cycle cost estimate for the 2020 census by September 2015. With about 8 months remaining until the preliminary design decision is to be made, and major tests already designed or completed (i.e., the 2014 Census test and the April 2015 optimizing self-response test), the Bureau has limited time to determine how these critical questions will be answered. Accordingly, until the Bureau establishes and implements clear plans for answering the Internet response rate and IT infrastructure questions, the Bureau will have limited information for beginning its development and implementation of systems and infrastructure. Also, as previously discussed, it is uncertain how complete or reliable the Bureau’s Internet response option and IT infrastructure cost estimates will be to inform key design decisions by this time.

High-Level Time Frames for Cloud Computing Decisions and Implementation Have Not Been Established

The OMB Federal Cloud Computing Strategy recognizes the importance of organizations planning effectively when selecting services to move to a cloud environment. The strategy recommends that organizations create roadmaps for cloud deployment and migration in order to prioritize services that minimize risks to the organization. NIST also recognizes that moving to a cloud environment is a business decision, where the organization’s business case should consider relevant factors such as transition and life-cycle costs and security and privacy requirements.

As previously mentioned, the Bureau’s existing IT infrastructure is not sufficient to process and store the large volume of anticipated electronic Internet survey responses for the 2020 census. Accordingly, Bureau officials stated that the Bureau plans to use a cloud environment to provide the needed capability. The Bureau has taken steps to research the use of the cloud to deliver the level of scalability needed to support the Internet response option for the 2020 census. Specifically, the Bureau has identified and prioritized 2020 census capabilities to potentially move into the cloud environment, such as Internet data collection. Additionally, the Bureau plans to conduct volume testing of the Centurion application in a cloud environment beginning in November 2014, in order to collect additional data to determine if it can meet the expected capacity in 2020. IT Directorate officials also said that they are working closely with federal agencies such as NIST and the Federal Aviation Administration to determine requirements for moving services to a cloud environment.

Although the Bureau expects cloud decisions to lay the groundwork for accommodating the volume of users expected for the 2017 Economic Census and 2020 Decennial Census, the Bureau has not established time frames to determine when key cloud computing decisions need to be made and actions need to be taken for the 2020 census, such as selecting, testing, and deploying a cloud environment that meets its needs for scalability, budget, security, and privacy protection of personally identifiable information. IT Directorate officials stated that by the third quarter of fiscal year 2015 the Bureau will develop a schedule for the IT Infrastructure 2020 Decennial Scale-Up project, which will include high- level time frames for a subset of relevant activities, such as conducting an analysis of alternatives to determine whether a cloud solution is the best alternative for addressing the scale-up. Additionally, IT Directorate officials told us that the Bureau plans to develop a strategy in 2016 to outline the overall approach for acquiring cloud solutions for the 2020 census, among other things.

The Bureau is planning to begin systems readiness testing in October 2018, which is when the Bureau has determined that the systems and processes for the 2020 census must be developed and ready for end-to- end system testing (approximately 3.7 years away). As we have previously reported, the federal government procurement process can require a significant amount of time, and agencies can face challenges in implementing cloud computing, such as meeting federal security requirements, obtaining guidance, acquiring knowledge and expertise to implement cloud services, certifying and accrediting vendors, and ensuring data portability and interoperability. Without established high- level time frames, Bureau officials will not know whether there is enough time to effectively implement a cloud environment. The Bureau has not yet established such time frames for the 2020 cloud implementation approach due to a lack of internal cloud computing expertise.

In an effort to offset this lack of internal expertise, the Bureau issued a task order in September 2014 to a contractor for assistance in assessing, analyzing, and recommending cloud computing technologies for the 2020 census. While this assistance may be helpful, until the Bureau, at a minimum, documents time frames for selecting, testing, and deploying its cloud environment, it will not know whether there is enough time to successfully implement a cloud solution that meets scalability, budget, security, and privacy protection of personally identifiable information needs for the 2020 census.

Gaps in IT Skills Continue to Exist

As our prior work and leading guidance recognize, having the right knowledge and skills is critical to the success of a program. In response to prior GAO recommendations on developing strategic workforce planning capabilities, the Bureau completed an enterprise-wide competency assessment in 2013 and identified several mission-critical gaps in technical competencies within the IT and 2020 census workforce that would be needed to support the Internet response option. In August 2014, the Bureau completed action plans and targets aimed at addressing the IT and 2020 census workforce competency gaps. The gaps related to the Internet response option and planned actions to address them are summarized in table 3.

Moving forward, the Bureau plans to monitor quarterly status reports on the implementation of these actions and closing competency gaps, beginning around December 2014. Fully implementing actions to close these competency gaps will be critical to ensuring the Bureau has the skills it needs to effectively develop and implement the Internet response option.

Conclusions

The introduction of an Internet response option for the 2020 census has the potential to offer numerous benefits, such as added convenience for households in an increasingly Internet-enabled population to respond to the survey, better quality data which can result in less follow-up work, and reduced costs associated with processing paper questionnaires. Identifying ways to increase participation via the Internet will help increase the benefits of this response mechanism.

While the Bureau’s initial estimate of approximately $73 million for the Internet response option was included in the fiscal year 2015 budget request, this estimate lacks reliability, which in turn, calls into question the reliability of the potential cost savings estimate of about $550 million to $1 billion for the optimizing self-response design category (which includes the Internet response option). Although the Bureau has several important cost estimating improvements under way, until the Bureau updates the Internet response option cost estimate to ensure that it meets best practices, the estimate will continue to be unreliable.

Additionally, the Bureau’s ability to effectively manage the scheduling, task, and capability challenges it faces in planning for an Internet response option will be critical to the success of the 2020 census. Specifically, without fully implementing our prior recommendations on integrating dependent activities into the schedule, the Bureau continues to be unable to estimate reliable dates and make informed decisions based on an accurate critical path.

Further, with about 8 months remaining before key design decisions need to be made, the Bureau has not established the methodologies for answering two Internet response option research questions that were deemed critical for the preliminary design decision—estimating the Internet self-response rate and determining the IT infrastructure for security and scalability needed. Accordingly, until clear plans for answering these questions are established and implemented, the Bureau will have limited information for beginning its development and implementation of systems and infrastructure.

Additionally, even though the Bureau acknowledges the need to acquire a cloud solution to compensate for the fact that its existing IT infrastructure is not sufficient to support a wide-scale Internet response option for the 2020 census, it has not defined the high-level time frames for when key cloud computing decisions need to be made and implemented. As a result, with systems readiness testing for the complete 2020 census design planned to begin in October 2018, it is uncertain how the Bureau can ensure that there is sufficient time to accomplish this objective.

Finally, the Bureau faces continuing gaps in mission-critical technical competencies, including cloud computing, security integration and engineering, and requirements development. Fully implementing the Bureau’s planned actions to close these gaps will be critical to ensuring that it has the skills needed to effectively deliver the 2020 census Internet response option.

Recommendations for Executive Action

To ensure that the Bureau is better positioned to deliver an Internet response option for the 2020 Decennial Census, we are recommending that the Secretary of Commerce direct the Under Secretary for Economic Affairs to direct the Director of the Census Bureau to take the following three actions: ensure that the estimated costs associated with the Internet response option are updated to reflect significant changes in the program and to fully meet the characteristics of a reliable cost estimate; ensure that the methodologies for answering the Internet response rate and IT infrastructure research questions are determined and documented in existing or future project plans in time to inform key design decisions; and develop high-level time frames for selecting, testing, and deploying a cloud environment to guide the Bureau’s approach to enabling scalability for the 2020 census.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Department of Commerce, which are reprinted in appendix II. The department neither agreed nor disagreed with our recommendations, but provided comments that are discussed in detail below.

In its comments, the department stated that our conclusion regarding the unreliability of estimated costs and savings for the Internet response option seemed to be based on the fact that the Bureau did not build a bottom-up estimate using documented parameters from previous censuses or research. The department also stated that it needed to conduct critical tests and research to inform key cost parameters and that it planned to revise the estimated costs and savings as research and testing efforts are completed. The department added that it believed its top-down estimates were sufficient for the purpose of identifying major cost drivers for the decennial census.

However, as stated in the report, our assessment took into account the Bureau’s top-down approach and that it had developed a less-rigorous, “rough-order-of-magnitude” estimate; we therefore performed a high-level analysis of the cost estimate and methodology. We also described what is expected of a rough-order-of-magnitude cost estimate and recognized that such an estimate is developed from limited data and in a short time. For example, regarding the lack of a work breakdown structure, our report notes that this should be initially set up when a program is established and updated with more detail over time as more information becomes known about the program.

Nevertheless, even when taking these factors into consideration, the Bureau’s rough-order-of-magnitude estimate for the Internet response option was not reliable when assessed against best practices that are applicable to such estimates. For example, as we state in our report, the Internet response option estimate was not accurate, in part, because when the Bureau revised selected components of its estimate in 2014, it did not revise the Internet response option portion, despite significant changes to the program since the original 2011 estimate. These changes included the realization that additional IT infrastructure, beyond the Bureau’s existing infrastructure, would be needed in order to support the scale-up needed for Internet response data processing and storage. Given the weaknesses identified in our report, we continue to maintain that the Bureau cannot be assured that its top-down estimate was sufficient for the purpose of identifying major cost drivers for the decennial census.

The department also recognized the need to improve its capabilities and skill sets in cost estimation and identified several planned and ongoing actions, such as standardizing a work breakdown structure, establishing enterprise guidance and policies for cost reporting, and continuing to hire and train certified cost estimators. As stated in our report, while these are important steps to institutionalize good estimating practices, until the Bureau updates the Internet response option cost estimate to ensure that it conforms to best practices, the estimate will continue to be unreliable. Accordingly, we maintain that actions to address our first recommendation are still needed.

Regarding our conclusions on key challenges associated with delivering an Internet response option, the department stated that it believed it had developed project plans, methodologies, and time frames for making decisions related to IT infrastructure needs, including the use of cloud computing to support 2020 census requirements. The department further stated that the wording or descriptions of these activities in its documentation may use internal jargon that may make them difficult to find for those who do not regularly work on these activities, and that it would work to mitigate this issue.

We disagree that a misunderstanding of the plans, methodologies, and time frames exists. During our review, we took appropriate measures to ensure that we had collected and analyzed the most current, complete, and accurate information available, including meetings with knowledgeable officials. Furthermore, at the conclusion of our review, we met with key officials from the 2020 Census and IT Directorates and confirmed the facts of our analysis, including the fact that project plans and methodologies for making decisions related to the IT infrastructure needs for the 2020 census did not exist.

We disagree that a misunderstanding of the plans, methodologies, and time frames exists. During our review, we took appropriate measures to ensure that we had collected and analyzed the most current, complete, and accurate information available, including meetings with knowledgeable officials. Furthermore, at the conclusion of our review, we met with key officials from the 2020 Census and IT Directorates and confirmed the facts of our analysis, including the fact that project plans and methodologies for making decisions related to the IT infrastructure needs for the 2020 census did not exist.

The department also commented that the documentation produced for the 2020 census design decision would include key dates for making the decisions (e.g., the need for and likely extent of cloud computing use), and revised estimates of key cost parameters (e.g., Internet response rates). The department stated that it had established a plan and an implementation team to produce this documentation, referred to as the 2020 Census Concept of Operations. While we recognize that the design decision would help inform some of the time frames related to cloud computing, our concern is that high-level time frames for when key decisions need to be made do not yet exist. Therefore, the Bureau does not have assurance that there will be enough time to overcome potential challenges in acquiring and effectively implementing cloud computing services, such as meeting federal security requirements, certifying and accrediting vendors, and ensuring data portability and interoperability.

Further, while we support the Bureau’s efforts to ensure that the outcome of the design decision is appropriately documented, our report highlights the need for documenting the research methodologies that will be used to make those design decisions. As stated in our report, with about 8 months remaining until the design decision and with major tests already designed or completed, the Bureau has limited time to determine how critical research questions will be answered. Consequently, we maintain that actions are still needed to respond to our recommendations to develop key methodologies and high-level time frames for cloud computing decisions.

The department provided two additional comments related to the background section of our report:

First, the department noted concern that discussions of differential Internet availability among historically hard-to-count populations imply that this could lead to reduced coverage of these populations in the 2020 census and stated that the Bureau believed that this was unlikely to result in less coverage of these populations. The department explained that knowledge of the differentials would help the Bureau to effectively deploy outreach, partnership, social media, and other advertising investments in 2020. It further stated that the Internet would not be the only response option offered in 2020 because individuals without access to, or desire to use, the Internet will still be able to respond on paper, by phone, or in person with a census interviewer. We agree that the differential Internet availability among historically hard-to-count populations should not imply that the introduction of an Internet response option would lead to reduced coverage. To clarify that point, we modified the report to state that the Internet response option’s effect on historically hard-to-count populations is unclear. We intentionally did not draw conclusions about what the potential impact would be. The discussion of varying Internet availability is included to illustrate why the impact of an Internet response option on historically hard-to-count populations is a relevant topic to examine, and the report also notes that the Bureau plans to offer multiple self-response modes for the 2020 census, including paper and telephone. This information is introduced to provide context to the discussion of the Bureau’s completed steps and further plans to examine the impact of introducing an Internet response option on hard-to-count populations.

Second, the department stated that it was uncertain whether the background section discussing prior related GAO reports took into account that some of the Bureau’s planned actions were not yet scheduled to be completed and expressed concern that this discussion implied that the Bureau had plans to complete all actions by this point in time. However, the section in question is intended only to summarize relevant prior GAO reports and recommendations and provide a brief status update on progress made to address these recommendations. As it is background information, we did not refer to or imply time frames for when the recommendations should be implemented.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Commerce, the Under Secretary for Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe the Census Bureau’s (Bureau) efforts to identify demographic groups likely to use Internet response and how they compare to historically hard-to-count populations, (2) assess the reliability of estimated costs and savings for the Internet response option, and (3) determine key challenges associated with delivering an Internet response option for the 2020 census.

To describe the Bureau’s efforts to examine the impact of an Internet response option on the historically hard-to-count populations, we identified and reviewed studies conducted by the Bureau on demographic groups likely to use the Internet response option and how they compare to historically hard-to-count populations. We also conducted a literature review to identify other relevant studies on hard-to-count populations, Internet usage among demographic groups, and the impact of an Internet response survey on hard-to-count populations. We interviewed Bureau officials on their plans to further assess the impact of an Internet response option on historically hard-to-count populations.

To assess the reliability of estimated costs and savings for the Internet response option, we obtained and analyzed the Bureau’s documentation supporting the 2020 census rough-order-of-magnitude life-cycle cost estimate, the Internet response option portion of the cost estimate, and the potential cost savings estimate. We compared the cost estimating methodology and documentation against best practices for developing reliable cost estimates identified in GAO’s Cost Estimating and Assessment Guide.account that the Bureau developed the 2020 census cost estimate as a less-rigorous, “rough-order-of-magnitude” cost estimate, and therefore performed a high-level analysis of the Bureau’s cost estimate and methodology. We also interviewed Bureau officials to verify that our findings were accurate and discussed their approach to estimating costs and potential savings. Our report notes the instances where reliability impacts the quality of the cost estimate.

When applying the best practices, we took into To identify key challenges to delivering an Internet response option for the 2020 census, we identified relevant experts within the Bureau’s key advisory groups: the National Academy of Sciences, National Advisory Committee, and Census Scientific Advisory Committee. These advisory groups consist of academic and industry experts from various fields, including information technology and Internet survey design, and they meet with the Bureau regularly to provide feedback on various areas, including the 2020 census program. We interviewed relevant experts from these advisory groups to discuss their perspectives on key challenges the Bureau faces in implementing an Internet response option.

We also analyzed documentation on the Bureau’s projects related to the Internet response option, such as project plans, schedules, risk registers, and monthly status reports; program-level documentation, such as 2020 census and IT strategy documents; and workforce action plans. We also interviewed Bureau officials from the 2020 census program and the IT Directorate to obtain information on progress made on the Internet response option. We compared the Bureau’s efforts against relevant guidance and best practices, such as those identified in GAO’s Schedule Assessment Guide, prior GAO work, the National Institute of Standards and Technology’s guidance on cloud computing, the Office of Management and Budget’s Federal Cloud Computing Strategy, and the Software Engineering Institute’s Capability Maturity Model® Integration, to determine if there were any gaps.

We aggregated the results to identify the key challenges the Bureau faces in delivering an Internet response option for the 2020 census and presented the preliminary challenges that we identified to experts and Bureau officials to obtain their feedback on the challenges. We also observed Bureau activities related to the Internet response option, including 2020 census program management reviews, advisory group meetings, demonstration of the Internet response application and pre- registration portal used for the 2014 census test, and operations of the 2014 census site test.

We conducted this performance audit from May 2014 to February 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the U.S. Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Carol R. Cha at (202) 512-4456 or chac@gao.gov.

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Shannin G. O’Neill (Assistant Director), Jason Lee, Jennifer Leotta, Lee McCracken, Dana Pon, and Jeanne Sung.