2020 Census: Additional Actions Would Help the Bureau Realize Potential Administrative Records Cost Savings
Released: 2015-10-20T00:00:00

Letter

Background

For decades, the Bureau has been considering how it could use administrative records to help reduce the decennial’s costs. We, the Bureau, and others have observed that some of the information needed for the census has already been collected by other government agencies in the course of administering their programs. Thus, accessing that information and using it to administer and, in some cases, provide the data for completing census forms, has the potential to reduce the cost of the decennial census, such as its expensive acquisition of temporary workspace and equipment to support fieldwork. Moreover, some of the information collected through administrative records could be more accurate than information collected through traditional methods, such as when respondents provide the Bureau with incomplete information (or no information at all), and when the Bureau’s enumerators need to interview neighbors or other “proxy” respondents to collect needed information. Depending on the source of the information, administrative records can help the Bureau with such things as whether a housing unit is occupied or vacant, improving the accuracy of the Bureau’s address list, and providing demographic information on household members.

As far back as 1970, the Bureau has made limited use of administrative records to help enumerate group quarters, such as college dormitories and prisons. More recently, the Bureau conducted limited experiments during the 2000 Census and found there was potential to use administrative records to assist with follow-up and other operations, but that further research would be needed. Additionally, since 2000, the Bureau has used addresses provided by the U.S. Postal Service (USPS) Delivery Sequence File (DSF) as a starting point to update its Master Address File, a data file that contains a list of all known living quarters in the United States and Puerto Rico. For the 2010 Census, the Bureau used administrative records to help enumerate some group quarters and select cases for an operation that followed up on potentially inaccurate census responses.

In 2009, the Bureau’s earliest planning for the 2020 Census considered a range of scenarios for using administrative records, from the most expensive option—a traditional census with extensive field follow-up with nonrespondents, and without increased use of administrative records—to the least expensive option—a census conducted entirely by administrative records. Considering concerns about cost and quality, the Bureau ruled out the extreme scenarios and began exploring a “hybrid” scenario that included a number of possible uses for administrative records. In planning for the 2020 Census, Bureau research and testing teams have been determining the possibilities, feasibilities, and cost and quality implications of various uses of administrative records, also including additional information, such as telephone numbers and addresses, obtained from commercial vendors.

Earlier this year in Maricopa County, Arizona, the Bureau conducted its 2015 Census Test to see how well it can use administrative records to reduce fieldwork and increase productivity for NRFU. The test also included a new field management structure and an enhanced Operations Control System supporting daily reassignments of cases. As part of this effort, the Bureau tested how well administrative records substituted for additional visits to collect information from nonresponding households and from proxies, such as neighbors; compared the cost and productivity of traditional follow-up methods to those relying on an enhanced operational control system, demonstrating the potential benefits of automating the assignment of work, scheduling the time of day for enumerators to conduct follow-up to determine when residents were most likely to be home, as well as efficient sequencing and routing of enumerator daily visits based on administrative records and information from other surveys; and provided ground experience with prototypes of systems with leased smartphone devices on which to collect data, which are not necessarily reflective of the systems it may acquire or develop for 2020.

A key benefit of a test like the 2015 Census Test is being able to identify potential problems with design alternatives. Untested systems and unpracticed procedures will inherently experience implementation issues that provide much of the basis on which lessons from such tests are typically drawn. The test results are one source of input to the Bureau’s preliminary design decisions for the 2020 Census. The Bureau included a description of its preliminary decisions in the 2020 Census Operational Plan it released on October 6, 2015. These decisions included using administrative records to identify vacant addresses in advance of follow- up field work and to enumerate nonresponding households when possible to reduce the need for repeated contact attempts during NRFU. The plan also described an updated lifecycle cost estimate for the 2020 Census, which we plan to review. The Bureau’s estimate of the total cost of the 2020 Census with the innovations it describes in its operational plan is $12.3 billion. The Bureau has more tests planned including a 2016 Census Test in selected areas within Harris County, Texas and Los Angeles County, California; a large test of address canvassing also in 2016; an additional site test in 2017 at an as yet undetermined location; and a 2018 end-to-end test—the equivalent of prior decennial cycles’ “dress rehearsal.”

Administrative Records Can Reduce 2020 Fieldwork, but the Bureau Will Need to Better Define Milestones and Deadlines to Help Manage Risks

Decennial Uses of Administrative Records Offer the Bureau the Opportunity to Reduce the Fieldwork Needed for the 2020 Census

In key planning documents, the Bureau describes a goal of using administrative records to reduce the field work involved in its NRFU operation. To that end, the Bureau plans to use data from internal and external sources, such as the 2010 Census, the United States Postal Service (USPS), and the Internal Revenue Service (IRS), in a number of ways, such as by identifying vacant housing units or enumerating households in cases of nonresponse. The Bureau has reported that the following three uses are key to the Bureau potentially saving up to $1.4 billion compared to traditional census methods. The Bureau tested each of these uses during its 2015 Census Test and has decided to use them.

Identify vacant housing units. The Bureau incurs a large part of the census’ cost while following up at residences that did not return a census questionnaire. To ensure a complete count, Bureau guidance in 2010 had enumerators visit some places up to six times to try to obtain a response. During the 2010 Census, enumerators visited 48 million housing units for follow-up at least once. This number included 14 million vacant housing units. One of the largest potential efficiency gains to the census may come from using administrative records to remove these vacant units from the follow-up workload. Preliminary findings from the Bureau’s 2015 site test found that administrative records identified 11.6 percent of the NRFU workload as vacant.

Identify and enumerate nonresponding housing units that are occupied. Another way the Bureau can reduce the NRFU workload is to use administrative records to count households that did not return census questionnaires. As part of its 2015 Census Test, the Bureau successfully enumerated households using various administrative records. The Bureau tested three approaches to counting nonresponding households during the 2015 site test. In the approach that most extensively used administrative records, the Bureau did not attempt any NRFU visits and enumerated all occupied households that had administrative records meeting a certain quality threshold. The enumerators who used this approach had an initial workload of approximately 29,000 households, and administrative records were used to enumerate more than 5,800 of these households—which reduced the workload for this approach by about 20 percent.

Predict best times to complete NRFU. One of the challenges the Bureau faces in NRFU is reaching a household at a time when someone is home. Catching respondents at home on an enumerator’s first visit reduces the need for more follow-up fieldwork. In the 2015 Census Test, the Bureau used administrative records in addition to information about how households had responded to other Bureau surveys to help determine the contact strategy for deciding if and when to interview a housing unit. For example, the Bureau used demographic information, such as age, from administrative records sources to determine the time of day to contact households.

The Bureau Has Identified Nine Additional Opportunities for Using Administrative Records in the 2020 Census

The Bureau has identified nine additional uses of administrative records that may help control cost or improve the quality of decennial census data or operations (see figure 1). The Bureau has not estimated cost savings for these nine uses, but has begun researching the feasibility of most of them.

As shown in the figure, these uses would occur during various points relative to data collection.

Before data collection. The first use listed in the figure—validate and update the address list—is one on which the Bureau is already working. The Bureau is drawing on address lists and map information from state, local, and tribal governments to update its own address list continuously throughout the decade, reducing the need for a more costly door-to-door canvassing during the 2 years prior to the census, as was done for the 2010 Census. According to Bureau officials, the Bureau is about to begin research on how better to use records to identify group quarters, such as dormitories, prisons, nursing homes, or homeless shelters; and to target its outreach, that is, encourage cooperation with the census. The Bureau uses special procedures to enumerate at these places, and administrative records could potentially reduce the time and effort spent getting ready for them.

During data collection. In addition to reducing the NRFU fieldwork, the Bureau is considering using administrative records to help ensure quality control of fieldwork such as by providing a near-real-time check on interviews of households more at risk of being missed in the census. This could reduce fieldwork and respondent burden or enable quality control reinterviews of respondents to target other types of quality concerns. The Bureau is also researching how administrative records can be used to help process responses it receives either on paper or over the Internet that do not have a census ID number on them (this activity is called non- ID processing). The Bureau may receive such responses from households that may have lost or never received mailings or other advance communication from the Bureau. The Bureau has done some testing on this use—in 2015 in the Savannah, Georgia, media market area, the Bureau invited test participants to respond over the Internet. The Bureau demonstrated that a large collation of administrative records from many sources was effective in helping the Bureau correct or fill in missing address information, which enabled the Bureau to better locate where those responses should be counted. The Bureau has other ongoing research into how other records may help the Bureau validate responses or the identities of those who submit responses as part of this processing.

After data collection. If the Bureau still does not have information on a housing unit after collecting data during field operations, it will attempt to impute the data—it has done this since 1970. According to Bureau officials, the Bureau plans to use administrative records to help improve imputation of three related types of data the Bureau fills in for these housing units. These data fields are (1) whether or not a unit is occupied, (2) what the count of the unit might be, and (3) the demographic characteristics of the residents. Finally, the Bureau is considering how administrative records might help the Bureau evaluate the accuracy of the census.

The Bureau’s research and testing on administrative records to help with the address list, the two uses within non-ID processing, and the three related imputation methods, are well underway. The Bureau reports having achieved some early success already by demonstrating records’ use in updating the address list and in locating respondents’ addresses as part of non-ID processing. Bureau officials said that they had not started research and testing on the three remaining potential uses because these uses are less likely to generate significant cost savings or they fall much later in the decennial cycle, and the Bureau considered them a lower priority for the limited funding available for research thus far in the decade. According to Bureau officials, the Bureau will begin research on the remaining uses during fiscal year 2016.

According to the Bureau, nearly every opportunity to use administrative records for the 2020 Census would involve more than one source of records and most opportunities would involve many sources. The Bureau has identified and obtained access to nearly all of the sources it believes it needs to leverage all of the opportunities it has identified, including the three uses (identify vacant housing units; identify and enumerate nonresponding housing units that are occupied; and predict best times to complete NRFU) the Bureau believes will generate a large portion of its estimated $1.4 billion savings from the cost of traditional methods. These sources are summarized in figure 2. The Bureau reported it has tested all of these sources already.

As of July 2015, the Bureau has memorandums of understanding in place with seven federal agencies governing the use of data from 15 different programs and activities.

The Bureau is leveraging data from state governments involved in three federal grant programs for low-income individuals that tend to serve sociodemographic groups (i.e., children and infants) that have historically been undercounted in the census, the Temporary Assistance for Needy Families (TANF), the Supplemental Nutrition Assistance Program (SNAP) and the smaller Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). Bureau officials believe that data maintained by states that administer the programs funded by these grants can reliably identify beneficiaries and their addresses, and can potentially add support to each of the potential uses the Bureau is considering. According to the Bureau, while it has agreements in place with nine states thus far to obtain their TANF, SNAP, or WIC program data, working individually with states can be time consuming. Several states have declined the Bureau’s requests to share data, citing various information technology limitations and resource constraints. The Bureau has invited all states to share data with it and officials said they are prepared to proceed with those that choose to participate.

As part of its program to validate and update its address list throughout the decade (rather than only during the 2 years prior to the census, as was done for the 2010 Census), the Bureau is seeking participation from state, local, and tribal governments. To participate, the governments must reliably maintain address lists, such as for the purposes of emergency response or property assessment, and be willing to share information with the Bureau. Thus far, to validate and update its address list, the Bureau has drawn on address lists and map information from more than 1,000 state, local, and tribal governments. Bureau officials have told us that they expect to receive reliable data covering about two-thirds of the more than 3,200 counties in the country through this program.

The Bureau is working to gain access to additional sources of records to better ensure the quality of the data it already has access to and to improve its ability to find “hard to count” groups. The additional sources include the following:

National Directory of New Hires (NDNH): NDNH is a national database of wage and employment information used for child support enforcement. Bureau officials believe that name and wage information from NDNH could help corroborate the tax data from IRS that the Bureau already has access to, improving the collective accuracy of the records. The President’s 2016 budget submission included a request for legislation that would authorize the Department of Health and Human Services to share NDNH data with the Bureau for statistical purposes such as the decennial census.

KidLink: KidLink is a database from the Social Security Administration (SSA) that links parent and child Social Security numbers for children born after 1998 in U.S. hospitals. It is valuable to the Bureau because children, and babies less than 1 year old in particular, have been historically undercounted. Bureau officials have said that access to this database could help identify another 1 million people. According to the Bureau, SSA raised issues about Bureau access to these data. The Census Bureau Director says that he will work with departmental staff and the Office of Management and Budget to explore an administrative solution that may provide the Bureau with access.

Obtaining access is challenging because federal, state, local, and tribal agencies have different authorities and policies governing what, whether, and how they share their administrative data. For example, the Bureau has access authority to IRS tax data. Yet for other data, such as NDNH, the Bureau is not authorized by statute to have access.

Bureau officials stated that they are examining ways to quantify the potential effect that their access to these additional sources could have on the 2020 Census. The Bureau estimated that the value of acquiring the NDNH and using it to corroborate data from IRS, in conjunction with other administrative and third-party data sources, would be approximately $157.5 million (using 2010 figures and dollars). This assumes no nonresponse follow-up visits for cases with administrative and/or third- party data, so that the actual savings would likely be less since the Bureau recently decided to make at least one follow-up visit before enumerating a household with administrative records. Bureau officials state that there is value in accessing these records for the Bureau’s other statistical surveys as well, and that even if they are unable to obtain the additional records in time for the 2020 Census, they would continue pursuing them for these other purposes, as well as for use in future censuses.

The Bureau Lacks Deadlines for Ensuring Administrative Records Decisions Are Made in Time

As of August 2015, the Bureau had not set deadlines to determine when to make final decisions on which of its 12 identified uses of administrative records it will implement for the 2020 Census, nor had it set deadlines for determining exactly which records from which sources it will tap in support of each use it implements. Moreover, the Bureau has no deadlines it can use against which to measure progress for obtaining access to its additional sources or scheduled milestones for when key steps may need to be taken to integrate them within 2020 preparations. For example, time will be needed to review files to ensure their fitness for use before the Bureau can integrate them into the census design. According to our scheduling guide, assurance of program success can be increased when management relies on credible schedules containing the complete scope of activities necessary to achieve established program objectives.

Bureau officials have stated that final decisions on the use of administrative records are needed by the end of fiscal year 2017 to be included in the Bureau’s 2018 end-to-end test. But, these deadlines do not appear in schedule documents. Deadlines for deciding on the remaining potential uses—either committing to move forward with them or abandoning them as possibilities for 2020— and for deciding how all other records will be used would help to ensure the Bureau is using its resources cost-effectively.

The Bureau is Addressing Challenges to Using Administrative Records for the 2020 Census; Implementing Our Previous Recommendations Could Help

The Bureau Has Processes in Place and Is Conducting Research and Testing to Ensure Quality of Records

Although the Bureau has no control over the accuracy of data provided to it by other agencies, it is responsible for ensuring that data it uses for 2020 Census are of sufficient quality for their planned uses. Data quality can involve the accuracy, relevance, and timeliness of the data.

Steps taken: The Bureau has taken many steps to ensure the quality of the records it is considering using for 2020.

The Bureau’s Center for Administrative Records Research and Application screens all administrative records the Bureau has obtained to ensure their fitness for use by assigning unique person and address identifiers to facilitate record linkage, evaluating biases associated with the linkages, and evaluating the quality and coverage of data.

Similarly, the Bureau’s Geography Division routinely screens address and map files provided by state, local, and tribal governments to determine if they satisfy preset minimum quality standards for completeness of address information. This helps to improve the master list of addresses.

The Bureau’s Administrative Records Modeling project team has researched several predictive models for identifying thresholds of sufficient quality for administrative records used in identifying occupied and vacant housing units during NRFU.

The Bureau has researched which combinations of records more fully cover the population. Among other findings, this research has helped the Bureau refine what combinations of administrative records work better to determine whether a housing unit is occupied or unoccupied, or for determining the number of people living in it.

Relatedly, in August 2015 the Bureau was measuring how well administrative records cover hard-to-count groups such as children and people who were born in other countries. The Bureau planned to finish this work by October 2015.

The Bureau reported that 2014 Census Site Test results found that administrative records matched some households better than others. For example, 65 percent of households with one adult and zero children matched to actual records, but only 33 percent of households with three adults and one or more child matched to administrative records.

Since records need to be timely to be most useful, according to the Bureau, it negotiated to obtain monthly files beginning in February 2015 from IRS tax returns for use in the 2015 Census Test, which it conducted around the Census Day of April 1, 2015. As a result, the Bureau obtained the records several months earlier in the calendar year than it had in the past.

Future plans: The Bureau plans comprehensive testing of all records during an end-to-end test of its 2020 Census design (to be conducted in 2018). The Bureau plans additional testing of administrative records for the 2016 Census Test in the Los Angeles and Houston metro areas, in a large test of address canvassing also in 2016, and in an additional site test in 2017 at an as yet undetermined location.

The Bureau reported it will review imputation models it used during prior censuses to determine how it can integrate information from administrative records into them in fiscal year 2016. Tests will be included in the 2016 Census Site Test.

The Bureau Has Experience Preventing Disclosure, but Implementing Our Prior Recommendations Can Improve Protections

We have previously reported that until the Bureau implements a complete and comprehensive security program, it will have limited assurance that its information and systems are being adequately protected against unauthorized access, use, disclosure, modification, disruption, or loss. In January 2013, we made 115 recommendations aimed at addressing weaknesses in that program. The Bureau expressed broad agreement and said it would work to find the best ways to address our recommendations. In July 2015, the Bureau reported that it experienced an information technology attack to gain access to the Federal Audit Clearinghouse that contains nonconfidential information on audit reporting packages from state and local governments, nonprofit organizations, and Indian tribes expending federal awards. Federal agencies use the single audit reports to ensure program compliance. The Bureau is making additional clearinghouse information available via the Internet next year. According to Bureau officials, the breach was limited to this database on a segmented portion of the Bureau’s network that does not touch administrative records or sensitive respondent data protected under Title XIII, and the hackers did not obtain the personally identifiable information of census and survey respondents.

Steps taken: The Bureau cited examples of its past long-standing experience in collecting data from other agencies and reporting on it as evidence of the Bureau’s ability to prevent disclosure of information from such sources:

Since 1972, the Bureau’s Survey of Business Owners has collected data on businesses from administrative records, including data from the Social Security Administration.

The Bureau’s Longitudinal Employer-Household Dynamics program produces information combining federal, state, and Bureau data on employers and employees. This program collects and secures administrative records information from all 50 U.S. states, including unemployment insurance earnings data.

During negotiations for access, the Bureau and the agency providing the data agree to data safeguards. For example, the Bureau’s agreement with IRS states that the Bureau will advise its employees of their responsibility for handling federal tax information, and will annually certify that all employees who access federal tax information have been advised about their obligation to protect the information. Further, the Bureau is required to provide annual reports to IRS that include the type of computer system and type of medium on which the data are contained. Once the Bureau obtains access to an administrative data source, it transfers the information that it needs to Bureau servers and maintains the information within the Bureau’s firewalls and information security infrastructure.

Our open recommendations underscore the importance of the Bureau safeguarding its systems. Bureau officials state that the Bureau has taken action on all 115 of our recommendations to improve its security program. In assessing the Bureau’s reported actions, we have reviewed documentation pertaining to 75 of the recommendations—58 of which we have confirmed have been addressed and 17 require additional actions and/or documentation from the Bureau. We are currently analyzing the extent to which the remaining 40 recommendations have been addressed by the Bureau and expect to complete that review by the end of 2015.

The Bureau Is Taking Steps to Maintain Transparency, but Implementing Our Prior Recommendation Can Help Support Dialogue on Public Concerns

A third challenge is the extent to which the public will accept the sharing of personal data across government agencies for the purposes of the census. We have previously reported on the need within the federal statistical system for broader public discussion on balancing trade-offs among competing values, such as quality, cost, timeliness, privacy, and confidentiality. Related concerns involve trust in the government and perceptions about burden on respondents as well the social benefits of agencies sharing data. We recommended in 2012 that the Bureau develop and implement an effective congressional outreach strategy, particularly on new design elements the Bureau is researching and considering as well as on cost-quality trade-offs of potential design decisions. The Bureau concurred with the recommendation and has taken a number of steps since that are likely to help inform congressional decision making, which we describe below.

Steps taken: In 2013, the Bureau contracted for regular polling of nationally representative individuals on the extent to which they prefer data to come from information already provided to federal and state governments or from a survey they fill out. Findings included that respondents were evenly divided when asked whether they prefer the Bureau to obtain someone’s name and age directly from the Social Security Administration rather than asking for this information on a questionnaire. In 2013, the Bureau began hosting quarterly program management reviews encouraging dialogue with oversight on selected technical aspects of the Bureau’s ongoing research and testing. These reviews are open to the public and viewable online over the Internet. These supplement the Bureau’s monthly status reports on ongoing research projects that the Bureau provides to Office of Management and Budget and, later, Congress.

Future steps: The Bureau is developing a communications campaign for 2020, which it will formally launch in 2016. The campaign will include information about how the Bureau intends to use administrative records in the 2020 Census. Given the many potential uses of administrative records the Bureau has identified, it will be important for the Bureau’s messaging to consider the range of uses. For example, some people may feel differently about the Bureau using administrative records for enumerating as opposed to targeting the time of day they will be contacted by the Bureau. Moving forward, to help support broader public discussion on trade-offs that the Bureau may need to make on the role of administrative records in the 2020 Census, the Bureau should address our prior recommendation to develop and implement an effective congressional outreach strategy, particularly on new design elements the Bureau is researching and considering, as well as on cost-quality trade- offs of potential design decisions.

In response to our 2012 recommendation, in November 2014 the Bureau provided us with a congressional engagement plan. The four-page plan brings together in one place a summary of the Bureau’s ongoing activity in this area, yet, by itself, lacks goals or strategies for attaining them, or accountability for who will work to implement them or when. We will continue monitoring the Bureau’s efforts to address this recommendation, particularly as they may depend on deadlines the Bureau may yet set for making final decisions about administrative records.

The Bureau Executed Its Recent Site Test According to Test Plan, but the Bureau Has Limited Information on How Some Implementation Issues Affected the Data Collected

The 2015 Site Test Was Carried Out on Schedule and Achieved Key Goals

The Bureau had several objectives for its 2015 Census Test. For example, the Bureau wanted to begin the process of developing a field operations control system that combined administrative records, technology, and available real-time data to improve the efficiency of field data collection. In addition, it planned to collect data on whether using administrative records could reduce NRFU workload and increase NRFU productivity. Senior Bureau officials told us that the test was also to provide data to help inform future cost estimates and design decisions.

The Bureau designed the 2015 Census Test to test three approaches to NRFU, and used one each within three different parts of the test sample: one that followed procedures similar to the 2010 Census and two that used experimental approaches. Each part of the sample had a workload of around 23,000 housing units.

Control panel. This panel followed NRFU procedures similar to those used in the 2010 Census. Like the 2010 Census, the operation was managed from a field office, enumerators compiled their timesheets by hand daily, and supervisory staff had regular face-to-face meetings with their enumerators.

Hybrid Administrative Records Removal Panel. For this panel, the Bureau used administrative records to identify vacant housing units and remove them from the NRFU workload before enumerators attempted contact. Then enumerators were to make one attempt to contact the remaining housing units. If the attempt was unsuccessful, the Bureau used administrative records to attempt to enumerate the housing units.

Full Administrative Records Removal Panel. This panel followed the same approach as the hybrid to remove vacant units from the NRFU workload. But the Bureau also attempted to use administrative records to count occupied households before enumerators attempted any contacts.

Bureau officials said they consider the 2015 Census Test a large success because it allowed them to employ a variety of new methods and advanced technologies that are under consideration for the 2020 Census. Our review of key tracking documents showed that the Bureau executed key milestones for the test early or on schedule. Additionally, Bureau officials stated that operational costs tracked closely to planned costs, and that actual field workload was within a few percentage points of that planned for each test panel.

Another success of the 2015 Census Test is that during our observations the Bureau maintained control over test panels so that they did not appear to influence each other. For example, field test managers with whom we met appeared largely unaware that other test panels existed or how their procedures may be different. According to the Bureau, the test also demonstrated the usefulness of continuing work on the enhanced operational control system.

Some of the tests biggest achievements were in demonstrating the feasibility of using administrative records: a total of 8,370 vacant units were identified for removal from the NRFU workload of 72,072 units (11.6 percent), a total of 14,312 cases (19.8 percent of workload) were identified as occupied through the use of administrative records. Depending on the test panel, this information was used instead of knocking on doors of neighbors or others when respondents were not home and enumeration attempts were exhausted, and the race/ethnicity for persons who did not provide it in their responses was identified.

These test results are linked to assumptions that need to be met to attain the Bureau’s cost savings estimate and assisted the Bureau in making its preliminary design decisions and updating its cost estimates. For example, the Bureau’s October 2015 operational plan reports that the Bureau will use an approach during NRFU like that for the hybrid administrative records removal panel during the 2015 test, making only one visit to nonresponding households where the Bureau has determined administrative records are good enough to complete the roster and enumerate the household.

Lack of Information on Site Test Implementation Issues May Affect Usefulness of Data for Future Decision Making

As part of its testing related to administrative records and proving capabilities of prototype systems implementing selected design features in the 2015 Census Test, the Bureau collected data on the extent to which administrative records reduced field data collection and improved productivity; two key cost-drivers in prior decennials. During the site test, the Bureau experienced specific implementation issues, and these in turn affected the measurement of the key cost drivers. We did not select a generalizable sample of the over 400 enumerators hired during the test to interview. Yet, our observations were consistent across the 30 enumerators we met and were echoed during the four debriefings the Bureau held with field staff near the conclusion of the site test.

We observed enumerators spending extra time dealing with enumeration devices that were not working properly, lacked connectivity, or that they may not have been using properly. Enumerators also told us about experiences similar to our observations. These issues affected measures related to total hours spent on the NRFU operation.

We observed several instances of enumerators being assigned to enumerate repeatedly the same nonresponding or nonexistent households, which the design alternative would not have permitted had it been implemented correctly. This affected measures related to reducing contact attempts.

We observed inefficiencies in automated route management, poor cellular service leading to limited or no access to smart phone maps while enumerators tried to locate addresses, and, to a lesser extent, enumerators having difficulties closing out or completing their cases, leading to additional visits. These issues are likely to have decreased various productivity measures.

Additionally, we observed situations where multiple enumerators were visiting multiunit structures and gated communities without coordination and communication between each other. This affected measures related to reducing contact attempts.

Based on our discussions with enumerators about these issues, it appears that the incidence of the issues was underreported to the Bureau, which had intended to track such issues. For example, all but 4 of the 30 temporary enumerators and field supervisors we spoke with said they tried their own temporary solutions to problems they faced, such as powering down and restarting their Bureau-issued phones when they froze, rather than notifying supervisors or the Bureau’s call-in number every time. We heard this repeated during the Bureau’s debriefings of enumerators near the end of the test. Additionally, enumerators we spoke with reported using the capability within their phones to record notes about specific cases with which they experienced implementation problems. According to senior managers responsible for managing the test implementation, during the test there was no systematic review of these notes planned other than when an enumerator had separately flagged a case as involving a dangerous situation, or was being let go for performance reasons and the enumerator’s work was being reviewed. After we discussed the notes with senior Bureau officials, they told us that they would explore the notes for information that would help the Bureau better understand the implementation issues. They also told us that during enumerators’ debriefings, they learned enumerators had not always known to whom to report what types of issues. Bureau training materials provided a toll-free number for enumerators to call regarding any technical issue if first calling the enumerator supervisor could not resolve it.

Bureau officials acknowledged that problems with the implementation of the test likely affected productivity measures to be used in calculating future cost estimates. According to leading practices reported in our cost estimation guide, information about the extent to which implementation issues affect test data should be collected, as they can be useful in future cost estimation. However, the effect on the measures is difficult to determine because the Bureau did not systematically keep data about the extent of the problems. Bureau officials said they are analyzing the results of the test to determine the extent to which identified implementation issues affected the productivity measure, so that they could better control for and understand the separate effects of implementation issues.

Moreover, in future tests, systematically collecting better information about the effect of implementation issues may help inform how the Bureau prioritizes specific design features underlying those issues. For example, if in future tests the Bureau systematically tracks the cases where, say, its automated routing may create problems for enumerators, the Bureau could know the extent to which that implementation issue affected its test measures, like average miles driven per case, and the resulting effect on cost. With such information, the Bureau could make a more informed decision about prioritizing its efforts and resources on ensuring that routing works in systems it ultimately develops or acquires, or whether to get rid of automated routing altogether. Such information can also help inform cost and risk analyses the Bureau may undertake based on the possibilities that such specific implementation issues may also occur during the 2020 Census. Accordingly, steps the Bureau can take to better capture information from enumerators about implementation issues should they arise in future tests will also help future cost estimation. Such steps might include additional or revised training elements for enumerators on what details to report and the logistics of reporting them: to whom—such as the help desk or their supervisors— where—such as within the notes capability on their phones or elsewhere—and how.

Key Assumptions for Cost Saving Estimates from Administrative Records are Logical; the Bureau Will Continue to Validate the Assumptions

The Bureau had five key assumptions that support its estimate that using administrative records for the 2020 enumeration will cost less than relying on a traditional design and methods (see table 1). Bureau officials responsible for cost estimation described each of the assumptions to us and showed us where the assumptions were represented in key planning documents.

The first three assumptions will only materialize if certain conditions are met. For example, to reduce the number of field offices, the Bureau will need to demonstrate that it can reduce the NRFU workload, as the workload is a key driver of the number of field offices needed. The Bureau articulated the last two assumptions as business decisions about its operations. We reviewed whether the assumptions were logically represented in the calculations used to produce the Bureau’s available cost estimates, and how the Bureau had supported the assumptions through prior Bureau experience, research, or testing. As shown in table 2, overall, the Bureau’s assumptions are logical and have support. The Bureau plans to further test some of these assumptions to validate the assumptions further or to identify needed revisions.

The Bureau’s planned tests in 2016 and beyond will inform several of these assumptions: Identify and remove vacant units from the NRFU workload. Now that the Bureau has successfully demonstrated it can identify and remove vacant units from the NRFU workload using administrative records, it plans to continue testing this activity to gather additional data on how much the NRFU workload can be reduced.

Reducing the number of field offices. Although the Bureau announced in October that it would open no more than 250 offices in 2020, the Bureau plans to conduct future large tests at other sites and to involve other operations in the tests to gather additional information about the number of field offices needed for 2020.

Limiting the number of NRFU visits. Future tests will provide the Bureau opportunities to try to control the maximum number of visits enumerators make, and determine if it can reduce implementation issues that caused repeat visits during the 2015 test.

Bureau officials told us that research is underway that will help validate that the new uses of administrative records will provide the benefits that the VDC and CFU operations formerly provided. They said that reliance on administrative records can provide an alternative to field visits intended simply to verify whether a housing unit exists or is vacant. Further, they said that the use of administrative records before and during the enumeration will remove the need for a CFU operation checking against the records after the enumeration, as was done in the 2010 Census.

While we were reviewing these cost assumptions, documentation was not always readily available, and Bureau reporting on one of the assumptions needed to be corrected. We were able to identify the needed support. Bureau staff said that, moving forward, they decided to change the methodology for future reporting on the cost estimate to involve more factors and variables, such as the ratio of field workers to supervisors they would need in 2020 in addition to the NRFU workload assumption. This change will help demonstrate the reliability of the estimates as well as ensure effective communication with others about them.

Bureau officials told us that the revised total life-cycle cost estimate the Bureau released on October 6, 2015, was developed with leading practices from our cost estimating and assessment guide. After the Bureau releases the underlying model, methodology, and supporting documents for the estimate, we anticipate reviewing them to assess their reliability.

Conclusions

Although administrative records have been discussed and used for the decennial census since the 1970s, the Bureau plans a more significant role for them to reduce the amount of data collection fieldwork to reduce the cost of the decennial census in 2020. The Bureau appears to have demonstrated the feasibility and potential effectiveness of administrative records for several uses during NRFU, which the Bureau estimates could save up to $1.4 billion compared to traditional census methods. The Bureau has also identified several additional opportunities to leverage administrative records to help improve the cost and quality of the 2020 Census. Yet, the Bureau will need to consider when to end research pursuits that show less promise for substantially reducing the census’ cost or meeting other 2020 goals so that it can focus resources on successfully refining and implementing activities that have greater potential. Knowing deadlines for when final go/no go decisions need to be made about which records the Bureau will use, how it will use them, and for which purposes will help ensure necessary activities are completed on time. Deadlines regarding still uncertain purposes or those involving records the Bureau is still pursuing, such as NDNH and KidLink, as well as those from some states, will also help the Bureau prioritize which activities—or records—to continue pursuing or to abandon if time becomes a constraint.

The Bureau’s 2015 Census Test was generally an operational success in that it provided much useful information to inform cost estimates and decisions about how to design future operations. Test results and the information about implementation issues we and the Bureau documented should prove useful as the Bureau moves forward to refine its business requirements for further testing. As the Bureau plans for its 2016 tests, getting better information from them about which fieldwork cases are not being implemented as planned could help the Bureau link future estimates of cost savings to, and prioritize, design features it wants or needs to work well in final systems it develops, or to eliminate problematic features it deems not worth the trouble. The Bureau could capture this information more systematically through the help desk, within the notes capability on the phones used for interviewing or elsewhere, and by ensuring enumerators received training on where to record the issues, who to contact, what details to include, and the importance of doing so. The bottom line is that the amount of time the Bureau has to research and test the range of options for 2020 and ensure their readiness is limited.

The Bureau has a lot of activity to follow through on in the time remaining, so it will be important to reduce the number of open options to ones the Bureau can manage well when turning the corner from research to a focus on development and implementation.

The Bureau’s early cost savings assumptions related to its use of administrative records are logical, and the Bureau is taking steps to develop further support for them. We plan to review the Bureau’s October 2015 cost estimate and latest estimates of savings from using administrative records after the Bureau makes supporting documentation available.

Recommendations for Executive Action

We recommend that the Secretary of Commerce direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to take the following two actions to help ensure the Bureau focuses its resources on those activities that show promise for substantially reducing enumeration cost.

Establish clearly documented deadlines for making final decisions about which records to use for what purposes, particularly for purposes not yet demonstrated as feasible or involving records it does not already have access to, such as NDNH and KidLink.

In advance of the 2016 Census Test and later tests, ensure systematic capture of information about fieldwork cases that experience problems by including information in enumerator training about where to record the issues, who to contact, what details to include, and the importance of doing so.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of the Department of Commerce for comment. In its written comments, reproduced in appendix II, the Department of Commerce concurred with our recommendations. The Department of Commerce also provided minor technical comments that were incorporated, as appropriate.

We are sending copies of the report to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO’s website at http://www.gao.gov.

If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The purpose of our review was to examine the Census Bureau’s (Bureau) plans for using administrative records for the 2020 Census. Specifically, our objectives were to review (1) the Bureau’s plans for using administrative records for 2020 and what opportunities and challenges the Bureau faces in using them; (2) the extent to which the Bureau’s key 2015 test of administrative records were implemented in accordance with its testing objectives and what the Bureau’s experience implementing selected aspects of the test was; and (3) key assumptions supporting the cost savings estimates to be achieved from administrative records.

For all objectives, we reviewed documentation from the Bureau on 2020 research and testing of administrative records, and reviewed documentary and testimonial evidence from Bureau officials responsible for research and testing the use of administrative records.

To address the first objective, we identified the administrative records the Bureau is considering and linked them to their possible uses the Bureau is considering for 2020. We identified which records the Bureau has access to and examined the Bureau’s authority under Title XIII and other statutes to use information from other agencies for the decennial census. We identified what decisions remain for the Bureau regarding administrative records as well as the Bureau’s timelines for making those decisions. We relied on our Schedule Assessment Guide as a source of criteria for assessing activity the Bureau plans for administrative records. We took steps to verify that the related schedule data we examined was reliably representing the Bureau’s schedule, such as by comparing the respective inclusion of major projects as well checking that activities occurred in both.

To address the second objective, we examined the Bureau’s 2015 Census Test, which took place in Maricopa County, Arizona. Using the Bureau’s test objectives as criteria, we conducted direct observations and interviews to assess the Bureau’s implementation of the test, and collected performance metrics on the test from the Bureau. We also documented where we observed implementation deviating from either what Bureau temporary enumerators had been trained to expect, or what we expected based on our prior experience with census field operations. We also identified problems appearing to arise with implementation. We conducted 30 in-field observations of Bureau enumerators conducting Nonresponse Follow-up during the test and interviewed Bureau employees managing the test. Our sample of field observations was stratified for geographic balance across the test site area, but was not designed necessarily to be generalizable. When examining the extent to which implementation issues may have affected measurement of key cost drivers in the 2015 Census Test, we relied on our Cost Estimating and Assessment Guide for criteria. We communicated implementation issues we observed in near-real time to Bureau officials for their consideration as they conducted their own evaluation of the test and ongoing related research.

To address the third objective, we inventoried the working cost assumptions provided by the Bureau and isolated those related to administrative records. To determine if assumptions were logical, we traced their incorporation into respective calculations within the cost model the Bureau used to produce the Bureau’s earlier cost estimates available to us at the time of the audit. We examined the support and justifications the Bureau had documented for each assumption, the related results of the 2014 and 2015 Census Tests, and related materials the Bureau made available to us during the audit. When examining the extent to which implementation issues may have affected measurement of key cost drivers in the 2015 Census Test, we relied on our Cost Estimating and Assessment Guide for criteria. We did not review the Bureau cost estimation methodology or the reliability of either its preliminary cost savings estimates, or any cost estimation material information that was part of the Bureau’s October 2015 release.

We conducted this performance audit from January 2015 through October 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ty Mitchell, Assistant Director; Jeffrey DeMarco; Robert Gebhart; Richard Hung; Andrea Levine; Donna Miller; Shannin O’Neill and Timothy Wexler.