2010 Census: Data Collection Operations Were Generally Completed as Planned, but Long-standing Challenges Suggest Need for Fundamental Reforms
Published: 2010-12-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Although the U.S. Census Bureau (Bureau) generally completed the field data collection phase of the 2010 Census consistent with its operational plans, at $13 billion, 2010 was the costliest census in the nation's history. Moving forward, it will be important to both refine existing operations as well as to reexamine the fundamental approach to the census to better address long-standing issues such as securing participation and escalating costs. As requested, this report reviews (1) the conduct of nonresponse follow-up (NRFU), where enumerators collect data from households that did not return their census forms, (2) the implementation of other field operations critical to a complete count, and (3) potential reexamination areas that could help produce a more cost-effective 2020 Census. The report is based on GAO's analysis of Bureau data and documents, surveys of local census office managers, and field observations.
section_title: What GAO Found
paragraphs: Nationally, the Bureau was well positioned to implement NRFU and subsequent field operations. The Bureau achieved a mail response rate of 63 percent, which was within its expectations, and recruited nearly 3.8 million total applicants for census jobs, which was 104 percent of its staffing goal. Moreover, the Bureau completed NRFU under budget, reportedly spending $1.59 billion on the operation, about $660 million (29 percent) less than the Bureau initially estimated. Most of the Bureau's local census offices (LCO) also completed NRFU ahead of the 10-week allotted time frame. Despite these operational successes, the Bureau encountered some notable challenges. For example, the pace of NRFU may have fostered a culture that tended to emphasize speed over quality, as those LCOs with higher percentages of less-complete questionnaires were more likely to have completed NRFU in 53 days or less (the average time LCOs took to complete NRFU). The Bureau also had to overcome issues with critical information technology (IT) systems. For example, performance problems with the IT system used to manage NRFU led to processing backlogs. Although the Bureau developed workarounds for the issue, it hindered the Bureau's ability to fully implement quality-assurance procedures as planned. The Bureau generally completed other follow-up operations designed to improve the accuracy of the data consistent with its plans. One of these activities was the vacant/delete check (VDC), where enumerators verified housing units thought to be vacant or nonexistent. The Bureau completed VDC two days ahead of schedule, but encountered duplicate addresses on the address list used for the operation, which could indicate a more systemic problem with the quality of the Bureau's address list. While it will be important to refine existing census-taking activities--many of which have been in place since 1970--results of prior censuses point to the fact that simply improving current methods will not bring about the reforms needed to control costs and maintain accuracy. The cost of conducting the census has, on average, doubled each decade since 1970. At the same time, because of demographic and attitudinal trends, securing a complete count has become an increasing challenge. As a result, a fundamental reexamination of the nation's approach to the census will be needed for a more cost-effective enumeration in 2020. Potential focus areas include new data collection methods; the tenure of the Census Director; and ensuring the Bureau's approaches to human-capital management, knowledge sharing, and other internal functions are aligned toward delivering more cost-effective outcomes. The Bureau recognizes that fundamental changes are needed and has already taken some important first steps, including developing a strategic plan. To help ensure the Bureau's efforts stay on track and to avoid problems it had in planning for prior censuses, it will be important for the Bureau to issue a comprehensive operational plan that includes performance goals, milestones, cost estimates, and other critical information that could be updated regularly.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Census Director refine NRFU and other field follow-up efforts by, among other things, emphasizing quality as much as speed during NRFU and by incorporating best practices in its IT acquisition-management policy. To help ensure reform efforts stay on track, the Bureau should develop an operational plan that integrates performance, budget, and other information. The Department of Commerce generally agreed with GAO's findings and recommendations.

section_title: Letter
paragraphs: One of the final acts of the decade-long census life cycle is to occur in the  remaining days of 2010 when, as required by law, the U.S. Census Bureau  (Bureau) is to release to the President the state population counts used to  apportion Congress. Although some additional work and more data  releases lie ahead, and information on the accuracy of the count is not  scheduled to be available until early 2012, this much is clear: the Bureau  generally completed the enumeration phase of the 2010 Census on  schedule and consistent with its operational plans, and largely surmounted  a series of risks that jeopardized the success of the headcount.
As you know, an operationally successful census was no small  accomplishment. Various social and demographic trends such as an  increasingly diverse population and a distrust of government made a  complete count an extraordinary challenge in 2010. At the same time, the  Bureau had to overcome a variety of internal management challenges  including shortcomings with critical information technology (IT) systems.
We have long reported that the decennial census is a shared national  undertaking, where the Bureau, Congress, government agencies at all  levels, private organizations, and ultimately the public at large, all play  vital roles in securing a complete count. That the Bureau completed key  operations on schedule, obtained an acceptable participation rate, and is  on track for meeting legally mandated deadlines for reporting population  figures is a tremendous credit to the people of this nation for completing  their census forms and cooperating with enumerators; the hundreds of  thousands of career and temporary Bureau employees who diligently  implemented a vast array of census-taking activities, often under difficult  circumstances; public, private, tribal, and nonprofit organizations of all  sizes for voluntarily partnering with the Bureau and raising awareness of  the census; and finally to Congress, which provided the necessary support  while holding the Bureau accountable for results.
Despite these impressive achievements, the 2010 Census required an  unprecedented commitment of resources, including recruiting more than  3.8 million total applicants—roughly equivalent to the entire population of  Oregon—for its temporary workforce; and it escalated in cost from an  initial estimate of $11.3 billion in 2001 to around $13 billion, the most  expensive population count in our nation’s history. Further, our oversight  of the 1990, 2000, and now 2010 Censuses suggests that the fundamental  design of the enumeration—in many ways unchanged since 1970—is no  longer capable of delivering a cost-effective headcount given the nation’s  increasing diversity and other sociodemographic trends.
Indeed, beginning in 1990, we reported that rising costs, difficulties in  securing public participation, and other long-standing challenges required  a revised census methodology, a view that was shared by other  stakeholders. For 2010, the Bureau eliminated the long-form questionnaire  in an effort to boost response rates, and refined other census-taking  activities, but the basic approach to the enumeration is essentially the  same as it was 40 years ago, and achieving acceptable results using these  conventional methods has required an increasingly larger investment of  fiscal resources, resources that in the coming years will become  increasingly scarce.
In short, as the nation turns the corner on the 2010 Census, it will be vitally  important to both identify lessons learned from the current decennial  census to improve existing census-taking activities, as well as to  reexamine and perhaps fundamentally transform the way the Bureau  plans, tests, implements, monitors, and evaluates future enumerations in  order to address long-standing challenges.
As requested, this report assesses the implementation of (1) nonresponse  follow-up (NRFU), the largest and most costly census field operation,  where the Bureau sends enumerators to collect data from households that  did not mail back their census forms, and (2) other key follow-up field  operations that were critical for ensuring a complete count; and   (3) identifies key questions and focus areas that will be important for the  Bureau, Congress, and census stakeholders to consider going forward now  that planning for the next enumeration is underway.
This report is one of three we are releasing today. Among the other two,  one focuses on the Bureau’s efforts to reach out to and enumerate hard-to- count populations, while the other examines the implementation of  operations aimed at reducing census coverage errors. Both reports  identify preliminary lessons learned, as well as potential focus areas for  improvement.
In reviewing NRFU, we examined the pace of production, the  fingerprinting of census workers as part of a background check, and the  performance of a critical automated system. The follow-up operations we  reviewed for this report include the vacant/delete check (VDC), where the  Bureau verifies the status of housing units flagged earlier in the census as  being unoccupied or nonexistent; and census coverage measurement  (CCM), where the Bureau assesses the completeness and accuracy of the  census count.
For all three objectives, we (1) analyzed Bureau cost and progress data as  well as planning and other pertinent documents; (2) conducted periodic  surveys of the Bureau’s 494 local census office (LCO) managers using a  series of online questionnaires that asked about their experience in  managing LCO activities; and (3) made field observations at 28 locations  across the country selected for various factors such as their geographic  and demographic diversity, and including parts of such urban areas as  Atlanta, Boston, Chicago, Detroit, New Orleans, New York City, San  Francisco, and Tucson, as well as less-populated areas such as Meridian,  Mississippi, and New Castle, Delaware. We also interviewed Bureau  officials at headquarters and LCO managers and staff, and reviewed our  prior work on the planning and implementation of the 1990, 2000, and 2010  Censuses. Moreover, to help inform a reexamination of the nation’s  approach to the census, in addition to the above, we reviewed our prior  work on governmentwide reexamination, as well as leading practices and  attributes in the areas of IT management, organizational performance,  collaboration, stewardship, and human capital. Appendix I includes  additional information on our scope and methodology and a list of LCOs  we visited. Data presented in this report measuring operational timeliness  and data quality were drawn from Bureau management and operational  data systems. To assess the reliability of the data, we reviewed Bureau  electronic documentation to gain information about the data and their  sources, and followed up with agency officials knowledgeable about the  data in cases where we had questions about potential errors or  inconsistencies. On the basis of our efforts, we determined that the data  were sufficiently reliable for the purposes of supporting the findings and  recommendations in this report.
We conducted this performance audit from December 2009 until  December 2010 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audits to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
On December 7, 2010, the Secretary of Commerce provided written  comments on a draft of this report (see app. II). The Department of  Commerce generally agreed with the overall findings and  recommendations of the report. In addition, the Secretary of Commerce  provided the Bureau’s technical comments and suggestions where  additional context might be needed, and we revised the report to reflect  these comments where appropriate.
subsections: section_title: Background
paragraphs: The decennial census is a constitutionally mandated enterprise critical to  our nation. Census data are used to apportion congressional seats, redraw  congressional districts, and help allocate hundreds of billions of dollars in  federal aid to state and local governments each year. A complete count of  the nation’s population is an enormous challenge requiring the successful  alignment of thousands of activities, hundreds of thousands of temporary  employees, and millions of forms. Indeed, over the past year, in an effort  to secure a complete count, the Bureau mailed out questionnaires to about  120 million housing units for occupants to complete and mail back; hand- delivered approximately 12 million questionnaires—mostly in rural  locations as well as in areas along the Gulf Coast affected by recent  hurricanes—for residents to fill out and return by mail; went door-to-door  collecting data from the approximately 46.6 million households that did  not mail back their census forms; and conducted operations aimed at  counting people in less-conventional dwellings such as migrant-worker  housing, boats, tent cities, homeless shelters, nursing homes, dormitories,  and prisons. In short, the decennial census is large, logistically complex,  and, at a cost now estimated at around $13 billion, expensive.
In developing the 2010 Census, the Bureau faced three significant internal  challenges: critical IT systems had performance problems during testing,  cost-estimates lacked precision, and some key operations were not tested  under census-like conditions. These were some of the issues that led us to  designate the 2010 Census a GAO high-risk area in 2008.
Although every census has its decade-specific difficulties,  sociodemographic trends such as concerns over personal privacy, more  non-English speakers, and more people residing in makeshift and other  nontraditional living arrangements make each decennial increasingly  challenging and do not bode well for the cost-effectiveness of future  counts. As shown in figure 1, the cost of enumerating each housing unit  has escalated from around $16 in 1970 to around $98 in 2010, in constant  2010 dollars (an increase of over 500 percent). At the same time, the mail  response rate—a key indicator of a successful census—has declined from  78 percent in 1970 to 63 percent in 2010. The mail response rate is an  important figure because it determines the NRFU workload and ultimately,  NRFU costs. In many ways, the Bureau has to invest substantially more  resources each decade just to match the prior decennial’s response rate.
In our earlier work on high-performing organizations, we noted that the  federal government must confront a range of new challenges to enhance  performance, ensure accountability, and position the nation for the future.  Nothing less than a fundamental transformation in the people, processes,  technology, and environment used by federal agencies to address public  goals will be necessary to address public needs. Ultimately, however, the  federal government needs to change its culture to be more results- oriented. For the Bureau, as with all federal agencies, this means ensuring,  among other things, that its culture embraces results rather than outputs;  follows matrixes rather than stovepipes; forms partnerships rather than  protecting turf; focuses on risk management rather than risk avoidance;  and takes proactive approaches rather than behaving reactively.
section_title: NRFU Was Generally Successful; Refinements Could Improve Procedures for 2020
subsections: section_title: The Bureau Met Its Response Rate Goal, but Recruited More Enumerators Than Needed and Should Revisit Its Staffing Model
paragraphs: Nationally, in terms of workload (as determined by the mail response rate)  and staffing levels, the Bureau was well positioned to implement NRFU.  With respect to the response rate, the Bureau expected a level of 59  percent to 65 percent. The actual mail response rate on April 19, when the  Bureau initially determined the universe of houses to visit for NRFU, was  just over 63 percent, well within the Bureau’s range of estimates. This  translated into an initial workload of 48.6 million housing units.
Achieving this response rate was an important accomplishment as the  nation’s population is growing steadily larger, more diverse, and according  to the Bureau, increasingly difficult to find and reluctant to participate in  the census. High response rates are essential because they save taxpayer  dollars. According to the Bureau, for every 1 percentage point increase in  mail response in 2010, the Bureau saved $85 million that would otherwise  have been spent on the follow-up efforts. According to the Bureau, it costs  42 cents to mail back each census form in a postage-paid envelope,  compared with an average estimate of around $57 for field activities  necessary to enumerate each housing unit in person. Moreover, mail  returns tend to have better-quality data, in part because as time goes on  after Census Day (April 1), people move, or may have difficulty recalling  who was residing with them.
As illustrated in figure 2, the Bureau met its expected response rate in all  but 11 states. The highest response rate (71.7 percent) was in Minnesota,  while the lowest response rate (51 percent) was in Alaska. At the same  time, response rates in all but two states—Hawaii and South Carolina—as  well as the District of Columbia, declined anywhere from 0.8 to 8.2  percentage points when compared to 2000, thus underscoring the  difficulty the Bureau will face in the future in trying to sustain response  rates. The mail response rate is important because it helps the Bureau  determine the housing units that failed to mail back the census  questionnaires, and thus are included in the NRFU workload.
The mail response rate differs from the participation rate in that it is  calculated as a percentage of all housing units in the mail-back universe,  including those that are later found to be nonexistent or unoccupied. In  contrast, the participation rate is the percentage of forms mailed back by  households that received them and is a better measure of cooperation with  the census. According to a Bureau press release dated October 21, 2010,  the nation achieved a final mail participation rate of 74 percent, matching  the final mail participation rate that was achieved for the 2000 Census.  Compared to 2000, participation rates for 22 states and the District of  Columbia, either met or exceeded their 2000 Census rate.
Key factors aimed at improving the mail response rate included the  mailing of an advance letter, a reminder postcard, and an aggressive  marketing and outreach program. In addition, this is the first decennial  census the Bureau sent a second or “replacement” questionnaire to  households. Replacement questionnaires were sent to around 25 million  households in census tracts that had the lowest response rates in the 2000  Census, and 10 million replacement questionnaires were sent to  nonresponding households in other census tracts that had low-to- moderate response rates in 2000.
With respect to staffing levels, the Bureau set a recruitment goal of nearly  3.7 million total applicants and achieved 104 percent of this goal by April  25, 2010, recruiting more than 3.8 million total applicants, almost a week  prior to the start of NRFU (once the Bureau had an adequate pool of  candidates for 2010, it attempted to limit the number of additional  applicants, taking such steps as discontinuing the advertising of census  jobs in mailed-out census materials).
According to the Bureau, based on past experience, it set its recruiting  goal at five times the number of persons that needed to be trained to  ensure it had an ample pool of candidates in specific areas with specific  skills, as well as to ensure it had a sufficient supply of enumerators during  the course of its field operations. The Bureau’s approach was similar to  that used for the 2000 Census despite vast differences in the economy.  During the 2000 Census, the Bureau was recruiting in the midst of one of  the tightest labor markets in nearly three decades. In contrast, during the  2010 Census, the Bureau was recruiting workers during a period of high  unemployment. While having too few enumerators could affect the  Bureau’s ability to complete NRFU on schedule, overrecruiting has its own  costs. For example, there are costs associated with administering and  processing the test taken at the time an individual applies for a census job,  as well as a $2 charge to have a name background check run on all  applicants. Overrecruiting can also be burdensome on applicants as they  need to find a test site and take a test before they can be hired for a census  job—a job that because the Bureau has overrecruited, may not be  available. In looking forward to 2020, it will be important for the Bureau to  more precisely refine its recruiting model based on lessons learned from  the labor markets in both 2000 and 2010, and use this information to  develop more accurate recruiting targets. It will also be important for the  Bureau to adhere to recruiting goals so that additional costs are not  incurred.
section_title: The Bureau Completed NRFU $660 Million under Budget
paragraphs: The Bureau budgeted that NRFU would cost around $2.25 billion.  However, by the end of the operation, the Bureau reported using  approximately $1.59 billion, which was 29 percent lower than budgeted.  The Bureau, with congressional approval, also set up a contingency fund  of $574 million to cover additional expenses that could have been caused  by unfavorable weather and other unforeseen events. However, in the end,  contingency money was not needed to complete NRFU.
While the Bureau conducted NRFU under budget, the difference between  actual and expected NRFU costs also highlights the need for the Bureau to  develop an accurate cost model in order to establish more credible cost  estimates for 2020. In addition to NRFU, other census operations had  substantial variances between their initial cost estimates and their actual  costs. In our 2008 report, we noted that the Bureau had insufficient  policies and procedures and inadequately trained staff for conducting  high-quality cost estimation for the decennial census, and recommended  that the Bureau take a variety of steps to improve the credibility and  accuracy of its cost estimates, including performing sensitivity and  uncertainty analyses. The Bureau generally agreed with our  recommendation and is taking steps to address them.
section_title: Most Local Census Offices Finished NRFU ahead of Schedule, but the Bureau’s Ambitious Production Schedule May Have Produced Mixed Results
paragraphs: In conducting NRFU, it is important for enumerators to follow Bureau  procedures for collecting complete and accurate data while keeping  production on schedule so that subsequent activities can begin as planned.  Timely completion of NRFU is also important because as time goes on,  people move or might have difficulty remembering who was living in a  household on Census Day.
The Bureau went to great lengths to obtain complete data directly from  household members. For example, Bureau procedures generally called for  enumerators to make six attempts to reach each household on different  days of the week at different times until they obtained needed information  on that household. However, in cases where household members could  not be contacted or refused to answer all or part of the census  questionnaire, enumerators were permitted to obtain data via proxy (a  neighbor, building manager, or other nonhousehold member presumed to  know about its residents), or if an enumerator after the required six  attempts was unable to collect data from either the household or a proxy  respondent then the enumerator submitted the incomplete questionnaire  to the LCO (this is referred to as a “closeout interview”). Closeout  interviews are processed at Bureau headquarters where statistical  methods are used to determine household information.
For the 2010 Census, NRFU began May 1 and was scheduled to finish July  10, 2010. However, a majority of LCOs generally finished their NRFU  workloads ahead of this 10-week time frame. For example, by June 28,  2010, week 8 of the NRFU operation, 342 of the Bureau’s 494 LCOs (almost  69 percent), had completed 100 percent of their workload. Figure 3 shows  the production levels over the course of NRFU.
A number of factors helped most LCOs complete NRFU ahead of schedule.  For example, the Bureau removed almost 2 million late mail returns prior  to the start of NRFU, reducing the follow-up workload from 48.6 million to  46.6 million housing units (a 4 percent reduction in NRFU workload). The  removal of the late mail returns resulted in a 1.5 percent increase in mail  response rate, saving approximately $127.5 million (based on the Bureau’s  estimate that a 1 percentage point increase in the mail response rate would  decrease workload costs by around $85 million).
Another factor that was instrumental to the success of NRFU was  retaining a sufficiently skilled workforce. Because of high unemployment  rates, turnover was far lower than anticipated. Advertising census jobs  locally helped to ensure an adequate number of applicants, and according  to the Bureau, mileage reimbursement may have been lower, in part  because enumerators lived in and had local knowledge about the  neighborhoods they were assigned. Further, people may have been more  willing to cooperate with enumerators who were from their own  community. For example, at a Native American village in New Mexico,  local enumerators were aware that according to the community’s customs  it was considered offensive to launch into business without first engaging  in conversation. In addition, local enumerators in hurricane-affected rural  areas of Louisiana were able to successfully locate households based on  their knowledge of the geography. For example, based on his familiarity  with the area, one enumerator we observed was able to locate an assigned  household not included on a census map using only a brief description,  such as “a white house with green roof away from the road.”
The Bureau also used weekly production goals that helped LCOs focus on  the need to stay on schedule and to track their progress. However, several  measures we reviewed underscored the challenge that LCOs faced in  hitting these production goals while still maintaining data quality.
Significantly, our analysis of Bureau data found that the fast pace of NRFU  was associated with the collection of less-complete household data.  Indeed, after controlling for such variables as response rate and local  enumeration challenges, we found that LCOs with higher percentages of  proxy interviews and closeout interviews were more likely to have  finished NRFU in 53 days or less (the average amount of time LCOs took  to complete their NRFU workloads) compared to LCOs with lower  percentages of proxy and closeout interviews. As noted above, proxy  interviews contain data provided by a nonhousehold member (e.g., a  neighbor) and may thus be less reliable than information collected directly  from a household member, while a closeout interview is one where no  interview is conducted and household information is later determined  using statistical methods at Bureau headquarters during data processing.
The pace of NRFU and its potential effect on data quality was also  reflected in the responses of a number of LCO managers we surveyed. For  example, although almost half of the LCO managers responding to our  survey said they were satisfied with their ability to meet production goals  while maintaining data quality, almost 30 percent of respondents were  dissatisfied with their ability to meet production goals while maintaining  data quality (around 20 percent responded that they were neither satisfied  nor dissatisfied). Some of the LCO managers commented that they felt  undue pressure to finish the operation early (sometimes a month earlier  than planned) and as a result, finishing early could have come at the  expense of accuracy.
In one example, an LCO manager noted that it appeared as though the  LCOs were in a race to finish NRFU as fast as possible, even if the best  data were not received. Another LCO manager said that even though his  office was surpassing the daily production goals, he was still being  pressured to finish faster, and that accuracy was not mentioned. Also, LCO  managers expressed frustration at production goals being changed  frequently or unexpectedly moved earlier.
Further, during our field visits, some LCO managers we spoke with at the  start of NRFU were concerned about meeting production goals as there  were not enough assignment area (AA) binders containing maps and  address registers for every enumerator due to problems with the Bureau’s  Paper-Based Operations Control System (PBOCS), a key IT system that we  discuss below. To ensure that enumerators had sufficient work, some  crew leaders split-up AA binders between two or more enumerators. This  is contrary to Bureau procedures which require enumerators to have their  own AA binder. When the binders are split, only one enumerator has the  required maps. Without maps an enumerator is unable to determine an  assignment area’s boundaries and ensure that the locations of all housing  units are accurately recorded, which can affect data quality.
Later in NRFU, managers at two LCOs we visited said they felt pressure to  finish NRFU ahead of schedule. At one LCO, managers explained that the  regional office wanted to finish NFRU by June 12, or approximately 4  weeks ahead of schedule. However, at that LCO they were only 85 percent  complete by week 5, and because NRFU procedures instruct enumerators  to make up to six attempts to contact a household, they were not sure how  they were going to finish by week 5 without having to accept more refusals  and incomplete interviews—leading to potentially more proxy and  closeout interviews, thus reducing data quality.
At the other LCO, production goals were stretched 15 percentage points  above the national goal in order to complete NRFU ahead of schedule. One  of the field supervisors at that office told us that he was able to meet the  revised production goals by having enumerators share their workload. For  example, in the morning, one enumerator would work the AA, and any  remaining cases were given to another enumerator in the evening to  complete. While this approach might have enhanced efficiency, the sharing  of enumerator assignments makes it more difficult for the Bureau’s  quality-assurance procedures to identify enumerators that are not  following procedures and may need to be retrained. Under the Bureau’s  procedures, AAs are to be assigned to one enumerator at a time.
In late-May 2010, while NRFU was still underway, we discussed the pace  of the operation with Bureau officials, and whether enumerators were  more often accepting less-complete household information. In response,  Bureau officials notified the LCOs and reminded them of the importance  of following prescribed procedures. Moving forward, as the Bureau  conducts its evaluations of its 2010 NRFU operation and begins planning  for 2020, it will be important for Bureau officials to closely examine the  quality of data collected during NRFU and the pace of the operation, and  determine whether it is placing appropriate emphasis on both objectives.
section_title: The Bureau Improved Its Procedures for Fingerprinting Employees, but More Work Is Needed
paragraphs: To better screen its workforce of hundreds of thousands of temporary  census workers, the Bureau fingerprinted its temporary workforce for the  first time in the 2010 Census. In past censuses, temporary workers were  only subject to a name background check that was completed at the time  of recruitment. The Bureau, however, encountered problems capturing  fingerprints during address canvassing, an operation that the Bureau  conducted in the summer of 2009 to verify every address in the country.  According to the Bureau, 22 percent of the approximately 162,000 workers  hired for address canvassing had unclassifiable prints, or fingerprints that  were incomplete or unreadable. The Federal Bureau of Investigation (FBI)  determined that this problem was generally the result of errors that  occurred when the prints were first taken at the LCOs that affected the  readability of the two fingerprint cards that were created for each  individual.
To address these problems, the Bureau improved its training procedures  and purchased additional equipment in order to fingerprint some 580,000  NRFU temporary employees. Specifically, the Bureau refined training  manuals used to instruct LCO staff on how to take fingerprints, scheduled  fingerprint training closer to when the prints were captured, and increased  the length of training. Further, the Bureau used an oil-free lotion during  fingerprinting that is believed to raise the ridges on fingertips to improve  the legibility of the prints.
The Bureau also revised its procedures to digitally capture a third and  fourth set of fingerprints when the first two sets of fingerprint cards could  not be read. The Bureau purchased around a thousand digital fingerprint  scanners (see fig. 4) for this new effort. The Bureau estimated that this  additional step could reduce the percentage of temporary workers with  unclassifiable prints from 22 percent to approximately 10 to 12 percent, or  an estimated 60,000 to 72,000 temporary workers for NRFU. As of May 25,  2010, it reduced the percentage of temporary workers with unclassifiable  prints to 8.6 percent of 635,251, or approximately 54,000 temporary  workers.
Fingerprint cards were sent from each LCO to the Bureau’s National  Processing Center (NPC) in Indiana where they were scanned and sent to  the FBI. We visited the NPC during peak processing and observed that  NPC was able to adequately handle the workload without any glitches.  However, capturing fingerprints at training sites did not go as well. Some  LCOs mentioned that collecting fingerprints took more time than  expected, thus reducing the time available for enumerator field training. In  our observations, at one LCO it took an extra 2 hours to fingerprint  enumerators, and at another fingerprinting took so long it carried over to  the next day (which put the NRFU instructor behind schedule).  Furthermore, almost 50 percent of LCO managers responding to our  survey reported dissatisfaction with fingerprinting procedures, compared  to about 30 percent of LCO managers who were satisfied. For example,  LCO managers commented that they did not have enough time to train  staff conducting the fingerprinting or did not have adequate fingerprinting  supplies, such as cards and ink pads. Several LCO managers said that the  process was time-consuming, yet the additional time spent did not  produce higher-quality prints, possibly because staff did not have  fingerprinting expertise. Although some LCO managers said they would  have preferred more digital fingerprinting, others reported that the digital  fingerprint scanners did not work well and were time-consuming to use. In  looking forward to 2020, the Bureau should revise or modify training so  that field staff are provided with numerous practice opportunities for  collecting fingerprints prior to each operation.
section_title: Workarounds Helped Mitigate PBOCS Issues, but Continuing Problems Hampered the Implementation of Key Quality-Assurance Procedures
paragraphs: Since 2005, we have reported on weaknesses in the Bureau’s management  and testing of key 2010 Census IT systems. Although the IT systems  ultimately functioned well enough for the Bureau to carry out the census,  workarounds developed to address performance problems with PBOCS— a workflow-management system crucial for the Bureau’s field  operations—adversely affected the Bureau’s ability to implement key  quality-assurance procedures as planned.
In June 2005, we noted that the Bureau had not fully implemented key  practices important to managing IT, including investment management,  system development and management, and enterprise architecture  management. As a result, we concluded that the Bureau’s IT investments  were at increased risk of mismanagement, and were more likely to  experience cost and schedule overruns and performance shortfalls.
As development of the IT systems progressed, these problems were  realized. For example, the Field Data Collection Automation program,  which included the development of handheld computers to collect  information for address canvassing and NRFU, experienced substantial  schedule delays and cost increases. As a result, the Bureau later decided  to abandon the planned use of handheld data collection devices for NRFU  and reverted to paper questionnaires. The Bureau developed PBOCS to  manage the operation. However, as we stated in several testimonies, even  with the approach of Census Day, PBOCS had not demonstrated the ability  to function reliably under full operational loads required to complete  NRFU. We noted that the limited amount of time remaining to improve its  reliability before it would be needed for key operations created a  substantial challenge for the Bureau.
Although the Bureau worked aggressively to improve PBOCS  performance, the system experienced significant issues at the start of  NRFU. For example, despite efforts to upgrade its hardware and software,  PBOCS continued to experience system outages, slow performance, and  problems generating and maintaining timely progress reports. The Bureau  has attributed these issues, in part, to the compressed development and  testing schedule, as well as to inadequate performance and interface  testing.
To mitigate the system’s performance issues, the Bureau implemented  various workarounds. For example, the Bureau frequently restricted the  number of hours that PBOCS was available to users in order to implement  software upgrades and perform other system maintenance activities. In  addition, the Bureau restricted the number of concurrent PBOCS users at  each LCO to help reduce demand on the system. These restrictions often  limited the number of concurrent users to 3 to 5 users per LCO, or about  1,500 to 2,500 total users. According to a Bureau official with  responsibility for PBOCS, the system was originally intended to provide  access for over 7,000 concurrent users. While these workarounds  improved the reliability of PBOCS, LCO managers who responded to our  survey were consistently dissatisfied with the restrictions on the number  of users allowed at one time, and many commented that the restrictions  adversely affected their ability to keep up with the workload. Further, the  limitations on the number of concurrent users, combined with PBOCS  outages and slow performance, delayed the shipping of questionnaires to  the data capture centers and resulted in a peak backlog of nearly 12  million questionnaires at the LCOs.
The substantial backlog of questionnaires hampered the Bureau’s ability to  effectively monitor productivity and data quality during NRFU as planned.  Nearly 75 percent of LCO manager survey respondents were dissatisfied  with the usefulness of PBOCS reports to plan and monitor work during  NRFU. A dissatisfied respondent wrote in that the unavailability of reports  greatly hampered his LCO’s ability to conduct NRFU in an efficient  manner. Almost 80 percent of responding LCO managers indicated that  their LCO needed to put forth a substantial amount of extra effort to  manually prepare reports to track productivity outside of PBOCS. The use  of manual processes increased costs at the LCOs and raised the risk of  human error.
The backlog of questionnaires also hampered the Bureau’s ability to  conduct NRFU reinterviews, a quality-assurance operation designed to  identify enumerators who intentionally or unintentionally produced data  errors. PBOCS was to select a sample of cases from each enumerator’s  completed workload, and these cases would be reinterviewed by another  enumerator. Once cases were selected, a quality-assurance enumerator  attempted to reinterview the original NRFU respondents in an effort to  verify that accurate data was collected during the initial NRFU interview.
However, the backlog of questionnaires delayed the selection of  reinterview cases and, as a result, some could not be conducted. For  example, in areas with large populations of college students, the Bureau  conducted NRFU early in order to maximize the probability of  enumerating people before they were likely to move out from where they  were living on Census Day. In some of those cases, reinterviews could not  be conducted since the students had moved out by the time an enumerator  was given the case for reinterview. In addition, it also took longer to detect  and retrain an enumerator with performance problems. For example, LCO  staff reported to us that, because of the delay carrying out reinterviews, it  was often too late to retrain enumerators because they had already  finished their assignments and were released before the errors were  identified. In cases where an enumerator had intentionally falsified work,  the enumerator was supposed to be released and all his or her work was  to be redone. However, because of the PBOCS delays, falsified cases were  sometimes identified after the enumerator was finished with his or her  assignment, requiring their entire assignment area to be reenumerated.
Identifying errors and falsifications early in the operation would have  minimized the number of housing units that needed to be reworked and  reduced the burden for respondents. For example, an LCO manager told  us that her office was not able to detect an enumerator’s falsification until  after NRFU, when the enumerator had already moved on to the next  operation, requiring the LCO to rework nearly 200 cases. According to our  survey, approximately 30 percent of LCO managers who experienced  backlogs reported that they had substantial difficulty detecting errors or  fraudulent interviewing as a result of the backlog, while more than 20  percent reported moderate difficulty and nearly 50 percent reported slight  to no difficulty detecting errors or fraudulent interviewing as a result of  the backlog.
The implementation of various workarounds helped the Bureau  successfully complete NRFU. However, the lack of a fully functioning  PBOCS limited the Bureau’s ability to effectively monitor productivity or  implement quality-assurance procedures as documented in its operational  plans.
More generally, as the Bureau prepares for 2020, among other actions it  will be important for it to continue to improve its ability to manage its IT  investments. Leading up to the 2010 Census, we made numerous  recommendations to the Bureau to improve its IT management practices  by implementing best practices in risk management, requirements  development, and testing, as well as establishing an IT acquisition- management policy that incorporates best practices. While the Bureau  implemented many of our recommendations, it did not implement our  broader recommendation to institutionalize these practices at the  organizational level. The challenges experienced by the Bureau in  acquiring and developing IT systems during the 2010 Census further  demonstrate the importance of establishing and enforcing a rigorous IT  acquisition management policy Bureau-wide. In addition, it will be  important for the Bureau to improve its ability to consistently perform key  IT management practices, such as IT investment management, system  development and management, and enterprise architecture management.  The effective use of these practices can better ensure that future IT  investments will be pursued in a way that optimizes mission performance.
section_title: Key Follow-up Operations Were Generally Completed as Planned
subsections: section_title: Vacant/Delete Check Operation Finished ahead of Schedule but over Budget
paragraphs: To help ensure that people are counted only once and in the right place, as  well as to collect complete and correct information about them, after  NRFU the Bureau conducts a number of operations designed to improve  the accuracy of the data. One of these operations is the VDC operation,  where enumerators verified the Census Day status of vacant and deleted  (nonexistent) housing units. VDC also attempts to enumerate late  additions to the Bureau’s address file, such as newly constructed housing,  and units for which the mail-out questionnaire was returned blank or  incomplete. The Bureau refers to these additional addresses as  supplemental cases. VDC has the potential to boost the accuracy of the  census, especially among traditionally undercounted populations. A  similar operation in 2000 found that 22 percent of housing units previously  identified as vacant, and 25 percent of those previously flagged for  deletion, were indeed occupied. Changing the status of these units led to a  net gain of 3.1 million people in the 2000 population count.
The Bureau completed the VDC operation on August 23, slightly ahead of  the original planned completion date of August 25, but also over budget.  The Bureau spent about $281 million on VDC, approximately 15 percent  over its baseline budget of $244 million. Bureau officials we spoke to  attributed the operation’s progress to the retention of experienced NRFU  staff for VDC. They noted that VDC staff were knowledgeable about  procedures and the locations in which they worked, and required less  training than they would have if they had been newly hired. With respect  to the cost overruns, the Bureau is analyzing why VDC exceeded its  budget. According to a Bureau official, additional costs may be related to  VDC cases being located farther apart than expected (which would require  more staff time and mileage reimbursement) and to enumerators adding  more new addresses than expected.
The VDC workload of 8.7 million housing units (5.6 million units vacant or  flagged for deletion, 2.9 million supplemental addresses, and 0.2 million  additions during the operation) was substantially less than the Bureau’s  previous estimate of 10.4 to 15.4 million units. During our review we found  that while the Bureau had updated its total cost estimates for VDC, it had  not adjusted day-to-day cost and progress expectations for VDC to  account for the reduced workload. Not having the most recent targets for  VDC could have impeded the Bureau’s ability to effectively monitor the  progress of enumerators in the field. We discussed this with Bureau  officials, and in mid-July they revised VDC cost and progress estimates to  account for the smaller workload, as well as other changes, including an  earlier start date and reduced staffing.
Further, during our field observations, LCO staff told us that some VDC  supplemental addresses had already been enumerated as occupied units  during NRFU. These supplemental addresses were slightly different from  the NRFU addresses (e.g., 123 Main Street versus 123A Main Street) and  appeared to be duplicate addresses. Duplicate addresses are supposed to  be checked during field verification (an operation to confirm the existence  of certain housing units added to the Bureau’s address file) and should not  have been in the VDC workload. Because the issue could indicate a  nationwide problem, we notified Bureau officials, and in response they  instituted a new procedure to identify and process duplicate addresses  without making a follow-up visit to the housing unit. Identifying duplicate  addresses before they get enumerated a second time is important because  unnecessarily visiting a housing unit previously counted can reduce the  accuracy of census data and will increase costs.
In order to assess the reasons why VDC ran over budget, and as  recommended in our June 2008 report, it will be important for the Bureau  to document lessons learned for cost elements whose actual costs differ  from the estimate. Knowing this will allow the Bureau to develop a more  accurate cost estimate for VDC in 2020. In addition, to ensure the accuracy  of data collected during VDC, it will be important for the Bureau to  research how duplicates were inadvertently included in the VDC  workload, as this data will help the Bureau compile a better address list  for VDC operations in 2020.
section_title: Census Coverage Measurement Redesigned with Smaller Sample to Reduce Nonsampling Errors
paragraphs: The Bureau attempts to conduct a complete and accurate count of the  nation’s population; nonetheless, some degree of coverage error is  inevitable because of the inherent complexity of counting the nation’s  large and diverse population and limitations in census-taking methods.  These census coverage errors can take a variety of forms, including a  person missed (an undercount), a person counted more than once (an  overcount), or a person who should not have been counted, such as a child  born after Census Day (another type of overcount). And because census  data are central to so many critical functions, it is essential to assess  census accuracy and improve the process when possible.
Statistical measurements of census coverage are obtained by comparing  and matching the housing units and people counted by an independent  sample or CCM survey to those counted by the census in and around the  sample areas. The Bureau has developed separate address lists—one for  the entire nation of over 134 million housing units that it is using to  conduct the census and one for coverage-measurement sample areas—and  is collecting each set of data through independent operations. The Bureau  collected its CCM data from households in sample areas nationwide, as  part of an operation that began in the middle of August and was completed  in October 2010.
In our April 2010 report, we noted that in December 2009 the Bureau made  numerous changes to the design of CCM that would reduce nonsampling  error—such as human errors made when recording data during  interviews—in CCM and its resulting estimates of census accuracy, thus  providing census data users with more-reliable estimates. These changes  include increasing quality-assurance reinterviewing, hiring more CCM  supervisors, and adding training for interviewers to improve interview  techniques for local or other special situations (such as interviewing  people who became homeless or have had to move frequently during the  housing crisis). The December decision also reduced the CCM sample size  by nearly 45 percent. The Bureau believes that this reduction will generate  cost savings to pay for changes to reduce nonsampling error. We believe  that, overall, these changes are reasonable efforts to improve survey  quality. The Bureau’s reduction in sample size will reduce precision of the  estimates, yet the proposed changes should reduce nonsampling errors  and thus provide users with more-reliable estimates.
Another challenge highlighted in our April 2010 report on CCM was  determining the optimal time to collect data for some 170,000 housing  units during person interviewing (PI), which began August 14 and ended  October 16, 2010. The issue is that if the Bureau starts PI too early, it  increases the chance that it overlaps with census data collection, possibly  compromising the independence of the two different sets of data and  introducing what is referred to as a “contamination bias” error into CCM  data. However, if the Bureau starts PI too late, it increases the chance that  respondents will not accurately remember household information from  Census Day, introducing error (known as “recall bias”) in the CCM count.  In that report we recommended that the Bureau assess the trade-offs  between starting early and introducing contamination bias or starting later  and risking recall bias. The Bureau responded that it planned to study and  measure some recall errors, but that there was no study planned to  measure contamination bias in 2010 due to concerns with the possible  contamination of census results in the study area. However, since both  types of errors—contamination bias and recall bias—could affect the  Bureau’s conclusions about the accuracy of the census, it will be  important for the Bureau to implement our recommendation and assess  the trade-offs between the two types of biases in timing decisions.  Moreover, this assessment could help the Bureau better inform the  optimal timing for future census and coverage-measurement data  collection operations.
section_title: Fundamental Reforms Will Be Needed for a More Cost-Effective Census in 2020
paragraphs: While it will be important to assess and revamp existing census-taking  activities, the results of prior enumerations underscore the fact that simply  refining current methods—many of which have been in place for  decades—will not bring about the reforms needed to control costs while  maintaining accuracy given ongoing and newly emerging societal trends.  Since 1970, the Bureau has used essentially the same approach to count  the vast majority of the population. The Bureau develops an address list of  the nation’s housing units and mails census forms to each one for  occupants to fill out and mail back. Over time, because of demographic  and attitudinal trends, securing an acceptable response rate has become  an increasing challenge, and the Bureau has spent more money with each  census in order to secure a complete count. Indeed, the cost of conducting  the census has, on average, doubled each decade since 1970, in constant  2010 dollars. If that rate of cost escalation continues into 2020, the nation  could be facing a $30 billion census.
Despite the nation’s greater investment in each census, the results are  often no better than the previous decennial. For example, as noted earlier,  while the unit cost of the census jumped from an average of around $70 in  2000 to around $98 in 2010, the mail response rate declined in 48 states.  Our concerns about the rising cost and diminishing returns of the census  are not new. In the mid-1990s, for example, we and others concluded that  the established approach for taking the census in 1990 had exhausted its  potential for counting the population cost-effectively and that fundamental  design changes were needed to reduce census costs and improve the  quality of data collected.
A fundamental reexamination of the nation’s approach to the census will  require the Bureau to rethink its approach to planning, testing,  implementing, monitoring, and evaluating the census, and addressing such  questions as, why was a certain program initiated? What was the intended  goal? Have significant changes occurred that affect its purpose? Does it  use prevailing leading practices?
As one example, a critical factor affecting the cost of the census is the  necessity for the Bureau to follow up on nonresponding housing units. The  hourly wages of enumerators, their productivity, mileage reimbursement,  and the need, in some cases, to return several times to an address to  obtain a response can all drive up costs. Administrative records from other  government agencies including driver licenses and school records can, if  used in lieu of making multiple visits to a housing unit, significantly  control costs. However, the Bureau would first need to resolve a number  of questions including the quality and the coverage of the information  supplied by the records and the policy and legal implications of accessing  them.
On the basis of our earlier work on high-performing organizations,  fundamental reforms will mean ensuring that the Bureau’s organizational  culture and structure, as well as its approach to strategic planning, human- capital management, internal collaboration, knowledge sharing, capital  decision making, risk and change management, and other internal  functions are aligned toward delivering more cost-effective outcomes.  Indeed, some of the operational problems that occurred during the 2010  and prior censuses are symptomatic of deeper organizational issues. For  example, the lack of staff skilled in cost-estimation during the 2010 Census  points to inadequate human-capital planning, while IT problems stemmed  from not fully and consistently performing certain functions including IT  investment management.
Going forward, it will be important for the Bureau, Congress, and other  stakeholders to reach consensus on a number of reexamination areas,  including the following, which have particular implications for controlling  costs and improving accuracy:    Which data collection approaches, the Internet and administrative records  among them, have potential to improve data quality without compromising  other Bureau goals and mandates such as confidentiality and timeliness?
To what extent can private-sector and other sources of information such  as maps, address lists, and geographic databases be employed to help  support the census?
How can the Bureau enhance how it partners with government and  nongovernmental organizations, data users, grassroots organizations, and  advisory groups to obtain their input and possibly better leverage their  knowledge and services? What is the best way of maintaining  congressional and stakeholder involvement and dialog throughout the  course of the decade?
What opportunities exist for the Bureau to leverage innovations in  technology and social media to more fully engage census stakeholders and  the general public throughout the decade on census issues, possibly  identifying more cost-effective methods?
To what extent can the Bureau use the American Community Survey—an  ongoing Bureau survey of population and housing characteristics that is  conducted throughout the decade—as a platform to test new census  methods and systems?
What are the implications of the Bureau’s goal to conduct the 2020 Census  at a lower cost than the 2010 Census on a cost per housing unit basis,  adjusted for inflation? For example, how would this spending limit affect  such considerations as accountability and data quality?
How can the Bureau best balance the acquisition of advanced technology,  some of which might not be fully mature until later in the decade, with the  need to commit to particular systems sufficiently early in the decade to  ensure the systems are fully tested and will work under census-like  conditions?
To what extent can the Bureau control costs and improve accuracy by  targeting census-taking activities using local response rate and  sociodemographic information from the 2010 Census, as well as other data  sources and empirical evidence?
What options exist for controlling the costs of particularly labor-intensive  operations such as NRFU and building the Bureau’s master address list  without sacrificing accuracy?
Can stakeholders reach agreement on a set of criteria that could be used  to weigh the trade-offs associated with the need for high levels of accuracy  on the one hand, and the increasing cost of achieving that accuracy on the  other hand?
The Bureau, recognizing that it cannot afford to continue operating the  way it does unless it fundamentally changes its method of doing business,  has already taken some important first steps in addressing these questions  as well as other areas. For example, the Bureau is looking to reform  certain aspects of its IT systems planning, in part to ensure that the  technical infrastructure needed for 2020 will be tested many times before  operations begin. The Bureau is also rebuilding its research directorate to  lead early planning efforts, and has plans to assess and monitor the skills  and competencies needed for the 2020 headcount and evaluate the  feasibility of administrative records.
Further, the Bureau has already developed a strategic plan for 2020 and  other related documents that, among other things, lay out the structure of  the Bureau’s planning efforts; outline the Bureau’s mission and vision for  2020 and the goals the Bureau seeks to meet to accomplish its mission;  and describe the Bureau’s plans for the research and testing phase of the  next enumeration.
The Bureau’s early planning efforts are noteworthy given the Bureau’s  long-standing challenges in this area. For example, in 1988, just prior to  the 1990 Census, we noted that the Bureau’s planning efforts generally  started late, experienced delays, were incomplete, and failed to fully  explore innovative approaches. Planning for the 2000 Census also had its  shortcomings. According to the Bureau, staff with little operational  experience played key roles in the design process, which resulted in  impractical reform ideas that could not be implemented. We also noted  that the 2000 Census suffered from a persistent lack of priority-setting,  coupled with minimal research, testing, and evaluation documentation to  promote informed and timely decision making. And, while the planning  process for the 2010 Census was initially more rigorous than for past  decennials, in 2004 we reported that the Bureau’s efforts lacked a  substantial amount of supporting analysis, budgetary transparency, and  other information, making it difficult for us, Congress, and other  stakeholders to properly assess the feasibility of the Bureau’s design and  the extent to which it could lead to greater cost-effectiveness compared to  alternative approaches. As a result, in 2004, we recommended that the  Bureau develop an operational plan for 2010 that consolidated budget,  methodological, and other relevant information into a single,  comprehensive document.
Although the Bureau later developed specific performance targets and an  integrated project schedule for 2010, the other elements we recommended  were only issued piecemeal, if available at all, and were never provided in  a single, comprehensive document. Because this information was critical  for facilitating a thorough, independent review of the Bureau’s plans, as  well as for demonstrating to Congress and other stakeholders that the  Bureau could effectively design and manage operations and control costs,  we believe that had it been available, it could have helped stave off, or at  least reduce, the IT and other risks that confronted the Bureau as Census  Day drew closer.
The Bureau’s strategic plan for 2020, first issued in 2009, is a “living”  document that will be updated as planning efforts progress. As the  approach for 2020 takes shape, it will be important for the Bureau to avoid  some of the problems it had in documenting the planning process for the  2010 Census, and pull all the planning elements together into a tactical  plan or road map. This will help ensure the Bureau’s reform initiatives stay  on track, do not lose momentum, and coalesce into a viable path toward a  more cost-effective 2020 Census. On the basis of our work on planning for  the 2010 Census, a road map for 2020 could include, but not be limited to,  the following elements that could be updated on a regular basis:    specific, measurable performance goals, how the Bureau’s efforts,  procedures, and projects would contribute to those goals, and what  performance measures would be used;    descriptions of how the Bureau’s approaches to human-capital  management, organizational structure, IT acquisitions, and other internal  functions are aligned with the performance goals;    an assessment of the risks associated with each significant decennial  operation, including the interrelationships between the operations and a  description of relevant mitigation plans;    detailed milestone estimates for each significant decennial operation,  including estimated testing dates, and justification for any changes to  milestone estimates;   detailed life-cycle cost estimates of the decennial census that are credible,  comprehensive, accurate, and well-documented as stipulated by Office of  Management and Budget and GAO guidance; and     a detailed description of all significant contracts the Bureau plans to enter  into and a justification for the contracts.
A comprehensive road map could generate several important benefits. For  example, it could help ensure a measure of transparency and facilitate a  more collaborative approach to planning the next census. Specifically, an  operational plan could function as a template for 2020 giving stakeholders  a common framework to assess and comment on the design of the census  and its supporting infrastructure, the resources needed to execute the  design, and the extent to which it could lead to greater cost-effectiveness  compared to alternative approaches. Further, it could be used to monitor  the Bureau’s progress in implementing its approach, and hold the agency  accountable for results. Importantly, to the extent the plan—or aspects of  it—are made available using social media tools, it could prompt greater  and perhaps more constructive civic engagement on the census, by  fostering an ongoing dialog involving individuals and communities of  stakeholders throughout the decade. On December 8, 2010, the Senate  approved a bill, the Census Oversight Efficiency and Management Reform  Act of 2010. If enacted, this bill, among its other provisions, would  require the Director of the Census to submit an annual comprehensive  status report on the next decennial census, beginning with the 2020  decennial census, to the appropriate congressional committees. The  specific requirements in the bill for the annual plan include most of the  elements discussed above.
Given the magnitude of the planning and transformation efforts facing the  Bureau, another reexamination question is that of long-term stewardship  governing the endeavor. Specifically, as the research, development, and  testing efforts for 2020 will play out over the decade-long census life cycle,  what is the optimal way to ensure continuity and accountability for an  enterprise that takes years to complete and extends beyond the tenure of  many elected political leaders?
Although the Director of the Census Bureau can, in concept, provide a  measure of continuity, of the 11 census directors that have served since  July 1969 (not including the current director), the average tenure was  around 3 years, and only one director has served more than 5 years.  Moreover, in the decade leading up to the 2010 Census, the Bureau was led  by four different directors and several acting directors. The turnover in the  Bureau’s chief executive officer position makes it difficult to develop and  sustain efforts that foster change, produce results, mitigate risks, and  control costs over the long term.
Currently, census directors are nominated by the President with Senate  confirmation. At the same time, the heads of a number of executive  agencies serve fixed appointments, including the Director of the Office of  Personnel Management (4 years), the Commissioner of Labor Statistics (4  years), and the Commissioner of Internal Revenue (5 years).
The census bill, recently passed by the Senate and discussed above,  includes a provision for a 5-year tenure for the Census Director. We  believe that the continuity resulting from a fixed-term appointment could  provide the following benefits to the Bureau:   Strategic vision. The Director needs to build a long-term vision for the  Bureau that extends beyond the current decennial census. Strategic  planning, human-capital succession planning, and life-cycle cost estimates  for the Bureau all span the decade.
Sustaining stakeholder relationships. The Director needs to continually  expand and develop working relationships and partnerships with  governmental, political, and other professional officials in both the public  and private sectors to obtain their input, support, and participation in the  Bureau’s activities.
Accountability. The life-cycle cost for a decennial census spans a decade,  and decisions made early in the decade about the next decennial census  guide the research, investments, and tests carried out throughout the  entire 10-year period. Institutionalizing accountability over an extended  period may help long-term decennial initiatives provide meaningful and  sustainable results.
Overall, the obstacles to conducting a cost-effective census have grown  with each decade, and as the Bureau looks toward the next enumeration,  it might confront its biggest challenge to date. As the Bureau’s past  experience has shown, early investments in planning can help reduce the  costs and risks of its downstream operations. Therefore, while Census Day  2020 is 10 years away, it is not too early for stakeholders to start  considering the reforms needed to help ensure the next headcount is as  cost-effective as possible.
section_title: Conclusions
paragraphs: Although the complete results of the 2010 Census are still some years  away, several preliminary lessons learned for the next enumeration have  already begun to emerge. They include the benefits of a replacement  questionnaire, the removal of late mail returns from the NRFU workload,  and hiring locally. Focus areas for improvement include revisiting the  Bureau’s staffing model, ensuring the Bureau emphasizes quality as well as  production during NRFU, better IT management, and ensuring a high- quality address file is used to carry out VDC operations.
That said, perhaps the most important lesson learned comes from the  collective experience gained from the 1990, 2000, and now 2010  enumerations: the Bureau goes to great lengths each decade to improve  specific census-taking activities, but these incremental modifications have  not kept pace with societal changes that make the population increasingly  difficult to locate and count cost-effectively. Therefore, as the Bureau  looks toward 2020, it will be important for it to reexamine both the  fundamental design of the enumeration, as well as its management and  culture to ensure that the Bureau’s business practices and systems  enhance its capacity to conduct an accurate count, control costs, manage  risks, and be more nimble in adapting to social, demographic,  technological, and other changes that can be expected in the years ahead.
The Bureau is taking some initial steps toward rethinking the census. At  the same time, past experience has shown that the Bureau cannot plan and  execute a successful enumeration on its own. Indeed, the noteworthy  achievements of the 2010 Census occurred because of the shared efforts of  the Bureau, and its parent organizations the Department of Commerce and  the Economics and Statistics Administration, Congress, and thousands of  other parties. It will be important for these and additional stakeholders to  maintain their focus on the census throughout the decade in order to  achieve desired results. Certain census reforms could require legislative  changes, and any new procedures will need to be thoroughly vetted,  tested, and refined. Although the next enumeration is 10 years away, the  groundwork for building a new census infrastructure is already under way.  The bottom line is that while the urgency of the 2010 Census has subsided,  it is by no means any less important to the nation.
section_title: Recommendations for Executive Action
paragraphs: As the Bureau plans for the next decennial census in 2020, in order to  support efforts to reexamine the fundamental design of the decennial  census, and help refine existing operations should they be used again in  the 2020 Census, we recommend that the Secretary of Commerce direct  the Under Secretary of the Economics and Statistics Administration, as  well as the Census Director, to take the following six actions:    To help enhance the Bureau’s performance and accountability, improve  the transparency of the planning process, gauge whether the Bureau is on- track toward a more cost-effective 2020 Census, and foster greater public  dialog about the census, the Bureau should develop an operational plan or  road map for 2020 that integrates performance, budget, methodological,  schedule, and other information that would be updated as needed and  posted on the Bureau’s Web site and other social media outlets, and  develop a mechanism that allows for and harnesses input from census  stakeholders and individuals.
To refine its approach to recruiting, the Bureau should evaluate current  economic factors that are associated with and predictive of employee  interest in census work, such as national and regional unemployment  levels, and use these available data to determine the potential temporary  workforce pool and adjust its recruiting approach.
To help ensure that the Bureau’s procedures for NRFU result in the  collection of high-quality data, the Bureau’s procedures for the timely  completion of NRFU should emphasize the collection of high-quality data  and proper enumeration techniques as much as speed.
To improve the fingerprinting process of temporary workers, the Bureau  should revise or modify training so that field staff are provided with  numerous practice opportunities for collecting fingerprints prior to each  operation.
To ensure that the Bureau improves its ability to manage future IT  acquisitions, the Bureau should immediately establish and enforce a  system-acquisition management policy that incorporates best practices in  system- and software-acquisition management.
To help ensure the Bureau compiles an accurate address list for VDC  operations in 2020, the Bureau should research how duplicate addresses  were inadvertently included in the VDC workload.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce provided written comments on a draft of this  report on December 7, 2010. The comments are reprinted in appendix II.  The Department of Commerce generally agreed with the overall findings  and recommendations of the report. In addition, the Secretary of  Commerce provided the Bureau’s technical comments and suggestions  where additional context might be needed, and we revised the report to  reflect these comments where appropriate.
The Bureau noted that our report did not acknowledge the steps it took to  modify its recruiting plans prior to NRFU. However, we do discuss the  Bureau’s modifications to its recruiting plans. Specifically, we stated that  “once the Bureau had an adequate pool of candidates for 2010, it  attempted to limit the number of additional applicants, taking such steps  as discontinuing the advertising of census jobs in mailed out census  materials.”
The Bureau also commented that it wanted to discuss our analysis that  found that the fast pace of NRFU was associated with the collection of  less-complete household data, noting that its own analysis of a similar  question did not yield the same finding. On December 7, 2010, we met with  Bureau staff to discuss the methodologies and variables used in each  analysis. After discussing our methodology and results, Bureau staff  explained that their analysis was preliminary and not as comprehensive as  our analysis. Further, they acknowledged that they used a different  methodology and different variables.
The Bureau, in commenting on our finding related to fingerprinting  temporary workers, noted that it was unclear as to ways in which  extending training, which usually requires spending more time and money,  would streamline fingerprinting efforts. To clarify this section we changed  the body of the report. The text now reads, “In looking forward to 2020,  the Bureau should revise or modify training so that field staff are provided  with numerous practice opportunities for collecting fingerprints prior to  each operation.”
We are sending copies of this report to the Secretary of Commerce, the  Under Secretary of Economic Affairs, the Director of the U.S. Census  Bureau, and interested congressional committees. The report also is  available at no charge on GAO’s Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please  contact Robert Goldenkoff at (202) 512-2757 or by e-mail at  goldenkoffr@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to assess the implementation of (1)  nonresponse follow-up (NRFU), the largest and most costly census field  operation, where the U.S. Census Bureau (Bureau) sends enumerators to  collect data from households that did not mail back their census forms,  and (2) other key follow-up field operations that were critical for ensuring  a complete count; and (3) identify key questions and focus areas that will  be important for the Bureau, Congress, and census stakeholders to  consider going forward now that planning for the next enumeration is  underway.
To meet our objectives, we used a combination of approaches and  methods to examine the conduct of these operations. These included  statistical analyses; surveys of the Bureau’s 494 local census office (LCO)  managers; analysis of mail response and form check-in rates from Bureau  cost and progress systems; interviews with key bureau headquarters  officials and LCO managers and staff; observation of LCO’s NRFU  operations; and reviews of relevant documentation including our prior  work on the planning and implementation of the 1990, 2000, and 2010  Censuses.
To examine the factors that affected the implementation of NRFU and  vacant/delete check operations (VDC), we interviewed LCO managers and  other supervisory staff and observed operations at 28 LCOs we visited  across the country. We selected LCOs because they were located in hard- to-count areas as determined by data from the 2000 Census. To make these  selections, we also used other factors such as their percentage of rural  population to obtain diversity in urban/rural populations, and proximity to  hurricane-affected areas. Selections for VDC observations were based  primarily on locations with high rates of vacant and delete classifications,  and they were chosen to include a mix of urban, suburban, and rural LCO  located in all regions of the country. (See below for a complete list of the  offices we visited.) During these visits, which took place from April to July  2010, we observed office operations to see how office staff were  processing questionnaires using the Paper-Based Operations Control  System (PBOCS) and capturing fingerprints with live scanners, attended  enumerator training, and observed enumerators in the field go door-to- door to collect census data for NRFU, NRFU reinterview, and VDC.  Because offices were judgmentally selected, our findings from these visits  cannot be projected to the universe of LCOs.
To obtain a national perspective on the conduct of NRFU and other field  data collection operations, we conducted a panel survey of all 494 LCO  managers from March to August 2010 using six questionnaires. The survey  was designed to examine (1) factors that affect the cost and performance  of local data collection efforts, and (2) LCO managers’ satisfaction with  information technology (IT) systems and other management support  functions. Response rate was at least 75 percent for each survey  questionnaire.
The practical difficulties of developing and administering a survey may  introduce errors—from how a particular question is interpreted, for  example, or from differences in the sources of information available to  respondents when answering a question. Therefore, we included steps in  developing and administering the questionnaire to minimize such errors.  For instance, we conducted pretests to check that (1) the questions were  clear and unambiguous, (2) terminology was used correctly, (3) the  questionnaire did not place an undue burden on agency officials, (4) the  information could feasibly be obtained, and (5) the survey was  comprehensive and unbiased. Pretest sites were selected for each wave to  emphasize variation among urban and rural LCOs. Pretests were  conducted over the phone, mostly as cognitive pretests in which the  respondent completed the survey during the pretest. We made changes to  the content and format of the questionnaire after review by a GAO survey  expert and after each of the pretests, based on the feedback we received.
To examine whether the pace of NRFU was associated with the collection  of less-complete data, in addition to the efforts described above, we  analyzed Bureau proxy and closeout rates, and the time it took for an LCO  to complete the NRFU workload. In order to determine whether the  durations of 2010 NRFU production activities were associated with lower- quality work, we conducted regression analyses using data from the  Bureau’s Cost and Progress System, PBOCS, and Matching Review and  Coding System (MaRCS). These analyses assessed whether indicators of  lower-quality enumeration such as the collection of proxy data from a  neighbor and closeout interviews, where a housing unit is occupied but no  interview was obtained, were associated with the number of days that the  LCO spent conducting NRFU production activities, after adjusting for  other factors associated with the timeliness of completion and workload.  We used two regression models: one model tested the association between  the number of days it took each LCO to complete 100 percent of its  workload and quality factors; the other regression model tested the  association between quick completion and quality factors. We also  analyzed cost data weekly for both NRFU and VDC to determine whether  those operations were within their respective budgets.
To assess the reliability of the data, we reviewed Bureau electronic  documentation to gain information about the data and their sources. We  examined data from the Bureau’s Cost and Progress, PBOCS, and MaRCS  systems to check for logical errors and inconsistencies, and followed up  with agency officials knowledgeable about the data in cases where we had  questions about potential errors or inconsistencies, and to inquire about  the accuracy and completeness of the entry and processing of the data.  Values are updated by the Bureau throughout the operations, and may be  revised by the Bureau even after the operations close. On the basis of our  efforts, we determined that the data were sufficiently reliable for the  purposes of this engagement.
Finally, to identify preliminary steps the Bureau can take to help transform  its management and culture, we reviewed our prior work on  governmentwide reexamination, as well as leading practices and attributes  in the areas of IT management, organizational performance, collaboration,  stewardship, and human capital. In addition, we reviewed census planning  material, prior GAO work on census planning and development efforts,  and spoke with Bureau officials about their needs and plans for  management and cultural transformation.
We conducted this performance audit from December 2009 until  December 2010 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audits to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Local Census Offices Visited in This Review
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Other key contributors to this report include Lisa Pearson, Assistant  Director; Mark Abraham; David Bobruff; Benjamin Crawford; Sara Daleski;  Dewi Djunaidy; Vijay D’Souza; Elizabeth Fan; Ronald Fecso; Robert  Gebhart; Richard Hung; Kristen Lauber; Andrea Levine; Ty Mitchell;  Kathleen Padulchick; Michael Pahr; Tind Ryen; Jonathan Ticehurst;  Timothy Wexler; Holly Williams; Monique Williams; and Kate Wulff.
section_title: Related GAO Products
paragraphs: 2010 Census: Key Efforts to Include Hard-to-Count Populations Went  Generally as Planned; Improvements Could Make the Efforts More  Effective for Next Census. GAO-11-45. Washington, D.C.: December 14,  2010.  2010 Census: Follow-up Should Reduce Coverage Errors, but Effects on  Demographic Groups Need to Be Determined. GAO-11-154. Washington,  D.C.: December 14, 2010.  2010 Census: Plans for Census Coverage Measurement Are on Track, but  Additional Steps Will Improve Its Usefulness. GAO-10-324. Washington,  D.C.: April 23, 2010.  2010 Census: Data Collection Is Under Way, but Reliability of Key  Information Technology Systems Remains a Risk. GAO-10-567T.  Washington, D.C.: March 25, 2010.  2010 Census: Key Enumeration Activities Are Moving Forward, but  Information Technology Systems Remain a Concern. GAO-10-430T.  Washington, D.C.: February 23, 2010.  2010 Census: Census Bureau Continues to Make Progress in Mitigating  Risks to a Successful Enumeration, but Still Faces Various Challenges.  GAO-10-132T. Washington, D.C.: October 7, 2009.  2010 Census: Census Bureau Should Take Action to Improve the  Credibility and Accuracy of Its Cost Estimate for the Decennial Census.  GAO-08-554. Washington, D.C.: June 16, 2008.  2010 Census: Census at Critical Juncture for Implementing Risk  Reduction Strategies. GAO-08-659T. Washington, D.C.: April 9, 2008.
Information Technology: Significant Problems of Critical Automation  Program Contribute to Risks Facing 2010 Census. GAO-08-550T.  Washington, D.C.: March 5, 2008.
Information Technology: Census Bureau Needs to Improve Its Risk  Management of Decennial Systems. GAO-08-259T. Washington, D.C.:  December 11, 2007.
Information Technology Management: Census Bureau Has Implemented  Many Key Practices, but Additional Actions Are Needed. GAO-05-661.  Washington, D.C.: June 16, 2005.  21st Century Challenges: Reexamining the Base of the Federal  Government. GAO-05-325SP. Washington, D.C.: February 1, 2005.
Information Technology Investment Management: A Framework for  Assessing and Improving Process Maturity. GAO-04-394G. Washington,  D.C.: March 1, 2004.
Comptroller General’s Forum, High-Performing Organizations: Metrics,  Means, and Mechanisms for Achieving High Performance in the 21st  Century Public Management Environment. GAO-04-343SP. Washington,  D.C.: February 13, 2004.
Human Capital: Key Principles for Effective Strategic Workforce  Planning. GAO-04-39. Washington, D.C.: December 11, 2003.