2000 Census: Best Practices and Lessons Learned for More Cost-Effective Nonresponse Follow-up
Released: 2002-02-11T00:00:00

Letter

Background

In conducting nonresponse follow-up, the bureau has historically faced the twin challenge of (1) collecting quality data (by obtaining complete and accurate information directly from household members) while (2) finishing the operation on schedule, before error rates can increase as people move or have trouble recalling who was living at their homes on Census Day (April 1), as well as keeping subsequent operations on-track. Nonresponse follow-up was scheduled to begin on April 27, 2000, and end 10 weeks later, on July 7, 2000.

Local census offices generally finished their nonresponse follow-up workloads ahead of the bureau’s 10-week schedule. As shown in figure 1, of the bureau’s 511 local offices in the 50 states, 463 (91 percent) finished nonresponse follow-up by the end of the eighth week of the operation, consistent with the bureau’s internal stretch goals. Moreover, nine local offices completed their workloads in as little as 5 weeks or less.

The timely completion of nonresponse follow-up in 2000 stands in sharp contrast to the bureau’s experience during the 1990 Census. As shown in figure 2, at the end of the 6-week scheduled time frame for nonresponse follow-up during the 1990 Census, the bureau had not completed the operation. In fact, as of two days prior to the scheduled end date, just two local census offices had completed the operation and the bureau had only completed about 72 percent of its 34 million household follow-up workload. It took the bureau a total of 14 weeks to complete the entire operation. By comparison, as noted above, the bureau completed nonresponse follow-up in less than 10 weeks during the 2000 Census.

Figure 2 also highlights the drop-off in production that occurs during the later weeks of nonresponse follow-up. According to the bureau, the decline occurs because unresolved cases at the end of nonresponse follow- up are typically the most difficult to reach, either because they are uncooperative or are rarely at home and are unknown to neighbors.

Scope and Methodology

To meet our objectives, we used a combination of approaches and methods to examine the conduct of nonresponse follow-up. These included statistical analyses; interviews with key bureau headquarters officials, regional census center officials, and local census office managers and staff; observations of local census offices’ nonresponse follow-up operations; and reviews of relevant documentation.

To examine the factors that contributed to the timely completion of nonresponse follow-up, we interviewed local census office managers and other supervisory staff at 60 local census offices we visited across the country. These offices generally faced specific enumeration challenges when nonresponse follow-up began in late April, and were thus prone to operational problems that could affect data quality (see app. I for a complete list of the offices we visited). Specifically, these offices had (1) a larger nonresponse follow-up workload than initially planned; (2) multiple areas that were relatively hard-to-enumerate, such as non-English-speaking groups; and (3) difficulties meeting their enumerator recruiting goals. During these visits, which took place in June and July 2000, we also observed office operations to see how office staff were processing questionnaires; at 12 of these offices we attended enumerator training; and at 31 offices we reviewed key reinterview documents in a given week during nonresponse follow-up. The local census offices we visited represent a mix of urban, suburban, and rural locations. However, because they were judgmentally selected, our findings from these visits cannot be projected to the universe of local census offices.

To obtain a broader perspective of the conduct of nonresponse follow-up, we used the results of our survey of a stratified random sample of managers at 250 local census offices. The survey—which asked these managers about the implementation of a number of key field operations— is generalizable to the 511 local census offices located in the 50 states. We obtained responses from managers at 236 local census offices (about a 94 percent overall response rate). All reported percentages are estimates based on the sample and are subject to some sampling error as well as nonsampling error. In general, percentage estimates in this report for the entire sample have confidence intervals ranging from about ± 4 to ± 5 percentage points at the 95 percent confidence interval. In other words, if all managers in our local census office population had been surveyed, the chances are 95 out of 100 that the result obtained would not differ from our sample estimate in the more extreme cases by more than ± 5 percent.

To examine whether the pace of nonresponse follow-up was associated with the collection of less complete data, in addition to the efforts described above, we analyzed bureau data on the weekly progress of nonresponse follow-up. Specific measures we analyzed included the time it took local census offices to finish nonresponse follow-up and the proportion of their cases completed by (1) “close-out” interviews, where questionnaires only contain basic information on the status of the housing unit (e.g., whether it was occupied), or (2) “partial” interviews, which contain more information than a close-out interview but are still less than complete. The completeness of the data collected by enumerators is one measure of the quality of nonresponse follow-up, and these two measures were the best indicators of completeness available from the database. We included data from the 511 offices located in the 50 states and controlled for enumeration difficulty using an index measure developed by the bureau. We did not include any outliers that the bureau identified as erroneous (for example, outliers resulting from coding errors).

We did our audit work at the local census offices identified in appendix I and their respective regional census centers; bureau headquarters in Suitland, Maryland; and Washington, DC, from March 2000 through September 2001. Our work was done in accordance with generally accepted government auditing standards.

We requested comments on a draft of this report from the Secretary of Commerce. On January 10, 2002, the Secretary forwarded the bureau’s written comments on the draft (see app. II) which we address at the end of this report.

The Bureau Used an Aggressive Outreach and Promotion Campaign and Other Strategies to Boost the Mail Response Rate but Public Cooperation Remains Problematic

Key to the bureau’s timely completion of nonresponse follow-up in 2000 was a higher than expected initial mail response rate that decreased the bureau’s follow-up workload. In addition to reducing the staff, time, and money required to complete the census count, the bureau’s past experience and evaluations suggest that the quality of data obtained from questionnaires returned by mail is better than the data collected by enumerators.

To help raise the mail response rate, the bureau (1) hired a consortium of private-sector advertising agencies, led by Young & Rubicam, to develop a national, multimedia paid advertising program, and (2) partnered with local governments, community groups, businesses, nongovernmental organizations, and other entities to promote the census on a grassroots basis (we discuss the bureau’s partnership program in more detail in our August 2001 report). The outreach and promotion campaign encouraged people to complete their census questionnaires by conveying the message that census participation helped their communities. The bureau also helped boost the mail response rate by using simplified questionnaires, which was consistent with our past suggestions, and by developing more ways to respond to the census, such as using the Internet.

The bureau achieved an initial mail response rate of about 64 percent, which was about 3 percentage points higher than the 61 percent response rate the bureau expected when planning for nonresponse follow-up.This, in turn, resulted in a nonresponse follow-up workload of about 42 million housing units, which was about 4 million fewer housing units than the bureau would have faced under its planning assumption of a 61 percent mail response rate.

In addition to surpassing its national response rate goals, the bureau exceeded its own expectations at the local level. Of the 511 local census offices, 378 (74 percent) met or exceeded the bureau's expected response rate. In so doing, these offices reduced their nonresponse follow-up workloads from the expected levels by between 54 and 58,329 housing units. The remaining 133 offices (26 percent) did not meet their expected response rate and the workload at these offices increased from their expected levels by between 279 and 33,402 housing units.

Securing Public Participation While Controlling Costs Remains a Considerable Challenge for the 2010 Census

The bureau’s success in surpassing its response rate goals was noteworthy given the formidable societal challenges it faced. These challenges included attitudinal factors such as concerns over privacy, and demographic trends such as more complex living arrangements. However, as the bureau plans for the next census in 2010, it faces the difficulty of boosting public participation while keeping costs manageable.

As we noted in our December 2001 report, although the bureau achieved similar response rates in 1990 and 2000 (65 percent in 1990 and 64 percent in 2000), the bureau spent far more money on outreach and promotion in 2000: about $3.19 per household in 2000 compared to $0.88 in 1990 (in constant fiscal year 2000 dollars), an increase of 260 percent. Moreover, the societal challenges the bureau encountered in 1990 and 2000 will probably be more complex in 2010, and simply staying on par with the 2000 response rate will likely require an even greater investment of bureau resources.

Further, while the mail response rate provides a direct indication of the nonresponse workload, it is an imperfect measure of public cooperation with the census as it is calculated as a percentage of all forms in the mail- back universe from which the bureau received a questionnaire. Because the mail-back universe includes housing units that the bureau determines are nonexistent or vacant during nonresponse follow-up, a more precise measure of public cooperation is the mail return rate, which excludes vacant and nonexistent housing units. According to preliminary bureau data, the mail return rate for the 2000 Census was 72 percent, a decline of 2 percentage points from the 74 percent mail return rate the bureau achieved in 1990. As shown in figure 3, in 2000, the bureau reduced, but did not reverse, the steady decline in public cooperation that has occurred with each decennial census since the bureau first initiated a national mail- out/mail-back approach in 1970. Bureau officials said they would further examine the reasons for the decline in the return rate as part of its Census 2000 evaluations.

In addition, as shown in figure 4, the results to date show that just three states increased their mail return rates compared to the 1990 Census. Overall, preliminary bureau data shows the change in mail return rates from 1990 through 2000 ranged from an increase of about 1 percentage point in Massachusetts and California to a decline of about 9 percentage points in Kentucky.

The bureau’s outreach and promotion efforts will also face the historical hurdle of bridging the gap that exists between the public’s awareness of the census on the one hand, and its motivation to respond on the other. Various polls conducted for the 2000 Census suggested that the public’s awareness of the census was over 90 percent; and yet, as noted earlier, the actual return rate was much lower—72 percent of the nation’s households. The bureau faced a similar issue in 1990 when 93 percent of the public reported being aware of the census, but the return rate was 74 percent. In our previous work, we noted that closing this gap would be a significant challenge for the bureau, and as the bureau plans for the 2010 Census, it will be important for it to explore approaches that more effectively convert the public’s awareness of the census into a willingness to respond.

Flexible Human Capital Strategies Helped the Bureau Meet Its Recruitment Goals

A second factor that was instrumental to the operational success of nonresponse follow-up was an ample and sufficiently skilled enumerator workforce. Based on anticipated turnover and the expected workload to carry out its four largest field data collection operations—of which nonresponse follow-up was the largest—the bureau set a recruitment goal of 2.4 million qualified applicants. In addition to the sheer volume of recruits needed, the bureau's efforts were complicated by the fact that it was competing for employees in a historically tight national labor market. Nevertheless, when nonresponse follow-up began on April 27, the bureau had recruited over 2.5 million qualified applicants.

The bureau surmounted its human capital challenge with an aggressive recruitment strategy that helped make the bureau a more attractive employer to prospective candidates and ensured a steady stream of applicants. Key ingredients of the bureau’s recruitment efforts included the following: 1. A geographic pay scale with wages set at 65 to 75 percent of local prevailing wages (from about $8.25 to $18.50 per hour for enumerators). The bureau also used its flexibility to raise pay rates for those census offices that were encountering recruitment difficulties.

For example, a manager at one of the Charlotte region’s local census offices told us that the office was having difficulty obtaining needed staff in part because census wages were uncompetitive. According to this manager, the region approved a pay increase for the office’s enumerators and office clerks, which helped the office obtain staff. In all, when nonresponse follow-up began, the bureau raised pay rates for field staff at eight local offices to address those offices’ recruiting challenges. 2. Partnerships with state, local, and tribal governments, community groups, and other organizations to help recruit employees and provide free facilities to test applicants. For example, Clergy United, an organization representing churches in the Detroit metropolitan area, provided space for testing census job applicants in December 1998. The organization even conducted pre-tests several days before each bureau-administered test so those applicants could familiarize themselves with the testing format. 3. A recruitment advertising campaign, which totaled over $2.3 million, that variously emphasized the ability to earn good pay, work flexible hours, learn new skills, and do something important for one’s community. Moreover, the advertisements were in a variety of languages to attract different ethnic groups, and were also targeted to different races, senior citizens, retirees, and people seeking part-time employment. The bureau advertised using traditional outlets such as newspaper classified sections, as well as more novel media including Internet banners and messages on utility and credit card bills. 4. Obtaining exemptions from the majority of state governments so that individuals receiving Temporary Assistance for Needy Families, Medicaid, and selected other types of public assistance would not have their benefits reduced when earning census income, thus making census jobs more attractive. At the start of nonresponse follow-up, 44 states and the Virgin Islands had granted an exemption for one or more of these programs. 5. Encouraging local offices to continue their recruiting efforts throughout nonresponse follow-up, regardless of whether offices had met their recruiting goals, to ensure a steady stream of available applicants.

The bureau matched these initiatives with an ongoing monitoring effort that enabled bureau officials to rapidly respond to recruiting difficulties. For example, during the last 2 weeks of April, the bureau mailed over 5 million recruiting postcards to Boston, Charlotte, and other locations where it found recruitment efforts were lagging.

Based on the results of our local census office visits, it is clear that the bureau’s human capital strategy had positive outcomes. Of the 60 local census offices we visited, officials at 59 offices provided useable responses to our question about whether their offices had the type of staff they needed to conduct nonresponse follow-up, including staff with particular language skills to enumerate in targeted areas. Officials at 54 of the 59 offices said they had the type of staff they needed to conduct nonresponse follow-up. For example, officials in the Boston North office said they hired enumerators who spoke Japanese, Vietnamese, Portuguese, Spanish, French, Russian, and Chinese, while Pittsburgh office officials said they had enumerators that knew sign language to communicate with deaf residents.

Managers at local census offices we surveyed provided additional perspective on recruiting needed field staff. As shown in figure 5, 30 percent of the respondents believed that the bureau’s ability to recruit and hire high-quality field staff needed no improvements. While managers at 52 percent of the local offices commented that some improvement to the recruiting and hiring process was needed and another 17 percent commented that a significant amount of improvement was needed, their suggestions varied. Managers’ suggestions generally related to various hiring practices, such as a greater use of face-to-face interviews to select managers at local census offices and earlier recruitment advertising.

Once nonresponse follow-up began, bureau officials tracked production rates as the primary measure of whether local offices had met their staffing goals. For example, bureau officials said that both bureau headquarters and regional census center staff monitored local census offices’ production daily. If an office was not meeting its production goals, bureau headquarters officials said they worked with regional census personnel, who in turn worked with the local census office manager, to determine the reasons for the shortfall and the actions necessary to increase production. Possible actions included bringing in enumerators from neighboring local census offices.

Overall, preliminary bureau data shows that about 500,000 enumerators worked on nonresponse follow-up. Nationally, the bureau established a hiring goal of 292,000 enumerator positions for nonresponse follow-up, which represented two people working approximately 25 hours per week for each position and assumed 100 percent turnover, according to bureau officials. The bureau has not yet analyzed how many enumerators charged at least 25 hours per week during nonresponse follow-up. Moreover, according to a senior bureau official, the bureau has not decided whether it will do such an analysis for 2010 planning purposes. According to this official, because the bureau hired about 500,000 enumerators and completed the operation a week ahead of schedule, they believe the bureau generally met its hiring goal.

Local Census Offices Planned in Advance for Specific Enumeration Challenges

A third factor that contributed to the timely completion of nonresponse follow-up was preparing in advance for probable enumeration challenges. To do this, the bureau called on local census offices and their respective regional census centers to develop action plans that, among other things, identified hard-to-enumerate areas within their jurisdictions, such as immigrant neighborhoods, and propose strategies for dealing with those challenges. These strategies included such methods as paired/team enumeration for high-crime areas, and hiring bilingual enumerators. While this early planning effort helped local census offices react to a variety of enumeration challenges, the currency and accuracy of the nonresponse follow-up address lists and maps remained problematic for a number of local census offices.

Most Local Offices Used Action Plans to Address Enumeration Challenges

Of the 60 local census offices we visited, officials at 55 offices provided useable responses to our question about how, if at all, their offices used their action plan for hard-to-enumerate areas during nonresponse follow- up. Officials at 51 of 55 offices said their offices used the strategies in their action plan to address the enumeration challenges they faced.

At the offices we visited, a frequently cited enumeration challenge was gaining access to gated communities or secure apartment buildings. Officials at 42 of the 60 offices we visited identified this as a problem. To address it, officials said they developed partnerships with building management and community leaders, among other strategies. In an Atlanta office, for example, local officials said they sent letters to managers of gated communities that stressed the importance of the census. Similarly, officials in a Chicago office said they personally phoned managers of secure apartment buildings. When enumerators from a Milwaukee local census office encountered problems accessing locked apartment buildings, local census officials told us that the City of Milwaukee sent aldermen to visit the building managers and encourage them to participate in the census.

Another common enumeration challenge appeared to be obtaining cooperation from residents—cited as a difficulty by officials at 34 of the 60 offices we visited. One problem they noted was obtaining responses to the long-form questionnaire—either in its entirety or to specific items, such as income-related questions--which, according to local census officials, some residents found to be intrusive.

Enumerators also encountered residents who were unwilling to participate in the census because of language and cultural differences, or their fears of government. The bureau’s standardized training for enumerators included procedures for handling refusals. Local census officials encouraged public participation with a variety of approaches as well. For example, census officials in Cleveland and Cincinnati said they provided additional training for enumerators on how to handle refusals and practiced what was taught in mock interviews. Officials in other census offices said they partnered with local community leaders who subsequently helped reach out to hard- to-enumerate groups, hired people who were bilingual or otherwise trusted and known by residents, and held media campaigns. Overall, according to bureau data, close to 470,000 households of the approximately 42 million making up the nonresponse follow-up workload (about 1 percent), refused to participate in the census.

The Accuracy and Currency of Nonresponse Follow-up Address Lists and Maps Appeared to Be Problematic

Of the 60 local census offices we visited, officials at 52 offices provided useable responses to our question about whether their offices’ nonresponse follow-up address list reflected the most accurate and current information. Officials at 21 of the 52 offices said that their lists generally were not accurate and current. Nationwide, as shown in figure 6, based on our survey of local census office managers, we estimate that managers at approximately 50 percent of local census offices believed that some improvement was needed in the accuracy of address lists for nonresponse follow-up. We estimated that managers at about 22 percent of local census offices believed that a significant amount of improvement was needed.

Among the more frequent problems managers cited were duplicate addresses and changes not being made from prior operations. For example, at a local census office in the Seattle region, managers said that some addresses were residences or businesses that had been gone for 10-15 years and should have been deleted in previous census operations but were not.

Local census officials we visited cited problems with the accuracy of the census maps as well. Of the 60 local census offices we visited, officials at 58 offices provided useable responses to our question about whether the most accurate and current information was reflected on the nonresponse follow- up maps. Officials at about a third of local census offices—21 of 58 offices—said the nonresponse follow-up maps did not reflect the most accurate and current information.

Further, as shown in figure 7, based on our survey of local census office managers, at about 41 percent of the offices, managers believed that some improvement was needed in maps for nonresponse follow-up. At about 23 percent of the offices, managers believed that a significant amount of improvement was needed in these maps.

Managers who commented that improvements were needed to the nonresponse follow-up maps said the maps were difficult to use, not updated from prior operations, and contained errors. For example, an official at a local census office in the Atlanta region said that some roads on the map did not exist or were not oriented correctly on the census maps. To address this difficulty, local office staff purchased commercial maps or used the Internet to help them locate some housing units.

The bureau developed its master address list and maps using a series of operations that made incremental updates designed to continuously improve the completeness and accuracy of the master address file and maps. A number of these updates occurred during nonresponse follow-up when enumerators encountered, for example, nonexistent or duplicate housing units, or units that needed to be added to the address list. As a result, the bureau was expecting some discrepancies between the nonresponse follow-up address list and what enumerators found in the field when they went door-to-door, which could account for some of the local census officials’ perceptions.

Another factor that affected the currency of the nonresponse follow-up address list was the cut-off date for mail-back responses. The bureau set April 11, 2000, as the deadline for mail-back responses for purposes of generating the address list for nonresponse follow-up. In a subsequent late mail return operation, the bureau updated its field follow-up workload by removing those households for which questionnaires were received from April 11 through April 18. However, according to bureau officials, the bureau continued to receive questionnaires, in part because of an unexpected boost from its outreach and promotion campaign. For example, by April 30—less than 2 weeks after the bureau removed the late mail returns that it had checked-in as of April 18--the bureau received 773,784 additional questionnaires. Bureau headquarters officials told us it was infeasible to remove the late returns from the nonresponse follow-up address lists and thus, enumerators needed to visit these households.

The cost of these visits approached $22 million, based on our earlier estimate that a 1-percentage point increase in workload could add at least $34 million in direct salary, benefits, and travel costs to the price tag of nonresponse follow-up. In addition, the bureau’s data processing centers then had to reconcile the duplicate questionnaires. According to officials at some local offices we visited, the visits to households that had already responded confused residents who questioned why enumerators came to collect information from them after they had mailed back their census forms.

The Bureau’s Stretch Goals to Complete Nonresponse Follow- up May Have Produced Mixed Results

To help ensure that local census offices completed nonresponse follow-up on schedule, the bureau developed ambitious interim stretch goals. These goals called on local census offices to finish 80 percent of their nonresponse follow-up workload within the first 4 weeks of the operation and be completely finished by the end of the eighth week. Under the bureau’s master schedule, local census offices had 10 weeks to complete the operation.

Local Census Office Managers Cited Both Positive and Negative Effects of the Nonresponse Follow-up Schedule on the Quality of the Operation

Our survey of local census office managers asked what impact, if any, scheduling pressures to complete nonresponse follow-up had on the quality of the operation. On the one hand, as shown in figure 8, about 41 percent of the local census office managers believed that scheduling pressures had little or no impact on the quality of the operation, while about 17 percent believed that such pressure had a positive or significantly positive impact. At a local census office in the New York region, for example, the local census office manager stated that, "pressuring people a little gave them the motivation to produce.” Managers in local census offices located in the Dallas region commented that the schedule “kept people on their toes and caused them to put forth their best effort" and that it “had a positive impact, particularly on quality.”

On the other hand, managers at a substantial number of local census offices had the opposite view. As shown in figure 8, about 40 percent of the respondents believed that scheduling pressure during nonresponse follow- up had a negative or significantly negative impact on the quality of the operation.

Of those managers who believed that the pressure to complete nonresponse follow-up adversely affected the quality of the operation, a common perception appeared to be that production was emphasized more than accuracy and that the schedule required local census offices to curtail procedures that could have improved data quality. For example, managers at some local census offices told us that the bureau’s regional census centers encouraged competition between local census offices by, among other actions, ranking local census offices by their progress and distributing the results to local managers. Managers at some local census offices believed that such competition fostered a culture where quantity was more important than quality. As one manager told us, the bureau’s ambitious nonresponse follow-up schedule led the manager “to put enormous pressure on people in the field to complete the operation quickly, and this affected the quality of data.” However, none of the managers we surveyed cited specific examples of where corners were cut or quality was compromised.

The Pace of Nonresponse Follow-up Was Not Associated with the Collection of Less Complete Data

One measure of the quality of nonresponse follow-up is the completeness of the data collected by enumerators. The bureau went to great lengths to obtain complete data directly from household members. Bureau procedures generally called for enumerators to make up to three personal visits and three telephone calls to each household on different days of the week at different times until they obtained needed information on that household.

However, in cases where household members could not be contacted or refused to answer all or part of the census questionnaire, enumerators were permitted to obtain data via proxy (a neighbor, building manager, or other nonhousehold member presumed to know about its residents), or collect less complete data than called for by the census questionnaire. Such data include (1) “closeout” interviews, where questionnaires only contain the information on the status of the housing unit (e.g., whether or not it was occupied), and the number of residents and (2) “partial” interviews, which contain more information than a closeout interview but less than a completed questionnaire.

There were several well-publicized breakdowns in these enumeration procedures at a small number of local census offices that took short cuts to complete their work (which the bureau later took steps to rectify). Nationally, however, our analysis of bureau data found no statistically significant association between the week individual local census offices finished their nonresponse follow-up workload and the percentage of partial or closeout interviews they reported, after controlling for the enumeration difficulty level of each local office’s area (at the time of our review, the bureau did not have information on data collected via proxy interviews).

Neither did we find a statistically significant relationship between the week that local census offices finished their nonresponse follow-up workload and the amount of residual workload, they had, if any. The residual workload consisted of households that were part of the original follow-up workload, but from which the bureau did not receive a questionnaire from the local census offices, and thus had not been processed through data capture. According to bureau data, 519 local offices had to conduct residual nonresponse follow-up on 121,792 households.

Similarly, we did not find an association between week-to-week “spikes” in local census offices’ production and the percentage of either partial or closeout interview data reported. Spikes or surges in production could indicate that local census offices were cutting corners to complete their workloads by a specific deadline. Nationally, we found no relationship between the number of questionnaires finished each week and either the percentage of those finished that were closeout interviews or partial interviews.

Overall, as shown in figure 9, as nonresponse follow-up progressed, the proportion of closeout and partial interview data collected relative to the amount of questionnaires finished remained relatively constant.

Moreover, only a small percentage of most local census offices’ nonresponse follow-up workload was finished using closeout and partial interviews. As shown in figure 10, of the 499 local offices where reliable closeout data were available, 413 (83 percent) reported that less than 2 percent of their questionnaires were finished in this manner, while 19 offices (4 percent) reported 5 percent or more of their finished nonresponse follow-up work as closeout interviews. For partial interviews, of the 508 offices where reliable data were available, 267 (53 percent) reported collecting less than 2 percent of such data, while 47 offices (9 percent) reported 5 percent or more of their finished work as partial interviews. The median percentages of closeout and partial interviews were .8 percent and 1.9 percent, respectively.

At those local census offices that had substantially higher levels of closeout and partial interview data than other offices, the bureau said that some of this was understandable given the enumeration challenges that these census offices faced. For example, according to the bureau, the relatively high partial interview rate at a New York local office (3.8 percent of that office’s finished nonresponse follow-up workload) was in line with the regional average of 2.2 percent, partly due to the difficulty that staff had in gaining access to apartment buildings. Once building managers gave enumerators access and they were able to obtain information from proxies, the number of refusals may have decreased, but the number of partial interviews increased because the proxies could not provide complete information.

Still, as noted above, some local census offices inappropriately used certain enumeration techniques. For example, the Hialeah, Florida, office reported finishing its nonresponse follow-up workload in 5 weeks—well ahead of the 8-week stretch goals and 10 weeks allotted for the operation. The Homestead, Florida, office—where Hialeah-trained enumerators were later transferred to help complete nonresponse follow-up—reported finishing its workload in 7 weeks. The Commerce Department’s Office of the Inspector General later found that Hialeah-trained enumerators did not make the required number of visits and telephone calls before contacting a proxy for information, and did not properly implement quality control procedures designed to detect data falsification. The bureau responded to these findings by, among other actions, reworking over 64,000 questionnaires from the Hialeah and Homestead offices.

Questions Surround Whether Certain Reinterview Procedures Were Implemented as Intended

To help ensure that enumerators followed proper enumeration procedures and were not falsifying data, the bureau “reinterviewed” households under certain circumstances to check enumerators’ work. As such, reinterviews were a critical component of the bureau’s quality assurance program for nonresponse follow-up. If falsification was detected during a reinterview, the local office was to terminate the enumerator and redo all of the enumerator’s work. Enumerators making inadvertent errors were to correct their mistakes and be retrained. The bureau conducted three types of reinterviews: 1. Random reinterviews were to be performed on a sample of enumerators’ work during the early weeks of their employment. Seven randomly selected questionnaires from each enumerator’s first 70 cases were to have been reinterviewed. 2. Administrative reinterviews checked the work of enumerators whose performance in certain dimensions (e.g., the number of partial interviews conducted) differed significantly from that of other enumerators employed in the same area—and there was no justification for the difference. In such cases, enumerators could be fabricating data. According to the bureau, administrative tests were designed to identify enumerators who were making errors that were more likely to occur toward the end of the operation, after the random check of enumerators’ initial work. They were conducted at the discretion of local census officials. 3. Supplemental reinterviews were to be conducted at the discretion of local census officials when they had some basis for concern about the quality of an enumerator’s work.

On the basis of our work and that of the bureau, we found that local census office officials often used their discretion to not conduct administrative and supplemental reinterviews and thus, a number of local offices did not conduct such reinterviews. At those offices, once the random check of enumerators’ initial work was completed, there were no additional checks specifically designed to catch enumerators suspected of falsifying data. This raises questions about the reinterview program’s ability to ensure the quality of enumerators’ work over the full duration of their employment on nonresponse follow-up.

Local Managers Often Decided Against Conducting Administrative Reinterviews

Of the 520 local census offices, 52 offices (10 percent) conducted no administrative and no supplemental reinterviews, according to bureau data.An additional 14 offices (3 percent) conducted no administrative reinterviews, and an additional 231 offices (44 percent) conducted no supplemental reinterviews.

A chief in the bureau’s Quality Assurance Office expressed concern about the adequacy of quality assurance coverage toward the end of nonresponse follow-up for offices that did not conduct administrative and supplemental reinterviews. According to this official, this meant that once random reinterviews were completed at those offices, there were no additional checks specifically designed to detect fabricated data. Although enumerators’ immediate supervisors were to check enumerators’ work daily, these reviews were generally designed to identify enumerators who were completing questionnaires incorrectly (e.g., not following the proper question sequence and writing illegibly), whereas administrative and supplemental reinterviews were aimed at identifying enumerators who were intentionally falsifying data.

Bureau officials said that at those local census offices that did not conduct any administrative reinterviews, local census office managers could conduct supplemental reinterviews if warranted. However, managers employed this option infrequently. Of the 66 local offices that did not conduct any administrative reinterviews, just 14 conducted supplemental reinterviews.

Reasons that local census managers could use—as specified by the bureau—for not conducting an administrative reinterview included (1) the enumerator no longer worked in the area for which the administrative test was conducted; (2) the enumerator’s work was characteristic with the area (e.g., the enumerator reported a large number of vacant housing units and the area had a large number of seasonal housing units); or (3) other reason, with an accompanying explanation. Managers were to document their decision on the bureau’s administrative reinterview trouble reports listing the suspect enumerators.

Our analysis of a week’s worth of administrative reinterview trouble reports at 31 local census offices found that while a number of enumerators were flagged for administrative reinterviews, local census office officials typically decided against conducting them. Specifically, of the 3,784 enumerators identified for possible reinterview, local officials subjected the work of 154 enumerators (4 percent) to reinterviews, and passed on 3,392 enumerators (90 percent). For 306 of the 3,874 enumerators (8 percent) listed on the administrative trouble reports we reviewed, there was no indication of a final decision on whether or not to subject the future work of these enumerators to administrative reinterview.

Overall, local census offices conducted far fewer administrative reinterviews than the bureau had anticipated. Local census offices conducted 276,832 administrative reinterviews—146,993 (35 percent) fewer than the 423,825 administrative reinterviews the bureau had expected based on a number of factors, including the number of cases completed per hour during the 1990 Census, and the estimated workload in 2000. Whether this was due to better quality work on the part of enumerators, or local managers deciding against subjecting enumerators’ work to reinterviews, is unknown. However, as administrative reinterviews were designed to detect fabrication and other quality problems more likely to occur toward the end of nonresponse follow-up after the random check of enumerators’ initial work, it will be important for the bureau to examine whether local census offices properly conducted administrative reinterviews, and thus ensure the quality of nonresponse follow-up data throughout the duration of the operation.

Conclusions

Although nonresponse follow-up was fraught with extraordinary managerial and logistical challenges, the bureau generally completed nonresponse follow-up consistent with its operational plan—a remarkable accomplishment given the scope and complexity of the effort. Our review highlighted several strategies that were key to the bureau’s success including (1) an aggressive outreach and promotion campaign and other efforts aimed at boosting the mail response rate and lowering the bureau’s nonresponse follow-up workload; (2) a flexible recruiting strategy that made the bureau a competitive employer in a tight national labor market; (3) advance planning for addressing location-specific enumeration challenges; and (4) ambitious stretch goals that encouraged local managers to accelerate the pace of the operation. It will be important for the bureau to document the lessons learned from these initiatives and use them to help inform planning efforts for the next decennial census in 2010.

It will also be important for the bureau to address the continuing significant challenges that were revealed by the conduct of nonresponse follow-up in 2000, including achieving an acceptable response rate (and thus lowering the bureau’s follow-up workload) while controlling costs; reversing the downward trend in public participation in the census, in part by converting the relatively large number of people who are aware of the census into census respondents; keeping the address list and maps used for nonresponse follow-up finding the right mix of incentives to motivate local census offices to complete nonresponse follow-up on schedule without compromising data quality; and ensuring that reinterview procedures provide sufficient quality assurance coverage through the full duration of enumerators’ employment on nonresponse follow-up.

Recommendations for Executive Action

As the bureau plans for the next national head count in 2010, we recommend that the Secretary of Commerce ensure that the bureau take the following actions to help ensure that nonresponse follow-up is conducted as cost effectively as possible:

Identify and refine lessons learned from the 2000 nonresponse follow-up operation and apply them to the bureau’s plans for the 2010 Census.

Assess to the extent practicable, why people who were aware of the census did not return their census questionnaires and develop appropriate marketing countermeasures to bridge the gap between their awareness of the census on the one hand, and their motivation to respond on the other.

Develop and test procedural and technological options that have the potential to generate a more accurate and up-to-date address list and set of maps for nonresponse follow-up. As part of this effort, the bureau should explore how to refresh the nonresponse follow-up address list more frequently, even as nonresponse follow-up is underway, so that enumerators would not have to make costly visits to late-responding households. The bureau also needs to examine the methods it uses in activities that precede nonresponse follow-up to develop and update the nonresponse address list and associated maps. Specifically, the bureau should determine the extent to which updates that should have been made were properly reflected in the nonresponse follow-up list and maps, and take appropriate corrective actions to address any problems it identifies.

Ensure that the bureau’s procedures and incentives for the timely completion of nonresponse follow-up emphasize the collection of quality data and proper enumeration techniques as much as speed.

Examine the bureau’s reinterview procedures—particularly as they relate to the discretion given to local census officials—to help ensure that the procedures are sufficient for consistently and reliably detecting potential problems throughout the duration of enumerators’ employment on nonresponse follow-up.

Agency Comments and The Secretary of Commerce forwarded written comments from the Bureau Our Evaluation of the Census on a draft of this report. The bureau concurred with all five of our recommendations and had no specific comments on them. The bureau also clarified several key points and provided additional information and perspective, which we incorporated in our report as appropriate.

The bureau noted that, in addition to the locked apartment buildings that we cited in the Results in Brief section of our report, gated communities were also an enumeration challenge. While the body of the report already contained this information, we added it to the Results in Brief section as well.

Our draft report stated: “One reason for the errors in the nonresponse follow-up address lists was that the bureau found it was infeasible to remove late-responding households. As a result, enumerators needed to visit over 773,000 households that had already mailed back their questionnaires. . . .” The bureau commented that it made a conscious decision to conduct these visits based on logistical concerns and, as a result, the bureau believes that our use of the terms “errors” and “needlessly” do not take this into consideration and are misleading.

Because the bureau could not refresh its nonresponse follow-up address list to reflect households that responded after April 18, the bureau had no choice but to send enumerators to those households and collect the information in-person. However, the term “needed to” better characterizes the bureau’s lack of options and we revised the text accordingly. We also deleted the term “errors.”

In response to our finding that 52 local census offices did not conduct any reinterviews after an initial random check of enumerators’ work, the bureau commented that the initial random check was not a minimal activity in that it involved reinterviewing up to seven cases per enumerator. The bureau also noted that there were no operational requirements to conduct a specific number of administrative or supplemental reinterviews. We agree with the bureau’s comments. Indeed, the draft report already included information on the number of initial random reinterviews the bureau conducted and the discretionary nature of administrative and supplemental reinterviews. Nevertheless, it is also true, as we note in our report, that once those 52 local census offices completed the seven random reinterviews, there were no additional checks specifically designed to catch enumerators suspected of falsifying data. Moreover, we reported that nationwide, local census offices conducted far fewer administrative reinterviews than the bureau had expected. As we note in the report, whether this was due to the quality of enumerators’ work or local managers using their discretion and opting not to subject enumerators’ work to reinterviews, is unknown.

With respect to the bureau’s monitoring of local census office’s productivity, the bureau noted that headquarters officials did not work directly with local census office staff as noted in the draft; rather, headquarters personnel worked with the bureau’s regional census centers, and they in turn worked with the local offices. We revised the text to reflect this information.

With respect to our observation that several local census offices had to quickly respond to unanticipated challenges, such as working with nonresponse follow-up address lists and maps that were not accurate or current, the bureau commented that there were standard procedures in the nonresponse follow-up enumerator manual on how to deal with map/register discrepancies. We verified this and revised the text accordingly.

In describing the steps that local census officials took to encourage public participation in the census, we noted that census officials in Cleveland and Cincinnati said they provided additional training for enumerators on how to handle refusals. The bureau noted that standardized training was provided, across the nation, on options for handling refusals, and information was also provided in the nonresponse follow-up enumerator manual. We verified this information and added it to the report.

The bureau commented that the address list and map difficulties that enumerators encountered were not nonresponse problems because, as we note in the report, and the bureau agrees, they should have been dealt with in earlier census operations. Nevertheless, the problems did not surface until nonresponse follow-up when enumerators encountered duplicate and nonexistent addresses, and were less productive as a result. For this reason, the report recommends that the bureau examine the methods it uses in activities that precede nonresponse follow-up to ensure the address lists and maps used for nonresponse follow-up are accurate and up-to-date.

In response to our statement that nonresponse follow-up was to help verify changes to the address list from earlier address list development operations, the bureau commented that nonresponse follow-up was conducted to enumerate households from which it did not receive a completed questionnaire; map and address updates were incidental. We agree with the bureau on the primary purpose of nonresponse follow-up and revised the text to better reflect this point. However, the bureau’s program master plan for the master address file includes nonresponse follow-up as one of a number of address list development and maintenance operations, and the bureau expected enumerators to update maps and address registers as needed as part of their field visits.

The bureau said it could not confirm data in our draft report on the number of vacant and deleted units identified during nonresponse follow-up and suggested removing this information. Although we obtained the data directly from the bureau, given the bureau’s concerns, we deleted the section.

In commenting on the fact that we did not find a statistically significant relationship between the week that local census offices finished their follow-up workload and the amount of their residual workload, the bureau stated that the report needed to reflect the fact that residual nonresponse consisted of housing units for which completed questionnaires had not been processed through data capture. We revised the draft accordingly.

The bureau noted that assistant managers for field operations, among other local census officials, could request supplemental reinterviews, and not just field operations supervisors as we stated in our report. We revised our draft to include this information.

With respect to our findings concerning the reinterview program’s ability to detect problems, particularly at the end of nonresponse follow-up, the bureau commented that there was turnover in the enumerator workforce; consequently, with new hires, random reinterviews were conducted during all stages of the operation. As we note in the report, 52 local census offices (about 10 percent of all local offices), did not conduct any administrative and supplemental reinterviews. Thus, once these offices completed the random reinterviews on the initial work of newly hired enumerators, there were no additional checks specifically designed to catch enumerators suspected of falsifying data. We added language to better clarify this point.

The bureau said that it was uncertain as to the methodology and documentation used for deriving figures on the number of reinterviews the bureau conducted. We obtained the data from the bureau’s cost and progress system.

The bureau stated that there was no evidence that data quality was compromised to motivate on-time completion of nonresponse follow-up. Our research suggests that the impact of the bureau’s incentives to motivate timeliness was less clear-cut given the fact that, as we note in our report, (1) about 40 percent of the local census office managers believed that scheduling pressures had a negative or significantly negative impact on the quality of nonresponse follow-up, and (2) a small number of local census offices took short-cuts to complete their work (which the bureau later took steps to rectify). Thus, while we agree with the bureau that maintaining data quality should be a given in determining motivational elements, the extent to which the bureau accomplished this goal for nonresponse follow-up appeared to have had mixed results.

In commenting on our conclusion that it will be important for the bureau to ensure that reinterview procedures provide sufficient quality assurance through the full duration of nonresponse follow-up, the bureau noted that the reinterview operation must be designed to provide sufficient quality assurance coverage. We revised the text accordingly We are sending copies of this report to the Honorable Dan Miller and Carolyn B. Maloney, House of Representatives, and those in other interested congressional committees; the Secretary of Commerce; and the Acting Director of the Bureau of the Census. Copies will be made available to others on request. Major contributors to this report are included in appendix III. If you have any questions concerning this report, please call me on (202) 512-6806.

Local Census Offices Included in This Review

Local Census Offices in the Census Bureau’s Atlanta Region

Local Census Offices in the Census Bureau’s Boston Region

Local Census Offices in the Census Bureau’s Charlotte Region

Local Census Offices in the Census Bureau’s Chicago Region

Local Census Offices in the Census Bureau’s Dallas Region

Local Census Offices in the Census Bureau’s Denver Region

Local Census Offices in the Census Bureau’s Kansas City Region

Local Census Offices in the Census Bureau’s Philadelphia Region

Local Census Offices in the Census Bureau’s Los Angeles Region

Local Census Offices in the Census Bureau’s New York Region

Local Census Offices in the Census Bureau’s Seattle Region

Comments from the Secretary of Commerce

GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to those named above, the following headquarters staff made key contributions to this report: Wendy Ahmed; Tom Beall; James Fields; Rich Hung; Lily Kim; J. Christopher Mihm; Victoria E. Miller; Vicky L. Miller; Ty Mitchell; Anne Rhodes-Kline; Lynn Wasielewski; Susan Wallace.

The following staff from the Western Regional Office also contributed to this report: James Bancroft; Robert Bresky; Arthur Davis; Julian Fogle; Araceli Hutsell; RoJeanne Liu; Elizabeth Dolan; Thomas Schulz; Nico Sloss; Cornelius Williams.

The following staff from the Central Regional Office also contributed to this report: Richard Burrell; Michael De La Garza; Maria Durant; Donald Ficklin; Ron Haun; Arturo Holguin, Jr.; Reid Jones; Stefani Jonkman; Roger Kolar; Tom Laetz; Miquel Salas; Enemencio Sanchez; Jeremy Schupbach; Melvin Thomas; Richard Tsuhara; Theresa Wagner; Patrick Ward; Linda Kay Willard; Cleofas Zapata, Jr.

The following staff from the Eastern Regional Office also contributed to this report: Cammillia Campbell; Lara Carreon; Betty Clark; Johnetta Gatlin-Brown; Marshall Hamlett; Carlean Jones; Janet Keller; Cameron Killough; Jean Lee; Christopher Miller; S. Monty Peters; Sharon Reid; Matthew Smith.

Related GAO Products on the Results of the 2000 Census and Lessons Learned for a More Cost-Effective Census in 2010

2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed (GAO-02-26, December 31, 2001). 2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau (GAO-02-30, December 28, 2001). 2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990 Census (GAO-02-31, December 11, 2001). 2000 Census: Better Productivity Data Needed for Future Planning and Budgeting (GAO-02-4, October 4, 2001). 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations (GAO-01-579, August 20, 2001).

Decennial Censuses: Historical Data on Enumerator Productivity Are Limited (GAO-01-208R, January 5, 2001). 2000 Census: Information on Short- and Long-Form Response Rates (GAO/GGD-00-127R, June 7, 2000).

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO E-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily e-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

Visit GAO’s Document Distribution Center

To Report Fraud, Waste, and Abuse in Federal Programs

Web site: www.gao.gov/fraudnet/fraudnet.htm, E-mail: fraudnet@gao.gov, or 1-800-424-5454 or (202) 512-7470 (automated answering system).

Public Affairs