Bureau of Justice Statistics: Quality Guidelines Generally Followed for Police-Public Contact Surveys, but Opportunities Exist to Help Assure Agency Independence
Released: 2007-04-30T00:00:00

Letter

Background

Overview of BJS Mission and Organization

BJS was established by the Justice Systems Improvement Act of 1979. In 1995, OMB identified BJS as one of 10 principal statistical agencies within the federal government. As defined by OMB, the statistical activities of statistical agencies include the planning of statistical surveys and studies; and the collection, processing, or tabulation of statistical data for publication, dissemination, research, analysis, or program management and evaluation. BJS publishes annual data on criminal victimization, populations under correctional supervision, and federal criminal offenders and case processing. It provides periodic data series on the administration of law enforcement agencies and correctional facilities, prosecutorial practices and policies, state court case processing, felony convictions, the characteristics of correctional populations, criminal justice expenditure and employment, civil case processing in state courts, and special studies on other criminal justice topics.

BJS is organizationally located within the Department of Justice’s Office of Justice Programs (see fig. 1). The highest-ranking executives of BJS (BJS Director) and the department’s Office of Justice Programs (Assistant Attorney General) are both noncareer officials appointed by the President and confirmed by the Senate. Within BJS, only the Director is a noncareer appointee.

The BJS Police-Public Contact Survey

BJS initiated the Police-Public Contact Survey pursuant to a mandate in the Violent Crime Control and Law Enforcement Act of 1994, which required the Attorney General to collect information on the use of excessive force by law enforcement officers. The data were to be used only for research or statistical purposes and were not to contain any information that could reveal the identity of the victim or any law enforcement officer. BJS fielded its first pilot survey in 1996 with the goal of better understanding the types and frequency of contacts between the police and the public, and the conditions under which force may be threatened or used. The pilot survey consisted of 6,421 respondents. The three subsequent surveys (in 1999, 2002, and 2005) consisted of 80,543, 76,910, and 63,943 respondents, respectively. Multiple reports and press releases may be issued in connection with any of the surveys. The years in which reports and a single press release associated with the 1999 and 2002 surveys were issued are shown in table 1.

Quality Guidelines for Statistical Products

Over the last several years, various types of guidance have been developed to help federal agencies such as BJS ensure the integrity of statistical information. In 1992, in response to requests from Congress and others as to what constitutes an effective statistical agency, the National Research Council began issuing best-practice guidelines. According to the Committee on National Statistics, which authored the guidelines, the guidelines have been widely cited and used by Congress and federal agencies, and have shaped legislation and executive actions to establish and evaluate statistical agencies. These recommended guidelines, which BJS and other statistical agencies may choose to voluntarily follow, cover the review, approval, and dissemination processes of products issued by federal statistical agencies. In its guideline document, Principles and Practices for Federal Statistical Agencies, the National Research Council indicated, among other things, that statistical agencies should provide high-quality data, take a strong position of independence, be perceived to be free of political interference and policy advocacy, and strive for wide dissemination of their results. In particular, according to the National Research Council, the quality guidelines are to cover the review process and include verification of sources and results, disclosure of limitations, and accuracy of results; approval process and include who has authority over the content and timing of the release of a product, and separation of policy from statistical information; and dissemination process and include the usability of information and its accessibility to a wide range of people.

In February 2002 and September 2006, pursuant to the Information Quality Act of 2001, OMB issued policy and procedural guidance to federal agencies, including statistical agencies such as BJS, directing them to develop their own quality guidelines to help maximize the quality, objectivity, utility, and integrity of the information they disseminate. OMB stated that it was essential that federal statistics be collected, processed, and published in a manner that guarantees and inspires confidence in their reliability. Specifically, OMB directed federal agencies to “adopt a basic standard of quality … as a performance goal,” and “take appropriate steps to incorporate information quality criteria into agency dissemination practices.”

In response to OMB’s February 2002 guidance, the Department of Justice, Office of Justice Programs, and BJS issued their own guidelines later that year. BJS issued a second edition of its guidelines in 2005. In formulating its guidelines, BJS stated that it sought to provide the public with additional information regarding its methods for ensuring the quality, utility, objectivity, and integrity of the statistics it publicly disseminates. As a component of the Department of Justice, BJS is governed by its own data quality guidelines, as well as the information quality guidelines promulgated by the Office of Justice Programs, Department of Justice, and Office of Management and Budget. The Department of Justice’s Information Quality Guidelines are intended to (1) provide the department’s components with a foundation for developing their own, more detailed procedures, (2) provide guidance to component staff, and (3) inform the public of the agency’s policies and procedures.

The Office of Justice Programs’ information quality guidelines require its components—including BJS—to (1) assess the usefulness of the information to be disseminated to the public by continuously monitoring information needs, developing new information sources, or revising existing methods, models, and information products where appropriate; (2) ensure disseminated information is accurate, clear, complete, reproducible, and presented in an unbiased manner by using reliable data sources, sound analytical techniques, and documenting methods and data sources; and (3) protect information from unauthorized access, corruption, or revision. As in the case of the Department of Justice’s guidelines, the Office of Justice Programs provides guidance to its components in developing their own, more specific quality guidelines.

BJS Followed Nearly All Available Quality Guidelines to Help Ensure Accuracy and Integrity of Products Issued from Police- Public Contact Surveys

BJS Fully Followed All Applicable Quality Guidelines for the Four Reports Issued from the 1999 and 2002 Police- Public Contact Surveys

For all four reports issued from the two Police-Public Contact Surveys, we found that BJS fully followed all of the review, approval, and dissemination guidelines available at the time of issuance. We considered a guideline to have been fully followed if our independent analysts determined that all aspects of the guideline were followed. (Our methodology for how we determined the extent to which BJS followed the guidelines is explained in app. I.) The extent to which BJS followed applicable, available guidelines when it issued its Police-Public Contact Survey reports is shown in table 2.

For the first report issued from the 1999 survey, we found that BJS voluntarily followed the National Research Council’s 10 applicable existing guidelines; for the second report, we found that BJS voluntarily followed those 10, as well as 2 additional guidelines issued since the first report, for a total of 12. For each of the two reports based on the 2002 survey, we found that BJS followed all 23 available data quality guidelines that had by then been issued by the National Research Council, the Department of Justice, the Office of Justice Programs, and BJS itself.

The data quality guidelines that BJS followed describe how agencies should review statistical information, obtain the approval of key decision makers, and publicly disseminate the information. While not all of the guideline-issuing organizations addressed the review, approval, and dissemination process, in total across the four organizations—the National Research Council, Department of Justice, Office of Justice Programs, and Bureau of Justice Statistics—all three areas were addressed. Some examples of the guidelines that BJS fully followed in its report issuance process are listed below. (For a complete list of all available data quality guidelines, see appendix II.)

Components of the Department of Justice and Office of Justice Programs will review all information dissemination products for their quality (including objectivity, utility, and integrity) before they are disseminated.

All BJS reports and other statistical products must be subject to an objective and appropriate verification process conducted by qualified BJS staff other than the author of the report.

The statistical agency has recognition by policy officials outside the statistical agency of its authority to release statistical information without prior clearance.

The statistical agency has authority for professional decisions over the scope, content, and frequency of data compiled, analyzed, or published. The objectivity of BJS statistics must be vigilantly protected at all times by BJS staff.

On the basis of our analysis, BJS successfully followed all applicable quality guidelines for these survey-based statistical reports, which both BJS and we consider to be statistical products covered by the guidelines. Thus, we believe the agency took proper steps to help ensure the accuracy and integrity of the review, approval, and dissemination processes associated with issuing public reports based on the two surveys we reviewed. BJS concurred with our analysis. All of the reports were posted to the BJS Web site, where the information is to be accessible to the general public.

BJS Fully Followed 7 of 10 Applicable Data Quality Guidelines for the Single Press Release Issued from Its Police-Public Contact Surveys

For the single press release that was issued—that is, the 2001 press release based on BJS’s 1999 Police-Public Contact Survey—we determined that BJS fully followed 7 of the 10 applicable National Research Council guidelines available at the time. The 7 federal data quality guidelines that BJS fully followed are listed below. (1) A statistical agency should develop an understanding of the validity and accuracy of its data and convey the resulting measures of quality to users in ways that are comprehensible to nonexperts. (2) A statistical agency should use modern statistical theory and sound statistical practice in all technical work. (3) A statistical agency has maintenance of a clear distinction between statistical information and policy interpretations of such information by the President, the secretary of the department, or others in the executive branch. (4) A statistical agency should follow good practice, in reports and other data releases, in documenting concepts, definitions, data collection methodologies, and measures of uncertainty, and in discussing possible sources of error. (5) Effective dissemination programs include policies for the preservation of data that guide what data to retain and how they are to be archived for future secondary analysis. (6) An agency should have an established publications policy that describes, for a data collection program, the types of reports and other data released to be made available, the audiences served, and the frequency of release. (7) Dissemination of data and information (basic series, analytic reports, press releases, public use tapes) should be timely and public. Avenues of dissemination should be chosen to reach as broad a public as reasonably possible.

There were 3 other applicable quality guidelines that we determined BJS was not in a position to follow in connection with this issued press release, and a press release based on the 2002 Police-Public Contact Survey findings was not issued. It is important to note that, for reasons discussed later in this report, BJS officials did not believe these guidelines were applicable to its press releases in the first place.

Differing Views on Applicability of Guidelines Raised Questions about Clarity, while Roles of Certain Noncareer Appointees during Product Issuance Process Affected BJS’s Independence

Two key factors affected whether and how BJS followed quality guidelines during the review, approval, and dissemination of products issued from the 1999 and 2002 Police-Public Contact Surveys. First, while BJS believed, as noted earlier, that its survey reports were statistical products covered by the quality guidelines, it did not believe that the survey-related press release was a statistical product covered by the quality guidelines. BJS cited a lack of specificity in the National Research Council’s guidelines as a basis for this conclusion. We believe, however, that while BJS’s interpretation of the guidelines was not unreasonable, there was nonetheless sufficient evidence for a different interpretation; namely, that this press release was a statistical product, that the available guidelines did apply, and that BJS was not in a position to meet 3 of 10 guidelines for the single press release issued from the 1999 survey, owing to a second factor. This second factor was the role that certain noncareer appointees outside BJS have the ability to play, pursuant to Department of Justice policy, in the product issuance process. In certain instances, the roles of these non-BJS officials meant that BJS was not in a position to fully follow all guidelines related to agency independence, and this holds the potential for future actual or perceived political interference in BJS’s product issuance process for statistical products.

BJS Interpreted Guidelines as Not Applying to Press Releases, Raising Questions about the Applicability and Clarity of Certain Guidelines

In both written documentation and oral comments, BJS officials stated that they believed they were in full conformance with the National Research Council’s guidelines and disagreed with our determination that the agency was not in a position to follow 3 of 10 guidelines for the 2001 Police-Public Contact Survey press release that was issued from the 1999 survey. The guidelines that we determined BJS was not in the position to fully follow all pertain to the agency’s independence and, in particular, to its control over the issuance of press releases. These guidelines were: (1) The statistical agency has recognition by policy officials of its authority to release statistical information without prior clearance. (2) The statistical agency has authority for professional decisions over the scope, content, and frequency of data compiled, analyzed, or published. (3) The release of information should not be subject to actual or perceived political interference. In particular, the timing of the public release of data should be the responsibility of the statistical agency.

BJS officials asserted that, based on their interpretation of the National Research Council’s guidelines, BJS press releases did not qualify as statistical products and, therefore, press releases did not fall within the purview of the council’s guidelines. They also asserted that neither BJS’s own quality guidelines, nor those issued by the Department of Justice and the Office of Justice Programs, apply to BJS press releases. Both BJS and Office of Justice Programs officials stated that the applicability of the council’s guidelines to BJS press releases was, at a minimum, open to question because the council did not state that press releases are data disseminations. In other words, according to BJS and the Office of Justice Programs, press releases are not publications of data, but rather they are simply announcements that a data publication is forthcoming. In its communications with us, BJS stated that many of the guidelines do not apply to press releases but apply only to statistical products.

Based on its content rather than its label as a press release, and notwithstanding that the policies and procedures for developing and issuing products labeled by the Office of Justice Programs as press releases differed from policies and procedures for products it labeled as statistical products, we believe there is sufficient evidence for us to conclude that the press release issued from the 1999 Police-Public Contact Survey qualified as a statistical product to which the National Research Council’s quality guidelines appropriately apply. Our analysis of this press release indicated that it was a data-based statistical product, more than simply an announcement that a data publication was forthcoming. In its entirety, the press release consisted of 20 sentences and one table describing the survey’s statistical findings; 3 sentences on the survey’s methodology; and 5 sentences on who prepared the report and how to obtain copies. We found that this press release was a compilation of statistical data that contained no interpretations, conclusions, or policy statements. (See Appendix III for a reproduction of the press release.) In accordance with the council’s guidelines, the release maintained “a clear distinction between statistical information and policy interpretations of such information.”

To understand whether the National Research Council was purposeful in not stating that its guidelines were applicable to statistical agency press releases, we contacted the council to seek clarification. Officials from the council’s Committee on National Statistics, which authored the data quality guidelines, stated that although the Principles and Practices document did not specifically state that the guidelines covered the content, scope, and timing of press releases issued by statistical agencies, it was not the committee’s intent to exclude press releases from the guidelines. They stated that, in their view, press releases issued by BJS are statistical products to which it is appropriate to apply the guidelines.

BJS and we agree that the National Research Council’s guidelines apply, in general, to statistical products. In asserting that the press release that BJS jointly issued with the Department of Justice and Office of Justice Programs was not a statistical product, BJS correctly noted that the National Research Council did not explicitly state that the guidelines covered press releases. However, given the strong statistical content of the Police-Public Contact Survey press release, we did not believe that such an explication was necessary. Nonetheless, we acknowledge that it is not unreasonable for BJS to reach a different conclusion given the lack of specificity that existed in the council’s printed guidelines. Because BJS’s own data quality guidelines, issued in 2002, state that they “govern all justice statistics that BJS produces and disseminates for the general public, including all statistics that are featured in BJS publications, on the website, and in BJS press releases,” we considered the BJS guidelines to be applicable to press releases, as well. BJS, however, did not hold this view.

It is important to note that we are not finding fault with BJS for the conclusions it drew with respect to the applicability of the quality guidelines to its press release issuance process because the National Research Council’s guidelines were not explicit on this matter. Indeed, we noted in our May 2006 report on data quality that 2 of 14 statistical agencies we surveyed stated that there was ambiguity as to whether a statistical press release was a statistical product, and if so, whether statistical agencies could issue them without first getting releases cleared at the departmental level. BJS was among the 14 statistical agencies surveyed, but it was not one of the two agencies reporting ambiguity in whether a statistical press release was a statistical product. Overall, we believe that BJS made a good faith effort to follow the guidelines it deemed to be applicable to the Police-Public Contact Survey products.

Deciding which guidelines a statistical agency like BJS should follow is further complicated by the fact that BJS’s parent organizations—the Department of Justice and Office of Justice Programs—have explicitly stated that their own guidelines do not apply to press releases. However, these organizations’ guidelines are intended to be broadly applicable to both statistical and nonstatistical agencies. For example, the Department of Justice comprises 38 separate component organizations that produce a variety of types of information, both statistical and nonstatistical in nature. The Office of Justice Programs is composed of 6 bureaus and program offices, and these, too, produce both statistical and nonstatistical information. Because we believe that press releases issued by the department and the Office of Justice Programs may in some, but not all, instances be statistical products, we do not hold the view that statistical guidelines should be universally applicable to all press releases issued by the Department of Justice and Office of Justice Programs. However, because different interpretations can arise, we believe that clarification regarding which guidelines should be applied under which circumstances—and, specifically, to press releases—would be helpful to statistical agencies that are in situations similar to BJS’s.

To address potential discrepancies such as these, in a May 2006 report on the quality of federal data, we recommended that to help improve governmentwide data dissemination practices that would further safeguard the integrity of federal statistical data, OMB should consider how best to address the gaps we identified between agencies’ data dissemination practices and the National Research Council’s guidelines. We noted in that report that OMB, in concert with federal statistical agencies, was developing a governmentwide directive on the release and dissemination of statistical products that, according to OMB officials, parallels the council’s and other generally accepted dissemination practices. We pointed out that it will be important for OMB’s directive to consider, for example, whether the directive should cover principal statistical agencies only, the statistical functions of all agencies, or only statistical products. OMB officials indicated that the guidance is intended to help ensure that statistical products are policy-neutral, timely, and accurate. We recommended that, among other things, OMB include in this directive (1) clear definitions of what is and is not covered, (2) the extent to which agencies should document their data dissemination guidance and how often the guidance should be reviewed, (3) the amount of flexibility agencies have in implementing OMB’s guidance, and (4) procedures for monitoring agencies’ adherence to its directive. To the extent that statistical agencies appropriately follow these practices, the directive could promote more consistent adherence to practices that facilitate broader dissemination of statistical data and enhance its credibility. Although OMB did not provide comments on the recommendations in our 2006 report, an OMB official told us that as of January 2007, OMB was still working on this directive. We believe it remains important for OMB to complete its directive on the release and dissemination of statistical products in order to help safeguard the integrity of federal statistical data, reduce the likelihood that the type of disagreement discussed in this report would recur, and help assure both the actual and perceived independence of BJS.

The Roles Played by Certain Outside Noncareer Appointees in the Press Release Process Affected BJS Carrying Out Its Role as an Independent Statistical Agency

The second key factor that affected whether and how BJS followed guidelines concerned the involvement of noncareer appointees outside of BJS in the press release issuance process, and had implications for BJS’s independence as a statistical agency. Specifically, we determined that BJS was not in a position to fully follow the 3 National Research Council guidelines listed in the previous section for the 2001 press release based on the 1999 survey (the only applicable, available data quality guidelines in place in 2001) because certain noncareer appointees outside of BJS and within the Department of Justice, are vested—pursuant to the Department of Justice’s and Office of Justice Programs’ policies defining the roles and responsibilities of their noncareer appointees—with the ability to participate in the review, approval, and dissemination of press releases. In certain cases, the roles and responsibilities of these noncareer appointees precluded BJS from being in the position to fully follow certain guidelines. The Assistant Attorney General within the Department of Justice’s Office of Justice Programs has general statutory responsibilities with respect to coordinating the activities of that office and its various components, such as BJS. These statutory provisions do not specifically address the Office of Justice Programs’ role with respect to the review, approval, and dissemination of press releases. However, under departmental policy, noncareer appointees within the Department of Justice and outside of BJS have the ability to participate in the press release issuance process.

Table 3 shows the type of involvement that the Assistant Attorney General in the Office of Justice Programs and other noncareer appointees generally have had in the press release review, approval, and dissemination process. Appendix IV describes in more detail the responsibilities of these various officials associated with review, approval, and dissemination procedures for both BJS reports and press releases.

With respect to the first of the three guidelines, which calls for a statistical agency to have authority to release information without prior clearance, it is our view that BJS was not in a position to follow this independence- related guideline at all because it did not have the ability to do so. This is because press releases are subject to review and approval by not only the BJS Director, but also by other Department of Justice noncareer appointees. Outside of BJS, the noncareer appointees participating in the clearance process are located in the Department of Justice’s Office of Justice Programs (these include the Office’s Chief of Staff, Deputy Assistant Attorney General, and Assistant Attorney General) and Office of Public Affairs.

The current Assistant Attorney General and two former Assistant Attorneys General in the Office of Justice Programs told us that there is no written, formal policy or guidance that bounds their input and decision- making roles and responsibilities with respect to BJS press releases. BJS and OJP officials indicated that the Office of Justice Programs’ Assistant Attorney General has ultimate responsibility for the review and approval of BJS press releases. Press releases are issued jointly on letterhead listing BJS and the Department of Justice. The current BJS Director confirmed that publication and dissemination functions for press releases are considered to be within the Assistant Attorney General’s oversight authority. Because the National Research Council stated that an aspect of independence includes “recognition by policy officials outside the statistical agency of its authority to release statistical information without prior clearance,” we concluded that BJS was not in the position to follow this guideline because, as we have stated, we believe the Police-Public Contact Survey press release was a statistical product that BJS could not issue independently.

In practice, the ways in which Assistant Attorneys General of the Office of Justice Programs have exercised their authority have varied. For instance, one former Office of Justice Programs’ Assistant Attorney General told us that she placed “self-imposed” limits on her decisions to modify the content of a BJS press release based on her awareness of congressional support for, and her own belief in, the independence of statistical agencies. The current Office of Justice Programs’ Assistant Attorney General told us that she reviews only press releases that contain quotes from the Attorney General. She said that since she assumed her position in 2005, there have been no BJS press releases that have quoted the Attorney General, and she has relied on her Deputy Assistant Attorney General, the BJS Director, and others to ensure the accuracy and clarity of press releases. Nevertheless, the BJS Director must obtain the approval of the Office of Justice Programs’ Assistant Attorney General and other Justice noncareer appointees to issue a press release.

With respect to the second guideline, pertaining to the agency’s decisions over the scope, content, and frequency of data compiled, analyzed, or published, we found that BJS was not in a position to fully follow this independence-related guideline. Specifically, we found that BJS could exercise professional decisions about the frequency of data analyzed and published (within available budgets), but did not always have complete control over the scope and content of survey press releases to be issued. As noted above, this was due to the fact that press releases are joint products of BJS, the Office of Justice Programs, and the Department of Justice, and noncareer appointees outside of BJS can become involved in the press release process. BJS’s situation with respect to this second guideline came to the fore during the drafting of a press release in 2005 based on the 2002 Police-Public Contact Survey. The press release that BJS sought to publish would have included the following statistical findings from the accompanying Police-Public Contact Survey report: (1) there was no statistically significant difference between the rates that white and minority drivers reported being stopped by police, and (2) once stopped, a larger percentage of black and Hispanic minority drivers reported police using or threatening to use force against them than did whites. The then-BJS Director and the then-Acting Assistant Attorney General had a difference of opinion regarding the presentation of the second statistical finding, which was included in the Police-Public Contact Survey report. Despite reported efforts on the part of both parties to negotiate alternative language with respect to the content of the press release, they could not resolve their differences and the BJS Director decided that a press release would not be issued. The current BJS Director told us that it is “inconceivable” that the Assistant Attorney General would issue a press release without the BJS Director’s prior approval.

According to current BJS officials (both career and noncareer) and the Office of Justice Programs’ Office of Communications staff, during the period 1996-2006, this was the only instance in which a BJS press release was prepared but not issued because the Office of Justice Programs and BJS could not agree on the contents. In all other instances during this period, according to these officials, when the parties disagreed on the content of a press release, they were able to resolve their differences.

With respect to the third guideline, pertaining to actual or perceived political interference and the timing of a release, we similarly believe BJS was in not in a position to fully follow this independence-related guideline for the 2001 press release, which, as discussed earlier, we believe to be a statistical product. Although we found no evidence of political interference with the timing of the 2001 survey press release issued from the 1999 survey, we found that BJS does not have complete control over the timing of press releases, as recommended by the National Research Council. Since both noncareer appointees and career officials in the Office of Justice Programs and the Department of Justice have a role in reviewing and approving BJS press releases, they can affect the date that a press release is issued. According to BJS, career and noncareer appointees outside of BJS can delay the issuance of a press release for reasons having nothing to do with political interference, such as a determination that the press release is not sufficiently newsworthy at the time that it was designated to be issued.

On balance, we believe that the noncareer appointees who played decision-making roles in the Police-Public Contact Survey press release process that we reviewed acted within the scope of the roles and responsibilities accorded them under Department of Justice policies, and that BJS made a reasonable effort to adhere to all applicable data quality guidelines. The fact that certain noncareer officials have the ability to make decisions that affect BJS’s ability to fully meet federal data quality guidelines suggests, however, that the potential exists for BJS’s review, approval, and dissemination process for statistical products to be subject to political interference. Thus, certain actions by noncareer appointees— though made on the basis of professional judgment—could put them at odds with the very guidelines designed to ensure the statistical independence and integrity of agencies such as BJS.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Justice for review and comment. On March 13, we received written comments on the draft report from the Office of Justice Programs’ Assistant Attorney General, and the comments are reproduced in full in appendix VI.

In her letter, the Assistant Attorney General affirmed several of our findings and agreed that a need exists for clear definitions about what quality guidelines cover. She noted that competing interpretations exist about what constitutes a statistical product and that the federal statistical community would benefit from clarity in this area.

However, the Assistant Attorney General disagreed with our characterization of the 2001 Police-Public Contact Survey press release as a statistical product and, therefore, with our conclusion that the National Research Council’s quality guidelines applied to this press release. The Assistant Attorney General stated that “a press release … is a public relations announcement issued to encourage media coverage. The mere presence of statistics in a press release does not transform a press release into a statistical product.” We do not believe and have not stated that the mere presence of statistics in a press release in and of itself transforms it into a statistical product any more than we believe or have stated that labeling a document lacking in statistics but called a statistical product necessarily transforms it into one. The Assistant Attorney General also stated that we “mischaracterized” BJS’s data quality guidelines as applying to press releases because the guidelines apply only to the statistics contained in BJS press releases, and because BJS conforms with OMB, the Department of Justice, and the Office of Justice Programs in considering press releases to be outside the scope of the guidelines. For the following reasons, we maintain that we made a sound decision in applying BJS’s guidelines to the Police-Public Contact Survey press release: (1) BJS’s guidelines state that they “govern all justice statistics that BJS produces and disseminates for the general public, including all statistics that are featured in BJS publications, on the website, and in BJS press releases;” and (2) the Police-Public Contact Survey press release was made up almost entirely of survey statistics, indicating to us that it was a statistical product. In determining that the Police-Public Contact Survey was a statistical product, we felt that the content of the press release was a more important determinant than the label attached to it, or the fact that the processes and staff involved in developing the press release were different from those in BJS reports.

The Assistant Attorney General also noted that the National Research Council’s written guidelines did not explicitly cover press releases. Because we agree, we contacted the National Research Council and consulted with officials of the Council’s Committee on National Statistics (the authoring committee of the Principles and Practices). The officials concurred with our view that BJS press releases referring to statistical products (as opposed to press releases about the announcement of a new agency head, for example) are statistical products to which it is appropriate to apply the guidelines. Although the Principles and Practices document did not specifically state that the guidelines covered the content, scope, and timing of press releases issued by statistical agencies, according to these officials it was not the Committee’s intent to exclude such press releases from the guidelines.

The Assistant Attorney General also felt that the draft report overstated the potential threats to BJS’s independence because we used the term “statistical products” to refer to press releases. She was concerned with our observation that the potential exists for BJS’s review, approval, and dissemination process for statistical products to be subject to political interference since noncareer officials can affect BJS’s ability to meet federal data quality guidelines. We stand by this conclusion. Department of Justice policy permits noncareer appointees within the Department but outside of BJS to participate in the press release process. At the same time however, we are unaware of anything that prevents future modifications to that policy to similarly allow noncareer appointees to participate in BJS’s report issuance. Thus, we believe that we have correctly assessed the risk of potential or actual threats to BJS’s independence.

Finally, the Assistant Attorney General stated that even if the council’s written guidelines explicitly applied to press releases, the BJS director would not adhere to them and no current law can make him do so. We recognize that they are voluntary and not legally required and never have said otherwise.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of the report to the Attorney General, the Director of the Office of Management and Budget, and other interested parties. In addition, the report will be available at no charge on GAO’s home page at http://www.gao.gov.

Please contact Brian Lepore at (202) 512-4523 or leporeb@gao.gov if you or your staff have any questions concerning this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

This report addresses the following two objectives for the 1999 and 2002 Police-Public Contact Surveys, the two surveys for which products had been issued as of February 2007: (1) To what extent did the Bureau of Justice Statistics (BJS) follow available guidelines to help ensure the accuracy and integrity of the review, approval, and dissemination of reports and press releases based on its surveys? (2) What key factors affected whether and how BJS followed available guidelines? In addition, we provide information on scope and methodology changes in the Police- Public Contact Surveys over time (see app. V).

To address the first objective, regarding the extent to which BJS followed guidelines, we obtained quality guideline documents from BJS, the Department of Justice’s Office of Justice Programs, the Department of Justice, and the National Research Council. The guidelines that we obtained from these organizations covered the period between February 2001, when the first product based on the 1999 Police-Public Contact Survey was issued, and June 2006, when the most recent product based on the 2002 survey was issued. We included these federal organizations in our review because BJS is a component of the Office of Justice Programs, which in turn is a component of the Department of Justice, and BJS considers itself to be “governed by” the information quality guidelines of these organizations. We included the National Research Council in our review because it is a widely recognized organization that issued guidelines that were intended to be statements of best practice and provide information on what constitutes an effective statistical organization. We also reviewed guidance and directives issued by the Office of Management and Budget (OMB) because OMB is charged with issuing governmentwide policy and procedural guidance to federal agencies, which are then encouraged to issue their own implementation guidelines.

We took several steps to determine the extent to which BJS followed the specific quality guidelines that it, the Office of Justice Programs, the Department of Justice, and the National Research Council had issued. From the documents provided by these four organizations, a GAO analyst initially identified a total of 63 guidelines that pertained to product review, approval, and dissemination processes. For verification purposes, a GAO methodologist also reviewed the guideline documents. The GAO methodologist agreed with the auditor that all 63 guidelines were appropriate for inclusion in our review. Because many of the guidelines issued by the four organizations were similar and overlapping, the GAO auditor reduced the list to 24 nonduplicative guidelines. The GAO methodologist again reviewed the work of the auditor, and in all cases agreed with the auditor that similar guidelines were being appropriately grouped.

We then developed a data collection instrument to determine whether BJS was following guidelines for the 1999 and 2002 Police-Public Contact Surveys, on which information could be recorded as to whether BJS fully followed, partially followed, or did not at all follow each of the guidelines. We defined “fully” as all aspects of the guideline being followed; “partially” as some, but not all, aspects of the guideline being followed; and “not at all” as no aspects of the guideline being followed. We asked BJS to complete a separate data collection instrument with respect to each of its 1999 and 2002 Police-Public Contact Survey reports and one press release, and to support each response by providing documentary evidence. To decrease the burden on BJS, GAO analysts completed the data collection instrument for 9 of the 24 guidelines, for which we already had sufficient information (for example, documents describing agency processes and procedures, and interviews regarding the roles and responsibilities of noncareer appointees). We provided our assessments regarding these guidelines to BJS and asked officials to either confirm or not confirm them. Two GAO analysts reviewed BJS’s responses and all available supporting documentary and testimonial evidence, and determined whether BJS fully, partially, or did not at all meet each guideline. We provided our findings to BJS for review and comment.

BJS’s 1996 and 2005 Police-Public Contact Surveys were outside the scope of our work. We excluded the 1996 survey because that was a relatively small-scale pilot study; and we excluded the 2005 survey, the most recent Police-Public Contact Survey conducted, because no reports or press releases have yet been issued from this work.

To address the second objective, regarding key factors that affected whether and how BJS followed guidelines, we reviewed processes and procedures that described the review, approval, and dissemination processes for BJS-generated reports and press releases, with particular interest in identifying the roles of noncareer appointees involved in each of these processes. We also reviewed pertinent statutory provisions relating to the roles and responsibilities of officials with respect to BJS. We conducted in-person interviews with, or obtained written responses to our questions from, noncareer appointees in BJS, the Office of Justice Programs, and the Department of Justice’s Office of Public Affairs. Specifically, we conducted in-person interviews with the current BJS Director and the BJS Director who was involved in the disagreement with the Acting Assistant Attorney General, as well as with the current Assistant Attorney General and Deputy Assistant Attorney General in the Office of Justice Programs. We obtained detailed written responses to our questions from a former BJS Director, the Acting Assistant Attorney General who was involved in the disagreement with the BJS Director, and two former Assistant Attorneys General from the Office of Justice Programs. We conducted a telephone interview with the current Deputy Director of the Department of Justice’s Office of Public Affairs. Among other things, we asked these noncareer appointees to provide us with information about BJS’s process for reviewing, approving, and disseminating reports and press releases; the roles and responsibilities of noncareer appointees in that process; changes, if any, that had occurred in the roles played by noncareer appointees; procedures used to help ensure that BJS reports and press releases were accurate, reliable, and unbiased; and any factors that may have affected BJS’s independence in the product issuance process. Finally, we reviewed the guidelines of BJS, the Department of Justice’s Office of Justice Programs, the Department of Justice, and the National Research Council to determine that they reflected the product issuance processes and to consolidate them in order to eliminate duplication.

To determine what changes, if any, have occurred in the scope and methodology of the Police-Public Contact Surveys between 1996 and 2006, which we present in appendix V, we initially developed a matrix of key scope and methodology dimensions, based on a review of the standard social science literature. We then conducted interviews and reviewed documents with respect to these dimensions, for all four Police-Public Contact Surveys—the 1996 pilot survey and the surveys of 1999, 2002, and 2005. We interviewed the current BJS Director and a former BJS Director, and available report authors and the key statistician participating in administrations of the survey, to ascertain their views concerning the intended scope of the four surveys, the methodologies used, scope and methodology changes that were made, and reasons for any changes. We also obtained written responses to our questions from these officials. We conducted a detailed documentary review of the scoping and methodology sections of the issued Police-Public Contact Survey reports and press releases, and extracted information about changes in the data collection instruments used (for example, the numbers and types of questions asked about searches and the use of force). In addition, we reviewed documents prepared by the American Statistical Association and the U.S. Bureau of the Census, which conducted field tests to ensure that Police-Public Contact Survey questions were appropriately devised. In cases where we noted that changes had been made between surveys, we reviewed Census Bureau documentation and interviewed staff and officials at BJS.

It was beyond the scope of this review to address any personnel issues that may have arisen in connection with the disagreement over the content of the 2005 draft press release based on the 2002 Police-Public Contact Survey. We conducted our work between April 2006 and January 2007 in accordance with generally accepted government auditing standards.

Appendix II: Applicable Quality Guidelines for Statistical Products Issued by the Bureau of Justice Statistics and Other Agencies

BJS followed numerous recommended data quality guidelines designed to help ensure the accuracy and integrity of the Police-Public Contact Survey products that it issued in 2001, 2002, 2005, and 2006 based on its 1999 and 2002 surveys. The product issuance guidelines were used to aid BJS’s efforts to review, approve, and disseminate these statistical products to the public and others.

The guidelines were issued at various points in time by the following organizations: the National Research Council, the Bureau of Justice Statistics, the Department of Justice, and the Office of Justice Programs.

In addition to reviewing the guidelines of these four organizations, we also reviewed guidelines and directives issued by the Office of Management and Budget (OMB). However, we did not specifically assess BJS’s practice with respect to following OMB’s guidelines because OMB issued governmentwide policy and procedural guidance to federal agencies that called for agencies to develop their own implementing guidelines.

Table 4 shows the guidelines that were available at the time BJS’s 1999 and 2002 Police-Public Contact Survey products were issued, and which guidelines BJS was in a position to follow.

Appendix III: Press Release Issued in 2001 Based on BJS’s 1999 Police-Public Contact Survey

Appendix IV: BJS Processes for the Review, Approval, and Dissemination of Police-Public Contact Survey Products

Review, Approval, and Dissemination Process for Reports Issued from Police-Public Contact Surveys

Since the inception of the Police-Public Contact Survey in 1996, the BJS Director has been the single noncareer appointee who has had a decision- making role in BJS’s review, approval, and dissemination processes for reports. The BJS Director is a noncareer presidential appointee subject to Senate confirmation. Figure 2 provides an overview of the process followed by BJS in the review, approval, and dissemination of Police- Public Contact Survey reports.

As indicated by the figure, the BJS report author and supervisor prepare the draft report for review and approval. The BJS Director reviews the draft, requests any changes, approves the final draft, and transmits a memorandum of notification through the Office of Justice Program’s Assistant Attorney General up the chain of command to the Attorney General. The memorandum contains an abstract of the report, selected survey findings, and a projected release date for the report. BJS sets the release date for 30 days from the date that the Assistant Attorney General signs the memorandum of notification. The report is posted to the Web site at that time, or sooner if the date and time are specified in the notification memo.

Review, Approval, and Dissemination Process for Press Releases Issued from Police-Public Contact Surveys

In contrast to the process followed for survey reports, several noncareer appointees are involved in the organizational review, approval, and dissemination process, as shown in figure 3.

As indicated in the figure, in addition to the Director of BJS, there are three noncareer appointees within the Office of Justice Programs who participate in the review and approval process—the Chief of Staff, the Deputy Assistant Attorney General, and the Assistant Attorney General, and at least one noncareer appointee within Department of Justice headquarters: the Director of the Office of Public Affairs.

The BJS report author and supervisor jointly work with staff from the Office of Justice Program’s Office of Communications to prepare the press release. The BJS director reviews the draft press release, requests any changes, approves the final draft, and transmits the press release up the chain of command to the Office of Justice Programs’ Assistant Attorney General for review and approval. Following approval by the Assistant Attorney General, the Department of Justice’s Office of Public Affairs reviews the press release for clarity, and the BJS Director then verifies that the information in the press release is accurate. The Department of Justice’s Office of Public Affairs is then responsible for disseminating the press release to Congress, the media, and executive department press offices, while BJS is responsible for disseminating the press release through its Web site.

Appendix V: Changes in the Scope and Methodology of the Police-Public Contact Surveys

BJS has conducted four Police-Public Contact Surveys as supplements to the National Crime Victimization Survey. The first Police-Public Contact Survey was conducted as a pilot in 1996. Three subsequent, more extensive surveys were conducted at 3-year intervals: 1999, 2002, and 2005. Although we do not discuss the 2005 survey in this report because no reports or press releases have yet been issued from this survey, we present information on the 2005 survey in this appendix because information is available on this survey’s scope and methodology. The scope of the Police- Public Contact Surveys has consistently expanded over time, while the methodology has remained generally consistent.

The Scope of the Police- Public Contact Survey Has Expanded with Each Additional Survey

The pilot survey was designed to test whether the survey could be effectively used as a supplement to the National Crime Victimization Survey to collect data on (a) the types of contacts the public have with the police, and (b) police use of force. To conduct this test, BJS employed a representative sample of 6,421 U.S. residents. The pilot survey yielded useful information on the various types of contacts the public had with the police, and whether force was used by the police. However, the sample size of the pilot survey was not sufficiently large for BJS to draw inferences about the extent to which the population at large would report that they experienced “excessive” use of force by police.

For its Police-Public Contact Survey in 1999, BJS increased the sample size to a representative sample of 80,543 U.S. residents. The scope of the survey was further enhanced by adding questions about traffic stops (the most common form of public-police contact, as determined in the 1996 pilot survey), and including a question on whether the police used excessive force during any contact with the public. BJS officials told us that they added the traffic stop questions, at least in part, to “address the growing public concern about racial profiling in connection with traffic stops.”

In its 2002 survey, BJS expanded and refined its survey questions further. Specifically, according to BJS officials, they added questions that would help BJS estimate the extent to which U.S. residents nationwide would say that (1) they were stopped by the police while driving, (2) they or their vehicle were searched by the police without their permission during a traffic stop, and (3) they were arrested as a result of the search. In addition, BJS officials said that they added questions to estimate differences, if any, among racial groups in their rates of traffic stops at various times of the day, and whether police used force in situations where persons were engaged in such behaviors as arguing with, cursing, or disobeying police.

In 2005, according to BJS officials, the scope of the Police-Public Contact Survey was further extended in several ways, including the following: (a) respondents were asked whether they had been arrested for driving under the influence of alcohol during the year (in order to make comparisons with Federal Bureau of Investigation (FBI) arrest rates, so that potential undercounting rates could be determined); (b) respondents were permitted to group themselves into any combination of racial categories (rather than choosing a single category) to better refine respondent demographic status; (c) respondents were asked whether police used force during any of their police contacts during the year, as opposed to the more limiting question in 2002, which was directed only toward the most recent contact with police; and (d) respondents were provided open-ended response fields on the survey instrument to indicate any ways they believed that the police had acted inappropriately toward them.

The Police-Public Contact Survey Methodology Has Remained Consistent Following the 1996 Survey

The 1999, 2002, and 2005 Police-Public Contact Surveys have consistently maintained a similar methodology. The methodological dimensions of the surveys that have remained consistent are geographic coverage, target population, sampling design, data collection method, sample size/response rate, survey administration, and sample characteristics (as indicated in table 5). To illustrate, all three surveys involved selecting nationally representative, stratified, multi-cluster samples from the population of U.S. residents 16 years of age or older. The data were collected either through face-to-face or computer-assisted telephone interviews. The surveys were administered during the last 6 months of the year, and the demographic characteristics of the samples were similar across time periods.

The 1996 pilot survey differed in several ways from the subsequent three surveys. Specifically, the pilot survey included residents younger than 16, included far fewer people than the subsequent surveys, and limited the sampling to individuals who had participated in the last round of the National Crime Victimization Survey. In addition, the percentage of face- to-face interviews was lower, and the survey administration period was shorter, and during a different time of the year, than in the other three surveys.

Appendix VI: Comments from the Department of Justice

Appendix VII: GAO Contact and Acknowledgments

Acknowledgments

In addition to the above, Evi. L. Rezmovic, Assistant Director; Ronald S. Fecso, Chief Statistician; Jared A. Hermalin; Karen A. Jarzynka; Amanda K. Miller; Amy L. Bernstein; Geoffrey R. Hamilton; Robert Alarapon; and Tracy J. Harris made key contributions to this report.