Bureau of Justice Statistics: Quality Guidelines Generally Followed for Police-Public Contact Surveys, but Opportunities Exist to Help Assure Agency Independence
Published: 2007-03-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Bureau of Justice Statistics (BJS), a statistical agency of the Department of Justice's Office of Justice Programs, produces a recurring national Police-Public Contact Survey documenting contacts between the police and the public, including instances involving the use or threat of force by police. BJS issues public reports and sometimes press releases from survey results. For reports and a press release issued from the 1999 and 2002 surveys (the most recent available), GAO reviewed (1) the extent to which BJS followed quality guidelines to ensure the accuracy and integrity of its survey-related products, and (2) factors that affected whether and how BJS followed available guidelines. GAO reviewed applicable federal data quality guidelines, policy and procedure documents, and interviewed current and former officials familiar with BJS.
section_title: What GAO Found
paragraphs: BJS followed nearly all quality guidelines for its 1999 and 2002 Police-Public Contact Surveys. Specifically, for the four public reports issued from these surveys, BJS fully followed all data quality guidelines available for reviewing statistical information, obtaining the approval of key decision makers, and publicly disseminating information. These guidelines were issued by the National Research Council, Department of Justice, Justice's Office of Justice Programs, and BJS itself. GAO believes that because BJS followed these guidelines, proper steps were taken to help ensure the accuracy and integrity of the reports. BJS followed 7 of the 10 quality guidelines available for the one press release issued from its 1999 survey, but was not in a position to fully follow 3 other guidelines for reasons discussed below. Two key factors affected whether and how BJS followed quality guidelines. The first concerned different interpretations about certain guideline applicability. BJS considered its survey-related reports--but not its press releases--to be statistical products covered by the National Research Council's guidelines. BJS cited a lack of specificity in these guidelines, which did not specifically state that they were applicable to statistical agency press releases, as a basis for concluding that the survey press releases need not conform to guidelines for statistical products. We believe BJS's position was not unreasonable, and did not find fault with the agency. However, we determined nonetheless that the single press release issued from the 1999 survey was a statistical product, and therefore believe the council's guidelines appropriately applied. Second, certain noncareer appointees outside BJS may, in accordance with Justice Department policy, make decisions about the review, approval, and dissemination of press releases, and BJS press releases are jointly issued with the Justice Department, with input from its Office of Justice Programs. Both conditions can potentially affect BJS's independence. Owing to these conditions, BJS was not, in our view, in a position to meet 3 council quality guidelines related to statistical agency independence, including that it be able to issue statistical products without prior clearance, and control the scope and content of its products. Justice affirmed several of GAO's findings but disagreed with certain GAO conclusions about the applicability of guidelines to a press release. Justice's detailed comments and GAO's response are contained in the report.

section_title: Letter
paragraphs: The Bureau of Justice Statistics (BJS), a statistical agency within the  The Bureau of Justice Statistics (BJS), a statistical agency within the  Department of Justice’s Office of Justice Programs, collects, analyzes,  Department of Justice’s Office of Justice Programs, collects, analyzes,  publishes, and disseminates information on crime, criminal offenders,  publishes, and disseminates information on crime, criminal offenders,  victims of crime, and the operation of justice systems at all levels of  victims of crime, and the operation of justice systems at all levels of  government. A critical aspect of the agency’s mission is to produce  government. A critical aspect of the agency’s mission is to produce  impartial, timely, and accurate statistics. An ongoing BJS data collection  impartial, timely, and accurate statistics. An ongoing BJS data collection  activity is the Police-Public Contact Survey—one of many statistical  activity is the Police-Public Contact Survey—one of many statistical  surveys the agency administers. This recurring, nationally representative  surveys the agency administers. This recurring, nationally representative  survey of the public is designed to document contacts between the police  survey of the public is designed to document contacts between the police  and the public and the outcomes of those contacts, including instances  and the public and the outcomes of those contacts, including instances  involving the use of force or the threat of force by the police. BJS issues  involving the use of force or the threat of force by the police. BJS issues  reports to the public based on the results of this survey, and these may be  reports to the public based on the results of this survey, and these may be  accompanied by a press release. To date, BJS has conducted the survey  accompanied by a press release. To date, BJS has conducted the survey  four times—a pilot survey was conducted in 1996, and more extensive  four times—a pilot survey was conducted in 1996, and more extensive  surveys were done in 1999, 2002, and 2005.surveys were done in 1999, 2002, and 2005.
At the time of our review, BJS had not issued any products from the 2005 survey.
As a result of this incident, some members of Congress, the research  community, and the media raised questions about the methods and  procedures BJS follows in reviewing and approving reports and other  products based on the agency’s statistical information, and the extent of  BJS’s independence as a statistical agency.
In light of the questions raised about BJS’s ability to meet its self-stated  mission as a statistical agency committed to maintaining public trust and  confidence in its statistics, we reviewed what processes are in place at BJS  to help the agency ensure the integrity and independence of its statistical  studies. This report specifically addresses the following two questions for  the 1999 and 2002 Police-Public Contact Surveys, the two surveys for  which products had been issued as of February 2007: (1) To what extent  did BJS follow guidelines to help ensure the accuracy and integrity of the  review, approval, and dissemination of reports and press releases based  on its surveys? (2) What key factors affected whether and how BJS  followed available guidelines?
To assess the extent to which BJS followed guidelines to help ensure the  accuracy and integrity of its product issuance processes for the 1999 and  2002 Police-Public Contact Surveys, we reviewed quality guidelines  pertaining to the review, approval, and dissemination of written products  issued by the National Research Council (a private, nonprofit institution  that is a component of the congressionally chartered National Academies),  the Department of Justice, the Office of Justice Programs, and BJS itself.  The guidelines that we reviewed covered the period between February  2001, when the first product based on the 1999 survey was issued, and  June 2006, when the most recent product based on the 2002 survey was  issued. We initially identified a total of 63 guidelines from the four  organizations. Because the organizations had a number of similar  guidelines, we eliminated the overlap by reducing the list to 24  nonduplicative guidelines. To determine the extent to which BJS followed  each guideline, we developed a data collection instrument for recording  information on whether BJS fully, partially, or did not at all follow the  guideline. For BJS’s 1999 and 2002 Police-Public Contact Survey reports  and the one press release that was issued, we asked BJS to assess the  extent to which it followed each of 15 guidelines, and provide us with  documentary evidence supporting its assessment. We completed the  instrument for the remaining 9 guidelines for which we already had  sufficient information from BJS and Office of Justice Programs documents  and interviews, and asked BJS to confirm or not confirm our assessments.  Two GAO analysts reviewed BJS’s responses and all available supporting  documentary and testimonial evidence, and made a final determination on  the extent to which we believed BJS followed each guideline. We provided  our findings to BJS for review and comment. We also reviewed guidelines  issued by the Office of Management and Budget (OMB), but did not  specifically assess BJS’s practice with respect to following OMB’s  guidelines, because OMB issued governmentwide policy and procedural  guidance to federal agencies that called for agencies to develop their own  agency-specific guidelines. To determine key factors that affected whether  and how BJS followed applicable guidelines, we reviewed BJS procedural  documents and interviewed both current and former officials involved in  the review, approval, and dissemination of BJS products, including past  and present BJS directors and assistant attorneys general within the Office  of Justice Programs. In addition, we reviewed pertinent statutory  provisions relating to the roles and responsibilities of officials with respect  to BJS. It was beyond the scope of this review to address any personnel  issues that may have arisen in connection with the disagreement over the  content of the draft press release. Appendix I provides additional details  on our scope and methodology. We conducted our work between April  2006 and February 2007 in accordance with generally accepted  government auditing standards.
In addition, in response to your interest regarding changes that have  occurred in the scope and methodology of the Police-Public Contact  Surveys between 1996 and 2005, and the reasons for any changes,  appendix V provides detailed information on this issue.
subsections: section_title: Background
subsections: section_title: Overview of BJS Mission and Organization
paragraphs: BJS was established by the Justice Systems Improvement Act of 1979. In  1995, OMB identified BJS as one of 10 principal statistical agencies within  the federal government. As defined by OMB, the statistical activities of  statistical agencies include the planning of statistical surveys and studies;  and the collection, processing, or tabulation of statistical data for  publication, dissemination, research, analysis, or program management  and evaluation. BJS publishes annual data on criminal victimization,  populations under correctional supervision, and federal criminal offenders  and case processing. It provides periodic data series on the administration  of law enforcement agencies and correctional facilities, prosecutorial  practices and policies, state court case processing, felony convictions, the  characteristics of correctional populations, criminal justice expenditure  and employment, civil case processing in state courts, and special studies  on other criminal justice topics.
BJS is organizationally located within the Department of Justice’s Office of  Justice Programs (see fig. 1). The highest-ranking executives of BJS (BJS  Director) and the department’s Office of Justice Programs (Assistant  Attorney General) are both noncareer officials appointed by the President  and confirmed by the Senate. Within BJS, only the Director is a noncareer  appointee.
section_title: The BJS Police-Public Contact Survey
paragraphs: BJS initiated the Police-Public Contact Survey pursuant to a mandate in  the Violent Crime Control and Law Enforcement Act of 1994, which  required the Attorney General to collect information on the use of  excessive force by law enforcement officers. The data were to be used  only for research or statistical purposes and were not to contain any  information that could reveal the identity of the victim or any law  enforcement officer. BJS fielded its first pilot survey in 1996 with the goal  of better understanding the types and frequency of contacts between the  police and the public, and the conditions under which force may be  threatened or used. The pilot survey consisted of 6,421 respondents. The  three subsequent surveys (in 1999, 2002, and 2005) consisted of 80,543,  76,910, and 63,943 respondents, respectively. Multiple reports and press  releases may be issued in connection with any of the surveys. The years in  which reports and a single press release associated with the 1999 and 2002  surveys were issued are shown in table 1.
section_title: Quality Guidelines for Statistical Products
paragraphs: Over the last several years, various types of guidance have been developed  to help federal agencies such as BJS ensure the integrity of statistical  information. In 1992, in response to requests from Congress and others as  to what constitutes an effective statistical agency, the National Research  Council began issuing best-practice guidelines. According to the  Committee on National Statistics, which authored the guidelines, the  guidelines have been widely cited and used by Congress and federal  agencies, and have shaped legislation and executive actions to establish  and evaluate statistical agencies. These recommended guidelines, which  BJS and other statistical agencies may choose to voluntarily follow, cover  the review, approval, and dissemination processes of products issued by  federal statistical agencies. In its guideline document, Principles and  Practices for Federal Statistical Agencies, the National Research Council  indicated, among other things, that statistical agencies should provide  high-quality data, take a strong position of independence, be perceived to  be free of political interference and policy advocacy, and strive for wide  dissemination of their results. In particular, according to the National  Research Council, the quality guidelines are to cover the    review process and include verification of sources and results,  disclosure of limitations, and accuracy of results;    approval process and include who has authority over the content and  timing of the release of a product, and separation of policy from  statistical information; and    dissemination process and include the usability of information and its  accessibility to a wide range of people.
In February 2002 and September 2006, pursuant to the Information Quality  Act of 2001, OMB issued policy and procedural guidance to federal  agencies, including statistical agencies such as BJS, directing them to  develop their own quality guidelines to help maximize the quality,  objectivity, utility, and integrity of the information they disseminate. OMB  stated that it was essential that federal statistics be collected, processed,  and published in a manner that guarantees and inspires confidence in their  reliability. Specifically, OMB directed federal agencies to “adopt a basic  standard of quality … as a performance goal,” and “take appropriate steps  to incorporate information quality criteria into agency dissemination  practices.”
In response to OMB’s February 2002 guidance, the Department of Justice,  Office of Justice Programs, and BJS issued their own guidelines later that  year. BJS issued a second edition of its guidelines in 2005. In formulating  its guidelines, BJS stated that it sought to provide the public with  additional information regarding its methods for ensuring the quality,  utility, objectivity, and integrity of the statistics it publicly disseminates.  As a component of the Department of Justice, BJS is governed by its own  data quality guidelines, as well as the information quality guidelines  promulgated by the Office of Justice Programs, Department of Justice, and  Office of Management and Budget. The Department of Justice’s  Information Quality Guidelines are intended to (1) provide the  department’s components with a foundation for developing their own,  more detailed procedures, (2) provide guidance to component staff, and  (3) inform the public of the agency’s policies and procedures.
The Office of Justice Programs’ information quality guidelines require its  components—including BJS—to (1) assess the usefulness of the  information to be disseminated to the public by continuously monitoring  information needs, developing new information sources, or revising  existing methods, models, and information products where appropriate;  (2) ensure disseminated information is accurate, clear, complete,  reproducible, and presented in an unbiased manner by using reliable data  sources, sound analytical techniques, and documenting methods and data  sources; and (3) protect information from unauthorized access,  corruption, or revision. As in the case of the Department of Justice’s  guidelines, the Office of Justice Programs provides guidance to its  components in developing their own, more specific quality guidelines.
section_title: BJS Followed Nearly All Available Quality Guidelines to Help Ensure Accuracy and Integrity of Products Issued from Police- Public Contact Surveys
subsections: section_title: BJS Fully Followed All Applicable Quality Guidelines for the Four Reports Issued from the 1999 and 2002 Police- Public Contact Surveys
paragraphs: For all four reports issued from the two Police-Public Contact Surveys, we  found that BJS fully followed all of the review, approval, and  dissemination guidelines available at the time of issuance. We considered  a guideline to have been fully followed if our independent analysts  determined that all aspects of the guideline were followed. (Our  methodology for how we determined the extent to which BJS followed the  guidelines is explained in app. I.) The extent to which BJS followed  applicable, available guidelines when it issued its Police-Public Contact  Survey reports is shown in table 2.
For the first report issued from the 1999 survey, we found that BJS  voluntarily followed the National Research Council’s 10 applicable  existing guidelines; for the second report, we found that BJS voluntarily  followed those 10, as well as 2 additional guidelines issued since the first  report, for a total of 12. For each of the two reports based on the 2002  survey, we found that BJS followed all 23 available data quality guidelines  that had by then been issued by the National Research Council, the  Department of Justice, the Office of Justice Programs, and BJS itself.
The data quality guidelines that BJS followed describe how agencies  should review statistical information, obtain the approval of key decision  makers, and publicly disseminate the information. While not all of the  guideline-issuing organizations addressed the review, approval, and  dissemination process, in total across the four organizations—the National  Research Council, Department of Justice, Office of Justice Programs, and  Bureau of Justice Statistics—all three areas were addressed. Some  examples of the guidelines that BJS fully followed in its report issuance  process are listed below. (For a complete list of all available data quality  guidelines, see appendix II.)
Components of the Department of Justice and Office of Justice  Programs will review all information dissemination products for their  quality (including objectivity, utility, and integrity) before they are  disseminated.
All BJS reports and other statistical products must be subject to an  objective and appropriate verification process conducted by qualified  BJS staff other than the author of the report.
The statistical agency has recognition by policy officials outside the  statistical agency of its authority to release statistical information  without prior clearance.
The statistical agency has authority for professional decisions over the  scope, content, and frequency of data compiled, analyzed, or published.    The objectivity of BJS statistics must be vigilantly protected at all times  by BJS staff.
On the basis of our analysis, BJS successfully followed all applicable  quality guidelines for these survey-based statistical reports, which both  BJS and we consider to be statistical products covered by the guidelines.  Thus, we believe the agency took proper steps to help ensure the accuracy  and integrity of the review, approval, and dissemination processes  associated with issuing public reports based on the two surveys we  reviewed. BJS concurred with our analysis. All of the reports were posted  to the BJS Web site, where the information is to be accessible to the  general public.
section_title: BJS Fully Followed 7 of 10 Applicable Data Quality Guidelines for the Single Press Release Issued from Its Police-Public Contact Surveys
paragraphs: For the single press release that was issued—that is, the 2001 press release  based on BJS’s 1999 Police-Public Contact Survey—we determined that  BJS fully followed 7 of the 10 applicable National Research Council  guidelines available at the time. The 7 federal data quality guidelines that  BJS fully followed are listed below.  (1) A statistical agency should develop an understanding of the validity  and accuracy of its data and convey the resulting measures of quality  to users in ways that are comprehensible to nonexperts.  (2) A statistical agency should use modern statistical theory and sound  statistical practice in all technical work.  (3) A statistical agency has maintenance of a clear distinction between  statistical information and policy interpretations of such information  by the President, the secretary of the department, or others in the  executive branch.  (4) A statistical agency should follow good practice, in reports and other  data releases, in documenting concepts, definitions, data collection  methodologies, and measures of uncertainty, and in discussing  possible sources of error.  (5) Effective dissemination programs include policies for the preservation  of data that guide what data to retain and how they are to be archived  for future secondary analysis.  (6) An agency should have an established publications policy that  describes, for a data collection program, the types of reports and other  data released to be made available, the audiences served, and the  frequency of release.  (7) Dissemination of data and information (basic series, analytic reports,  press releases, public use tapes) should be timely and public. Avenues  of dissemination should be chosen to reach as broad a public as  reasonably possible.
There were 3 other applicable quality guidelines that we determined BJS  was not in a position to follow in connection with this issued press  release, and a press release based on the 2002 Police-Public Contact  Survey findings was not issued. It is important to note that, for reasons  discussed later in this report, BJS officials did not believe these guidelines  were applicable to its press releases in the first place.
section_title: Differing Views on Applicability of Guidelines Raised Questions about Clarity, while Roles of Certain Noncareer Appointees during Product Issuance Process Affected BJS’s Independence
paragraphs: Two key factors affected whether and how BJS followed quality guidelines  during the review, approval, and dissemination of products issued from  the 1999 and 2002 Police-Public Contact Surveys.  First, while BJS  believed, as noted earlier, that its survey reports were statistical products  covered by the quality guidelines, it did not believe that the survey-related  press release was a statistical product covered by the quality guidelines.  BJS cited a lack of specificity in the National Research Council’s  guidelines as a basis for this conclusion. We believe, however, that while  BJS’s interpretation of the guidelines was not unreasonable, there was  nonetheless sufficient evidence for a different interpretation; namely, that  this press release was a statistical product, that the available guidelines  did apply, and that BJS was not in a position to meet 3 of 10 guidelines for  the single press release issued from the 1999 survey, owing to a second  factor. This second factor was the role that certain noncareer appointees  outside BJS have the ability to play, pursuant to Department of Justice  policy, in the product issuance process. In certain instances, the roles of  these non-BJS officials meant that BJS was not in a position to fully follow  all guidelines related to agency independence, and this holds the potential  for future actual or perceived political interference in BJS’s product  issuance process for statistical products.
subsections: section_title: BJS Interpreted Guidelines as Not Applying to Press Releases, Raising Questions about the Applicability and Clarity of Certain Guidelines
paragraphs: In both written documentation and oral comments, BJS officials stated  that they believed they were in full conformance with the National  Research Council’s guidelines and disagreed with our determination that  the agency was not in a position to follow 3 of 10 guidelines for the 2001  Police-Public Contact Survey press release that was issued from the 1999  survey. The guidelines that we determined BJS was not in the position to  fully follow all pertain to the agency’s independence and, in particular, to  its control over the issuance of press releases. These guidelines were:  (1) The statistical agency has recognition by policy officials of its  authority to release statistical information without prior clearance.  (2) The statistical agency has authority for professional decisions over the  scope, content, and frequency of data compiled, analyzed, or  published.  (3) The release of information should not be subject to actual or perceived  political interference. In particular, the timing of the public release of  data should be the responsibility of the statistical agency.
BJS officials asserted that, based on their interpretation of the National  Research Council’s guidelines, BJS press releases did not qualify as  statistical products and, therefore, press releases did not fall within the  purview of the council’s guidelines. They also asserted that neither BJS’s  own quality guidelines, nor those issued by the Department of Justice and  the Office of Justice Programs, apply to BJS press releases. Both BJS and  Office of Justice Programs officials stated that the applicability of the  council’s guidelines to BJS press releases was, at a minimum, open to  question because the council did not state that press releases are data  disseminations. In other words, according to BJS and the Office of Justice  Programs, press releases are not publications of data, but rather they are  simply announcements that a data publication is forthcoming. In its  communications with us, BJS stated that many of the guidelines do not  apply to press releases but apply only to statistical products.
Based on its content rather than its label as a press release, and  notwithstanding that the policies and procedures for developing and  issuing products labeled by the Office of Justice Programs as press  releases differed from policies and procedures for products it labeled as  statistical products, we believe there is sufficient evidence for us to  conclude that the press release issued from the 1999 Police-Public Contact  Survey qualified as a statistical product to which the National Research  Council’s quality guidelines appropriately apply. Our analysis of this press  release indicated that it was a data-based statistical product, more than  simply an announcement that a data publication was forthcoming. In its  entirety, the press release consisted of 20 sentences and one table  describing the survey’s statistical findings; 3 sentences on the survey’s  methodology; and 5 sentences on who prepared the report and how to  obtain copies. We found that this press release was a compilation of  statistical data that contained no interpretations, conclusions, or policy  statements. (See Appendix III for a reproduction of the press release.) In  accordance with the council’s guidelines, the release maintained “a clear  distinction between statistical information and policy interpretations of  such information.”
To understand whether the National Research Council was purposeful in  not stating that its guidelines were applicable to statistical agency press  releases, we contacted the council to seek clarification. Officials from the  council’s Committee on National Statistics, which authored the data  quality guidelines, stated that although the Principles and Practices  document did not specifically state that the guidelines covered the  content, scope, and timing of press releases issued by statistical agencies,  it was not the committee’s intent to exclude press releases from the  guidelines. They stated that, in their view, press releases issued by BJS are  statistical products to which it is appropriate to apply the guidelines.
BJS and we agree that the National Research Council’s guidelines apply, in  general, to statistical products. In asserting that the press release that BJS  jointly issued with the Department of Justice and Office of Justice  Programs was not a statistical product, BJS correctly noted that the  National Research Council did not explicitly state that the guidelines  covered press releases. However, given the strong statistical content of the  Police-Public Contact Survey press release, we did not believe that such  an explication was necessary. Nonetheless, we acknowledge that it is not  unreasonable for BJS to reach a different conclusion given the lack of  specificity that existed in the council’s printed guidelines. Because BJS’s  own data quality guidelines, issued in 2002, state that they “govern all  justice statistics that BJS produces and disseminates for the general  public, including all statistics that are featured in BJS publications, on the  website, and in BJS press releases,” we considered the BJS  guidelines to  be applicable to press releases, as well. BJS, however, did not hold this  view.
It is important to note that we are not finding fault with BJS for the  conclusions it drew with respect to the applicability of the quality  guidelines to its press release issuance process because the National  Research Council’s guidelines were not explicit on this matter. Indeed, we  noted in our May 2006 report on data quality that 2 of 14 statistical  agencies we surveyed stated that there was ambiguity as to whether a  statistical press release was a statistical product, and if so, whether  statistical agencies could issue them without first getting releases cleared  at the departmental level. BJS was among the 14 statistical agencies  surveyed, but it was not one of the two agencies reporting ambiguity in  whether a statistical press release was a statistical product. Overall, we  believe that BJS made a good faith effort to follow the guidelines it  deemed to be applicable to the Police-Public Contact Survey products.
Deciding which guidelines a statistical agency like BJS should follow is  further complicated by the fact that BJS’s parent organizations—the  Department of Justice and Office of Justice Programs—have explicitly  stated that their own guidelines do not apply to press releases. However,  these organizations’ guidelines are intended to be broadly applicable to  both statistical and nonstatistical agencies. For example, the Department  of Justice comprises 38 separate component organizations that produce a  variety of types of information, both statistical and nonstatistical in nature.  The Office of Justice Programs is composed of 6 bureaus and program  offices, and these, too, produce both statistical and nonstatistical  information. Because we believe that press releases issued by the  department and the Office of Justice Programs may in some, but not all,  instances be statistical products, we do not hold the view that statistical  guidelines should be universally applicable to all press releases issued by  the Department of Justice and Office of Justice Programs. However,  because different interpretations can arise, we believe that clarification  regarding which guidelines should be applied under which  circumstances—and, specifically, to press releases—would be helpful to  statistical agencies that are in situations similar to BJS’s.
To address potential discrepancies such as these, in a May 2006 report on  the quality of federal data, we recommended that to help improve  governmentwide data dissemination practices that would further  safeguard the integrity of federal statistical data, OMB should consider  how best to address the gaps we identified between agencies’ data  dissemination practices and the National Research Council’s guidelines.  We noted in that report that OMB, in concert with federal statistical  agencies, was developing a governmentwide directive on the release and  dissemination of statistical products that, according to OMB officials,  parallels the council’s and other generally accepted dissemination  practices. We pointed out that it will be important for OMB’s directive to  consider, for example, whether the directive should cover principal  statistical agencies only, the statistical functions of all agencies, or only  statistical products. OMB officials indicated that the guidance is intended  to help ensure that statistical products are policy-neutral, timely, and  accurate. We recommended that, among other things, OMB include in this  directive (1) clear definitions of what is and is not covered, (2) the extent  to which agencies should document their data dissemination guidance and  how often the guidance should be reviewed, (3) the amount of flexibility  agencies have in implementing OMB’s guidance, and (4) procedures for  monitoring agencies’ adherence to its directive. To the extent that  statistical agencies appropriately follow these practices, the directive  could promote more consistent adherence to practices that facilitate  broader dissemination of statistical data and enhance its credibility.  Although OMB did not provide comments on the recommendations in our  2006 report, an OMB official told us that as of January 2007, OMB was still  working on this directive. We believe it remains important for OMB to  complete its directive on the release and dissemination of statistical  products in order to help safeguard the integrity of federal statistical data,  reduce the likelihood that the type of disagreement discussed in this  report would recur, and help assure both the actual and perceived  independence of BJS.
section_title: The Roles Played by Certain Outside Noncareer Appointees in the Press Release Process Affected BJS Carrying Out Its Role as an Independent Statistical Agency
paragraphs: The second key factor that affected whether and how BJS followed  guidelines concerned the involvement of noncareer appointees outside of  BJS in the press release issuance process, and had implications for BJS’s  independence as a statistical agency.  Specifically, we determined that BJS  was not in a position to fully follow the 3 National Research Council  guidelines listed in the previous section for the 2001 press release based  on the 1999 survey (the only applicable, available data quality guidelines in  place in 2001) because certain noncareer appointees outside of BJS and  within the Department of Justice, are vested—pursuant to the Department  of Justice’s and Office of Justice Programs’ policies defining the roles and  responsibilities of their noncareer appointees—with the ability to  participate in the review, approval, and dissemination of press releases.  In certain cases, the roles and responsibilities of these noncareer  appointees precluded BJS from being in the position to fully follow certain  guidelines. The Assistant Attorney General within the Department of  Justice’s Office of Justice Programs has general statutory responsibilities  with respect to coordinating the activities of that office and its various  components, such as BJS. These statutory provisions do not specifically  address the Office of Justice Programs’ role with respect to the review,  approval, and dissemination of press releases. However, under  departmental policy, noncareer appointees within the Department of  Justice and outside of BJS have the ability to participate in the press  release issuance process.
Table 3 shows the type of involvement that the Assistant Attorney General  in the Office of Justice Programs and other noncareer appointees generally  have had in the press release review, approval, and dissemination process.  Appendix IV describes in more detail the responsibilities of these various  officials associated with review, approval, and dissemination procedures  for both BJS reports and press releases.
With respect to the first of the three guidelines, which calls for a statistical  agency to have authority to release information without prior clearance, it  is our view that BJS was not in a position to follow this independence- related guideline at all because it did not have the ability to do so. This is  because press releases are subject to review and approval by not only the  BJS Director, but also by other Department of Justice noncareer  appointees. Outside of BJS, the noncareer appointees participating in the  clearance process are located in the Department of Justice’s Office of  Justice Programs (these include the Office’s Chief of Staff, Deputy  Assistant Attorney General, and Assistant Attorney General) and Office of  Public Affairs.
The current Assistant Attorney General and two former Assistant  Attorneys General in the Office of Justice Programs told us that there is no  written, formal policy or guidance that bounds their input and decision- making roles and responsibilities with respect to BJS press releases. BJS  and OJP officials indicated that the Office of Justice Programs’ Assistant  Attorney General has ultimate responsibility for the review and approval  of BJS press releases. Press releases are issued jointly on letterhead listing  BJS and the Department of Justice. The current BJS Director confirmed  that publication and dissemination functions for press releases are  considered to be within the Assistant Attorney General’s oversight  authority. Because the National Research Council stated that an aspect of  independence includes “recognition by policy officials outside the  statistical agency of its authority to release statistical information without  prior clearance,” we concluded that BJS was not in the position to follow  this guideline because, as we have stated, we believe the Police-Public  Contact Survey press release was a statistical product that BJS could not  issue independently.
In practice, the ways in which Assistant Attorneys General of the Office of  Justice Programs have exercised their authority have varied. For instance,  one former Office of Justice Programs’ Assistant Attorney General told us  that she placed “self-imposed” limits on her decisions to modify the  content of a BJS press release based on her awareness of congressional  support for, and her own belief in, the independence of statistical  agencies. The current Office of Justice Programs’ Assistant Attorney  General told us that she reviews only press releases that contain quotes  from the Attorney General. She said that since she assumed her position in  2005, there have been no BJS press releases that have quoted the Attorney  General, and she has relied on her Deputy Assistant Attorney General, the  BJS Director, and others to ensure the accuracy and clarity of press  releases. Nevertheless, the BJS Director must obtain the approval of the  Office of Justice Programs’ Assistant Attorney General and other Justice  noncareer appointees to issue a press release.
With respect to the second guideline, pertaining to the agency’s decisions  over the scope, content, and frequency of data compiled, analyzed, or  published, we found that BJS was not in a position to fully follow this  independence-related guideline. Specifically, we found that BJS could  exercise professional decisions about the frequency of data analyzed and  published (within available budgets), but did not always have complete  control over the scope and content of survey press releases to be issued.  As noted above, this was due to the fact that press releases are joint  products of BJS, the Office of Justice Programs, and the Department of  Justice, and noncareer appointees outside of BJS can become involved in  the press release process. BJS’s situation with respect to this second  guideline came to the fore during the drafting of a press release in 2005  based on the 2002 Police-Public Contact Survey. The press release that  BJS sought to publish would have included the following statistical  findings from the accompanying Police-Public Contact Survey report: (1)  there was no statistically significant difference between the rates that  white and minority drivers reported being stopped by police, and (2) once  stopped, a larger percentage of black and Hispanic minority drivers  reported police using or threatening to use force against them than did  whites. The then-BJS Director and the then-Acting Assistant Attorney  General had a difference of opinion regarding the presentation of the  second statistical finding, which was included in the Police-Public Contact  Survey report. Despite reported efforts on the part of both parties to  negotiate alternative language with respect to the content of the press  release, they could not resolve their differences and the BJS Director  decided that a press release would not be issued. The current BJS Director  told us that it is “inconceivable” that the Assistant Attorney General would  issue a press release without the BJS Director’s prior approval.
According to current BJS officials (both career and noncareer) and the  Office of Justice Programs’ Office of Communications staff, during the  period 1996-2006, this was the only instance in which a BJS press release  was prepared but not issued because the Office of Justice Programs and  BJS could not agree on the contents. In all other instances during this  period, according to these officials, when the parties disagreed on the  content of a press release, they were able to resolve their differences.
With respect to the third guideline, pertaining to actual or perceived  political interference and the timing of a release, we similarly believe BJS  was in not in a position to fully follow this independence-related guideline  for the 2001 press release, which, as discussed earlier, we believe to be a  statistical product. Although we found no evidence of political  interference with the timing of the 2001 survey press release issued from  the 1999 survey, we found that BJS does not have complete control over  the timing of press releases, as recommended by the National Research  Council. Since both noncareer appointees and career officials in the Office  of Justice Programs and the Department of Justice have a role in reviewing  and approving BJS press releases, they can affect the date that a press  release is issued. According to BJS, career and noncareer appointees  outside of BJS can delay the issuance of a press release for reasons having  nothing to do with political interference, such as a determination that the  press release is not sufficiently newsworthy at the time that it was  designated to be issued.
On balance, we believe that the noncareer appointees who played  decision-making roles in the Police-Public Contact Survey press release  process that we reviewed acted within the scope of the roles and  responsibilities accorded them under Department of Justice policies, and  that BJS made a reasonable effort to adhere to all applicable data quality  guidelines. The fact that certain noncareer officials have the ability to  make decisions that affect BJS’s ability to fully meet federal data quality  guidelines suggests, however, that the potential exists for BJS’s review,  approval, and dissemination process for statistical products to be subject  to political interference. Thus, certain actions by noncareer appointees— though made on the basis of professional judgment—could put them at  odds with the very guidelines designed to ensure the statistical  independence and integrity of agencies such as BJS.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Justice for review  and comment. On March 13, we received written comments on the draft  report from the Office of Justice Programs’ Assistant Attorney General,  and the comments are reproduced in full in appendix VI.
In her letter, the Assistant Attorney General affirmed several of our  findings and agreed that a need exists for clear definitions about what  quality guidelines cover.  She noted that competing interpretations exist  about what constitutes a statistical product and that the federal statistical  community would benefit from clarity in this area.
However, the Assistant Attorney General disagreed with our  characterization of the 2001 Police-Public Contact Survey press release as  a statistical product and, therefore, with our conclusion that the National  Research Council’s quality guidelines applied to this press release. The  Assistant Attorney General stated that “a press release … is a public  relations announcement issued to encourage media coverage. The mere  presence of statistics in a press release does not transform a press release  into a statistical product.” We do not believe and have not stated that the  mere presence of statistics in a press release in and of itself transforms it  into a statistical product any more than we believe or have stated that  labeling a document lacking in statistics but called a statistical product  necessarily transforms it into one.  The Assistant Attorney General also  stated that we “mischaracterized” BJS’s data quality guidelines as applying  to press releases because the guidelines apply only to the statistics  contained in BJS press releases, and because BJS conforms with OMB, the  Department of Justice, and the Office of Justice Programs in considering  press releases to be outside the scope of the guidelines. For the following  reasons, we maintain that we made a sound decision in applying BJS’s  guidelines to the Police-Public Contact Survey press release: (1) BJS’s  guidelines state that they “govern all justice statistics that BJS produces  and disseminates for the general public, including all statistics that are  featured in BJS publications, on the website, and in BJS press releases;”  and (2) the Police-Public Contact Survey press release was made up  almost entirely of survey statistics, indicating to us that it was a statistical  product.  In determining that the Police-Public Contact Survey was a  statistical product, we felt that the content of the press release was a more  important determinant than the label attached to it, or the fact that the  processes and staff involved in developing the press release were different  from those in BJS reports.
The Assistant Attorney General also noted that the National Research  Council’s written guidelines did not explicitly cover press releases.  Because we agree, we contacted the National Research Council and  consulted with officials of the Council’s Committee on National Statistics  (the authoring committee of the Principles and Practices).  The officials  concurred with our view that BJS press releases referring to statistical  products (as opposed to press releases about the announcement of a new  agency head, for example) are statistical products to which it is  appropriate to apply the guidelines.  Although the Principles and  Practices document did not specifically state that the guidelines covered  the content, scope, and timing of press releases issued by statistical  agencies, according to these officials it was not the Committee’s intent to  exclude such press releases from the guidelines.
The Assistant Attorney General also felt that the draft report overstated  the potential threats to BJS’s independence because we used the term  “statistical products” to refer to press releases. She was concerned with  our observation that the potential exists for BJS’s review, approval, and  dissemination process for statistical products to be subject to political  interference since noncareer officials can affect BJS’s ability to meet  federal data quality guidelines. We stand by this conclusion.  Department  of Justice policy permits noncareer appointees within the Department but  outside of BJS to participate in the press release process.  At the same  time however, we are unaware of anything that prevents future  modifications to that policy to similarly allow noncareer appointees to  participate in BJS’s report issuance. Thus, we believe that we have  correctly assessed the risk of potential or actual threats to BJS’s  independence.
Finally, the Assistant Attorney General stated that even if the council’s  written guidelines explicitly applied to press releases, the BJS director  would not adhere to them and no current law can make him do so.  We  recognize that they are voluntary and not legally required and never have  said otherwise.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from its  issue date. At that time, we will send copies of the report to the Attorney  General, the Director of the Office of Management and Budget, and other  interested parties. In addition, the report will be available at no charge on  GAO’s home page at http://www.gao.gov.
Please contact Brian Lepore at (202) 512-4523 or leporeb@gao.gov if you  or your staff have any questions concerning this report. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses the following two objectives for the 1999 and 2002  Police-Public Contact Surveys, the two surveys for which products had  been issued as of February 2007: (1) To what extent did the Bureau of  Justice Statistics (BJS) follow available guidelines to help ensure the  accuracy and integrity of the review, approval, and dissemination of  reports and press releases based on its surveys? (2) What key factors  affected whether and how BJS followed available guidelines? In addition,  we provide information on scope and methodology changes in the Police- Public Contact Surveys over time (see app. V).
To address the first objective, regarding the extent to which BJS followed  guidelines, we obtained quality guideline documents from BJS, the  Department of Justice’s Office of Justice Programs, the Department of  Justice, and the National Research Council. The guidelines that we  obtained from these organizations covered the period between February  2001, when the first product based on the 1999 Police-Public Contact  Survey was issued, and June 2006, when the most recent product based on  the 2002 survey was issued. We included these federal organizations in our  review because BJS is a component of the Office of Justice Programs,  which in turn is a component of the Department of Justice, and BJS  considers itself to be “governed by” the information quality guidelines of  these organizations. We included the National Research Council in our  review because it is a widely recognized organization that issued  guidelines that were intended to be statements of best practice and  provide information on what constitutes an effective statistical  organization. We also reviewed guidance and directives issued by the  Office of Management and Budget (OMB) because OMB is charged with  issuing governmentwide policy and procedural guidance to federal  agencies, which are then encouraged to issue their own implementation  guidelines.
We took several steps to determine the extent to which BJS followed the  specific quality guidelines that it, the Office of Justice Programs, the  Department of Justice, and the National Research Council had issued.  From the documents provided by these four organizations, a GAO analyst  initially identified a total of 63 guidelines that pertained to product review,  approval, and dissemination processes. For verification purposes, a GAO  methodologist also reviewed the guideline documents. The GAO  methodologist agreed with the auditor that all 63 guidelines were  appropriate for inclusion in our review. Because many of the guidelines  issued by the four organizations were similar and overlapping, the GAO  auditor reduced the list to 24 nonduplicative guidelines. The GAO  methodologist again reviewed the work of the auditor, and in all cases  agreed with the auditor that similar guidelines were being appropriately  grouped.
We then developed a data collection instrument to determine whether BJS  was following guidelines for the 1999 and 2002 Police-Public Contact  Surveys, on which information could be recorded as to whether BJS fully  followed, partially followed, or did not at all follow each of the guidelines.  We defined “fully” as all aspects of the guideline being followed; “partially”  as some, but not all, aspects of the guideline being followed; and “not at  all” as no aspects of the guideline being followed. We asked BJS to  complete a separate data collection instrument with respect to each of its  1999 and 2002 Police-Public Contact Survey reports and one press release,  and to support each response by providing documentary evidence. To  decrease the burden on BJS, GAO analysts completed the data collection  instrument for 9 of the 24 guidelines, for which we already had sufficient  information (for example, documents describing agency processes and  procedures, and interviews regarding the roles and responsibilities of  noncareer appointees). We provided our assessments regarding these  guidelines to BJS and asked officials to either confirm or not confirm  them. Two GAO analysts reviewed BJS’s responses and all available  supporting documentary and testimonial evidence, and determined  whether BJS fully, partially, or did not at all meet each guideline. We  provided our findings to BJS for review and comment.
BJS’s 1996 and 2005 Police-Public Contact Surveys were outside the scope  of our work. We excluded the 1996 survey because that was a relatively  small-scale pilot study; and we excluded the 2005 survey, the most recent  Police-Public Contact Survey conducted, because no reports or press  releases have yet been issued from this work.
To address the second objective, regarding key factors that affected  whether and how BJS followed guidelines, we reviewed processes and  procedures that described the review, approval, and dissemination  processes for BJS-generated reports and press releases, with particular  interest in identifying the roles of noncareer appointees involved in each  of these processes. We also reviewed pertinent statutory provisions  relating to the roles and responsibilities of officials with respect to BJS.  We conducted in-person interviews with, or obtained written responses to  our questions from, noncareer appointees in BJS, the Office of Justice  Programs, and the Department of Justice’s Office of Public Affairs.  Specifically, we conducted in-person interviews with the current BJS  Director and the BJS Director who was involved in the disagreement with  the Acting Assistant Attorney General, as well as with the current  Assistant Attorney General and Deputy Assistant Attorney General in the  Office of Justice Programs. We obtained detailed written responses to our  questions from a former BJS Director, the Acting Assistant Attorney  General who was involved in the disagreement with the BJS Director, and  two former Assistant Attorneys General from the Office of Justice  Programs. We conducted a telephone interview with the current Deputy  Director of the Department of Justice’s Office of Public Affairs. Among  other things, we asked these noncareer appointees to provide us with  information about BJS’s process for reviewing, approving, and  disseminating reports and press releases; the roles and responsibilities of  noncareer appointees in that process; changes, if any, that had occurred in  the roles played by noncareer appointees; procedures used to help ensure  that BJS reports and press releases were accurate, reliable, and unbiased;  and any factors that may have affected BJS’s independence in the product  issuance process. Finally, we reviewed the guidelines of BJS, the  Department of Justice’s Office of Justice Programs, the Department of  Justice, and the National Research Council to determine that they  reflected the product issuance processes and to consolidate them in order  to eliminate duplication.
To determine what changes, if any, have occurred in the scope and  methodology of the Police-Public Contact Surveys between 1996 and 2006,  which we present in appendix V, we initially developed a matrix of key  scope and methodology dimensions, based on a review of the standard  social science literature. We then conducted interviews and reviewed  documents with respect to these dimensions, for all four Police-Public  Contact Surveys—the 1996 pilot survey and the surveys of 1999, 2002, and  2005. We interviewed the current BJS Director and a former BJS Director,  and available report authors and the key statistician participating in  administrations of the survey, to ascertain their views concerning the  intended scope of the four surveys, the methodologies used, scope and  methodology changes that were made, and reasons for any changes. We  also obtained written responses to our questions from these officials. We  conducted a detailed documentary review of the scoping and methodology  sections of the issued Police-Public Contact Survey reports and press  releases, and extracted information about changes in the data collection  instruments used (for example, the numbers and types of questions asked  about searches and the use of force). In addition, we reviewed documents  prepared by the American Statistical Association and the U.S. Bureau of  the Census, which conducted field tests to ensure that Police-Public  Contact Survey questions were appropriately devised. In cases where we  noted that changes had been made between surveys, we reviewed Census  Bureau documentation and interviewed staff and officials at BJS.
It was beyond the scope of this review to address any personnel issues  that may have arisen in connection with the disagreement over the content  of the 2005 draft press release based on the 2002 Police-Public Contact  Survey. We conducted our work between April 2006 and January 2007 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Applicable Quality Guidelines for Statistical Products Issued by the Bureau of Justice Statistics and Other Agencies
paragraphs: BJS followed numerous recommended data quality guidelines designed to  help ensure the accuracy and integrity of the Police-Public Contact Survey  products that it issued in 2001, 2002, 2005, and 2006 based on its 1999 and  2002 surveys. The product issuance guidelines were used to aid BJS’s  efforts to review, approve, and disseminate these statistical products to  the public and others.
The guidelines were issued at various points in time by the following  organizations:  the National Research Council,  the Bureau of Justice Statistics,  the Department of Justice, and  the Office of Justice Programs.
In addition to reviewing the guidelines of these four organizations, we also  reviewed guidelines and directives issued by the Office of Management  and Budget (OMB). However, we did not specifically assess BJS’s practice  with respect to following OMB’s guidelines because OMB issued  governmentwide policy and procedural guidance to federal agencies that  called for agencies to develop their own implementing guidelines.
Table 4 shows the guidelines that were available at the time BJS’s 1999 and  2002 Police-Public Contact Survey products were issued, and which  guidelines BJS was in a position to follow.
section_title: Appendix III: Press Release Issued in 2001 Based on BJS’s 1999 Police-Public Contact Survey
section_title: Appendix IV: BJS Processes for the Review, Approval, and Dissemination of Police-Public Contact Survey Products
subsections: section_title: Review, Approval, and Dissemination Process for Reports Issued from Police-Public Contact Surveys
paragraphs: Since the inception of the Police-Public Contact Survey in 1996, the BJS  Director has been the single noncareer appointee who has had a decision- making role in BJS’s review, approval, and dissemination processes for  reports. The BJS Director is a noncareer presidential appointee subject to  Senate confirmation. Figure 2 provides an overview of the process  followed by BJS in the review, approval, and dissemination of Police- Public Contact Survey reports.
As indicated by the figure, the BJS report author and supervisor prepare  the draft report for review and approval. The BJS Director reviews the  draft, requests any changes, approves the final draft, and transmits a  memorandum of notification through the Office of Justice Program’s  Assistant Attorney General up the chain of command to the Attorney  General. The memorandum contains an abstract of the report, selected  survey findings, and a projected release date for the report. BJS sets the  release date for 30 days from the date that the Assistant Attorney General  signs the memorandum of notification. The report is posted to the Web  site at that time, or sooner if the date and time are specified in the  notification memo.
section_title: Review, Approval, and Dissemination Process for Press Releases Issued from Police-Public Contact Surveys
paragraphs: In contrast to the process followed for survey reports, several noncareer  appointees are involved in the organizational review, approval, and  dissemination process, as shown in figure 3.
As indicated in the figure, in addition to the Director of BJS, there are  three noncareer appointees within the Office of Justice Programs who  participate in the review and approval process—the Chief of Staff, the  Deputy Assistant Attorney General, and the Assistant Attorney General,  and at least one noncareer appointee within Department of Justice  headquarters: the Director of the Office of Public Affairs.
The BJS report author and supervisor jointly work with staff from the  Office of Justice Program’s Office of Communications to prepare the press  release. The BJS director reviews the draft press release, requests any  changes, approves the final draft, and transmits the press release up the  chain of command to the Office of Justice Programs’ Assistant Attorney  General for review and approval. Following approval by the Assistant  Attorney General, the Department of Justice’s Office of Public Affairs  reviews the press release for clarity, and the BJS Director then verifies  that the information in the press release is accurate. The Department of  Justice’s Office of Public Affairs is then responsible for disseminating the  press release to Congress, the media, and executive department press  offices, while BJS is responsible for disseminating the press release  through its Web site.
section_title: Appendix V: Changes in the Scope and Methodology of the Police-Public Contact Surveys
paragraphs: BJS has conducted four Police-Public Contact Surveys as supplements to  the National Crime Victimization Survey. The first Police-Public Contact  Survey was conducted as a pilot in 1996. Three subsequent, more  extensive surveys were conducted at 3-year intervals: 1999, 2002, and 2005.  Although we do not discuss the 2005 survey in this report because no  reports or press releases have yet been issued from this survey, we present  information on the 2005 survey in this appendix because information is  available on this survey’s scope and methodology. The scope of the Police- Public Contact Surveys has consistently expanded over time, while the  methodology has remained generally consistent.
subsections: section_title: The Scope of the Police- Public Contact Survey Has Expanded with Each Additional Survey
paragraphs: The pilot survey was designed to test whether the survey could be  effectively used as a supplement to the National Crime Victimization  Survey to collect data on (a) the types of contacts the public have with the  police, and (b) police use of force. To conduct this test, BJS employed a  representative sample of 6,421 U.S. residents. The pilot survey yielded  useful information on the various types of contacts the public had with the  police, and whether force was used by the police. However, the sample  size of the pilot survey was not sufficiently large for BJS to draw  inferences about the extent to which the population at large would report  that they experienced “excessive” use of force by police.
For its Police-Public Contact Survey in 1999, BJS increased the sample size  to a representative sample of 80,543 U.S. residents. The scope of the  survey was further enhanced by adding questions about traffic stops (the  most common form of public-police contact, as determined in the 1996  pilot survey), and including a question on whether the police used  excessive force during any contact with the public. BJS officials told us  that they added the traffic stop questions, at least in part, to “address the  growing public concern about racial profiling in connection with traffic  stops.”
In its 2002 survey, BJS expanded and refined its survey questions further.  Specifically, according to BJS officials, they added questions that would  help BJS estimate the extent to which U.S. residents nationwide would say  that (1) they were stopped by the police while driving, (2) they or their  vehicle were searched by the police without their permission during a  traffic stop, and (3) they were arrested as a result of the search. In  addition, BJS officials said that they added questions to estimate  differences, if any, among racial groups in their rates of traffic stops at  various times of the day, and whether police used force in situations  where persons were engaged in such behaviors as arguing with, cursing, or  disobeying police.
In 2005, according to BJS officials, the scope of the Police-Public Contact  Survey was further extended in several ways, including the following: (a)  respondents were asked whether they had been arrested for driving under  the influence of alcohol during the year (in order to make comparisons  with Federal Bureau of Investigation (FBI) arrest rates, so that potential  undercounting rates could be determined); (b) respondents were  permitted to group themselves into any combination of racial categories  (rather than choosing a single category) to better refine respondent  demographic status; (c) respondents were asked whether police used  force during any of their police contacts during the year, as opposed to the  more limiting question in 2002, which was directed only toward the most  recent contact with police; and (d) respondents were provided open-ended  response fields on the survey instrument to indicate any ways they  believed that the police had acted inappropriately toward them.
section_title: The Police-Public Contact Survey Methodology Has Remained Consistent Following the 1996 Survey
paragraphs: The 1999, 2002, and 2005 Police-Public Contact Surveys have consistently  maintained a similar methodology. The methodological dimensions of the  surveys that have remained consistent are geographic coverage, target  population, sampling design, data collection method, sample size/response  rate, survey administration, and sample characteristics (as indicated in  table 5). To illustrate, all three surveys involved selecting nationally  representative, stratified, multi-cluster samples from the population of U.S.  residents 16 years of age or older. The data were collected either through  face-to-face or computer-assisted telephone interviews. The surveys were  administered during the last 6 months of the year, and the demographic  characteristics of the samples were similar across time periods.
The 1996 pilot survey differed in several ways from the subsequent three  surveys. Specifically, the pilot survey included residents younger than 16,  included far fewer people than the subsequent surveys, and limited the  sampling to individuals who had participated in the last round of the  National Crime Victimization Survey. In addition, the percentage of face- to-face interviews was lower, and the survey administration period was  shorter, and during a different time of the year, than in the other three  surveys.
section_title: Appendix VI: Comments from the Department of Justice
section_title: Appendix VII: GAO Contact and Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the above, Evi. L. Rezmovic, Assistant Director; Ronald S.  Fecso, Chief Statistician; Jared A. Hermalin; Karen A. Jarzynka; Amanda K.  Miller; Amy L. Bernstein; Geoffrey R. Hamilton; Robert Alarapon; and  Tracy J. Harris made key contributions to this report.