Justice Impact Evaluations: One Byrne Evaluation Was Rigorous; All Reviewed Violence Against Women Office Evaluations Were Problematic
Released: 2002-03-07T00:00:00

Letter

Background

The Justice Assistance Act of 1984 (P.L. 98-473) created OJP to provide federal leadership in developing the nation’s capacity to prevent and control crime, administer justice, and assist crime victims. OJP carries out its responsibilities by providing grants to various organizations, including state and local governments, Indian tribal governments, nonprofit organizations, universities, and private foundations. OJP comprises five bureaus, including BJA, and seven program offices, including VAWO. In fulfilling its mission, BJA provides grants for programs and for training and technical assistance to combat violent and drug-related crime and help improve the criminal justice system. VAWO administers grants to help prevent and stop violence against women, including domestic violence, sexual assault, and stalking. During fiscal years 1995 through 2001, BJA and VAWO awarded about $943 million to fund 700 Byrne and 1,264 VAWO discretionary grants.

One of BJA’s major grant programs is the Byrne Program. BJA administers the Byrne program, just as its counterpart, VAWO, administers its programs. Under the Byrne discretionary grants program, BJA provides federal financial assistance to grantees for educational and training programs for criminal justice personnel; for technical assistance to state and local units of government; and for projects that are replicable in more than one jurisdiction nationwide. During fiscal years 1995 through 2001, Byrne discretionary grant programs received appropriations of about $385 million.

VAWO was created in 1995 to carry out certain programs created under the Violence Against Women Act of 1994. The Victims of Trafficking and Violence Prevention Act of 2000 reauthorized most of the existing VAWO programs and added new programs. VAWO programs seek to improve criminal justice system responses to domestic violence, sexual assault, and stalking by providing support for law enforcement, prosecution, courts, and victim advocacy programs across the country. During fiscal years 1995 through 2001, VAWO’s five discretionary grant programs that were subject to program evaluation were (1) STOP (Services, Training, Officers, and Prosecutors) Violence Against Indian Women Discretionary Grants, (2) Grants to Encourage Arrest Policies, (3) Rural Domestic Violence and Child Victimization Enforcement Grants, (4) Domestic Violence Victims’ Civil Legal Assistance Grants, and (5) Grants to Combat Violent Crimes Against Women on Campuses. During fiscal years 1995 through 2001, about $505 million was appropriated to these discretionary grant programs.

As already mentioned, NIJ is the principal research and development agency within OJP, and its duties include developing, conducting, directing, and supervising Byrne and VAWO discretionary grant program evaluations. Under 42 U.S.C. 3766, NIJ is required to “conduct a reasonable number of comprehensive evaluations” of the Byrne discretionary grant program. In selecting programs for review under section 3766, NIJ is to consider new and innovative approaches, program costs, potential for replication in other areas, and the extent of public awareness and community involvement. According to NIJ officials, the implementation of various types of evaluations, including process and impact evaluations, fulfills this legislative requirement. Although legislation creating VAWO does not require evaluations of the VAWO discretionary grant programs, Justice’s annual appropriations for VAWO during fiscal years 1998 through 2002 included monies for NIJ research and evaluations of violence against women. In addition, Justice has promulgated regulations requiring that NIJ conduct national evaluations of two of VAWO’s discretionary grant programs. As with the Byrne discretionary programs, NIJ is not required by statute or Justice regulation to conduct specific types of program evaluations, such as impact or process evaluations.

The Director of NIJ is responsible for making the final decision on which Byrne and VAWO discretionary grant programs to evaluate; this decision is based on the work of NIJ staff in coordination with Byrne or VAWO program officials. Once the decision has been made to evaluate a particular program, NIJ issues a solicitation for proposals for grant funding from potential evaluators. When applications or proposals are received, an external peer review panel comprising members of the research and relevant practitioner communities is convened. Peer review panels identify the strengths, weaknesses, and potential methodologies to be derived from competing proposals. When developing their consensus reviews, peer review panels are to consider the quality and technical merit of the proposal; the likelihood that grant objectives will be met; the capabilities, demonstrated productivity, and experience of the evaluators; and budget constraints. Each written consensus review is reviewed and discussed with partnership agency representatives (e.g., staff from BJA or VAWO). These internal staff reviews and discussions are led by NIJ’s Director of the Office of Research and Evaluation who then presents the peer review consensus reviews, along with agency and partner agency input, to the NIJ Director for consideration and final grant award decisions. The NIJ Director makes the final decision regarding which application to fund.

Scope and Methodology

To meet our objectives, we conducted our work at OJP, BJA, VAWO, and NIJ headquarters in Washington, D.C. We reviewed applicable laws and regulations, guidelines, reports, and testimony associated with Byrne and VAWO discretionary grant programs and evaluation activities. In addition, we interviewed responsible OJP, NIJ, BJA, and VAWO officials regarding program evaluations of discretionary grants. As agreed with your offices, we focused on program evaluation activities associated with the Byrne and VAWO discretionary grant programs. In particular, we focused on the program evaluations of discretionary grants that were funded during fiscal years 1995 through 2001.

To address our first objective, regarding the number, type, status of completion, and award amount of Byrne and VAWO discretionary grant program evaluations, we interviewed NIJ, BJA, and VAWO officials and obtained information on Byrne and VAWO discretionary grant programs and program evaluations. Because NIJ is responsible for carrying out program evaluations of Byrne and VAWO discretionary grant programs, we also obtained and analyzed NIJ data about specific Byrne and VAWO discretionary grant program evaluations, including information on the number of evaluations as well as the type, cost, source of funding, and stages of implementation of each evaluation for fiscal years 1995 through 2001. We did not independently verify the accuracy or completeness of the data that NIJ provided.

To address the second objective, regarding the methodological rigor of the impact evaluation studies of Byrne and VAWO discretionary grant programs during fiscal years 1995 through 2001, we initially identified the impact evaluations from the universe of program evaluations specified by NIJ. We excluded from our analysis any impact evaluations that were in the formative stage of development—that is, the application had been awarded but the methodological design was not yet fully developed. As a result, we reviewed four program evaluations.

For the four impact evaluations that we reviewed, we asked NIJ to provide any documentation relevant to the design and implementation of the impact evaluation methodologies, such as the application solicitation, the grantee’s initial and supplemental applications, progress notes, interim reports, requested methodological changes, and any final reports that may have become available during the data collection period. We also provided NIJ with a list of methodological issues to be considered in our review and requested them to submit any additional documentation that addressed these issues. We used a data collection instrument to obtain information systematically about each program being evaluated and about the features of the evaluation methodology. We based our data collection and assessments on generally accepted social science standards. We examined such factors as whether evaluation data were collected before and after program implementation; how program effects were isolated (i.e., the use of nonprogram participant comparison groups or statistical controls); and the appropriateness of sampling and outcome measures. Two of our senior social scientists with training and experience in evaluation research and methodology separately reviewed the evaluation documents and developed their own assessments before meeting jointly to discuss the findings and implications. This was done to promote a grant evaluation review process that was both independent and objective.

To obtain information on the approaches that BJA, VAWO, and NIJ used to disseminate program evaluation results, we requested and reviewed, if available, relevant handbooks and guidelines on information dissemination, including, for example, NIJ’s guidelines. We also reviewed BJA, VAWO, and NIJ’s available print and electronic products as related to their proven programs and evaluations, including two NIJ publications about Byrne discretionary programs and their evaluation methodologies and results.

We conducted our work between February 2001 and December 2001 in accordance with generally accepted government auditing standards.

We requested comments from Justice on a draft of this report in January 2002. The comments are discussed near the end of this letter and are reprinted as appendix III.

Number, Type, Status of Completion, and Award Amount of Byrne and VAWO Discretionary Grant Program Evaluations

During fiscal years 1995 through 2001, NIJ awarded about $6 million to carry out five Byrne and five VAWO discretionary grant program evaluations. NIJ awarded evaluation grants using mostly funds transferred from BJA and VAWO. Specifically, of the approximately $1.9 million awarded for one impact and four process evaluations of the Byrne discretionary program, NIJ contributed about $299,000 (16 percent) and BJA contributed about $1.6 million (84 percent). VAWO provided all of the funding (about $4 million) to NIJ for all program evaluations of five VAWO discretionary grant programs. According to NIJ, the five VAWO program evaluations included both impact and process evaluations. Our review of information provided by NIJ showed that 6 of the 10 program evaluations—all 5 VAWO evaluations and 1 Byrne evaluation—included impact evaluations. The remaining four Byrne evaluations were exclusively process evaluations that measured the extent to which the programs were working as intended. As of December 2001, only one of these evaluations, the impact evaluation of the Byrne CAR Program, had been completed. The remaining evaluations were in various stages of implementation. Table 1 lists each of the five Byrne program evaluations and shows whether it was a process or an impact evaluation, its stage of implementation, the amount awarded during fiscal years 1995 through 2001, and the total amount awarded since the evaluation was funded.

Table 2 lists each of the five VAWO program evaluations and shows that it was both a process and an impact evaluation, its stage of implementation, and the amount awarded during fiscal years 1995 through 2001, which is the total amount awarded.

Methodological Problems Have Adversely Affected Three of Four Impact Evaluations

Our review showed that methodological problems have adversely affected three of the four impact evaluations that have progressed beyond the formative stage. All three VAWO evaluations that we reviewed demonstrated a variety of methodological limitations, raising concerns as to whether the evaluations will produce definitive results. The one Byrne evaluation was well designed and used appropriate data collection and analytic methods. We recognize that impact evaluations, such as the type that NIJ is managing, can encounter difficult design and implementation issues. In the three VAWO evaluations that we reviewed, program variation across sites has added to the complexity of designing the evaluations. Sites could not be shown to be representative of the programs or of particular elements of these programs, thereby limiting the ability to generalize results; the lack of comparison groups hinders the ability to minimize the effects of factors external to the program. Furthermore, data collection and analytical problems compromise the ability of evaluators to draw appropriate conclusions from the results. In addition, peer review committees found methodological problems in two of the three VAWO evaluations that we considered.

The four program evaluations are multiyear, multisite impact evaluations. Some program evaluations used a sample of grants, while others used the entire universe of grants. For example, the Grants to Encourage Arrests Policies Program used 6 of the original 130 grantee sites. In contrast, in the Byrne Children at Risk impact evaluation, all five sites participated. As of December 2001, NIJ had already received the impact findings from the Byrne Children at Risk Program evaluation but had not received impact findings from the VAWO discretionary grant program evaluations.

Impact Evaluations Are Difficult to Successfully Design and Implement

An impact evaluation is an inherently difficult task, since the objective is to isolate the effects of a particular program or factor from all other potential contributing programs or factors that could also effect change. Given that the Byrne and VAWO programs are operating in an ever changing, complex environment, measuring the impact of these specific Byrne and VAWO programs can be arduous. For example, in the evaluation of VAWO’s Rural Domestic Violence Program, the evaluator’s responsibility is to demonstrate how the program affected the lives of domestic violence victims and the criminal justice system. Several other programs or factors besides the Rural Domestic Violence Program may be accounting for all or part of the observed changes in victims’ lives and the criminal justice system (e.g., a co-occurring program with similar objectives, new legislation, a local economic downturn, an alcohol abuse treatment program). Distinguishing the effects of the Rural Domestic Violence Program requires use of a rigorous methodological design.

Project Variation within the VAWO Programs Complicates Evaluation Design and Implementation

All three VAWO programs permitted their grantees broad flexibility in the development of their projects to match the needs of their local communities. According to the Assistant Attorney General, this variation in projects is consistent with the intent of the programs’ authorizing legislation. We recognize that the authorizing legislation provides VAWO the flexibility in designing these programs. Although this flexibility may make sense from a program perspective, the resulting project variation makes it more difficult to design and implement a definitive impact evaluation of the program. Instead of assessing a single, homogeneous program with multiple grantees, the evaluation must assess multiple configurations of a program, thereby making it difficult to generalize about the entire program. Although all of the grantees’ projects under each program being evaluated are intended to achieve the same or similar goals, an aggregate analysis could mask the differences in effectiveness among individual projects and thus not result in information about which configurations of projects work and which do not.

The three VAWO programs exemplify this situation. The Arrest Policies Program provided grantees with the flexibility to develop their respective projects within six purpose areas: implementing mandatory arrest or proarrest programs and policies in police departments, tracking domestic violence cases, centralizing and coordinating police domestic violence operations, coordinating computer tracking systems, strengthening legal advocacy services, and educating judges and others about how to handle domestic violence cases. Likewise, the STOP Grants Program encouraged tribal governments to develop and implement culture-specific strategies for responding to violent crimes against Indian women and provide appropriate services for those who are victims of domestic abuse, sexual assault, and stalking. Finally, the Rural Domestic Violence Program was designed to provide sites with the flexibility to develop projects, based on need, with respect to the early identification of, intervention in, and prevention of woman battering and child victimization; with respect to increases in victim safety and access to services; with respect to enhancement of the investigation and prosecution of crimes of domestic violence; and with respect to the development of innovative, comprehensive strategies for fostering community awareness and prevention of domestic abuse. Because participating grant sites emphasized different project configurations, the resulting evaluation may not provide information that could be generalized to a broader implementation of the program.

Site Selection Not Shown to be Representative within the VAWO Programs

The sites participating in the three VAWO evaluations were not shown to be representative of their programs. Various techniques are available to help evaluators choose representative sites and representative participants within those sites. Random sampling of site and participant selection are ideal, but when this is not feasible, other purposeful sampling methods can be used to help approximate the selection of an appropriate sample (e.g., choosing the sample in such proportions that it reflects the larger population--stratification). At a minimum, purposeful selection can ensure the inclusion of a range of relevant sites.

As discussed earlier, in the case of the Arrest Policies Program, six purpose areas were identified in the grant solicitation. The six grantees chosen for participation in the evaluation were not however, selected on the basis of their representativeness of the six purpose areas or the program as a whole. Rather, they were selected on the basis of factors related solely to program “stability;” that is; they were considered likely to receive local funding after the conclusion of federal grant funding, and key personnel would continue to participate in the coordinated program effort. Similarly, the 10 Rural Domestic Violence impact evaluation grantees were not selected for participation on the basis of program representativeness or the specific purpose areas discussed earlier. Rather, sites were selected by the grant evaluator on the basis of “feasibility”; specifically, whether the site would be among those participants equipped to conduct an evaluation. Similarly, the STOP Violence Against Indian Women Program evaluation used 3 of the original 14 project sites for a longitudinal study; these were not shown to be representative of the sites in the overall program. For another phase of the evaluation, the principal investigator indicated that grantee sites were selected to be geographically representative of American Indian communities. While this methodology provides for inclusion of a diversity of Indian tribes in the sample from across the country, geography as a sole criterion does not guarantee representativeness in relation to many other factors.

Lack of Comparison Groups in the VAWO Evaluations Hinders Ability to Isolate the Effects of the Programs

Each of the three VAWO evaluations was designed without comparison groups—a factor that hinders the evaluator’s ability to isolate and minimize external factors that could influence the results of the study. Use of comparison groups is a standard practice employed by evaluators to help determine whether differences between baseline and follow-up results are due to the program under consideration or to some other programs or external factors. For example, as we reported in 1997, to determine whether a drug court program has been effective in reducing criminal recidivism and drug relapse, it is not sufficient to merely determine whether those participating in the drug court program show changes in recidivism and relapse rates. Changes in recidivism and relapse variables between baseline and program completion could be due to other external factors, irrespective of the drug court program (e.g., the state may have developed harsher sentencing procedures for those failing to meet drug court objectives). If, however, the drug court participant group is matched at baseline against another set of individuals, “the comparison group” who are experiencing similar life circumstances but who do not qualify for drug court participation (e.g., because of area of residence), then the comparison group can help in isolating the effects of the drug court program. The contrasting of the two groups in relation to recidivism and relapse can provide an approximate measure of the program’s impact.

All three VAWO program impact evaluations lacked comparison groups. One issue addressed in the Arrest Policies Program evaluation, for example, was the impact of the program on the safety and protection of the domestic violence victim. The absence of a comparison group, however, makes it difficult to firmly conclude that change in the safety and protection of participating domestic abuse victims is due to the Arrest Policies Program and not to some other external factors operating in the environment (e.g., economic changes, nonfederal programs such as safe houses for domestically abused women, and church-run support programs). Instead of using comparison groups, the Arrest Policies Program evaluation sought to eliminate potential competing external factors by collecting and analyzing extensive historical and interview data from subjects and by conducting cross-site comparisons; the latter method proved unfeasible.

The STOP Violence Against Indian Women Discretionary Grant Program has sought in part, to reduce violent crimes against Indian women by changing professional staff attitudes and behaviors. To do this, some grantees created and developed domestic violence training services for professional staff participating in site activities. Without comparison groups, however, assessing the effect of the STOP training programs is difficult. Attitudes and behaviors may change for myriad reasons unrelated to professional training development initiatives. If a treatment group of professional staff receiving the STOP training had been matched with a comparison group of professional staff that was similar in all ways except receipt of training, there would be greater confidence that positive change could be attributed to the STOP Program. Similarly, the lack of comparison groups in the Rural Domestic Violence evaluation makes it difficult to conclude that a reduction in violence against women and children in rural areas can be attributed entirely, or in part, to the Rural Domestic program. Other external factors may be operating.

VAWO Data Collection and Analytical Problems Evident during Grant Implementation

All three VAWO impact evaluations involved data collection and analytical problems that may affect the validity of the findings and conclusions. For example, we received documentation from NIJ on the STOP Grant Program for Reducing Violence Against Indian Women showing that only 43 percent of 127 grantees returned a mail survey. In addition, only 25 percent of 127 tribes provided victim outcome homicide and hospitalization rates—far less than the percentage needed to draw broad- based conclusions about the intended goal of assessing victim well being. In the Arrest Policies evaluation, NIJ reported that the evaluators experienced difficulty in collecting pre-grant baseline data from multiple sites and the quality of the data was oftentimes inadequate, which hindered their ability to statistically analyze change over time. In addition, evaluators were hindered in several work areas by lack of automated data systems; data were missing, lost, or unavailable; and the ability to conduct detailed analyses of the outcome data was sometimes limited. For the Rural Domestic Violence evaluation, evaluators proposed using some variables (e.g., number and type of awareness messages disseminated to the community each month, identification of barriers to meeting the needs of women and children, and number of police officers who complete a training program on domestic violence) that are normally considered to relate more to a process evaluation than an impact evaluation. NIJ noted that outcome measurement indicators varied by site, complicating the ability to draw generalizations. NIJ further indicated that the evaluation team did not collect baseline data prior to the start of the program, making it difficult to identify change resulting from the program.

VAWO Peer Review Committees Expressed Concerns about Two Evaluations

NIJ does not require applicants to use particular evaluation methodologies. NIJ employs peer review committees in deciding which evaluation proposals to fund. The peer review committees expressed concerns about two of the three VAWO program evaluation proposals (i.e., those for the Arrest Policies and Rural Domestic Violence programs) that were subsequently funded by NIJ. Whereas NIJ funded the Arrest Policies evaluation as a grant, NIJ funded the Rural Domestic Violence evaluation as a cooperative agreement so that NIJ could provide substantial involvement in conducting the evaluation.

A peer-review panel and NIJ raised several concerns about the Arrest Policies Program evaluation proposal. These concerns included issues related to site selection, victim interviewee selection and retention in the sample, and the need for additional impact measures and control variables. The grant applicant’s responses to these issues did not remove concerns about the methodological rigor of the application, thus calling into question the ability of the grantee to assess the impact of the Arrest Policies Program. For example, the grantee stated that victim interviewee selection was to be conducted through a quota process and that the sampling would vary by site. This would not allow the evaluators to generalize program results. Also, the evaluators said that they would study communities at different levels of “coordination” when comparison groups were not feasible, but they did not adequately explain (1) how the various levels of coordination would be measured, (2) the procedures used to select the communities compared, and (3) the benefits of using this method as a replacement for comparison groups. NIJ subsequently funded this evaluation, and it is still in progress.

A peer review committee for the Rural Domestic Violence and Child Victimization Enforcement Grant Program evaluation also expressed concerns about whether the design of the evaluation application, as proposed, would demonstrate whether the program was working. In its consensus review notes, the peer review committee indicated that the “ability to make generalizations about what works and does not work will be limited.” The peer review committee also warned of outside factors (e.g., unavailability of data, inaccessibility of domestic violence victims) that could imperil the evaluation efforts of the applicant. Based on the peer review committee’s input, NIJ issued the following statement to the applicant: “As a national evaluation of a major programmatic effort we hope to have a research design and products on what is working, what is not working, and why. We are not sure that the proposed design will get us to that point.” We reviewed the grant applicant’s response to NIJ’s concern in its application addendum and found that the overall methodological design was still not discussed in sufficient detail or depth to determine whether the program was working. Although the Deputy Director of NIJ’s Office of Research and Evaluation asserted that this initial application was only for process evaluation funding, our review of available documents showed that the applicant had provided substantial information about both the process and impact evaluation methodologies in the application and addendum. We believe that the methodological rigor of the addendum was not substantially improved over that of the original application. The Deputy Director told us that, given the “daunting challenge faced by the evaluator,” NIJ decided to award the grant as a cooperative agreement. Under this arrangement, NIJ was to have substantial involvement in helping the grantee conduct the program evaluation. The results of that evaluation have not yet been submitted. The evaluator’s draft final report is expected no earlier than April 2002.

Byrne Evaluation Was Successfully Designed and Implemented

In contrast to the three VAWO impact evaluations, the Byrne impact evaluation employed methodological design and implementation procedures that met a high standard of methodological rigor, fulfilling each of the criteria indicated above. In part, this may reflect the fact that Byrne’s CAR demonstration program, unlike the VAWO programs, was according to the Assistant Attorney General, intended to test a research hypothesis, and the evaluation was designed accordingly. CAR provided participants with the opportunity to use a limited number of program services (e.g., family services, education services, after-school activities) that were theoretically related to the impact variables and the prevention and reduction of drug use and delinquency. As a result, the evaluation was not complicated by project heterogeneity. All five grantees participated in the evaluation. High-risk youths within those projects were randomly selected from targeted neighborhood schools, providing student representation. Additionally, CAR evaluators chose a matched comparison group of youths with similar life circumstances (e.g., living in distressed neighborhoods and exposed to similar school and family risk factors) and without access to the CAR Program. Finally, no significant data collection implementation problems were associated with the CAR Program. The data were collected at multiple points in time from youths (at baseline, at completion of program, and at one year follow-up) and their caregivers (at baseline and at completion of program). Self-reported findings from youths were supplemented by the collection of more objective data from school, police, and court records on an annual basis, and rigorous test procedures were used to determine whether changes over time were statistically significant.

Additionally, CAR’s impact evaluation used control groups, a methodologically rigorous technique not used in the three VAWO evaluations. To further eliminate the effects of external factors, youths in the targeted neighborhood schools were randomly assigned either to the group receiving the CAR Program or to a control group that did not participate in the program. Since the CAR Program group made significant gains over the same-school group and the matched comparison group not participating in the program, there was good reason to conclude that the CAR Program was having a beneficial effect on the targeted audience.

Appendix I provides summaries of the four evaluations.

Conclusions

Despite great interest in assessing results of OJP’s discretionary grant programs, it can be extremely difficult to design and execute evaluations that will provide definitive information. Our in-depth review of one Byrne and three VAWO impact evaluations that have received funding since fiscal year 1995 has shown that, in some cases, the flexibility that can be beneficial to grantees in tailoring programs to meet their communities’ needs has added to the complexities of designing impact evaluations that will result in valid findings. Furthermore, the lack of site representativeness, appropriate comparison groups, and problems in data collection and analysis may compromise the reliability and validity of some of these evaluations. We recognize that not all evaluation issues that can compromise results are easily resolvable, including issues involving comparison groups and data collection. To the extent that methodological design and implementation issues can be overcome, however, the validity of the evaluation results will be enhanced. NIJ spends millions of dollars annually to evaluate OJP grant programs. More up-front attention to the methodological rigor of these evaluations will increase the likelihood that they will produce meaningful results for policymakers. Unfortunately, the problematic evaluation grants that we reviewed are too far along to be radically changed. However, two of the VAWO evaluation grants are still in the formative stage; more NIJ attention to their methodologies now can better ensure useable results.

Recommendations for Executive Action

We recommend that the Attorney General instruct the Director of NIJ to assess the two VAWO impact evaluations that are in the formative stage to address any potential methodological design and implementation problems and, on the basis of that assessment, initiate any needed interventions to help ensure that the evaluations produce definitive results. We further recommend that the Attorney General instruct the Director of NIJ to assess its evaluation process with the purpose of developing approaches to ensure that future impact evaluation studies are effectively designed and implemented so as to produce definitive results.

Agency Comments and Our Evaluation

We provided a copy of a draft of this report to the Attorney General for review and comment. In a February 13, 2002, letter, the Assistant Attorney General commented on the draft. Her comments are summarized below and presented in their entirety in appendix III.

The Assistant Attorney General agreed with the substance of our recommendations and said that NIJ has begun, or plans to take steps, to address them. Although it is still too early to tell whether NIJ’s actions will be effective in preventing or resolving the problems we identified, they appear to be steps in the right direction. With regard to our first recommendation—that NIJ assess the two VAWO impact evaluations in the formative stage to address any potential design and implementation problems and initiate any needed intervention to help ensure definitive results—the Assistant Attorney General noted that NIJ has begun work to ensure that these projects will provide the most useful information possible. She said that for the Crimes Against Women on Campus Program evaluation, NIJ is considering whether it will be possible to conduct an impact evaluation and, if so, how it can enhance its methodological rigor with the resources available. For the Civil Legal Assistance Program evaluation, the Assistant Attorney General said that NIJ is working with the grantee to review site selection procedures for the second phase of the study to enhance the representativeness of sites. The Assistant Attorney General was silent about any additional steps that NIJ would take during the later stages of the Civil Legal Assistance Program process evaluation to ensure the methodological rigor of the impact phase of the study. However, it seems likely that as the process evaluation phase of the study continues, NIJ may be able to take advantage of additional opportunities to address any potential design and implementation problems.

With regard to our second recommendation—that NIJ assess its evaluation process to develop approaches to ensure that future evaluation studies are effectively designed and implemented to produce definitive results—the Assistant Attorney General stated that OJP has made program evaluation, including impact evaluations of federally funded programs, a high priority. The Assistant Attorney General said that NIJ has already launched an examination of NIJ’s evaluation process. She also noted that, as part of its reorganization, OJP plans to measurably strengthen NIJ’s capacity to manage impact evaluations with the goal of making them more useful for Congress and others. She noted as an example that OJP and NIJ are building measurement requirements into grants at the outset, requiring potential grantees to collect baseline data and track the follow-up data through the life of the grant. We have not examined OJP’s plans for reorganizing, nor do we have a basis for determining whether OJP’s plans regarding NIJ would strengthen NIJ’s capacity to manage evaluations. However, we believe that NIJ and its key stakeholders, such as Congress and the research community, would be well served if NIJ were to assess what additional actions it could take to strengthen its management of impact evaluations regardless of any reorganization plans.

In her letter, the Assistant Attorney General pointed out that the report accurately describes many of the challenges facing evaluators when conducting research in the complex environment of criminal justice programs and interventions. However, she stated that the report could have gone further in acknowledging these challenges. The Assistant Attorney General also stated that the report contrasts the Byrne evaluation with the three VAWO evaluations and obscures important programmatic differences that affect an evaluator’s ability to achieve “GAO’s conditions for methodological rigor.” She pointed out that the Byrne CAR Program was intended to test a research hypothesis and that the evaluation was designed accordingly, i.e., the availability of baseline data were ensured; randomization of effects were stipulated as a precondition of participation; and outcome measures were determined in advance on the basis of the theories to be tested. She further stated that, in contrast, all of the VAWO programs were (1) highly flexible funding streams, in keeping with the intention of Congress, that resulted in substantial heterogeneity at the local level and (2) well into implementation before the evaluation started. The Assistant Attorney General went on to say that it is OJP’s belief that evaluations under less than optimal conditions can provide valuable information about the likely impact of a program, even though the conditions for methodological strategies and overall rigor of the CAR evaluation were not available.

We recognize that there are substantive differences in the intent, structure, and design of the various discretionary grant programs managed by OJP and its bureaus and offices. And, as stated numerous times in our report, we acknowledge not only that impact evaluation can be an inherently difficult and challenging task but also that measuring the impact of these specific Byrne and VAWO programs can be arduous, given that they are operating in an ever changing, complex environment. We agree that not all evaluation issues that can compromise results are easily resolvable, but we firmly believe that, with more up-front attention to design and implementation issues, there is a greater likelihood that NIJ evaluations will provide meaningful results for policymakers. Absent this up-front attention, questions arise as to whether NIJ is (1) positioned to provide the definitive results expected from an impact evaluation and (2) making sound investments given the millions of dollars spent on these evaluations.

The Assistant Attorney General also commented that although our report discussed “generally accepted social science standards,” it did not specify the document that articulates these standards or describe our elements of rigor. As a result, the Assistant Attorney General said, OJP had to infer that six elements had to be met to achieve what “GAO believes” is necessary to “have a rigorous impact evaluation.” Specifically, she said that she would infer that, for an impact evaluation to be rigorous would require (1) selection of homogenous programs, (2) random or stratified site sampling procedures (or selection of all sites), (3) use of comparison groups, (4) high response rates, (5) available and relevant automated data systems that will furnish complete and accurate data to evaluators in a timely manner, and (6) funding sufficient to accomplish all of the above. Furthermore, the Attorney General said that it is rare to encounter all of these conditions or be in a position to engineer all of these conditions simultaneously; and when all of these conditions are present, the evaluation would be rigorous. She also stated that it is possible to glean useful, if not conclusive, evidence of the impact of a program from an evaluation that does not rise to the standard recommended by GAO because of the unavoidable absence of “one or more elements.”

We agree that our report did not specify particular documents that articulate generally accepted social science standards. However, the standards that we applied are well defined in scientific literature. All assessments of the impact evaluations we reviewed were completed by social scientists with extensive experience in evaluation research. Throughout our report, we explain our rationale and the criteria we used in measuring the methodological rigor of NIJ’s impact evaluations. Furthermore, our report does not suggest that a particular standard or set of standards is necessary to achieve rigor, nor does it suggest that other types of evaluations, such as comprehensive process evaluations, are any less useful in providing information on how a program is operating. In this context, it is important to point out that the scope of our work covered impact evaluations of Byrne and VAWO discretionary grant programs— those designed to assess the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program.

We differ with the Assistant Attorney General with respect to the six elements cited as necessary elements for conducting an impact evaluation. Contrary to the Assistant Attorney General’s assertion, our report did not state that a single homogeneous program is a necessary element for conducting a rigorous impact evaluation. Rather, we pointed out that heterogeneity or program variation is a challenge that adds to the complexity of designing an evaluation. In addition, contrary to her assertion, the report did not assert that random sampling or stratification was a necessary element for conducting a rigorous evaluation; instead it stated that when random sampling is not feasible, other purposeful sampling methods can be used. With regard to comparison groups, the Assistant Attorney General’s letter asserted that GAO standards required using groups that do not receive program benefits as a basis of comparison with those that do receive such benefits. In fact, we believe that the validity of evaluation results can be enhanced through establishing and tracking comparison groups. If other ways exist to effectively isolate the impacts of a program, comparison groups may not be needed. However, we saw no evidence that other methods were effectively used in the VAWO impact evaluations we assessed.

The Assistant Attorney General also suggested that we used a 75 percent or greater response rate for evaluation surveys as a standard of rigor. In fact, we did not—we simply pointed out that NIJ documents showed a 43 percent response rate on one of the STOP Grant Program evaluation surveys. This is below OMB’s threshold response rate level—the level below which OMB particularly believes nonresponse bias and statistical problems could affect surveys. Given OMB guidance, serious questions could be raised about program conclusions drawn from the results of a survey with a 43 percent response rate. In addition, the Assistant Attorney General suggested that, by GAO standards, she would have to require state, local, or tribal government officials to furnish complete and accurate data in a timely manner. In fact, our report only points out that NIJ reported that evaluators were hindered in carrying out evaluations because of the lack of automated data systems or because data were missing, lost, or unavailable—again, challenges to achieving methodologically rigorous evaluations that could produce meaningful and definitive results.

Finally, the Assistant Attorney General’s letter commented that one of the elements needed to meet “all of GAO’s conditions” of methodological rigor is sufficient funding. She stated that more rigorous impact evaluations cost more than those that provide less scientific findings, and she said that OJP is examining the issue of how to finance effective impact evaluations. We did not assess whether funding is sufficient to conduct impact evaluations, but we recognize that designing effective and rigorous impact evaluations can be expensive—a condition that could affect the number of impact evaluations conducted. However, we continue to believe that with more up-front attention to the rigor of ongoing and future evaluations, NIJ can increase the likelihood of conducting impact evaluations that produce meaningful and definitive results.

In addition to the above comments, the Assistant Attorney General made a number of suggestions related to topics in this report. We have included the Assistant Attorney General’s suggestions in the report, where appropriate. Also, the Assistant Attorney General provided other comments in response to which we did not make changes. See appendix III for a more detailed discussion of the Assistant Attorney General’s comments.

We are sending copies of this report to the Chairman and the Ranking Minority Member of the Senate Judiciary Committee; to the Chairman and Ranking Minority Member of the House Judiciary Committee; to the Chairman and Ranking Minority Member of the Subcommittee on Crime, House Committee on the Judiciary; to the Chairman and the Ranking Minority Member of the House Committee on Education and the Workforce; to the Attorney General; to the OJP Assistant Attorney General; to the NIJ Director; to the BJA Director; to the VAWO Director; and to the Director, Office of Management and Budget. We will also make copies available to others on request.

If you or your staff have any questions about this report, please contact John F. Mortin or me at (202) 512-8777. Key contributors to this report are acknowledged in appendix IV.

Appendix I: Summaries of the Impact Evaluations of Byrne and VAWO Programs

Evaluation findings Assessment of evaluation This evaluation has several limitations. (1) The choice of the 10 impact sites is skewed toward the National Evaluation of the Rural Domestic Violence and Child Victimization Grant Program COSMOS Corporation The Violence Against Women Office’s (VAWO) Rural Domestic Violence Program, begun in fiscal year 1996, has funded 92 grantees through September 2001. The primary purpose of the program is to enhance the safety of victims of domestic abuse, dating violence, and child abuse. The program supports projects that implement, expand, and establish cooperative efforts between law enforcement officers, prosecutors, victim advocacy groups, and others in investigating and prosecuting incidents of domestic violence, dating violence, and child abuse; provide treatment, counseling, and assistance to victims; and work with the community to develop educational and prevention strategies directed toward these issues. The impact evaluation began in July 2000, with a final report expected no earlier than April 2002. Initially, 10 grantees were selected to participate in the impact evaluation; 9 remain in the evaluation. Two criteria were used in the selection of grant participants: the “feasibility” of earlier site-visited grantees to conduct an outcome evaluation and VAWO recommendations based on knowledge of grantee program activities. Logic models were developed, as part of the case study approach, to show the logical or plausible links between a grantee’s activities and the desired outcomes. The specified outcome data were to be collected from multiple sources, using a variety of methodologies during 2- to- 3-day site visits (e.g., multiyear criminal justice, medical, and shelter statistics were to be collected from archival records; community stakeholders were to be interviewed; and grantee and victim service agency staff were to participate in focus groups). At the time of our review, this evaluation was funded at $719,949. The National Institute of Justice (NIJ) could not separate the cost of the impact evaluation from the cost of the process evaluation. Too early to assess. technically developed evaluation sites and is not representative of either all Rural Domestic Violence Program Grantees, particular types of projects, or delivery styles. (2) The lack of comparison groups will make it difficult to exclude the effect of external factors, such as victim safety and improved access to services, on perceived change. (3) Several so-called short-term outcome variables are in fact process variables (e.g., number of police officers who complete a training program on domestic violence, identification of barriers to meeting the needs of women and children). (4) It is not clear how interview and focus group participants are to be selected, (5) Statistical procedures to be used in the analyses have not been sufficiently identified. The NIJ peer review committee had concerns about whether the evaluation could demonstrate that the program was working. NIJ funded the application as a cooperative agreement because a substantial amount of agency involvement was deemed necessary to meet the objectives of the evaluation.

Evaluation findings Assessment of evaluation This evaluation has several limitations: the absence of a representative sampling frame for site National Evaluation of the Arrest Policies Program Institute for Law and Justice (ILJ) The purpose of VAWO’s Arrest Policies Program is to encourage states, local governments, and Indian tribal governments to treat domestic violence as a serious violation of criminal law. The program received a 3-year authorization (fiscal years 1996 through 1998) at approximately $120 million to fund grantees under six purpose areas: implementing mandatory arrest or proarrest programs and policies in police departments, tracking domestic violence cases, centralizing and coordinating police domestic violence operations, coordinating computer tracking systems, strengthening legal advocacy services, and educating judges and others about how to handle domestic violence cases. Grantees have flexibility to work in several of these areas. At the time the NIJ evaluation grant was awarded, 130 program grantees had been funded; the program has since expanded to 190 program grantees. The impact evaluation began in August 1998, with a draft final report due in March 2002. Six grantees were chosen to participate in the impact evaluation. Each of the six sites was selected on the basis of program “stability,” not program representativeness. Within sites, both quantitative and qualitative data were to be collected and analyzed to enable better understanding of the impact of the Arrest Program on offender accountability and victim well being. This process entailed reviewing data on the criminal justice system’s response to domestic violence; tracking a random sample of 100 offender cases, except in rural areas, to determine changes in offender accountability; conducting content analyses of police incident reports to assess change in police practices and documentation; and interviewing victims or survivors at each site to obtain their perceptions of the criminal justice system’s response to domestic violence and its impact on their well-being. ILJ had planned cross-site comparisons and the collection of extensive historical and interview data to test whether competing factors could be responsible for changes in arrest statistics. At the time of our review, this evaluation was funded at $1,130,574. NIJ could not separate the cost of the impact evaluation from the cost of the process evaluation. Too early to assess. selection, the lack of comparison groups, the inability to conduct cross-site comparisons, and the lack of a sufficient number of victims in some sites to provide a perspective on the changes taking place in domestic violence criminal justice response patterns and victim well-being. In addition, there was difficulty collecting pre-grant baseline data, and the quality of the data was oftentimes inadequate, limiting the ability to measure change over time. Further, automated data systems were not available in all work areas, and data were missing, lost, or unavailable. An NIJ peer review committee also expressed some concerns about the grantee’s methodological design.

Evaluation findings Assessment of evaluation Methodological design and implementation issues may cause difficulties in attributing program impact. A Impact Evaluation of STOP Grant Programs for Reducing Violence Against Indian Women The University of Arizona VAWO’s STOP (Services, Training, Officers, and Prosecutors) Grant Programs for Reducing Violence Against Indian Women Discretionary Grant Program was established under Title IV of the Violent Crime Control and Law Enforcement Act of 1994. The program’s principal purpose is to reduce violent crimes against Indian women. The program, which began in fiscal year 1995 with 14 grantees, encourages tribal governments to develop and implement culture-specific strategies for responding to violent crimes against Indian women and providing appropriate services for those who are victims of domestic abuse, sexual assault, and stalking. In this effort, the program provided funding for the services and training, and required the joint coordination, of nongovernmental service providers, law enforcement officers, and prosecutors hence the name, the STOP Grant Programs for Reducing Violence Against Indian Women. The University of Arizona evaluation began in October 1996 with an expected final report due in March 2002. The basic analytical framework of this impact evaluation involves the comparison of quantitative and qualitative pre-grant case study histories of participating tribal programs with changes taking place during the grant period. Various data collection methodologies have been adopted (at least in part, to be sensitive to the diverse Indian cultures): 30-minute telephone interviews, mail surveys, and face-to-face 2- to 3 day site visits. At the time of our review, this evaluation was funded at $468,552. NIJ could not separate the cost of the impact evaluation from the cost of the process evaluation. Too early to assess. number of methodological aspects of the study remain unclear: the site selection process for “in-depth case study evaluations;” the methodological procedures for conducting the longitudinal evaluation; the measurement, validity, and reliability of the outcome variables; the procedures for assessing impact; and the statistical tests to be used for determining significant change. Comparison groups are not included in the methodological design. In addition, only 43 percent of the grantees returned the mail survey, only 25 percent could provide the required homicide and hospitalization rates; and only 26 victims of domestic violence and assault could be interviewed (generally too few to measure statistical change). Generalization of evaluation results to the entire STOP Grant Programs for Reducing Violence Against Indian Women will be difficult, given these problems.

Longitudinal Impact Evaluation of the Strategic Intervention for High Risk Youth (a.k.a. The Children at Risk Program) The Urban Institute The Children at Risk (CAR) Program, a comprehensive drug and delinquency prevention initiative funded by the Bureau of Justice Assistance (BJA), the Office of Juvenile Justice and Delinquency Prevention (OJJDP), the Center on Addiction and Substance Abuse, and four private foundations, was established to serve as an experimental demonstration program from 1992 to 1996 in five grantee cities. Low-income youths (11 to 13 years old) and their families, who lived in severely distressed neighborhoods at high-risk for drugs and crime, were targeted for intervention. Eight core service components were identified: case management, family services, education services, mentoring, after- school and summer activities, monetary and nonmonetary incentives, community policing, and criminal justice and juvenile intervention (through supervision and community service opportunities). The goals of the program were to reduce drug use among targeted families and improve the safety and overall quality of life in the community. The evaluation actually began in 1992, and the final report was submitted in May 1998. The study used both experimental and quasi-experimental evaluation designs. A total of 671 youths in target neighborhood schools were randomly assigned to either a treatment group (which received CAR services and the benefit of a safer neighborhood) or to a control group (which received only a safer neighborhood). Comparison groups (n=203 youths) were selected from similar high-risk neighborhoods by means of census tract data; comparison groups did not have access to the CAR Program. Interviews were conducted with youth participants at program entry (baseline), program completion (2 years later), and 1-year after program completion. A parent or caregiver was interviewed at program entry and completion. Records from schools, police, and courts were collected annually for each youth in the sample as a means of obtaining more objective data. The total evaluation funding was $1,034,732. Youths participating in CAR were significantly less likely than youths in the control group to have used gateway and serious drugs, to have sold drugs, or to have committed violent crimes in the year after the program ended. CAR youths were more likely than youths in the control and comparison groups to report attending drug and alcohol abuse programs. CAR youths received more positive peer support than controls, associated less frequently with delinquent peers, and were pressured less often by peers to behave in antisocial ways. CAR households used more services than control group households, but the majority of CAR households did not indicate using most of the core services available.

Assessment of evaluation CAR is a methodologically rigorous evaluation in both its design and implementation. The evaluation findings demonstrate the value of the program as a crime and drug prevention initiative.

Appendix II: NIJ’s Guidelines for Disseminating Discretionary Grant Program Evaluation Results

NIJ has the primary role of disseminating Byrne and VAWO discretionary grant program evaluation results of evaluations managed by NIJ, according to NIJ, BJA, and VAWO officials, because NIJ is responsible for conducting these types of evaluations. NIJ is authorized to share the results of its research with federal, state, and local governments. NIJ also disseminates information on methodology designs.

NIJ’s Dissemination Practices

NIJ’s practices for disseminating program evaluation results are specified in its guidelines. According to the guidelines, once NIJ receives a final evaluation report from the evaluators and the results of peer reviews have been incorporated, NIJ grant managers are to carefully review the final product and, with their supervisor, recommend to the NIJ Director which program results to disseminate and the methods for dissemination. Before making a recommendation, grant managers and their supervisors are to consider various criteria, including policy implications, the nature of the findings and research methodology, the target audience and their needs, and the cost of various forms of dissemination. Upon receiving the recommendation, the Director of NIJ is to make final decisions about which program evaluation results to disseminate. NIJ’s Director of Planning and Management said that NIJ disseminates program evaluation results that are peer reviewed, are deemed successful, and add value to the field.

Once the decision has been made to disseminate program evaluation results and methodologies with researchers and practitioners, NIJ can choose from a variety of publications, including its Research in Brief; NIJ Journal–At a Glance: Recent Research Findings; Research Review; NIJ Journal–Feature Article; and Research Report. In addition, NIJ provides research results on its Internet site and at conferences. For example, using its Research in Brief publication, NIJ disseminated impact evaluation results on the Byrne Children at Risk (CAR) program to 7,995 practitioners and researchers, including state and local government and law enforcement officials; social welfare and juvenile justice professionals; and criminal justice researchers. In addition, using the same format, NIJ stated that it distributed the results of its process evaluation of the Byrne Comprehensive Communities Program (CCP) to 41,374 various constituents, including local and state criminal and juvenile justice agency administrators, mayors and city managers, leaders of crime prevention organizations, and criminal justice researchers. NIJ and other OJP offices and bureaus also disseminated evaluation results during NIJ’s annual conference on criminal justice research and evaluation. The July 2001 conference was attended by 847 public and nonpublic officials, including criminal justice researchers and evaluation specialists from academic institutions, associations, private organizations, and government agencies; federal, state, and local law enforcement, court, and corrections officials; and officials representing various social service, public housing, school, and community organizations.

In addition to NIJ’s own dissemination activities, NIJ’s Director of Planning and Management said that it allows and encourages its evaluation grantees to publish their results of NIJ-funded research via nongovernmental channels, such as in journals and through presentations at professional conferences. Although NIJ requires its grantees to provide advance notice if they are publishing their evaluation results, it does not have control over its grantees’ ability to publish these results. NIJ does, however, require a Justice disclaimer that the “findings and conclusions reported are those of the authors and do not necessarily reflect the official position or policies of the U.S. Department of Justice.” For example, although NIJ has not yet disseminated the program evaluation results of the three ongoing VAWO impact evaluations that we reviewed, one of the evaluation grantees has already issued, on its own Internet site, 9 of 20 process evaluation reports on the Arrests Policies evaluation grant. The process evaluations were a component of the NIJ grantee’s impact evaluation of the Arrest Policies Program. Because the evaluations were not completed, NIJ required that the grantee’s publication of the process evaluations be identified as a draft report pending final NIJ review.

Dissemination of NIJ’s Byrne Discretionary Grants: Comprehensive Communities Program and Children at Risk Program

As discussed earlier, NIJ publishes the results of its evaluations in several different publications. For example, NIJ used the Research in Brief format to disseminate evaluation results for two of the five Byrne discretionary grant programs Comprehensive Communities Program (CCP) and Children at Risk Program (CAR) that were evaluated during fiscal years 1995 through 2001. Both publications summarize information including each program’s evaluation results, methodologies used to conduct the evaluations, information about the implementation of the programs themselves, and services that the programs provided. CCP’s evaluation results were based on a process evaluation. Although a process evaluation does not assess the results of the program being evaluated, it can provide useful information that explains the extent to which a program is operating as intended. The NIJ Research in Brief on the Byrne CAR Discretionary Grant Program provides a summary of issues and findings regarding the impact evaluation. That summary included findings reported one year after the end of the program, in addition to a summary of the methodology used to conduct the evaluation, the outcomes, the lessons learned, and a major finding from the evaluation.

Appendix III: Comments from the Department of Justice

Following are GAO’s comments on the Department of Justice’s February 13, 2002, letter.

GAO Comments

1. We have amended the text to further clarify that BJA administers the Byrne program, just as its counterpart, VAWO, administers its programs (see page 4). However it is important to point out that regardless of the issues raised by OJP, the focus of our work was on the methodological rigor of the evaluations we reviewed, not the purpose and structure of the programs being evaluated. As discussed in our Scope and Methodology section, our work focused on program evaluation activities associated with Byrne and VAWO discretionary grant programs generally and the methodological rigor of impact evaluation studies associated with those programs in particular. To make our assessment, we relied on NIJ officials to identify which of the program evaluations of Byrne and VAWO grant programs were, in fact, impact evaluation studies. We recognize that there are substantial differences among myriad OJP programs that can make the design and implementation of impact evaluations arduous. But, that does not change the fact that impact evaluations, regardless of differences in programs, can benefit from stronger up-front attention to better ensure that they provide meaningful and definitive results. 2. We disagree with OJP’s assessment of our report’s treatment of program variation. As discussed earlier, the scope of our review assessed impact evaluation activities associated with Byrne and VAWO discretionary grant programs, not the programs themselves. We examined whether the evaluations that NIJ staff designated as impact evaluations were designed and implemented with methodological rigor. In our report we observe that variations in projects funded through VAWO programs complicate the design and implementation of impact evaluations.

According to the Assistant Attorney General, this variation in projects is consistent with the intent of the programs’ authorizing legislation. We recognize that the authorizing legislation provides VAWO the flexibility in designing these programs. In fact, we point out that although such flexibility may make sense from a program perspective, project variation makes it much more difficult to design and implement a definitive impact evaluation. This poses sizable methodological problems because an aggregate analysis, such as one that might be constructed for an impact evaluation, could mask the differences in effectiveness among individual projects and therefore not result in information about which configurations of projects work and which do not. 3. We have amended the Results in Brief to clarify that peer reviews evaluated proposals. However, it is important to note that while the peer review committees may have found the two VAWO grant applications to be the most superior, this does not necessarily imply that the impact evaluations resulting from these applications were well designed and implemented. As discussed in our report, the peer review panel for each of the evaluations expressed concerns about the proposals that were submitted, including issues related to site selection and the need for additional impact measures and control variables. Our review of the documents NIJ made available to us, including evaluators’ responses to peer review comments, led to questions about whether the evaluators’ proposed methodological designs were sufficient to allow the evaluation results to be generalized and to determine whether the program was working. 4. We have amended the Background section of the report to add this information (see page 6). 5. As discussed in OJP’s comments, we discussed external factors that could account for changes that the Rural Program evaluation observed in victims’ lives and the criminal justice system. We did so not to critique or endorse activities that the program was or was not funding, but to demonstrate that external factors may influence evaluation findings. To the extent that such factors are external, the Rural Program evaluation methodology should account for their existence and attempt to establish controls to minimize their affect on results (see page 14). We were not intending to imply that alcohol is a cause for domestic violence, as suggested by the Assistant Attorney General, but we agree that it could be an exacerbating factor that contributes to violence against women. 6. As discussed earlier, we recognize that there are substantive differences in the intent, structure, and design of the various discretionary grant programs managed by OJP and its bureaus and offices. Also, as stated numerous times in our report, we acknowledge not only that impact evaluation can be an inherently difficult and challenging task but also that measuring the impact of these specific Byrne and VAWO programs can be arduous given that they are operating in an ever changing, complex environment. We agree that not all evaluation issues that can compromise results are easily resolvable, but we firmly believe that with more up- front attention to design and implementation issues, there is a greater likelihood that NIJ impact evaluations will provide meaningful results for policymakers.

Regarding the representativeness of sites, NIJ documents that were provided during our review indicated that sites selected during the Rural Program evaluation were selected on the basis of feasibility, as discussed in our report— specifically, whether the site would be among those participants equipped to conduct an evaluation. In its comments, OJP stated that the 6 sites selected for the impact evaluation were chosen to maximize geographical and purpose area diversity while focusing on sites with high program priority. OJP did not provide any additional information that would further indicate that the sites were selected on a representative basis. OJP did, however, point out that the report does not address how immensely expensive the Arrest evaluation would have become if it had included all 130 sites. We did not address specific evaluation site costs because we do not believe that there is a requisite number of sites needed for any impact evaluation to be considered methodologically rigorous. Regarding OJP’s comment about the flexibility given to grantees in implementing VAWO grants, our report points out that project variation complicates evaluation design and implementation. Although flexibility may make sense from a program perspective, it makes it difficult to generalize about the impact of the entire program. 7. We used the drug court example to illustrate, based on our past work, how comparison groups can be used in evaluation to isolate and minimize external factors that could influence the study results. We did not, nor would we, suggest that any particular unit of analysis is appropriate for VAWO evaluations since the appropriate unit of analysis is dependent upon the specific circumstances of the evaluation. We were only indicating that since comparison groups were not utilized in the studies, the evaluators were not positioned to demonstrate that change took place as a result of the program. 8. We do not dispute that VAWO grant programs may provide valuable outputs over the short term. However, as we have stated previously, the focus of our review was on the methodological rigor of impact evaluations--those evaluations that are designed to assess the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. Given the methodological issues we found, it is unclear whether NIJ will be able to discern long-term effects due to the program. 9. As stated in our report, we acknowledge not only that impact evaluation can be an inherently difficult and challenging task, but that measuring the impact of Byrne and VAWO programs can be arduous given the fact that they are operating in an ever changing, complex environment. We agree that not all evaluation issues that can compromise results are easily resolvable, but we firmly believe that, with more up-front attention to design and implementation issues, there is a greater likelihood that NIJ evaluations will provide meaningful results for policymakers. As we said before, absent this up-front attention, questions arise as to whether NIJ is (1) positioned to provide the definitive results expected from an impact evaluation and (2) making sound investments given the millions of dollars spent on these evaluations. If NIJ believes that the circumstances of a program are such that it cannot be evaluated successfully (in relation to impact) they should not proceed with an impact evaluation. 10. We have amended the footnote to state that from fiscal year 1995 through fiscal year 1999, this program was administered by VAWO. As of fiscal year 2000, responsibility for the program was shifted to OJP’s Corrections Program Office (see page 5). 11. In regard to the number of grants, we have amended the text to reflect that the information NIJ provided during our review is the number of grantees, not the number of grants (see pages 25 and 26). We have also amended our report to reflect some of the information provided in VAWO’s description of the Rural Domestic Violence Program to further capture the essence of the program (see page 25). 12. We disagree. We believe that separating the cost of the impact and process evaluations is more than a matter of bookkeeping. Even though the work done during the process phase of an evaluation may have implications for the impact evaluation phase of an evaluation, it would seem that, given the complexity of impact evaluations, OJP and NIJ would want to have in place appropriate controls to provide reasonable assurance that the evaluations are being effectively and efficiently carried out at each phase of the evaluation. Tracking the cost of these evaluation components would also help reduce the risk that OJP’s, NIJ’s, and, ultimately, the taxpayer’s investment in these impact evaluations is not wasted. 13. As discussed earlier, we recognize that there are substantive differences in the intent, structure, and design of the various discretionary grant programs managed by OJP and its bureaus and offices, including those managed by VAWO. Our report focuses on the rigor of impact evaluations of grant programs administered by VAWO and not on the program’s implementing legislations. Although flexibility may make sense from a program perspective, it makes it difficult to develop a well designed and methodologically rigorous evaluation that produces generalizeable results about the impact of the entire program. 14. Our report does not suggest that other types of evaluations, such as comprehensive process evaluations, are any less useful in providing information about how well a program is operating. The scope of our review covered impact evaluations of Byrne and VAWO discretionary grant programs—those designed to assess the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program.

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to the above, Wendy C. Simkalo, Jared A. Hermalin, Chan My J. Battcher, Judy K. Pagano, Grace A. Coleman, and Ann H. Finley made key contributions to this report.