Justice Impact Evaluations: One Byrne Evaluation Was Rigorous; All Reviewed Violence Against Women Office Evaluations Were Problematic
Published: 2002-03-05T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: Discretionary grants awarded under the Bureau of Justice Assistance's (BJA) Byrne Program help state and local governments make communities safe and improve criminal justice. Discretionary grants awarded under BJA's Violence Against Women Office (VAWO) programs are aimed at improving criminal justice system responses to domestic violence, sexual assault, and stalking. The National Institute of Justice (NIJ) awarded $6 million for five Byrne Program and five VAWO discretionary grant program evaluations between 1995 and 2001. Of the 10 programs evaluated, all five VAWO evaluations were designed to be both process and impact evaluations of the VAWO programs. Only one of the five Byrne evaluations was designed as an impact evaluation and the other four evaluations were process evaluations. GAO's in-depth review of the four impact evaluations since fiscal year 1995 showed that only one of these--the evaluation of the Byrne Children at Risk Program--was methodologically sound. The other three evaluations, all of which examined VAWO programs, had methodological problems.

section_title: Letter
paragraphs: This report responds to your request that we review selected aspects of the U.S. Department of Justice’s (Justice) Office of Justice Programs’ (OJP) program evaluations of discretionary grants awarded by OJP’s Bureau of Justice Assistance’s (BJA) Byrne Program (Byrne) and the Violence Against Women Office (VAWO). Between fiscal years 1997 and 2000, Byrne and VAWO discretionary grant awards grew, in constant fiscal year 2000 dollars, about 85 percent—from about $105 million to approximately $194 million. These funds were awarded directly to various organizations, such as state and local governments, either on a competitive basis or pursuant to legislation allocating funds through congressional earmarks or direction. Discretionary grants awarded under the Byrne program were designed to help state and local governments make communities safe and improve criminal justice.  Discretionary grants awarded under VAWO programs are aimed at improving criminal justice system responses to domestic violence, sexual assault, and stalking. Questions have been raised, however, regarding what these and other Justice grant programs have accomplished.
To address your request, we are reporting on (1) the number, type, status of completion, and award amount of Byrne and VAWO discretionary grant program evaluations during fiscal years 1995 through 2001 and (2) the methodological rigor of the impact evaluation studies of Byrne and VAWO discretionary grant programs during fiscal years 1995 through 2001. In addition, information that you requested on OJP’s approaches to disseminating evaluation results about Byrne and VAWO discretionary  grant programs is presented in appendix II. Our review covered  discretionary grant program evaluations managed by Justice’s National  Institute of Justice (NIJ). Program evaluations are systematic studies that  are conducted periodically or ad hoc to assess how well a program is  working. These studies can include impact evaluations—designed to  assess the net effect of a program by comparing program outcomes with  an estimate of what would have happened in the absence of the program—  and process evaluations—designed to assess the extent to which a  program is operating as intended. NIJ is OJP’s principal research and  development agency and is responsible for evaluating Byrne and VAWO  programs. For Byrne program evaluations, NIJ awarded funding to  evaluators using its own funds and funds made available through BJA. For  VAWO program evaluations, NIJ awarded funding to evaluators using  funds made available exclusively through VAWO.
subsections: section_title: Background
paragraphs: The Justice Assistance Act of 1984 (P.L. 98-473) created OJP to provide  federal leadership in developing the nation’s capacity to prevent and  control crime, administer justice, and assist crime victims. OJP carries out  its responsibilities by providing grants to various organizations, including  state and local governments, Indian tribal governments, nonprofit  organizations, universities, and private foundations. OJP comprises five  bureaus, including BJA, and seven program offices, including VAWO. In  fulfilling its mission, BJA provides grants for programs and for training  and technical assistance to combat violent and drug-related crime and  help improve the criminal justice system. VAWO administers grants to help  prevent and stop violence against women, including domestic violence,  sexual assault, and stalking. During fiscal years 1995 through 2001, BJA  and VAWO awarded about $943 million to fund 700 Byrne and 1,264 VAWO  discretionary grants.
One of BJA’s major grant programs is the Byrne Program. BJA  administers the Byrne program, just as its counterpart, VAWO, administers  its programs. Under the Byrne discretionary grants program, BJA provides  federal financial assistance to grantees for educational and training  programs for criminal justice personnel; for technical assistance to state  and local units of government; and for projects that are replicable in more  than one jurisdiction nationwide. During fiscal years 1995 through 2001,  Byrne discretionary grant programs received appropriations of about $385  million.
VAWO was created in 1995 to carry out certain programs created under  the Violence Against Women Act of 1994. The Victims of Trafficking and  Violence Prevention Act of 2000 reauthorized most of the existing VAWO  programs and added new programs. VAWO programs seek to improve  criminal justice system responses to domestic violence, sexual assault,  and stalking by providing support for law enforcement, prosecution,  courts, and victim advocacy programs across the country. During fiscal  years 1995 through 2001, VAWO’s five discretionary grant programs that  were subject to program evaluation were (1) STOP (Services, Training,  Officers, and Prosecutors) Violence Against Indian Women Discretionary  Grants, (2) Grants to Encourage Arrest Policies, (3) Rural Domestic  Violence and Child Victimization Enforcement Grants, (4) Domestic  Violence Victims’ Civil Legal Assistance Grants, and (5) Grants to Combat  Violent Crimes Against Women on Campuses. During fiscal years 1995  through 2001, about $505 million was appropriated to these discretionary  grant programs.
As already mentioned, NIJ is the principal research and development  agency within OJP, and its duties include developing, conducting,  directing, and supervising Byrne and VAWO discretionary grant program  evaluations. Under 42 U.S.C. 3766, NIJ is required to “conduct a reasonable  number of comprehensive evaluations” of the Byrne discretionary grant  program. In selecting programs for review under section 3766, NIJ is to  consider new and innovative approaches, program costs, potential for  replication in other areas, and the extent of public awareness and  community involvement. According to NIJ officials, the implementation of  various types of evaluations, including process and impact evaluations,  fulfills this legislative requirement. Although legislation creating VAWO  does not require evaluations of the VAWO discretionary grant programs,  Justice’s annual appropriations for VAWO during fiscal years 1998 through  2002 included monies for NIJ research and evaluations of violence against  women. In addition, Justice has promulgated regulations requiring that  NIJ conduct national evaluations of two of VAWO’s discretionary grant  programs. As with the Byrne discretionary programs, NIJ is not required  by statute or Justice regulation to conduct specific types of program  evaluations, such as impact or process evaluations.
The Director of NIJ is responsible for making the final decision on which  Byrne and VAWO discretionary grant programs to evaluate; this decision is  based on the work of NIJ staff in coordination with Byrne or VAWO  program officials. Once the decision has been made to evaluate a  particular program, NIJ issues a solicitation for proposals for grant  funding from potential evaluators. When applications or proposals are  received, an external peer review panel comprising members of the  research and relevant practitioner communities is convened. Peer review  panels identify the strengths, weaknesses, and potential methodologies to  be derived from competing proposals. When developing their consensus  reviews, peer review panels are to consider the quality and technical merit  of the proposal; the likelihood that grant objectives will be met; the  capabilities, demonstrated productivity, and experience of the evaluators;  and budget constraints. Each written consensus review is reviewed and  discussed with partnership agency representatives (e.g., staff from BJA or  VAWO). These internal staff reviews and discussions are led by NIJ’s  Director of the Office of Research and Evaluation who then presents the  peer review consensus reviews, along with agency and partner agency  input, to the NIJ Director for consideration and final grant award  decisions. The NIJ Director makes the final decision regarding which  application to fund.
section_title: Scope and Methodology
paragraphs: To meet our objectives, we conducted our work at OJP, BJA, VAWO, and  NIJ headquarters in Washington, D.C. We reviewed applicable laws and  regulations, guidelines, reports, and testimony associated with Byrne and  VAWO discretionary grant programs and evaluation activities. In addition,  we interviewed responsible OJP, NIJ, BJA, and VAWO officials regarding  program evaluations of discretionary grants. As agreed with your offices,  we focused on program evaluation activities associated with the Byrne  and VAWO discretionary grant programs. In particular, we focused on the  program evaluations of discretionary grants that were funded during fiscal  years 1995 through 2001.
To address our first objective, regarding the number, type, status of  completion, and award amount of Byrne and VAWO discretionary grant  program evaluations, we interviewed NIJ, BJA, and VAWO officials and  obtained information on Byrne and VAWO discretionary grant programs  and program evaluations. Because NIJ is responsible for carrying out  program evaluations of Byrne and VAWO discretionary grant programs,  we also obtained and analyzed NIJ data about specific Byrne and VAWO  discretionary grant program evaluations, including information on the  number of evaluations as well as the type, cost, source of funding, and  stages of implementation of each evaluation for fiscal years 1995 through  2001. We did not independently verify the accuracy or completeness of the  data that NIJ provided.
To address the second objective, regarding the methodological rigor of the  impact evaluation studies of Byrne and VAWO discretionary grant  programs during fiscal years 1995 through 2001, we initially identified the  impact evaluations from the universe of program evaluations specified by  NIJ. We excluded from our analysis any impact evaluations that were in  the formative stage of development—that is, the application had been  awarded but the methodological design was not yet fully developed. As a  result, we reviewed four program evaluations.
For the four impact evaluations that we reviewed, we asked NIJ to provide  any documentation relevant to the design and implementation of the  impact evaluation methodologies, such as the application solicitation, the  grantee’s initial and supplemental applications, progress notes, interim  reports, requested methodological changes, and any final reports that may  have become available during the data collection period. We also provided  NIJ with a list of methodological issues to be considered in our review and  requested them to submit any additional documentation that addressed  these issues. We used a data collection instrument to obtain information  systematically about each program being evaluated and about the features  of the evaluation methodology. We based our data collection and  assessments on generally accepted social science standards. We examined  such factors as whether evaluation data were collected before and after  program implementation; how program effects were isolated (i.e., the use  of nonprogram participant comparison groups or statistical controls); and  the appropriateness of sampling and outcome measures. Two of our senior  social scientists with training and experience in evaluation research and  methodology separately reviewed the evaluation documents and  developed their own assessments before meeting jointly to discuss the  findings and implications. This was done to promote a grant evaluation  review process that was both independent and objective.
To obtain information on the approaches that BJA, VAWO, and NIJ used to  disseminate program evaluation results, we requested and reviewed, if  available, relevant handbooks and guidelines on information  dissemination, including, for example, NIJ’s guidelines. We also reviewed  BJA, VAWO, and NIJ’s available print and electronic products as related to  their proven programs and evaluations, including two NIJ publications  about Byrne discretionary programs and their evaluation methodologies  and results.
We conducted our work between February 2001 and December 2001 in  accordance with generally accepted government auditing standards.
We requested comments from Justice on a draft of this report in January  2002. The comments are discussed near the end of this letter and are  reprinted as appendix III.
section_title: Number, Type, Status of Completion, and Award Amount of Byrne and VAWO Discretionary Grant Program Evaluations
paragraphs: During fiscal years 1995 through 2001, NIJ awarded about $6 million to  carry out five Byrne and five VAWO discretionary grant program  evaluations. NIJ awarded evaluation grants using mostly funds transferred  from BJA and VAWO. Specifically, of the approximately $1.9 million  awarded for one impact and four process evaluations of the Byrne  discretionary program, NIJ contributed about $299,000 (16 percent) and  BJA contributed about $1.6 million (84 percent). VAWO provided all of the  funding (about $4 million) to NIJ for all program evaluations of five VAWO  discretionary grant programs. According to NIJ, the five VAWO program  evaluations included both impact and process evaluations. Our review of  information provided by NIJ showed that 6 of the 10 program  evaluations—all 5 VAWO evaluations and 1 Byrne evaluation—included  impact evaluations. The remaining four Byrne evaluations were  exclusively process evaluations that measured the extent to which the  programs were working as intended.  As of December 2001, only one of  these evaluations, the impact evaluation of the Byrne CAR Program, had  been completed. The remaining evaluations were in various stages of  implementation. Table 1 lists each of the five Byrne program evaluations  and shows whether it was a process or an impact evaluation, its stage of  implementation, the amount awarded during fiscal years 1995 through  2001, and the total amount awarded since the evaluation was funded.
Table 2 lists each of the five VAWO program evaluations and shows that it  was both a process and an impact evaluation, its stage of implementation,  and the amount awarded during fiscal years 1995 through 2001, which is  the total amount awarded.
section_title: Methodological Problems Have Adversely Affected Three of Four Impact Evaluations
paragraphs: Our review showed that methodological problems have adversely affected  three of the four impact evaluations that have progressed beyond the  formative stage. All three VAWO evaluations that we reviewed  demonstrated a variety of methodological limitations, raising concerns as  to whether the evaluations will produce definitive results. The one Byrne  evaluation was well designed and used appropriate data collection and  analytic methods. We recognize that impact evaluations, such as the type  that NIJ is managing, can encounter difficult design and implementation  issues. In the three VAWO evaluations that we reviewed, program variation  across sites has added to the complexity of designing the evaluations. Sites  could not be shown to be representative of the programs or of particular  elements of these programs, thereby limiting the ability to generalize  results; the lack of comparison groups hinders the ability to minimize the  effects of factors external to the program. Furthermore, data collection  and analytical problems compromise the ability of evaluators to draw  appropriate conclusions from the results. In addition, peer review  committees found methodological problems in two of the three VAWO  evaluations that we considered.
The four program evaluations are multiyear, multisite impact evaluations.  Some program evaluations used a sample of grants, while others used the  entire universe of grants. For example, the Grants to Encourage Arrests  Policies Program used 6 of the original 130 grantee sites. In contrast, in the  Byrne Children at Risk impact evaluation, all five sites participated. As of  December 2001, NIJ had already received the impact findings from the  Byrne Children at Risk Program evaluation but had not received impact  findings from the VAWO discretionary grant program evaluations.
subsections: section_title: Impact Evaluations Are Difficult to Successfully Design and Implement
paragraphs: An impact evaluation is an inherently difficult task, since the objective is  to isolate the effects of a particular program or factor from all other  potential contributing programs or factors that could also effect change.  Given that the Byrne and VAWO programs are operating in an ever  changing, complex environment, measuring the impact of these specific  Byrne and VAWO programs can be arduous. For example, in the  evaluation of VAWO’s Rural Domestic Violence Program, the evaluator’s  responsibility is to demonstrate how the program affected the lives of  domestic violence victims and the criminal justice system. Several other  programs or factors besides the Rural Domestic Violence Program may be  accounting for all or part of the observed changes in victims’ lives and the  criminal justice system (e.g., a co-occurring program with similar  objectives, new legislation, a local economic downturn, an alcohol abuse  treatment program). Distinguishing the effects of the Rural Domestic  Violence Program requires use of a rigorous methodological design.
section_title: Project Variation within the VAWO Programs Complicates Evaluation Design and Implementation
paragraphs: All three VAWO programs permitted their grantees broad flexibility in the  development of their projects to match the needs of their local  communities.  According to the Assistant Attorney General, this variation  in projects is consistent with the intent of the programs’ authorizing  legislation. We recognize that the authorizing legislation provides VAWO  the flexibility in designing these programs.  Although this flexibility may  make sense from a program perspective, the resulting project variation  makes it more difficult to design and implement a definitive impact  evaluation of the program. Instead of assessing a single, homogeneous  program with multiple grantees, the evaluation must assess multiple  configurations of a program, thereby making it difficult to generalize about  the entire program. Although all of the grantees’ projects under each  program being evaluated are intended to achieve the same or similar goals,  an aggregate analysis could mask the differences in effectiveness among  individual projects and thus not result in information about which  configurations of projects work and which do not.
The three VAWO programs exemplify this situation. The Arrest Policies  Program provided grantees with the flexibility to develop their respective  projects within six purpose areas: implementing mandatory arrest or  proarrest programs and policies in police departments, tracking domestic  violence cases, centralizing and coordinating police domestic violence  operations, coordinating computer tracking systems, strengthening legal  advocacy services, and educating judges and others about how to handle  domestic violence cases. Likewise, the STOP Grants Program encouraged  tribal governments to develop and implement culture-specific strategies  for responding to violent crimes against Indian women and provide  appropriate services for those who are victims of domestic abuse, sexual  assault, and stalking. Finally, the Rural Domestic Violence Program was  designed to provide sites with the flexibility to develop projects, based on  need, with respect to the early identification of, intervention in, and  prevention of woman battering and child victimization; with respect to  increases in victim safety and access to services; with respect to  enhancement of the investigation and prosecution of crimes of domestic  violence; and with respect to the development of innovative,  comprehensive strategies for fostering community awareness and  prevention of domestic abuse. Because participating grant sites  emphasized different project configurations, the resulting evaluation may  not provide information that could be generalized to a broader  implementation of the program.
section_title: Site Selection Not Shown to be Representative within the VAWO Programs
paragraphs: The sites participating in the three VAWO evaluations were not shown to  be representative of their programs. Various techniques are available to  help evaluators choose representative sites and representative participants  within those sites. Random sampling of site and participant selection are  ideal, but when this is not feasible, other purposeful sampling methods can  be used to help approximate the selection of an appropriate sample (e.g.,  choosing the sample in such proportions that it reflects the larger  population--stratification). At a minimum, purposeful selection can ensure  the inclusion of a range of relevant sites.
As discussed earlier, in the case of the Arrest Policies Program, six  purpose areas were identified in the grant solicitation. The six grantees  chosen for participation in the evaluation were not however, selected on  the basis of their representativeness of the six purpose areas or the  program as a whole. Rather, they were selected on the basis of factors  related solely to program “stability;” that is; they were considered likely to  receive local funding after the conclusion of federal grant funding, and key  personnel would continue to participate in the coordinated program effort.  Similarly, the 10 Rural Domestic Violence impact evaluation grantees were  not selected for participation on the basis of program representativeness  or the specific purpose areas discussed earlier. Rather, sites were selected  by the grant evaluator on the basis of “feasibility”; specifically, whether the  site would be among those participants equipped to conduct an  evaluation. Similarly, the STOP Violence Against Indian Women Program  evaluation used 3 of the original 14 project sites for a longitudinal study;  these were not shown to be representative of the sites in the overall  program. For another phase of the evaluation, the principal investigator  indicated that grantee sites were selected to be geographically  representative of American Indian communities. While this methodology  provides for inclusion of a diversity of Indian tribes in the sample from  across the country, geography as a sole criterion does not guarantee  representativeness in relation to many other factors.
section_title: Lack of Comparison Groups in the VAWO Evaluations Hinders Ability to Isolate the Effects of the Programs
paragraphs: Each of the three VAWO evaluations was designed without comparison  groups—a factor that hinders the evaluator’s ability to isolate and  minimize external factors that could influence the results of the study. Use  of comparison groups is a standard practice employed by evaluators to  help determine whether differences between baseline and follow-up  results are due to the program under consideration or to some other  programs or external factors. For example, as we reported in 1997, to  determine whether a drug court program has been effective in reducing  criminal recidivism and drug relapse, it is not sufficient to merely  determine whether those participating in the drug court program show  changes in recidivism and relapse rates. Changes in recidivism and relapse  variables between baseline and program completion could be due to other  external factors, irrespective of the drug court program (e.g., the state may  have developed harsher sentencing procedures for those failing to meet  drug court objectives). If, however, the drug court participant group is  matched at baseline against another set of individuals, “the comparison  group” who are experiencing similar life circumstances but who do not  qualify for drug court participation (e.g., because of area of residence),  then the comparison group can help in isolating the effects of the drug  court program. The contrasting of the two groups in relation to recidivism  and relapse can provide an approximate measure of the program’s impact.
All three VAWO program impact evaluations lacked comparison groups.  One issue addressed in the Arrest Policies Program evaluation, for  example, was the impact of the program on the safety and protection of  the domestic violence victim. The absence of a comparison group,  however, makes it difficult to firmly conclude that change in the safety and  protection of participating domestic abuse victims is due to the Arrest  Policies Program and not to some other external factors operating in the  environment (e.g., economic changes, nonfederal programs such as safe  houses for domestically abused women, and church-run support  programs). Instead of using comparison groups, the Arrest Policies  Program evaluation sought to eliminate potential competing external  factors by collecting and analyzing extensive historical and interview data  from subjects and by conducting cross-site comparisons; the latter method  proved unfeasible.
The STOP Violence Against Indian Women Discretionary Grant Program  has sought in part, to reduce violent crimes against Indian women by  changing professional staff attitudes and behaviors. To do this, some  grantees created and developed domestic violence training services for  professional staff participating in site activities. Without comparison  groups, however, assessing the effect of the STOP training programs is  difficult. Attitudes and behaviors may change for myriad reasons unrelated  to professional training development initiatives. If a treatment group of  professional staff receiving the STOP training had been matched with a  comparison group of professional staff that was similar in all ways except  receipt of training, there would be greater confidence that positive change  could be attributed to the STOP Program. Similarly, the lack of  comparison groups in the Rural Domestic Violence evaluation makes it  difficult to conclude that a reduction in violence against women and  children in rural areas can be attributed entirely, or in part, to the Rural  Domestic program. Other external factors may be operating.
section_title: VAWO Data Collection and Analytical Problems Evident during Grant Implementation
paragraphs: All three VAWO impact evaluations involved data collection and analytical  problems that may affect the validity of the findings and conclusions. For  example, we received documentation from NIJ on the STOP Grant  Program for Reducing Violence Against Indian Women showing that only  43 percent of 127 grantees returned a mail survey. In addition, only 25  percent of 127 tribes provided victim outcome homicide and  hospitalization rates—far less than the percentage needed to draw broad- based conclusions about the intended goal of assessing victim well being.  In the Arrest Policies evaluation, NIJ reported that the evaluators  experienced difficulty in collecting pre-grant baseline data from multiple  sites and the quality of the data was oftentimes inadequate, which  hindered their ability to statistically analyze change over time. In addition,  evaluators were hindered in several work areas by lack of automated data  systems; data were missing, lost, or unavailable; and the ability to conduct  detailed analyses of the outcome data was sometimes limited. For the  Rural Domestic Violence evaluation, evaluators proposed using some  variables (e.g., number and type of awareness messages disseminated to  the community each month, identification of barriers to meeting the needs  of women and children, and number of police officers who complete a  training program on domestic violence) that are normally considered to  relate more to a process evaluation than an impact evaluation. NIJ noted  that outcome measurement indicators varied by site, complicating the  ability to draw generalizations. NIJ further indicated that the evaluation  team did not collect baseline data prior to the start of the program, making  it difficult to identify change resulting from the program.
section_title: VAWO Peer Review Committees Expressed Concerns about Two Evaluations
paragraphs: NIJ does not require applicants to use particular evaluation  methodologies. NIJ employs peer review committees in deciding which  evaluation proposals to fund. The peer review committees expressed  concerns about two of the three VAWO program evaluation proposals (i.e.,  those for the Arrest Policies and Rural Domestic Violence programs) that  were subsequently funded by NIJ. Whereas NIJ funded the Arrest Policies  evaluation as a grant, NIJ funded the Rural Domestic Violence evaluation  as a cooperative agreement so that NIJ could provide substantial  involvement in conducting the evaluation.
A peer-review panel and NIJ raised several concerns about the Arrest  Policies Program evaluation proposal. These concerns included issues  related to site selection, victim interviewee selection and retention in the  sample, and the need for additional impact measures and control  variables. The grant applicant’s responses to these issues did not remove  concerns about the methodological rigor of the application, thus calling  into question the ability of the grantee to assess the impact of the Arrest  Policies Program. For example, the grantee stated that victim interviewee  selection was to be conducted through a quota process and that the  sampling would vary by site. This would not allow the evaluators to  generalize program results. Also, the evaluators said that they would study  communities at different levels of “coordination” when comparison groups  were not feasible, but they did not adequately explain (1) how the various  levels of coordination would be measured, (2) the procedures used to  select the communities compared, and (3) the benefits of using this  method as a replacement for comparison groups. NIJ subsequently funded  this evaluation, and it is still in progress.
A peer review committee for the Rural Domestic Violence and Child  Victimization Enforcement Grant Program evaluation also expressed  concerns about whether the design of the evaluation application, as  proposed, would demonstrate whether the program was working. In its  consensus review notes, the peer review committee indicated that the  “ability to make generalizations about what works and does not work will  be limited.” The peer review committee also warned of outside factors  (e.g., unavailability of data, inaccessibility of domestic violence victims)  that could imperil the evaluation efforts of the applicant. Based on the  peer review committee’s input, NIJ issued the following statement to the  applicant: “As a national evaluation of a major programmatic effort we  hope to have a research design and products on what is working, what is  not working, and why. We are not sure that the proposed design will get us  to that point.” We reviewed the grant applicant’s response to NIJ’s concern  in its application addendum and found that the overall methodological  design was still not discussed in sufficient detail or depth to determine  whether the program was working.  Although the Deputy Director of NIJ’s  Office of Research and Evaluation asserted that this initial application was  only for process evaluation funding, our review of available documents  showed that the applicant had provided substantial information about  both the process and impact evaluation methodologies in the application  and addendum. We believe that the methodological rigor of the addendum  was not substantially improved over that of the original application.  The  Deputy Director told us that, given the “daunting challenge faced by the  evaluator,” NIJ decided to award the grant as a cooperative agreement.  Under this arrangement, NIJ was to have substantial involvement in  helping the grantee conduct the program evaluation. The results of that  evaluation have not yet been submitted. The evaluator’s draft final report  is expected no earlier than April 2002.
section_title: Byrne Evaluation Was Successfully Designed and Implemented
paragraphs: In contrast to the three VAWO impact evaluations, the Byrne impact  evaluation employed methodological design and implementation  procedures that met a high standard of methodological rigor, fulfilling  each of the criteria indicated above.  In part, this may reflect the fact that  Byrne’s CAR demonstration program, unlike the VAWO programs, was  according to the Assistant Attorney General, intended to test a research  hypothesis, and the evaluation was designed accordingly. CAR provided  participants with the opportunity to use a limited number of program  services (e.g., family services, education services, after-school activities)  that were theoretically related to the impact variables and the prevention  and reduction of drug use and delinquency. As a result, the evaluation was  not complicated by project heterogeneity. All five grantees participated in  the evaluation. High-risk youths within those projects were randomly  selected from targeted neighborhood schools, providing student  representation. Additionally, CAR evaluators chose a matched comparison  group of youths with similar life circumstances (e.g., living in distressed  neighborhoods and exposed to similar school and family risk factors) and  without access to the CAR Program. Finally, no significant data collection  implementation problems were associated with the CAR Program. The  data were collected at multiple points in time from youths (at baseline, at  completion of program, and at one year follow-up) and their caregivers (at  baseline and at completion of program). Self-reported findings from  youths were supplemented by the collection of more objective data from  school, police, and court records on an annual basis, and rigorous test  procedures were used to determine whether changes over time were  statistically significant.
Additionally, CAR’s impact evaluation used control groups, a  methodologically rigorous technique not used in the three VAWO  evaluations. To further eliminate the effects of external factors, youths in  the targeted neighborhood schools were randomly assigned either to the  group receiving the CAR Program or to a control group that did not  participate in the program. Since the CAR Program group made significant  gains over the same-school group and the matched comparison group not  participating in the program, there was good reason to conclude that the  CAR Program was having a beneficial effect on the targeted audience.
Appendix I provides summaries of the four evaluations.
section_title: Conclusions
paragraphs: Despite great interest in assessing results of OJP’s discretionary grant  programs, it can be extremely difficult to design and execute evaluations  that will provide definitive information. Our in-depth review of one Byrne  and three VAWO impact evaluations that have received funding since fiscal  year 1995 has shown that, in some cases, the flexibility that can be  beneficial to grantees in tailoring programs to meet their communities’  needs has added to the complexities of designing impact evaluations that  will result in valid findings. Furthermore, the lack of site  representativeness, appropriate comparison groups, and problems in data  collection and analysis may compromise the reliability and validity of  some of these evaluations. We recognize that not all evaluation issues that  can compromise results are easily resolvable, including issues involving  comparison groups and data collection. To the extent that methodological  design and implementation issues can be overcome, however, the validity  of the evaluation results will be enhanced. NIJ spends millions of dollars  annually to evaluate OJP grant programs. More up-front attention to the  methodological rigor of these evaluations will increase the likelihood that  they will produce meaningful results for policymakers. Unfortunately, the  problematic evaluation grants that we reviewed are too far along to be  radically changed. However, two of the VAWO evaluation grants are still in  the formative stage; more NIJ attention to their methodologies now can  better ensure useable results.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Attorney General instruct the Director of NIJ to  assess the two VAWO impact evaluations that are in the formative stage  to address any potential methodological design and implementation  problems and, on the basis of that assessment, initiate any needed  interventions to help ensure that the evaluations produce definitive  results. We further recommend that the Attorney General instruct the  Director of NIJ to assess its evaluation process with the purpose of  developing approaches to ensure that future impact evaluation studies are  effectively designed and implemented so as to produce definitive results.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a copy of a draft of this report to the Attorney General for  review and comment. In a February 13, 2002, letter, the Assistant Attorney  General commented on the draft. Her comments are summarized below  and presented in their entirety in appendix III.
The Assistant Attorney General agreed with the substance of our  recommendations and said that NIJ has begun, or plans to take steps, to  address them. Although it is still too early to tell whether NIJ’s actions will  be effective in preventing or resolving the problems we identified, they  appear to be steps in the right direction. With regard to our first  recommendation—that NIJ assess the two VAWO impact evaluations in  the formative stage to address any potential design and implementation  problems and initiate any needed intervention to help ensure definitive  results—the Assistant Attorney General noted that NIJ has begun work to  ensure that these projects will provide the most useful information  possible. She said that for the Crimes Against Women on Campus Program  evaluation, NIJ is considering whether it will be possible to conduct an  impact evaluation and, if so, how it can enhance its methodological rigor  with the resources available.  For the Civil Legal Assistance Program  evaluation, the Assistant Attorney General said that NIJ is working with  the grantee to review site selection procedures for the second phase of the  study to enhance the representativeness of sites.  The Assistant Attorney  General was silent about any additional steps that NIJ would take during  the later stages of the Civil Legal Assistance Program process evaluation  to ensure the methodological rigor of the impact phase of the study.  However, it seems likely that as the process evaluation phase of the study  continues, NIJ may be able to take advantage of additional opportunities  to address any potential design and implementation problems.
With regard to our second recommendation—that NIJ assess its evaluation  process to develop approaches to ensure that future evaluation studies are  effectively designed and implemented to produce definitive results—the  Assistant Attorney General stated that OJP has made program evaluation,  including impact evaluations of federally funded programs, a high priority.  The Assistant Attorney General said that NIJ has already launched an  examination of NIJ’s evaluation process. She also noted that, as part of its  reorganization, OJP plans to measurably strengthen NIJ’s capacity to  manage impact evaluations with the goal of making them more useful for  Congress and others.  She noted as an example that OJP and NIJ are  building measurement requirements into grants at the outset, requiring  potential grantees to collect baseline data and track the follow-up data  through the life of the grant. We have not examined OJP’s plans for  reorganizing, nor do we have a basis for determining whether OJP’s plans  regarding NIJ would strengthen NIJ’s capacity to manage evaluations.  However, we believe that NIJ and its key stakeholders, such as Congress  and the research community, would be well served if NIJ were to assess  what additional actions it could take to strengthen its management of  impact evaluations regardless of any reorganization plans.
In her letter, the Assistant Attorney General pointed out that the report  accurately describes many of the challenges facing evaluators when  conducting research in the complex environment of criminal justice  programs and interventions. However, she stated that the report could  have gone further in acknowledging these challenges. The Assistant  Attorney General also stated that the report contrasts the Byrne evaluation  with the three VAWO evaluations and obscures important programmatic  differences that affect an evaluator’s ability to achieve “GAO’s conditions  for methodological rigor.” She pointed out that the Byrne CAR Program  was intended to test a research hypothesis and that the evaluation was  designed accordingly, i.e., the availability of baseline data were ensured;  randomization of effects were stipulated as a precondition of  participation; and outcome measures were determined in advance on the  basis of the theories to be tested.  She further stated that, in contrast, all of  the VAWO programs were (1) highly flexible funding streams, in keeping  with the intention of Congress, that resulted in substantial heterogeneity at  the local level and (2) well into implementation before the evaluation  started. The Assistant Attorney General went on to say that it is OJP’s  belief that evaluations under less than optimal conditions can provide  valuable information about the likely impact of a program, even though  the conditions for methodological strategies and overall rigor of the CAR  evaluation were not available.
We recognize that there are substantive differences in the intent, structure,  and design of the various discretionary grant programs managed by OJP  and its bureaus and offices. And, as stated numerous times in our report,  we acknowledge not only that impact evaluation can be an inherently  difficult and challenging task but also that measuring the impact of these  specific Byrne and VAWO programs can be arduous, given that they are  operating in an ever changing, complex environment. We agree that not all  evaluation issues that can compromise results are easily resolvable, but  we firmly believe that, with more up-front attention to design and  implementation issues, there is a greater likelihood that NIJ evaluations  will provide meaningful results for policymakers. Absent this up-front  attention, questions arise as to whether NIJ is (1) positioned to provide the  definitive results expected from an impact evaluation and (2) making  sound investments given the millions of dollars spent on these evaluations.
The Assistant Attorney General also commented that although our report  discussed “generally accepted social science standards,” it did not specify  the document that articulates these standards or describe our elements of  rigor. As a result, the Assistant Attorney General said, OJP had to infer that  six elements had to be met to achieve what “GAO believes” is necessary to  “have a rigorous impact evaluation.” Specifically, she said that she would  infer that, for an impact evaluation to be rigorous would require (1)  selection of homogenous programs, (2) random or stratified site sampling  procedures (or selection of all sites), (3) use of comparison groups, (4)  high response rates, (5) available and relevant automated data systems  that will furnish complete and accurate data to evaluators in a timely  manner, and (6) funding sufficient to accomplish all of the above.  Furthermore, the Attorney General said that it is rare to encounter all of  these conditions or be in a position to engineer all of these conditions  simultaneously; and when all of these conditions are present, the  evaluation would be rigorous.  She also stated that it is possible to glean  useful, if not conclusive, evidence of the impact of a program from an  evaluation that does not rise to the standard recommended by GAO  because of the unavoidable absence of “one or more elements.”
We agree that our report did not specify particular documents that  articulate generally accepted social science standards.  However, the  standards that we applied are well defined in scientific literature. All  assessments of the impact evaluations we reviewed were completed by  social scientists with extensive experience in evaluation research.  Throughout our report, we explain our rationale and the criteria we used  in measuring the methodological rigor of NIJ’s impact evaluations.  Furthermore, our report does not suggest that a particular standard or set  of standards is necessary to achieve rigor, nor does it suggest that other  types of evaluations, such as comprehensive process evaluations, are any  less useful in providing information on how a program is operating. In this  context, it is important to point out that the scope of our work covered  impact evaluations of Byrne and VAWO discretionary grant programs—  those designed to assess the net effect of a program by comparing  program outcomes with an estimate of what would have happened in the  absence of the program.
We differ with the Assistant Attorney General with respect to the six  elements cited as necessary elements for conducting an impact evaluation.  Contrary to the Assistant Attorney General’s assertion, our report did not  state that a single homogeneous program is a necessary element for  conducting a rigorous impact evaluation.  Rather, we pointed out that  heterogeneity or program variation is a challenge that adds to the  complexity of designing an evaluation.  In addition, contrary to her  assertion, the report did not assert that random sampling or stratification  was a necessary element for conducting a rigorous evaluation; instead it  stated that when random sampling is not feasible, other purposeful  sampling methods can be used. With regard to comparison groups, the  Assistant Attorney General’s letter asserted that GAO standards required  using groups that do not receive program benefits as a basis of comparison  with those that do receive such benefits. In fact, we believe that the  validity of evaluation results can be enhanced through establishing and  tracking comparison groups. If other ways exist to effectively isolate the  impacts of a program, comparison groups may not be needed. However,  we saw no evidence that other methods were effectively used in the VAWO  impact evaluations we assessed.
The Assistant Attorney General also suggested that we used a 75 percent  or greater response rate for evaluation surveys as a standard of rigor. In  fact, we did not—we simply pointed out that NIJ documents showed a 43  percent response rate on one of the STOP Grant Program evaluation  surveys. This is below OMB’s threshold response rate level—the level  below which OMB particularly believes nonresponse bias and statistical  problems could affect surveys. Given OMB guidance, serious questions  could be raised about program conclusions drawn from the results of a  survey with a 43 percent response rate. In addition, the Assistant Attorney  General suggested that, by GAO standards, she would have to require  state, local, or tribal government officials to furnish complete and accurate  data in a timely manner. In fact, our report only points out that NIJ  reported that evaluators were hindered in carrying out evaluations  because of the lack of automated data systems or because data were  missing, lost, or unavailable—again, challenges to achieving  methodologically rigorous evaluations that could produce meaningful and  definitive results.
Finally, the Assistant Attorney General’s letter commented that one of the  elements needed to meet “all of GAO’s conditions” of methodological rigor  is sufficient funding. She stated that more rigorous impact evaluations cost  more than those that provide less scientific findings, and she said that OJP  is examining the issue of how to finance effective impact evaluations. We  did not assess whether funding is sufficient to conduct impact evaluations,  but we recognize that designing effective and rigorous impact evaluations  can be expensive—a condition that could affect the number of impact  evaluations conducted. However, we continue to believe that with more  up-front attention to the rigor of ongoing and future evaluations, NIJ can  increase the likelihood of conducting impact evaluations that produce  meaningful and definitive results.
In addition to the above comments, the Assistant Attorney General made a  number of suggestions related to topics in this report. We have included  the Assistant Attorney General’s suggestions in the report, where  appropriate. Also, the Assistant Attorney General provided other  comments in response to which we did not make changes.  See appendix  III for a more detailed discussion of the Assistant Attorney General’s  comments.
We are sending copies of this report to the Chairman and the Ranking  Minority Member of the Senate Judiciary Committee; to the Chairman and  Ranking Minority Member of the House Judiciary Committee; to the  Chairman and Ranking Minority Member of the Subcommittee on Crime,  House Committee on the Judiciary; to the Chairman and the Ranking  Minority Member of the House Committee on Education and the  Workforce; to the Attorney General; to the OJP Assistant Attorney  General; to the NIJ Director; to the BJA Director; to the VAWO Director;  and to the Director, Office of Management and Budget. We will also make  copies available to others on request.
If you or your staff have any questions about this report, please contact  John F. Mortin or me at (202) 512-8777. Key contributors to this report are  acknowledged in appendix IV.
section_title: Appendix I: Summaries of the Impact Evaluations of Byrne and VAWO Programs
paragraphs: Evaluation findings  Assessment of evaluation	 This evaluation has several limitations. (1) The choice of the 10 impact sites is skewed toward the  National Evaluation of the Rural Domestic Violence and Child Victimization Grant Program  COSMOS Corporation  The Violence Against Women Office’s (VAWO) Rural Domestic Violence Program, begun in fiscal year  1996, has funded 92 grantees through September 2001. The primary purpose of the program is to  enhance the safety of victims of domestic abuse, dating violence, and child abuse.  The program  supports projects that implement, expand, and establish cooperative efforts between law enforcement  officers, prosecutors, victim advocacy groups, and others in investigating and prosecuting incidents of  domestic violence, dating violence, and child abuse; provide treatment, counseling, and assistance to  victims; and work with the community to develop educational and prevention strategies directed toward  these issues.  The impact evaluation began in July 2000, with a final report expected no earlier than April 2002.  Initially, 10 grantees were selected to participate in the impact evaluation; 9 remain in the evaluation.  Two criteria were used in the selection of grant participants: the “feasibility” of earlier site-visited  grantees to conduct an outcome evaluation and VAWO recommendations based on knowledge of  grantee program activities. Logic models were developed, as part of the case study approach, to show  the logical or plausible links between a grantee’s activities and the desired outcomes. The specified  outcome data were to be collected from multiple sources, using a variety of methodologies during 2- to- 3-day site visits (e.g., multiyear criminal justice, medical, and shelter statistics were to be collected from  archival records; community stakeholders were to be interviewed; and grantee and victim service  agency staff were to participate in focus groups). At the time of our review, this evaluation was funded at  $719,949. The National Institute of Justice (NIJ) could not separate the cost of the impact evaluation  from the cost of the process evaluation.  Too early to assess.  technically developed evaluation sites and is not representative of either all Rural Domestic  Violence Program Grantees, particular types of projects, or delivery styles. (2) The lack of comparison  groups will make it difficult to exclude the effect of external factors, such as victim safety and improved  access to services, on perceived change. (3) Several so-called short-term outcome variables are in fact  process variables (e.g., number of police officers who complete a training program on domestic  violence, identification of barriers to meeting the needs of women and children). (4) It is not clear how  interview and focus group participants are to be selected, (5) Statistical procedures to be used in the  analyses have not been sufficiently identified. The NIJ peer review committee had concerns about  whether the evaluation could demonstrate that the program was working. NIJ funded the application as  a cooperative agreement because a substantial amount of agency involvement was deemed necessary  to meet the objectives of the evaluation.
Evaluation findings  Assessment of evaluation	 This evaluation has several limitations: the absence of a representative sampling frame for site  National Evaluation of the Arrest Policies Program  Institute for Law and Justice (ILJ)  The purpose of VAWO’s Arrest Policies Program is to encourage states, local governments, and Indian  tribal governments to treat domestic violence as a serious violation of criminal law. The program  received a 3-year authorization (fiscal years 1996 through 1998) at approximately $120 million to fund  grantees under six purpose areas: implementing mandatory arrest or proarrest programs and policies in  police departments, tracking domestic violence cases, centralizing and coordinating police domestic  violence operations, coordinating computer tracking systems, strengthening legal advocacy services,  and educating judges and others about how to handle domestic violence cases. Grantees have flexibility  to work in several of these areas. At the time the NIJ evaluation grant was awarded, 130 program  grantees had been funded; the program has since expanded to 190 program grantees.  The impact evaluation began in August 1998, with a draft final report due in March 2002. Six grantees  were chosen to participate in the impact evaluation. Each of the six sites was selected on the basis of  program “stability,” not program representativeness. Within sites, both quantitative and qualitative data  were to be collected and analyzed to enable better understanding of the impact of the Arrest Program  on offender accountability and victim well being. This process entailed reviewing data on the criminal  justice system’s response to domestic violence; tracking a random sample of 100 offender cases, except  in rural areas, to determine changes in offender accountability; conducting content analyses of police incident  reports to assess change in police practices and documentation; and interviewing victims or survivors at each  site to obtain their perceptions of the criminal justice system’s response to domestic violence and its impact  on their well-being. ILJ had planned cross-site comparisons and the collection of extensive historical and  interview data to test whether competing factors could be responsible for changes in arrest statistics. At  the time of our review, this evaluation was funded at $1,130,574. NIJ could not separate the cost of the  impact evaluation from the cost of the process evaluation.  Too early to assess.  selection, the lack of comparison groups, the inability to conduct cross-site comparisons, and the lack of  a sufficient number of victims in some sites to provide a perspective on the changes taking place in  domestic violence criminal justice response patterns and victim well-being. In addition, there was  difficulty collecting pre-grant baseline data, and the quality of the data was oftentimes inadequate,  limiting the ability to measure change over time. Further, automated data systems were not available in  all work areas, and data were missing, lost, or unavailable. An NIJ peer review committee also  expressed some concerns about the grantee’s methodological design.
Evaluation findings  Assessment of evaluation	 Methodological design and implementation issues may cause difficulties in attributing program impact. A  Impact Evaluation of STOP Grant Programs for Reducing Violence Against Indian Women  The University of Arizona  VAWO’s STOP (Services, Training, Officers, and Prosecutors) Grant Programs for Reducing Violence  Against Indian Women Discretionary Grant Program was established under Title IV of the Violent Crime  Control and Law Enforcement Act of 1994. The program’s principal purpose is to reduce violent crimes  against Indian women. The program, which began in fiscal year 1995 with 14 grantees, encourages  tribal governments to develop and implement culture-specific strategies for responding to violent crimes  against Indian women and providing appropriate services for those who are victims of domestic abuse,  sexual assault, and stalking. In this effort, the program provided funding for the services and training,  and required the joint coordination, of nongovernmental service providers, law enforcement officers, and  prosecutors hence the name, the STOP Grant Programs for Reducing Violence Against Indian Women.  The University of Arizona evaluation began in October 1996 with an expected final report due in March  2002. The basic analytical framework of this impact evaluation involves the comparison of quantitative  and qualitative pre-grant case study histories of participating tribal programs with changes taking place  during the grant period. Various data collection methodologies have been adopted (at least in part, to be  sensitive to the diverse Indian cultures): 30-minute telephone interviews, mail surveys, and face-to-face  2- to 3 day site visits. At the time of our review, this evaluation was funded at $468,552. NIJ could not  separate the cost of the impact evaluation from the cost of the process evaluation.  Too early to assess.  number of methodological aspects of the study remain unclear: the site selection process for “in-depth  case study evaluations;” the methodological procedures for conducting the longitudinal evaluation; the  measurement, validity, and reliability of the outcome variables; the procedures for assessing impact; and  the statistical tests to be used for determining significant change. Comparison groups are not included in  the methodological design. In addition, only 43 percent of the grantees returned the mail survey, only 25  percent could provide the required homicide and hospitalization rates; and only 26 victims of domestic  violence and assault could be interviewed (generally too few to measure statistical change).  Generalization of evaluation results to the entire STOP Grant Programs for Reducing Violence Against  Indian Women will be difficult, given these problems.
Longitudinal Impact Evaluation of the Strategic Intervention for High Risk Youth (a.k.a. The  Children at Risk Program) The Urban Institute  The Children at Risk (CAR) Program, a comprehensive drug and delinquency prevention initiative  funded by the Bureau of Justice Assistance (BJA), the Office of Juvenile Justice and Delinquency  Prevention (OJJDP), the Center on Addiction and Substance Abuse, and four private foundations, was  established to serve as an experimental demonstration program from 1992 to 1996 in five grantee cities.  Low-income youths (11 to 13 years old) and their families, who lived in severely distressed  neighborhoods at high-risk for drugs and crime, were targeted for intervention. Eight core service  components were identified: case management, family services, education services, mentoring, after- school and summer activities, monetary and nonmonetary incentives, community policing, and criminal  justice and juvenile intervention (through supervision and community service opportunities). The goals of  the program were to reduce drug use among targeted families and improve the safety and overall quality  of life in the community.  The evaluation actually began in 1992, and the final report was submitted in May 1998. The study used  both experimental and quasi-experimental evaluation designs. A total of 671 youths in target  neighborhood schools were randomly assigned to either a treatment group (which received CAR  services and the benefit of a safer neighborhood) or to a control group (which received only a safer  neighborhood). Comparison groups (n=203 youths) were selected from similar high-risk neighborhoods  by means of  census tract data; comparison groups did not have access to the CAR Program. Interviews  were conducted with youth participants at program entry (baseline), program completion (2 years later),  and 1-year after program completion. A parent or caregiver was interviewed at program entry and  completion. Records from schools, police, and courts were collected annually for each youth in the  sample as a means of obtaining more objective data. The total evaluation funding was $1,034,732.  Youths participating in CAR were significantly less likely than youths in the control group to have used  gateway and serious drugs, to have sold drugs, or to have committed violent crimes in the year after the  program ended. CAR youths were more likely than youths in the control and comparison groups to  report attending drug and alcohol abuse programs. CAR youths received more positive peer support  than controls, associated less frequently with delinquent peers, and were pressured less often by peers  to behave in antisocial ways. CAR households used more services than control group households, but  the majority of CAR households did not indicate using most of the core services available.
Assessment of evaluation	 CAR is a methodologically rigorous evaluation in both its design and implementation. The evaluation  findings demonstrate the value of the program as a crime and drug prevention initiative.
section_title: Appendix II: NIJ’s Guidelines for Disseminating Discretionary Grant Program Evaluation Results
paragraphs: NIJ has the primary role of disseminating Byrne and VAWO discretionary  grant program evaluation results of evaluations managed by NIJ, according  to NIJ, BJA, and VAWO officials, because NIJ is responsible for conducting  these types of evaluations. NIJ is authorized to share the results of its  research with federal, state, and local governments. NIJ also disseminates  information on methodology designs.
subsections: section_title: NIJ’s Dissemination Practices
paragraphs: NIJ’s practices for disseminating program evaluation results are specified  in its guidelines. According to the guidelines, once NIJ receives a final  evaluation report from the evaluators and the results of peer reviews have  been incorporated, NIJ grant managers are to carefully review the final  product and, with their supervisor, recommend to the NIJ Director which  program results to disseminate and the methods for dissemination. Before  making a recommendation, grant managers and their supervisors are to  consider various criteria, including policy implications, the nature of the  findings and research methodology, the target audience and their needs,  and the cost of various forms of dissemination. Upon receiving the  recommendation, the Director of NIJ is to make final decisions about  which program evaluation results to disseminate. NIJ’s Director of  Planning and Management said that NIJ disseminates program evaluation  results that are peer reviewed, are deemed successful, and add value to the  field.
Once the decision has been made to disseminate program evaluation  results and methodologies with researchers and practitioners, NIJ can  choose from a variety of publications, including its Research in Brief; NIJ  Journal–At a Glance: Recent Research Findings; Research Review; NIJ  Journal–Feature Article; and Research Report. In addition, NIJ provides  research results on its Internet site and at conferences. For example, using  its Research in Brief publication, NIJ disseminated impact evaluation  results on the Byrne Children at Risk (CAR) program to 7,995 practitioners  and researchers, including state and local government and law  enforcement officials; social welfare and juvenile justice professionals;  and criminal justice researchers. In addition, using the same format, NIJ  stated that it distributed the results of its process evaluation of the Byrne  Comprehensive Communities Program (CCP) to 41,374 various  constituents, including local and state criminal and juvenile justice agency  administrators, mayors and city managers, leaders of crime prevention  organizations, and criminal justice researchers. NIJ and other OJP offices  and bureaus also disseminated evaluation results during NIJ’s annual  conference on criminal justice research and evaluation. The July 2001  conference was attended by 847 public and nonpublic officials, including  criminal justice researchers and evaluation specialists from academic  institutions, associations, private organizations, and government agencies;  federal, state, and local law enforcement, court, and corrections officials;  and officials representing various social service, public housing, school,  and community organizations.
In addition to NIJ’s own dissemination activities, NIJ’s Director of  Planning and Management said that it allows and encourages its  evaluation grantees to publish their results of NIJ-funded research via  nongovernmental channels, such as in journals and through presentations  at professional conferences. Although NIJ requires its grantees to provide  advance notice if they are publishing their evaluation results, it does not  have control over its grantees’ ability to publish these results. NIJ does,  however, require a Justice disclaimer that the “findings and conclusions  reported are those of the authors and do not necessarily reflect the official  position or policies of the U.S. Department of Justice.” For example,  although NIJ has not yet disseminated the program evaluation results of  the three ongoing VAWO impact evaluations that we reviewed, one of the  evaluation grantees has already issued, on its own Internet site, 9 of 20  process evaluation reports on the Arrests Policies evaluation grant. The  process evaluations were a component of the NIJ grantee’s impact  evaluation of the Arrest Policies Program. Because the evaluations were  not completed, NIJ required that the grantee’s publication of the process  evaluations be identified as a draft report pending final NIJ review.
section_title: Dissemination of NIJ’s Byrne Discretionary Grants: Comprehensive Communities Program and Children at Risk Program
paragraphs: As discussed earlier, NIJ publishes the results of its evaluations in several  different publications. For example, NIJ used the Research in Brief format  to disseminate evaluation results for two of the five Byrne discretionary  grant programs Comprehensive Communities Program (CCP) and  Children at Risk Program (CAR) that were evaluated during fiscal years  1995 through 2001. Both publications summarize information including  each program’s evaluation results, methodologies used to conduct the  evaluations, information about the implementation of the programs  themselves, and services that the programs provided. CCP’s evaluation  results were based on a process evaluation. Although a process evaluation  does not assess the results of the program being evaluated, it can provide  useful information that explains the extent to which a program is  operating as intended. The NIJ Research in Brief on the Byrne CAR  Discretionary Grant Program provides a summary of issues and findings  regarding the impact evaluation. That summary included findings reported  one year after the end of the program, in addition to a summary of the  methodology used to conduct the evaluation, the outcomes, the lessons  learned, and a major finding from the evaluation.
section_title: Appendix III: Comments from the Department of Justice
paragraphs: Following are GAO’s comments on the Department of Justice’s February 13, 2002,  letter.
subsections: section_title: GAO Comments
paragraphs: 1.	 We have amended the text to further clarify that BJA administers the Byrne  program, just as its counterpart, VAWO, administers its programs (see page 4).  However it is important to point out that regardless of the issues raised by OJP,  the focus of our work was on the methodological rigor of the evaluations we  reviewed, not the purpose and structure of the programs being evaluated. As  discussed in our Scope and Methodology section, our work focused on program  evaluation activities associated with Byrne and VAWO discretionary grant  programs generally and the methodological rigor of impact evaluation studies  associated with those programs in particular. To make our assessment, we relied  on NIJ officials to identify which of the program evaluations of Byrne and VAWO  grant programs were, in fact, impact evaluation studies. We recognize that there  are substantial differences among myriad OJP programs that can make the design  and implementation of impact evaluations arduous. But, that does not change the  fact that impact evaluations, regardless of differences in programs, can benefit  from stronger up-front attention to better ensure that they provide meaningful and  definitive results.  2.	 We disagree with OJP’s assessment of our report’s treatment of program variation.  As discussed earlier, the scope of our review assessed impact evaluation activities  associated with Byrne and VAWO discretionary grant programs, not the programs  themselves. We examined whether the evaluations that NIJ staff designated as  impact evaluations were designed and implemented with methodological rigor.  In  our report we observe that variations in projects funded through VAWO programs  complicate the design and implementation of impact evaluations.
According to the Assistant Attorney General, this variation in projects is  consistent with the intent of the programs’ authorizing legislation. We recognize  that the authorizing legislation provides VAWO the flexibility in designing these  programs.  In fact, we point out that although such flexibility may make sense  from a program perspective, project variation makes it much more difficult to  design and implement a definitive impact evaluation. This poses sizable  methodological problems because an aggregate analysis, such as one that might  be constructed for an impact evaluation, could mask the differences in  effectiveness among individual projects and therefore not result in information  about which configurations of projects work and which do not.  3.	 We have amended the Results in Brief to clarify that peer reviews evaluated  proposals. However, it is important to note that while the peer review committees  may have found the two VAWO grant applications to be the most superior, this  does not necessarily imply that the impact evaluations resulting from these  applications were well designed and implemented. As discussed in our report, the  peer review panel for each of the evaluations expressed concerns about the  proposals that were submitted, including issues related to site selection and the  need for additional impact measures and control variables. Our review of the  documents NIJ made available to us, including evaluators’ responses to peer  review comments, led to questions about whether the evaluators’ proposed  methodological designs were sufficient to allow the evaluation results to be  generalized and to determine whether the program was working.  4.	 We have amended the Background section of the report to add this information  (see page 6).  5.	 As discussed in OJP’s comments, we discussed external factors that could  account for changes that the Rural Program evaluation observed in victims’ lives  and the criminal justice system. We did so not to critique or endorse activities that  the program was or was not funding, but to demonstrate that external factors may  influence evaluation findings. To the extent that such factors are external, the  Rural Program evaluation methodology should account for their existence and  attempt to establish controls to minimize their affect on results (see page 14).  We  were not intending to imply that alcohol is a cause for domestic violence, as  suggested by the Assistant Attorney General, but we agree that it could be an  exacerbating factor that contributes to violence against women.  6.	 As discussed earlier, we recognize that there are substantive differences in the  intent, structure, and design of the various discretionary grant programs managed  by OJP and its bureaus and offices. Also, as stated numerous times in our report,  we acknowledge not only that impact evaluation can be an inherently difficult and  challenging task but also that measuring the impact of these specific Byrne and  VAWO programs can be arduous given that they are operating in an ever changing,  complex environment. We agree that not all evaluation issues that can  compromise results are easily resolvable, but we firmly believe that with more up- front attention to design and implementation issues, there is a greater likelihood  that NIJ impact evaluations will provide meaningful results for policymakers.
Regarding the representativeness of sites, NIJ documents that were provided  during our review indicated that sites selected during the Rural Program  evaluation were selected on the basis of feasibility, as discussed in our report—  specifically, whether the site would be among those participants equipped to  conduct an evaluation. In its comments, OJP stated that the 6 sites selected for the  impact evaluation were chosen to maximize geographical and purpose area  diversity while focusing on sites with high program priority. OJP did not provide  any additional information that would further indicate that the sites were selected  on a representative basis.  OJP did, however, point out that the report does not  address how immensely expensive the Arrest evaluation would have become if it  had included all 130 sites.  We did not address specific evaluation site costs  because we do not believe that there is a requisite number of sites needed for any  impact evaluation to be considered methodologically rigorous. Regarding OJP’s  comment about the flexibility given to grantees in implementing VAWO grants,  our report points out that project variation complicates evaluation design and  implementation. Although flexibility may make sense from a program perspective,  it makes it difficult to generalize about the impact of the entire program.  7.	 We used the drug court example to illustrate, based on our past work, how  comparison groups can be used in evaluation to isolate and minimize external  factors that could influence the study results.  We did not, nor would we, suggest  that any particular unit of analysis is appropriate for VAWO evaluations since the  appropriate unit of analysis is dependent upon the specific circumstances of the  evaluation. We were only indicating that since comparison groups were not  utilized in the studies, the evaluators were not positioned to demonstrate that  change took place as a result of the program.  8.	 We do not dispute that VAWO grant programs may provide valuable outputs over  the short term.  However, as we have stated previously, the focus of our review  was on the methodological rigor of impact evaluations--those evaluations that are  designed to assess the net effect of a program by comparing program outcomes  with an estimate of what would have happened in the absence of the program.  Given the methodological issues we found, it is unclear whether NIJ will be able  to discern long-term effects due to the program.  9.	 As stated in our report, we acknowledge not only that impact evaluation can be an  inherently difficult and challenging task, but that measuring the impact of Byrne  and VAWO programs can be arduous given the fact that they are operating in an  ever changing, complex environment. We agree that not all evaluation issues that  can compromise results are easily resolvable, but we firmly believe that, with  more up-front attention to design and implementation issues, there is a greater  likelihood that NIJ evaluations will provide meaningful results for policymakers.  As we said before, absent this up-front attention, questions arise as to whether NIJ  is (1) positioned to provide the definitive results expected from an impact  evaluation and (2) making sound investments given the millions of dollars spent  on these evaluations.  If NIJ believes that the circumstances of a program are such  that it cannot be evaluated successfully (in relation to impact) they should not  proceed with an impact evaluation.  10.  We have amended the footnote to state that from fiscal year 1995 through fiscal  year 1999, this program was administered by VAWO. As of fiscal year 2000,  responsibility for the program was shifted to OJP’s Corrections Program Office  (see page 5).  11.  In regard to the number of grants, we have amended the text to reflect that the  information NIJ provided during our review is the number of grantees, not the  number of grants  (see pages 25 and 26).  We have also amended our report to  reflect some of the information provided in VAWO’s description of the Rural  Domestic Violence Program to further capture the essence of the program (see  page 25).  12.  We disagree. We believe that separating the cost of the impact and process  evaluations is more than a matter of bookkeeping. Even though the work done  during the process phase of an evaluation may have implications for the impact  evaluation phase of an evaluation, it would seem that, given the complexity of  impact evaluations, OJP and NIJ would want to have in place appropriate controls  to provide reasonable assurance that the evaluations are being effectively and  efficiently carried out at each phase of the evaluation. Tracking the cost of these  evaluation components would also help reduce the risk that OJP’s, NIJ’s, and,  ultimately, the taxpayer’s investment in these impact evaluations is not wasted.  13.  As discussed earlier, we recognize that there are substantive differences in the  intent, structure, and design of the various discretionary grant programs managed  by OJP and its bureaus and offices, including those managed by VAWO.  Our  report focuses on the rigor of impact evaluations of grant programs administered  by VAWO and not on the program’s implementing legislations. Although flexibility  may make sense from a program perspective, it makes it difficult to develop a well  designed and methodologically rigorous evaluation that produces generalizeable  results about the impact of the entire program.  14.  Our report does not suggest that other types of evaluations, such as  comprehensive process evaluations, are any less useful in providing information  about how well a program is operating. The scope of our review covered impact  evaluations of Byrne and VAWO discretionary grant programs—those designed to  assess the net effect of a program by comparing program outcomes with an  estimate of what would have happened in the absence of the program.
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the above, Wendy C. Simkalo, Jared A. Hermalin, Chan My J.  Battcher, Judy K. Pagano, Grace A. Coleman, and Ann H. Finley made key  contributions to this report.