Juvenile Justice Grants: DOJ Should Take Additional Actions to Strengthen Performance and Fraud Risk Management
Released: 2019-12-18T00:00:00

Letter

Background

OJJDP Programs and Grants

In fiscal year 2018, OJJDP administered 16 programs, which collectively included 53 grant solicitations and 295 associated awards. As shown in table 1, the 16 programs included the Title II Formula Grant Program, two discretionary grant programs for youth mentoring and missing and exploited children that awarded the most funding, and 13 other discretionary grant programs to support efforts such as helping opioid- affected youth, victims of child abuse, and girls in the juvenile justice system. For a complete list of OJJDP programs and grants in fiscal year 2018, see appendix I.

On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted various requirements to strengthen accountability and oversight in OJJDP grant programs, including grants to states under the Title II Formula Grant Program. The Reform Act expressed the sense of Congress that OJJDP must restore meaningful enforcement, and states must exercise vigilant oversight, to ensure compliance with core requirements of the Title II Formula Grant Program. Among the measures the Reform Act put into place to achieve this goal was a requirement that states maintain an “effective” system of monitoring compliance—a requirement that became effective for fiscal year 2020 awards. In contrast, the requirement prior to fiscal year 2020 was for states to maintain an “adequate” system of monitoring compliance, which was the requirement that applied to the grant performance period reviewed for this report (October 2015 through September 2018).

This report is one of many called for by the Reform Act to improve oversight of OJJDP grant programs, including the Title II Formula Grant Program. Other oversight requirements in the Reform Act are addressed to OJJDP; OJP’s Office of Audit, Assessment, and Management; and the DOJ OIG. Appendix III provides information on selected Reform Act requirements that relate to accountability and oversight in the Title II Formula Grant Program, and the status of efforts to implement them.

OJP Grant Monitoring and Compliance Auditing Activities

All OJP grantees and awards are subject to “grant monitoring.” Grant monitoring consists of (1) programmatic and (2) financial monitoring, and according to OJP officials, helps ensure the programmatic and financial integrity and accountability of grantees. OJP policy requires programmatic desk reviews on all open awards each fiscal year and “in- depth” monitoring—consisting of enhanced programmatic desk reviews or site visits—on at least 10 percent of the total number and dollar amount of open and active awards annually. In addition, OJP’s Office of the Chief Financial Officer plans to financially monitor at least 10 percent of the award population annually.

States awarded Title II Formula Grants are subject to an additional form of monitoring—“compliance auditing”—which is conducted by OJJDP annually to fulfill statutory requirements unique to the program. Specifically, “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. See appendix IV for more information on compliance auditing in the Title II Formula Grant Program associated with the performance period we reviewed.

Both the grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits (see figure 1).

Fraud Risk Management and Related Guidance

Fraud and “fraud risk” are distinct concepts. Fraud—obtaining something of value through willful misrepresentation—is challenging to detect because of its deceptive nature. Fraud risk (which is a function of likelihood and impact) exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. Fraud risk management is a process for ensuring program integrity by continuously and strategically mitigating the likelihood and impact of fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if actual fraud has not yet been identified or occurred.

According to federal standards and guidance, executive-branch agency managers—including those at DOJ, OJP, and OJJDP—are responsible for managing fraud risks and implementing practices for addressing those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks.

We issued our Fraud Risk Framework in July 2015. The Fraud Risk Framework provides a comprehensive set of leading practices, arranged in four components, which serve as a guide for agency managers developing efforts to combat fraud in a strategic, risk-based manner. The Fraud Risk Framework is also aligned with Principle 8 (“Assess Fraud Risk”) of Standards for Internal Control in the Federal Government. The Fraud Reduction and Data Analytics Act of 2015 requires agencies to establish financial and administrative controls that are aligned with the Fraud Risk Framework’s leading practices. In addition, guidance under Office of Management and Budget Circular A- 123 affirms managers should adhere to the leading practices identified in the Fraud Risk Framework.

OJJDP Has Goals and Measures to Assess the Performance of Individual Programs, But Has Not Set Corresponding Performance Targets

OJJDP Has Established Goal Statements and Performance Measures for Individual Programs

OJJDP has established goal statements and performance measures for individual programs. Goal statements broadly convey a program’s overall intent, and performance measures assess program outputs or outcomes. Figure 2 below shows current goal statements and performance measures for the Youth Mentoring Program, for example.

OJJDP requires grantees to report on performance measures which are listed in grant solicitations. OJJDP has designated 15 of its measures as “core” measures that are generally applicable across most OJJDP programs. OJJDP aggregates data from some of these core measures across all applicable programs collectively to assess progress toward office-level targets. Targets represent outputs or outcomes expressed as a numeric goal. For example, OJJDP has office-level targets for the percent of youth who offend and reoffend, as well as the percent of grantees that have implemented an evidence-based program. OJJDP obtains, reports on, and uses OJJDP program performance data, as shown in figure 3 below.

OJJDP Is Taking Steps to Address Limitations in Grantee-Submitted Program Performance Data

We reviewed a selection of performance data that grantees provided to OJJDP for three programs: Title II Formula Grant Program, Youth Mentoring Program, and Gang Prevention Program. As a result of our review—which covered data from October 2015 through December 2018—we determined that these data were not sufficiently reliable for the purpose of providing examples of performance results in this report. For example, we found that grantees reported inconsistent information, such as different numbers of youth served for different performance measures within the same data collection period. Further, some performance measures double-count youth when presenting data by year. OJJDP collects data from discretionary grantees during two 6-month reporting periods and some measures are not designed to be aggregated across time periods.

Nevertheless, we found that OJJDP was aware of most of the issues we identified and is taking steps to improve the reliability of grantee- submitted data. For instance, beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process to identify inconsistent or otherwise questionable data and reach out to grantees for verification. Appendix II includes information about the issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of these data.

OJJDP Has Not Set Program-Level Targets, Which Limits Its Ability to Assess Progress toward Program Goals

While OJJDP has established goal statements and performance measures, it has not set numeric targets by which it can assess progress for each individual program. For example, one stated goal of the Title II Formula Grant Program is to prevent youth already in the juvenile justice system from reoffending. OJJDP’s annual Title II Formula Grant Program performance reports state the percentage of youth who reoffended, the performance measure for this goal; however, the reports do not provide a target against which to evaluate whether the result reflects progress toward the stated goal.

Further, while OJJDP has set several office-wide targets for all programs collectively, these targets may not be appropriate for assessing the progress of individual programs because programs vary in size. For example, according to grantee-submitted performance data, the Title II Formula Grant Program served approximately 100,000 youth in fiscal year 2018, while the Gang Prevention Program served about 1,000 youth in calendar year 2018. Thus, office-level targets, while useful, may be more representative of the performance of OJJDP’s largest programs and obscure the results of individual programs.

Since June 2019, OJJDP has been reviewing individual program goal statements and performance measures as part of an OJP-wide review, but this review does not include setting program-level targets. OJJDP officials said that they have not set program-level targets for two reasons: (1) there was uncertainty about whether OJJDP had the authority to do so (versus OJP’s Office of the Chief Financial Officer), and (2) setting such targets has not been a priority for OJJDP in the past, in part, due to a lack of resources. In October 2019, OJP clarified that the OJJDP Administrator has authority for OJJDP performance measurement, including setting program-level and office-level targets. OJJDP officials also stated that program oversight has recently become a higher priority and they plan to bring on new staff; and they agreed with the need to set program-level targets.

Tracking performance measures against established numeric targets is a leading practice in performance management. Numeric targets establish standards against which federal programs can measure progress towards goals because comparisons can be easily made between projected performance and actual results. Thus, updating program goal statements and performance measures would be more effective with related numeric targets. One goal of the OJP-wide review is to increase accountability for achieving results. Setting program-level targets could help OJJDP meet this goal by establishing a clear means by which progress toward goals can be measured.

DOJ Has Taken Steps to Consider Fraud Risks Relevant to OJJDP Grant Programs, but Has Not Determined a Fraud Risk Tolerance

Tools Used in OJP’s Grant Monitoring and Compliance Auditing Efforts Provide Insight on Grant Risks Affecting OJJDP Grant Programs

According to OJP officials, some of the tools it uses in grant monitoring and compliance auditing consider fraud risk affecting OJJDP grant programs. As previously discussed, OJP’s grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits. To carry out these efforts, OJP relies on various tools to assess the overall risk of grantees and awards. Figure 4 provides additional information on the tools that provide insight on fraud risks, according to OJP officials.

According to OJP officials, pre-award and ongoing risk assessment processes that apply to all OJJDP grantees are the primary way in which the office identifies fraud risks. The bullets below describe the nature of the primary tools used during these risk assessment processes for all OJJDP grantees and how they provide insights into potential fraud risks.

Financial Capability Questionnaire: This questionnaire includes 28 questions designed to provide insight on the financial systems and internal controls a grantee has in place prior to receiving an award. OJP developed the current version of this questionnaire in part as a response to a 2013 DOJ OIG audit, according to OJP officials. All applicants for OJJDP awards are required to fill out this questionnaire, and new grantees’ pre-award risk scores and corresponding risk levels rely, in part, on the applicants’ responses to it. These include detailed questions related to the capabilities of the applicant’s financial management system, such as whether it has the capability to record expenditures by budget cost categories. According to OJP officials, if an applicant’s accounting system cannot do so, the opportunity for fraud increases because the commingling of funds between budget categories would make it difficult to determine whether federal funds were spent in accordance with the approved budget. The questionnaire also includes items related to procurement, travel policy, and subrecipient management and monitoring. These questions may similarly indicate increased fraud risk depending on how the applicant responds. For instance, according to OJP officials, questions related to procurement are designed to determine whether the applicant employs a fair, transparent, and competitive procurement process. If an applicant’s procurement standards do not meet these criteria, the likelihood of fraudulent activity may increase.

Grant Assessment Tool: The Grant Assessment Tool helps assess open/active OJJDP awards and grantees against 38 risk criteria. OJP officials identified 14 of these criteria as being indicators of potential fraud risks, such as the results of recent audits, whether the award has subawards or subcontracts, and whether grantees have completed progress reports on time. Officials explained that progress report delinquencies, for example, may be an indicator that a recipient does not have adequate internal controls to handle federal awards, which may provide a greater opportunity for fraud. The Grant Assessment Tool generates a risk score and corresponding monitoring priority for each open/active grantee and award quarterly. According to OJP officials, Grant Assessment Tool criteria have evolved over time in response to common audit or monitoring findings, as well as ongoing coordination with the DOJ OIG, as discussed later.

For the Title II Formula Grant Program, grantees are also assigned risk assessment scores and audited using the compliance auditing tools described in the bottom half of figure 4. OJP officials stated that certain responses to any of the questions in the tools used during compliance auditing may indicate inadequate program management or weak internal controls, which may increase the risk of fraudulent activity.

OJP officials use a variety of other tools during grant monitoring to monitor each OJJDP grantee or award (see tools in the top half of figure 4). The tools used differ depending on the type and level of monitoring being performed. According to OJP officials, all of the monitoring and auditing tools include detailed questions that provide insight about the strength of a grantee’s internal controls. In cases where a grantee is unable to provide adequate documentation in response to these questions, OJP officials stated that they may have weak internal controls which may increase the risk of fraudulent activity.

OJP Coordinates with the DOJ OIG

OJP regularly coordinates with the DOJ OIG on issues related to fraud risk affecting OJJDP grant programs through meetings, trainings, and reviews of OIG audits. Specifically, according to officials, staff from OJP’s Office of Audit, Assessment, and Management meet with the OIG two to three times per year to discuss fraud allegations and ongoing fraud investigations related to OJJDP grant programs. These discussions assist OJP officials in determining their monitoring or auditing plans, but also provide insights about the types of issues that are being referred to the OIG for investigation, which can then inform needed changes to monitoring and auditing tools, according to OJP officials. The OIG also provides training to OJP staff every other year on how to identify and report potential fraud. Additionally, Office of Audit, Assessment, and Management officials regularly review OIG audit findings pertaining to individual grantees and compare them to grant monitoring findings for the same grantees. According to OJP officials, the purpose of these reviews is to evaluate where OJP may be able to improve its monitoring processes. Officials stated that one example of an outcome from such a review is the previously discussed Financial Capability Questionnaire.

DOJ Has Not Determined the Department’s Fraud Risk Tolerance

According to JMD officials, the department is implementing fraud risk management requirements through an iterative process that will be completed over multiple years, which leverages the department’s overall ERM processes. Leading practices for planning and conducting fraud risk assessments acknowledge that assessing fraud risks is an iterative process. According to JMD officials, as a result of this iterative approach, the fraud risk assessments JMD conducted in fiscal years 2017 and 2018 did not fully align with selected leading practices in the Fraud Risk Framework, which include documenting a fraud risk profile. Although the Department continued to implement additional leading practices from the Fraud Risk Framework by developing a fraud risk profile as part of the fiscal year 2019 fraud risk assessment, the fraud risk profile did not include any consideration of DOJ’s fraud risk tolerance—another leading practice identified by the Fraud Risk Framework.

2017 Fraud Risk Assessment

In fiscal year 2017, JMD conducted an assessment of fraud risks that consisted of four facilitated discussion groups with relevant officials. One discussion group was tailored to the specific concerns of grant programs—which included grant programs managed by OJJDP—as recommended by leading practices of the Fraud Risk Framework. During the grant-focused facilitated discussion group, participants identified five fraud scenarios relevant to DOJ grant programs, such as misdirection of funds, which occurs when a recipient deliberately misdirects funds in a manner inconsistent with the purpose outlined in the award agreement. After participants reached consensus on the fraud scenarios, they then ranked the inherent risk of each scenario, in alignment with leading practices of the Fraud Risk Framework.

DOJ officials also told us that participants ranked residual risk using a voting tool, which required the participants to understand the various management controls in place to address a particular fraud risk. However, the documented outcomes of these discussions did not identify specific fraud risk controls or the extent to which those controls mitigate specific fraud risks, and as a result, it is unclear how these residual risk values were determined. Further, according to officials responsible for managing the fiscal year 2017 effort, the facilitated discussion group did not determine a specific and measurable fraud risk tolerance for DOJ grant programs generally nor discuss specific fraud risk management activities or controls for any specific grant programs. As discussed in greater detail later in this report, leading practices for fraud risk management state that managers should determine a fraud risk tolerance and Standards for Internal Control in the Federal Government states that managers should define risk tolerances in specific and measurable terms so they are clearly stated and can be measured.

2018 Fraud Risk Assessment

The fiscal year 2018 effort to assess fraud risk at DOJ consisted of a brief survey about fraud-related issues that was distributed to all DOJ components and was more limited in nature than the fiscal year 2017 effort. Specifically, the fiscal year 2018 questionnaire did not ask the components to identify inherent or residual risks, as was done in the prior year. Further, for the grants category, JMD included four of the five fraud risk areas that were addressed in fiscal year 2017. However, the fiscal year 2018 questionnaire did specifically ask components to identify key fraud risk management activities designed to prevent, detect, or respond to fraud, which had not been part of the fiscal year 2017 effort. Including information about control activities indicates additional maturation of JMD’s fraud risk management activities, but the questionnaire did not ask components to consider the extent to which these control activities mitigate the likelihood and impact of risk—as recommended by leading practices of the Fraud Risk Framework. The questions asked in fiscal year 2018 are shown in table 2 below.

According to JMD officials, the components’ responses to this questionnaire were summarized for internal purposes, but no additional analysis or work, such as defining a fraud risk tolerance and documenting a fraud risk profile, was completed for any of the categories listed in table 2. Further, JMD officials stated that the department does not believe a full-scale fraud risk assessment is warranted annually, and the fiscal year 2018 effort was designed to build on the prior year’s assessment.

According to officials, this is part of the department’s iterative approach to implementing fraud risk management requirements, which, consistent with leading practices, may not necessarily incorporate all relevant leading practices in each iteration.

2019 Fraud Risk Assessment and Profile

In December 2019, JMD officials provided the final summary of the fiscal year 2019 fraud risk assessment, which included a fraud risk profile as recommended by leading practices. According to JMD officials, to conduct the fiscal year 2019 fraud risk assessment, JMD officials first created a fraud risk profile template using information from the 2017 and 2018 assessments. Figure 5 shows an excerpt of JMD’s draft fraud risk profile template, as of September 2019.

After senior leadership reviewed the pre-populated template, JMD held a facilitated discussion with representatives from each component to evaluate the risk information presented in the template for each topic area. JMD identified several risks for each topic area, including the same five risks for the grants area that were identified in fiscal year 2017.

Based on information provided, the fiscal year 2019 fraud risk assessment and resulting fraud risk profile incorporate many leading practices of the Fraud Risk Framework. These include consideration of inherent fraud risk, current fraud risk controls and their suitability (the extent to which control activities mitigate the likelihood and impact of risk), and residual fraud risk. However, the 2019 fraud risk profile did not determine a measurable fraud risk tolerance, or prioritize residual risk against that tolerance for any of the assessed categories, including grants. Managers’ defined risk tolerance may depend on the circumstances of individual programs and other objectives beyond mitigation of fraud risks. Leading practices for fraud risk management state that managers should define a fraud risk tolerance, examine the suitability of existing fraud controls, and then prioritize residual fraud risks. In doing so, managers should consider the extent to which existing control activities mitigate inherent risks and whether the remaining risks exceed managers’ tolerance. Based on this analysis and the defined risk tolerance, managers then rank residual risks in order of priority, and determine their responses, if any, to mitigate those risks that exceed their risk tolerance.

JMD officials stated that they did not yet define the department’s fraud risk tolerance for any of the assessed categories because they view it as the next step in the maturation of DOJ’s fraud risk assessment processes. However, JMD did not provide details or documentation of its plans to develop a specific and measurable fraud risk tolerance for the next iteration of their fraud risk assessments. Although following an iterative approach to fraud risk management is consistent with leading practices, until DOJ defines a measurable fraud risk tolerance for the assessed categories, the department may not effectively allocate limited resources to address fraud risks—including those associated with OJJDP grant programs. Specifically, by determining a measurable fraud risk tolerance for the grants category and assessing identified residual fraud risks against that tolerance to prioritize these risks, the department will help ensure that OJJDP’s grant programs are not vulnerable to greater risks than DOJ is willing to tolerate. Doing so will also provide assurance that OJJDP does not unintentionally over-allocate limited funding to address fraud risks the department is willing to tolerate.

Plans for Conducting Future Fraud Risk Assessments

According to JMD officials, they are in the process of awarding a contract that will result in an implementation plan for addressing fraud risk management requirements in the future. Specifically, in July 2019, JMD released a Request for Quotes for a Blanket Purchase Agreement in support of DOJ’s implementation of OMB Circular A-123. One of the deliverables JMD expects to order under this agreement is an implementation plan for addressing fraud risk management requirements, which will include developing a plan for conducting regular fraud risk assessments consistent with leading practices for fraud risk management. According to officials, they expect to award the agreement by the end of calendar year 2019, after which the contractor will perform task orders issued by JMD that will include details related to the methodology, timeframes, and staffing associated with each deliverable. Because neither the award nor the task orders were in place at the time of our review, we cannot determine whether DOJ’s planned efforts will fully align with the leading practices of the Fraud Risk Framework, but we will continue to monitor DOJ’s efforts during related ongoing work.

Conclusions

In fiscal year 2018, OJJDP made 295 awards totaling over $290 million to support programs intended to ensure youth are held appropriately accountable and empower youth to live productive lives. Performance measurement helps ensure funding achieves such outcomes and fraud risk management helps ensure funding is not improperly diverted from this intended purpose. Both of these management principles facilitate stewardship and accountability for federal funds.

Since June 2019, OJJDP has been reviewing and updating goal statements and performance measures for individual programs. While OJJDP has office-level targets, it does not have program-level targets. Program-level targets (specific numeric goals) would help OJJDP assess progress toward individual program goals and increase accountability for achieving positive outcomes.

Over the past few years, DOJ has taken steps to consider fraud risk for all DOJ grants, including OJJDP’s. Determining a fraud risk tolerance—and assessing residual fraud risk against that tolerance to prioritize these risks—would help OJP calibrate resources to address grant fraud risk for OJJDP programs, helping ensure that resources are not under- or over- allocated.

Recommendations for Executive Action

We are making two recommendations, including one to OJJDP and one to JMD. Specifically: The OJJDP Administrator should set performance targets for individual grant programs. (Recommendation 1)

The Assistant Attorney General for Administration should ensure that future department-level fraud risk profiles (1) determine the department’s fraud risk tolerance for DOJ grants—which include OJJDP grant programs, and (2) prioritize residual fraud risks based on an assessment against that tolerance, consistent with leading practices in GAO’s Fraud Risk Framework. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of this report to DOJ for review and comment. In an email message, an official within JMD stated that the Department concurred with our recommendations. In written comments provided by OJP, which are reproduced in appendix V, the agency concurred with our recommendation that it set performance targets for individual OJJDP programs. Specifically, OJP stated that the OJJDP Administrator will set performance targets, to the extent practical, for OJJDP's current and new grant programs. Further, the OJJDP Administrator will ensure that the performance targets are reviewed annually. We believe this action, if implemented, would address our recommendation. DOJ also concurred with our second recommendation to include a fraud risk tolerance for DOJ grants in future department-level fraud risk profiles, but did not provide details as to how they will address it. DOJ also provided technical comments, which we have incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Attorney General, and other interested parties. In addition, the report is available at no charge on GAO’s website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact Gretta Goodwin at (202) 512-8777 GoodwinG@gao.gov or Rebecca Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Office of Juvenile Justice and Delinquency Prevention Programs and Grants in Fiscal Year 2018

Under the Department of Justice Appropriations Act, 2018, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) administered 16 grant programs in fiscal year 2018, corresponding with a total of 53 grant solicitations and 295 awards. OJJDP issued certain grant solicitations to carry out the purposes of more than one program. Table 3 shows OJJDP’s fiscal year 2018 programs and associated information, including the primary source of authority for each program, as identified by OJJDP.

Appendix II: GAO Review of Performance Data and Steps Being Taken to Improve Data Reliability

To provide examples of performance results from three of the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) 16 programs funded in fiscal year 2018, we assessed the reliability of key data elements and methods used to calculate selected performance measures. We determined that data grantees submitted to OJJDP for the programs we selected were not sufficiently reliable for this purpose. Nevertheless, we present rounded numbers of “youth served” for two of the programs we selected (2018 data only), as we found these data to be reliable for the specific purpose of illustrating differences in the relative sizes of the programs. This appendix discusses our approach for selecting programs to review, identifying the timeframe of review, and selecting performance measures to review. It also discusses issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of data submitted by grantees.

Programs Reviewed

We selected three of OJJDP’s 16 programs funded in fiscal year 2018 and listed in appendix I. Specifically, we selected the Title II Formula Grant Program because it is OJJDP’s only formula grant program and because it is a comprehensive nationwide program. We selected the Youth Mentoring and Gang Prevention programs from among the 15 discretionary grant programs as a result of the following process: we ranked programs by total amount awarded during fiscal years 2016 through 2018 and randomly selected one program from the top 50 percent and one from the bottom 50 percent, and we excluded programs that we had recently reviewed in other reports—such as the Tribal Youth and Victims of Child Abuse programs.

Timeframe of Review

We obtained performance data for the selected programs from OJJDP from October 2015 through December 2018. We chose this timeframe because 2018 was the latest full calendar year for which data were available, and looking back three years was sufficient to capture variations in the programs’ funding levels.

Performance Measures Reviewed

We selected seven of OJJDP’s 15 core performance measures for review. Specifically, we selected measures that focus on youth and that objectively measure short-term outputs or outcomes, as follows: number of program youth served number and percent of youth with whom an evidence-based program number and percent of youth completing program requirements number and percent of program youth who offend (short-term) number and percent of program youth who re-offend (short-term) number and percent of program youth who are victimized (short-term); number and percent of program youth who are re-victimized (short- term)

Not all of these performance measures are applicable to all programs. Performance measures that assess the number and percent of program youth who offend, re-offend, and are victimized measure both short-term and long-term outcomes, and we focused only on short-term because of the challenge that grantees face in tracking youth after they exit programs, as explained to us by OJJDP officials.

Review Process

We reviewed relevant performance measures, their definitions, and the methodology for calculating them. We reviewed data cleaning and validation procedures that OJJDP uses to verify data provided by grantees. We also tested grantee-submitted data provided to us by OJJDP for missing data, outliers, and inconsistencies. For example, we tested for illogical values, such as different numbers of youth served within the same time period, or numerators that were higher than the denominators (i.e. more youth successfully exiting the program than the total number exiting). Finally, we interviewed knowledgeable OJJDP officials and contractors on several occasions.

Results of Review

We determined that the data were not sufficiently reliable for the purpose of providing examples of program performance results due to several limitations (bulleted below). Although not every limitation applies to every program we reviewed, the number and significance of the collective issues identified led us not to use the data as examples of performance results in this report.

Performance measure results for discretionary programs are often double-counted when presenting data by year. OJJDP collects data from discretionary grantees every 6-months; however, some performance measures are not designed to be aggregated across time periods. OJJDP acknowledges that aggregating data from two, 6-month time periods often results in double-counting when presenting annualized data.

Some grantees report inconsistent numbers of youth served. For instance, in the Gang Prevention program, the total number of youth served from January through June 2018 was reported as 387 and 267 for different performance measures in the same data set (within one reporting period). We also identified instances where grantees reported inconsistent numbers of youth across reporting periods.

Some grantees do not respond to requests to verify questionable data. According to OJJDP officials grantees may lack resources or staff capacity to collect data and track youth’s outcomes. As a result, some grantees may submit incorrect data or submit data after the reporting deadline. Beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process whereby contractors flag inconsistent or otherwise questionable data and reach out to grantees for clarification. However, according to officials, grantees do not face any consequences if they do not respond to requests from OJJDP for data verification. For instance, for Youth Mentoring program data submitted by grantees for July through December 2018, OJJDP contractors flagged 122 of 630 grantees as having potentially inaccurate data, but subsequently received responses from only 74 of the 122 grantees.

Inconsistent performance measure definitions. We found inconsistencies in the definitions for the following performance measures: (1) number and percent of program youth who offend (short-term), and (2) number and percent of program youth who re- offend (short-term). For both measures, one documented definition states that they apply only to youth who offend or re-offend during the reporting period, and another documented definition states that it also applies to youth who exited the program 0-6 months ago.

Reporting on a subset of youth not representative of all program youth. OJJDP uses the number of youth “tracked” as the denominator for several of its performance measures, including the number of youth who offend and reoffend. According to OJJDP, the number of youth tracked for the offend and reoffend measures should ideally be the same number as total youth served by a program. However, the number of youth tracked is usually lower than the number of youth served. As a result, the measure often reflects a subset of youth that, according to one official, may have characteristics that are not well understood—such as youth or families who are more willing to be tracked because they have not offended recently—and thus may skew the results.

OJJDP Steps to Improve Reliability of Grantee- Submitted Data

OJJDP acknowledges there are concerns with the quality of the grantee- reported performance measures data. According to OJJDP officials, the limitations are the result of several challenges which they are addressing:

Replacing outdated data collection tool. Officials said their current data collection tool is outdated and can be unwieldy and confusing for grantees. Along these lines, the tool only includes automated error checks for a limited amount of data fields and does not include an auto-populate feature, which would prevent grantees from entering illogical or inconsistent data. However, according to Office of Justice Program (OJP) officials, as of October 2019, OJP is designing a new data collection tool for all OJP components—including OJJDP—that will include automated error checks and an auto-populate feature, and they plan to implement this tool beginning in October 2020.

Updating performance measures. Officials said that some performance measures are also outdated, such as those that result in duplication when reported annually. Officials also said that some performance measures are confusing to grantees. Nevertheless, as part of an ongoing OJP-wide review of performance measures, OJJDP is in the process of reviewing and updating all OJJDP performance measures and plans to provide updated definitions and instructions to grantees.

Increasing grantee response rate for data verification. To increase grantee response rates to data verification requests, OJJDP reported that it is exploring possible consequences for grantees if they do not respond, such as increased scrutiny by OJJDP staff who oversee awards or temporary withholding of funds until verifications are submitted. According to officials, whatever approach (or approaches) they decide on, they will implement them by March 2020.

Appendix III: Selected Oversight Requirements Related to the Title II Formula Grant Program

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) within the Department of Justice is responsible for administering grant programs under the Juvenile Justice and Delinquency Prevent Act of 1974. One of these programs, the Title II Formula Grant Program, authorizes the award of formula grants to states to develop programs for juveniles and improve their juvenile justice systems. On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted amendments to the Title II Formula Grant Program, including new accountability and oversight requirements for grantees and OJJDP. The amendments were not effective until the fiscal year 2020 grant award cycle and did not apply to the period of performance we evaluated for this report, which was through fiscal year 2018. Table 4 summarizes the accountability and oversight requirements now in effect for the Title II Formula Grant Program and the status of OJJDP’s efforts to implement them.

The Reform Act also requires several evaluations and assessments to help strengthen OJJDP’s internal controls and identify fraud, waste or abuse in its programs. Table 5 summarizes selected oversight requirements related to the Title II Formula Grant Program.

Appendix IV: Compliance Auditing Applicable to the Title II Formula Grant Program Prior to Fiscal Year 2020

The Title II Formula Grant Program Prior to Fiscal Year 2020

The Title II Formula Grant Program—so called because it was authorized by Title II of the Juvenile Justice and Delinquency Prevention Act of 1974 (JJDPA)—is a state formula grant program, administered by the Office of Juvenile Justice and Delinquency Prevention (OJJDP). The program has been amended several times since 1974—most recently, by the Juvenile Justice Reform Act of 2018 (“Reform Act”), which also called for this evaluation of OJJDP’s performance. The performance data we reviewed (which covers Title II Formula Grants from October 2015 through September 2018) corresponds with statutory requirements in effect at that time, not the current requirements, as amended by the Reform Act, which apply to grant awards made in fiscal year 2020 and subsequent fiscal years. To be consistent with the data we reviewed, this appendix presents information on program requirements that applied prior to fiscal year 2020. Because these requirements are no longer current, we will differentiate them from those that are by citing the superseded edition of the U.S. Code in which they appear—(2012 & Supp. V 2018)—in comparison to the 2018 Main Edition (2018), which contains the provisions now in force.

Statutory Basis for Compliance Auditing

The term “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. During the grant application process, the four core requirements are among several (previously 28, now 33) that a state’s 3-year plan must satisfy for the state to be eligible for award. However, unlike the other eligibility requirements, the four core requirements can trigger a reduction to a state’s grant allocation unless the state maintains compliance during performance. States must provide adequate systems of monitoring their compliance with three of the four core requirements—i.e. those related to when and where juveniles may be detained in detention or correctional facilities—and OJJDP must audit the adequacy of states’ compliance monitoring systems. OJJDP must also determine whether states maintained compliance with each of the four core requirements and, if not, OJJDP must reduce the state’s allocation the following fiscal year by at least 20 percent for each core requirement that the state failed to meet.

During the period covered in our review (i.e., prior to fiscal year 2020), the four core requirements subject to compliance auditing were: 1. Deinstitutionalization of status offenders—which prohibits states from using secure detention or correctional facilities to hold juveniles charged with status offenses (except for a listed few). This requirement also applies to juveniles not charged with an offense but who enter the justice system as aliens or as dependent, neglected or abused youths. 2. Separation of juveniles from adult inmates—which prohibits a state from detaining or confining juveniles protected by the deinstitutionalization of status offenders requirement (see above), or juveniles who are alleged or found to be delinquent, in any institution where they have contact with adult inmates. 3. Removal of juveniles from adult jails and lockups—which prohibits a state from detaining or confining juveniles in adult jails or lockups, except in limited circumstances and for specified periods of time, and only if the juvenile has no contact with adult inmates. 4. Addressing disproportionate minority contact—which requires a state to address the disproportionate number of minority youth who come into contact with the juvenile justice system.

OJJDP’s Compliance Auditing Process

OJJDP’s compliance auditing process during the majority of the period of our review is set forth in a 2017 OJJDP policy document. According to this policy, OJJDP conducts a comprehensive assessment and makes a determination whether the state is in compliance with each of the four core requirements. The comprehensive assessment includes verification of the data submitted, an analysis of the data submitted by the state to evaluate compliance with each of the four core requirements, and a review to assess the adequacy of internal controls over the state’s compliance monitoring process for collecting and reporting compliance monitoring data.

According to this policy, the OJJDP Administrator issues correspondence annually regarding final compliance determinations. These determinations include, as necessary, specific details regarding why a state was determined to be out of compliance with any of the four core requirements or the required compliance monitoring system. Per the policy, a state’s formula grant funding will be reduced by 20 percent for each core requirement with which OJJDP has determined a state to be out of compliance. Additionally, if OJJDP determines that the state has an inadequate system of monitoring, the state may have receipt of its formula grant funding withheld or may be deemed ineligible for a formula grant award.

Finally, according to this policy, OJJDP conducts field audits on a rotating schedule. The purpose of the field audits is to confirm state compliance monitoring activity and practices through direct onsite observation and file review, and to identify needed areas for technical assistance. OJJDP anticipates, with available funding and resources, that every state will receive a field audit every three years.

Appendix V: Comments from the Office of Justice Programs

Appendix VI: GAO Contact and Staff Acknowledgements

GAO Contacts

Staff Acknowledgements

In addition to the contacts named above, Tonnyé Conner-White (Assistant Director), Jonathan Oldmixon (Assistant Director), Jeff Jensen (Analyst-in-Charge), James Ashley, Dominick Dale, Christine Davis, Caroline DeCelles, Elizabeth Dretsch, Eric Hauswirth, Elizabeth Kowalewski, Ben Licht, Jan Montgomery, Heidi Nielson, and Abby Volk, made key contributions to this report.