The author(s) shown below used Federal funding provided by the U.S. 

Department of Justice to prepare the following resource: 

Document Title: Multi-Method Study on Risk Assessment 

Implementation and Youth Outcomes in 

the Juvenile Justice System 

Author(s): Christopher Sullivan, Ph.D., Clare Strange, 

MSW, Carrie Sullivan, MA, Jamie 

Newsome, Ph.D., Melissa Lugo, MS, Derek 

Mueller, MS, Amber Petkus, MS, Bryan 

Holmes, MS, Holly Lonergan, MS, James 

McCafferty, Ph.D. 

Document Number: 252927 

Date Received: May 2019 

Award Number: 2014-MU-FX-0006 

This resource has not been published by the U.S. Department of 

Justice. This resource is being made publically available through the 

Office of Justice Programs’ National Criminal Justice Reference 

Service. 

Opinions or points of view expressed are those of the author(s) and 

do not necessarily reflect the official position or policies of the U.S. 

Department of Justice. 
 

 

 

Final Technical Report 

Multi-Method Study on Risk Assessment Implementation and Youth Outcomes 

in the Juvenile Justice System 

Christopher Sullivan, PhD 

Clare Strange, MSW 

Carrie Sullivan, MA 

Jamie Newsome, PhD 

Melissa Lugo, MS 

Derek Mueller, MS 

Amber Petkus, MS 

Bryan Holmes, MS 

Holly Lonergan, MS 

James McCafferty, PhD 

University of Cincinnati 

Center for Criminal Justice Research 

April 2019 

This project was supported by Award No. 2014-MU-FX-0006 awarded by the Office of Juvenile 

Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. The 

opinions, findings, and conclusions or recommendations expressed in this report are those of the 

author and do not necessarily reflect those of the Department of Justice. 

^Correspondence to christopher.sullivan@uc.edu 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.
 

Table of Contents 

Acknowledgements......................................................................................................................... 6 

Executive Summary ....................................................................................................................... 7 

Study Methods........................................................................................................................... 8 

Data and Measures ................................................................................................................. 10 

Juvenile Justice Personnel In-Person Interview Data. .......................................................... 10 

Web-based Survey Data........................................................................................................ 11 

Youth Case Records.............................................................................................................. 11 

Follow-Up Survey with Youths............................................................................................ 12 

Analytic Procedures................................................................................................................ 12 

Summary of Key Findings ..................................................................................................... 13 

Juvenile Justice Personnel Interviews and Surveys. ............................................................. 13 

Youths Case Records and Follow-Up Survey with Youths.................................................. 14 

Usage Studies........................................................................................................................ 15 

Discussion................................................................................................................................. 16 

Conclusion ............................................................................................................................... 18 

Final Report Introduction and Overview.................................................................................... 20 

Brief Background for Project ................................................................................................ 22 

The Evolution of Risk and Needs Assessment. .................................................................... 22 

The Status of JRNA. ............................................................................................................. 22 

Variability in Findings. ......................................................................................................... 24 

The Importance of Implementation. ..................................................................................... 25 

Description of the Ohio Youth Assessment System (OYAS) .............................................. 27 

Objectives and Research Questions ...................................................................................... 30 

Overview of Research Design and Data Collection ............................................................. 31 

Overview of Report Sections and Usage Studies.................................................................. 32 

Section I. Implementation of Juvenile Risk and Needs Assessment: Views from Juvenile Justice 

Personnel...................................................................................................................................... 34 

Development of the Interview Guide..................................................................................... 34 

Semi-Structured Interview Methods and Site Visit Process ............................................... 36 

Site Visits.............................................................................................................................. 38 

Semi-Structured Interview Notes and Analytic Process. ...................................................... 41 

Juvenile Justice Personnel Web-Based Surveys................................................................... 43 

Interview and Web-Based Survey Overlap........................................................................... 46 

Descriptive Statistics and State-Specific Characteristics .................................................... 46 

Agency Setting...................................................................................................................... 47 

Job Title. ............................................................................................................................... 49 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

2 
 

 

Job Classification.................................................................................................................. 51 

Proximity to the OYAS......................................................................................................... 54 

Job Tenure............................................................................................................................. 56 

State Context for Implementation and Usage ...................................................................... 62 

Timeline for OYAS Implementation. ................................................................................... 62 

Written Policies and Procedures. .......................................................................................... 63 

Juvenile Justice Personnel Interview Findings .................................................................... 74 

How Agencies Use the OYAS.............................................................................................. 74 

Sharing OYAS Assessment Information. ............................................................................. 80 

Aggregate Data Usage. ......................................................................................................... 82 

Attitudes Toward Risk and Needs Assessment and the OYAS............................................ 84 

Overall Satisfaction and Levels of Support for the OYAS................................................... 91 

Qualitative Evidence in Support of the OYAS. .................................................................... 93 

Qualitative Evidence for OYAS Concerns and Areas in Need of Improvement.................. 97 

Degree of Assessment Implementation and Views on Implementation Processes............. 104 

Process of Buy-In for the OYAS. ....................................................................................... 105 

Modifications to Usage/Implementation............................................................................. 111 

Qualitative Evidence in Areas for Improvement for the Implementation of the OYAS. ... 113 

Quality Assurance Measures............................................................................................... 116 

General Conclusions from the Interview Data.................................................................... 120 

Web-Based Survey Findings ................................................................................................ 122 

Overall Satisfaction with the OYAS................................................................................... 123 

The Perceived Usefulness of the OYAS............................................................................. 125 

Perceptions of Benefit to Youths. ....................................................................................... 127 

Perceptions of the OYAS and Consistency. ....................................................................... 129 

Perceptions of OYAS Implementation. .............................................................................. 130 

Broader Attitudes Toward OYAS Implementation. ........................................................... 132 

Strengths and Limitations of the OYAS. ............................................................................ 135 

OYAS Administrator Scoring............................................................................................. 140 

General Conclusions from the Web-Based Survey Data.................................................... 143 

Integrated Analysis of the Interview and Web-Based Survey Data................................. 147 

Job Characteristics. ............................................................................................................. 149 

General Attitudes About Risk and Needs Assessment. ...................................................... 152 

Agency Assessment Strategies. .......................................................................................... 156 

OYAS Information Usage................................................................................................... 157 

OYAS Strengths and Limitations. ...................................................................................... 158 

General Conclusions from the Integrated Analysis of the Interview and Survey Data. ..... 159 

Usage Study 1: Support for the OYAS and Integrity in the Juvenile Risk and Needs 

Assessment Process ............................................................................................................... 161 

Method. ............................................................................................................................... 163 

Analytic Plan....................................................................................................................... 167 

Results................................................................................................................................. 168 

Summary of Support for the OYAS and Integrity Usage Study......................................... 170 

Summary of Section I ........................................................................................................... 172 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

3 
 

Section II. Juvenile Risk and Needs Assessment Usage, Process, and Case Decisions.......... 172 

Methodology .......................................................................................................................... 173 

Comprehensive Assessment Sample................................................................................... 174 

Youth Follow-Up Sample................................................................................................... 182 

Results .................................................................................................................................... 194 

Description of Comprehensive Assessment Sample........................................................... 195 

Descriptive Statistics for Youth Follow-Up Interviews. .................................................... 203 

Alignment of Youth Follow-up and Comprehensive Assessment Samples. ...................... 209 

Usage Study 2: Youth Appraisals by Race and Geography.............................................. 211 

Method ................................................................................................................................ 214 

Results................................................................................................................................. 215 

Summary of Youth Appraisals by Race and Geography..................................................... 218 

Usage Study 3: Race, Ethnicity, and Validity of the Residential OYAS Tool................. 219 

Method ................................................................................................................................ 222 

Analytic Plan....................................................................................................................... 222 

Results................................................................................................................................. 223 

Summary of Race, Ethnicity, and Validity of the OYAS Residential Tool. ...................... 229 

Usage Study 4: Professional Override of Assessment Level ............................................. 230 

Method and Analytic Plan. ................................................................................................. 231 

Results................................................................................................................................. 232 

Summary of Override Substudy.......................................................................................... 239 

Usage Study 5: Mental Health and Substance Use Information and Treatment Matching 

................................................................................................................................................. 240 

Method and Analytic Plan. ................................................................................................. 241 

Results................................................................................................................................. 243 

Summary of Treatment Matching....................................................................................... 245 

Usage Study 6: OYAS Strengths and Barriers................................................................... 246 

Method ................................................................................................................................ 249 

Analytic Plan....................................................................................................................... 250 

Results................................................................................................................................. 251 

Summary of OYAS Strengths and Barriers. ....................................................................... 261 

Summary of Section II.......................................................................................................... 262 

Section III. Youths’ Outcomes and Juvenile Risk and Needs Assessment.............................. 263 

Risk Assessment, Juvenile Justice Decisions, and Youths’ Outcomes............................. 263 

Analysis of Case Decisions and Outcomes Based on JRNA.............................................. 265 

Results................................................................................................................................. 269 

Summary of Case Decisions and Outcomes Based on JRNA ............................................ 290 

Analysis of Youth Follow-Up Interview Data .................................................................... 292 

Summary of Analysis of Youth Follow-Up Interview Data............................................... 301 

IV. Summary and Conclusion ................................................................................................... 303 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

4 
 

 

 

 

 

 

Limitations............................................................................................................................. 303 

Sampling ............................................................................................................................. 303 

Measures and Data Collection ............................................................................................ 306 

Analytic Limitations ........................................................................................................... 308 

Key Findings.......................................................................................................................... 309 

Key Findings and Discussion from Juvenile Justice Personnel Interviews and Surveys ... 309 

Key Findings and Discussion from Youth Case Records and Follow-Up Interviews........ 315 

Analysis of the Linkage Between Risk Assessment, Juvenile Justice Decisions, and Recidivism 

............................................................................................................................................. 320 

Implications and Recommendations ................................................................................... 322 

Training............................................................................................................................... 326 

Usage................................................................................................................................... 331 

Monitoring .......................................................................................................................... 336 

Future Research and Evaluation ......................................................................................... 338 

Conclusion ............................................................................................................................. 340 

References .................................................................................................................................. 342 

Appendices.................................................................................................................................. 352 

Appendix A. Juvenile Justice Personnel Semi-Structured Interview Guide................... 352 

Appendix B. Juvenile Justice Personnel Web-Based Survey............................................ 361 

Appendix C. Code Definitions for Personnel-Generated Strengths of the OYAS 

(Interviews and Surveys)...................................................................................................... 370 

Appendix D. Code Definitions for Personnel-Generated Limitations of the OYAS 

(Interviews and Surveys)...................................................................................................... 372 

Appendix E. Standardized Results for OYAS Usefulness Confirmatory Factor Analysis 

................................................................................................................................................. 374 

Appendix F. OYAS Administrator Sample Characteristics for Vignette and Usefulness 

Analysis .................................................................................................................................. 375 

Appendix G. Overview of Requested Record Data Fields ................................................ 376 

Appendix H. Youth Follow-Up Interview........................................................................... 380 

Appendix I. Variation of Substance Use Indicators on the Disposition, Residential, and 

Reentry Tools ........................................................................................................................ 394 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

5 
 

 

 

 

 

 

 

Acknowledgements 

The research team is grateful to many individuals for their assistance in completing this 

project. Namely, the guidance we have received from our current and previous OJJDP Social 

Science Analysts, Benjamin Adams and Jennifer Tyson, has been instrumental. We are indebted 

to staff who helped secure state-level approvals and set up site visits, those staff members who 

either completed an interview with us or filled out the on-line survey, and those staff who sent us 

requested data, as well as the youth who completed a follow-up survey with us. Additionally, we 

would like to thank the following agencies and individuals: 

University of Cincinnati 

Leah Butler, Katelyn Cain, Erin Cochran, Kevonyah Edwards, Gina Gibbs, Edward Latessa, 

Joshua Long, Jennifer Lux, Ryan Moore, Ben Overberg, John Schwartz, Brian Verkamp 

Indiana 

Indiana Judicial Center, Indiana Supreme Court, Christine Blessinger, Michelle Goodman, Mary 

Kay Hudson, Rebecca Humphrey, Mark Kniola, Kara Mahuron, Jamie Mann, Richard Payne, 

Sarah Schelle, Shannon Schmaltz, Jane Seigel 

Arizona 

Administrative Office of the Courts, Department of Juvenile Corrections, Sandra Acosta, Celena 

M. Angstead, Bruce Baus, Tania Martinez Collier, John Crabtree, Cynthia Goertz, Dany 

Hernandez, Joseph Kelroy, Marjorie Lucas-Voegelin, Scott Mabery, Eric Meaux, Amy Stuart, 

Teresa Tschupp, John Vivian, Howard Wykes 

Ohio 

Ohio Department of Youth Services, James Darnell, Melissa Dunphy, Kevin Floyd, Ryan Gies, 

Robert Hammond, Steve Koenig, Nathan Lawson, Michael Orlando, Anthony Panzino, Harvey 

Reed, Eric Shafer, Bruce Sowards, Hannah Thomas 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

6 
 

 

 

 

 

 

 

 

Executive Summary 

Multi-Method Study on Risk Assessment Implementation and Youth Outcomes in the 

Juvenile Justice System 

Risk assessment research is extensive, but tends to be concentrated in certain areas, namely 

the relationship between risk and official records of recidivism. As several important questions 

warrant consideration to offer insight on the usage and impact of risk and needs assessment with 

justice-involved youths, this research project builds on existing studies but seeks to focus more 

extensively on implementation and usage. To help answer these outstanding questions, the Office 

of Juvenile Justice and Delinquency Prevention (OJJDP) funded the University of Cincinnati 

Center for Criminal Justice Research (CCJR) to complete a project entitled the Multi-Method 

Study on Risk Assessment Implementation and Youth Outcomes in the Juvenile Justice System. 

This study pursued four main objectives: 

1. Describe and assess risk and needs assessment usage and implementation practices at 

different juvenile justice decision-points (i.e., intake/diversion, detention, disposition, 

residential intake, and reentry), using a multi-state sample of sites at different stages of 

adoption, to develop recommendations on best practices in training, monitoring, and 

usage. 

2. Assess court and programming decision-making outcomes based on variation in risk 

and needs assessment usage and implementation practices across agencies and states. 

3. Evaluate how the implementation of risk and needs assessments across multiple stages 

of the juvenile justice system impacts recidivism among juvenile offenders. 

4. Evaluate justice-based and developmentally relevant youths’ outcomes based on

variation in assessment-based decisions that reflect the usage, monitoring, and 

implementation of standardized risk and needs assessments. 

The project focuses on the implementation of one particular juvenile risk and needs 

assessment (JRNA), the Ohio Youth Assessment System (OYAS) (Latessa, Lovins, & Ostrowski, 

2009; Lovins & Latessa, 2013), across different juvenile justice decision-points and at agencies at 

various stages in the implementation process, which permitted a more nuanced and specific set of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

analyses and subsequent recommendations. However, as they are based in broad and pervasive 

themes from the juvenile justice personnel views and agency practices, the results and 

recommendations also generalize to the state-, county-, or agency-wide adoption of other JRNAs. 

The report presents a varied set of analyses using comprehensive sets of data in line with 

the research objectives described above. These include early sections devoted to explicating key 

themes from juvenile justice personnel interviews and web-based surveys. Later sections of the 

report present findings from the analysis of the comprehensive sample of case records and youths’

follow-up interviews. In addition to the main analyses, the middle portion of the report contains 

overviews of several “usage” studies that help to illustrate important points in understanding risk 

and needs assessment “in action” in the juvenile justice system. Finally, the report reiterates some 

of the key findings of the research project and identifies important data limitations to set an 

interpretive context around the results. The study’s concluding section then turns to a series of 

recommendations regarding JRNA training, usage, monitoring, and future evaluation and research 

based on the synthesis of key findings from the study. 

Study Methods 

The research team interviewed juvenile justice personnel at various agencies across the three 

states in various stages of the implementation process. In-person interviews (N = 217) were 

conducted with personnel from 22 juvenile justice agencies. The agencies were purposively 

selected so that we could assess how the full suite of OYAS tools was being implemented and 

ensure a mix of different types of juvenile agencies and staff with various experiences with and 

views of the OYAS. In the sites, we also purposively selected personnel who had experience 

directly related to the OYAS. For example, through planning its implementation, administering 

the assessment, using its results to make juvenile justice-related decisions, or training other staff 

on the assessment. Interviewees included court (n = 37); correctional/secure treatment (n = 44); 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

probation (n = 105); parole (n = 14); and state-level (n = 17) personnel. An interview guide was 

developed in order to answer questions specific to implementation context, policies, and practices 

in each state. Additionally, the research team reviewed the policies and procedures documents for 

each of the three state-level administrative bodies in order to identify trends in these policies 

around assessment, and the facilitators and barriers to assessment implementation and use. 

To capture data from a more generalizable sample, a web-based survey on assessment use 

and practices was distributed to a everyone in who was registered as an administrator of the OYAS 

in each of the three states and other juvenile justice personnel who interacted with the results of 

the assessment (e.g., judges, magistrates) who were not included in the interview sample. The webbased surveys were sent to an email list of juvenile justice personnel via Qualtrics. Names and 

emails were provided by state-level administrative bodies in each state. In total, 1,013 surveys 

were completed resulting in a 32.4% response rate. Survey respondents included staff from court 

(n = 102), correctional/secure treatment (n = 132), probation (n = 592), parole (n = 7), state-level 

(n = 8), and individuals whose agency setting could not be inferred from their job title due to the 

anonymous nature of the survey data (n = 157) or whose job title was missing (n = 15). Though 

job roles varied, all personnel in the survey sample administered the OYAS and/or used its results 

to inform their decisions. Respondents who administered the OYAS were randomly assigned a 

vignette to score, which served as a measure of reliability in scoring relative to the manual. To 

triangulate certain findings, some of the questions from the semi-structured interviews were also 

included on the web-based survey. Special care was taken to analyze these particular results 

comparatively across the samples. 

To ensure adequate representation of youths at varying stages of the juvenile justice process, 

a large sample of youths was selected through stratified random sampling techniques. Youths who 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

9 
 

 

 

 

 

 

 

 

 

 

 

 

 

resided in State 1 or State 2 were eligible for selection if they were assessed in 2014 or 2015. 

Youths in State 3 were eligible if their assessments took place in 2013 through 2015 for juvenile 

corrections and 2016 or 2017 for juvenile court cases. The first stage of sampling involved 

selecting counties from which youth would be selected. Due to the large number of counties in 

State 1 and State 2, counties were stratified based on assessment usage and were randomly selected 

from each stratum to participate in the study. The number of counties in State 3 is limited relative 

to the other states and all counties were included. Youths within the selected counties were then 

stratified based on the type of assessment used. This process resulted in a sample of 6,222 youths 

being selected for the study. 

A smaller subsample of youths was randomly selected from the larger sample to participate 

in follow-up interviews conducted by research staff over the telephone. Similar to the process used 

to select the larger sample of youths, cases were stratified based on the year and tool used to 

complete the assessment. Research staff undertook an intensive location and follow-up procedure 

that led to a sample of 131, which was 9 percent of the 1,402 cases for which states and local 

agencies provided contact information. The effective interview response rate was 20.4 percent 

when factoring in only those cases for which research staff had some contact with the youth or a 

parent (e.g., this does not include cases with inaccurate or outdated contact information). 

Data and Measures 

Juvenile Justice Personnel In-Person Interview Data. The interviews, which lasted 

roughly 30 to 60 minutes, were conducted using a 55-question semi-structured interview that 

allowed for elaboration from the interviewees. The questions touched on themes relevant to JRNA 

implementation literature, including: agency and staff characteristics; approach to youth 

assessment; the OYAS implementation process; and youth assessment practices. Following 

grounded theory (Corbin & Strauss, 2008), the research team analyzed the open-ended questions 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

10 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

by assigning codes to words and phrases, and grouping them accordingly as patterns emerged. 

Codes were inductively assigned based on their latent content, which was done using qualitative 

data analysis software (QDA Miner). The quantitative analysis of the close-ended, standardized 

questions (i.e., rating scales, or “yes,” “no,” or “unsure” responses) was completed using SPSS.

Web-based Survey Data. The full survey contained 52 items, though the number of 

questions respondents were asked depended on their role in the agency and connection to the 

OYAS. The questions covered themes regarding personnel characteristics, assessment use and 

practices, and the implementation process. Those respondents who administered any of the OYAS 

tools were asked to score a randomly-assigned vignette. All analyses of the survey data (including 

some qualitative analysis of the open-ended responses) were completed using SPSS. These 

analyses followed the same general processes as the interview data. 

Youth Case Records. Data for the youths selected for the comprehensive assessment 

sample consist of official records provided by agencies in each state. For youths in State 1 and 

State 2, the research team created an annotated spreadsheet with instructions that was shared with 

agencies through a secure File Transfer Protocol (FTP) up/download process. The spreadsheets 

contained identifying and case information about the youths to aid agency staff in selecting the 

appropriate cases for data extraction. Agencies were asked to complete the spreadsheets by adding 

the requested information. State 3 provided their information on a hard drive which was physically 

transferred to a research team member. In a few State 1 and State 2 jurisdictions, research staff 

were granted access to case record management systems in order to extract the requested data or 

were provided larger amounts of data that were downloaded from such systems from which the 

requested data were identified and extracted. For the smaller subsample of youths selected for the 

follow-up interviews, contact information was also requested. This was collected in the same 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

11 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

manner just described. Once the requested data were received by the research team, they underwent 

an extensive cleaning process prior to being merged together for analysis. 

Follow-Up Survey with Youths. Youths selected for follow-up interviews were contacted 

by phone by research staff and asked to participate in the study. Youths over 18-years of age 

provided verbal consent for participation. Parents (Guardians) were asked to provide verbal 

consent for those under 18-years old and these youths were also asked to provide verbal 

assent/consent to participate. Youths participated in a semi-structured interview consisting of 46 

questions across eight content areas: education/employment; family, living arrangements, and 

neighborhood; peer associations; situational awareness; beliefs; substance use; contact with the 

criminal justice system and JRNA; and treatment services. Interviews typically took approximately 

30 minutes to complete, and youths were sent a $15 gift card to a restaurant to compensate them 

for their time. 

Analytic Procedures 

By design, the study reported here is based on various sources and types of data. The 

juvenile justice personnel interviews and survey data were coded and analyzed with a mix of 

quantitative and iterative, qualitative methods. The analysis of the case record and youth interview 

data proceeded in varied fashion depending on the objective of the particular analysis. We utilize 

varying analytic methods to answer questions under each of the objectives relevant to those sets 

of data. Those analytic methods range from basic descriptive methods and bivariate tests to convey 

information about our samples and subsamples utilized in particular analyses to several types of 

multivariate analyses. Multivariate models were usually based on logistic regression given the 

nature of many study measures. These models allowed for estimation of key relationships while 

conditioning on other factors that could plausibly impact the inferences from those analyses. 

Measurement models and path/mediation models were also utilized in answering some questions. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

12 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of Key Findings 

Juvenile Justice Personnel Interviews and Surveys. This portion of the study identified 

a solid foundation upon which juvenile justice agencies may build an effective and sustainable 

implementation process. We found that most personnel were satisfied with the OYAS and believe 

it benefits the agency and the youths that they work with. However, they perceive their own 

satisfaction to be higher than that of their peers, which may speak to organizational culture and the 

messaging around the OYAS. More generally speaking, personnel were in agreement that risk and 

needs assessments enhance fairness in the juvenile justice decision-making process. Personnel also 

perceived the OYAS to provide useful information regarding criminogenic needs, but not about 

non-criminogenic needs. In general, agencies used the assessment information that was gathered 

in important ways (e.g., establishing supervision levels), but did not optimize its use (e.g., examine 

aggregate OYAS data). 

Validation was an important aspect to establishing buy-in among staff for the accuracy of 

the tool. While two out of the three states had the OYAS validated on their specific population, 

personnel saw numerous limitations of the OYAS that they perceived to lead to the need for 

overrides. Further, issues of validity and reliability were some of the most commonly cited 

limitations in the OYAS across each state. This perception on behalf of many personnel had 

implications for their buy-in for the tool and usage of its information. Most personnel indicated 

that they were told the reasons that their agency adopted the OYAS, but did not feel like specific 

steps were taken to establish buy-in. Training was cited as the main strategy for garnering buy-in 

among the staff. Additionally, formal quality assurance measures were decidedly absent. This may 

contribute to a lack of sustainability in the implementation process, as a lack of quality assurance 

processes reduces the quality of the information gathering process, and therefore also limits the 

usefulness of the information for juvenile justice decisions. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

13 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Specific to the web-based surveys, OYAS administrators consistently provided more 

negative views than non- administrators of the assessment, its usefulness, and its impact on their 

jobs. Despite providing slightly more negative views of the OYAS than non-administrators, OYAS 

administrators still tended to express positive overall sentiments of the assessment and its impact 

on their agency and the youths they work with. The scoring of vignettes in the web-based survey 

found some concerns with the accuracy of OYAS scoring. Only half OYAS administrators (n = 

228) scored their assigned vignette correctly (52.2%). This may feed directly into the high level of 

concern reported concerning reliability and validity of the OYAS. 

Youths Case Records and Follow-Up Survey with Youths. In addition to implementing 

risk and needs assessments, the study also investigated how the information was used in ways that 

could impact the experiences youths have in the juvenile justice system. Modern assessments such 

as the OYAS are designed to inform decision-making, which is assumed to improve outcomes for 

youths. With regard to dispositions, the results indicated that when risk levels based on OYAS 

scores are used to match youths to dispositions the likelihood of the youths recidivating is 

impacted. For example, when state commitment relative to non-custodial sanctions is used with 

higher risk youths, a small negative effect on the likelihood of recidivism is observed. Other results 

illustrate complex relationships among risk, disposition/placement, and recidivism that are 

informative about the relationship between risk assessment and justice decisions. 

In the analyses investigating the use of domain scores to match youths to appropriate 

treatment, the findings were generally inconsistent with what would be expected based on theory 

underlying JRNA. Overall, participation in various interventions was not associated with 

reductions in recidivism, or significantly increased the odds of youths having a new adjudication. 

When domain scores were examined in relation to the treatment youths received, scores in the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

14 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Prosocial Skills domain were most consistently related to treatment decisions. In other words, 

higher scores on this criminogenic need area increased the odds that youths would receive a 

treatment intervention. There was little evidence to suggest that the relationships between 

criminogenic needs and recidivism are mediated through various treatment options, however. 

Many of the self-reported outcomes for the youths were positive when looking at the 

various attitudinal scales measured during the follow-up interviews. Most of the 131 youths were 

in school or working at the follow-up point as well. These results did not vary much across the 

different initial OYAS risk level groups. Youths frequently received treatment during the juvenile 

justice process or follow-up period, and the moderate and higher risk groups tended to report 

greater involvement in treatment across most of the categories included in the interview protocol. 

Still, despite some positive results in self-reported attitudes, interviewees also reported a good deal 

of later contact with the juvenile and criminal justice system and many self-reported drug or 

alcohol use. There are some potentially useful interrelationships among the various self-report 

responses that are investigated in exploratory analyses. 

Usage Studies. In addition to the central analyses, the breadth of the data allowed us to 

carry out six subsidies to deepen the understanding of more nuanced aspects of the juvenile justice 

process. Usage study 1 evidenced that individuals reporting more favorable views of the OYAS 

scored the web-based survey scoring vignettes more accurately than those with less favorable 

views of the OYAS. In usage study 2, assessment classifications varied significantly among sites 

in our sample of juvenile court jurisdictions and Non-White youths were significantly more likely 

to receive a higher risk classification than their White counterparts. Usage study 3 identified some 

item, domain, and overall risk score performance differences across race and ethnicity, suggesting 

the need for further consideration of invariance across those subgroups of youths. In usage study 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

4, we discovered that while overrides are allowed by most agencies, only 3.4 percent of 

assessments were overridden, and almost always in the upward direction. Usage study 5 revealed 

that youths who were noted as having mental health or substance use needs were significantly 

more likely to receive treatment in these areas. Finally, in usage study 6, we determined that the 

optional strengths and barriers included on the OYAS tools were being used with some regularity 

and that these were noted at higher rates for higher risk youth and youth who progressed further 

into the system. 

Discussion 

The results of the current report help to further set the stage for the effective and sustainable 

implementation of JRNA. A set of recommendations based on these results are provided to identify 

some potentially effective strategies for moving forward. Despite the comprehensive and specific 

nature of these results and recommendations, there are still contextual elements to which those in 

charge of planning and carrying out the implementation must pay special attention, and adjust their 

strategies accordingly. Namely, state (and county) context matters for implementation. Those in 

leadership are advised to take heed of state and county initiatives that are in place and could help 

to ‘embed’ the assessment process and its information into what is already being carried out. To

that end, risk and needs assessment implementation does not occur in a vacuum, and therefore 

there will always be a residual impact of prior initiatives on the general reception of the tool. These 

elements will differ between localities and should be considered prior to full implementation. 

At this stage of their evolution, JRNA are often not implemented onto a “blank state” as

personnel and agency administrators may already have other experiences with assessment tools 

and other research-to-practice initiatives. The steps taken prior to, during, and after the initial 

rollout of a risk and needs assessment will set the tone for its reception. Formal quality assurance 

practices will ensure fidelity to the assessment, as well as sustainability in its implementation over 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

16 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

time. Overall, policies, training practices, automated systems functioning and integration, and 

assessment information usage practices will vary from agency to agency, and personnel 

perceptions of the assessment will be impacted uniquely by these conditions. 

Aside from state and local context, personnel characteristics are also relevant to 

implementation. Personnel using risk and needs assessments work in many different agency 

settings, have unique job roles, varying degrees of tenure in the field, and varying current and past 

experience with risk and needs assessments. Staff in leadership positions are encouraged to 

examine these characteristics, and consider the ways in which this may impact their perceptions 

(e.g., Have the personnel bought-in? Have they ever used any assessments before? Do they believe 

that the assessment is accurate and its information useful? How will their daily work be impacted 

by the tool? Are they able to seamlessly use and share this information? Are the trainings tailored 

specifically to them?). All of these characteristics can impact their level of buy-in with the tool, as 

well as their propensity to use and share the assessment information effectively. 

The fourth objective of the study involved evaluating justice-based and developmentally 

relevant youths’ outcomes based on variation in assessment-based decisions that reflect the usage, 

monitoring, and implementation of standardized risk and needs assessments. The result from the 

juvenile justice record portion of the analysis suggested that risk level and juvenile court 

dispositions have an impact on youths’ recidivism and these are generally in ways anticipated by

prior research and the underlying logic of JRNA. For example, when state commitment was used 

appropriately, it had a slight negative effect on the likelihood of recidivism for higher risk youths. 

In other instances where there was less alignment of risk level and disposition there tended to be 

increased recidivism, suggesting that supervision and other placement decisions may likely to lead 

to less desirable outcomes in those instances. We also examined processes that are expected to 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

17 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

occur around the domain level needs information from the OYAS (e.g., education and 

employment, substance use). JRNA processes should provide information that can be used to 

strategically match youths to treatment and services that address key needs, which in turn is 

intended to reduce recidivism. The results of the analysis generally suggest that matching 

criminogenic needs to treatment does not consistently occur in the juvenile justice systems that we 

studied as much as anticipated based on the underlying logic of JRNA. Case management and 

treatment matching is a key objective of contemporary risk and needs assessment, but these results 

suggest a need for more focus on this in future research and development of practice. It also 

requires further consideration of how risk and needs are balanced in processing and treating justice 

involved-youths. In general, this formal process-based modeling helps to better illustrate the 

manner in which different elements of measured risk and needs might impact eventual case 

outcomes. 

Although limited some by data concerns, the follow up interviews identify some insightful 

relationships pertinent to how the juvenile justice process can utilize the assessment information 

collected early on to inform decisions that impact positive attitudes and other developmental 

outcomes later on—effectively breaking the relationship between risk and later poor outcomes. 

This is illustrated in part based on the procriminal attitudes example that was explored in order to 

illustrate some tentative relationships among JRNA, treatment, and different categories of youth 

outcomes. It is, however, that the assessment process is viewed as part of a series of sequential 

steps where intermediate outcomes, like change in values or prosocial efficacy, are measured and 

responded to strategically throughout. 

Conclusion 

The results of this study should help to build processes for improving the quality and use 

of JRNA. As a state or agency is rolling out new assessment practices, or looking to improve their 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

18 
 

 

 

 

 

 

 

 

 

 

 

 

 

current risk and needs assessment processes, they should consider key implementation and 

sustainability facilitators: careful planning that includes establishing support for the risk and needs 

assessment amongst a variety of stakeholders, creating realistic but detailed usage and 

implementation guidelines and policies and procedures, sharing information with all those who 

will be impacted by the use of the assessment, sufficient training and post-training support, and 

beneficial quality assurance practices that will help ensure the risk and needs assessment is 

completed correctly and used to its full potential. 

The analysis of case record data across three states and dozens of agencies provide greater 

insight on the ways in which risk assessment information is used in practice. In some cases that 

use fit with the underlying logic of assessment practice (and existing research); however, in other 

instances the evidence that the tools were being used as intended was not as clear. The youth 

follow-up interview data help to reinforce the importance of focusing a portion of the discussion 

of JRNA on outcomes besides official recidivism. The questions asked on the interview serve both 

as endpoints for more developmental juvenile justice in themselves and as important leverage 

points in attaining positive justice-related goals (i.e., reductions in recidivism). Collectively, the 

findings from the study—both supportive and critical—offer some insight on how risk and needs 

assessment can be used as an engine to help generate better outcomes for youths and the juvenile 

justice agencies whom they encounter. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

19 
 

 

 

 

 

 

 

 

 

 

 

 

 

Final Report Introduction and Overview 

Juvenile justice system processing requires a series of discretionary decisions about youths 

who have become involved with juvenile courts and, subsequently, corrections agencies. Like the 

adult criminal justice system, juvenile courts and corrections have largely moved from reliance 

solely on unstructured, professional judgment to relying more on structured assessment 

processes—or a blend of the two (Shlonsky & Wagner, 2005)—as a precursor to making those 

decisions. Referring to juvenile probation systems, Wachter (2015) reports that more than twothirds of U.S. states have now adopted standard statewide assessments and many others have 

regional or local assessment instruments. He goes on to mention that risk assessments are often 

“the foundation of evidence-based practices, enhancing efforts to treat offenders, reduce 

recidivism, and ultimately increase public safety.” More broadly, proponents of the risk-needsresponsivity model suggest that effective treatment requires systematic assessment as a first step 

(Bonta, 2002). Presumably, this heightens the importance of assessment in the juvenile justice 

process as it can have cascading effects on a range of decisions by court and correctional 

personnel—and therefore on youth outcomes. 

The implementation of juvenile risk and needs assessments (JRNA) and associated 

procedures and policies require increased attention to ensure that they are used effectively in 

practice. These tools frequently contribute to more systematic decision-making, but several 

dimensions of related policy and procedure require further examination and analysis to ensure that 

the processes adopted and implemented are effective, efficient, and fair when applied in the 

juvenile justice system. This is essential as a starting point as juvenile justice seeks to become 

more developmental in its approach to youths’ sanctions and treatment (National Research 

Council, 2013; Scott & Steinberg, 2009; Sullivan, 2019). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

20 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This research project builds on existing studies of JRNA, but seeks to focus more 

extensively on implementation and usage according to the logic of how the risk and needs 

assessment process is meant to work from an information-gathering and decision-making 

standpoint. That process assumes that practitioners (1) have access to a validated tool; (2) they 

accurately use this tool to gather youths’ information; (3) subsequently use that information to 

make decisions about disposition and treatment; and, finally, (4) this process results in better 

juvenile justice and developmental outcomes (e.g., new referrals and placements and school and 

work engagement) (see Figure 1). 

Figure. 1. Logic Model of Risk and Needs Assessment 

Access provided to 

a validated risk and 

needs assessment 

tool 

Tool used to gather 

information and 

evaluate risk and 

needs 

Information 

gathered informs 

dispositional/ case 

decision-making 

(e.g., referrals) 

Youth receives 

more appropriate 

sanctions/treatment 

Reduced 

recidivism and 

better case/ 

personal/ 

developmental 

outcomes 

1 2 3 4 5 

This study examines these sequenced stages by collecting and analyzing data from juvenile 

justice personnel interviews and surveys, youths’ records spanning various juvenile justice system 

involvement, and follow-up interviews with selected youths. Three states that have utilized the 

same risk and needs assessment tool, the Ohio Youth Assessment System (OYAS) (Latessa, 

Lovins, & Ostrowski, 2009; Lovins & Latessa, 2013): Arizona, Indiana, and Ohio were included.1 

Data were collected from four samples: (1) purposively-selected juvenile justice agencies (N = 

22)2 and personnel within them (N = 217); (2) a larger sample survey of individuals who conduct 

OYAS assessments or who use the information that results from the assessments (N = 1,013); (3) 

1 To protect the confidentiality of the states, counties, and sites that contributed to this research project, their names 

are henceforth de-identified. 2 A previous publication (Vincent et al., 2018)stated that 23 agencies were included. During final coding checks, one 

individual was reassigned to a state agency affiliation as opposed to a distinct site. This did not materially affect 

findings presented in that research bulletin. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

21 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

a comprehensive sample of youths who had assessments conducted in selected sites between 2013 

and 2017 (N = 6,222); and (4) a subsample of 131 youths who participated in detailed follow-up 

interviews. We then carried out various analyses to answer questions pertinent to the adoption, 

implementation, usage, and outcomes of JRNA. 

Brief Background for Project 

The Evolution of Risk and Needs Assessment. The development of risk assessments can 

be viewed as spanning four generations (Bonta, 1996). First generation risk assessments are 

clinical in nature and rely on the professional judgment of counselors and case managers to 

determine the offender’s dangerousness, supervision level, and treatment needs (Van Voorhis & 

Salisbury, 2013). Attempting to reduce some of the subjectivity surrounding first generation 

assessments, second generation assessments are based on empirical research but are limited as they 

focus predominately on criminal history items. While these historical factors are predictive of 

recidivism, they are static (i.e. cannot be changed), which does not correspond well with 

intervention efforts aimed at addressing delinquent behavior (Gendreau, Little, & Goggin, 1996). 

Incorporating both static and dynamic (i.e., factors that can be targeted for change) factors, third 

generation assessments include youths’ risk factors and needs. Following the logic described

above, modern fourth-generation risk and needs assessments continue to examine youths’ risk and

needs, while also identifying responsivity factors and incorporating case management strategies to 

better inform intervention efforts (Andrews, Bonta, & Wormith, 2006). 

The Status of JRNA. With this evolution, a variety of risk and needs assessments can now 

be tapped to systematically gather information on youths. Many of these assessments have been 

the subject of predictive validity evaluations that support their use. Validated assessments include 

the Youth Level of Service/Case Management Inventory—YLS/CMI (e.g., Flores & Travis, 2004), 

the Structured Assessment of Violence Risk in Youth—SAVRY (e.g., Childs et al., 2013), the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

22 
 

 

 

 

 

 

 

 

 

Youth Assessment and Screening Instrument—YASI (e.g., Orbis Partners, 2007), and the Positive 

Achievement Change Tool—PACT (e.g., Barnoski, 2004). Additionally, some states and local 

jurisdictions have developed their own empirically-based risk and needs assessments. Examples 

of these tools include the Ohio Youth Assessment System—OYAS (e.g., Lovins & Latessa, 2013) 

and the North Carolina Assessment of Risk—NCAR (e.g., Schwalbe, Frasier, & Day, 2007). 

Many of these risk tools rely—at least in part—on the “Central Eight” risk factors

established by Andrews and Bonta (2010). These factors comprise juvenile justice history, 

personality attributes, antisocial peers, antisocial attitudes, family relationships, employment and 

education, and substance use. The vast majority of research in this area indicates that using 

measures derived from the Central Eight will improve risk prediction and facilitate effective 

intervention. For example, Flores and Travis’ (2004) comprehensive review of the YLS/CMI 

found that the Central Eight were significantly related to recidivism when measured at various 

points in the juvenile justice system (e.g., probation, detention, and institutional commitment). 

Similar results were reported in other YLS/CMI studies (e.g., McGrath & Thompson, 2012), as 

well as research on the OYAS (Latessa et al., 2009). 

In addition to the individual studies supporting the predictive validity of particular risk and 

needs instruments, recent meta-analyses have confirmed their value as well (e.g., Olver, Stockdale, 

& Wormith, 2009; Schwalbe, 2007). For example, Schwalbe’s (2007) meta-analysis reviewed 42 

studies that examined the predictive validity of 28 risk assessments. His findings suggested that 

the majority of the tools used by juvenile justice agencies were predictive of recidivism. 

Specifically, the overall effect size for juvenile risk assessments (r = .25) was moderate and similar 

to effect sizes estimated by meta-analyses of adult risk assessments (e.g., Gendreau et al., 1996). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

23 
 

 

 

 

 

 

 

 

 

 

 

Variability in Findings. This recent work demonstrates that (1) risk assessment has 

evolved considerably in recent decades and (2) a variety of instruments may be useful in informing 

juvenile justice case decisions. Still, studies have found differences in the strength of predictive 

validity across different risk assessment tools, as well as variability across studies with the same 

instrument (e.g., Rossegger et al., 2013). As a bottom line, Schwalbe (2007) reported an “Area

Under the Curve” statistic of 0.64 in his meta-analysis of predictive validity studies (see Figure 2). 

This suggests that validated tools predict youths’ recidivism significantly better than chance (0.50), 

but the 0.36 proportion of missed predictions nevertheless shows that there is room for 

improvement in JRNA practice. 

Figure 2. Summary of Average Effect and Variability in Juvenile Risk Assessments 

1.00 

0.90 

0.80 

0.70 

0.60 

0.50 

Prediction Error 

0.64 

Null Effect=0.50 

0.40 

0.30 

Area Under the Curve (AUC) Effect Size 0.20 

0.10 

0.00 

Adapted from data in Schwalbe, C. (2007). Risk assessment for juvenile justice: A meta-analysis. Law and Human 

Behavior, 31, 449-462. 

There are several possible reasons for these findings. For instance, the variation might be 

due to the stage of the assessment and agency involved (e.g., intake, probation agency, residential 

facility), the resources available to the agency conducting the assessment, the type of youths served 

within that agency (e.g., females vs. males, whites vs. ethnic minorities) (Andrews et al., 2011), 

and if the assessment is being conducted correctly. Given the variation observed across individual 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

cases, agencies, juvenile justice personnel, and states, the question of “which risk and needs 

assessment tool is the most accurate and effective at predicting recidivism?” is not as important as 

“how do implementation processes and usage impact the effectiveness of risk and needs 

assessment in juvenile justice decision-making?”

The Importance of Implementation. Various challenges in moving research to practice 

have become more apparent as the juvenile and criminal justice systems have become increasingly 

evidence-based. Berman and Fox (2010) have compared evidence-based reforms in justice settings 

to new start-up businesses, which have limited probability of success. Amassing relevant research 

evidence is only one part of the process of effectively introducing policy to the field. And, given 

the challenges in even reaching that point, it surprisingly may be the easier part of the evidenceto-implementation process (Fixsen, Blase, Metz, & Van Dyke, 2009; 2013; Laub, 2016). To avoid 

problems in introducing such practices and stagnation following initiation, this “other half” of the

process must be considered carefully in assessing any juvenile justice practice. Going a step 

further, while implementation in the juvenile justice system inherently brings unique obstacles 

(Sullivan, 2019), risk and needs assessment and their usage is perhaps more challenging still as it 

does not merely necessitate instituting a program or policy, but rather requires shifts in 

information-gathering and decision-making among personnel in the juvenile justice system. Thus, 

introduction of these tools must overcome natural inclinations in human judgment and decisionmaking (see, e.g., Bell, Raiffa, & Tversky, 1988; Tversky & Kahneman, 1974) and the study and 

monitoring of this implementation process is even more important than is typically the case. 

The concept of implementation in risk and needs assessment refers to a variety of activities. 

These include staff training, the perceived value of the assessment by staff members and agencies, 

usage practices, and quality assurance processes (Vincent, Guy, & Grisso, 2012). Shortcomings of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

25 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

implementing risk and needs assessments in practice have been linked to reduced predictive 

validity estimates (e. g., Flores, Lowenkamp, Holsinger, & Latessa, 2006), as well as inappropriate 

decision-making in case management (e.g., Vincent et al., 2012). In a study of the predictive 

validity of the LSI-R, Flores and his colleagues (2006) reported that the predictive validity of the 

assessment tool was significantly associated with the fidelity of implementation in correctional 

agencies. That is, agencies with strong risk and needs assessment implementation processes in 

place (e.g., proper staff training) had better predictive validity. 

Moreover, Vincent et al. (2012) found that simply training staff on the value of risk and 

needs assessments did not guarantee appropriate case management decision-making. Rather, 

establishing agency policies regarding the use of assessments in case planning was needed. This 

necessitates a holistic view of both the assessment tool and decision-making process that depends 

on it (see also, Vincent et al., 2018). Vincent and colleagues (2012) further indicated that when 

juvenile probation offices properly implemented risk and needs assessment, it led to a reduction in 

secure placement rates and use of intensive community supervision, which allowed for better 

resource allocation throughout the system. This early implementation research reflects the growing 

understanding that developing scientific knowledge, as is the case with validated risk and needs 

assessment, is only one half of moving research to practice to improve individual case outcomes 

(Fixsen et al., 2009). 

Researchers have emphasized the need to more formally consider questions of 

implementation in the context of risk and needs assessment (e.g., Luong & Wormith, 2011; Seave, 

2011; Flores et al., 2006). Given the upstream role that risk and needs assessment and classification 

plays in informing later aspects of effective interventions, it is essential that this gap in the research 

be filled, as without proper implementation, agencies “will not be able to take full advantage of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

26 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

evidence-based practices” (Seave, 2011, p. 140), which will limit possible gains in reducing 

youths’ recidivism and promoting positive developmental outcomes. A growing body of research 

in other fields offers analysis and insight on the implementation process that should be useful to 

juvenile justice. Still there are unique features of juvenile justice processes and decisions—

especially in risk and needs assessment—that warrant focused attention to improve practice and 

youths’ outcomes. 

Description of the Ohio Youth Assessment System (OYAS) 

The OYAS (Latessa, Lovins, & Ostrowski, 2009; Lovins & Latessa, 2013) was developed 

based on the principles of effective intervention, and follows the risk, need, and responsivity 

(RNR) model of offender treatment (Andrews et al., 1990). It was developed by the University of 

Cincinnati Corrections Institute (UCCI) in conjunction with Ohio’s Department of Youth

Services.3 The state requested a tool that could be used by all juvenile justice actors across the state 

and benefit both the youths and the state as a whole. It is unique in that it is designed to 

appropriately measure criminogenic risk and needs at five stages of the juvenile justice system: (1) 

diversion, (2) detention, (3) disposition, (4) residential, and (5) reentry. The following offers a 

brief description of each tool: 

• Diversion Tool: The Diversion Tool is designed to assist juvenile courts in identifying 

youths who can be safely diverted from further formal processing in the juvenile justice 

system. This tool consists of 6 items and can be completed from a file review or face-toface interview in approximately 10 minutes. 

• Detention Tool: The Detention Tool is used with youths being considered for secure 

detention. It allows for low risk youths to remain in the community prior to court hearings. 

This instrument consists of 6 items and is completed through a brief face-to-face interview, 

which takes approximately 10 minutes. 

3 Two initial members of the research team were involved in the development and implementation of the OYAS tools, 

but neither contributed to the writing and analysis of the final report, its conclusions, or its recommendations. Further, 

the research staff for this study were not responsible for the training and implementation policies that were the central 

focus of this research. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

27 
 

 

 

 

 

 

 

 

 

 

 

 

 

• Disposition Screener: A short six item Disposition Screener is used to quickly identify 

low risk youths post-adjudication. The instrument is designed to be administered through 

structured interview and file review. If a youth is low risk on the screener, no further 

assessment is needed. If a youth is moderate/high on the screener then a full Disposition 

Tool is recommended. 

• Disposition Tool: For those youths adjudicated by the court, the Disposition Tool assesses 

their risk to reoffend, identifies important criminogenic needs, as well as barriers to 

treatment, and provides insight for case planning. It covers seven domains: criminal 

history, family, education/employment, prosocial skills, substance abuse/mental 

health/personality, peers, and antisocial attitudes. 

• Residential Tool: The Residential Tool affords residential programs the ability to 

accurately assess the youths’ level of risk and corresponding criminogenic needs. Similar 

to the Disposition Tool, the Residential Tool includes seven domain areas. 

• Reentry Tool: The Reentry Tool reassesses youths after an extended stay in a residential 

program. It is based on the same domains as the Disposition and Residential Tools but is 

scored based on the youths’ progress in programming. The Reentry Tool is used in 

conjunction with post-release supervision and treatment to ensure successful transition 

into the community. 

Information needed to score each of the tools comes from the same sources. For the shorter 

tools (i.e., Diversion, Detention, Disposition Screener), not all elements are required): case 

information and juvenile justice history records, a youth self-report form, a youth interview, and 

collateral information sources as needed (e.g., talking with parents, teachers, or treatment 

providers). Once the information is collected, the tool is scored by consulting a standardized 

manual. The total number of points is then linked to a risk level (low, moderate, and high), which 

is calculated differently for males versus females. For the longer tools (i.e., Disposition, 

Residential and Reentry), scores and levels for each of the domains are also calculated. Tools can 

be completed electronically or using paper copies and hand scoring. 

This system of assessment tools is comprehensive in scope and adapts to the various needs 

of the youths and staff responsible for decision-making, supervision, and intervention at the 

various stages of the juvenile justice system. Studies of the predictive validity of the OYAS have 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

28 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

been encouraging. For example, several studies (e.g., Labreque & Schweitzer, 2013; Lovins & 

Latessa, 2013; McCafferty, 2016; McCafferty, 2013) have confirmed that using different tools at 

each point in the juvenile justice system can provide a more accurate and comprehensive 

understanding of risk and needs levels for individual youth and that assessment results can be used 

to facilitate more effective decision-making on the part of juvenile justice officials. 

UCCI provides training in the OYAS and was responsible for training staff in the three 

states included in this study. A two-day in-person training with certification is required prior to 

being able to administer any of the OYAS tools. The two-day training includes information on the 

following: the principles of effective correctional intervention, actuarial assessment, the content 

and use of the OYAS, interviewing techniques, and hands-on practice opportunities. Specifically, 

trainees practice scoring the OYAS by watching multiple videos of interviews and completing 

individual and group scoring before reviewing scoring with the trainers. After these practice 

opportunities, certification includes a written and video test which involves scoring an OYAS tool 

based on a video of an OYAS interview. Passing these two tests requires scoring the video within 

three points of the “true score” (+/-) according to the OYAS scoring guide (which is available to 

the trainees at the time of the test), and scoring no more than five items incorrectly. It is important 

to note that all five tools are included in the two-day training, and once certified, staff are able to 

conduct all five tools. 

In order to build internal capacity, a five-day training of trainers (ToT) is also available. 

These trainings permit individuals to train additional personnel at their own agencies in 

administering the OYAS. In order to qualify for the ToT, staff are required to have administered 

at least 15 to 20 assessments prior to the ToT. The first three days are spent with the UCCI trainers 

practicing and delivering the training material. The last two days are reserved for the ToT 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

29 
 

 

 

 

 

 

 

 

 

 

 

participants to deliver the OYAS training to new users. After the ToT is completed, and newly 

trained trainers are classified as either a lead trainer, a co-trainer, or not certified as trainer. 

Certification and ToT trainings were provided to the three states in this study. 

Objectives and Research Questions 

Risk and needs assessment research is extensive, but tends to be concentrated in certain 

areas, namely the relationship between risk and official records of recidivism. Nevertheless, 

several important questions warrant consideration to offer insight on the usage and impact of risk 

and needs assessment with justice-involved youths. Given the background and existing literature, 

this study pursued four main objectives: 

1. Describe and assess risk and needs assessment usage and implementation practices at 

different juvenile justice decision-points (i.e., intake/diversion, detention, disposition, 

residential intake, and reentry), using a multi-state sample of sites at different stages of 

adoption, to develop recommendations on best practices in training, monitoring, and 

usage. 

2. Assess court and programming decision-making outcomes based on variation in risk 

and needs assessment usage and implementation practices across agencies and states. 

3. Evaluate how the implementation of risk and needs assessments across multiple stages 

of the juvenile justice system impacts recidivism among juvenile offenders. 

4. Evaluate justice-based and developmentally relevant youths’ outcomes based on 

variation in assessment-based decisions that reflect the usage, monitoring, and 

implementation of standardized risk and needs assessments. 

To meet these objectives, we carried out a comprehensive, multisite study of agencies 

implementing the same assessment system, the OYAS, to consider agency implementation in 

practice and its impact on juvenile justice decisions and youths’ outcomes.4 

4 For simplicity, we refer to the Ohio Youth Assessment System as the OYAS in the course of the report, but the 

secondary adopter states refer to the OYAS with their own state label. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

30 
 

 

 

 

 

 

 

 

 

 

Overview of Research Design and Data Collection 

The study comprised four separate streams of related, but distinct, data. Each of the key 

study components and analytic plans are described in the report where appropriate, but Figure 3 

provides an overview and this section briefly describes the multi-method research plan that was 

used to meet the four research objectives. 

We collected survey and interview data from relevant sites and personnel to describe and 

assess the implementation practices each state has adopted. In conjunction with the juvenile 

justice personnel data collection, we reviewed policies and procedures for implementing the 

OYAS to further characterize and contextualize implementation processes at multiple decision 

points across the three states and agencies within them. We then sampled and analyzed case records 

across juvenile justice decision points (e.g., diversion, disposition, reentry) and agencies (e.g., 

court, corrections) to link implementation and case and agency-level outcomes. This helped us to 

examine how variation in implementation practices affects system-based outcomes. Recidivism 

records were collected for large, stratified samples of cases in each state (N = 6,222) to examine 

case outcomes relative to the variation and nature of implementation at the agency level. The 

effective analytic sample sizes for case disposition and treatment matching modeling were 

approximately n=4805 and n=3584 records, respectively, due to measure availability (e.g., 

available recidivism data, collection of treatment information, knowledge of case disposition and 

outcome) and matching within records (e.g., linking assessment to disposition and recidivism 

records). Finally, we evaluated developmentally-relevant youths’ outcomes based on variation in

system-level decisions and implementation processes. A random subsample of cases that were 

assessed with an OYAS tool was drawn and those youths were then interviewed to obtain more 

extensive self-report, case outcome, and treatment receipt data (N = 131). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

31 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

- -

--

-

-

=

Figure 3. Overview of Design and Data Collection for Study 

JJ Personnel Interviews 

217 interviews in 22 

agencies in 3 states 

In depth, semi structured 

interviews (55 questions) 

Institutional and 

community corrections, 

courts, state 

administration 

Identified by state 

contacts based on study 

objectives 

Developed 

questions/themes from 

past research and current 

objectives 

JJ Personnel Surveys 

1,013 responses from 3 

states 

52 questions; policies, 

practices, usage, barriers 

and facilitators of 

implementation 

Scoring vignette 

questions for those who 

administer the tool(s) 

Distribution by 

researchers and states 

response rate was 32.4% 

Mostly fixed items, but 

some open ended 

questions 

Agency Case 

Records 

Data from 3 States and 

55 Counties 

Extraction from state and 

local record systems 

Stratified random 

samples based on 

assessment prevalence 

and site; total of 6,222 

cases 

Measures on assessment, 

case characteristics, case 

dispositions, and 

outcomes 

Utilize relevant cases for 

analysis of each decision 

point 

Youth Follow-Up 

Interviews 

Random subsample from 

Agency Case Records 

Contact information 

from agency records and 

comprehensive searches 

A total of 1,402 cases 

were available for 

contact 

N 131 completed 

interviews 

46 item interview; some 

items match OYAS 

domain areas/other 

standardized assessments 

Overview of Report Sections and Usage Studies 

This report is divided into three main substantive sections where we identify the specific 

methods used and results of relevant analyses for the different research objectives. Section I 

focuses primarily on implementation of OYAS in the three states and local agencies within those 

states (Objective #1) and therefore draws heavily on practitioner views based on data from juvenile 

justice personnel interviews and surveys. It then covers themes about interviewees’ and

respondents’ perceptions of the OYAS and risk and needs assessment more generally, usage of the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

32 
 

 

 

 

 

 

 

 

 

 

results of the risk and needs assessment process, views of the implementation process, and 

identification of facilitators or barriers in implementation and sustainability. Additionally, this 

section includes a usage study of the relationship between personnel support for the OYAS, and 

the integrity of the risk and needs assessment process. 

Section II of the report focuses on multiple analyses that tap into different aspects of risk 

and needs assessment usage—from basic questions such as whether risk scores and levels are 

related to case outcomes, to the degree to which the full potential of the process is being reached 

in making treatment decisions. This section predominately examines case record data, but will 

integrate juvenile justice personnel perspectives to contextualize and extend key findings where 

appropriate. The second section also includes a series of substudies (i.e., usage studies) that 

compare the usage of risk and needs assessment across sites to the objectives that the agencies 

(and researchers) had in mind when first promoting the use of structured risk and needs assessment 

(Objective #2). 

The third and fourth research objectives are covered in Section III of the report, which 

generally pertains to “outcomes” of the risk and needs assessment process via the decisions that 

are made by juvenile justice officials (Objective #3 and Objective #4). The final piece, Section 

IV, the summary and conclusions section, takes stock of the various findings pertinent to the 

research objectives. After identifying some limitations to help contextualize the findings, we 

present and discuss the key lessons learned from the study—focusing especially on 

recommendations for federal, state, and local agencies interested in improving the efficiency and 

effectiveness of juvenile justice decision-making and youths’ outcomes. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

33 
 

 

 

 

 

 

 

 

 

 

Section I. Implementation of Juvenile Risk and Needs Assessment: Views from Juvenile 

Justice Personnel 

As highlighted above, implementation research in JRNA is somewhat limited in its amount 

and scope at this point. Consequently, documenting the perspectives of those in the field is a main 

objective in establishing a sense of how to best move risk and needs assessment tools and processes 

to practice. This is likewise beneficial in identifying facilitators or challenges in implementation. 

This section seeks to cover themes about personnel perceptions of the OYAS and risk and needs 

assessment more generally, usage of the results of the risk and needs assessment process, views of 

the implementation process, and identification of facilitators or barriers in implementation and 

sustainability. Subsequently, a substudy entitled “Support for the OYAS and Integrity in the 

Juvenile Risk and Needs Assessment Process” explicitly examines the relationship of personnel 

attitudes regarding the OYAS, and the accuracy with which they score the tool. 

To describe and assess the implementation practices each state has adopted regarding the 

OYAS, the research team systematically collected survey and interview data from relevant sites 

and personnel. In addition, data on policies and procedures for implementing the OYAS was 

reviewed to characterize implementation processes at multiple decision points. All subsequent data 

and analyses in this section of the report are specific to the juvenile justice personnel interview 

and web-based survey portions of this study. 

Development of the Interview Guide 

The research team developed the semi-structured guide specifically to help answer the 

research questions in the proposal. Generally speaking, the goal of the juvenile justice personnel 

interviews was to gather information so that we could describe and assess OYAS implementation 

and usage practices in the three states. The goal includes identifying trends in agency policies and 

practices around assessment, facilitators to assessment implementation and use, and barriers to 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

34 
 

 

 

 

 

 

assessment implementation and use. These measures were created based on the general objectives 

of risk and needs assessment (e.g., increase consistency, facilitate assigning risk levels, resource 

management), the training processes involved with the assessment system used by the three states, 

and with prior implementation research (see, e.g., Aarons, Hurlburt, & Horwitz, 2011; Fixsen et 

al., 2009; Meyers, Durlak, & Wandersman, 2012). 

While the interview guide was geared to fit the objectives for this study, we also consulted 

known sources. Some interview (and web-based survey) items and scales were modified from 

three sources. First, the Survey for Probation and Parole Officers (Hubbard, Travis, & Latessa, 

2001) was used to help establish how the OYAS was being used. Second, the Community 

Organizational Assessment Tool (COAT) was used to inform themes about the planning and 

implementation process around JRNA (Pratt & Hernandez, 2003). Third, the National Criminal 

Justice Treatment Practices Survey (NCJTPS) (Taxman, Young, Wiersema, Rhodes, & Mitchell, 

2007) helped to identify aspects of the agencies and their climate conducive to implementation 

and sustainability. In combination, this knowledge was used to identify and refine concepts 

relevant to risk and needs assessment implementation. The interview guide was drafted during 

team meetings and then reviewed independently by each research staff member. Research team 

meetings then included using the drafted guide to practice the interview for timing and flow, and 

to ensure that the main questions sufficiently captured needed information. Finally, the interview 

was piloted at the first two site visits in State 1. Feedback from the interview teams was collected 

and then the interview guide was finalized. 

The interview guide contained 55 questions and was divided into several main areas: 

agency and staff characteristics (14 questions;); agency approach to assessment (12 questions;); 

the OYAS implementation process (10 questions;); and assessment practices (19 questions). The 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

35 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

questions were predominantly open-ended (e.g., “please describe your involvement with the 

OYAS”). Some questions were likert scale items (e.g., “on a scale of 1 to 5, with 1 meaning 

completely disagree and 5 meaning completely agree, please rate your agreement with the 

following statement”). Further, some questions were asked to elicit a “yes,” “no,” or “other” set 

response (e.g., every meeting, occasionally, seldom, never); however, interviewers probed every 

response for details. For example, one question, “Are staff formally trained on the OYAS?” was 

followed by “Please describe the process.” As a result, almost all of the items required coding for

emergent themes. The full interview guide can be found in Appendix A. 

Semi-Structured Interview Methods and Site Visit Process 

The research team worked with state agency personnel in each of the three states to 

determine the specific sites to visit. This was completed in a series of meetings and 

communications with each state, either in-person, over the phone, via email, or a combination 

thereof. The research team worked with the state agency personnel to ensure variation in the site 

selection process. First, in each state, we requested a mix of different types of agencies so we could 

see how the full suite of OYAS tools was being implemented and also to understand the potential 

varying experiences of those at different stages of the process and those at different types of 

juvenile justice agencies. In each state, we ensured that the state level agencies/actors were 

included as they frequently were involved in setting policy and championed adoption and 

implementation. We also ensured that there was at least one residential institution and one parole 

location in each state. The remaining sites were reserved for county-level agencies that included 

the juvenile court, probation, and in some cases, detention or secure treatment centers. Second, we 

selected a mix of sites concerning the use, experience, and support of the tool and assessment 

process. We specifically worked with the state agency personnel to include both early and late 

adopters in the sample. Third, we ensured that we captured sites in each state that are considered 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

36 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

urban, suburban, and rural, which also ensured variety in the size of the site (i.e., the number of 

staff employed, as well as the number of youths served by a given agency). After each site was 

identified, the state agency personnel linked the research team to a main point of contact at the 

site. Once the connection was made, a phone call was scheduled to explain the study and facilitate 

participation. All of the sites that were identified agreed to participate in the study.5 

During the call, the research team asked the contact to describe the site, the staff, and the 

use of the OYAS in the organization. During this call and subsequent calls and emails, staff were 

purposely selected for interviews. Staff that conducted the assessments, used the assessment results 

to make decisions, or used the OYAS information in their daily work were prioritized for 

interviews. In smaller sites, all staff that fell into this category were interviewed. In larger sites, 

however, we relied on the contact to identify staff to interview. In these instances, the research 

team asked the site to ensure sufficient participation in different ways. To illustrate, for positions 

that were unique or specific to the OYAS, we made sure to interview those individuals. For staff 

titles that had multiple positions, we worked with the site contact to interview a representative 

cross section of staff members who had the same job title. For example, if there were six units in 

probation department, we interviewed a mix of managers, supervisors, and line staff across the 

units. In courts with multiple juvenile judges and magistrates, we interviewed at least one from 

each category. In treatment programs that had multiple clinicians, we interviewed approximately 

half of them. Through this process, we ensured that staff in different roles and with varying 

exposure to the assessment were interviewed—so as to gather responses from the spectrum of 

experiences and routine exposure to the assessment. The variety of site staff that were interviewed 

5 This is likely due in part to the purposive nature of the sampling strategy as well the guidance of state-level officials 

with respect to which sites may be best for the study. Consequently, there may be some selection artifacts in the 

participating sites. Still, the sites required varying degrees of negotiation, including questions-and-response from the 

research team, before final permission was granted. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

37 
 

 

 

 

 

 

 

can be seen in Table 1.1. Interviews with relevant juvenile justice personnel were semi-structured 

and consisted of questions about a series of topics discussed in a conversational fashion (Patton, 

2001). Depending upon the role of the staff being interviewed, interviews lasted 30 minutes to 120 

hours, with the majority of interviews falling within the 45 to 60 minute range. 

Site Visits. In addition to the one-on-one, in-person interviews, reviews of relevant 

documents and policies were conducted during the on-site visits by research staff who had training 

in these forms of data collection. Site visits ranged from one to three days in length. The size of 

the teams varied from two to five research staff. Table 1.1 outlines the characteristics of the 

interview sample, including the state where interviews were conducted, the agency name and type 

(e.g., court, probation, detention, state), number of interviews conducted at a given site, and the 

types of staff that were interviewed. The state and site names have been given pseudonyms (e.g., 

State 1, State 2, State, 3, Site A, Site B, Site AA, Site BB, Site AAA, Site BBB) to protect the 

confidentiality of states, agencies, and personnel interviewed. 

Table 1.1. Interview Sample Description: Agency Types, Number of Interviews, and Staff Titles 

State Site Agency 

Type 

Number of 

Interviews 

Types of Staff Interviewed 

State 1 (n = 71) 

Site A State 4 Deputy Director of Facility Operations and 

Programs 

Director of Parole 

Quality Improvement Administrator 

Management Analyst 

Site B Community 

Correctional 

Facility 

4 Director 

Clinical Coordinator 

Residential Care Staff/Training Coordinator 

Residential Care Supervisor 

Site C Court 

Probation 

Detention 

Family 

Resource 

Center 

23 Judges/Magistrates 

Defense Attorneys and Prosecutors 

Chief Probation Officer 

Court Intake Officers/Supervisors 

Placement Coordinator 

Special Programs Administrator 

Probation Officers/Supervisors 

Case Managers/Supervisors 

Aftercare Reentry Specialist 

Detention Officers/Supervisors 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

38 
Table 1.1. Interview Sample Description: Agency Types, Number of Interviews, and Staff Titles 

State Site Agency 

Type 

Number of 

Interviews 

Types of Staff Interviewed 

Mental Health Social Worker 

Site D Court 

Probation 

Detention 

Intervention 

Center 

Residential 

Treatment 

Center 

14 Magistrate 

Probation Officers/Supervisors 

Grant Coordinator/Planner 

Clinical Intake Coordinator 

Case Manager 

Treatment Coordinator 

Assessment Specialists 

Mentoring Coordinator 

Family Specialists 

Site E Parole 

Residential 

Treatment 

Center 

8 Regional Administrator 

Regional Director 

Probation Officers/Supervisors 

Operations Manager 

Site F Court 

Probation 

9 Judge/Magistrate 

Court Director 

Juvenile Probation Director 

Intake and Assessment Officers 

Probation Officers 

Diversion Officers 

Site G Institution 9 Superintendent 

Deputy Superintendent of Programs 

Psychology Assistants/Supervisors 

Social Workers/Supervisor 

Unit Manager 

Youth Specialist 

State 2 (n = 45) 

Site AA Institution 5 Program Director 

Psychiatric Social Service Specialists 

Intake Specialists 

Site BB Intake 

Facility 

4 Program Director 

Psychiatric Social Service Specialists 

Intake Specialists 

Site CC State 4 Executive Director 

Program Director 

Reintegration Specialists (i.e. parole officers) 

Site DD Court 

Probation 

Detention 

Reporting 

Center 

14 Judge 

Magistrate 

Chief PO/Superintendent 

Assistant Chief PO 

Probation Officers/Supervisors 

Site EE State 4 Executive Director 

Deputy Director 

Staff Attorney 

OYAS Trainer 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

39 
Table 1.1. Interview Sample Description: Agency Types, Number of Interviews, and Staff Titles 

State Site Agency 

Type 

Number of 

Interviews 

Types of Staff Interviewed 

Site FF Court 

Probation 

Treatment 

8 Deputy Chief Probation Officer 

Intake Supervisor 

Surveillance Officer 

Probation Officers/Supervisors 

Youth Services Executive Director 

Program Coordinator 

Clinical Director 

Clinical Case Manager 

Site GG Court 

Community 

Corrections 

6 Judge 

Director of Court Services 

Juvenile Services Coordinator 

Probation Officers/Supervisors 

Administrative Assistant 

State 3 (n = 101) 

Site AAA State 2 Juvenile Automation Manager 

Operations and Budget Manager 

Site BBB State 3 Community Corrections Bureau Administrator 

Parole Administrator 

Research, Development, and Planning 

Administrator 

Site CCC Parole 5 Parole Supervisor 

Parole Officers 

Family Services Coordinator 

Site DDD Intake 

Facility 

5 Reentry Chief Administrator 

Youth Program Officers 

Treatment Psychologist 

Assessment and Classification Coordinator 

Site EEE Court 

Probation 

Detention 

12 Judge 

Chief Probation Officer 

Treatment Coordinator 

Detention Services Director 

Detention Officers 

Lead Deputy Probation Officer 

Probation Officers/Supervisors 

Intake Probation Officer 

Diversion Officer 

Site FFF Court 

Probation 

Detention 

18 Director 

Division Director 

Probation Officers/Supervisors 

Intake Officer 

Pre-Disposition Report Writer 

Family Court Coordinator 

Director of Youth Justice Center 

Judge 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

40 
 

 

 

 

 

 

 

 

Table 1.1. Interview Sample Description: Agency Types, Number of Interviews, and Staff Titles 

State Site Agency 

Type 

Number of 

Interviews 

Types of Staff Interviewed 

Site GGG Court 

Probation 

Detention 

13 Judge 

Director 

Probation Officers/Supervisors 

Diversion Officer 

Site HHH Court 

Probation 

Detention 

43 Chief Probation Officer 

Research and Planning Director 

Management Analyst 

Staff Development Officer 

Probation Officers/Supervisors 

Treatment Officers 

Surveillance Officers 

Diversion Officers 

Judges/Commissioners 

Defense Attorneys and Prosecutors 

Public Advocate 

Semi-Structured Interview Notes and Analytic Process. As shown in Table 1.1, between 

June 23, 2015 and September 15, 2016 the University of Cincinnati research team met with 22 

different agencies and conducted a total of 217 semi-structured interviews throughout State 1, State 

2, and State 3. These data, along with state and agency documents (e.g. supervision standards, risk 

and needs assessment policies at the local and state level, OYAS-specific practice manuals), were 

analyzed in order to identify trends in agency policies and practices around assessment, facilitators 

to assessment implementation and use, and barriers to assessment implementation and use. The 

analysis took place in four phases, beginning with interviewing. The three subsequent phases 

analyzed data and reported findings at the site level, the state level, and overall with the last phase 

being represented by the current report. Those phases are summarized in Figure 1.1. 

During phase one of the analysis, research staff recorded detailed notes on interview 

responses (tape-recording was not used). Interviews were given an ID number, scanned, 

transcribed, and aggregated into one Excel document on a secure server for further review. 

Transcription occurred as soon after the interviewing process as possible. This generally occurred 

within eight weeks or sooner—depending on the number of interviews associated with a particular 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

41 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

site visit. The original interviewer was asked to resolve any discrepancies during the transcription 

process. The Excel file was then uploaded into QDA Miner 5 software for qualitative analysis. 

Following a grounded theory approach (Corbin & Strauss, 2008), the research team reviewed every 

open-ended question response individually and assigned codes to words and phrases and grouped 

them accordingly as patterns emerged. Codes were created inductively and assigned based on the 

meaning of the text, rather than on the frequency of characters/words (i.e., latent content). Lastly, 

the interviews were also uploaded into SPSS for quantitative analysis of the close-ended, 

standardized questions (i.e., rating scales, or “yes”, “no”, or “unsure” responses).

In phase two, staff grouped data into site-level reports, paying particular attention to trends 

or common features that generally define assessment policies, practices, and use, as well as those 

that are markers of implementation facilitators, barriers, or current or previous processes. 

Representative quotes and data clusters were used to illustrate and contextualize the key themes 

within this report and in subsequent reports. Grouping data facilitated some basic quantification of 

trends across and within sites (e.g., a certain percentage of interviewees made reference to a 

particular “limitation” or “strength” of the OYAS). These site-level reports were independently 

reviewed by all of the staff who conducted personnel interviews at the site. Staff feedback from 

independent review was collected and integrated into the site-level reports. The key themes in 

these reports provided structure for the descriptive analysis framework for subsequent reports. This 

process also identified relevant themes and observations for further interrogation in later analyses. 

In phase three, themes identified across site-level reports were conceptualized into statelevel reports, illustrated by representative quotes and data clusters. State-level reports were 

independently reviewed by four to five staff members, those of whom had participated in the initial 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

42 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

data collection phase and/or had reviewed the site-level reports. Staff feedback was collected and 

integrated into state-level reports. 

In the fourth and final phase of the qualitative data analysis process, the findings from each 

state-level report were compared and summarized here. This section of the final report discusses 

the overall findings in order to address the three research objectives within the context of the larger 

research project. Those sections of each report were also independently reviewed by the same staff 

members, those of whom had participated in the initial data collection phase and/or had reviewed 

the site- and state-level reports, and their feedback was integrated into this report. This included 

revisiting key themes to double check consistency and examples before they were used to reach 

conclusions about perceptions of risk and needs assessment use and implementation, obstacles to 

use and implementation, and facilitators of use and implementation. These, coupled with the 

results of the web-based survey (discussed below), led to a majority of the recommendations 

developed later in the report. A summary of the phases of the analytic process for the interview 

data is provided in Figure 1.1 below. 

Figure 1.1. Phases of the Analytic Process for the Interview Data 

• Transcribe • Quantify trends site themes into conclusions 

interview notes and within sites state-level reports •Revisit and check 

resolve • Draft and review • Staff review of themes 

discrepancies site reports state-level reports • Extract conclusions • Qualitative and •Research staff • Quantify trends and make 

quantitative review of site-level across sites recommendations 

analyses reports 

Juvenile Justice Personnel Web-Based Surveys 

In order to capture information about the use of the OYAS from a larger, more 

generalizable sample, a web-based survey on assessment use and practices was distributed to staff 

who were not included in the interview sample. The sample for the web-based survey contained 

Data Collection & Site-Level Reports State-Level Development of 

Phase 1 Phase 2 Phase 3 Phase 4 

Initial Analyses Reports Final Report • Gather 

•Conduct interviews representative •Identify themes • Synthesis of 

and record detailed quotes and data across site reports previous steps 

notes on responses clusters • Summarize cross- •Review of analytic 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

43 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

individuals who administer the assessment and/or use the assessment results to inform decisions 

regarding case disposition and treatment. The lists of individuals were generated by the state 

agencies in each of the three states and came from two primary sources: those individuals who 

were trained to conduct the OYAS assessments and those who were asked to use the information 

generated from the assessments. The sampling frame for the web-based survey included a variety 

of juvenile justice personnel (e.g., judges, supervisors, probation officers, case managers, 

treatment staff) across different types of agencies (e.g., secure detention or treatment facilities, 

probation and parole agencies, community-based service providers, and court-based settings).6 

The web-based survey contained items regarding assessment policies, practices, and usage 

that may serve as facilitators or barriers to implementation. There was a total of 52 items on the 

survey, but the number of questions answered depended on the respondent’s role in the agency

and connection to the OYAS. Skip patterns were programmed into the survey to streamline items 

that did not apply to a respondent. For example, those who indicated that they personally 

administer the assessments were asked to complete more items than those who indicated they do 

not administer the assessments. There were four sections of the survey, which included 1) general 

questions concerning assessment use and practices, 2) implementing the OYAS, 3) applying the 

assessment information (1 randomly assigned scoring vignette), and 4) staff demographic 

information. The complete survey can be found in Appendix B. 

The third section of survey, labeled “Applying the Assessment,” asked staff who indicated 

that they administer the assessment to complete one of three randomly-allocated vignettes. These 

were scoring vignettes based on the Juvenile Justice History (2 items), Peers and Social Support 

Network (6 items), and Pro-Social Skills (3 items) assessment domains from the Disposition Tool 

6 Judges and magistrates were not sent the survey and as such did not participate in the web-based survey in State 3. 

However, judges and magistrates did participate in State 2 and State 1. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

44 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(see Appendix B for a list of the vignettes). A vignette was selected using the randomization 

function in Qualtrics. If a respondent answered “Yes” to the item, “Do you administer any of the

assessment instruments?,” one of the three vignettes was presented to the respondent, with all three 

vignettes having an equal chance of being selected. In each vignette, respondents were asked to 

read a hypothetical case and score out the vignette. A link to the scoring guide was provided in the 

survey and the scoring items relevant to the vignette were listed. These assigned vignettes were 

identical to some of those used in OYAS training procedures. 

The survey was administered through Qualtrics, a well-established online survey 

development platform. Survey distribution varied: sometimes the state agency sent out an email 

with the anonymous survey links and sometimes University of Cincinnati research staff distributed 

the anonymous survey links through the Qualtrics system. Respondents included individuals who 

were certified in the OYAS, OYAS trainers, and relevant stakeholders. Those individuals who 

were interviewed during the site visits were not asked to participate in survey. After the release of 

the survey, a first reminder was sent out two weeks later. A second and final reminder was sent 

out four weeks after the survey release date that identified the closing date for the survey as six 

weeks after the survey release. Out of a total of 3,125 possible participants, 1,013 juvenile justice 

personnel completed the survey, resulting in a 32.4% response rate.7 

These data were downloaded directly from the Qualtrics platform into SPSS (V.23) for 

analysis. In contrast to the interview data analysis, all closed and open-ended questions for the 

survey data were analyzed in SPSS due to the sheer size of the sample (N = 1,013). However, the 

same inductive coding process using grounded theory (Corbin & Strauss, 2008) was used to 

7 During the survey administration, the streamlined judges and magistrates web-based survey was inadvertently sent 

to a quarter of the State 2 OYAS administrator sample. Therefore, potentially 47 individuals in State 2 who administer 

the OYAS received that link, and therefore answered fewer questions and were not prompted to score a vignette. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

45 
 

 

 

 

 

 

 

 

 

 

 

analyze the open-ended questions of interest. For this reason, the results for some of the openended questions are displayed somewhat uniquely across the interview and survey samples. 

Interview and Web-Based Survey Overlap. Throughout this section, key findings from 

the interviews and web-based surveys are presented. While most of the items on the interviews 

and surveys are unique, 14 questions overlap. These questions targeted agency-level information 

on usage practices (e.g., override practices), quality assurance processes, and respondent attitudes 

toward and experiences with the OYAS and its automated system. Where the questions overlap, 

we explore similarities and differences in responses between the interview and web-based survey 

samples. In the following section, interview and survey sample personnel job characteristics are 

listed and discussed in brief to set the context for the subsequent results sections. Information 

pertinent to state context for the implementation of the OYAS is also presented in the following 

section. The overlapping questions are listed explicitly, and their results are compared between the 

interview and survey samples in the “Integrated Analysis” section of this report, which also 

includes a thorough consideration of the implications for these results within and across the 

samples. 

Descriptive Statistics and State-Specific Characteristics 

The juvenile justice personnel had varying degrees of professional experience, and worked 

in a number of unique juvenile court, correctional, and administrative settings. They varied in their 

job tasks and titles, both within and between the larger state context. These characteristics, along 

with the differences in assessment implementation across agencies and states, likely impacted their 

usage and perceptions of risk and needs assessments generally, and more specifically, the OYAS. 

The following sections describe the agency settings in which the personnel in the interview and 

web-based survey samples worked, their job classifications and titles, tenure in the field, as well 

as their familiarity with the OYAS. Together, these observations inform the analysis by providing 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

46 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

the context in which the personnel interacted with the OYAS, and therefore establishes the baseline 

conditions in which the OYAS has been implemented in the three states. 

Agency Setting. The juvenile justice personnel agency settings, displayed in Table 1.2, fell 

into one of six categories: court, probation, parole, correctional/secure treatment facility, state, or 

“other.” While the first four categories are somewhat intuitive, the fifth category, “state,” indicates 

a statewide administrative agency, often responsible for selecting the OYAS as a risk and needs 

assessment tool and planning its broader implementation. Categories were designated based on the 

setting of the agency where an interviewee was employed or as indicated by the respondent.

8 

The interview sample has its own unique distribution of agency settings. Notably, about 

half (48.4%) of the interviewees worked in a probation setting. This fits with the court records data 

that are presented subsequently in the larger report which contain a high prevalence of Disposition 

Tool assessments, as well as a pattern seen more generally in terms of where JRNA tend to be 

conducted (Wachter, 2015). Therefore, results from the interview sample reflect the responses 

given by probation personnel more so than court, parole, correctional/secure treatment facility, or 

state personnel. With the next most common agency setting being correctional/secure treatment 

facilities (at 20.3%), the vast majority of interviewees worked in settings responsible for 

processing delinquency cases and making disposition and placement decisions, supervision and 

custody, case planning, and delivery of direct services. State-level interviewees were among the 

least common job settings, at 8.3 percent. As these individuals played a role in the selection of the 

OYAS and the planning/oversight of its statewide implementation, they offered a unique 

perspective that has been drawn out for emphasis periodically throughout the results. However, 

8 In the interest of anonymity, agency-specific information was not collected for the web-based survey sample. 

Therefore, their agency settings were approximated according to their job title, as well as any other information 

provided in the survey responses (e.g., for someone with a Probation Officer job title, it was assumed that they worked 

in a probation agency setting). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

47 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

this perspective also suggests a somewhat “removed” position from frontline implementation of 

the OYAS, which some interviewees have suggested is an impediment to their assisting agencies 

in troubleshooting. 

The distribution of agency settings for the interviewees changes slightly when analyzing 

the data by state (Table 1.2). State 2 had the lowest parole agency representation.9 State 3 had 

about three times the number of probation interviewees than State 1, and about 20 percent more 

than State 2. This is a function of the fact that one agency in State 3 was the largest county across 

all of the states to be included in the interview site selection. State 1 had a higher proportion of 

correctional/secure treatment facility interviewees than State 2 or State 3, at about 31 percent. This 

was due to the fact that two different types of state institutions were selected as interview sites in 

State 1. While about one-quarter of State 1 interviewees worked in a court setting, only 8.9 percent 

of State 2 interviewees were in a court setting—due to county agencies being referred to as juvenile 

justice and community corrections agencies. These unique distributions at the state level 

undoubtedly influenced the interviewees’ responses, which will be discussed as applicable 

throughout. 

The web-based survey sample has its own distribution of agency settings, which, like the 

interview data, largely fall in line with trends regarding where JRNA tends to occur. Probation was 

the most common job setting (59.3% of the sample), and “other” was the second most common 

(15.7%). The “other” job setting category had an inflated frequency due to the nature of the data. 

Since the research team did not know the name of an agency a respondent was from,10 respondents 

were given pre-set job titles to choose from, as well as an “other” option. When “other” was 

9 In State 2, only one individual from parole was interviewed, as the OYAS is conducted on each youth prior to their 

release from a residential facility and youths do not typically remain on parole long enough for a reassessment. 

10This was a further layer of protection to ensure confidentiality to participants. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

48 
 

 

 

 

 

 

 

 

 

 

 

selected, the research team determined the job setting from the provided job titles. However, some 

responses were not specific enough to estimate the setting. When analyzing the survey sample by 

state, probation remained the most common setting for all three states, though State 3 had a higher 

proportion of probation-based personnel than State 2 or State 1 (76%, 69.2%, and 40.4%, 

respectively). Combining the interview and survey samples, probation was the most common job 

setting, and the perceptions of the OYAS from personnel in this setting are more represented than 

those from court, state, parole, correctional/secure treatment, or “other” settings. However, this 

overrepresentation is to be expected since probation officers are likely the most common job type 

to administer the assessment in juvenile justice systems. 

Job Title. Interviewees were asked to provide their job title and basic duties, which 

resulted in a long and varied list. The research team used an inductive coding process to collapse 

the interviewee-provided job titles into those seen in Table 1.2. Because of this inductive process, 

the job titles for the interview and survey samples differed in their categorization and specificity. 

For example, all job titles in the interview sample containing “Director” that were not explicitly 

related to treatment, probation, parole, or detention were combined into the “Non-Treatment 

Director/Executive Director” category (4.1% of the sample). In addition, all job titles containing 

“Administrator” (e.g., Regional Administrator, Special Programs Administrator, Quality 

Improvement Administrator) were collapsed into the “Administrator” category (6% of the sample). 

For clarity, this category captured those in an “administrative” role, and does not represent those 

interviewees who administer the OYAS as part of their job duties. That information was captured 

subsequently under “Proximity to the OYAS”, and is explored more thoroughly in the “Web-Based 

Survey Results” section. Moving forward, all references to “administrative” staff refer to their 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

49 
 

 

 

 

 

 

 

 

 

general role within the agency, and not to whether they administer the OYAS. Personnel who 

administered the OYAS are referred to as “OYAS administrators.”

Though the regular duties of the personnel within each job title likely differed to a degree, 

the main function of their role was captured by the category. In the event that an interviewee’s

given job title was unclear, the research team looked to the description of their daily duties before 

making a final designation. Table 1.2 shows that the highest proportion of interviewees were 

Probation Officers (35.9% of the sample) or Probation Supervisors (11.1% of the sample). In 

addition, there was a notable proportion of Non-Supervisory Treatment Staff (13.8% of the 

sample). When analyzing job titles in the interview sample by state, State 1 had a larger proportion 

of Administrators (8.5%), parole representatives (including Officers and Supervisors, 5.6%), and 

Non-Supervisory Treatment Staff (22.5%), and a smaller proportion of Probation Officers (18.3%) 

than the other two states. State 2 had the largest proportion of Chief Probation Officers (8.8%), 

Probation Supervisors (13.3%), and Judges/Magistrates/Attorneys. Lastly, State 3 had the largest 

proportion of Probation Officers (45.5%). 

Unlike the interview sample, the web-based survey respondents were provided eleven 

options from which they were able to select their job title: State-Level Administrator, 

Judge/Magistrate, Supervisor, Intake Officer/Intake Staff, Unit Manager, Parole Officer, Detention 

Officer, Case Manager, Court Administrator, Probation Officer, and Other. Those who selected 

“Other” were prompted to manually list their job title, and these job titles were examined closely 

to determine their fit in one of the pre-designated categories. As shown in Table 1.2, Probation 

Officer was the most common job title (60.2% of the full sample), followed by Supervisor 

(General) at 13.8%. This category captured anyone who worked in a supervisor capacity, and was 

different than the supervisor categories developed within the interview data, which had more 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

50 
 

 

 

 

 

 

 

 

 

 

 

 

 

detail. In Table 1.2, a dash (-) represents a job title that did not apply to that sample. For example, 

as survey respondents did not have the option to specify which type of supervisor they were (e.g., 

probation, parole, detention), these rows contain a dash (-) in the survey columns. This is different 

from cells that contain a value of zero, which indicates that a given job title was absent from the 

sample. This demarcation applies to all of the fields in Table 1.2. 

When analyzing the survey sample job titles by state, Probation Officer remained the most 

common in all three states. The State 3 subsample had the highest proportion of Probation Officers 

when compared to the State 2 and State 1 (72.7%, 72%, and 42.9% of the sample, respectively). 

The Supervisor (General) job title was also common, but did not exceed 16 percent in any of the 

three state samples. Unlike in the interview data, “Other” was a common job title in the survey 

data, as survey respondents often elected to manually insert their job title, rather than choose from 

a pre-designated list. Some examples of “Other” job titles included: Research Analyst, IT 

Representative, Program Manager, and Compliance Coordinator. 

Job Classification. The juvenile justice personnel held positions in various locations, and 

within each location, unique job classifications. In this context, the variable “job classification”

differs from “job title” in that it represents the function, or “type” of day-to-day role the personnel 

fulfilled in their agency. For the interviews, these classifications were derived inductively by 

extracting job duties from the interviewee-provided job titles and descriptions. These job duties 

were then organized into broad groups that represent interviewees with similar job roles across 

agencies. For the surveys, the pre-designated job titles were collapsed into job classification 

categories. However, as these job titles did not lend themselves to designating frontline staff, we 

created a proxy measure to represent this category of job classification. All respondents who 

indicated that they a) carried a caseload, and/or b) administered the OYAS were grouped into the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

51 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“frontline” staff classification in the survey sample. The final job classification categories for both 

the interviews and surveys included administrator/director, judge/magistrate/attorney, frontline 

staff, supervisor/manager, and other.

11 As relevant throughout the report, the data were analyzed 

according to job classification. This was done in order to draw out important, meaningful 

distinctions in the results, which helped to give context, as well as inform the recommendations. 

For example, in the interview results, the overall level of satisfaction with the OYAS was analyzed 

by the frontline job classification with the hypothesis that frontline staff, having more proximity 

to the OYAS, would express stronger feelings of satisfaction or dissatisfaction with the OYAS. 

First, the administrator/director classification included those who were likely the most 

removed from the day-to-day work with youths. These were the people at the state, county, and 

agency level who made such “big-picture” decisions as adopting a risk and needs assessment, who 

may have created and enforced its policies and procedures, coordinated training and quality 

improvement efforts, or tracked assessment data. As illustrated in Table 1.2, the State 1 interview 

and survey samples had a larger portion of administrators/directors than State 2 or State 3 (about 

12% of the full State 1 sample). Therefore, responses from the State 1 reflected the values and 

opinions of this group more than those in the other two states. The administrator/director job 

classification was one of the less common job classifications within the survey data due to a 

difference in sampling strategy (5.7% of the sample). Whereas the interview sample was purposive 

in nature (i.e., the interviews targeted more of the “champions” of the OYAS, often including 

administrators and those in leadership positions at the county and state level), the survey sample 

was meant to capture more of the variation within the juvenile justice personnel population. This 

11 Only one interviewee was classified in the “other” job classification. This individual was a [non-officer] probation 

staff member who conducted data entry for OYAS assessments and reported this information to the state office. 

Therefore, this job title could not be accurately coded into any of the existing categories. There was, however, a higher 

proportion of “other” job classifications in the survey sample due to the anonymous nature of the data.

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

52 
 

 

 

 

 

 

 

 

 

 

 

difference in sampling strategy between the interviews and surveys is noted where relevant. 

The next classification, judge/magistrate/attorney, had more direct exposure to the youths 

than those personnel in the administrator/director classification. While these were not the 

individuals tasked with administering the OYAS, they likely came into daily contact with youths 

(in a court setting), and may have informed some of their decision-making according to risk and/or 

need level as determined by the OYAS. As seen in Table 1.2, judges/magistrates/attorneys made 

up nearly 10 percent of the interview sample, and were distributed relatively evenly across the 

three states (9.9% in State 1, 9.9% in State 3, and 8.9% in State 2). This group made up less of the 

survey sample, at about 5.7 percent. This was due in part to the fact that judges and magistrates 

did not participate in the survey in State 3. 

The frontline staff classification was assigned to those whose main job function was to 

work directly with youths in service provision, whether through treatment or supervision. Many 

of these individuals carried a caseload, and/or administered the OYAS, though it is important to 

note that frontline staff did not necessarily administer the OYAS. Even so, they were typically 

aware of the OYAS results (e.g., risk level and perhaps criminogenic and non-criminogenic needs), 

and were tasked with using this information in various ways, such as managing their caseloads by 

creating care plans or making referrals based on the OYAS results. This distinction is important 

because these personnel have likely had the most intimate experience with the OYAS, making 

them more keenly aware of the micro-level strengths or limitations of risk and needs assessment 

than personnel in other job classifications. In the same vein, these personnel may also have had 

less awareness of the “big picture” strengths, limitations, and usage of a risk and needs assessment. 

In the interview data, the State 3 subsample had the largest portion of frontline personnel, 

at 61.4 percent. About half of the State 1 and State 2 interview samples were frontline staff. In the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

53 
 

 

 

 

 

 

 

 

 

 

 

 

survey data, the proportion of frontline staff was much higher (roughly 72%). State 2 had the 

highest portion of frontline personnel, though it was comparable to State 3 (78.3% and 77.1% of 

the samples, respectively). As “frontline” was the most common job classification in both the 

interview and survey samples, it is important to note that their values, experiences, and opinions 

are the most represented throughout the report. 

The last classification was supervisor/manager. These interviewees were the “middlemanagers” of the sample, and also tasked with supervising a team of frontline staff. They were 

also often the staff that had the most familiarity with the OYAS policies and procedures, and some 

of these supervisors/managers served as OYAS trainers. As such, these personnel had a close 

understanding of the role of administrators/directors, as well as the frontline staff they supervised. 

In addition, they worked closely with youths and performed some of the same tasks as frontline 

staff. In the interview sample, one in five were supervisors/managers, though State 2 and State 1 

had a larger proportion than State 3 (24.4%, 22.5%, and 17.8%, respectively). This was the second 

most common job classification next to line staff. In the survey sample, supervisors/managers 

made up about 10 percent of the sample and were distributed relatively evenly across the three 

states. 

Proximity to the OYAS. Several items on the interview guide and web-based survey 

gauged personnel familiarity with and proximity to the OYAS.12 The research team generally 

defined proximity in terms of staff members’ degree of assessment usage in routine practice. For 

example, some personnel (e.g., judges/magistrates) may have seen the scores from the assessment 

during disposition procedures, but perhaps have not engaged with it like a probation officer who 

was responsible for conducting case investigations or who personally administered the OYAS. 

12 See Q39 under Youth Assessment Practices, and Q36 under Implementing the OYAS in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

54 
 

 

 

 

 

 

 

 

 

Similarly, some personnel used the automated system even if they did not personally administer 

the OYAS, sometimes in a supervisory capacity or for training purposes. Some of these individuals 

may have received OYAS information and used it in case planning or the provision of services. 

Table 1.2 shows that, while a small proportion of the interview sample were OYAS trainers 

or coordinators for their respective agencies (4.1% of the sample), a larger portion administered 

one or more of the tools as a part of their daily job function (54.4% of the sample). About 61 

percent used the online automated system to track OYAS information, including merging 

information from previous assessments, setting case plan goals, and generating youths’ progress 

reports. This automated system is meant to be the information management component of the 

OYAS system where results are recorded and stored, and both individual and group/agency-level 

reports are generated from this system. Personnel perceptions of the strengths, limitations, and 

overall implementation of the automated system is discussed separately in these results. 

State 1 had a higher proportion of interviewees who personally administered the OYAS 

(60.6% of the sample), compared to about 55 percent for State 3 and 42 percent for State 2. 

Whereas all three states had a similar proportion of OYAS trainers or coordinators, there was more 

variation in whether interviewees used the automated OYAS system. State 1 had a lower portion 

of interviewees who used the automated system, at about 64 percent. By contrast, between 78 to 

82 percent of State 3 and State 2 interviewees used the automated OYAS system. 

The survey sample had a slightly different distribution of proximity relative to the OYAS 

from the interview sample, as evidenced in Table 1.2. Approximately two-thirds of the survey 

sample administered the OYAS and/or used its automated system. State 2 had a higher proportion 

of OYAS administrators (83.7% of the sample), compared to 67.4 percent in State 3, and 72.6 

percent in State 1. Consequently, State 2 also had a higher proportion of personnel who used the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

55 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

automated system (84% of the sample). These results suggest that State 2 survey respondents, due 

to their close proximity to the OYAS, may be more intimately familiar with its processes, and its 

strengths and limitations than State 3 or State 1 survey respondents. 

Data were not collected on whether the survey respondents were OYAS trainers or 

coordinators and this information could not be estimated. Similarly, survey respondents did not 

have the option of responding “unsure” to survey items or prompts from interviews (as did the 

interview sample); therefore, any uncertainty may have manifested in skipping the question, thus 

reducing response rates for certain items. 

Job Tenure. In Table 1.2, job tenure represents the number of years the personnel 

estimated that they have worked in their current agency/facility. This did not necessarily represent 

their time in the field of juvenile justice, which may have exceeded the time in their current 

agency/facility. Therefore, job tenure in this context represents a conservative estimate of the 

interviewees’ and respondents’ time spent working in the juvenile justice field. 

Job tenure for the interview sample ranged from one year to 42 years, (mean [x̅]= 13.3, 

standard deviation [sd] = 8.25). The standard deviation value suggests that there was a moderate 

amount of variation in job tenure, and that the majority (about 70%) of the sample worked at their 

agencies between five and 21 years. State 1 (N = 71) interviewees’ job tenure ranged from one to 

38 years (x̅= 11.2, sd = 8.7), State 2 (N = 45) interviewees’ job tenure ranged from one to 42 years 

(x̅= 12.7, sd = 8.3), and State 3 (N = 99) interviewees’ job tenure ranged from one to 30 years (x̅

= 15.0, sd = 7.6). A one-way analysis of variance (ANOVA) revealed that State 3 interviewees 

had a significantly longer average job tenure than State 1 or State 2, also with less variation (F = 

4.747, df = 2, p < .05). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

56 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Job tenure for the survey sample ranged from zero (i.e., in the field for less than one year)13 

to 40 years (x̅= 13.9 years, sd = 8.5). Differences in average job tenure between the interview and 

survey samples was non-significant. The State 1 survey subsample job tenure ranged from zero to 

40 years, with the highest average job tenure of the three states (x̅= 14.2, sd = 8.8). The State 2 

survey subsample job tenure ranged from zero to 40 years, and had the lowest average job tenure, 

with the highest amount of variation in responses (x̅= 13.3, sd = 9.2). Lastly, the State 3 survey 

subsample job tenure ranged from zero to 34 years, and had a mean that fell in between State 1 

and State 2 (x̅= 14.0, sd = 7.0). These differences in average job tenure between the states in the 

survey sample were non-significant. However, it is accurate to suggest that on average, State 1 

respondents worked at their agencies/facilities for longer than State 2 or State 3 respondents. This 

length of experience may have impacted their perceptions of the OYAS and of risk and needs 

assessments in general, having potentially been employed in the juvenile justice field prior to the 

wide adoption and application of risk and needs assessments. Personnel with longer job tenure 

may also have been exposed to a wider array of risk and needs assessment tools and 

implementation experiences over the years. These conclusions could potentially be applied to any 

personnel who have worked in an agency/facility for an extended period of time. 

Table 1.2 contains the descriptive statistics for agency setting, job title, job classification, 

proximity to the OYAS, and job tenure for the interview and survey samples in each state. Note 

that the sample size decreased with questions that did not apply to some personnel. For example, 

if an interviewee or survey respondent indicated that they did not administer the OYAS or use its 

information, then the research team may not have asked if they use the OYAS automated system 

(an item which has a lower response rate than the others). 

13 Personnel with little on the job experience were most likely not selected for the interviews, which is why none of 

the interviewees reported less than one year of job tenure. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

57 
Table 1.2. Descriptive Statistics for Juvenile Justice Personnel Sample Job Characteristics 

In-Person Interviews Web-Based Surveys 

Variable State 1 State 2 State 3 State 1 State 2 State 3 

Agency Setting n n n 

(%) (%) (%) 

n n n 

(%) (%) (%) 

Court 18 4 15 

(25.4) (8.9) (14.9) 

73 28 1 

(18.0) (8.7) (0.4) 

Probation 17 21 67 

(23.9) (46.7) (66.3) 

164 222 206 

(40.4) (69.2) (76.0) 

Parole 8 1 5 

(11.3) (2.2) (5.0) 

5 2 0 

(1.2) (0.6) (0.0) 

Correctional/Secure 

Treatment 

22 13 9 

(31.0) (28.9) (8.9) 

85 30 17 

(20.9) (9.3) (6.3) 

State 6 6 5 

(8.5) (13.3) (5.0) 

5 2 1 

(1.2) (0.6) (0.4) 

Other 0 0 0 

(0.0) (0.0) (0.0) 

74 37 46 

(18.2) (11.5) (17.0) 

Total 71 45 101 

(100.0) (100.0) (100.0) 

406 321 271 

(100.0) (100.0) (100.0) 

Job Title n n n 

(%) (%) (%) 

n n n 

(%) (%) (%) 

Administrator 6 2 5 

(8.5) (4.4) (5.0) 

43 5 2 

(10.6) (1.6) (0.8) 

Attorney 1 1 4 

(1.4) (2.2) (4.0) - - -

Judge/Magistrate/ 

Commissioner 

6 4 5 

(8.5) (8.8) (5.0) 

25 25 0 

(6.1) (8.0) (0.0) 

Parole Officer 4 1 5 

(5.6) (2.2) (5.0) 

4 2 0 

(1.0) (0.6) (0.0) 

Parole Supervisor 2 0 0 

(2.8) (0.0) (0.0) - - -

Parole Director 2 0 0 

(2.8) (0.0.) (0.0) - - -

Probation Officer 13 18 46 

(18.3) (40.0) (45.5) 

175 224 197 

(42.9) (72.0) (72.7) 

Probation 

Supervisor 

8 6 10 

(11.3) (13.3) (9.9) 

5 3 5 

(1.2) (1.0) (1.8) 

Chief Probation 

Officer 

2 4 2 

(2.8) (8.8) (2.0) - - -

Detention/Facility 

Officer 

2 0 2 

(2.8) (0.0) (2.0) 

4 3 3 

(1.0) (1.0) (1.1) 

Detention/Facility 

Supervisor 

3 0 0 

(4.2) (0.0) (0.0) - - -

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

58 
Table 1.2. Descriptive Statistics for Juvenile Justice Personnel Sample Job Characteristics 

In-Person Interviews Web-Based Surveys 

Variable State 1 State 2 State 3 State 1 State 2 State 3 

Detention/Facility 1 0 3 

Director (1.4) (0.0) (3.0) - - -

Non-Supervisory 16 2 12 

Treatment Staff (22.5) (4.4) (11.9) 

16 12 1 

(3.9) (3.9) (0.4) 

Treatment 4 2 4 

Supervisor/Director (5.6) (4.4) (4.0) - - -

Non-Treatment 1 4 3 

Director (1.4) (8.8) (3.0) - - -

Supervisor - - - (General) 63 36 38 

(15.4) (11.6) (14.0) 

0 1 0 Other (0.0) (2.2) (0.0) 73 1 25 

(17.9) (0.3) (9.2) 

71 45 101 Total (100.0) (100.0) (100.0) 408 311 271 

(100.0) (100.0) (100.0) 

n n n Job Classification (%) (%) (%) n n n 

(%) (%) (%) 

Administrator/ 13 6 11 

Director (18.3) (13.3) (10.9) 

43 5 2 

(11.1) (2.0) (0.8) 

Judge/Magistrate/ 7 4 10 

Attorney (9.9) (8.9) (9.9) 

25 25 0 

(6.4) (10.0) (0.0) 

35 23 62 Frontline Staff (49.3) (51.1) (61.4) 249 195 182 

(64.2) (78.3) (77.1) 

16 11 18 Supervisor/Manager (22.5) (24.4) (17.8) 28 23 35 

(7.2) (9.2) (14.8) 

0 1 0 Other (0.0) (2.2) (0.0) 43 1 17 

(11.1) (0.4) (7.2) 

71 45 101 Total (100.0) (100.0) (100.0) 388 249 236 

(100.0) (100.0) (100.0) 

Proximity to the n n n 

OYAS (%) (%) (%) 

n n n 

(%) (%) (%) 

Administers OYAS 

43 19 56 Yes (60.6) (42.2) (55.4) 249 195 182 

(72.6) (83.7) (67.4) 

28 26 45 No (39.4) (57.8) (44.6) 94 38 88 

(27.4) (16.3) (32.6) 

0 0 0 Unsure (0.0) (0.0) (0.0) - - -

71 22 101 Total (100.0) (100.0) (100.0) 343 233 270 

(100.0) (100.0) (100.0) 

OYAS Coordinator/Trainer 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

59 
 

 

Table 1.2. Descriptive Statistics for Juvenile Justice Personnel Sample Job Characteristics 

In-Person Interviews Web-Based Surveys 

Variable State 1 State 2 State 3 State 1 State 2 State 3 

Yes 3 2 4 

(4.2) (4.4) (4.0) 

- - -

No 68 43 97 

(95.8) (95.6) (96.0) 

- - -

Unsure 0 0 0 

(0.0) (0.0) (0.0) 

- - -

Total 71 45 101 

(100.0) (100.0) (100.0) 

- - -

Uses Automated System 

Yes 42 29 63 

(63.6) (78.4) (81.8) 

163 142 148 

(68.8) (84.0) (77.5) 

No 23 8 14 

(34.8) (21.6) (18.2) 

74 27 43 

(31.2) (16.0) (22.5) 

Unsure 1 0 0 

(1.5) (0.0) (0.0) - - -

Total 66 37 77 

(100.0) (100.0) (100.0) 

237 169 191 

(100.0) (100.0) (100.0) 

Job Tenure (years) x̅ x̅ x̅

sd sd sd 

x̅ x̅ x̅

sd sd sd 

11.2 12.7 15.0 

8.7 8.3 7.6 

14.2 13.3 14.0 

8.8 9.2 7.0 

Due to missing data, totals are less than the reported sample size. 

In summary, the interview and survey sample job characteristics (e.g., agency setting, job 

title and classification, job tenure, and proximity to the OYAS) inform the results presented in this 

section. In the interview sample, nearly three-quarters worked in a probation setting or secure 

facility, while state-level administrators represented the lowest proportion of interviewees. 

Probation Officer and Probation Officer Supervisor were the two most common job titles among 

the interview sample, though there was also a high proportion of non-supervisory treatment staff. 

The frontline job classification (i.e., those personnel whose main job function put them in daily 

contact with the youths in an assessment, supervisory, case management, or treatment roles) were 

the most common. Nearly two-thirds of the interview sample administered the OYAS, and about 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

60 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

three-quarters used its automated system. The average tenure of the interview sample was 13.3 

years. All of these job characteristics varied between the states within the interview sample, and 

the implications of this variation are discussed where relevant throughout the report. 

Job characteristics also had a unique distribution across the web-based survey sample, and 

across the three states within that sample. The survey sample represented the general population 

of juvenile justice practitioners, whereas the interview sample was purposive, and therefore 

contained more people considered to be the “champions” of the OYAS (i.e., state-, county-, and 

agency-level administration and leadership, as well as OYAS trainers and coordinators). Less 

definitive conclusions can be made from some of the job-specific variables (e.g., job setting and 

job classification) within the anonymous survey data than the in-person interview data, as these 

were approximations based solely on the job titles provided. However, in general, it is clear that 

the majority of the survey sample respondents were based in a probation setting, and performed 

the duties of “frontline” staff, such as administering the OYAS and using its automated system. 

Subsample size for the states and across the interviews and surveys varies both within and 

across the results tables throughout the report. These varying sample sizes are most often due to 

one of two factors: 1) the personnel did not have a response to a given question and indicated that 

they did not know, or chose not to answer; or 2) due to some personnel characteristic (e.g., job 

tenure, job classification, job setting), a given question did not apply to them. For example, if 

survey respondents indicated that they did not administer the OYAS, they would not be asked 

which specific OYAS tools they were trained to administer. Or, for example, if interviewees did 

not work at an agency until after the OYAS was implemented, they may have chosen not to answer 

questions about the specific OYAS implementation processes at their agency. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

61 
 

 

 

 

 

 

 

 

 

 

 

 

 

State Context for Implementation and Usage 

Timeline for OYAS Implementation. While the sample characteristics provided some 

context for the results that follow, the state and agency-level factors should also be kept in mind 

in interpreting the key findings. State 1, State 2, and State 3 were in unique stages of implementing 

a statewide risk and needs assessment (see timelines in Figure 1.2), which gives further context to 

the results presented in the current report. These differences are evidenced by various state and 

county initiatives, validation processes, the existing service frameworks at the time of OYAS 

implementation (including various automated systems or previous risk and needs assessments), 

and the personnel responses. State 1 was unique from State 2 and State 3 in that it was the only 

state with a decentralized juvenile justice system, with each of its counties functioning 

independently from the state-level administrative body. As such, unlike the State 2 and State 3 

central administrative office, the State 1 central administrative office did not have authority over 

whether State 1 courts used the OYAS. 

In terms of state- and county-wide initiatives, all three states were similar in two ways. 

One, they all participated in certification and ToT training with UCCI. Two, they all had 

participated in the Annie E. Casey Foundation’s Juvenile Detention Alternatives Initiative (JDAI)

at some point around the implementation period. JDAI is an integrative effort focusing on the 

reallocation of public resources from incarcerating youths toward investment in youths, families, 

and communities. It also placed a premium on risk and needs assessment as a conduit to making 

better decisions about detaining youths (Steinhart, 2006). The processes and goals of JDAI were 

the most focal to State 2 personnel, as the initiative started in State 2 in 2006, and continued to 

spread through the counties through 2016. While State 3 was also a JDAI state replication site, 

most of the county processes were completed in 2014, prior to our data collection period. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

62 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Relatedly, while eight counties in State 1 participated in JDAI, they were not considered a state 

replication site, and the last county expansion took place in 2013. 

State 1 had an additional statewide initiative that has impacted the implementation and 

usage of the OYAS. State 1 created a program that functioned to divert youths from incarceration, 

and filtered these funds instead into evidence-based programs in the community. This program 

was connected to risk and needs assessment as youths’ placement in state facilities depended upon 

OYAS risk level, with more intensive interventions being reserved for high risk youth. As a result, 

there has been a significant reduction in incarceration rates for youths in State 1 in recent years. In 

general, this program has been influential in pushing State 1 counties and the state as a whole 

toward evidence-based practices, including the adoption and usage of a statewide risk and needs 

assessment. The impact of this program will be seen subsequently as the State 1 personnel tended 

to mention aggregate data usage and the allocation of resources within their agencies more than 

interviewees in the other two states. These unique implementation characteristics are summarized 

in Figure 1.2 below. 

Figure 1.2. State-Specific Timelines for OYAS Implementation 

State 1 State 2 State 3 

•2000's: Institution diversion •2011: JDAI implemented in •2006: JDAI implemented in programs developed select counties select counties •2009: Pilot counties adopt •2013: OYAS adopted statewide •2010: OYAS piloted in select OYAS counties •2014: JDAI expanded to more •2010: JDAI implemented in counties •2011: OYAS and web-based 

select counties system adopted statewide •2018: OYAS validation study •2010: OYAS adopted statewide for residential youth completed •2012: OYAS validated •2011: OYAS inter-rater (as part of this project) •2016: JDAI adopted reliability study •2018: Validation studies started statewide 

•2018: OYAS re-validation 

started 

Written Policies and Procedures. Aside from various sample characteristics and 

implementation processes, each state also had its own unique set of written policies and procedures 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

63 
 

 

 

 

 

 

 

 

 

 

regarding risk and needs assessment, with varying degrees of accessibility. While State 2 and State 

3 had readily available OYAS policies and procedures on their state agency websites, State 1’s

OYAS policies were not located on a publicly accessible platform. Instead, the research team 

obtained the State 1 policies from state contacts. Upon the research team analyzing these three sets 

of policies and procedures, several general commonalities and discrepancies were identified. 

Notably, all three states’ policies and procedures manuals reflected the principles of 

effective intervention (see Andrews et al., 1990) to varying degrees. State 3, the only state whose 

policy did not mention the OYAS by name, outlined the best practices for intensive and standard 

probation, including definitions of risk, criminogenic needs, and risk and needs assessment. Best 

practices for supervision and case planning (e.g., transferring youths from intensive to standard 

probation based on changes in their risk level, using the risk and needs assessment results to inform 

the “supervision plan”) were woven in alongside administrative guidelines such as budgetary 

requests, program eligibility, and caseload limits, making the State 3 policies and procedures 

manual by far the most extensive of the three states’, and the most expansive with respect to 

policies specific to risk and needs assessment itself. The manual served as a general guide for how 

agencies might have created their own policies surrounding risk and needs assessment, and did 

not, for the most part, give specific recommendations about timelines or usage. Two notable 

exceptions were that probation officers were to conduct a risk and needs assessment within 30 

days of having a youth assigned to their caseload (only if a youth did not receive one in the predisposition phase), and intensive supervision officers were to make treatment referrals within 30 

days of disposition as well. Overall, the State 3 policies and procedures manual was lengthy in 

comparison to State 1 and State 2, and though detailed in its presentation of evidence-based 

practices in supervision, it did not specifically name the OYAS. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

64 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The State 2 policies and procedures manual also focused on best practices and the 

underlying purpose of the implementation and use of an actuarial risk and needs assessment. This 

included guidance for case planning (e.g., tailoring case plans to reassessment results), 

responsivity assessments (e.g., conducting responsivity assessments as substance use, mental 

health, or sexual offending needs arise), as well as a user-friendly chart with clear guidelines for 

conducting each of the five OYAS tools (in addition to the Disposition Screener). While some of 

the practices were specific (e.g., reassessments were to occur at least every 6 months, and 

assessment information entered into the automated system within 30 days), others were vague. For 

instance, case plans were to be updated “as needed.” This language may have resulted in case plans 

that were less “living” documents, reducing their value in assessing change. This will be evidenced 

by the results in Table 1.6, where State 2 interviewees were the least likely to indicate that the 

OYAS was used to measure youths’ progress. The Disposition Screener and Tool had more 

specific timelines than the rest of the tools, likely due to the fact that there is a natural timeline to 

have these tools completed prior to adjudication so that their information may be used in court. 

Overall, the State 2 OYAS policy was easily accessible, simple to read, and followed the principles 

of effective intervention. 

By contrast, the State 1 written OYAS policies and procedures focused more on the various 

OYAS user-levels (e.g., administrative, inquiry, supervisor, super user). It did not provide details 

about best practices for localities, nor were the principles of effective intervention mentioned in 

detail. The State 1 manual did not include policies for each of the specific OYAS tools, but did 

provide more general guidance about training processes and user responsibilities. For example, 

users were to receive “booster training as deemed necessary.” Quality assurance practices were 

mentioned specifically, but focused more on the accuracy of the information entered into the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

65 
 

 

 

 

 

 

 

 

 

automated system, the quantity of overrides, and the timelines by which assessments should be 

completed, not necessarily concerning the quality of the assessment process itself. Overall, the 

OYAS policies and procedures manual for State 1 was easy to understand (though not readily 

available online), but did not go into depth about evidence-based practices, case planning, or the 

specifics of each of the OYAS screeners and assessments. The lack of specificity in these areas 

may be a result of the fact that the state authority does not set policy for the counties and expected 

its localities to develop their own policies related to the OYAS. 

Personnel responses on policies and procedures. As written, the policies and procedures 

sometimes stood in contrast to the personnel responses related to this topic. This is evidenced in 

Table 1.3, where the majority of interviewees (69.4%) indicated that there are written policies in 

place.

14 However, as will be seen subsequently, the interviewees generally did not specify the 

details of a given policy, or know where the policies could be located. 

Table 1.3. Interview Sample Descriptive Statistics for Written Documentation Regarding the 

OYAS 

Variable State 1 State 2 State 3 Combined 

Agency Has Written 

Policies in Place? n (%) n (%) n (%) n (%) 

Yes 

No 

Unsure 

Total 

25 (42.4) 

16 (27.1) 

18 (30.5) 

59 (100.0) 

30 (83.3) 

3 (8.3) 

3 (8.3) 

36 (100.0) 

56 (86.2) 

3 (4.6) 

6 (9.2) 

65 (100.0) 

111 (69.4) 

22 (13.8) 

27 (16.9) 

160 (100.0) 

Due to missing data, totals are less than the reported sample size. 

In Table 1.3, State 1 drove the variation in overall percent of the sample that indicated their 

agency had written documentation regarding the OYAS. State 1 had approximately half the 

proportion of affirmative responses as State 2 and State 3, at 42.4 percent. State 1 also had a much 

higher proportion of respondents who were unsure whether there was written documentation, at 

14 See Q37 under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

66 
 

 

 

 

 

 

 

 

 

 

 

 

30 percent (versus about 8% for State 2 and 9% for State 3). This may have been a product of the 

accessibility of said state policies and/or the degree to which policies were developed at the local 

level. 

Reassessment policies. All three states had reassessment policies written in to their risk 

and needs assessment policies and procedures manuals. The State 1 policy stated that 

reassessments should be completed no earlier than four months and no later than six months from 

the previous assessment. State 2 indicated that reassessments should occur every six months, or 

more frequently if permissible by “local policy,” giving leeway where local context may dictate 

the appropriate timeline for reassessments. State 3 provided less specific guidance, stating in their 

policy that probation officers should “periodically examine the needs of each juvenile” to

determine whether their supervision level should be modified. To varying degrees, these policies 

reflected the University of Cincinnati’s original design for the OYAS, which indicated that 

reassessments should occur every six months, or if a major life event occurs before this six-month 

timeline that would significantly change the score. 

Interviewees’ perceptions and knowledge of reassessment policies and procedures did not 

always coincide with the written policies and procedures described previously, and varied from 

agency to agency. There was also a degree of dissatisfaction expressed regarding reassessment 

policy, as some interviewees indicated that they are not conducted with enough regularity to 

capture the quickly changing circumstances of juveniles. Related, interviewees expressed other 

frustrations with these policies. For example, some noted that they are in need of particular 

guidance for which tool is appropriate for reassessing residential youths. Others stated that they 

were in need of guidance on when youths should be reassessed and what constitutes a “major life

event.” 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

67 
 

 

 

 

 

 

 

 

 

 

Interviewees were asked to identify and describe the OYAS reassessment process,15 and 

their responses are presented in Table 1.4. The difference in the response rates between the 

reassessment questions is partially explained by interviewees’ familiarity with reassessment

policies, along with built-in skip patterns. For example, if interviewees indicated there was not a 

reassessment policy in place, or that they were unsure if there was a reassessment policy, then they 

were not asked whether the said policy was followed. 

Table 1.4. Interview Sample Descriptive Statistics for Reassessment Policies 

Variable State 1 State 2 State 3 Combined 

Reassessment Policy in Place? n (%) n (%) n (%) n (%) 

Yes 33 (63.5) 31 (88.6) 60 (84.5) 124 (78.5) 

No 11 (21.2) 4 (11.4) 5 (7.0) 20 (12.7) 

Unsure 8 (15.4) 0 (0.0) 6 (8.5) 14 (8.9) 

Total 52 (100.0) 35 (100.0) 71 (100.0) 158 (100.0) 

Reassessment Policy Followed? n (%) n (%) n (%) n (%) 

Yes 21 (77.8) 15 (68.2) 37 (82.2) 73 (77.7) 

No 2 (7.4) 4 (18.2) 4 (8.9) 10 (10.6) 

Unsure 4 (14.8) 3 (13.6) 4 (8.9) 11 (11.7) 

Total 27 (100.0) 22 (100.0) 45 (100.0) 94 (100.0) 

Youths Leave Without 

Reassessment? n (%) n (%) n (%) n (%) 

Yes 18 (72.0) 15 (62.5) 29 (80.6) 62 (72.9) 

No 4 (16.0) 9 (37.5) 5 (13.9) 18 (21.2) 

Unsure 3 (12.0) 0 (0.0) 2 (5.6) 5 (5.9) 

Total 25 (100.0) 24 (100.0) 36 (100.0) 85 (100.0) 

Due to missing data, totals are less than the full reported sample size. 

Over 78 percent of interviewees indicated their agency had a reassessment policy in place. 

Similarly, over 77.7 percent reported that their agency followed the reassessment policies. 

However, 72.9 percent shared that there were instances when a youth would leave an agency 

without a reassessment. Similar to the findings depicted in Table 1.3 (regarding all written policies 

and procedures for the OYAS), State 1 drove the variation with respect to trends in reassessment 

15 See Q50a-c under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

68 
 

 

 

 

 

 

 

 

 

 

 

 

policy as well. In Table 1.4, State 1 had by far the smallest proportion of affirmative responses to 

whether the interviewees’ agency had a reassessment policy in place, at 63.5 percent. Again, State 

1 also had a higher proportion of interviewees stating that they were unsure whether a reassessment 

policy was in place, at about 15 percent (as opposed to about 9% for State 3, and 0% for State 2). 

Again, this may have been the product of both a lack of policy accessibility and/or lack of a local 

policy. It is possible that turnover may have contributed to this, as State 1 interviewees had the 

lowest average job tenure of the three states (~11 years). However, a Chi-Square test revealed no 

significant relationship between job tenure and awareness of a reassessment policy. 

In terms of whether the policies in place were followed, State 2 had the lowest proportion 

of affirmative answers, at about 68 percent. For the combined sample, about one in ten 

interviewees were unsure whether their reassessment policy was followed within their agency. 

Nearly three out of four interviewees indicated that youth sometimes left their agencies/facilities 

without a reassessment, though there was some variability between the states, with State 3 having 

the highest proportion of affirmative responses, at 80.6 percent (versus State 1 at about 72%, and 

State 2 at 62.5%). 

A Chi-Square test revealed that job classification is related to interviewee perceptions 

regarding reassessments, though it is not a statistically significant relationship. Particularly, 

frontline staff tended to agree more that some youths leave the agency without a reassessment 

(2 

(2) = 5.93, p = .052).16 This was the only job classification distinction that had suggestively 

different responses in the reassessment items. This may have been a reflection of frontline 

interviewees’ intimate knowledge of the day-to-day occurrences of an agency, while non-frontline 

staff may have been less familiar. 

16 The number in the parenthetical stands for degrees of freedom. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

69 
 

 

 

 

 

 

 

 

 

 

 

Override policies. 

17 As originally designed by the University of Cincinnati, those who 

administer the OYAS have the discretion to override the overall risk level for up to 10 percent of 

cases. The State 3 and State 2 policies and procedures manuals did not dictate a specific override 

policy. In contrast, the State 1 policy dictated that supervisors should check 10 percent of all 

assessments, to ensure that any overrides found in the process were appropriately done. In general, 

there was not much specific guidance on the use of overrides in the written policies and procedures 

for any of the three states. To help explore the nature and use of overrides, staff were asked a series 

of questions related to this topic (see Table 1.5).

18 

These questions explored the common reasons for overrides, the process to obtain an 

override, and whether certain types of juveniles and/or offenses lent themselves to an overreliance 

on overrides. The variation in response rates was attributed to the nature of the three questions. 

For example, if interviewees indicated that overrides were not allowed at their agency, then they 

were not asked the following two questions. Additionally, if interviewees were not OYAS 

administrators, then they were potentially unfamiliar with the practices and policies surrounding 

overrides. Survey respondents were asked if overrides were allowed at their agency, but were not 

asked if specific offenses required overrides, or whether the OYAS had limitations that led to the 

use of overrides. Therefore, the survey responses are displayed for the first item only in Table 1.5. 

Table 1.5. Interview and Survey Sample Descriptive Statistics for Override Policies 

Variable State 1 State 2 State 3 Combined 

Are Overrides Allowed at 

Agency? n (%) n (%) n (%) n (%) 

Interviews 

Yes 59 (96.7) 35 (97.2) 57 (76.0) 151 (87.8) 

No 2 (3.3) 0 (0.0) 14 (18.7) 16 (9.3) 

17 Overrides are not thoroughly explored here. More information concerning override decisions can be found in the 

Professional Override of Assessment substudy (Substudy 4). 18 See Q44d-f under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

70 
 

 

 

 

Table 1.5. Interview and Survey Sample Descriptive Statistics for Override Policies 

Variable State 1 State 2 State 3 Combined 

Unsure 0 (0.0) 1 (2.8) 4 (5.3) 5 (2.9) 

Total 61 (100.0) 36 (100.0) 75 (100.0) 172 (100.0) 

Surveys 

Yes 280 (86.7) 183 (90.6) 188 (85.5) 651 (87.4) 

No 43 (13.3) 19 (9.4) 32 (14.5) 94 (12.6) 

Unsure - - - -

Total 323 (100.0) 202 (100.0) 220 (100.0) 745 (100.0) 

Must Override 

Specific Offenses? n (%) n (%) n (%) n (%) 

(Interviews only) 

Yes 10 (22.7) 15 (60.0) 17 (41.5) 42 (38.2) 

No 25 (56.8) 9 (36.0) 19 (46.3) 53 (48.2) 

Unsure 9 (20.5) 1 (4.0) 5 (12.2) 15 (13.6) 

Total 44 (100.0) 25 (100.0) 41 (100.0) 110 (100.0) 

Specific Limitations of 

OYAS Lead to Overrides? n (%) n (%) n (%) n (%) 

(Interviews only) 

Yes 16 (43.2) 11 (47.8) 21 (63.6) 48 (51.6) 

No 17 (45.9) 12 (52.2) 8 (24.2) 37 (39.8) 

Unsure 4 (10.8) 0 (0.0) 4 (12.1) 8 (8.6) 

Total 37 (100.0) 23 (100.0) 33 (100.0) 93 (100.0) 

Due to missing data, totals are less than the reported sample size. 

Overall, 87 percent of interviewees and survey respondents confirmed that overrides were 

allowed at their agencies/facilities. However, it was unclear if there were official policies to be 

followed in the override process at the agency level. Some interviewees reported that supervisor 

approval was needed to conduct an override, while others indicated that staff retain total discretion. 

Despite an inconsistent perspective on override policies, interviewees indicated that overrides were 

infrequently used. A substantial amount of State 3 interviewees (nearly 19%) stated that overrides 

of OYAS risk level were not allowed, while nearly 100 percent of State 1 and State 2 interviewees 

indicated that overrides were allowed at their respective agencies. To illustrate, one State 2 

interviewee indicated his/her agency allows for overrides to the OYAS risk level (which is tied to 

the supervision level), and has written policies and procedures for this process. Staff do not 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

71 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

complete overrides themselves, but make recommendations to a supervisor, who will then decide 

whether to complete the override. 

In addition, for those who answered in the affirmative, there was much variation in 

whether specific offenses were automatically overridden (e.g., sex offenses). A substantially 

higher portion of State 3 interviewees perceived the OYAS to have specific limitations that lead 

to the need for overrides (at nearly 67%, compared to about 43% for State 1 and 48% for State 2). 

This may have been a product of three points. First, the State 3 staff reported serving a more diverse 

population of youths. Second, when the interviews were conducted, the OYAS had not yet been 

validated specifically for their state, while State 1 and State 2 have had the OYAS validated on 

their respective populations. Third, State 3 interviewees more commonly noted the need (or in 

some cases, the agency policy) to override sex offenders to high risk. Of the 64 interviewees that 

mentioned sex offenders scoring low as a common reason for overrides, 21.9 percent were from 

State 1, 29.7 percent were from State 2, and 48.4 percent were from State 3. However, this result 

was somewhat tempered by the web-based survey results, as a higher proportion of State 2 

respondents indicated that sex offenses was a common reason for the use of an override (84.1% 

affirmative, compared to 76.9% in State 3, and 70.8% in State 1). 

State-specific override practices. Interviewees in all three states enumerated the specifics 

of when overrides were likely to occur at their respective agencies. In State 3, overrides were 

typical when a youth had been adjudicated for a sex offense, or when collateral information (e.g., 

information provided from schools, families, court records) indicated an override was necessary. 

This reflects many interviewees’ perceptions of the “hierarchy of information” regarding OYAS 

scoring, as most value collateral information more than information from the youth interviews. 

State 3 interviewees also noted that supervisors must approve and sign any overrides to risk level. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

72 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In State 2, the interviewees indicated that overrides to risk level simply required a “good

reason.” Common reasons for overrides included offense-based characteristics such as sex 

offenses, and serious and violent offenses (particularly including the involvement of guns). In this 

way, the use of an override appeared to be compensation for what interviewees perceived to be the 

potential for a lack of punishment in court decision-making. In general, the State 2 interviewees 

indicated that overrides were not commonly used, but allowed if needed. 

Like State 3, State 1 interviewees indicated that conflicting answers from collateral 

information may lead to an override of risk level. State 1 interviewees also indicated that supervisor 

approval and a narrative justification for the override were required. Similar to State 2, other 

reasons for overrides were more charge-based, including sex offenses, human trafficking cases, 

guns/violent offenses, or a long history of juvenile justice involvement. Lastly, positive urine 

toxification tests and frequent disciplinary infractions inside residential facilities may also have 

triggered a risk level override. 

Among many of the interviews, there was some question as to the “hierarchy of 

information,” that is, which source it is appropriate to draw from most heavily when information 

from the youth interviews, case records, or other sources of collateral information conflict.19 For 

example, some interviewees indicated that information from the youths’self-report was the highest 

on the hierarchy, and therefore score the items according to how the youths complete the selfreport survey. Moreover, some interviewees have omitted questions from the interview that are 

also on the self-report, as a method of shortening the interview process. However, the self-report 

survey was designed as a supplementary source of information, and not a replacement of the 

19 As designed, if there is any confirmatory evidence of a risk/need factor, staff should score the items as a risk factor. 

Interview data indicate that in some sites, however, staff gave preference to different sources of information and may 

have been underscoring items and assessments, which may have led to some overrides. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

73 
 

 

 

 

 

 

 

 

 

 

 

 

 

interview guide. Other interviewees indicated that collateral information (e.g., school records, 

information from family) was the highest on the information hierarchy, and would use this 

information often as a justification for an override. Evidence of these patterns will be seen in the 

interview findings as well. 

Juvenile Justice Personnel Interview Findings 

How Agencies Use the OYAS. The OYAS was designed to assist in several decision 

points throughout the juvenile justice process. Uses of the OYAS may include identifying risk 

level, determining supervision levels, making treatment and service referrals, noting responsivity 

strengths and barriers, allocating resources within an agency, developing specialized caseloads, 

and/or monitoring youths and system progress in addressing risk and needs. However, usage 

practices may look different based on the job role of the interviewee, and their proximity to the 

OYAS. For example, judges and magistrates (nearly 10% of the sample) often receive a general, 

summarized version of the OYAS information, including major treatment needs and overall risk 

level. This information is most often included in the pre-sentencing investigation (PSI) report, 

which judges and magistrates may use to assign the type (e.g., family, drug, peers) and quantity of 

treatment. In general, the judges and magistrates place differing emphases on the OYAS 

information and the PSI report, and use this information in different ways (e.g., informing 

treatment mandates), showing how usage practices may differ, even within a given job role. 

Another example are treatment staff, who may not administer the OYAS, but likely use its 

information in treatment and aftercare plans, or to make placement decisions within facilities. 

During the interviews, staff identified numerous ways in which the OYAS was used 

throughout their agencies/facilities. Interviewees were asked if the OYAS assists in specific 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

74 
 

 

 

 

 

 

decision points in the juvenile justice system.20 The results for these questions are presented in 

Table 1.6. The variation in response rates may be indicative of the interviewees’ job roles, with 

those who were not frontline staff being potentially less familiar with the specific ways in which 

the OYAS information is used. Variation in responses is also noted in Table 1.7 regarding whether 

OYAS information is shared with outside referral agencies. 

Table 1.6. Interview Sample Descriptive Statistics for How Agencies Use the OYAS 

Does Your Agency Use the 

OYAS Assessment State 1 State 2 State 3 Combined 

Information to...

Allocate Resources in 

Agency? n (%) n (%) n (%) n (%) 

Yes 24 (38.1) 11 (29.7) 15 (22.4) 50 (29.9) 

No 23 (36.5) 22 (59.5) 37 (55.2) 82 (49.1) 

Unsure 16 (25.4) 4 (10.8) 15 (22.4) 35 (21.0) 

Total 63 (100.0) 37 (100.0) 67 (100.0) 167 (100.0) 

Develop Specialized 

Caseloads? n (%) n (%) n (%) n (%) 

Yes 21 (34.4) 7 (20.0) 13 (19.7) 41 (25.3) 

No 30 (49.2) 26 (74.3) 48 (72.7) 104 (64.2) 

Unsure 10 (16.4) 2 (5.7) 5 (7.6) 17 (10.5) 

Total 61 (100.0) 35 (100.0) 66 (100.0) 162 (100.0) 

Determine Supervision 

Level? n (%) n (%) n (%) n (%) 

Yes 45 (73.8) 31 (86.1) 56 (77.8) 132 (78.1) 

No 11 (18.0) 4 (11.1) 9 (12.5) 24 (14.2) 

Unsure 5 (8.2) 1 (2.8) 7 (9.7) 13 (7.7) 

Total 61 (100.0) 36 (100.0) 72 (100.0) 169 (100.0) 

Measure Youths’ Progress? n (%) n (%) n (%) n (%) 

Yes 33 (53.2) 18 (52.9) 49 (71.0) 100 (60.6) 

No 21 (33.9) 12 (35.3) 16 (23.2) 49 (29.7) 

Unsure 8 (12.9) 4 (11.8) 4 (5.8) 16 (9.7) 

Total 62 (100.0) 34 (100.0) 69 (100.0) 165 (100.0) 

Match Youths to Services? n (%) n (%) n (%) n (%) 

Yes 55 (88.7) 29 (80.6) 47 (68.1) 131 (78.4) 

No 5 (8.1) 6 (16.7) 18 (26.1) 29 (17.4) 

Unsure 2 (3.2) 1 (2.8) 4 (5.8) 7 (4.2) 

Total 62 (100.0) 36 (100.0) 69 (100.0) 167 (100.0) 

20 See Q43a-h under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

75 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1.6. Interview Sample Descriptive Statistics for How Agencies Use the OYAS 

Does Your Agency Use the 

OYAS Assessment State 1 State 2 State 3 Combined 

Information to...

Assist in Diversion, 

Disposition Decisions? n (%) n (%) n (%) n (%) 

Yes 39 (63.9) 29 (80.6) 35 (51.5) 103 (62.4) 

No 17 (27.9) 6 (16.7) 19 (27.9) 42 (25.5) 

Unsure 5 (8.2) 1 (2.8) 14 (20.6) 20 (12.1) 

Total 61 (100.0) 36 (100.0) 68 (100.0) 165 (100.0) 

Additional Uses? n (%) n (%) n (%) n (%) 

Yes 14 (35.0) 3 (17.6) 13 (30.2) 30 (30.0) 

No 18 (45.0) 13 (76.5) 29 (67.4) 60 (60.0) 

Unsure 8 (20.0) 1 (5.9) 1 (2.3) 10 (10.0) 

Total 40 (100.0) 17 (100.0) 43 (100.0) 100 (100.0) 

Due to missing data, totals are less than the reported sample size. 

As can be seen, the results varied greatly for these decision points. For example, 78 percent 

of interviewees indicated the OYAS is used to determine supervision levels, but other areas saw 

less consistency in perceived use (e.g., assist in diversion and disposition decisions). One State 2 

interviewee suggested specifically that there needs to be trainings on youths’ strengths and barriers 

to help identify them in the responsivity sections of the OYAS. In fact, a sizeable number of the 

interviewees responded that they did not fill out the strengths and barriers consistently (mainly 

because these are trained as being optional).21 As such, the information may not have been 

routinely used for case planning or other purposes. In turn, staff may not have been clued in to 

when to use additional responsivity assessments and work to match youths to services based on 

these factors. Related, many interviewees felt strongly that there should be more mental health 

21 Several of the OYAS tools (i.e., Disposition, Residential, and Reentry) have optional strengths and barriers listed 

in each of the dynamic need areas. This information is often considered optional and some staff do not complete these 

sections. Some of the strengths and barriers overlap with risk items. See usage study 6 (Usage of OYAS Strengths and 

Barriers) for more detail on usage of these items. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

76 
 

 

 

 

 

 

 

 

 

 

 

 

 

items on the OYAS. This concern could potentially be alleviated by the systematic use of the 

responsivity factor sections on the OYAS tools. 

According to the interviewees, the main two uses of the OYAS information were to 

determine supervision level and match youths to services (approximately 80% in both instances), 

which reflect a general alignment with the principles of risk, need, and responsivity (Andrews et 

al., 1990). However, interviewees expressed difficulty in several areas. First, there was difficulty 

particularly in adhering to the risk principle (which dictates, among other things, that youths are 

to receive a quantity of treatment that is appropriate for their risk level). For example, many 

interviewees, along with using OYAS information to match youths to treatment, expressed the 

desire to refer low risk youths to more intensive services. In some cases, this led to the use of an 

override to justify such service provision, and in other cases, the personnel generally described 

bypassing risk levels and assigning youths treatment quantities based on their own professional 

discretion. On the opposite end of this concern, many other interviewees found that community 

resources were hard to navigate and/or largely unavailable (particularly in rural areas). Therefore, 

they found it difficult to use the assessment information to dictate any amount of treatment 

referrals. 

Another concern regarding using OYAS information to determine supervision level was 

that this created a “net-widening” effect, as more youths at one particular agency were being 

classified as “high risk” than before. These patterns of classification generally depended on the 

agency where the personnel was interviewed, but overall, there were concerns expressed about the 

manner in which risk level was tied to supervision and service levels. Case planning was also 

frequently mentioned as a challenge. For example, an interviewee from State 3 expressed 

frustration with using OYAS assessment results in case planning. S/he noted that staff had “no 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

77 
 

 

 

 

 

 

 

 

formal training on how to create [a case plan], so everyone’s case plan looks different [and] 

include[s] different things, [we] can’t delete or update [the] case plan, so everything stays even if

[mistakes are made].” In examining the wealth of responses on assessment information usage, it 

is evident that the full utility of the OYAS is not yet in practice in any of the states. 

Notably, the four most common uses were reflected in much of the three states’ written 

policies and procedures (i.e., determining supervision level, measuring youths progress, matching 

youths to services, and assisting in diversion/disposition/release decisions), whereas the less 

common uses were not. This suggests that written policy may dictate much of the risk and needs 

assessment usage within an agency/facility. Still, even those uses were not consistently as high as 

might be expected according to theory underlying optimal risk and needs assessment usage (Vieira, 

Skilling, & Peterson-Badali, 2009; Vincent et al., 2012). On the whole, just over 60 percent of 

interviewees perceived the OYAS as being useful for case decisions and measuring youths 

progress. 

The most between-state variation in responses is observed in whether OYAS information 

is used to allocate resources within the agency, measure youths’ progress while on supervision, or 

assist in diversion/disposition/release decisions. Less than one-third of interviewees indicated that 

OYAS information is used to allocate resources within the agency (ranging from about 21% to 

38% between the three states). This may include the creation of specialized units (i.e., those 

dedicated solely to low or high risk), or dedicating more funds to specific programs that are needed 

by a wide range of youths (e.g., substance use treatment). Whether the OYAS information is used 

to measure youths’ progress was also highly dependent on state context. While about 61 percent 

of the total sample responded in the affirmative, State 3 was seemingly an outlier, with 72 percent 

responding in the affirmative (compared to 53% in State 2, and 55% in State 1). Despite the lower 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

78 
 

 

 

 

 

 

 

 

 

 

 

 

 

numbers in State 1 and State 2, one interviewee in State 2 noted that s/he uses the OYAS to measure 

progress by sharing the results over time with the youths and their families—indicating where they 

started and where they are currently. Related, one State 1 interviewee noted that s/he used the 

OYAS, in combination with another assessment, to help determine rewards and sanctions to 

include on behavioral contracts for youths. 

Lastly, there was much variation in response to whether the OYAS information assists in 

making diversion/disposition/release decisions. Here, 81 percent of State 2 interviewees indicated 

that OYAS information is used for these decisions (compared to 60% in State 1 and 52% in State 

3). One State 3 interviewee added context to these results, indicating that in his/her agency, the 

OYAS Diversion Tool was often conducted after diversion was already decided for a youth. 

Therefore, conducting the Diversion Tool felt like a “waste of time.” This speaks to the importance

of setting up assessment processes so that personnel are able to see its impact on juvenile justice 

decision-making. 

A large portion (21%) of the combined interviewee sample was uncertain as to whether the 

OYAS information is used to allocate resources within their respective agencies. Interviewees 

from all three states indicated that the OYAS is rarely used for additional purposes. Other purposes 

that were stated though, include using the OYAS scores to determine the rewards and 

consequences on behavioral contracts for youths, using the OYAS information (i.e., risk and needs 

levels) when explaining supervision and programming decisions to parents, explaining disposition 

decisions to victims and community stakeholders, justifying contact requirements or home 

confinement (e.g., low risk youths reporting once a month, high risk reporting once a week), and 

using OYAS information as a starting point for collaboration with outside service providers on 

treating high risk areas. These additional uses are consistent with some of the “balanced 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

79 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

supervision” models, such as those outlined in Maloney, Romig, and Armstrong (1988). As such, 

the results indicate that these states may be able to more fully capitalize on the information 

available from risk and needs assessments. 

Sharing OYAS Assessment Information. Interviewees were asked whether their 

respective agencies shared the information gained in conducting an OYAS assessment to facilitate 

the work of treatment agencies and other placement sites.22 Information sharing can help ensure 

that resources from these secondary agencies are not spent duplicating assessments and can save 

resources. The risk and needs information in the OYAS can also help an agency to adhere to the 

principles of effective intervention. For example, an agency could have separate groups for low 

risk youths or provide higher risk youths with more intensive treatment services. Table 1.7 below 

contains the interviewees’ responses concerning sharing the assessment information with other

agencies. 

Table 1.7. Interview Descriptive Statistics for Sharing OYAS Information 

Variable State 1 State 2 State 3 Combined 

Does Your Agency Share OYAS 

Information? n (%) n (%) n (%) n (%) 

Yes 31 (54.4) 16 (50.0) 30 (45.5) 77 (49.7) 

No 17 (29.8) 9 (28.1) 27 (40.9) 53 (34.2) 

Unsure 9 (15.8) 7 (21.9) 9 (13.6) 25 (16.1) 

Total 57 (100.0) 32 (100.0) 66 (100.0) 155 (100.0) 

Due to missing data, totals are less than the reported sample size. 

Only half of the total sample indicated that their agencies share OYAS information with 

outside providers, though, those who do indicated that they do not have difficulty in doing so.23 

This low proportion may be a product of barriers to sharing information (e.g., youths/parents 

having to sign releases of information), or the lack of a systematized method/requirement of 

22 See Q51 under Youth Assessment Practices in Appendix A. 23 Interviewees were also asked if they have had any issues in sharing OYAS information (Q51b. “Are there any issues

with sharing this information?”). Due to too little variation in responses (with only three affirmative answers in the 

sample), these results were omitted from Table 1.7. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

80 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

sharing information, or that the court has not educated the secondary agencies on how the 

information may help to inform service delivery. In addition, the low level of information sharing 

may be a product of low buy-in for the OYAS—if staff do not see the value in conducting an 

OYAS or using it to make decisions, they may not recognize any value in sharing it with other 

agencies or personnel involved in a youth’s case. 

State-specific OYAS information sharing. Interviewees from all three states indicated that 

they may share OYAS information with outside individuals for specific purposes. In State 3, 

interviewees indicated that a shared automated system allows other agencies to view OYAS 

information, including case plan information as well as assessment results. This information can 

be made available to outside service providers such as treatment staff, counselors, and attorneys. 

Some interviewees have made it a regular practice to send the OYAS information along with 

treatment referrals, whereas others wait for outside agencies to request the information. 

State 2 interviewees also indicated that they share OYAS information across juvenile 

justice agencies (e.g., counties, judges, probation, and parole agencies) via shared access to the 

OYAS automated system. In addition, the information is often shared with counselors and 

residential placement staff, but there is no systematized method. Lastly, some interviewees 

indicated that OYAS information is shared indirectly through pre-disposition reports, which are 

available to individuals working with youths. 

Unlike State 3 and State 2, some State 1 interviewees indicated that they share the results 

of the OYAS with the youths’ families. Information is also often shared with children’s services

and mental health providers. Like the other two states, State 1 interviewees can give stakeholders 

view-only access to the OYAS automated system, so that staff across agencies can freely access 

this information. It is evident that all three states have the ability to share OYAS information, but 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

81 
 

 

 

 

 

 

 

 

 

 

 

do not necessarily choose to do so and have not systematized this process. This is an area where 

agencies could further capitalize on the usefulness of JRNA information. 

Aggregate Data Usage. One of the benefits of conducting risk and needs assessments is 

that it allows states and local entities the ability to examine data produced during the assessment 

process. Interviewees were asked whether their agency uses aggregate OYAS data to inform 

decision-making at that level.24 The results are displayed in Table 1.8. Similar to the previous tables 

illustrating OYAS usage practices, the varying response rates in Table 1.8 below are likely due to 

variation in job roles. However, in the aggregate data usage context, it is more likely that missing 

data come from frontline interviewees’ potential unfamiliarity with data aggregation processes. As 

such, it is likely that interviewees in administrative roles were more familiar with this aspect of 

data management. 

Table 1.8. Interview Descriptive Statistics for Aggregate Data Usage 

Variable State 1 State 2 State 3 Combined 

Agency Uses Aggregate 

Data? n (%) n (%) n (%) n (%) 

Yes 

No 

Unsure 

Total 

22 (36.1) 

21 (34.4) 

18 (29.5) 

61 (100.0) 

10 (31.3) 

15 (46.9) 

7 (21.9) 

32 (100.0) 

10 (17.5) 

37 (64.9) 

10 (17.5) 

57 (100.0) 

42 (28.0) 

73 (48.7) 

35 (23.3) 

150 (100.0) 

Due to missing data, totals are less than the reported sample size. 

The overall average affirmative response across the states for using aggregate data was low 

(28%), with State 3 having the lowest proportion of affirmative responses at 17.5 percent (versus 

31.3% in State 2, and 36.1% in State 1). This may have been the product of two points. First, State 

3 was the last of the three states to adopt the OYAS, and may not have had a plan in place for 

aggregate data at the time of data collection for the current study. Second, there may have been a 

24 See Q35 under Implementing the OYAS in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

82 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

lack of faith in the accuracy of the information as the OYAS was not yet validated in the state. A 

large proportion of the interview sample (nearly one in four interviewees) were also unsure 

whether aggregate data are used in any way. These findings suggest that if aggregate OYAS 

information is being generated and utilized routinely, this is not being shared with the frontline 

staff and this may also affect satisfaction levels. Staff from two states were able to provide 

examples of limited uses of aggregate data. In State 1, an agency justified the allocation of an 

evening reporting center for high risk youths. In State 2, an agency used aggregate data to justify 

a special unit at a correctional facility. In addition, some interviewees in State 2 also indicated that 

aggregate OYAS data might be used in the future to justify paying for licensure training for key 

staff to be able to provide substance abuse services since a large number of their youths struggle 

with substance use. 

However, one frontline interviewee did note that aggregate data was shared with him/her 

at the agency, which ultimately reduced his/her confidence in the OYAS, as many of the “low 

risk” youths recidivated. This speaks to the balance of aggregating data during early 

implementation, while maintaining confidence in the tools and continuing proper usage practices. 

This also speaks to the importance of tempering expectations about the impact of a risk and needs 

assessment, as a degree of recidivism is to be expected at all risk levels, despite intervention efforts. 

State-specific aggregate data usage. Interviewees in all three states emphasized that 

aggregate data collection and usage seemed to be concentrated at the state level. State 3 

interviewees indicated that the central state office sends emails to the staff with aggregate OYAS 

information, though some interviewees complained that these data are outdated because the 

counties depend on the state office to aggregate the data and send it out to the counties. It is for 

this reason that interviewees claimed the aggregate information is not used for any specific 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

83 
 

 

 

 

 

 

 

 

 

 

 

 

 

purpose. State 3 interviewees agreed that it would be advantageous to be able to pull and share 

aggregate information as needed at the local level. One state-level State 3 interviewee expressed 

the desire to compare data on mental health outcomes and treatment records to aggregate OYAS 

data, in order to explore or confirm that the assessment results are matching more general service 

usage patterns of the youths in that area. 

The only two uses of aggregate data in State 2 interviews were mentioned above. In 

general, State 2 interviewees believed that JDAI efforts may collect and share aggregate OYAS 

information more than the individual counties, but none of the interviewees could elaborate on the 

process. The only use of aggregate data in State 1 interviews was mentioned above. As was stated 

in the “State Context for Implementation and Usage” section, State 1 collects data on programs 

that are implemented to keep youths in the community and related policy changes and this may 

include OYAS information. In general, State 1 appeared well-positioned, due to ongoing stateand county-level efforts, to collect and use aggregate OYAS data to improve supervision practices 

and treatment delivery to youths though it had not capitalized on that effort at the time the 

interviews took place. It has, however, promoted the implementation of the assessment system as 

helping to facilitate reducing the use of state residential placements relative to community-based 

alternatives. 

Attitudes Toward Risk and Needs Assessment and the OYAS. Table 1.9a shows the 

means and standard deviations for the three questions that gauged interviewees’ attitudes about

risk and needs assessment generally, and the OYAS specifically. Interviewees were asked to rate 

their attitudes and perceptions on a scale of 1 – 5 with 5 indicating greater support for each 

statement. Figure 1.3 outlines the detailed distributions of responses to these items for the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

84 
 

 

 

 

 

 

 

 

 

 

combined interview sample.

25 These questions gauged whether staff feel that the OYAS benefits 

the youths, their agencies, and whether it enhances fairness in the juvenile justice decision-making 

process. 

Table 1.9a. Interview Descriptive Statistics for the General Attitudes toward Risk and Needs 

Assessment and the OYAS 

Variable State 1 State 2 State 3 Combined 

x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

OYAS Benefits the 

Youths (N = 201) 3.75 (.87) 3.40 (1.21) 3.59 (0.99) 3.60 (1.01) 

OYAS Benefits the 

Agency (N = 192) 3.95 (.90) 3.57 (1.24) 3.77 (1.15) 3.79 (1.10) 

JRNAs Enhance Fairness 

in Decision-making 3.88 (.85) 4.05 (.97) 3.62 (1.10) 3.80 (1.00) 

(N = 208) 

Total 3.86 (.87) 3.67 (3.42) 3.66 (1.08) 3.73 (1.04) 

Due to missing data, totals are less than reported sample size. 

Analyzing the results by state, State 1 interviewees gave a higher average score on whether 

the OYAS benefits the youths, and whether it benefits their agencies. This may have been due to 

the fact that the OYAS was implemented first in State 1, and had a higher proportion of state-level 

administrators in the interviewee pool. State 2 interviewees gave the highest average rating for 

whether risk and needs assessments in general enhance fairness in juvenile justice decisionmaking. A Chi-Square test revealed no significant differences between the states in their average 

level of agreement on these three items. There appears to be consensus across the states with regard 

to JRNA increasing fairness and benefiting the agency, more so than benefiting the youth directly. 

Meaningful differences arose when analyzing the sample by various job classifications. 

For example, those who work on the “frontline” (i.e., working directly with youths in supervisory 

or treatment capacities on a daily basis), versus those who work in another juvenile justice capacity 

25 See Q15-17 under General Agency/Facility Approach to Youth Assessment in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

85 
 

 

 

 

 

 

 

 

 

 

 

 

 

(e.g., state-level administrator, attorney, treatment supervisor). In the combined sample, a ChiSquare test revealed that the only statistically significant difference between frontline and nonfrontline interviewees’ average ratings was in whether risk and needs assessments reduce bias in 

juvenile justice decision-making (2

(8) = 25.05, p < .05, Cramer’s V = .35). On average, nonfrontline employees rated this item a 4.13 (sd = .93), while frontline employees rated this item a 

3.54 (sd = .98). 

One interviewee gave context to these results, stating that “Line officers don’t really like

[the OYAS], [but] supervisors base everything off it.” This illustrates some of the potential 

skepticism of the OYAS among frontline personnel; however, their average score for this item, 

though significantly lower than non-frontline personnel, still indicated that they were in general 

agreement that risk and needs assessments reduce bias. In terms of “basing everything” off of the 

OYAS, this runs in contrast to the importance of frontline personnel recognizing that the OYAS 

is a piece of the “rehabilitation puzzle,” and is meant to interact with existing processes and 

services in order to most appropriately service the youths. This notion of an “end-all, be-all” tool 

may impact buy-in from personnel who expect a risk and needs assessment tool to go above and 

beyond its intended uses as a result of how it is promoted in rollout and training. The exercise of 

balance and sensitivity in risk and needs assessment is necessary to inform juvenile justice 

decision-making to the proper degree. Another frontline interviewee expressed concern that staff 

were “relying too much” on the risk score, and essentially attributing this to a “worse kid,” rather

than a youth with a higher risk for recidivism. 

Aside from differences in the frontline and non-frontline interviewee perceptions, another 

potentially meaningful distinction is between the supervisors/managers and the rest of the sample, 

who were referenced earlier in the report as the “middle-managers” (i.e., these individuals were 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

86 
 

 

 

 

 

 

 

 

 

 

 

 

often former frontline workers who are now charged with supervising staff and enforcing policies, 

potentially including those associated with the OYAS). After accounting for this distinction, a ChiSquare test revealed no significant differences between supervisors/managers and the rest of the 

full sample in their level of agreement that the OYAS benefits the youths (2 

(7) = 16.82, p < .05, 

Cramer’s V = .29) or benefits their agency (2 

(7) = 22.10, p < .05, Cramer’s V = .34). On average, 

supervisors/managers rated the “Benefits the Youths” item a 3.57 (sd = 1.13), and the rest of the 

sample rated it a 3.61 (sd = .98). On the “Benefits the Agency” item, the supervisors/managers 

gave an average rating of 3.82 (sd = 1.14), and the rest of the sample rated it a 3.78 (sd = 1.09). As 

such, it appears that being in a supervisory role produces a mixed effect in terms of its impact on 

interviewees’ perceptions of the OYAS, and of risk and needs assessment in general. 

Figure 1.3 below clearly illustrates the stronger levels of agreement among the combined 

interview sample that the OYAS benefits the agency, and that risk and needs assessment enhance 

fairness in the juvenile justice decision-making process. From this figure, it is clear that those who 

were in direct disagreement with any of these three items represented a smaller percentage of the 

sample. In the event that an interviewee listed a range in their response (e.g., “from three to 3.5”, 

or “from three to four”), the midpoint was used (e.g., 3.25 or 3.5, respectively), and the response 

was then rounded to the nearest category for the sake of display in Figure 1.3. 

Continuing with the interviewees’ general attitudes toward JRNA, Table 1.9b shows the 

results of four additional questions that more specifically focused on the OYAS. Interviewees were 

asked whether OYAS information is a valuable part of the decision-making process26, whether it 

26 See Q42a, Q44f, Q47, and Q48 under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

87 
 

 

 

 

 

 

 

 

 

provides useful information regarding criminogenic and non-criminogenic needs, and whether 

there are limitations to the OYAS that lead to the use of overrides.27 

Figure 1.3. Detailed Distribution of Interview Responses for the General Attitudes toward Risk 

and Needs Assessment and the OYAS 

100 

90 

80 

70 

60 

50 n = 78 n = 74 n = 78 

n = 7 n = 10 n = 6 n = 18 n = 13 n = 12 

n = 58 

n = 36 

n = 53 n = 40 

n = 59 n = 59 

0 

10 

20 

30 

40 

% Agreement Level 

Benefits the Youth (N = 201) Benefits the Agency (N = 192) Enhances Fairness in Decisionmaking (N = 208) 

Strongly Disagree Disagree Neutral Agree Strongly Agree 

Table 1.9b. Interview Descriptive Statistics for the General Attitudes about the OYAS 

Variable State 1 State 2 State 3 Combined 

Valuable Part of Decisionmaking Process28 n (%) n (%) n (%) n (%) 

Yes 36 (72.0) 31 (83.8) 49 (79.0) 116 (77.9) 

No 10 (20.0) 4 (10.8) 12 (19.4) 26 (17.4) 

Unsure 4 (8.0) 2 (5.4) 1 (1.6) 7 (4.7) 

Total 50 (100.0) 37 (100.0) 62 (100.0) 149 (100.0) 

Useful Information Regarding 

Criminogenic Needs n (%) n (%) n (%) n (%) 

Yes 49 (84.5) 28 (90.3) 61 (91.0) 138 (88.5) 

No 6 (10.3) 3 (9.7) 4 (6.0) 13 (8.3) 

Unsure 3 (5.2) 0 (0.0) 2 (3.0) 5 (3.2) 

Total 58 (100.0) 31 (100.0) 67 (100.0) 156 (100.0) 

Useful Information Regarding 

Non-Criminogenic Needs n (%) n (%) n (%) n (%) 

Yes 27 (58.7) 13 (56.5) 31 (52.5) 71 (55.5) 

No 13 (28.3) 8 (34.8) 23 (39.0) 44 (34.4) 

27 Throughout the interviews, personnel also mentioned overriding individual items on the OYAS due to youth 

dishonesty, though overrides to risk level were far more common, and also the proper way to conduct an override 

according to the intended design of the assessment. 28 The interview guide was revised after the State 1 interviews had already begun. Therefore, some State 1 interviewees 

(n = 8) responded to a slightly different question from question 42a listed in Table 1.9b. Instead of answering, “Do

you believe that the information from the OYAS is a valuable part of the decision-making process?”, some State 1

interviewees answered the following: “Do you think the OYAS provides valuable information?”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

88 
 

 

 

 

 

 

 

 

Table 1.9b. Interview Descriptive Statistics for the General Attitudes about the OYAS 

Variable State 1 State 2 State 3 Combined 

Unsure 6 (13.0) 2 (8.7) 5 (8.5) 13 (10.2) 

Total 46 (100.0) 23 (100.0) 59 (100.0) 128 (100.0) 

Limitations Lead to Overrides n (%) n (%) n (%) n (%) 

Yes 16 (43.2) 11 (47.8) 21 (63.6) 48 (51.6) 

No 17 (45.9) 12 (52.2) 8 (24.2) 37 (39.8) 

Unsure 4 (10.8) 0 (0.0) 4 (12.1) 8 (8.6) 

Total 37 (100.0) 23 (100.0) 33 (100.0) 93 (100.0) 

Due to missing data, totals may be less than the reported sample size. 

Compared to State 1 and State 3, State 2 interviewees expressed slightly more favorable 

attitudes toward the OYAS. Nearly 84 percent indicated it is a valuable part of the decision-making 

process, 92 percent indicated that the OYAS provides useful information regarding criminogenic 

needs, and about 48 percent indicated that the OYAS has specific limitations that lead to overrides. 

State 3 had a much higher proportion of interviewees than State 1 or State 2 who indicated that the 

OYAS has specific limitations that lead to overrides (at about 64%). This may be attributed to the 

fact that the OYAS was not yet validated in State 3 at the time of the interviews, and may have 

had legitimate limitations that lead to the need for overrides with their juvenile population (e.g., 

State 3 has more variation in youths’ demographics characteristics in comparison with State 1 or 

State 2—See Table 2.7 in Section II). Lastly, though the vast majority of interviewees from all 

three states expressed that the OYAS provides valuable information regarding criminogenic needs, 

only slightly more than half of the interviewees from each state indicated that the OYAS provides 

valuable information regarding non-criminogenic needs. This may be due to either the 

underutilization of the strengths and barriers sections of the tools by some staff or the feeling of 

some that how the OYAS measures strengths and barriers is not sufficient. There is also likely to 

be some variation across states and sites due to how staff are trained and oriented to that section 

of the tool. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

89 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Chi-Square tests were conducted to determine whether there were differences between the 

states on these items. No significant differences were found. These results suggest somewhat 

similar general attitudes toward the OYAS between the three states, when each state sample was 

analyzed collectively (i.e., when not considering interviewee characteristics such as job 

classification). Analyzing the full sample by job classification, the only significant difference in 

results for the four general attitudes items was between the non-supervisory treatment staff (i.e., 

those who provide the day-to-day substance use or mental health treatment to the youths), and staff 

in other roles. Non-supervisory treatment staff were significantly less likely to indicate that the 

OYAS provides valuable information (2 

(2) = 8.17, p < 05, Cramer’s V = .23). It is possible that 

these individuals’ job roles relied more on information from secondary assessments or responsivity 

assessments that target mental health or substance use. No other significant differences on these 

items by job classification were found. 

Interviewees elaborated on their concerns that the OYAS has limitations that lead to 

overrides. For example, some suggested that status offenders consistently receive a “low risk”

designation, but are typically the youths who are in need of the most services. These youths, for 

example, may be experiencing significant issues in the home or at school, but may not receive 

services since such cases are often diverted.

29 As risk level is often tied to service provision (i.e., 

program eligibility may require a higher risk level designation), overrides are sometimes used to 

compensate. This scenario highlights the balance between risk, criminogenic need, noncriminogenic need, and how some programmatic elements (i.e., eligibility requirements) may 

impede the accuracy of a youth’s assigned risk level. 

29 The OYAS diversion tool comprises mainly current offense and juvenile justice history items with two questions 

on familial arrest and parental supervision of the youth. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

90 
 

 

 

 

 

 

 

 

 

 

 

 

Overall Satisfaction and Levels of Support for the OYAS. Support for the OYAS was 

identified through two questions and is presented in Table 1.10. First, staff were asked to rate their 

personal satisfaction with the OYAS (1 being “not at all satisfied” and 5 being “very satisfied”).

30 

Second, staff were asked to rate the overall level of staff support for the use of the OYAS (1 being 

“not at all supportive” and 5 being “very supportive”).

31 This differed from the previous question 

slightly in that it gauged the interviewees’ perception of the way the OYAS is used throughout the 

agency, thus moving the focus from the interviewee to his/her colleagues. 

Table 1.10. Interview Descriptive Statistics for the Overall Satisfaction and Levels of Support 

for the OYAS 

Variable State 1 State 2 State 3 Combined 

Rate your opinion on the 

following from 1 (low) to x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

5 (high)...

Personal Satisfaction with 

the OYAS (N = 200) 3.75 (.74) 3.67 (0.82) 3.34 (0.92) 3.56 (.86) 

Overall Level of Staff 

Support for the Use of the 3.49 (1.01) 3.26 (1.13) 2.93 (1.14) 3.21 (1.11) 

OYAS (N = 200) 

Due to missing data, totals are less than the reported sample size. 

Many interviewees’ personal level of satisfaction was at a four or higher (35.5% of the full 

sample), with the mean level of satisfaction falling at 3.56 (sd = .87). Similarly, nearly half of the 

respondents (48%) rated staff support at a four or higher. The mean level of staff support from 200 

responses was 3.21 out of five (sd = 1.13). The average perceived staff support for the OYAS was 

not only lower than personal support for the OYAS, but it had more variation, indicating that 

interviewee responses on this item were more dispersed around the mean. Essentially, there was 

more consensus around personal support for the OYAS than there was regarding perceived overall 

staff support for the use of the OYAS. 

30 See Q54 under Youth Assessment Practices in Appendix A. 31 See Q22 under General Agency/Facility Approach to Youth Assessment in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

91 
 

 

 

 

 

 

 

There was meaningful variation between the states when analyzing the combined sample. 

There were significant differences in both the personal satisfaction with the OYAS (F = 5.216, df 

= 2, p < .05), and the perceived overall satisfaction with the OYAS (F = 4.397, df = 2, p < .05) 

between the states. State 3 presented the lowest average score for the level of staff support for the 

use of the OYAS, at 2.93, but a significantly higher personal level of satisfaction with the OYAS. 

State 1 interviewees exhibited a significantly higher perception of the overall satisfaction with the 

OYAS than State 2 or State 3 interviewees. Overall, personal satisfaction with the OYAS appeared 

to be higher than the perceived overall level of staff support for the use of the OYAS. Essentially, 

interviewees noted that their agency peers had more negative perceptions of the OYAS than they 

did personally. Figure 1.4 illustrates further the interviewees’ higher ranking of their own personal

satisfaction with the OYAS, over their perceived level of support for the use of the OYAS by their 

peers. Similar to Figure 1.3, the scores in Figure 1.4 are rounded to the nearest category for the 

sake of display. As can be seen, personal satisfaction with the OYAS trends toward the positive 

side of the rating scale while a greater degree of “neutrality” arises when rating the second item, 

as this became the modal category. 

Figure 1.4. Interview Detailed Distribution of Responses for the Overall Satisfaction and Levels 

of Support for the OYAS 

100 

90 

80 

70 

60 n = 97 

% in Satisfaction/Support Category 50 

40 n = 65 n = 60 n = 58 30 

20 n = 29 n = 16 10 n = 4 

0 

Personal Satisfaction with the OYAS (N = 200) Perceived Level of Staff Support for Use of OYAS (N 

= 200) 

n = 16 

n = 32 n = 23 

Not At All Satisfied/Supportive Not Satisfied/Supportive Neutral 

Satisfied/Supportive Very Satisfied/Supportive 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

92 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Qualitative Evidence in Support of the OYAS. After being asked their personal level of 

support, and their perceived level of support for the OYAS from other staff members, interviewees 

were asked to identify several strengths of the tools. Two questions were asked of staff during 

interviews: one to identify perceived strengths of the OYAS in general, and another to assess their 

perceived strengths in how their agency uses the instruments.

32 Responses were analyzed and 

summarized in a word cloud using QDA Miner (Figure 1.5), and code frequencies for the top five 

most frequent codes are displayed in Figure 1.6. The word cloud in Figure 1.5 represents the code 

frequencies from the full sample, while a more comparative analysis is made between the states in 

Figure 1.6. 

The 191 responses to these questions yielded 15 codes regarding the strengths of the OYAS 

and its usage within the agencies/facilities. These codes are graphically displayed in Figure 1.5. In 

this visual representation of code frequencies, the font size of each code is relative to their 

frequency in the interviews, with the more frequently named strengths being represented by the 

larger fonts, and the less frequently named strengths being represented by the smaller fonts. As 

one example of the inductive coding process, many interviewees indicated that the OYAS is 

helpful, but did not indicate a specific reason as to why or how it is helpful. Whereas these were 

not initially coded, it became clear that a significant portion of the interviewees had positive 

feelings toward the assessment that were not being represented in the results. Therefore, a “Helpful 

(not otherwise specified [NOS])” code was created and defined as “Any mention of support that 

characterizes the assessment, assessment process, or its results as generally helpful, but does not 

indicate a specific reason for why, or in what capacity (i.e., is not otherwise specified).”33 

32 See Q26 under General Agency/Facility Approach to Youth Assessment and Q52 under Youth Assessment Practices 

in Appendix A. 

33 Please refer to the Semi-structured interview and analytic process sections for general details on the qualitative data 

analysis process. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

93 
 

 

 

 

Figure 1.5. Frequent Interview Sentiments for OYAS Strengths Word Cloud (N = 191) 

The list of strengths from the full sample were as follows, with the case frequencies in the 

parenthetical: (1) “Helps decision-making and service provision” (n = 85, 44.5%); (2) 

“Fair/consistent/objective” (n = 71, 37.2%); (3) “Identifies needs” (n = 64, 33.5%), (4) 

“Useful/comprehensive information” (n = 59, 39.0%); (5) “Identifies risk level” (n = 31, 16.2%); 

(6) “Helpful with case plans” (n = 28, 14.7%); (7) “Establishes baseline/monitors progress” (n = 

25, 13.1%); (8) “Evidence-based, validated, reliable” (n = 20, 10.5%); (9) “Good

implementation/support” (n = 19, 9.9%); (10) “Easy to use/access/organized” (n = 17, 8.9%); (11) 

“Builds rapport” (n = 16, 8.4%); (12) “Helpful (NOS)” (n = 12, 6.3%); (13) “Dictates level of 

supervision” (n = 9, 4.7%); (14) “Automated system useful” (n = 8, 4.2%); and (15) “Reduces 

workload” (n = 2, 1.0%). A number of these strengths are likewise listed as limitations by other 

interviewees (e.g., validity/reliability, case planning, implementation), which is noted and further 

explored in the next subsection “Qualitative Evidence for OYAS Concerns and Areas in Need of 

Improvement.”

The interviewees cast a wider net for strengths of the OYAS than they did for its 

limitations, meaning that interviewees named more unique strengths, some of which, in turn, have 

lower frequencies (e.g., see “Automated system useful” or “Reduces workload” above). This 

pattern seems to indicate that, while interviewees were able to name a diverse array of the strengths 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

94 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

of the OYAS, there was more consensus regarding its perceived limitations (which perhaps bear 

more influence on interviewees’ general attitudes toward the assessment, as evidenced in Tables 

1.9a-b). To illustrate, one State 1 interviewee commented that the OYAS automated system is 

quick to use and clear, and interfaces with other statewide initiative platforms. However, many 

more respondents (as demonstrated below) mentioned the automated system under limitations. 

Continuing with the strengths of the OYAS, Figure 1.6 shows the results of the top five 

most frequent codes from the full sample analyzed by state.34 The code frequencies represent the 

percent of respondents within each state who named a given strength in their response to these 

open-ended questions. It is important to note again that each state has its own unique characteristics 

surrounding the implementation and usage of the OYAS, including state- or county-specific 

initiatives and policies (see Figure 1.2). Additionally, each state had a unique makeup of 

interviewees, where certain types of positions (e.g., frontline vs. administrator) or job locations 

(e.g., probation vs. court) are overrepresented in one state (e.g. State 3 had an overrepresentation 

of frontline staff in probation when compared to State 1 or State 3, see Table 1.2). These 

characteristics, as outlined previously, likely influenced the results displayed in Figure 1.6. 

Figure 1.6 reveals that state context matters when considering the strengths of the OYAS. 

For example, 43.6 percent of State 2 interviewees (n = 17), 48.5 percent of State 1 interviewees (n 

= 32), and 25.6 percent of State 3 interviewees (n = 22) named “Fair/consistent/objective” as a

strength of the OYAS. The fact that a smaller proportion State 3 interviewees named the OYAS 

as fair, consistent, or objective may be a reflection of validation concerns for their state. State 2 

had a lower proportion of interviewees that indicated that the OYAS provides 

useful/comprehensive information. Approximately 20.5 percent of State 2 interviewees (n = 8), 

34 See Appendix C for definitions of each code displayed in Figures 1.5 and 1.6. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

95 
 

 

 

 

 

 

34.9 percent of State 3 interviewees (n = 30), and 31.8 percent of State 1 interviewees (n = 21) 

named “Useful/comprehensive information” as a strength of the OYAS. 

Figure 1.6. Top Five OYAS Strengths for Interviews by State 

n = 38 

n = 32 

n = 24 n = 21 

n = 10 

n = 19 n = 17 

n = 11 n = 8 

n = 4 

n = 28 n = 22 n = 29 n = 30 

n = 17 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

State 3 (N = 86) 

% of Cases from State 

State 1 (N = 66) State 2 (N = 39) 

State 1 had the highest proportion of interviewees state that the OYAS helps in decisionmaking and service provision, at 57.6 percent of interviewees (n = 38), followed by nearly half 

(48.7%) of State 2 interviewees (n = 19), and only 32.6 percent of State 3 interviewees (n = 28) 

named “Helps decision-making/service provision” as a strength of the OYAS. One State 2 

interviewee stated that, since the adoption of the OYAS Diversion Tool his/her agency has begun 

to divert far more youths, which suggests that the Diversion Tool is working to keep low risk 

youths in the least restrictive intervention available. This stands in contrast, however, to the 

aforementioned point that some personnel override low risk youths to moderate or high risk for 

the purpose of designating them to a higher level of service, despite their low prevalence of 

criminogenic need. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

96 
 

 

 

 

 

 

 

 

 

 

 

 

 

In terms of strengths that reflect the principles of effective intervention (Andrews et al., 

1990), interviewees from all three states named “risk” and “need” related strengths. Around 10.3 

percent of State 2 interviewees (n = 4), 19.8 percent of State 3 interviewees (n = 17), and 15.2 

percent of State 1 interviewees (n = 10) named “Identifies risk level” as a strength of the OYAS. 

A total of 28.2 percent of State 2 interviewees (n = 11), 33.7 percent of State 3 interviewees (n = 

29), and 36.4 percent of State 1 interviewees (n = 24) named “Identifies needs” as a strength of the 

OYAS. Though there was some variation in frequencies across the states, this may reflect a general 

familiarity with the principles of effective intervention that may have potentially been gained 

through the OYAS training process. 

For nearly every strength listed in Figure 1.6, the State 2 results more closely mirrored 

those of State 1 over State 3. This may potentially come from the similarities in youths’

demographics across these two states when compared to State 3, or from the fact that the OYAS 

had been validated for both State 1 and State 2 but not for State 3 at the time of the interviews. As 

demonstrated in Figure 1.2, State 1 and State 2 also implemented the assessment earlier than State 

3, and therefore have had more experience with it. 

Qualitative Evidence for OYAS Concerns and Areas in Need of Improvement. 

Interviewees were also asked to name their concerns regarding the OYAS and the perceived 

limitations of the tools. Through the nature of open-ended questioning, some interviewee concerns 

were more relevant to the implementation process than to the OYAS as a risk and needs assessment 

tool. These implementation concerns are named here and discussed more explicitly later in the 

report (see “Degree of Assessment Implementation” and “Views on Implementation Processes”

sections), as the current section focuses on concerns with the assessment itself (including its 

contents, processes, outcomes, etc.). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

97 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

To elicit staff concerns, staff were asked to identify some areas in need of improvement 

that are specifically related to the OYAS tools. Second, staff were asked what they see as some of 

the limitations of the tools based on how their agency currently uses the OYAS.

35 Third, staff were 

asked what concerns are raised by other staff regarding the OYAS.

36 Similar to the strengths, the 

195 cases that responded to any one of these questions about concerns/limitations were analyzed 

using QDA Miner, and are displayed graphically in Figures 1.7 and 1.8. Ten codes emerged from 

these questions. This represents a more consolidated response than the strengths of the OYAS 

(which elicited 15 codes), in that the interviewees focused on a smaller pool of similar 

concerns/limitations. 

As with the strengths codes, the concerns/limitations codes were assigned inductively. For 

example, the “Lack of validity/reliability” code is assigned to “Any mention of concern regarding 

the validity or reliability of the assessment. This includes any mention of uncertainty or 

subjectivity in scoring, but does not include any mention of concern about sex offenders, which 

are captured in the ‘inaccurate for sex offenders’ code”. In this example, although concerns about 

sex offenders’ risk not being captured accurately would technically be a validity concern, it was 

clear that concerns about sex offenders were so prominent that this concern merited its own code. 

Otherwise, this information would have been lost within the “Lack of validity/reliability” code.37 

Notably, “Lack of validity/reliability” is the code that captures any concerns about the need for 

“professional discretion,” where interviewees may have mentioned generally that the OYAS does 

not produce accurate risk levels, and feel that their own judgment serves as a more accurate 

measure of risk in some cases. 

35 See Q26 under General Agency/Facility Approach to Youth Assessment, and Q53 under Youth Assessment 

Practices in Appendix A. 

36 See Q25 under General Agency/Facility Approach to Youth in Appendix A. 37 See Appendix D for a full list and definitions of the OYAS limitations codes. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

98 
 

 

 

 

 

 

 

 

 

 

 

 

All ten codes from the full sample are displayed in Figure 1.7, with the top five most 

frequently named codes displayed in Figure 1.8. Figure 1.7 represents the results from the full 

sample, whereas Figure 1.8 provides a more comparative analysis between the states. As with the 

strengths, the code frequencies for concerns/limitations are relative to their font size, with the more 

frequently named codes represented by the larger fonts and the less frequently named codes 

represented by the smaller fonts. 

The full list of limitations/concerns are as follows, with the case frequencies that named a 

given weakness/concern in the parenthetical: (1) “Lack of validity/reliability” (n = 95, 48.7%); (2) 

“Items on assessment” (n = 86, 44.1%); (3) “Poor implementation/training” (n = 84, 43.1%); (4) 

“Time-consuming” (n = 51, 26.2%); (5) “Utilization” (n = 44, 22.6%); (6) “Youths/family 

manipulate scores” (n = 31, 15.9%); (7) “Difficulty in case planning” (n = 27, 13.8%); (8) “Lack 

of buy-in” (n = 25, 12.8%); (9) “Inaccurate for sex offenders” (n = 24, 12.3%); and (10) 

“Automated system issues” (n = 24, 12.3%).Concerning the most frequently noted 

weakness/concern around validity/reliability, State 3 interviewees were especially concerned that 

the OYAS had not been validated on their youths, and many interviewees described feeling that 

the assessment information was inaccurate, and therefore irrelevant to their daily work. 

There are some characteristics of the limitations/concerns responses that are unique from 

the strengths of the OYAS. Namely, the consensus drawn by nearly half of the full sample that a 

“Lack of validity/reliability” is a major concern of the OYAS (48.7%). This suggests a larger 

preoccupation within the sample about the validity and/or reliability of the tools (and tangentially, 

professional discretion). This sentiment has important implications for usage practices, as 

interviewees who do not believe the OYAS provides accurate information may likely be resistant 

to using its information in a meaningful way (e.g., measuring youths progress). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

99 
 

 

 

 

 

 

 

 

 

As mentioned previously, multiple limitations/concerns are also represented within the 

strengths. This is particularly remarkable in light of the open-ended nature of the interview 

questions, as well as the inductive coding process. Essentially, the interviewees tended to have 

strong positive or negative feelings about some of the same elements of the OYAS. Case planning, 

implementation/training, and the automated system seem to be areas of contention among the 

interviewees, though typically, more perceived these elements as a concern/weakness than as a 

strength. To highlight one of these, the automated system, the current study revealed many issues 

with the “practicalities” of the automated OYAS system, namely that interviewees did not see the

automated system as having a user-friendly interface. For example, an interviewee from State 2 

requested capabilities for uploading case notes from the OYAS interview to provide context for 

anyone who needs to work a youth in the future. 

Figure 1.7. Frequent Interview Sentiments for OYAS Limitations/Concerns Word Cloud (N = 

195) 

State 1 and State 2 interviewees did not express much concern specifically regarding the 

process of case planning with the OYAS, though this was a substantial concern for State 3. Only 

one State 2 interviewee and three State 1 interviewees expressed concern regarding case planning 

in relation to the OYAS, while 25.8 percent of State 3 interviewees (n = 23) did name “Difficulty

in case planning” as a concern. Overall, there were concerns regarding case planning across the 

three states. These concerns were typically coupled with requests for more detailed case plan 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

100 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

training, specific to helping frontline staff with incorporating case plans based on the OYAS results 

into their daily practice. 

Figure 1.8 shows a comparison of the results of the top six most frequent codes for OYAS 

concerns and limitations analyzed by state. The code frequencies in Figure 1.8 represent the 

percent of respondents within a state who named a given weakness or concern in their response to 

these open-ended questions. Again, it is important to consider each state’s unique context 

surrounding implementation and their unique pool of interviewee characteristics. Figure 1.8 shows 

divergence between states in multiple concerns, particularly for the “Items on assessment,” “Lack 

of validity/reliability,” and “Time-consuming”. State 2 interviewees are disproportionately 

concerned over the items on the assessment, as 65.8 percent (n = 25) named this as a concern. Only 

38.2 percent of State 3 interviewees (n = 34), and 39.7 percent of State 1 interviewees (n = 27) 

named “Items on assessment” as a concern. Across the three states, interviewees reported “cutting 

out” questions from the OYAS that they felt to be repetitive or time-consuming in order to shorten 

the assessment. For example, multiple interviewees noted that they disliked the item on the OYAS 

Disposition Tool related to self-esteem.38 

Validation is clearly impactful for obtaining buy-in and readiness for change, which is a 

product of both organizational and individual factors (Backer, 1995). However, items on the 

assessment remained a large concern for 39.7 percent of State 1 interviewees as well, the first 

38 As part of the OYAS disposition tool, the youth are asked, “On a scale of 1 to 10, how cool do you consider 

yourself?” They are then asked, “If your friends had to rate you, how cool would they rate you on a scale of 1 to 10?”

According to the OYAS scoring guide, “The purpose of this item is to determine the youth’s level of inflated selfesteem. Score this item as 0 if the youth has an appropriate level of self-esteem and does not believe that he/she is 

better than the system. If the youth demonstrates inflated self-esteem, score this item as 1. Examples of inflated selfesteem, include statements that he/she is smarter than the system, can manipulate or ‘play’ the system, or they paint 

an image of him or herself that is higher than what others would offer. If the youth rates himself/herself as an 8 or 

higher and states that friends would rate him/her as an 8 or higher, score this item as 1. If the youth rates one of the 

questions as 8 or higher and 1 as 7 or lower, the interviewer should take into account the totality of the interview. If 

the youth rates both statements as 7 or lower, score this item as 0.”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

101 
 

 

 

 

 

 

 

 

 

adopter of the tools, indicating that validation may not account for 100 percent of the concern over 

the assessment items. There was evidence in the interviews that staff who conduct the OYAS did 

not believe that certain items (e.g., high self-esteem) should be equated with antisocial behavior. 

There is also evidence that staff do not feel the assessment items go into enough detail about such 

things as gang membership, and that the “antisocial” scenarios built into the assessment are 

unrealistic. Here, there is a clash between tool validation and the “face validity” of some of the 

tools’ items, which repeats itself in the survey data as well. 

Figure 1.8. Top Five OYAS Limitations/Concerns for Interviews by State 

100 

n = 37 

n = 27 n = 31 

n = 17 n = 17 

n = 19 

n = 25 

n = 16 

n = 5 n = 8 

n = 39 n = 34 n = 37 

n = 29 n = 19 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

% of Cases from State 

State 1 (N = 68) State 2 (N = 38) State 3 (N = 89) 

In a similar vein, interviewees were differentially concerned about the validity and 

reliability of the tools, that is, the ability to accurately and repeatedly measure one’s risk for 

recidivism using the OYAS. 50 percent of State 2 interviewees (n = 19), 43.8 percent of State 3 

interviewees (n = 39), and 54.4 percent of State 1 interviewees (n = 37) named “Lack of

validity/reliability” as a concern. While this was concerning to a larger portion of State 1 

interviewees (for which the tools had been validated at the time of the interviews), all three states 

were genuinely concerned with this piece. This may speak to the need for ongoing training to 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

102 
 

 

 

 

 

 

 

 

 

monitor and improve tool reliability, and clearer communication regarding the validation process 

and what this means for the individual items, as well as the assessment as a whole. 

Again, validation appears to play an important role in the buy-in process, but is not the sole 

determining factor, as evidenced by State 1 interviewees’ concern for the tool’s validity. 

Anecdotally, interviewees also expressed concern that the OYAS was “less valid” for certain 

subgroups, such as females, giving the example that high rates of substance use tend to increase 

females’ risk level to an inaccurate degree on the OYAS, whereas they have more “needs” instead 

of actual risk to reoffend. As indicated in the beginning of this section, there was also a great deal 

of concern across the three states (12.3%) that the OYAS was not valid for the sex offender 

population, though these concerns merited a unique code (“Inaccurate for sex offenders”).

The amount of time for completion of the more comprehensive OYAS tools (i.e., 

Disposition, Residential, Reentry) was also a concern for all three states, though more so for State 

3 and State 1 than State 2. 13.2 percent of State 2 interviewees (n = 5), 32.6 percent of State 3 

interviewees (n = 29), and 25 percent of State 1 interviewees (n = 17) named “Time-consuming”

as a concern. This may also tie in to issues with the items on the assessment, as seemingly 

“irrelevant” items (e.g., high self-esteem) may have exacerbated the interviewees’ perception that

the OYAS is too lengthy. 

In terms of implementation and training, 42.1 percent of State 2 interviewees (n = 16), 41.6 

percent of State 3 interviewees (n = 37), and 45.6 percent of State 1 interviewees (n = 31) named 

this as a concern. Adding context to this concern, one interviewee explained that, as a member of 

the detention staff, the trainings were too “probation-focused,” and not as applicable to their role

as they would have liked. Numerous other interviewees noted that the lag in time from training to 

rollout was not only concerning, but that it decreased buy-in amongst the staff. Overall, these 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

103 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

results suggest a similar amount of concern regarding implementation across the three states, at 

least in the context of open-ended questioning format used in the interviews. The following section 

explores implementation concerns with more detail. 

Degree of Assessment Implementation and Views on Implementation Processes. 

Interviewees were asked to rate their agency’s status with respect to the implementation of the

OYAS.

39 All three states gave a relatively high rating for perceived degree of assessment 

implementation, ranging from 4.02 to 4.42 out of 5 (means and standard deviations are listed in 

Table 1.11). The impact of this sense of “arrival” to full implementation is seen in terms of its 

impact on continuous quality improvement practices and attention to fidelity in risk and needs 

assessment (see “Quality Assurance Measures” section). 

Table 1.11. Interview Descriptive Statistics for the Degree of Assessment Implementation 

State 1 State 2 State 3 Combined Variable (N = 63) (N = 42) (N = 79) (N = 184) 

x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

Current Status With 

Respect to Assessment 4.47 (.76) 4.02 (.99) 4.10 (1.11) 4.21 (.99) 

Implementation Process 

Due to missing data, totals are less than the reported sample size. 

State 1 had the highest average rating for degree of assessment implementation, which was 

expected given the combination of its position as the first state in the current sample to implement 

the OYAS and the additional impetus for its use at the state level (youths have to have an OYAS 

completed before placement at a state facility; see the “State Context for Implementation and 

Usage section” for additional information). State 2 had the lowest average rating on this item, 

despite it being the second of the three states in the current sample to adopt and that the tools. State 

3 was more similar to State 2 in their ratings and this may have been due to the fact that the OYAS 

39 See Q5 under Agency & Staff Characteristics in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

104 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

had not been validated in State 3.40 A one-way ANOVA indicated a significant difference in ratings 

between the interviewees in the three states (F = 4.366, df = 2, p = .05). Across job classification 

within the full sample, there were also no significant differences. In this way, it appears that that 

sample, across state contexts and job classifications, maintained similar views regarding their 

agency’s status with respect to the OYAS implementation process, and that overall, the OYAS had 

been fully implemented. 

Process of Buy-In for the OYAS. Three questions identified staff perceptions of the buyin process for the OYAS. The responses to these questions are presented in Table 1.12. First, the 

interviewees were asked if they were ever told the specific reasons for using the OYAS.

41 Second, 

the interviewees were asked if anyone within the agency took specific steps to facilitate buy-in.42 

Lastly, interviewees were asked if resources were introduced specifically to ease the rollout of the 

OYAS.43 The variation in response rates to these questions can be partially explained by job tenure, 

as some of the interviewees were not at their current agency at the time of the OYAS rollout. 

Therefore, they did not have information regarding the OYAS buy-in process, at least at initial 

rollout. 

Table 1.12. Interview Descriptive Statistics for the Process of Buy-In for the OYAS 

Variable State 1 State 2 State 3 Total Sample 

Told Reasons for Using the OYAS? n (%) n (%) n (%) n (%) 

Yes 51 (75.0) 29 (76.3) 63 (70.8) 143 (73.3) 

No 12 (17.6) 6 (15.8) 23 (25.8) 41 (21.0) 

Unsure 5 (7.4) 3 (7.9) 3 (3.4) 11 (5.6) 

Total 68 (100.0) 38 (100.0) 89 (100.0) 195 (100.0) 

Steps Taken to Achieve Buy-in? n (%) n (%) n (%) n (%) 

Yes 35 (54.7) 16 (45.7) 35 (45.5) 86 (48.9) 

40 As part of this project, the OYAS Residential Tool and OYAS Reentry Tool were validated for a state agency in 

State 3 (see McCafferty, Newsome, & Sullivan, 2017 for further details). Separately, the OYAS is being validated for 

three counties in State 3 (but not by this research team). 

41 See Q20 under General Agency/Facility Approach to Youth Assessment in Appendix A. 42 See Q21 under General Agency/Facility Approach to Youth Assessment in Appendix A. 43 See Q27 under Implementing the OYAS in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

105 
 

 

 

 

 

 

 

 

 

 

 

Table 1.12. Interview Descriptive Statistics for the Process of Buy-In for the OYAS 

Variable State 1 State 2 State 3 Total Sample 

No 21 (32.8) 14 (40.0) 39 (50.6) 74 (42.0) 

Unsure 8 (12.5) 5 (14.3) 3 (3.9) 16 (9.1) 

Total 64 (100.0) 35 (100.0) 77 (100.0) 176 (100.0) 

Resources Introduced to Ease 

Rollout? n (%) n (%) n (%) n (%) 

Yes 30 (53.6) 24 (82.8) 48 (72.7) 102 (67.5) 

No 14 (25.0) 4 (13.8) 9 (13.6) 27 (17.9) 

Unsure 12 (21.4) 1 (3.4) 9 (13.6) 22 (14.6) 

Total 56 (100.0) 29 (100.0) 66 (100.0) 151 (100.0) 

Due to missing data, totals are less than the reported sample size. 

Table 1.12 shows particularly low numbers in light of what is known about effective 

implementation (Vincent et al., 2012; Vincent et al., 2018). Over 70 percent (73.3 percent) of 

interviewees reported being told at least one reason for implementing the OYAS, with a similar 

proportion in each state, though slightly lower in State 3 (70.8%). Commonly cited reasons for the 

implementation of the OYAS included the move to evidence-based practices, the need to identify 

youths’ risk levels, and the desire for a validated tool. One interviewee stated that the OYAS 

trainers clearly communicated in the training that a statewide risk and needs assessment had the 

potential to reduce the overall incarcerated population of youths, and was adopted, at least in part, 

for this purpose. About half (48.9%) responded in the affirmative that steps were taken to achieve 

buy-in. As is discussed subsequently, this low number may have contributed to the levels of overall 

staff buy-in for the OYAS in all three states. To illustrate, interviewees in State 3 stated that they 

would have appreciated the OYAS trainers making the connection between risk and needs 

assessment, evidence-based practices, and the JDAI initiative more concretely. A different State 3 

interviewee suggested the positive impact of starting trainings by asking the staff, “What do you

want the tool to do for you?”

Over 67 percent of interviewees affirmed that resources were dedicated specifically to the 

OYAS rollout. Though State 1 had the highest average rating for degree of assessment 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

106 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

implementation, the State 1 respondents indicated the lowest level of agreement that resources 

were dedicated to ease the rollout of the OYAS. This may have been a symptom of their greater 

distance from initial implementation, as interviewees may not recall the specifics of the strategies 

used (with State 1 being the first of the three states to implement the OYAS), but can also reflect 

a belief that more could have been done to facilitate the rollout of the assessment tools and process. 

There was one significant differences in the results between the states, indicating that 

interviewees from the three states shared similar views about whether they were told the reasons 

for the OYAS. The significant difference was on the item regarding resources dedicated to easing 

the roll-out (χ2

(4)= 9.71, p < .05, Cramer’s V = .18). State 2 had the highest affirmative response 

rate concerning resources introduced to ease rollout and State 1 the lowest. This is illustrated by 

the fact that once the OYAS was selected and adopted in State 2, the risk and needs assessment 

task force requested that there be in-state trainers for ease of training coordination and increased 

buy-in. In terms of job classification, there were no significant differences on these three items. 

As non-supervisory staff were significantly less likely to indicate that the OYAS provides useful 

information (see Table 1.9b), this particular job classification may be a) be less familiar with the 

purpose of the OYAS, and b) perceive less benefit in its use. 

Though there is little research on this topic specifically, general insight on diffusion of 

policy and practice suggests that additional knowledge of the tool and/or the implementation 

process should translate into support for or satisfaction with the tool, as it reaches the agency (see 

Greenhalgh, Robert, MacFarlane, Bate & Kyriakidou, 2004). In light of the variation in whether 

personnel were told the reasons for using the OYAS, and whether resources were dedicated 

specifically to its implementation (i.e., about 70% gave an affirmative response to either item, see 

Table 1.12), we used multivariate linear regression to test whether knowledge of the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

107 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

implementation process increased the overall satisfaction with the tool. We controlled for whether 

interviewees administered the assessment, how long they have worked in the field (for descriptive 

statistics of the “Overall Satisfaction” item, see Table 1.10, and for descriptive statistics of the 

“OYAS Administrator” and “Job Tenure” variables, see “Descriptive Statistics and State-Specific 

Characteristics”). This test allowed for some insight into the conditional probability of satisfaction 

with the tool, given the two explanatory variables that we operationalized to represent knowledge 

of the implementation process. Specifically, knowledge of the implementation process was 

measured by (1) being told the reasons for the use of the OYAS, and (2) being aware of specific 

resources that were dedicated to its initial rollout. Both of these variables were recoded into binary, 

yes/no answers. People who were unsure or were not asked the question (i.e., legitimate skips) 

were coded as “no”, resulting in 200 valid cases for this analysis. 

The results of this regression indicated no significant differences in overall satisfaction 

with the OYAS between those with knowledge of the implementation process, and those without 

(N = 200). Further, the explanatory power of the independent variables was rather weak (R2 = 

.024), and an F-test revealed the model itself was non-significant (F = 1.192, p = .315). We also 

completed an independent samples t-test to compare the mean levels of satisfaction for people 

coded as yes/no on either explanatory variable (i.e., “Told Reasons Why” and “Resources Used to 

Ease Rollout”). These tests were also non-significant (t = -.592, p = .555, and t = .123, p = .902, 

respectively). As such, there are two possible conclusions: first, our explanatory measures may not 

have fully captured the concept of knowledge of the implementation process; second, it is also 

possible that knowledge of the implementation process (as measured in the current analysis), has 

less impact on the level of satisfaction with the tool than other characteristics (e.g., job setting, job 

role, proximity to the assessment, experience with its uses in decision-making processes). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

108 
 

 

 

 

 

 

 

 

 

 

 

 

State-specific strategies for garnering staff buy-in for the OYAS. Interviewees in all three 

states suggested that trainings and recertifications were the main strategies for garnering staff buyin for the OYAS. In the same vein, it was common for interviewees to describe the trainings as 

“piecemeal” or “inconsistent,” suggesting that, though they were often the only attempt at 

establishing buy-in, they could hinder it in some cases. Here, the importance of high quality, 

comprehensive, and evidence-based training is notable. More in-depth training on the purpose of 

the OYAS (including the purpose of each individual tool) was also suggested by a number of 

interviewees, as well as cross-training, that includes individuals from different agencies in 

different roles. As such, training is a necessary but insufficient ingredient for obtaining buy-in for 

risk and needs assessment. One state-level State 3 interviewee suggested the importance of asking 

staff, “What do you want the tool to do for you?” for establishing buy-in prior to conducting 

trainings. Though, the value in doing so is decidedly in whether trainers and/or agency leadership 

adjust their approach according to trainees' responses. 

Aside from staff training and recertification, there were meaningful similarities and 

differences in the methods used by each state to achieve buy-in for the OYAS. Numerous State 2 

interviewees noted the value in having all staff trained (not just those who conduct the tools), and 

stated that this helps all staff to understand the risk and needs assessment process and its value at 

their agencies. The interviewees in State 2 also noted that a multi-agency assessment task force 

was created specifically to help harness buy-in from stakeholders from all parts of the juvenile 

justice decision-making process. These task force members played an integral role in selecting the 

assessment and planning its statewide implementation. State 3 interviewees described capitalizing 

on the enthusiasm of those staff who bought-in (such as supervisors or newer staff) to be 

influencers for those who had not yet bought-in. Additionally, question and answer sessions were 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

109 
 

 

 

 

 

 

 

 

 

 

 

held twice annually to encourage the staff to ask questions and challenge their beliefs or 

assumptions about the inaccuracy or lack of utility of the OYAS. Lastly, interviewees described a 

long build-up to implementation, where state office employees and supervisors discussed the 

OYAS long before its actual adoption, which reportedly helped fuel anticipation. 

State 2 interviewees noted that “higher-ups” in the county agencies and state office met 

with agency staff to discuss the OYAS prior to the start of training. At some agencies, all staff 

were trained in the OYAS (despite whether they conduct the assessments or not), so that they were 

familiar with its purpose and the process. The staff were trained to include OYAS information in 

case plans, reinforcing the notion that the information is important and useful and better enabling 

staff to monitor youths progress (though, State 2 did not use the OYAS case plan). Lastly, State 2 

interviewees described keeping a website with all OYAS documents and policies, and having an 

OYAS task force as two other methods that have helped to encourage staff buy-in with the tools. 

This task force was comprised of probation officers and local community corrections staff, 

institution staff, special court staff, a trial judge representative, and a state-level representative. 

Their responsibilities included communicating with the developers of the OYAS throughout the 

validation process, and then drafting OYAS policy regarding certification and eligibility. 

In State 1, state-level interviewees described their state central office as small, and having 

completed “natural buy-in” through seeing the entire process of the development, adoption, and 

implementation of the OYAS. Site visits from the central office to the individual counties have 

garnered buy-in by encouraging a dialogue between the state and the counties. In this way, state 

employees and local supervisors were able to discover staff misconceptions about the OYAS and 

address them. At the county level, some supervisors made continuous efforts to include the OYAS 

in daily conversations, referring back to the assessment at various decision points, (e.g., “What did 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

110 
 

 

 

 

 

 

 

 

 

 

 

 

the OYAS say?”). Supervisors found it helpful to frame the OYAS as a “piece of the puzzle”, and

not an end-all-be-all tool for dictating placement and supervision decisions. Lastly, the OYAS 

trainings were open to all staff who are interested in them, which meant that there was a mix of 

those mandated to training with those who were naturally interested (and therefore demonstrated 

some natural buy-in). It is unclear, however, how many staff who were not mandated took 

advantage of this opportunity. 

Modifications to Usage/Implementation. Interviewees were asked about any changes 

made to the usage or implementation practices since the OYAS was first introduced in their 

respective agencies.

44 Their responses are presented in Table 1.13. The variation in response rates 

for this question can likely be attributed to job tenure, with some interviewees not having been at 

their agency long enough to have witnessed modifications to OYAS usage/implementation since 

initial rollout. 

Table 1.13. Interview Descriptive Statistics for Modifications to Usage/Implementation 

Variable State 1 State 2 State 3 Combined 

Modifications to Usage or 

Implementation Practices Since n (%) n (%) n (%) n (%) 

OYAS First Introduced? 

Yes 23 (39.7) 13 (41.9) 35 (56.5) 71 (47.0) 

No 29 (50.0) 17 (54.8) 23 (37.1) 69 (45.7) 

Unsure 6 (10.3) 1 (3.2) 4 (6.5) 11 (7.3) 

Total 58 (100.0) 31 (100.0) 62 (100.0) 151 (100.0) 

Due to missing data, totals are less than the reported sample size. 

Overall, less than half (47.0%) of the total sample indicated that changes have been made 

since the rollout of the OYAS. A relatively large proportion of State 1 interviewees (10.3%) were 

uncertain if there have been modifications to the usage or implementation practices since the 

OYAS was first introduced in their state. State 3 interviewees exhibited a substantially higher level 

44 See Q30 under Implementing the OYAS in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

111 
 

 

 

 

 

 

 

 

 

 

 

of agreement (56.5%) that changes have been made since rollout, which again may be a reflection 

of the shorter period of time since implementation and staff may have been more familiar with 

changes that have taken place. 

State-specific examples of modifications to OYAS usage/implementation. The 

interviewees from all three states gave examples of modifications to the usage and implementation 

practices for the OYAS. One common modification was the addition of various OYAS tools as 

roll-out progressed, typically including the Disposition Screener and the Residential Tool. These 

were two tools that many interviewees indicated were not initially used when the OYAS was first 

implemented in their state, but were then adopted to better serve the youths. 

State 3 interviewees indicated that three OYAS tools have been added since the initial 

rollout (i.e., Residential, Reentry, and the Disposition Screener). Additionally, some interviewees 

noted that the State 3 interview guide had been revised for the purpose of conducting 

reassessments. In terms of case plans, some interviewees indicated that these had been updated 

and were then used as an addition to the assessment process; however, State 3 did not employ one 

uniform case plan, and this change may reflect a change at one or more local-level agencies. One 

interviewee noted that training practices had been improved in conjunction with the statewide 

training group to include new OYAS information as it was captured. 

Like State 3, State 2 interviewees named several modifications to the usage and 

implementation of the OYAS. First, a detention center was required to conduct OYAS assessments 

(presumably the Detention Tool) in place of a different diversionary tool that is not part of the 

OYAS, the Detention Risk Assessment Instrument (DRAI). Second, OYAS policies and 

procedures had been updated to reflect these changes, as well as the addition of the Disposition 

Screener in the state. Lastly, State 2 importantly became one of the first statewide participatory 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

112 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

entities of JDAI starting in 2006, and continued its expansion across the state through 2016. JDAI 

calls for a comprehensive reform strategy to reduce the reliance on secure confinement for youths. 

After becoming a part of this initiative, the counties in State 2 began to link supervision levels to 

OYAS risk levels, and use this information to incarcerate as few youths as possible. 

State 1 interviewees also indicated numerous modifications that had been made to the usage 

and implementation of the OYAS since rollout. For example, at one residential facility, the 

assessment was conducted by an intake social worker at the facility, as opposed to being conducted 

by someone from the county level at the time of referral. In addition, one interviewee indicated 

that reassessments were not completed every six months for youths in residential placement, but 

only prior to a youth leaving the facility for the community. In general, State 1 interviewees 

identified fewer modifications to usage and implementation, which may be an indication of State 

1 having used the OYAS the longest, and that its processes surrounding OYAS usage and 

implementation are, consequently, more solidified. 

Qualitative Evidence in Areas for Improvement for the Implementation of the OYAS. 

One of the key objectives of the study is to understand how implementation of the OYAS and 

related processes were perceived by staff. Notably, the research team wanted to understand the 

ways in which staff thought the implementation of the OYAS might be improved upon. These 

areas for improvement were identified through two open-ended questions. First, staff were asked 

how the implementation process might be improved.45 Second, staff were asked what they would 

suggest to other agencies that are looking to implement a risk and needs instrument.46 The 

responses to these questions are analyzed by state, as the OYAS was implemented independently 

in State 1, State 2, and then State 3. 

45 See Q28a under Implementing the OYAS in Appendix A. 46 See Q29 under Implementing the OYAS in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

113 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

State-specific views on OYAS implementation processes. State 1 was the first of the three 

states included in this study to adopt the OYAS, and some of the interviewees’ concerns about

implementation reflected the fact that more time has passed since the initial rollout. State 1 

interviewees voiced concerns about “housekeeping items,” including “top-down” communication 

from state and agency leadership about recertification timelines and how to set up and utilize the 

automated system. Furthermore, concerns from the interviewees were reported about the lack of 

integration of the OYAS automated system with their existing framework, noting that case 

planning became more difficult when having to use two separate automated systems (i.e., the 

OYAS automated system and another extant case planning system) that do not communicate. This 

speaks to the importance of integrating service systems such that personnel are able to use them 

efficaciously. 

In terms of training, State 1 interviewees expressed the need for more frequent booster 

trainings, and more trainings on the automated system and case planning, specifically. Related, 

interviewees also requested a more user-friendly interface with the automated system, with 

capabilities for uploading case notes from the OYAS interview. One State 1 interviewee noted that 

poor systems integration can reduce buy-in for a tool and its automated system, and how beneficial 

it would be to have a case management system that imports OYAS information directly, which 

would reduce the time staff spend inputting data. 

The State 1 interviewees also noted that the time period between training and “going live”

with the assessment was critical, and should be as short as possible. Interviewees were also 

concerned about quality assurance, and some suggested having a smaller team of OYAS assessors 

to encourage consistency, as opposed to training all or most staff at an agency to conduct the 

OYAS. Regardless of who receives the training, interviewees voiced concern that all staff should 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

114 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

be made aware of the purpose and utility of the OYAS, with more of a focus on how to use the 

information after conducting the assessment. 

State 2 interviewees provided many similar suggestions for improving the OYAS 

implementation process, including better communication about timelines and the benefits and 

potential uses of the tools. Interviewees also requested more frequent training opportunities and 

ongoing quality assurance efforts. In addition, State 2 interviewees were acutely aware of the 

importance of garnering staff buy-in, and suggested providing information sessions for all 

stakeholders (i.e., not just those who would be conducting or using OYAS information). One 

interviewee suggested a trial period for the tools, where staff could decide for themselves how it 

would be the most useful for a department. In this way, while the tools may be mandated in a state, 

each agency, department, and/or unit could have the opportunity to determine how to best 

introduce each tool for their setting. 

State 3 interviewees expressed considerable concern over technical issues with the 

automated system, and difficulty in retrieving its data and forms. In terms of training, some State 

3 interviewees called for more statewide booster trainings to facilitate inter-agency learning and 

collaboration. These same interviewees indicated that too much discretion in implementation was 

given to the individual counties, which promoted inconsistency in implementation. Contrarily, 

other State 3 interviewees stated that they would appreciate a more decentralized training process, 

whereby staff questions and concerns could be addressed more locally (and thus, in a timelier and 

more relevant fashion). Regardless of whether the process is centralized or not, it is clear from the 

interviews that there are certain necessary components for implementation (e.g., a streamlined, 

automated system, continuous booster trainings), and perhaps others where there could be local 

flexibility (e.g., quality assurance, promoting buy-in for secondary users). Another concern for 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

115 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

State 3 interviewees was that the OYAS Diversion Tool was being conducted and scored after 

diversion was decided, so staff lost buy-in because they did not feel the impact of the tool for 

decision-making. To this point, while it is important to emphasize the importance of timing 

assessments, it is also important to promote the wealth of uses for its information (i.e., not just 

diversion decisions). Lastly, and similar to State 1 and State 2, the State 3 interviewees expressed 

the need for more guidance on how to use the information gathered by the OYAS more 

meaningfully in supervision and case management. 

Quality Assurance Measures. Staff were asked if there are any quality assurance (QA) 

measures in place related to the implementation and use of the OYAS tools at their agencies.

47 

These responses are presented in Table 1.14. The variation in response rates is due to the relative 

prevalence of job roles and/or settings in the interviewee sample. As the majority of interviewees 

were considered frontline staff and worked within a probation setting, it is possible that they were 

unaware of background QA processes, or did not identify QA processes as such, but as the regular 

duties of their supervisor, for example. 

Despite over one-third of interviewees indicating that QA measures were in place, most 

agencies did not appear to engage in formal QA checks for the OYAS. These inconsistencies were 

somewhat reflective of the written policies for each state. In their written policy, State 2 did not 

formally mention QA procedures for the OYAS, but State 1 provided in-depth guidance. State 3 

fell somewhere in-between, providing non-specific, “big-picture” guidance for agencies to create 

their own QA processes. At some of the agencies, specific individuals were identified as dedicated 

QA staff. In these instances, staff noted how helpful it was to have one person they could turn to 

for assessment related questions. 

47 See Q37 under Youth Assessment Practices in Appendix A. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

116 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1.14. Interview Descriptive Statistics for Quality Assurance Measures 

Variable State 1 State 2 State 3 Combined 

Quality Assurance 

Measures in Place? n (%) n (%) n (%) n (%) 

Yes 

No 

Unsure 

Total 

20 (30.8) 

30 (46.2) 

15 (23.1) 

65 (100.0) 

11 (32.4) 

17 (50.0) 

6 (17.6) 

34 (100.0) 

31 (44.9) 

25 (36.2) 

13 (18.8) 

69 (100.0) 

62 (36.9) 

72 (42.9) 

34 (20.2) 

168 (100.0) 

Due to missing data, totals are less than the reported sample size. 

Of 167 responses represented in Table 1.14, only 63 interviewees (36.9%) indicated that 

quality assurance measures were in place. Follow-up questions revealed that these QA measures 

were more likely reflective of staff management techniques, meaning that they measured the 

frequency or timeline of conducting of an OYAS, and not the quality of that assessment process 

and its outcomes. For example, supervisors mainly monitored whether assessments were 

completed within a certain time frame.48 From the interview responses, many sites seemed to have 

little or no formal QA measures in place for the OYAS, specifically. Some staff indicated that 

supervisors were not trained or proficient in the OYAS, which presented challenges to monitoring 

staff on their use of the tools. One standout in this respect comes from one agency in State 3, where 

local trainers had previously sat in on interviews to monitor the assessment scoring and provide 

feedback to newly trained staff. 

There was a relatively small proportion of affirmative responses for all three states, ranging 

from about 30 percent in State 1, to 32 percent in State 2, and about 45 percent in State 3. Compared 

to others, these particular questions drew out the most uncertainty from interviewees, with an 

average of 12 percent being unsure if there were any OYAS QA processes in place. This 

uncertainty, combined with the general lack of QA processes, shows that there may be an overall 

perception that the OYAS has been implemented (i.e., the process is complete), rather than it being 

48 In fact, the automated system automatically reminds supervisors when staff have missed a deadline. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

117 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

an ongoing process that needs consistent monitoring and updating. This is consistent with the 

results in Table 1.11, which showed that the perceived degree of assessment implementation was 

rated just above a four out of five points for each state. 

Overall, the interviews noted some concerning practices that should have been identified 

in routine QA practices. For example, the research team found that staff were not consistently 

using the tools as designed. As described above, the longer assessment tools in the system require 

multiple data sources be used to score a youth’s risk and needs level. Namely, collateral 

information and a face-to-face interview are required, while a youth self-report questionnaire is 

recommended. During the interviews with juvenile justice personnel, it became clear that some 

localities used the self-report questionnaire in place of a face-to-face interview. However, the selfreport questionnaire was designed as a supplement to—not a replacement of—the interview guide. 

State-specific quality assurance measures. Interviewees from all three states described the 

formal and informal quality assurance (QA) processes that were in place at their respective 

agencies. As stated previously, interviewees from all three states were quick to name management 

practices as a form of quality assurance. Examples of this included supervisors reviewing the 

timing of reassessments to ensure they were conducted at least once every six months, checking 

the automated system to ensure a case plan was completed, or that overrides were not occurring 

too frequently. Some of these QA “checks” seemed to foster the feeling among interviewees that 

the OYAS assessments were being done in order to “check a box” and not for their intended 

purpose. Aside from these management practices, interviewees were mostly unable to name 

additional, more formal QA processes that were in place at their agencies, which suggests that 

such approaches were rare. While rare, staff were able to describe a few of these processes during 

interviews, and these are highlighted below. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

118 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

State 3 interviewees reported that many counties had their own QA systems in place, and 

were also encouraged by the state to have local trainers and to have supervisors observe OYAS 

interviews after the staff were trained and give feedback as needed; however, these did not occur 

with any regularity. Interviewees also described supervisors “signing off” on case plans once 

reviewing them for their content and assuring that they matched the results of the assessment. 

Random file checks also occurred for this purpose. Overall, though there was some indication of 

QA practices, it did not appear to be systematic within the agencies across the state. 

State 2 interviewees revealed a somewhat similar pattern to State 3, in that there did not 

appear to be a systematized process of quality assurance, but a more loosely coupled system of 

managerial checks. At the state level, audits were run to ensure that each county was conducting 

the OYAS within the proper time frames. At the local level, agencies were encouraged to create 

their own quality assurance processes. Some interviewees described supervisors sitting in on 

OYAS interviews and scoring them simultaneously to check for reliability. One site noted that an 

annual training allowed their agency to highlight and discuss common scoring problems using case 

examples and role playing. In another site, a quality assurance position was in the process of being 

created, but at the time of the site visits, there was not currently any one staff member dedicated 

to this purpose. 

State 1 interviewees also described informal quality assurance processes. For example, 

staff members were encouraged to track their own timelines for initial assessments and 

reassessments. In addition, if there were questions about scoring certain items, it was not 

uncommon that these would be discussed during staff meetings. Lastly, much like State 3 and State 

2, State 1 interviewees indicated that supervisors verify that scores are accurately recorded in the 

automated system and that assessments are completed on time, but there was not a strong, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

119 
 

 

 

 

 

 

 

 

 

 

 

 

 

consistent focus on the quality of the information gathering process, or decisions in usage or case 

management. 

General Conclusions from the Interview Data. The 217 juvenile justice personnel semistructured interviews conducted at 22 sites across three states in the current study included staff 

across many different job settings, titles, and classifications, as well as varying degrees of 

familiarity with the OYAS. In general, however, a large proportion of interviewees worked in 

probation settings as “frontline” staff, often tasked with conducting the OYAS and translating its 

results into case management and supervision decisions. There was also, however, a degree of 

variation in these job characteristics, as the interview sample was purposive, and contained a larger 

degree of OYAS “champions” than will be seen in the web-based survey data. These “champions”

of the OYAS may have come in the form of state or agency leadership, but also those frontline 

workers and supervisors who participated in the OYAS training/coordination efforts in their state 

and/or locality. Regardless of the job characteristics of the interview sample, most individuals 

worked at their agency for a significant amount of time, with the average job tenure in any given 

state exceeding 10 years. 

There are several key points to reiterate in light of the interview findings outlined in this 

report. First, there is strong baseline support for the use of JRNA, and its implications for 

increasing fairness in juvenile justice decision-making and service provision. Interviewees were 

also in general agreement that the OYAS benefits their agency, the youths with whom they work, 

and also provides valuable information regarding criminogenic needs. However, differences arose 

in the level of agreement with these items across job classifications, highlighting the need to be 

mindful of the impact of different job roles when implementing a risk and needs assessment. 

Individuals with different job classifications will have unique experiences with JRNA and 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

120 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

perceptions may diverge, which has implications for buy-in. This resonates with past work on the 

role that judges’ resistance or buy-in plays in effective risk and needs assessment implementation 

(Vincent et al., 2012). 

Second, risk and need assessment results information is being used in important ways, 

including determining youths supervision level, measuring youths progress, matching youths to 

services, and assisting in diversion/disposition/release decisions. There is room, however, for this 

information to be used in other ways (as supported in the risk and needs assessment literature), 

including matching youths to staff, developing specialized caseloads, or allocating resources 

within an agency. Using OYAS information in these other ways may bolster the level of support 

(i.e., buy-in) for its importance in the daily operations of juvenile justice agencies, and sharing 

OYAS information with outside agencies (for treatment purposes) may improve buy-in from 

stakeholders. Sharing OYAS information in a systematic way was not a common practice in any 

of the three states. 

Third, there is a strong sense among the interviewees that their agencies/facilities have 

fully implemented the OYAS. While this item drew the highest degree of consensus among the 

sample, this “sense of arrival” was contradicted by a lack of systematic quality assurance processes 

specific to the OYAS. It is essential to note the importance of the ongoing nature of the 

implementation process, as booster trainings and ongoing formal “quality checks” are necessary 

in order to maintain and/or improve the accuracy and consistency of scoring a risk and needs 

assessment. Several effective quality assurance measures and methods to obtain buy-in were 

outlined in the interviews, and will be highlighted in the implications and recommendations. 

However, these processes were lacking in all three of the states. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

121 
 

 

 

 

 

 

 

 

 

 

Fourth, an overwhelming amount of concerns with the OYAS relate to perceptions that the 

OYAS is not valid or reliable. While the OYAS had been validated in two of the three states at the 

time of data collection, increasing the validity and reliability of the assessment through continued 

training and QA processes may be key to promoting buy-in, which may lead to more optimal usage 

practices. The following section transitions from reporting in-person interview results, to reporting 

the web-based survey results. Finally, an integrated analyses of the two sets of the results follows. 

Web-Based Survey Findings 

As stated previously, a 52-question web-based survey was completed by 1,013 juvenile 

justice personnel. Survey completers included judges, supervisors, probation officers, case 

managers, and treatment staff across different types of agencies (e.g., secure detention/institution 

or treatment facilities, probation and parole agencies, and community-based service providers). 

The four main sections of the survey included: 1) general questions concerning assessment use 

and practices, 2) implementing the assessment, 3) applying the assessment information, and 4) 

staff information. For those respondents who indicated that they administer the OYAS, a 

randomly-allocated vignette was assigned for them to score as a test of reliability. Whereas the 

results of the interview data alone were presented in the previous sections, the results of the webbased survey data are presented here. At times, these data are analyzed by state to capture the 

nuanced impact of state context on the broader perceptions of risk and needs assessment and 

implementation across juvenile justice personnel. Other times, the full survey sample was analyzed 

across OYAS administrator/non-administrator roles to capture differences in perceptions of these 

unique relationships to the OYAS. Whereas figures are presented that contain the detailed 

distribution of frequencies for each of the following items across OYAS user role, state-specific 

frequencies are only displayed where significant differences were found. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

122 
 

 

 

 

 

 

 

 

 

 

Overall Satisfaction with the OYAS. Table 1.15 provides the descriptive statistics for 

respondents’ overall satisfaction with the OYAS, separated by those who administer the OYAS 

(i.e., OYAS administrator) and those who do not (i.e., OYAS non-administrator).49 The detailed 

distribution of satisfaction levels by OYAS user-role is also presented in Figure 1.9. This 

comparison of OYAS administrators and non-administrators is explored within the survey data as 

these two groups were large enough for a meaningful comparison, despite certain items having 

lower response rates. This distinction allowed the research team to draw out the potential impact 

of the OYAS administrators’ familiarity with the assessment on their perceptions of it. The item 

shown in Table 1.15 asked participants to rate their overall satisfaction with the OYAS tool(s) on 

a scale from 1 (not at all satisfied) to 5 (completely satisfied), and responses spanned this range. 

Generally, OYAS administrators reported lower average satisfaction levels (x̅= 2.96, sd = 1.11) 

than the non-administrators (x̅ = 3.13, sd = 1.10); however, Chi-Square analyses indicated no 

significant difference in ratings between these two groups. 

Of the state samples, State 3 had the lowest average overall satisfaction with the OYAS (x̅

= 2.93, sd = 1.10), and State 1 the highest (x̅= 3.08, sd = 1.03). Though the difference was nonsignificant, it illustrated the potential influence of tool validation on personnel perceptions in a 

given state, as the OYAS was validated in State 1, and not State 3. State 1 survey respondents may 

also have been aware that the tool was validated in their state, and therefore perceived it to be a 

more satisfactory tool. Anecdotally, it was common for State 3 interviewees to mention the lack 

of OYAS validation as a large concern for personnel in their state. Another potential reason State 

1 respondents reported higher overall satisfaction is that they have been using the OYAS the 

longest of the three states included in this study and respondents may be more accustomed to it. 

49 See Appendix B for the full web-based survey, including the text for all items presented in this section of the report. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

123 
 

 

 

 

 

Table 1.15. Survey Descriptive Statistics for the Overall Satisfaction with the OYAS 

Min. Max. x̅ sd 

Analysis by OYAS User-Role 

OYAS Administrators (N = 555) 1 5 2.96 1.06 

OYAS Non-Administrators (N = 176) 1 5 3.13 1.10 

Analysis by State 

State 1 (N = 350) 1 5 3.08 1.03 

State 2 (N = 283) 1 5 3.00 1.06 

State 3 (N = 232) 1 5 2.93 1.10 

Due to missing data, totals are less than the reported sample size. 

The detailed distribution of OYAS user-role responses to the “Overall Satisfaction” item 

in Table 1.15 are depicted in Figure 1.9 below. The modal response from OYAS administrators 

and non-administrators was a three, indicating neutral feelings toward the assessment. A higher 

proportion of non-administrators rated their level of satisfaction with the OYAS as “satisfied”

(28.4%) or “completely satisfied” (9.7%), whereas the OYAS administrators were more likely to 

indicate some level of dissatisfaction (30.2%). In general, these separate distributions are quite 

similar, indicating that OYAS administrators did not seem to have meaningfully different levels 

of satisfaction with the OYAS as non-administrators. 

Figure 1.9. Survey Detailed Distribution of Responses for Level of Overall Satisfaction with the 

OYAS by User Role 

100 

90 

80 

70 

60 

50 

n = 207 n = 64 40 

n = 17 

n = 28 

n = 29 

n = 151 n = 50 30 

n = 104 20 n = 64 n = 17 

10 

0 

OYAS Administrator (N = 555) OYAS Non-Administrator (N = 176) 

% Satisfaction Level 

Not at All Satisfied Not Satisfied Neutral Satisfied Completely Satisfied 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

124 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Perceived Usefulness of the OYAS. Table 1.16 displays the descriptive statistics for 

staffs’ responses to the following statement, “Please indicate your level of agreement with the

following statements related to implementing the OYAS in your agency: The use of the OYAS 

has made my job easier.” Responses ranged from 1 (strongly disagree) to 5 (strongly agree). The 

detailed distribution for this item is also presented in Figures 1.10a-b. Overall, those who do not 

administer the OYAS reported higher average agreement levels (x̅= 2.92, sd = 1.01) than those 

who do administer the tools (x̅= 2.85, sd = 1.06). A Chi-Square test indicated that this difference 

was significant, with a modest relationship between OYAS user role and whether the OYAS is 

perceived to make the respondents’ job easier (2 

(4) = 17.32, p < .05, Cramer’s V = .16). Therefore, 

while both groups tended to disagree or at best had neutral views that the use of the OYAS made 

their job easier, OYAS administrators were significantly more likely to disagree with that 

statement. 

An analysis by state revealed that, similar to the “Overall Satisfaction” item, State 3 

respondents provided the lowest average rating for whether the use of the OYAS makes their jobs 

easier (x̅= 2.74, sd = 1.10). The State 2 sample provided the highest average rating (x̅= 2.96, sd = 

1.00). Overall, the three state samples leaned toward disagreement with the statement, though the 

difference between states was significant (2

(8) = 18.17, p < .05). However, the results of a ChiSquare test indicated a weaker relationship between this item and state membership (Cramer’s V 

= .11) than between this item and OYAS user role. 

In Figure 1.10a below, it is noteworthy that the modal response for OYAS administrators 

and non-administrators is different. While OYAS non-administrators tended to indicate a “neutral”

response to the “Makes Job Easier” item (38.1%), OYAS administrators tended to indicate that 

they “agreed” with this statement (32.2%). However, within the administrator category, it was 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

125 
 

 

 

 

 

 

 

 

more likely, in general, that respondents indicated that they “disagree” (31.6%) or “strongly

disagree” (10.2%) with this statement, rather than “agree” (32.2%) or “strongly agree” (2.5%).

Overall, OYAS administrators provided more variation in response than their non-administrator 

counterparts, as evidenced by their respective standard deviations (sd = 1.06 and 1.01). 

Table 1.16. Survey Descriptive Statistics for the Use of the OYAS & “Makes Job Easier”

Min. Max. x̅ sd 

Analysis by OYAS User Role ^ 

OYAS Administrators (N = 512) 1 5 2.85 1.06 

OYAS Non-Administrators (N = 155) 1 5 2.92 1.01 

Analysis by State ^ 

State 1 (N = 318) 1 5 2.91 1.02 

State 2 (N = 245) 1 5 2.96 1.00 

State 3 (N = 214) 1 5 2.74 1.10 

% Agreement Level 

Due to missing data, totals are less than the reported sample size. ^ p < .05 

Figure 1.10a. Survey Detailed Distribution of Responses for Level of Agreement for the 

Usefulness of the OYAS by User Role 

100 

90 

80 

70 

60 

50 n = 59 40 n = 165 n = 162 

30 

n = 13 

n = 120 

n = 13 

n = 39 n = 36 

20 n = 52 

10 n = 8 

0 

OYAS Administrator (N = 512) OYAS Non-Administrator (N = 155) 

Strongly Disagree Disagree Neutral Agree Strongly Agree 

Figure 1.10b below contains the frequencies for the “Makes Job Easier” item across states,

as this distinction elicited significantly different responses (see Table 1.16). Here, it is apparent 

that State 3 respondents expressed stronger disapproval that the OYAS was useful in making their 

jobs easier, as they were the only state in which “agree” was not the modal response. Curiously,

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

126 
 

 

 

 

 

 

 

 

 

 

State 3 also had the highest proportion of respondents in the “strongly agree” category. These

individuals may be the “champions” of the assessment (i.e., those responsible for planning the

assessment implementation, training staff, and/or creating/carrying out OYAS policies). 

Figure 1.10b. Survey Detailed Distribution of Responses for Level of Agreement for the 

Usefulness of the OYAS by State 

100 

n = 25 n = 20 n = 27 

n = 98 n = 61 

n = 73 

n = 84 

n = 78 n = 54 n = 104 n = 80 

n = 49 

n = 7 n = 6 n = 11 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

State 3 (N = 214) 

% Agreement Level 

Strongly Agree 

State 1 (N = 318) State 2 (N = 245) 

Strongly Disagree Disagree Neutral Agree 

Perceptions of Benefit to Youths. Table 1.17 contains the descriptive statistics for 

respondents’ levels of agreement, from 1 (strongly disagree) to 5 (strongly agree), concerning 

whether the OYAS benefits the youths in their agencies. Figure 1.11 provides the detailed 

distribution of their responses, separated by the two samples of interest: OYAS administrators and 

non-administrators. Those who administer the OYAS reported a lower average level of agreement 

with this statement (x̅= 3.20, sd = .99) when compared to non-administrators (x̅= 3.42, sd = .95); 

however, this difference was non-significant. More so than the previous two items (“Overall 

Satisfaction with the OYAS” and “Level of Agreement for the Usefulness of the OYAS”), the two 

samples generally expressed agreement that the OYAS benefits the youths in their agencies. 

Analyzing the item by state, State 3 again provided the lowest average rating, though the 

difference was non-significant between states. In general, all three state samples leaned toward 

perceiving the OYAS to be beneficial to the youths in their agency, though State 2 respondents 

gave the highest average level of agreement with this item. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

127 
 

 

 

 

 

 

 

 

Table 1.17. Survey Descriptive Statistics for the OYAS Benefits Youths in Agency 

Min. Max. x̅ sd 

Analysis by User Role 

OYAS Administrators (N = 507) 1 5 3.20 .99 

OYAS Non-Administrators (N = 154) 1 5 3.42 .95 

Analysis by State 

State 1 (N = 316) 1 5 3.27 .95 

State 2 (N = 243) 1 5 3.32 .93 

State 3 (N = 212) 1 5 3.22 1.04 

Due to missing data, totals are less than the reported sample size. 

The detailed distribution of responses for the “Benefits Youths” item are displayed by user 

role in Figure 1.11 below. For both samples, “agree” was the modal response, followed by 

“neutral”. Notably, the non-administrator sample was more likely to “agree” (46.8%) or “strongly 

agree” (7.8%) with this statement than the OYAS administrator sample, and a larger portion of the 

administrator sample was more likely to “disagree” (16.6%) or “strongly disagree” (6.5%) than 

the non-administrator sample. However, as the overall divergence in ratings between samples was 

non-significant, it is clear that respondents generally agree that the OYAS benefits the youths in 

their agency. 

Figure 1.11. Survey Detailed Distribution of Responses for Level of Agreement for the OYAS 

Benefits Youths in Agency by User Role 

100 

90 

n = 33 n = 6 

n = 84 n = 20 

n = 165 n = 44 

n = 201 

n = 72 

n = 24 n = 12 

0 

10 

20 

30 

40 

50 

60 

70 

80 

% Agreement Level 

OYAS Administrator (N = 507) OYAS Non-Administrator (N = 154) 

Strongly Disagree Disagree Neutral Agree Strongly Agree 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

128 
 

 

 

 

 

 

 

 

 

 

 

 

Perceptions of the OYAS and Consistency. Table 1.18 illustrates the descriptive statistics 

for the respondents’ level of agreement from 1 (strongly disagree) to 5 (strongly agree) regarding 

whether the OYAS increases consistency in rules, policies, and procedures in assessment practices. 

Additionally, Figure 1.12 presents the detailed distribution of responses for this item by OYAS 

user role. While both samples generally agreed with this statement, the non-administrator sample 

indicated a higher average rating (x̅= 3.57, sd = .91) than the OYAS-administrators (x̅= 3.36, sd 

= .97). There was no significant difference in ratings between the user role samples. This 

“Consistency” item in particular retrieved the highest level of agreement among either sample 

when compared to the other items. The difference between the state-specific samples was also 

non-significant, with all three states generating a similar mean score and variation statistic. 

Table 1.18. Survey Descriptive Statistics for the OYAS Increases Consistency 

Min. Max. x̅ sd 

Analysis by User Role 

OYAS Administrators (N = 527) 1 5 3.36 .97 

OYAS Non-Administrators (N = 160) 1 5 3.57 .91 

Analysis by State 

State 1 (N = 329) 1 5 3.34 .96 

State 2 (N = 260) 1 5 3.50 .93 

State 3 (N = 218) 1 5 3.49 .98 

Due to missing data, totals are less than the reported sample size. 

Figure 1.12 displays the detailed distribution of responses by user role for the 

“Consistency” item in Table 1.18. The two samples shared the same modal category (i.e., agree) 

and in general shared a very similar distribution of responses. However, OYAS non-administrators 

did appear to be more enthusiastic in their agreement with this statement (63.1% of cases) than 

OYAS administrators (51.6%). Similar to the belief that the OYAS benefits the youths, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

129 
 

 

 

 

 

 

 

 

 

participants were in high agreement that the OYAS increases consistency in rules, policies, and 

procedures regarding assessment practices. 

Figure 1.12. Survey Detailed Distribution of Responses for the Level of Agreement for the 

OYAS Increases Consistency by User Role 

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

OYAS Non-Administrator (N = 160) 

Strongly Agree 

Perceptions of OYAS Implementation. Respondents were asked to rate their level of 

agreement that there is staff support for the implementation of the OYAS from 1 (strongly 

disagree) to 5 (strongly agree). The descriptive statistics are displayed in Table 1.19, and the 

detailed distribution of responses is also depicted in Figure 1.13. Those who administer the OYAS 

reported slightly lower average levels of agreement (x̅ = 3.35, sd = 1.00) compared to nonadministrators (x̅ = 3.39, sd = 1.05), but this difference was non-significant. This follows the 

pattern of some of the previous items, where OYAS administrators provided a slightly lower 

average score than non-administrators, but in general the samples’ perceptions converged. There 

was also no significant difference when analyzing the responses by state, though State 2 provided 

the highest average rating and State 3 the lowest. 

Though responses in the full sample ranged from 1 (strongly disagree) to 5 (strongly agree), 

in general it is agreed upon that there is staff support for the implementation of the OYAS, and % Satisfaction Level 

n = 233 

n = 25 n = 5 

n = 15 

n = 39 

n = 85 

n = 39 n = 16 

OYAS Administrator (N = 527) 

n = 157 

n = 73 

Strongly Disagree Disagree Neutral Agree 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

130 
 

 

 

 

 

 

 

this level of perceived support from staff was higher than the average responses provided in the 

“Personal Support for the OYAS” item. This is in contrast to the interview data, where personal 

level of satisfaction with the OYAS was rated higher than the perceived level of staff support for 

the use of the OYAS. It is possible that this was the result of a “method effect,” whereby 

interviewees, by the nature of being face-to-face with an interviewer, were more apt to frame their 

own perception of the OYAS more positively (see, e.g., Newman et al., 2002). Additionally, as 

the interview sample was purposive, it potentially contained more state and local “champions” of 

the assessment than in the survey sample. 

Table 1.19. Survey Descriptive Statistics for Staff Support for Implementing the OYAS 

Min. Max. x̅ sd 

Analysis by User Role 

OYAS Administrators (N = 509) 1 5 3.35 1.00 

OYAS Non-Administrators (N = 155) 1 5 3.39 1.05 

Analysis by State 

State 1 (N = 314) 1 5 3.39 .99 

State 2 (N = 245) 1 5 3.46 .91 

State 3 (N = 215) 1 5 3.27 1.08 

Due to missing data, totals are less than the reported sample size. 

Figure 1.13 below contains the detailed distribution of responses from the “Staff Support”

item in Table 1.19 by OYAS user role. It reveals a relatively similar distribution of responses 

between the OYAS administrator and non-administrator samples for this item. The largest 

difference was in the “strongly agree” category, where non-administrators expressed a higher level 

of enthusiasm (9%) than OYAS administrators (5.7%). Less than one-quarter of either sample 

disagreed that there is staff support for the implementation of the OYAS. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

131 
 

 

 

 

 

 

 

Figure 1.13. Survey Detailed Distribution of Responses for Level of Agreement for Staff Support 

for Implementing the OYAS by User Role % Agreement Level 

n = 31 n = 9 

n = 73 n = 25 n = 120 n = 32 

n = 75 

n = 29 n = 14 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

OYAS Non-Administrator (N = 155) 

Neutral Agree Strongly Agree 

n = 256 

OYAS Administrator (N = 509) 

Strongly Disagree Disagree 

Broader Attitudes Toward OYAS Implementation. The web-based survey also asked 

respondents to report their level of agreement with a series of statements regarding the 

implementation of the OYAS in their agency from 1 (strongly disagree) to 5 (strongly agree). Here, 

we examine the collective responses of the full sample, and do not separate the results by OYAS 

administrators and non-administrators. This is in order to gauge overall staff attitudes toward 

implementation more broadly, with items involving the assessment itself, the policies surrounding 

the assessment, and the implementation environment both inside and beyond the individual 

agencies. Significant differences in responses between the states are noted as they occur. Some 

items target agreement with the presence of empirically-supported implementation practices at 

respondents’ agencies (e.g., a written protocol regarding the OYAS), whereas other target 

agreement for their absence (e.g., formal training not being offered). 

Table 1.20 provides the descriptive statistics for the implementation items and Figure 1.14 

displays the detailed distribution of responses. The items in Table 1.20 are in order of the highest 

average level of agreement to the lowest. Respondents had the highest level of agreement that there 

is a protocol for how to use the OYAS (x̅= 3.96, sd = .76), and that there are clear guidelines for 

when to use the OYAS (x̅= 3.90, sd = .80). These two items reflect the preparation level of the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

132 
 

 

 

 

 

 

 

 

 

 

individual agencies in implementing the OYAS, as written policies and procedures are considered 

a best practice for implementing a risk and needs assessment (Vincent et al., 2012). 

There is also a high level of agreement that the OYAS tools are easy to read, interpret, and 

use (x̅= 3.59, sd = .99). While respondents tend to also agree that finding resources to properly 

address youths’ needs is difficult (x̅= 3.15, sd = 1.10) and that achieving staff buy-in is challenging 

(x̅= 3.14, sd = 1.00), respondents tend to disagree that formal OYAS training is not routinely 

offered (x̅= 2.98, sd = 1.13), or that there is inconsistency in the implementation of the OYAS (x̅

= 2.65, sd = 1.05). The “Formal training in the OYAS is not routinely offered” item in particular 

spawned a larger standard deviation (sd = 1.13) than any other item, highlighting local level 

variation in implementation. 

Table 1.20. Survey Descriptive Statistics for the Broader Attitudes Toward Implementation 

Min. Max. x̅ sd 

There is a protocol for how to use the OYAS (N = 772) 1 5 3.96 .76 

There are clear guidelines for when to use the OYAS (N = 775) 1 5 3.90 .80 

The OYAS tools are easy to read, interpret, and use (N = 775) 1 5 3.59 .99 

Finding resources to properly address youths’ needs identified 

from the OYAS is difficult (N = 773) 1 5 3.15 1.10 

Staff buy-in is hard to achieve (N = 776) 1 5 3.14 1.00 

Formal training in the OYAS is not routinely offered (N = 771) 1 5 2.98 1.13 

Implementation of the OYAS is not consistent (N = 773) 1 5 2.65 1.05 

Due to missing data, totals are less than the reported sample size. 

When analyzing these data by state, three important differences arise in attitudes toward 

implementation. First, a Chi-Square test revealed that State 3 respondents were significantly more 

likely to agree that there is a protocol in place for the OYAS (2 

(8) = 18.38, p < .05, Cramer’s V = 

.11). This may speak to the importance of not only having a protocol in place, but having one that 

makes sense to those who will be referring to it and abiding by its contents (i.e., State 3 respondents 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

133 
 

 

 

 

 

 

 

 

 

 

 

consistently expressed less approval of the OYAS and its uses, yet showed the highest level of 

agreement that a protocol is in place). 

The second important difference in broader attitudes toward implementation between the 

state samples involved the resources available for treatment. State 3 interviewees were 

significantly more likely to agree that resources for treatment are difficult to find (x̅= 3.41, sd = 

1.12, 2 

(8) = 42.32, p < .001). This item in particular shows the strongest relationship to state 

membership than any other implementation item (Cramer’s V = .17). This barrier to treatment may 

impact personnel perceptions of the implementation and usefulness of any JRNA, and as in this 

study, specifically the OYAS. State 1 provided the lowest level of agreement with this item (x̅= 

2.91, sd = 1.04), indicating that these personnel are highly satisfied with the treatment resources 

available to the youths in their agencies/facilities. This may speak to the efforts mentioned 

previously in the report in the State Context for Implementation and Usage. 

The last major difference in broad attitudes toward implementation involved the 

availability of training. A Chi-Square test showed a significant difference for this item between 

the three states (2 

(8) = 27.71, p < .05, Cramer’s V = .13), with State 1 providing the highest 

average rating (x̅= 3.15, sd = 1.11), and State 3 the lowest (x̅= 2.75, sd = 1.12). As such, State 3 

respondents agreed significantly more that formal OYAS training is routinely offered. Once more, 

this suggests the importance of not only providing regular training opportunities, but ensuring their 

quality or applicability to staff. This is evidenced by State 3’s generally low levels of satisfaction 

for the OYAS and its uses, but yet high level of agreement that training is available. 

In order of the highest level of agreement from left to right across the X-axis, Figure 1.14 

contains the detailed distribution of responses to the implementation items in Table 1.20. This 

figure illustrates the unique variation in responses for each implementation item, with “Training”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

134 
 

 

 

 

 

 

 

having the largest standard deviation (sd = 1.13), and “Protocol” having the smallest (sd = .76). 

“Disagree” was the modal response to the “Resources,” “Training,” and “Implementation” items, 

whereas “neutral” was the modal response to the “Staff Buy-In” item. In general, some items reveal 

an enthusiastic level of agreement (e.g., “Protocol,” “Guidelines,” and “OYAS Instruments”),

while others elicit a more tempered response, and may depend more so on local context. 

Figure 1.14. Survey Detailed Distribution of Responses for Level of Agreement for Broader 

Attitudes toward Implementation Items 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

% Level of Agreement 

Strongly Disagree Disagree Uncertain Agree Strongly Agree 

Strengths and Limitations of the OYAS. Staff were provided an opportunity to list the 

strengths and limitations of the OYAS based on how their agency uses the assessment (Tables 

1.21a-b). Specifically, respondents were provided with a series of text entry boxes in which they 

could list up to three strengths and three limitations. This analysis of respondent-generated OYAS 

strengths and limitations has a different appearance from its corresponding results section within 

the interview results. This is because the interview strengths and limitations data were analyzed 

using qualitative data analysis software (QDA Miner). Because of the larger size of the web-based 

survey sample, strengths and limitations were coded within SPSS using a similar inductive process. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

135 
 

 

 

 

 

 

 

 

 

 

 

It is noteworthy that, despite the separate analysis processes, both the interview and survey samples 

generated many of the same themes, which will be discussed further in the “Integrated Analysis”

section. 

Of the 1,013 respondents who completed the survey, about 44 percent (n = 449) provided 

at least one strength. A total of 1,029 strengths were provided by these 449 respondents across the 

three items. Therefore, those who chose to answer the question by providing a strength of the 

OYAS tended to provide two or three strengths. There was wide variability across those responses, 

with the most frequently cited strengths detailed in Table 24a. Some of these strengths involved 

the content and structure of the tools, the usefulness of its completed product, or implementation 

process. 

Included in the Table 1.21a is any strength that was named by five percent or more of the 

sample that provided at least one strength (N = 449). A review of the responses for the strengths 

suggests the OYAS provides overall system fairness, consistency, and objectivity (n = 180); is 

helpful with case/treatment planning (n = 121); determines risk level and supervision level (n = 

118); provides useful/comprehensive information (n = 110); and identifies youths' needs (i.e., risk 

factors) (n = 84). Additionally, the respondents identified that the OYAS is easy, clear, and/or 

quick to administer (n = 62); aids in juvenile justice and service provision decision-making (n = 

57); provides valid (i.e., accurate) results (n = 25); helps to monitor youths progress (n = 24); or 

has some sort of other strength that is not otherwise specified (NOS) (n = 61). NOS strengths often 

included positive, yet non-specific sentiments toward the OYAS (e.g., “It is a good tool”). This 

list of common strengths reveals that the OYAS is seen by many to provide useful information that 

guides treatment and court decision-making, and increases consistency/reduces bias in those 

decisions. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

136 
 

 

 

 

 

 

 

 

Table 1.21a does not depict the less common strengths reported, including: the OYAS helps 

to improve resource allocation through the respondents’ agencies (n = 19); the OYAS 

training/implementation/support was of high quality (n = 17); the OYAS provides valuable 

information regarding both risk and needs (n = 11); and that there is a high level of buy-in from 

the staff (n = 10). Notably, many of these strengths, particularly those that are less common, are 

seen by other respondents as limitations, as evidenced in Table 1.21b. 

Table 1.21a. Survey Descriptive Statistics for Respondent-Generated OYAS Strengths 

Strength (n = # of cases) 

Fair/consistent/objective 

(n = 180) 

% of Survey Completers 

Who Named Strength(s) 

(N = 449) 

40.1 

% of All Survey 

Completers 

(N = 1,013) 

17.8 

Case/treatment planning (n =121) 26.9 11.9 

Risk/supervision level (n = 118) 26.3 11.6 

Useful/comprehensive information 

(n = 110) 24.5 10.9 

Identifies needs (n = 84) 18.7 8.3 

Easy/clear/quick to administer (n = 62) 13.8 6.2 

Helpful (not otherwise specified) 

(n = 61) 

Decision-making/service provision 

(n = 57) 

13.6 

12.7 

6.0 

5.6 

Valid results (n = 25) 5.6 2.5 

Monitor youths progress (n = 24) 5.3 2.4 

Of the 1,013 survey completers, about 42 percent provided at least one limitation of the 

OYAS, and many provided more (N = 427). This is less than, but also similar to, the proportion of 

the sample that provided at least one strength. These responses yielded a total of 840 limitations 

across the three items (Table 1.21b), indicating that most respondents provided more than one 

limitation, and these limitations are concentrated among a smaller pool of concerns than the 

strengths. Respondent-generated limitations represent concerns about the tool itself, its associated 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

137 
 

 

 

 

 

 

 

 

 

 

 

implementation processes, and even some concerns that may lie beyond the scope of the tools and 

the agency entirely (e.g., a lack of local resources to refer youths for treatment). 

The most commonly named limitations included: the OYAS fails to provide 

comprehensive information in areas such as family and education issues, mental health needs, 

and/or substance abuse issues (n = 122); inaccurate or invalid results regarding risk level (n = 95); 

there is inconsistent administration and scoring of the tools (i.e., low reliability) (n = 94); the tools 

are too time-consuming to administer and complete (n = 52); the agency has poor usage practices 

(n = 50); a lack of staff buy-in (n = 50); inadequate training and written policies (n = 47); a lack 

of resources available for treatment (n = 33); youths or family score manipulation (i.e., dishonesty) 

during the information gathering process (n = 31); the OYAS is inaccurate for sex offenders (n = 

27); difficulties in using the OYAS information for case planning (n = 23); and other not otherwise 

specified (NOS) limitations (n = 87). NOS limitations often included negative, yet non-specific 

sentiments towards the OYAS, such as “it is not good.”

As mentioned above, some of these items were listed as both a strength and a limitation 

of the OYAS. For example, consistency across the system and determining risk level and 

supervision levels was cited as a strength. However, some staff indicated that there were 

inconsistencies across how probation officers administer the OYAS, conduct overrides, or 

calculate risk scores, as well as inconsistencies in administrative policies. Additionally, a similar 

number of respondents named the scope of information provided by the OYAS as a strength (n = 

110) and a limitation (n = 112). This highlights the importance of considering local context in the 

implementation process, as this may contribute to staff perceptions of the tool. 

Some less common concerns (i.e., not endorsed by 5% or more of the limited sample) 

emerged alongside those just mentioned, including a lack of quality assurance processes (n = 18); 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

138 
 

 

 

 

 

 

 

concerns about reassessments being too time-consuming, or done improperly or too infrequently 

(n = 15); there is an overreliance on the tool’s results relative to professional discretion (n = 11); 

and the improper use of overrides (n = 8). 

Table 1.21b. Survey Descriptive Statistics for Respondent-Generated OYAS Limitations 

Limitation (n = # of cases) 

% of Survey 

Completers Who 

Named Limitation(s) 

(N = 427) 

% of All Survey 

Completers 

(N = 1,013) 

Incomprehensive information (n = 112) 26.2 11.1 

Lack of validity (n = 95) 22.2 9.4 

Inconsistent administration and scoring 

(i.e., low reliability) (n = 94) 22.0 9.3 

Other limitation (NOS) (n = 87) 20.4 8.6 

Time-consuming (n = 52) 12.2 5.1 

Poor usage practices (n = 50) 11.7 4.9 

Lack of staff buy-in (n = 50) 11.7 4.9 

Inadequate training and written policies 

(n = 47) 

Lack of resources available for treatment 

(n = 33) 

Youths/family manipulate scores (n = 31) 

11.0 

7.7 

7.3 

4.6 

3.3 

3.1 

Inaccurate for sex offenders (n = 27) 6.3 2.7 

Difficulty in case planning (n = 23) 5.4 2.3 

Staff were also asked to provide any specific suggestions for improving the OYAS and/or 

processes in their agency. Of the 1,013 survey participants, only 145 individuals provided feedback 

for improving the OYAS and/or procedures concerning the OYAS. Common suggestions 

included: making the automated system more user-friendly and integrated into other online 

systems; allowing more room for professional judgment to change the risk level of a youth in 

instances where that is necessary (i.e., many staff feel that the OYAS “under classifies” many 

youths); including more mental health and sexual offending indicators in the assessment; providing 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

139 
 

 

 

 

 

 

 

 

 

 

 

 

 

more frequent training and certification on the tools; designating certain staff to be OYAS 

administrators to ensure consistency and accuracy in administration; fostering consistency in 

implementation across counties of a given state; providing more convenient ways for staff to 

access aggregate OYAS data; and training outside stakeholders on the OYAS (e.g., attorneys, 

judges, prosecutors). 

OYAS Administrator Scoring. Risk and needs assessment can greatly improve case 

processing, but it needs to be implemented well in order to do so (Vincent et al., 2018). The webbased survey offered an opportunity to test reliability in a basic way (i.e., consistency across 

administrators) in scoring the survey by virtue of a simulated vignette in the domains of Juvenile 

Justice History, Peers and Social Support Network, or Pro-Social Skills. The following results 

pertain only to those individuals who indicated that they administer any of the OYAS tools (n = 

626), and the results are presented for the full sample of OYAS administrators (i.e., not divided by 

state). Those who did not indicate that they administer the OYAS were not asked to answer the 

vignettes (presented in Tables 1.22, Table 1.23, and Figure 15) as they did not apply. Table 1.22 

provides the distribution of respondents who reported they use any of the OYAS tools. The results 

indicate that the Disposition Tool was by far the most commonly used tool, with 82.9 percent of 

the sample endorsing its use. The Diversion and Residential Tools were the second most 

commonly used, with 36.8 percent and 33.9 percent, respectively. The Detention Tool had the 

lowest frequency of use (n = 94), which is partially explained by the fact that the tool is rarely used 

in State 3. Additionally, many survey respondents used multiple OYAS tools. 

Table 1.22. Survey Descriptive Statistics for the Distribution of OYAS Assessment Tool Use 

OYAS Tool Frequency Percent 

Diversion 188 36.8 

Detention* 94 18.8 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

140 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1.22. Survey Descriptive Statistics for the Distribution of OYAS Assessment Tool Use 

OYAS Tool Frequency Percent 

Disposition 494 82.9 

Residential 172 33.9 

Reentry 99 20.7 

Totals in Table 1.22 exceed the total possible sample size of 626 due to some respondents administering more 

than one assessment tool. 

*The Detention Tool is routinely used in State 1 and State 2, but rarely used in State 3. 

The descriptive statistics for the scoring of the three possible vignettes given to OYAS 

administrators are presented in Table 1.23. These reflect vignettes given during OYAS training 

session and therefore they are benchmarked. Correct scores were determined by the research team 

by consulting the published OYAS scoring guide. For the Juvenile Justice History vignette, the 

average score was 2.01 (sd = .70), which is close to the correct score of 2. The average score for 

the Peers & Social Support Networks vignette (x̅= 5.46, sd = 0.62) was also trending towards the 

correct score (Total = 6). OYAS administrators who received the Pro-Social Skills vignette also 

had an average score close to the correct score of 2 (x̅= 1.81, sd = 0.80). In general, average 

vignette scores revealed that OYAS administrators are able to apply the scoring guide to retrieve 

a fairly accurate risk score for these three domains/vignettes, but also suggest some degree of 

variability generally in line with the degree to which interpretation was necessary. The juvenile 

justice history score is very close on average in the degree to which it was scored correctly relative 

to the others. 

Table 1.23. Survey Descriptive Statistics for the OYAS Administrator Vignettes 

Min. Max. x̅ sd Correct 

Score50 

Juvenile Justice History 

(N = 143) 0.0 3.0 2.01 0.70 2 

50 The correct score refers to the true total score for each vignette. This is the score that should have been assigned 

based on the OYAS scoring guide. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

141 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1.23. Survey Descriptive Statistics for the OYAS Administrator Vignettes 

Correct Min. Max. x̅ sd Score50 

Peers & Social Support 

Networks (N = 149) 3.0 6.0 5.46 0.62 6 

Pro-Social Skills 0.0 3.0 1.81 0.80 2 (N = 143) 

Due to missing data, totals are less than the reported sample size. 

Figure 15 presents descriptive statistics for the distance away from the correct score (i.e., 

scoring discrepancy) for all three vignettes. Specifically, this represents the difference between 

what the OYAS administrators scored the vignette and the correct total score. Therefore, those 

with a difference of -2 suggest that these individuals provided a total score that was two points 

below the correct score from the manual. Underscoring would indicate less risk for a youth in 

comparison to over-scoring, which is indicative of more risk. 

The results show that only half OYAS administrators scored their assigned vignette 

correctly (52.2%). The largest proportion of survey respondents scored the Juvenile Justice History 

vignette correctly (60.1%), and no respondents scored the vignette three or more below the total 

score. It was, however, more common for respondents to score the Juvenile Justice History 

vignette one or more points above the total score (21.7%) than the other two vignettes. For the 

Peers & Social Support Networks vignette, only slightly more than half of the respondents (52.4 

%) scored the vignette correctly. For respondents who had a scoring discrepancy, this trended in 

the direction of “under-scoring” the vignette, indicating that respondents were more likely to assess 

less risk for the Peers & Social Support Networks domain than the other two domains. For the ProSocial Skills vignette, 55.3 percent of the respondents scored this domain incorrectly. These 

incorrect scores were distributed both above and below the correct score of two. Overall, these 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

142 
 

 

 

 

 

 

 

 

 

 

 

 

 

scores suggest that, while there is some correspondence with the manual, there are also some 

discrepancies that require further attention and which might be addressed in future booster training. 

Past research suggests that personnel perceptions of a risk and needs assessment tool can 

impact its integrity (Vincent et al., 2012). Relevant to the current study, this suggests that those 

with more supportive attitudes toward the OYAS may score it with more accuracy. In the current 

section, we have seen that there is some variation in accuracy among the personnel sample when 

scoring the vignettes. Following the integrated analysis of the juvenile justice personnel interviews 

and surveys is a substudy entitled “Support for the OYAS and Integrity in the Juvenile Risk and

Needs Assessment Process.” This section more fully explores the relationship between perception 

of the OYAS and its integrity. 

Figure 15. Survey Detailed Distribution of Vignette Scoring Discrepancies 

100 

90 

80 

70 n = 86 

Juvenile Justice History (N = 143) Peers & Social Support Networks Pro-Social Skills (N = 143) 

(N = 149) 

Scored -3 below benchmark Scored -2 below benchmark 

Score -1 below benchmark Scored correctly 

Scored +1 or more above benchmark 

General Conclusions from the Web-Based Survey Data. The findings presented in this 

n = 0 n = 1 n = 0 n = 4 n = 7 n = 6 

n = 22 

n = 63 

n = 44 

n = 78 

n = 64 

n = 31 

n = 0 

n = 29 

0 

10 

20 

30 

40 

50 

60 

% in Scoring Category 

section provided a general sense of attitudes toward the OYAS and general perceptions in key 

areas concerning the usefulness, benefits, support for, and consistency of the OYAS. Specifically, 

attitudes and perceptions concerning the OYAS were examined across two groups (i.e., OYAS 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

143 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

administrators and non-administrators), as well as across the three states. We also explored the 

consistency of vignette scores among OYAS administrators, as well as respondent-generated 

strengths and limitations of the OYAS, and suggestions for improving the implementation process. 

The results lead to several general conclusions. There is generally a mixed level of 

agreement/support for all items on the survey, as respondents’ answers ranged from 1 (low, 

completely disagree, or not at all unsatisfied) to 5 (high, completely agree, or completely satisfied). 

While answers on the extreme ends of this range tended to be outliers when considering the 

responses of the full sample, it is important to note that individuals with strong feelings toward the 

OYAS may have more influence over the perceptions of their peers, positive or negative. 

When asked to rate their level of personal satisfaction with the OYAS, the average 

response from OYAS administrators and non-administrators leaned toward being satisfied. OYAS 

administrators tended to give a lower average response, but this difference was non-significant. 

The modal response for either group was “neutral.” There were also no significant differences for 

this item between the states, though it was noteworthy that State 3, as the only state that did not 

have the OYAS validated on their population at the time of data collection, reported the lowest 

average response for personal satisfaction with the OYAS. 

In terms of the perceived usefulness of the OYAS in making personnel jobs “easier,” 

OYAS administrators were significantly more likely to disagree with this statement. Expectedly, 

OYAS administrators and non-administrators provided a different modal response; the nonadministrators more often indicated a “neutral” rating (i.e., 3), and the administrators more often 

indicated “agreement” (i.e., 4) with the statement. However, OYAS administrators as a group were 

almost exactly as likely to indicate that they “agreed” with the statement as they were to indicate 

that they “disagreed” that the OYAS makes their job easier. Across the states, State 3 provided a 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

144 
 

 

 

 

 

 

 

 

 

 

significantly lower average response than the other two states. However, there was a stronger 

relationship between OYAS user role and this item than state membership. 

There was a high level of agreement that the OYAS benefits the youths in respondents’

agencies, and no significant difference in ratings between OYAS user roles or between the three 

states. Similarly, respondents indicated a high level of agreement that the OYAS increases 

consistency in rules, policies, and procedures regarding assessment practices. This, next to the 

OYAS benefitting the youths, is the most consistently supported item in the web-based survey 

results, and was also mentioned as a strength of the OYAS in the open-ended question portion. 

There was also a high level of agreement that there is staff support for the implementation 

of the OYAS, which was rated higher on average than the personal level of satisfaction with the 

OYAS. This runs in contrast to the interview data, where personal satisfaction was rated higher 

than perceived staff support. There was no significant difference in scores between OYAS 

administrators and non-administrators, nor between the state-specific samples. In general, State 3 

respondents provide lower average ratings on most items in the web-based survey results, but this 

difference is rarely statistically significant. 

Respondents were in high agreement that there is a written protocol for how to use the 

OYAS, and that guidelines are in place for when to use the tools. They also tended to agree that 

the OYAS is easy to read, interpret, and use. Other items drew a more mixed response, which 

highlighted some differences in local context regarding implementation practices and staff 

attitudes (e.g., training not being routinely offered or the implementation being inconsistent). 

These items were analyzed by state to identify the source of these contextual differences. 

When analyzing these broader implementation items by state, State 3 respondents were 

significantly more likely to agree that there is a protocol in place for the OYAS. State 1 respondents 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

145 
 

 

 

 

 

 

 

 

 

 

 

 

 

provided the lowest average level of agreement with this item. State 3 interviewees were also 

significantly more likely to agree that community resources for treatment are difficult to find. This 

item in particular showed the strongest relationship to state membership than any other 

implementation item, again highlighting the importance of local context. 

Respondents were asked to list the strengths and limitations of the OYAS. The most 

common strengths reflected the OYAS itself and its processes, as well as the usefulness of its 

finished product. They noted that the OYAS increases consistency and reduces bias in the 

supervision process. Additionally, they noted the usefulness of OYAS information for treatment 

planning, which is contrasted with the aforementioned concern about a lack of treatment resources 

in the community. It is possible that as the treatment resources in the community increase, so will 

the perceived usefulness and overall satisfaction with the assessment system. 

Similarly, the most common limitations also involved the OYAS itself, as well as its 

completed product. Respondents expressed the most concern that the OYAS does not collect the 

proper/comprehensive information (e.g., mental health, substance use, family factors), which may 

lead to an inaccurate risk score. This sentiment is relevant to, and impactful for, establishing buyin among staff and stakeholders. Where there were many respondents who indicated that the 

OYAS provides good information, there were about the same number who indicated the contrary. 

In terms of completeness of information, this is where having supplementary need and responsivity 

assessments in place is important, so as to capture such information that respondents perceived to 

be lacking in the risk and needs assessment. 

The OYAS Diversion Tool was by far the most commonly used tool, and Detention the 

least. With only approximately half of the survey respondents scoring the randomly-assigned 

vignettes correctly, OYAS administrators were only moderately accurate in scoring the OYAS. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

146 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

There may have also been some selection artifacts, as respondents who decided to complete the 

vignette may have been more apt to know about the OYAS tools and may have been more 

confident in their ability to score it well. Depending upon the vignette, there was evidence of both 

underscoring and overscoring youths. The vignette results revealed the opportunity for ongoing 

trainings to address the items that users may be inaccurately scoring most often (e.g., the ProSocial Skills vignette was the least likely to be scored correctly, and the majority of respondents 

scored this domain incorrectly). Difficulty in scoring may also contribute to the feeling that the 

OYAS provides inaccurate/invalid results, which was one of the respondent-generated limitations 

provided. Increasing scoring accuracy through improving initial training in key areas and by 

providing more consistent and frequent ongoing training has the potential to increase buy-in for 

the tool. The relationship between support for the OYAS and scoring accuracy is explored in the 

in this section as well, as one potential avenue for variation to enter the scoring process. 

Integrated Analysis of the Interview and Web-Based Survey Data 

There were 14 questions on the web-based survey that overlapped with those asked in the 

juvenile justice personnel interviews. These questions involved how the OYAS information is 

used, respondents’ experience with overrides, quality assurance processes, the OYAS automated 

system and other assessments, as well as respondents’ general attitudes about risk and needs 

assessment, and attitudes specific to the OYAS. Specifically, the interviews and surveys shared 

six questions on how the agency uses assessment information (i.e., does your agency use the OYAS 

assessment information to): 

1. Allocate resources to particular parts of the agency? 

2. Develop specialized caseloads? 

3. Determine supervision level? 

4. Measure youths progress in reducing risk and needs? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

147 
 

 

 

 

 

 

 

5. Match youths to appropriate treatment services? 

6. Assist in diversion/disposition/release decisions? 

There are seven other questions in common. These included: 

7. Are there any quality assurance processes that your agency engages in concerning 

the OYAS? 

8. Do you use any additional assessments? 

9. Does the agency allow for overrides? 

10. On a scale from 1 to 5, with 1 being you are not satisfied and 5 being you are 

completely satisfied, please rate your overall satisfaction with the assessment 

system. 

11. On a scale from 1 to 5, with 1 meaning it does not at all reduce bias and 5 being it 

removes all bias, rate of your level of agreement on the statement that risk and 

needs assessments reduce bias. 

12. On a scale from 1 to 5, with 1 meaning it does not benefit youths at all and 5 

being it fully benefits youths, rate of your level of agreement on the statement that 

the OYAS benefits youths. 

13. Do you have experience with the automated system? 

In the analyses that follow, the interview and survey data are analyzed as two complete, 

yet, unique groups, and are sometimes combined to draw out the results from the juvenile justice 

personnel in general, regardless of the source of the information (i.e., survey or interview). 

Whereas the web-based survey sample was previously analyzed across the OYAS 

administrator/non-administrator distinction, the sample is analyzed as a whole in the current 

section so as to compare the results more directly to the interview data. 

It should be noted that, in many places where the interviewees were able to indicate they 

were “unsure” of an answer, the survey respondents did not have this opportunity. Therefore, 

where applicable, the limited number of “unsure” responses from the interview data were not 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

148 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

included in these comparative results. Because of this, it is possible that some survey respondents 

who felt unsure about an answer were not able to express their ambiguity, and answered the 

questions with a distinct “yes” or a “no” or intentionally skipped the question. Though there is no 

way to know the exact number of survey respondents who felt ambiguity in response to a given 

question, it is probable that, like the interview data, this would have been a small percentage of the 

sample. 

Job Characteristics. Though the interviewees and survey respondents were all considered 

juvenile justice personnel, the characteristics of work responsibilities and titles were somewhat 

unique across each sub-sample. The descriptive statistics for job characteristics for both samples 

were presented originally in Table 1.2, separated by state. Below, Table 1.24 outlines the job 

characteristics for the interview and survey samples as a whole, setting the context in which the 

average response from interviewees and survey respondents may differ or converge. The top two 

most frequent job settings for either sample were probation and correctional/secure treatment 

facility. Conversely, the interview sample had a higher proportion of court, parole, and state 

agency personnel, whereas the survey sample had a higher proportion of “other” job settings. This 

is due to the nature of the web-based survey, where job settings were inferred from job titles and 

some job titles were not specific enough to be able to do so. The interviewee sample provided 

much more detail in terms of job titles, tasks, and settings, therefore eliminating the need for an 

“other” category. 

In terms of job classification, the interview sample had a higher proportion of 

administrators/directors, judges/magistrates/attorneys, and supervisors/managers. The survey 

sample had a higher proportion of frontline staff, as well as “other” job classifications. This 

revealed the purposive selection of the interview sample, which was meant to include a range of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

149 
 

 

 

 

 

 

 

job roles, but more so individuals who were familiar with the OYAS and its implementation and 

use (i.e., those who drove adoption). This included those in leadership positions who led the 

planning and execution of the OYAS implementation in their state or agency. Therefore, the results 

from the interview sample were more representative of the opinions of those who have had these 

experiences. The survey sample was more representative of the juvenile justice practitioner 

population, which is weighted toward frontline, probation-based staff. Therefore, results from the 

survey sample were more representative of the opinions of those who were interacting daily with 

the OYAS and had to translate its results into treatment and supervision decisions. The survey 

sample had a higher proportion of personnel who administer the OYAS than the interview sample, 

though use of its automated system is relatively evenly split across the samples. This is possible 

as even those who do not administer the OYAS may have access to its automated system for 

supervisory or treatment purposes. 

Table 1.24. Descriptive Statistics for Job Characteristics Across Interview and Survey Samples 

Variable Interviews Surveys Total Sample 

(N = 217) (N = 1,013) (N = 1,230) 

Job Setting n (%) n (%) n (%) 

Court 37 (17.1) 102 (10.2) 139 (11.4) 

Probation 105 (48.4) 592 (59.3) 697 (57.4) 

Parole 14 (6.5) 7 (0.7) 21 (1.7) 

Correctional/Secure Treatment Facility 44 (20.3) 132 (13.2) 176 (14.5) 

State 17 (7.8) 8 (0.8) 25 (2.1) 

Other 0 (0.0) 157 (15.7) 157 (12.9) 

Total 217 (100.0) 998 (100.0) 1,215 (100.0) 

Job Classification n (%) n (%) n (%) 

Administrator/Director 30 (13.8) 50 (5.7) 80 (7.3) 

Judge/Magistrate/Attorney 21 (9.7) 50 (5.7) 71 (6.5) 

Frontline Staff 120 (55.3) 626 (71.7) 746 (68.4) 

Supervisor/Manager 45 (20.7) 86 (9.9) 131 (12.0) 

Other 1 (0.5) 61 (7.0) 62 (5.7) 

Total 217 (100.0) 873 (100.0) 1,090 (100.0) 

Administer OYAS n (%) n (%) n (%) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

150 
 

 

 

 

 

 

 

 

 

 

Table 1.24. Descriptive Statistics for Job Characteristics Across Interview and Survey Samples 

Variable Interviews Surveys Total Sample 

(N = 217) (N = 1,013) (N = 1,230) 

Yes 118 (54.4) 626 (74.0) 744 (70.0) 

No 99 (45.6) 220 (26.0) 319 (30.0) 

Total 217 (100.0) 846 (100.0) 1,063 (100.0) 

Use Automated System n (%) n (%) n (%) 

Yes 134 (74.9) 453 (75.9) 587 (75.6) 

No 45 (25.1) 144 (24.1) 189 (24.4) 

Total 179 (100.0) 597 (100.0) 776 (100.0) 

Job Tenure x̅(sd) x̅(sd) x̅(sd) 

13.26 (8.25) 13.86 (8.47) 13.72 (8.41) 

Due to missing data, totals are less than the reported sample size. 

Each of these settings may bring a unique set of job tasks and characteristics that both 

dictate the personnels’ daily interactions with the OYAS, as well as their opinions on the usefulness 

of the tool. For example, juveniles’ risk level’s (as dictated by the OYAS) may determine their 

frequency of contact with their probation officer. For probation officers, this may be beneficial or 

a hindrance to the supervision process. Another example is the “state” employees, whose 

involvement with the OYAS is largely administrative and oversight-driven. These individuals may 

have a better sense of the big-picture implementation of the OYAS, but less daily practice in 

conducting the OYAS assessments and translating the results into case planning decisions. The 

interviewees and survey respondents also had varying degrees of tenure in the field of juvenile 

justice, however these did not differ significantly. In general, the average juvenile justice 

interviewee or respondent had been in the field for over 13 years, and therefore has a great deal of 

experience. Many of these individuals may have worked in the field prior to the widespread 

adoption of risk and needs assessment, and could have experienced a major shift in the strategies 

and methods behind juvenile justice supervision over the years. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

151 
 

 

 

 

 

 

 

 

 

 

 

 

In summary, survey respondents have been working in the field of juvenile justice for 

slightly longer than interviewees, were more likely to be frontline staff working in a probation 

setting, and were more likely to personally administer the OYAS to youths. To the contrary, though 

these same job characteristics were common in the interview sample also, the interviewees were 

more likely than the survey respondents to be administrators/directors, 

judges/magistrates/attorneys, or supervisors/managers working in a court, parole, or state agency 

setting. The two samples taken together, the majority of personnel worked in a probation job 

setting, were considered frontline staff, administered the OYAS, and used its automated system. 

Therefore, they have had an intimate familiarity with the OYAS, its processes, and their respective 

agency’s usage practices. They have also worked in the field for a significant amount of time on 

average, and have likely had to adapt to major shifts in juvenile justice supervision over time, 

including (but not limited to) the wide adoption of risk and needs assessments. 

General Attitudes About Risk and Needs Assessment. Table 1.25 below outlines the 

general attitudes of the juvenile justice personnel interview and survey samples regarding risk and 

needs assessment, as well as some that are more specific to the OYAS. Survey respondents gave 

significantly lower ratings than the interviewees on their personal level of satisfaction with the 

OYAS, whether risk and needs assessments reduce bias in juvenile justice decision-making, and 

whether the OYAS specifically benefits the youths. However, it is noteworthy that the average 

response on all three of these items leaned toward agreement. 

In the combined sample, personnel rated their level of agreement that risk and needs 

assessments reduce bias and that the OYAS benefits the youths higher than their personal level of 

satisfaction with the OYAS. This mirrors what was presented in the interview and survey results 

sections of the report thus far. While personnel were able to acknowledge the tool’s usefulness in

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

152 
 

 

 

 

 

 

 

 

 

 

creating more equitable and beneficial supervision strategies for the youths they serve, they also 

expressed a degree of dissatisfaction with the OYAS itself, and/or its associated processes or 

implementation. A Chi-Square test revealed no significant differences in the items in Table 1.25 

when analyzing the combined personnel sample by state. Therefore, the source of the information 

(i.e., survey or interview) appears to be the more influential factor for these items. As shown in 

Table 1.25, interviewees rated all three items significantly higher than survey respondents. 

Table 1.25. Descriptive Statistics for General Attitudes about Risk and Needs Assessment and 

the OYAS Across Interview and Survey Samples 

Variable Min. Max. x̅ sd 

Overall Satisfaction with OYAS ^ 

In-person Interview (N = 200) 1 5 3.56 0.86 

Web-based Survey (N = 865) 1 5 3.01 1.06 

Combined JJP sample 1 5 3.12 1.04 

(N = 1,065) 

Risk and Needs Assessments Reduce Bias ^^ 

In-person Interview (N = 208) 1 5 3.80 1.00 

Web-based Survey (N = 797) 1 5 3.20 1.04 

Combined JJP sample 

(N = 1,005) 1 5 3.32 1.06 

OYAS Benefits the Youths ^^^ 

In-Person Interview (N = 201) 1 5 3.60 1.01 

Web-based Survey (N = 771) 1 5 3.27 0.97 

Combined JJP sample (N = 972) 1 5 3.34 0.99 

Due to missing data, totals are less than reported sample size. 

^ p < .001, 2

(5) = 60.71, Cramer’s V = .24 

^^ p < .001, 2

(4) = 84.60, Cramer’s V = .29 

^^^ p < .001, 2

(4) = 48.28, Cramer’s V = .22 

Figures 1.16a-c outline the detailed distribution of responses for the three items in Table 

1.25. Figure 1.16a below shows that there is a degree of neutrality towards the OYAS overall, as 

this was the modal category for the full sample. However, a higher proportion of interviewees 

indicated that they were satisfied with the OYAS than survey respondents (roughly 60% vs. 34%, 

respectively). As mentioned previously, this may be the product of the purposive sampling method 

used for the interview sample, which contains more of the individuals who championed the use of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

153 
 

 

 

 

 

a risk and needs assessment tool in their state, and likely played a role in planning its 

implementation. It may also be a byproduct of interviewees rating items higher due to the interview 

format. 

Figure 1.16a. Comparison of Interviews and Surveys: Overall Satisfaction with the OYAS 

(N = 1,065) 

100 

% in Category 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

Completely Not 

Satisfied 

Not Satisfied Neutral Satisfied Completely Satisfied 

Interview (N = 200) Survey (N = 865) 

Figure 1.16b below outlines the detailed distribution of juvenile justice personnel responses 

to whether risk and needs assessments reduce bias in juvenile justice decision-making. Though the 

modal category for both the interview and survey sample was “agree,” a higher proportion of

interviewees than survey respondents either “agreed” or “strongly agreed” with this statement 

(66% vs. 44%). Therefore, though the averages in Table 1.25 revealed that both samples tended to 

agree with this statement, the frequency distribution below highlights the degree of enthusiasm 

with which each sample provided their responses. Interviewees tended to agree more strongly that 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

154 
 

 

 

 

 

risk and needs assessments reduce bias whereas this may be a slightly weaker belief for survey 

respondents on average. 

Figure 1.16b. Comparison of Interviews and Surveys: Risk and Needs Assessments Reduce Bias 

(N = 1,005) 

100 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

Strongly Disagree Neutral Agree Strongly Agree 

% in Category 

Survey (N = 797) 

Disagree 

Interview (N = 208) 

Lastly, Figure 1.16c below outlines the detailed distribution of responses as to whether the 

OYAS benefits the youths in their agency. Once more, the interview sample had a higher 

proportion indicating agreement with this statement (roughly 66% to 45%), while the survey 

sample had more disagreement with this statement (26% to 8%). However, based on overall 

prevalence, both samples leaned toward agreement that the OYAS is beneficial to the youths in 

their agencies. In all three items depicted in Figures 13a, 13b, and 13c, juvenile justice personnel 

expressed more positive sentiment regarding the OYAS and risk and needs assessment in general, 

which lends support to the notion that there is a foundation upon which to build and grow staff 

buy-in for a risk and needs assessment tool. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

155 
 

 

 

Figure 1.16c. Comparison of Interviews and Surveys: OYAS Benefits the Youths (N = 972) 

100 

90 

80 

0 

10 

20 

30 

40 

% in Category 70 

60 

50 

Strongly Disagree Disagree Neutral Agree Strongly Agree 

Interview (N = 201) Survey (N = 771) 

Agency Assessment Strategies. Figure 1.17 below depicts data regarding juvenile justice 

personnel perceptions of their respective agency’s strategies and practices surrounding the OYAS. 

Specifically, the samples were asked whether their agencies allow for overrides to the OYAS, use 

additional assessments, and have formal quality assurance measures in place. While overrides 

appeared to be allowed in the vast majority of agencies, personnel indicated that additional 

assessments are used to supplement the OYAS far less frequently. 

Figure 1.17. Comparison of Interviews and Surveys: Agency Assessment Strategies 

100 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

% Affirmative Answer 

Interview Survey 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

156 
 

 

 

 

 

 

 

 

 

 

Even less common were formal quality assurance measures, which only about one-third of 

the full sample suggested are in place. These results indicate that there are gaps where the 

implementation literature would support the use of these important strategies, which have their 

own implications for increasing post-rollout sustainability of a risk and needs assessment. Whereas 

the use of responsivity assessments helps to tailor interventions to a youth’s strengths and barriers,

as well as capture information that many of the personnel perceive to be lacking in the OYAS, 

formal quality assurance measures (examples provided in “Implications and Recommendations”) 

serve as the backbone to the entire implementation process, and should be established in all 

agencies that adopt a risk and needs assessment. 

OYAS Information Usage. The last comparison between the interview and survey data is 

in how agencies use the assessment information. Figure 1.18 depicts overall responses to a number 

of items that are common uses of risk and needs assessment information, some of which are 

supported by the risk and needs assessment literature (Vincent et al., 2012). Four of these items 

were relatively prevalent in responses across the samples, including using the assessment 

information to determine supervision level, measure youths progress, match youths to treatment, 

and assist in juvenile justice decision-making. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

157 
 

 

 

 

 

 

 

 

Figure 1.18. Comparison of Interviews and Surveys: Assessment Information Usage Practices 

0 

10 

20 

30 

40 

50 

60 

70 

80 

Match Youth to 

Treatment 

(N = 944) 

Assist in 

Decision-making 

(N = 926) 

% Affirmative Response 

100 

90 

Allocate Develop Determine Measure Youth 

Resources Specialized Supervision Level Progress (N 

(N = 891) Caseloads (N = 943) = 928) 

(N = 923) 

Interview Survey 

It was far less common that personnel indicated their agencies use the assessment 

information to allocate resources within the agencies or to develop specialized caseloads. These 

trends remained relatively stable throughout the responses of the interview and survey samples 

and across the three states. Therefore, it is clear that OYAS information is being used in the more 

basic sense of establishing treatment and supervision objectives, but the data may be utilized to a 

greater extent in overall planning and case management. 

OYAS Strengths and Limitations. The personnel-generated strengths and limitations of 

the OYAS were similar across the interview and survey samples, which illustrates their potency in 

risk and needs assessment implementation. Many of these themes referenced the OYAS itself, 

while others pertained to the broader picture of implementation. For strengths, it was clear that 

personnel sensed that the OYAS increases fairness and encourages objectivity in juvenile justice 

decision-making. In this way, personnel did not seem to be against the concept of using actuarial 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

158 
 

 

 

 

 

 

 

 

 

 

risk and needs assessment to promote consistency, but rather expressed their concerns regarding 

the way that JRNAs (and the OYAS) are implemented and used. Personnel in both samples also 

noted that the OYAS information is helpful in making juvenile justice decisions (e.g., disposition, 

diversion). 

Personnel had more consensus regarding the limitations of the OYAS, most commonly 

mentioning concerns about specific items on the assessment, the validity of the OYAS, and the 

reliability with which it is administered and scored. The prevalence of these concerns across the 

samples has major implications for how personnel perceive and use OYAS information. Increasing 

the validity and reliability of OYAS assessments (along with personnel perceptions of validity and 

reliability) is therefore key to promoting optimal usage practices. Lastly, the personnel were in 

agreement that the OYAS is time-consuming, which speaks to the busy nature of their daily work, 

and lends itself to “shortcuts” (e.g., cutting assessment questions, overlooking collateral 

information) that may generate validity and reliability questions. As such, some OYAS limitations 

are of a somewhat recursive in nature, effectively “snowballing” with other perceived, interrelated, 

limitations to affect the implementation and use of the tool in practice. 

General Conclusions from the Integrated Analysis of the Interview and Survey Data. 

Analysis of the integrated samples reveals several key conclusions. First, there is a solid foundation 

upon which to build optimal implementation strategies and practices. Throughout the report, 

personnel attitudes and perceptions of the OYAS and risk and needs assessment generally 

highlighted that they agree with the potential benefits of a risk and needs assessment 

implementation, including its potential to reduce bias and enhance fairness in juvenile justice 

decision-making. However, the results diverged somewhat when examining some of the more 

micro-level policies and practices related to risk and needs assessment. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

159 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Second, formal quality assurance measures came up rarely in interview and survey results, 

which suggests that personnel may be “on their own” in terms of administering the OYAS 

consistently and acquiring an accurate measure of risk. When these practices go awry, personnel 

are understandably left to feel that the OYAS is an inaccurate tool, or does not provide information 

that is useful to them in their daily work with youths. The development of formal and systematic 

quality assurance at the local- and state-level is therefore of the utmost importance, and will be 

discussed in the implications and recommendations. 

Third, the OYAS information is being used in line with some key system objectives, which 

juvenile justice personnel believe has helped to streamline and enhance fairness in the treatment 

of youths. Across the interview and survey samples, OYAS information was consistently used to 

determine supervision level and match youths to treatment, which are two key elements according 

to the principles of effective intervention. However, the data suggest that agencies are not taking 

full advantage of the potential uses of OYAS information, which may also contribute to some 

personnel sensing of a lack of utility for OYAS information. As OYAS information usage and 

sharing increases in an agency, it is likely that buy-in for the importance of this information will 

increase. However, it is important to keep in perspective the purpose and scope of a risk and needs 

assessment tool, which is only one piece of the “rehabilitation puzzle,” and is intended to more 

broadly affect information gathering and decision making in conjunction with other elements of 

the juvenile justice implementation context. 

Lastly, the results diverged in various places according to state context, as well as sample 

job characteristics. Several state-level initiatives were outlined in Figure 1.2 that suggested a 

unique implementation environment in each of the three states included in the current study. 

Additionally, Table 1.2 outlined the job characteristics for each sample that indicate meaningful 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

160 
 

 

 

 

 

 

 

 

 

 

 

 

 

differences within the counties and agencies in each state. In many cases, these contextual 

differences impacted the perceptions and usage of the OYAS. This became particularly apparent 

when examining the perceptions of frontline staff and non-frontline staff. Their unique job roles 

force them to interact with the tools in different ways, which leads to divergent opinions on its 

usefulness in supervising youths and promoting fairness throughout that process. Therefore, it is 

important to strike a balance between honoring local context, and promoting uniformity in a 

statewide risk and needs assessment implementation. 

Usage Study 1: Support for the OYAS and Integrity in the Juvenile Risk and Needs 

Assessment Process 

As mentioned in the beginning of this section of the report, juvenile justice personnel 

surveys (in this case, web-based surveys) can provide insight regarding perceptions of JRNA usage 

that can impact the integrity of the tool. The introduction of randomized vignettes and assessment 

scoring into the web-based survey allowed us to gather some information on the application of the 

OYAS among the respondents who administer it. 

Risk and needs assessment implementation is not only contingent on administrative tasks 

(i.e., updating policy and procedures, providing training), but also how receptive staff are to using 

the assessment system. Ultimately, perceptions about risk and needs assessment tools may shape 

the integrity of their implementation (Vincent et al., 2012). Considering that practitioners play a 

central role in fostering change, understanding attitudes towards risk and needs assessments is 

important (Fixsen et al., 2009). A risk and needs assessment tool may be implemented and 

mandated in an agency; however, if there is little support for the tool’s use, the integrity of the tool 

may be impacted. Negative attitudes towards innovation—in this instance adopting the OYAS in 

case assessment and management—may hinder their proper use (Corrigan et al., 1998). Negative 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

161 
 

 

 

 

 

 

 

 

 

 

 

attitudes toward administering assessment tools may lead staff to inappropriately score the tools, 

which can ultimately impact case management and the allocation of treatment services. 

While research assessing perceptions specific to the OYAS is limited, studies have 

examined attitudes toward risk and needs assessments and how these views impact the utility of 

the tool. Schneider, Ervin, & Snyder-Joy (1996) assessed a number of articles spanning 15 years, 

and conducted in-depth interviews with probation officers and supervisors from Oklahoma 

Department of Corrections. A questionnaire was also sent out to all probation and parole offices 

in the state (N = 296, responses rate 60%, final N = 179). The goal of the questionnaire was to 

provide insight from practitioners on specific issues regarding risk and needs assessment tools. 

Results indicated that there was a strong correlation between the officer’s belief that risk and need 

instruments are useful and perceptions of its effectiveness. This study did not examine how 

perceptions impact the implementation of the tool, however. 

Similarly, Shook & Sarri (2007) assessed how the utility of a tool may drive perceptions. 

The study examined the Structured Decision-Making’s (SDM) usefulness at specific decisionmaking points (e.g., pretrial detention, post-adjudication placement) among court professionals. 

Survey questions tapped into court professionals’ opinions of specific reasons for the 

implementation of SDM and their perceptions of whether SDM was meeting these goals. Items 

pertaining to the key reasons for SDM implementation included whether the tool (1) ensured 

appropriate placement, (2) reserved commitment for most serious offenders, and (3) made 

placement and decisions consistent. These items spanned both positive and negative perceptions 

concerning the tool. Overall, support for a tool can also stem from its utility in providing a uniform 

framework for decision-making and this may impact whether it is used for placement decisions 

(Shook & Sarri, 2007). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

162 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

These prior studies highlight how perceptions of the usefulness of a risk and needs 

assessment tool may impact views concerning its effectiveness and ultimately, how it is used. We 

extend this research by looking at whether support for risk and needs assessments and clarity on 

their usefulness may impact the integrity of their use. We first measure and describe the level of 

support for the OYAS and understanding of its usefulness. Then, using multinomial logistic 

regression, we assess whether the level of support is associated with how accurately practitioners 

score it. 

Method. As previously noted, the web-based survey included a sample 1,013 juvenile 

justice personnel. The present analysis is primarily concerned with those who administer the 

OYAS tool(s). Therefore, any participants who indicated “yes” to the survey item, “do you

administer any of the instruments?” were selected. The final sample of those who administer the 

OYAS included 626 participants from the three states. 

Measures 

OYAS Usefulness. A one-factor measure was proposed for OYAS usefulness. First, 

correlations were examined to assess the relationship between key Likert measures concerned with 

how personnel view the usefulness of the OYAS. Eight 5-point Likert scale items, ranging from

strongly disagree to agree, assessed the usefulness of the OYAS in the general areas of identifying

criminogenic needs, determining supervision levels, developing case plans, decision-making, and

treatment progress. Table 1.26 provides the descriptive statistics for the Likert items captured by

the OYAS usefulness factor. The internal consistency was also examined to determine how closely

related the set of items are as a group. Cronbach’s alpha was equal to 0.91, which is an acceptable

level (Cronbach, 1951; Bland & Altman, 1997). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

163 
 

 

 

Table 1.26. Descriptive Statistics for OYAS Usefulness Items (N = 626) 

Variable Strongly 

Disagree Disagree Uncertain Agree Strongly Agree Total51 

n n n n n n 

(%) (%) (%) (%) (%) (%) 

OYAS helps identify 

youths’ criminogenic 

needs 

16 

(2.6) 

46 

(7.3) 

73 

(11.7) 

323 

(51.6) 

69 

(11.0) 

527 

(84.2) 

OYAS is useful in 

determining youths’ 

supervision levels 

21 

(3.4) 

61 

(8.1) 

55 

(8.8) 

299 

(47.8) 

89 

(14.2) 

515 

(82.3) 

OYAS is useful in 

providing information to 

develop comprehensive 

case plans 

22 

(3.5) 

55 

(8.8) 

89 

(14.2) 

295 

(47.1) 

56 

(8.9) 

517 

(82.6) 

OYAS helps determine 

appropriate interventions 

based on youths’ risk 

levels 

22 

(3.5) 

74 

(11.8) 

72 

(11.5) 

308 

(49.2) 

51 

(8.1) 

527 

(84.2) 

OYAS is useful in 

matching youth to 

appropriate treatment 

services 

20 

(3.2) 

68 

(10.9) 

99 

(15.8) 

283 

(45.2) 

49 

(7.8) 

519 

(82.9) 

OYAS is useful in 

measuring progress in 

addressing youths’ risks

and needs 

17 

(2.7) 

68 

(10.9) 

91 

(14.5) 

292 

(46.6) 

50 

(8.0) 

518 

(82.7) 

OYAS is useful in 

assisting in diversion, 

disposition, placement, 

or release decisions 

21 

(3.4) 

74 

(11.8) 

83 

(13.3) 

285 

(45.5) 

55 

(8.8) 

518 

(82.7) 

The use of the OYAS 52 162 120 165 13 512 

has made my job easier (8.3) (25.9) (19.2) (26.2) (2.1) (81.2) 

We conducted a confirmatory factor analysis (CFA) in MPlus to further evaluate this 

usefulness factor (Muthѐn & Muthѐn, 1998-2018). Overall, the model fit the data well, which is 

51 Totals may not reflect overall sample size (N = 626) due to missing data. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

164 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

evidenced by a Kaiser-Meyer-Olkin (KMO) close to 1, comparative fit index (CFI)/Tucker-Lewis 

index (TLI) greater than the .95 threshold, and a Chi-Square test of model fit that was an 

improvement over the baseline model (2

(20) = 97.79, p < .01). The standardized factor loadings 

range from .76 to .86 and were statistically significant. The root mean squared error of 

approximation was acceptable (RMSEA = 0.08) (MacCallum, Browne, & Sugawara, 1996). 

Additionally, the weighted root mean square residual (WRMR) approached a value close to 1 

which is indicative of good model fit (Brown, 2006). Generally, these values were in line with 

conventional cutoffs (e.g., Hu & Bentler, 1999). Full model fit information and individual factor 

loadings are provided in Appendix E. 

Integrity. As described above, case vignettes, which were based on OYAS training 

processes, were randomly assigned to those who indicated “yes” to the question about whether

they administer tool(s). The outcome of interest is consistency with the manualized score for each 

case. One of three vignettes was randomly assigned to each OYAS administrator within the 

structure of the survey. These vignettes are based on domains in the Disposition Tool and measure 

whether administrators can accurately score a hypothetical case. The three OYAS Disposition 

domains used for the vignettes were the following: (1) Pro-Social Skills, (2) Juvenile Justice 

History, and (3) Peers and Social Support Network. For each vignette, a link was provided that 

contained the scoring guide for that domain. OYAS administrators read a hypothetical case and 

scored their respective domain based upon the provided information. Table 1.27 shows the 

distribution of respondents across each vignette. Out of the 626 self-reported administrators, only 

70 percent completed the vignette portion of the survey (N = 437). The proportion of cases for 

each vignette were relatively similar at approximately 23 percent of the sample for each vignette. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

165 
 

 

 

 

 

 

 

 

 

 

Table 1.27. Distribution of Administrators by Vignette (N = 626) 

Number of OYAS % of Total Vignette Administrators Sample 

Juvenile Justice History 145 23.2 

Peers and Social Support Network 149 23.8 

Pro-Social Skills 143 22.8 

Total 437 69.8 

Scored Vignette Difference. In order to quantitatively measure integrity, a scored vignette

difference measure was created. Measuring movement away from the true score or “scoring errors”

is not unique to this study. Van der Knaap et al. (2012) examined the Dutch RISc assessment tool 

(Recidivism Assessment Scale) (see also Rocque & Plummer-Beale, 2014). Separate assessments 

by two probation officers were conducted then compared to one another. Their agreements and 

disagreements were summed together and turned into a proportion. The scale of 0.00 to 1.00 for 

the proportion of agreement was given cutoffs for comparison: slight (0.00-0.20), fair (0.21-0.40), 

moderate (0.41-0.60), substantial (0.61-0.80), or almost perfect (0.81-1.00) (based on Landis and 

Koch, 1977). 

Given the three different vignettes, the difference between what the administrators scored 

the vignette and the true score was created. A scale was created rather than simply looking at 

whether staff scored the vignette correctly or not since there is a margin of error built into the 

OYAS tools. In training and for certification to be an OYAS assessor, staff can score within +/- 3 

of the benchmark score and still accurately assess the youth. While we are only focusing on one 

domain of the Disposition Tool, we wanted to follow this logic and allow for room the true score 

(i.e., benchmark score). First, each vignette was summed based on what score OYAS 

administrators provided. A benchmark score was also created, which is the correct total for each 

vignette. The scored vignette difference variable is the difference between the respondent’s score

and the benchmark score. This variable provides an ordinal scale that captures movement away 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

166 
 

 

 

 

 

 

 

 

 

 

 

 

from the benchmark score. A smaller proportion of the sample scored three (n = 1) or two (n = 17) 

below the benchmark score with the majority scoring one below the benchmark score (n = 130). 

Given this, those who scored three, two, and one below the benchmark score were combined into 

one category. Also, underscoring is similar across all vignettes such that scoring under the 

benchmark indicates less risk for the youth in comparison to over-scoring, which is indicative of 

more risk. Table 1.28 provides the distribution for scored vignette difference. 

Table 1.28. Distribution of Scored Vignette Difference 

Scored Vignette Difference n %

Scored -3, -2, or -1 below benchmark score 148 33.9

Scored correctly 229 52.4

Scored +1 above the benchmark score 60 13.7

Total 437 100.0

Control Variables. Some vignettes were longer than others, therefore, 3 dummy variables 

were created to control for any differences. The Pro-Social Skills vignette is the reference category. 

Gender was also included as a control variable. Other relevant control variables include work 

experience (in years), highest education level obtained, and whether an administrator was retrained 

in the OYAS. All of these controls may influence a respondent’s familiarity with administering 

the OYAS such that older individuals with higher education levels and who received retraining 

may be more likely to score the vignette correctly. 

Analytic Plan. A multinomial logistic regression was estimated to assess the impact of 

staff support on integrity in risk and needs assessment scoring. Before the analysis was conducted, 

the proportional odds assumption was investigated to assess whether the predictors have the same 

effect on the odds for each category. In order to meet this assumption, the test of parallel lines in 

SPSS should be not be significant. The parallel lines test indicated that the full model with controls 

violated this assumption (2 = 33.94(8), p < .001). The analyses then defaulted to multinomial 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

167 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

logistic regression due to the violation of the proportional odds assumption. Therefore, separate 

coefficients are estimated for each category rather than assuming that the regression coefficients 

are the same for all 4 categories (e.g., -2, -1, 0, 1 under/over benchmark score) (Norusis, 2012). 

Multinomial logistic regression contains the same assumption as binary logistic regression (i.e., 

independence among the predictor categories and a categorical dependent variable), but the 

outcome has more than two categories. 

Results 

Sample Demographics. The relevant controls used in the multinomial logistic regression 

models and additional sample descriptors are shown in Appendix F. Probation and parole officers 

comprised the majority of the sample (73%). A little less than half of the sample is from State 1 

(40%). About half of the sample indicated they had a bachelor’s degree (50.3%). On average, staff

have worked in the field for approximately 15 years (x̅= 14.9, sd = 8.4). Females constituted 45 

percent of the sample. Also, 43 percent indicated that they had some retraining on the OYAS 

tool(s) (i.e., booster training or re-certification; see Table 1.29). 

Multivariate Model with Support and Controls. The statistically significant -2 log 

likelihood Chi-Square test indicates that the full model fit the data well (p < .001). This model has 

a Nagelkerke pseudo r-square of .25.52 OYAS usefulness significantly and negatively predicted the 

relative risk of scoring below the benchmark score relative to those who scored the vignette 

correctly. Specifically, as perceived usefulness increases there are lower odds of being in the 

"underscored" category than in the "correct" group (-28%). Additionally, the model suggests that 

receiving the Juvenile Justice History vignette significantly and negatively predicted the relative 

52 Job title was not included as a control because it was not statistically significant in a bivariate test. Most participants 

indicated that they were a probation or parole officer leaving little variation in the other job title categories reported. 

The job title measure was further refined with collapsed categories and dummy variables were also created for each 

job title. Findings still suggested that job title had no substantive impact on our findings. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

168 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

risk of falling into the “-3, -2, or -1 difference” category relative to the “correct score” category. 

This indicates that those who received the Pro-Social Skills vignette were more likely to score -3, 

-2, or -1 below the benchmark score. In part, this may be due to the fact that items in this domain 

are more difficult to score (i.e., somewhat more subjective than Juvenile Justice History). Only 

one measure had a significant relationship with the +1 Difference Score Outcome. Education level 

significantly and negatively predicted the relative likelihood of scoring 1 above the benchmark 

score compared to those who correctly scored the vignette. Specifically, those with higher levels 

of education were less likely to score 1 over the benchmark score than correctly compared to those 

with lower levels of education. 

Table 1.29. Multinomial Logistic Regression Predicting Scored Vignette Difference (N = 402) 

Variable -3, -2, or -1 Difference Score +1 Difference Score 

Independent Variables b(SE) OR b(SE) OR 

OYAS Usefulness -.33(.12)** .72 .17(.20) 1.19 

Control Variables b(SE) OR b(SE) OR 

Peers Vignette .32(.27) 1.38 -22.56(.00) 

JJ History Vignette -.97(.32)** .38 -.30(.33) .74 

Work Experience .14(.09) 1.15 .07(.12) 1.07 

Male -.17(.24) .84 .50(.35) 1.65 

Education Level .26(.20) 1.30 -.62(.27)* .54 

Retraining .06(.24) 1.06 .20(.33) 1.22 

Intercept -1.71(.98) - -.31(1.27) -

Model Fitting Information 

-2 Log Likelihood 681.49 

Chi-Square (df) *** 194.47(14) 

Nagelkerke R2 .25*** 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

169 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

*p < .05, **p < .01, ***p < .001; Odds Ratio for “Peers Vignette” not shown due to sparse cells in difference score 

category. 

Summary of Support for the OYAS and Integrity Usage Study. Overall, the results 

from the present study suggest that assessment administrators with higher levels of perceived 

support for utility of the assessment system (i.e., OYAS usefulness) score the domains more 

accurately than those with lower levels of perceived support. Those who see the utility of the 

OYAS and believe in its ability to help identify supervision levels, criminogenic needs, and assist 

in case planning tended to score youth correctly rather than those who have less belief in their 

usefulness. Taking stock in what the OYAS is designed to do can influence integrity or rather the 

degree to which a tool is scored accurately, as suggested in the present study. Those who 

underscore the tool may not see the utility of using it to its full potential and may believe they 

think they know what is best for a youth. This in turn, may lead staff to not engage in proper 

information gathering during the interview to effectively score out the tools. 

While deviation from the benchmark score is a solid measure of integrity and has been 

used in other studies, the usefulness measures may have more robust effects if other outcome 

measures were included, such as the use of overrides. If a PO is continuously overriding a tool 

(without a specific cause, such as if the youth was a sex offender or had a gun specification) then 

the integrity of the tool may be impacted such that he or she is using their own judgment to move 

a youth to a different risk level. This may induce variation within and across agencies. Also, quality 

assurance practices should also be explored since it is an integrity check on whether staff are 

accurately scoring the tool and using it to drive decision-making. 

There are some limitations in this portion of the study. Only one domain from the Disposition 

Tool was scored by participants. Having participants score a full assessment may better help us to 

fully understand perceived support for the usefulness of the tool and its impact on integrity. This 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

170 
 

 

 

 

 

 

 

 

 

 

 

 

 

may allow for a greater margin of error to properly disentangle the effect of usefulness. There was 

an approximately 30 percent non-response rate (N = 179) for the vignettes. Specifically, these 

respondents did not answer any of the vignette items. Out of the 179 cases that did not participate 

in the vignettes, close to 90 percent of these cases (N = 158) did not finish the web-based survey. 

Upon further examination, approximately 60 percent of those who did not finish the survey failed 

to go past the scaled items concerning the usefulness of the OYAS, which appear prior to the 

vignette portion of the survey. It may be that those with more negative attitudes toward the OYAS 

did not complete this portion, and consequently the vignette portion, which could potentially 

impact our results. It is likely, however, that such a selection effect would tend to push less 

enthusiastic OYAS administrators out of our current analytic sample, further depressing the 

prevalence of correct scoring, which is about 52 percent in the current sample. A small number of 

cases did not fully complete the vignettes (N = 10). These individuals only partially scored items 

across the vignette. The usage study in this report focused on those who completed the vignettes 

in order to further investigate the scoring on the vignette items. 

Despite the potential limitations, this report is a first step in unpacking important JRNA 

implementation and system usage patterns. The analysis of relationships between attitudes toward 

implementation and general perception of the OYAS and, in turn, the tool’s perceived usefulness 

and administrator performance in these final portions of this first section of the report highlight 

the complex relationship between implementation processes, juvenile justice personnel 

perceptions, and assessment usage. It is therefore important that we must look at things 

comprehensively in order to understand how the different pieces fit together and eventually affect 

case processing, decision-making, and outcomes. We turn further attention to that task in the next 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

171 
 

 

 

 

 

 

 

 

 

 

sections of the report, which focus heavily on records of what happens to youths in the system as 

a result of risk and needs assessment and what it means for their outcomes. 

Summary of Section I 

Section I focused on the first study objective: to describe and assess risk and needs 

assessment usage and implementation practices at different juvenile justice decision-points. 

Overall, the interviews and surveys noted some positivity towards the OYAS, but also highlighted 

numerous ways to improve the use of the tool and ways to improve the implementation process. 

Our first usage case study also suggested that support for the tool and its usefulness may affect 

how personnel score it. As such, if sites can increase support for the assessment—especially 

through thoughtful implementation strategies—the validity and reliability of the assessment may 

be higher. Another main focus in the first objective was to develop recommendations on best 

practices in training, monitoring, and usage. We will come back to this topic at the end of the report 

in Section IV. 

Section II of the report moves us toward Study Objectives two and three: (2) assess court 

and programming decision-making outcomes based on variation in risk and needs assessment 

usage and implementation practices across agencies and states; and (3) evaluate how the 

implementation of risk and needs assessments across multiple stages of the juvenile justice system 

impacts recidivism among juvenile offenders. 

Section II. Juvenile Risk and Needs Assessment Usage, Process, and Case Decisions 

The perceptions of those in the field are essential in describing and evaluating 

implementation, as they cannot be easily disentangled from the new practice. Fixsen et al., (2009: 

532) note that frequently “the practitioner is the intervention” and this is definitely true in case

assessment and decision-making in juvenile justice. The insights of those involved in planning, 

using assessments, and making decisions offer the context in which other elements of the process 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

172 
 

 

 

 

 

 

 

 

 

 

 

 

 

must be considered. At the same time, that insight is only one step in moving beyond a “black 

box” understanding of risk and needs assessment in relation to youths’ outcomes. This 

comprehensive study of implementation and use goes beyond practitioner views to focus on the 

mechanisms through which an initial assessment might affect decision-making at the case and 

aggregate-level. 

We utilize the case record data available for this project to consider several examples of 

risk and needs assessment usage through five targeted usage studies. Together these two elements 

offer insight on different elements of usage by first presenting an overview of our study data, and 

then additional usage cases that help to explore different facets of JRNA and its application in the 

juvenile justice system. These usage cases include an analysis of geographic and subgroup 

consistency in decisions (Usage Study #2); measurement and predictive validity of the OYAS in 

different race/ethnicity groups (Usage Study #3); an assessment of patterns from cases where 

professional judgment was used to override the result of the OYAS process (Usage Study #4), an 

analysis of the linkage between overall and domain risk scores and the processing and treatment 

decisions they are intended to affect—including an emphasis on specific problems like substance 

abuse and mental health (Usage Study #5); and consideration of primary and second-order uses of 

the information from the tools and assessment processes (e.g., strengths and barriers; Usage Study 

#6). Each of these studies draws on the case record data provided by agencies in each of the three 

state sites with the intent of extending the understanding of applied JRNA. Prior to reviewing each 

usage study, an overview of the methodology and general description of the composition of the 

samples of youths is provided. 

Methodology 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

173 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The research objectives related to assessment usage and youths’ outcomes were met using 

a combination of official record data and interviews with youths. This section describes the 

sampling, data collection and management, and measures pertinent to those elements of the study. 

This sets the stage for several different analyses described in the next two sections of this report. 

Comprehensive Assessment Sample. Upon the writing of the proposal for this project, 

existing records indicated that during 2013 over 20,000 assessments were completed in State 1, 

25,000 in State 2, and 15,000 in State 3. This gave the research team an estimate for the number 

of records to expect. Youths assessed with the OYAS in the three participating states during 2013 

through 2017 were included in the study. We received two years of data from each state across 

these four years: 2014 and 2015 in States 1 and 2, and 2013 through 2017 in State 3. This resulted 

in a sampling frame of over 100,000 cases. The sample of cases spans the five stages at which the 

OYAS is administered to allow for sufficient power to analyze assessment and outcome data from 

intake through the back end of the juvenile justice process (Cohen, 1992). Specifically, a greater 

percentage of disposition and reentry cases were selected and extracted relative to their prevalence 

in the sampling frame, as proportionately fewer youths reach those stages. In State 2, for example, 

approximately 11,000 Diversion Tool assessments were completed in 2013 compared to about 

8,000 Disposition and roughly 1,600 Reentry Tool assessments. 

Two-Stage Sampling Frame. For each state, counties were first selected based on their 

assessment usage (total assessment use/population size x 100). Counties were split into three strata 

based on assessment use: 1) low, 2) moderate, and 3) high. For State 1 and State 2, 31 counties 

were randomly selected from each state based on their total number of counties using a random 

number generator approach. The 31 counties served as our base to create the final comprehensive 

assessment sample for each state. From the 31 counties, 24 counties were randomly selected for 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

174 
 

 

 

 

 

 

 

 

 

 

 

our final sample. The remaining seven counties not selected served as replacement counties if we 

were unable to obtain data from one of the initially selected counties. The two-stage sampling 

frame did not apply to State 3 because it has few counties compared to State 1 and State 2, which 

each have dozens of counties. As a result, all State 3 counties were included in our comprehensive 

assessment sample. 

For all three states, cases were then randomly selected from the chosen counties. Cases 

were initially proportioned out per state across the different assessment tool types: 1) Diversion –

200 cases, 2) Detention – 200 cases, 3) Disposition – 600 cases, 4) Residential - 500 cases, 6) 

Reentry – 500 cases. The data were also cross-checked for any duplicate cases to ensure only one 

case per youth was represented. However, due to the use of replacement counties and the data 

received from study sites, these totals varied slightly across states. From the sampling frame of 

over 100,000 for the years indicated above, we selected roughly 2,000 cases per state for a total 

final sample of 6,222 cases. Table 2.1 presents the distribution of cases across the three study 

sites. 

Table 2.1 Distribution of Cases by OYAS Tool and State 

OYAS Tool State 1 State 2 State 3 Combined 

n (%) n (%) n (%) n (%) 

Diversion 

Detention 

Disposition 

Residential 

Reentry 

Total 

207 (10.1) 

179 (8.8) 

672 (32.9) 

479 (23.5) 

505 (24.7) 

2,042 (100.0) 

166 (10.0) 

181 (10.5) 

459 (26.5) 

459 (26.5) 

465 (26.9) 

1,730 (100.0) 

840 (34.3) 

10 (0.4) 

1,073 (43.8) 

508 (20.7) 

19 (0.8) 

2,450 (100.0) 

1,213 (19.5) 

370 (5.9) 

2,204 (35.4) 

1,446 (23.2) 

989 (15.9) 

6,222 (100.0) 

For State 1, all seven replacement counties and additional counties from the overall 

comprehensive assessment sample were used to replace counties that declined to participate 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

175 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

resulting in a total of 42 potential counties.53 For State 2, the initial 24 randomly selected counties 

for the data request process was eventually constrained to 21 counties due to data request 

agreements. Out of a total pool of 21 potential counties for State 2, we received data for 14 

counties. The remaining 7 counties did not participate in the study and replacement counties were 

not available due to the data sharing agreement with the state. 

As noted above, the two-stage sampling frame did not apply to State 3. Cases for State 3 

data were received at two different points. Record data were first received from the juvenile court 

system. This sample included 13 counties, which covered most of the state. The selection process 

described above was applied to the court record data. A second wave of data was received from 

the corrections agencies in the state. This sample included cases from all counties in State 3. The 

corrections sample was reduced to cases with assessment information (N = 680) and then a random 

subsample of 450 cases was extracted for the comprehensive assessment sample. The random 

sample of 450 cases represents a similar proportion of corrections cases received from State 1 

(N=459) and State 2 (N=650). 

Data Collection. Data collection strategies for the comprehensive assessment sample 

varied somewhat across the three states and agencies. This was often driven by agency preferences 

around file transfers. In those cases, we prioritized obtaining data for the core measures (outlined 

below). In most cases we sent requests to local agencies through a main point of contact at each 

site. These requests included an overview of the study, our institutional review board and state 

approvals, and data security protocols. In State 1 and State 2, each participating agency was sent a 

Microsoft Excel worksheet detailing the data requested (see Appendix G). Within these 

worksheets, the research team provided information for each selected youth (all categories in the 

53 Since we received data from State 1 first, the sample for State 1 went beyond our original sampling intentions as 

we were unsure of our ability to receive the needed data from State 2 and State 3. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

176 
 

 

 

 

 

 

 

 

 

 

 

“youth information” section of Appendix G) from the participating agency, to assist in identifying 

applicable case records for data extraction. The research team was readily available to answer 

follow-up questions from contacts at each agency. Data sharing procedures varied across agencies, 

but generally a secure File Transfer Protocol up/download process was used in which agency 

contacts placed the data in a secure, encrypted folder as they extracted information from their 

systems. Agencies completed these data requests at their convenience and finished worksheets 

were then submitted to the research team. In a few cases, physical data drives or record 

management system access credentials were provided to research staff to retrieve relevant case 

data. For State 3, all of the comprehensive assessment sample data (across all counites) was 

received on a physical data drive. Therefore, the research team did not have to reach out to each 

county in State 3 and obtain spreadsheets. The data received were then added to the existing data 

drawn from the assessment system. Table 2.2 provides an overview of record data response rates 

across sites for each state. 

Table 2.2. Site Response Rate for Comprehensive Assessment Sample by State 

State 1 State 2 State 354 

Number Sites that Provided Data 30 14 15 

Total Possible Sites 42 21 15 

Response Rate (%) 71.4 66.7 100.0 

To assist agency staff in selecting the correct information to send to the research team, 

descriptions were included for each category of data. Appendix G outlines the information 

originally requested from each agency and the variable definitions provided. Most agencies simply 

provided the data in the manner requested. However, a small sample of agencies in State 1 and 

State 2 chose to complete the request by extracting data from their record information system for 

54 Two of the counties in State 3 were only accounted for in the State Corrections agency sample, however. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

177 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

a full range of cases. Additionally, all cases for the years requested were obtained in State 3. In 

cases where agencies provided full data files (i.e., all case record data provided for requested 

years), we then extracted our requested cases from the larger pool of data. 

Agency Data Management and Cleaning. Upon receipt of data from a participating 

agency, an extensive data cleaning and management procedure was undertaken. To track data 

cleaning completion and ensure quality control, all data were first preserved in their original form. 

Changes after this point were only made to working copies. While data were typically received in 

Microsoft Excel worksheets, they were converted to a data management/analysis file (typically 

SPSS) soon after for more efficient data management, recoding, duplication checks, and 

examination of summary statistics to identify potential anomalies. Prior to this conversion, 

spreadsheets were streamlined by removing any information that would not be part of the final 

data set (e.g. agency instructions, headers, notes, extra spaces or youth information). Files with 

substantial missing data were discussed in data management meetings. The research team 

proceeded on a case-by-case basis, and continually worked to build a protocol for later data sets in 

the process. In addition to checking for completion, data were screened to ensure agency 

information was correctly classified. Corrections were then made to any data categorized 

incorrectly. This sometimes necessitated follow-ups with agencies to ensure clarity in data 

definitions. 

Once converted into SPSS format, data cleaning commenced using a syntax template as a 

guide. This syntax outlined a variety of data edits meant to standardize the content provided by the 

agency. Edits included renaming, truncating, and (re)coding agency variables. All string variables 

were recoded into numeric variables and each numeric category was logged in a state codebook to 

prevent overlapping codes. Unique numeric codes were assigned chronologically based on the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

178 
 

 

 

 

 

 

 

 

 

 

 

 

information contained in each agency file. As new data arrived, the codebook was consulted during 

the data cleaning process. Any previously-assigned numeric codes that were relevant were re-used, 

while new information was assigned a code based on the next number available in the numeric 

coding sequence. Any acronyms or short-hand information used by sites was also clarified and 

rectified as part of the cleaning process. The formatting and type (i.e., string, numeric, or date) was 

also standardized, including variable properties like labels and missing data values, for all data. 

After the cleaning process for each file was completed, data from all agencies in each state 

were merged into one large state file. State-level data were then examined and decisions regarding 

the content of the final data (e.g. coding, formatting) were made. In some cases, agencies provided 

additional case or record information that was not originally requested. During data cleaning, this 

information was preserved until the finalization process, when decisions were made regarding its 

utility to study objectives. These decisions were based on the data availability and coverage that 

appeared in the state-level files as a whole and the main research objectives. Once the final changes 

were carried out, all state-level files were merged together into one large database. That data set 

was then utilized in the analyses in the next sections of this report. 

Overview of Case Record Measures. In order to assess the impact of risk and needs 

assessment implementation across different stages of the juvenile justice system on relevant 

youths’ outcomes, a number of measures were extracted for the comprehensive assessment sample. 

Below is an overview of the core measurement areas. For a full description of variables, please see 

Appendix H for the comprehensive assessment sample codebook. 

Demographics. A number of demographics were obtained in the case record including date 

of birth, age, gender, race, ethnicity, and county of residence. The comprehensive assessment 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

179 
 

 

 

 

 

 

 

 

 

 

 

 

sample also has a state identifier variable for youths. The demographic information was obtained 

through the OYAS database maintained in each state. 

Risk and needs assessment data. The crux of the data synthesis involved linking the 

assessment data from state repositories to focal offense information, treatment, and recidivism 

outcomes of interest. As noted above, the latter often came from local agencies. The assessment 

data included an assessment identification number and the assessment date. The data also included 

an assessment status variable that indicated whether an assessment was completed, in progress, 

needed review, had not been started, or had a review completed. Only two cases had assessments 

that were noted as incomplete. Risk assessment scores were obtained for all other youths in the 

sample. Relevant assessment information represented in the data includes the tool type (i.e., 

Detention, Disposition, Diversion, Reentry, and Residential), the overall score, risk level, and 

override risk level. Scores were also provided for each of the OYAS domains for the Disposition, 

Residential, and Reentry Tools. The domains are the same across the tools, but the items that make 

up the domains differ based on the tool. The domains are: Juvenile Justice History; Family and 

Living Arrangements; Peers and Social Support Network; Education and Employment; Pro-Social 

Skills; Substance Abuse, Mental Health, and Personality; and Values, Beliefs, and Attitudes. 

Focal offense information. One objective of the present study was to request offense 

information associated with a youth’s assessment. The focal offense information included case

number, offense description, and offense level. We also received the following dates: offense date, 

arrest date, court date, and adjudication date. For the focal offense, the comprehensive assessment 

sample also outlined the adjudication status for the focal offense, adjudication type, disposition, 

and youth status. The adjudication status variable is concerned with whether there was an 

adjudication for the focal offense (yes/no). The adjudication type measure is a categorical indicator 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

180 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

that includes responses such as “admission of guilt,” “dismissed,” and “no contest.” The most

serious disposition was also reported which included being waived to adult court and whether a 

case was dismissed, terminated, or closed with the former being most serious. Disposition 

information also indicated whether a youth was placed on probation or residential treatment, house 

arrest, committed, diverted, or received other sanctions like community service. Youth status 

details whether a youth’s case was active, inactive, suspended, waived, and other relevant focal

case information. 

Treatment and other referrals information. The treatment information received is 

concerned with the focal offense. Specifically, it captured whether a youth was referred to specific 

interventions as dictated by the OYAS or if other sanctions were applied. To varying degrees 

across sites, the treatment information submitted included provider, program name, and the type 

of treatment received.55 The treatment provider information specified supplemental information 

regarding whether the provider was a rehabilitation center, substance abuse recovery center, 

mental health facility, and other behavioral healthcare providers. Treatment program names 

included but were not limited to: Thinking for a Change (T4C), cognitive behavioral therapy, drug 

education class, Aggression Replacement Therapy (ART), and Functional Family Therapy (FFT). 

Additionally, the treatment received outlines the type of treatment, such as counseling, educational 

and employment services, family services, sex offender treatment, religious services, substance 

abuse, anger management, and treatment for antisocial attitudes. Aside from treatment, we also 

requested information regarding additional sanctions or program referrals assigned to a youth. 

These include information such as electronic monitoring, restitution, informal sanctions (e.g., 

55 No treatment indicators were provided for State 3. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

181 
 

 

 

 

 

 

 

 

 

 

 

 

apology letter, essay, litter pickup), community service, mentoring, and other court ordered 

sanctions. Those indicators at times overlap with the disposition measures described above. 

Recidivism offense information. Four measures of recidivism were extracted for the youths 

in the comprehensive assessment sample including: (1) new arrests (juvenile and/or adult); (2) new 

adjudications (juvenile and/or adult); (3) return to custody (juvenile commitment or adult 

incarceration); and (4) subsequent probation, parole, or technical violation (juvenile and/or adult). 

Specifics regarding the recidivism offense were also obtained, such as the offense code, offense 

code description, degree, offense level, count, and offense type. Typically, the analyses in this 

report are based on new adjudications and/or returns to custody as those measures had the best 

coverage. 

Youth Follow-Up Sample. The youth follow-up sample is a subset of the larger 

comprehensive assessment sample. The purpose of this sample was to select a smaller proportion 

of youths to contact to obtain more detailed information through interviews regarding education, 

employment, family, pro-social skills, substance use, recidivism, and treatment services received. 

The interviews were researcher-administered over the phone. The youth follow-up sample was 

originally designed to include 300 total youth interviews. Cases were randomly selected from the 

comprehensive assessment sample. For each state, 250 cases were selected per year. The intent 

was to ensure representation of the five tool types. The youth follow-up cases followed similar 

proportions to the comprehensive assessment sample, which included: 1) Diversion – 50 cases, 2) 

Detention – 50 cases, 3) Disposition – 150, 4) Residential – 124 cases, and 5) Reentry – 126. This 

breakdown totals to 500 cases per state. 

Development of Youth Follow-up Interview. The research team developed the semistructured youth follow-up interview in order to extensively study a subset of youths. Generally 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

182 
 

 

 

 

 

 

 

 

 

speaking, the youth follow-ups allows for a more detailed understanding of the impact of 

assessment, treatment, and self-reported system involvement. The youth follow-up interview was 

drafted during team meetings and then reviewed independently by each research staff member. 

The drafted protocol was then tested during research team meetings for timing, flow, and to ensure 

that the main questions sufficiently captured needed information. As noted below, items and 

subscales were drawn from existing measurement tools where possible. 

The youth follow-up interview contained 46 questions and was divided into eight different 

content areas: (1) education/employment (15 questions); (2) family, living arrangements, and 

neighborhood (6 questions); (3) peer associations (2 questions); (4) situational awareness (4 

questions); (5) beliefs (2 questions); (6) substance use (4 questions); (7) contact with the criminal 

justice system and OYAS assessment (5 questions); and (8) treatment services (8 questions). The 

full youth follow-up interview protocol can be found in Appendix H. Responses to items varied 

and included yes/no responses, statements that corresponded to Likert scales, and open-ended 

questions. In order to maintain continuity in measurement, the interview items were developed, in 

part, based upon the OYAS self-report. It not only taps into pertinent criminogenic risks and needs, 

but also allowed us to follow up with the youths to determine reductions in risk and needs. The 

interview also provided a collateral source of information regarding the type, frequency, and 

quality of treatment and supervision. Youths were also able to provide additional insights on some 

relevant developmental outcomes that were not easily measured with agency record data. 

Youth Follow-Up Interview Administration. When contact was made with a prospective 

participant, several procedures were followed to ensure completion of the interview. For youths 

under 18, staff established contact with their guardians, who were read the study consent and 

information form to ensure consent was obtained prior to the interview. All of the consent forms 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

183 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

for this portion of the study are located in Appendix J. In the event that guardian consent was not 

needed (or after it was obtained), youths were read the study assent/consent form and given 

information about the interview using a scripted information sheet. Once the youths’ agreement to

participate was obtained, they were read a scripted brief of the study and assured that they could 

stop answering questions and discontinue at any time and did not have to answer any questions if 

they did not want wish. The interview was then administered via phone and took about 30 minutes 

on average—but this depended on each youth’s level of disclosure and the pertinence of certain 

areas of the protocol (e.g., interviewees who were not in school or employed would skip a 

proportion of the questions). Each question was read by a staff member, who then listed all 

possible answers and recorded the youth’s response. Upon completion of the interview, these

responses were entered into an SPSS database for later analysis. Youths also chose a $15 restaurant 

gift card to receive as an expression of appreciation for participating in the study. Gift cards were 

then sent to the youths along with a hand-written “thank you” note and an informational flyer. 

Youth Follow-Up Sample Data Collection. Final totals differ from the initial proportions 

due to agencies declining to participate or inability to locate contact data for particular youths. 

Ultimately, a random subsample of 1,402 of cases from the comprehensive assessment sample was 

targeted for these interviews following the disposition of their focal case or facility release. These 

cases were stratified across the three states. The number of cases vary by state depending on the 

number of agencies that provided those data. Table 2.3 provides the distribution of sites that 

provided contact data for cases. 

Table 2.3. Response Rate for Youth Contact Data by State 

State 1 State 2 State 356 

Number of Sites that Provided Contact Data 26 9 13 

56 State 3 data for youth follow-up and interviewing was available for 13 counties. Youth follow-up cases were 

randomly extracted from this total sample rather than reaching out to each county and requesting contact data. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

184 
 

 

 

 

 

 

 

 

 

 

 

 

Table 2.3. Response Rate for Youth Contact Data by State 

State 1 State 2 State 356 

Number of Sites Asked to Provide Contact Data 33 16 13 

Response Rate (%) 78.8 56.3 100.0 

For State 1 only, the self-report subsample was bolstered with random sampling with 

replacement. State 2 and State 3 only involved one data request process, and we were not able to 

request any additional data or use replacement counties. Additionally, records and contact 

information were provided by state and local agencies on a rolling basis such that the effective 

field time for potential interviewees in State 1 was significantly longer than State 2 and State 3, 

and that State 2 was longer than in State 3. Administrative data available for all cases assessed 

during the time period of interest were used to evaluate and adjust for non-response and attrition. 

Table 2.4 provides the response rates for each state. 

Table 2.4. Summary of Youth Contacts, Interviews Completed, and Response Rates 

State 1 State 2 State 357 

Number of Youth Interviewed 79 25 27 

Total Possible Youth 704 198 500 

Response Rate (%) 11.2 12.6 5.4 

Youth Follow-Up Training. Interviewers were required to undergo human subjects 

research training prior to working in the on-campus call center. Staff were required to attend 

several in-person trainings, during which they were paired with more experienced interviewers. 

These trainings involved shadowing experienced staff, reviewing calling procedures and codes of 

conduct, and learning search strategies for contacting unreachable youths. Staff were shown 

around the call center, conducted practice calls, and discussed record-keeping procedures (e.g., 

57 State 3 data for youth follow-up and interviewing became available in March 2018 and data collection was closed 

in August 2018. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

185 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

call log, gift card tracking, interview completion). After completing training, new staff entered the 

call center shift rotation. Call center staff were typically scheduled to conduct interviews between 

9am and 9pm Monday-Saturday from April 2016 to August 2018. Call schedules were created 

each month for interview administrators to optimize the distribution of the calls that were made. 

Youth follow-up call center procedures. Interviews were conducted via a University of 

Cincinnati phone line inside of a private campus office. Approximately 30-35 hours per week were 

spent making calls when the call center was active. Time not spent making calls was devoted to 

other follow-up related tasks, like searching public records and social media profiles for 

prospective participants who were selected for contact but who were unreachable. Additionally, 

public court records were searched for unreachable youths or for any youths who study staff were 

told were incarcerated (e.g. during a phone call attempt). Staff recorded all call and search attempts 

in an electronic and paper call log. 

As previously mentioned, individuals were selected for follow-up from a sample of youths 

who came in contact with participating agencies during the window of time covered by the 

comprehensive assessment sample. The study team requested contact information (i.e., phone 

numbers, addresses, and parent/guardian names) on file in each jurisdiction for selected youths. 

This information was then used in attempts to contact the youths and their parents to obtain assent 

and consent for completion of the interview. Working phone numbers were called at least once 

every 2-4 weeks for a period of at least three months. If a response was obtained during that time 

(i.e. if someone picked up the phone), staff used a script to ask the responder if the individual was 

interested in participating in the follow-up interview. If a youth was 18 or older, staff sought to 

talk with the individual in question. However, if a youth was younger than 18, staff sought to talk 

with their parent or guardian. In the event that a voicemail was reached, staff left a response using 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

186 
 

 

 

 

 

 

 

 

 

 

 

 

 

a voicemail script during every other contact attempt. The record files were used to confirm ages 

for youths relative to current dates in order to ensure that appropriate protocols were followed for 

consent/assent. 

The overview of contact attempts and associated cases that were/were not reached are 

provided in Table 2.5. If no response was obtained, the number provided was non-working, or no 

number was provided or located, youths were deemed “unreachable” and moved to a “search for 

new information” file. The research team utilized a number of resources to track and identify 

additional contact information for youths (i.e., VineLink victim notification, county court/jail 

databases, WhitePages Premium, and Facebook). The general approach taken in this process was 

modeled on previous descriptions of best practices in locating hard-to-reach youth populations 

(see, e.g., McCuller, Sussman, Holiday, Craig, & Dent, 2002; Wright, Allen, & Devine, 1995). 

Public record databases were searched for information on these individuals by cross-checking any 

prior contact information provided for the youths or their parents/guardians with novel 

information. If potential new working phone numbers were found, they were recorded, and the 

youths’ files were moved back into the call queue. This process might be carried out multiple times 

for youths that were challenging to contact (e.g., phone number was changed or disconnected). On 

some occasions, the call-center staff would locate a valid mobile phone number during the 

WhitePages Premium search. 

Study staff also searched Facebook for any profiles that could be matched to existing 

information for unreachable youths. If no Facebook profile or updated contact information could 

be found, youths were moved to “unable to contact” status and no further attempt was made to 

reach them. However, if a matching profile was found, staff sent a scripted message and a study 

flyer. This message briefed potential participants on the purpose of the study, provided the study 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

187 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

contact information, asked about their interest in the study, and encouraged each individual to 

reach out if they had questions. Youths under the age of 18 were also told that if they wanted to 

participate, they would need to have a parent or guardian available to give consent prior to 

participation. Facebook messages were attempted up to three times within a period of at least two 

weeks between attempted contacts. If youths responded, they were asked if they had any questions 

or would like to schedule a time to do the interview over the phone. If no response was received 

after at least two Facebook contact attempts, and no updated contact information was available for 

the youths at that time, their files were moved to “unable to contact” status and all attempts to 

reach them ceased. 

At the beginning of the study, the team also contacted hard-to-reach youths via text 

message using Google Voice. A total of 102 messages were sent to the youths and/or their parents 

with a valid mobile phone number. The majority of these texts were sent to youths from State 1 

(82.4%). This method was phased out as the research staff had limited success in accurately 

identifying valid mobile phone numbers and received relatively few responses from parents and/or 

youths. 

On some occasions, study staff were able to reach family members of selected youths, only 

to be informed that they were incarcerated. In the event that the youth’s case was in juvenile court, 

no public records existed for the case. Therefore, study staff asked family members for an estimate 

of when youths would be released and followed up again when the specified dates had passed. 

Furthermore, when incarcerated youths had cases in adult court, their cases were able to be tracked 

through public court records until their release. After three weeks in the community, their files 

were placed back in the call queue and attempts to make contact re-started. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

188 
 

 

 

 

 

 

 

 

 

 

 

Youth Follow-Up Sample Contact Attempts. Table 2.5 provides an overview of contact 

attempts made to youths in the follow-up sample by state. A total of 9,297 phone calls were made 

to 1,405 youths included in the youth follow-up sample. Research staff made phone contact with 

either the youths or their guardian(s) during 734 phone calls of the 5,789 call attempts in State 1 

(12.7% of call attempts resulted in contact). Each youth in State 1 was called approximately eight 

times on average. Youths in State 2 were called approximately nine times, on average. 

Approximately four voicemail messages were left for each youth in State 2. The research team 

was able to make phone contact with 107 youths and/or their guardian out of the 198 youths in 

State 2 (54.0%). The fewest calls were made to the sample of 500 youths from State 3 (N = 1,594; 

17.1% of all calls).58 The youths and/or their guardians were reached on 289 of the 1,594 call 

attempts (18.1%). The research team attempted an average of 3.2 calls per youth and left an 

average of 1.7 voicemails per youth. Phone contact was made with 209 youths and/or guardians in 

State 3 (41.8%). 

Table 2.5. Descriptive Statistics for Youth Follow-Up Sample – Youth Contact Methods 

State 1 Variable (N = 707) State 2 

(N = 198) 

State 3 

(N = 500) 

Combined 

(N = 1,405) 

n (%) n (%) n (%) n (%) 

Calls Attempted 5,879 (63.2) 1,824 (19.6) 1,594 (17.1) 9,297 (100.0) 

Voicemails (VM) Left 1,589 (49.8) 747 (23.4) 853 (26.7) 3,189 (100.0) 

Phone Contact with Youth 209 (60.4) 53 (15.3) 84 (24.3) 346 (100.0) 

Phone Contact with 587 (57.9) Guardian 187 (18.5) 239 (23.6) 1,013 (100.0) 

Any Phone Contact 734 (58.9) 224 (18.0) 289 (23.2) 1,247 (100.0) 

Any Phone Contact per Case 325 (50.7) 107 (16.7) 209 (32.6) 641 (100.0) 

Searched for Additional 817 (66.5) Contact 290 (23.6) 121 (9.9) 1228 (100.0) 

Google Voice Text 84 (82.4) 16 (15.7) 2 (1.9) 102 (100.0) 

58 As previously noted, contact information for youths from State 3 was not received until later in the study timeline. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

189 
 

 

 

 

 

 

 

 

 

 

Table 2.5. Descriptive Statistics for Youth Follow-Up Sample – Youth Contact Methods 

Variable State 1 

(N = 707) 

State 2 

(N = 198) 

State 3 

(N = 500) 

Combined 

(N = 1,405) 

Facebook Message1 546 (70.4) 177 (22.8) 53 (6.8) 776 (100.0) 

x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

Number of Calls per Case 8.3 (7.4) 9.2 (6.6) 3.2 (1.4) 6.6 (6.4) 

Number of VM per Case 2.2 (3.1) 3.8 (4.1) 1.7 (1.5) 2.3 (2.9) 

1 Includes: responses and > 2 week follow-up messages. 

Notes: n = number, x̅= mean, sd = standard deviation. 

Among youths that were challenging to locate, a total of 1,228 searches were conducted by 

the call-center staff. During the searches, the research team often located Facebook accounts for 

the youths. A total of 776 Facebook messages were sent to youths across the three states, which 

includes: initial messages, follow-up messages to youths at least 2 weeks after the initial message, 

and responses to messages sent from the youths. The use of Facebook resulted in the completion 

of 12 additional interviews (9.2% of the total number of interviews). 

Table 2.6 provides the response rates for each state. The combined response rate for 

completed interviews across the three states was 9.3% (N = 131). The response rate was the highest 

among youths in State 2 (12.6%). A relatively similar percentage of youths from State 1 completed 

the phone interview (11.2%). The response rate was the lowest for youths in State 3 (5.4%). Of 

the 1,405 youths in the follow-up sample, 204 youths declined to participate in the interview 

(14.5%). A small percentage of youths in the overall follow-up sample were identified as being in 

a detention center or secure confinement facility during the follow-up period (6.2%). Adjusting 

the response rate to account for whether any phone contact was made with the youths and/or their 

parents yields a higher response rate (20.4%) compared to the rate presented in Table 2.4. 

Table 2.6. Descriptive Statistics for Youth Follow-Up Sample – Interview Response 

Variable State 1 

(N = 707) 

n (%) 

State 2 

(N = 198) 

n (%) 

State 3 

(N = 500) 

n (%) 

Combined 

(N = 1,405) 

n (%) 

190 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 2.6. Descriptive Statistics for Youth Follow-Up Sample – Interview Response 

Variable State 1 

(N = 707) 

State 2 

(N = 198) 

State 3 

(N = 500) 

Combined 

(N = 1,405) 

Completed Interview 79 (11.2) 25 (12.6) 27 (5.4) 131 (9.3) 

Declined to Participate 107 (15.1) 28 (14.1) 69 (13.8) 204 (14.5) 

Detention/Incarceration 66 (9.3) 7 (3.5) 14 (2.8) 87 (6.2) 

Adjusted Response Rate1 (24.3) (23.4) (12.9) (20.4) 

1 Accounts for whether any phone contact was made with youths and/or their parents. 

Overview of follow-up interview measures. We draw in part from the measures from the 

CAS case record data described above in analyses with the 131 youth whom we interviewed and 

so this section identifies the key measures from the youth follow-up tool that we developed and 

utilized in the study. 

Summary scales and indices. We created a series of summary measures that reflect youths’

different attitudes and beliefs at the point at which they were interviewed. In some cases, these 

measures reflected elements of the OYAS self-report tool in order to establish some sense of 

whether youths had changed between the time of their assessments and the point at which they 

were interviewed. In other cases, however, they simply reflect relevant dimensions of development 

that might be affected by youths’ experiences in the system (which in turn may have been

influenced by the risk and needs assessment process). 

The Prosocial Family measure is an average of items coded such that higher scores suggest 

more positive family relationships and lower scores suggest less positive relationships with 

family.59 The scale comprises five items, such as “my family is important to me,” “I feel safe with 

my family,” and “my family wants me to stay out of trouble.” A single factor solution explained

55 percent of the variance among the items, which loaded from 0.51 to 0.81 in exploratory factor 

59 Unless noted otherwise, possible responses ranged from 1 (strongly disagree) to 5 (strongly agree) on the items that 

comprise these composite measures. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

191 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

analysis (EFA) with Principal Axis Factoring. The Cronbach’s alpha for the measure was 0.75. 

Both of these values exceed their respective cut-offs of 0.40 (factor loadings) and 0.70 (Cronbach’s 

alpha). Similarly, a four-item Prosocial Peer score was developed based on Likert-scale responses 

to questions such as, “my friends are good role models” and “my friends want me to stay out of 

trouble.” This measure had a Cronbach’s alpha score of 0.77 and a single factor accounted for 59

percent of the variance/covariance in these items with item loadings from 0.50 to 0.77. 

A mean scale was computed to capture youths’ Attitudes toward the Juvenile Justice 

System and the services that they may have received during the course of their involvement. This 

measure comprised four items with a Cronbach’s alpha of 0.87. A single factor explained 71 

percent of the variance/covariance of the included items, which had individual loadings from 0.60 

to 0.93 on an underlying latent variable. The items in the measure included, “I am receiving (or 

received) the kind of help I need,” “juvenile justice personnel have helped me get the services I 

need,” and “the services I have received helped me to stay out of trouble with the law.”

We also developed an average score (k = 5 items) that reflected youths’ Attitudes toward 

Work and School. This was only computed for the 99 interviewees who responded that they were 

involved in either of the two at the time of the interview. The two scales had alphas of 0.86 and 

0.77, respectively, and their loadings were generally above .40. The underlying factors in each 

explained 65 and 53 percent of item variance, respectively. Where a youth was involved in both 

work and school, we prioritized the latter score given the age range of the sample. The items 

included “my school[work] is worthwhile,” “I feel committed to school[work],” and “Even though

school[work] can be difficult, it's worth it in the long run.”

The Procriminal Attitudes measure includes nine items such as “people should be allowed 

to use illegal drugs without any legal consequences,” “there are some good things about gangs,”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

192 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

and “you have to get even with people who do not respect you.” The Cronbach’s alpha was 0.77

and a single factor explained about 36 percent of the variance in EFA. Single item loadings ranged 

from 0.37 to 0.66. 

The mean of seven items from the Texas Christian University (TCU) Adolescent Thinking 

scale was used to measure interviewees’ thinking styles and thinking errors, situational control, 

and efficacy in problem solving (Knight, Becan, Landrum, Joe, & Flynn, 2014). Example items 

included, “I am confident that I can walk away from a fight” and “I am confident that I can resist 

pressure from friends to do things that may get me in trouble.” The measure had a Cronbach’s

alpha of 0.83 and a single factor explained 55 percent of the variance in the pool of items. The 

individual factor loadings ranged from 0.40 to 0.80. 

Employment and school. We utilized two dichotomous measures (0 = “No,” 1 = “Yes”) to 

capture whether the youth was employed or attending school at the time of the interview. This 

measure was merged in analyses to reflect whether the interviewee was involved in either (or both) 

of the two. 

Justice involvement. During the interview, youths were also asked a number of questions 

about their behavior and experiences in the past year (or termination from the court or release if 

that was a shorter time period). These measures tapped into the frequency with which they had 

been “stopped by police,” “arrested,” “adjudicated delinquent or convicted,” “placed on

probation,” or “recommitted to detention or an institution.” These variables were measured as

counts initially, but collapsed into dichotomous (0 = “No”, 1 = “Yes”) measures for the purposes 

of the reported analyses. 

Substance use. The interview included several questions on youths’ recent substance use

during the follow-up period. These included “alcohol use,” “marijuana use,” and “harder drug 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

193 
 

 

 

 

 

 

 

 

 

 

 

use” (e.g., cocaine, heroin). Item responses were generally coded as frequencies, but were 

collapsed into dichotomous categories (0 = “No”, 1 = “Yes”) for these analyses. 

Self-reported treatment and programming. Several items were utilized to capture whether 

the youths received any treatment and programming in the juvenile justice process (0 = “No”, 1 =

“Yes”). As in the case record data, categories included, “substance use,” “anger management,”

“cognitive behavioral therapy (CBT),” “counseling,” “family treatment,” or “mental health 

treatment.” We also collapsed these measures to reflect whether the youth received “any” of these

treatments during juvenile justice involvement or in the follow-up period. 

Overview of analyses. We utilize varying analytic methods in this section and the next to 

answer key questions of interest under each of the study Research Objectives using these followup data and the case records described above. Those are described in more depth at each relevant 

point in the report below. Generally, we draw from basic descriptive methods to convey 

information about our samples and subsamples utilized in particular analyses. We also isolate 

important relationships with bivariate hypothesis tests (i.e., t, F, and 2) and related measures of 

association in order to highlight key points and also to set up further multivariate modeling. The 

analytic plan comprises several types of multivariate analyses, usually based on multivariate 

logistic regression given the nature of our measures. These models allow us to assess relevant 

questions related to Research Objectives 3 and 4 while conditioning on other factors that could 

plausibly impact the estimates and inferences from those analyses. We also utilize path and causal 

mediation and measurement models that allow us to understand usage and outcome processes 

wherever possible. 

Results 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

194 
 

 

 

 

 

 

 

 

 

 

 

 

Description of Comprehensive Assessment Sample. The descriptive statistics for the 

youths and case characteristics for the comprehensive assessment sample are displayed in Table 

2.7. The largest proportion of youths in the sample were from State 3 (39.3%), followed by State 

1 (32.8%), with the smallest proportion selected from State 2 (27.8%). Approximately threefourths of the sample (76%) were male, and the majority of youths in the combined sample were 

Caucasian (55.5%). African Americans made up a sizeable portion of the sample as well at 31.1 

percent, and were the dominant racial category in the State 1 subsample. The remaining youths 

were identified as other races or multiracial. On average, youths were approximately 16 years-old 

at the time they were assessed. The most common offense types in the combined sample were 

violent or sexual offenses (28%), followed by property crimes (23.9%). Drug or alcohol offenses 

were also common (15%), and were the most frequently reported offense type in the State 3 

subsample. The majority of youths were adjudicated for a felony (37%) or misdemeanor (34.4%). 

Table 2.7. Descriptive Statistics for Comprehensive Assessment Sample – Youth and Case 

Characteristics 

Variable State 1 

(N = 2,042) 

State 2 

(N = 1,730) 

State 3 

(N = 2,450) 

Combined 

(N = 6,222) 

Gender n (%) n (%) n (%) n (%) 

Male 

Female 

1,609 (78.8) 

433 (21.2) 

1,342 (77.6) 

388 (22.4) 

1,776 (72.5) 

674 (27.5) 

4,727 (76.0) 

1,495 (24.0) 

Race n (%) n (%) n (%) n (%) 

Caucasian 

African American 

Asian 

Hispanic 

Native American 

Native Hawaiian or 

Pacific Islander 

871 (42.7) 

1,087 (53.2) 

3 (0.1) 

0 (0.0) 

4 (0.2) 

2 (0.1) 

940 (54.3) 

669 (38.7) 

2 (0.1) 

0 (0.0) 

3 (0.2) 

1 (0.1) 

1,641 (67.0) 

176 (7.2) 

7 (0.3) 

211 (8.6) 

191 (7.8) 

4 (0.2) 

3,452 (55.5) 

1,932 (31.1) 

12 (0.2) 

211 (3.4) 

198 (3.2) 

7 (0.1) 

Multi-racial 

(Caucasian and 

African American) 

Multi-racial 

35 (1.7) 0 (0.0) 0 (0.0) 35 (0.6) 

(Caucasian and 

Asian) 

1 (0.0) 0 (0.0) 0 (0.0) 1 (0.0) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

195 
 

 

 

 

 

Table 2.7. Descriptive Statistics for Comprehensive Assessment Sample – Youth and Case 

Characteristics 

State 1 State 2 State 3 Combined Variable (N = 2,042) (N = 1,730) (N = 2,450) (N = 6,222) 

Multi-racial 

(Unknown) 6 (0.3) 0 (0.0) 0 (0.0) 6 (0.1) 

Other 33 (1.6) 114 (6.6) 21 (0.9) 168 (2.7) 

Missing 0 (0.0) 1 (0.1) 199 (8.1) 200 (3.2) 

Ethnicity n (%) n (%) n (%) n (%) 

Hispanic 75 121 211 407 

(3.7) (7.0) (8.6) (6.5) 

Non-Hispanic 1,901 1,544 239 3684 

(93.1) (89.2) (9.8) (59.2) 

Missing 66 65 2,000 2,131 

(3.2) (3.8) (81.6) (34.2) 

Age x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

16.06 (1.74) 15.64 (1.43) 15.77 (1.71) 15.83 (1.66) 

Offense Type n (%) n (%) n (%) n (%) 

Violent/Sex 753 (36.9) 623 (36.0) 369 (15.1) 1,745 (28.0) 

Property 497 (24.3) 617 (35.7) 371 (15.1) 1,485 (23.9) 

Weapon 61 (3.0) 71 (4.1) 50 (2.0) 182 (2.9) 

Drug/Alcohol 65 (3.2) 201 (11.6) 669 (27.3) 935 (15.0) 

VOC, PV, or 

Probation 8 (0.4) 69 (4.0) 46 (1.9) 123 (2.0) 

Other 111 (5.4) 22 (1.3) 109 (4.4) 242 (3.9) 

Status/Unruly/DC 243 (11.9) 127 (7.3) 96 (3.9) 466 (7.5) 

Missing 304 (14.9) 0 (0.0) 740 (30.2) 1,044 (16.8) 

Offense Level n (%) n (%) n (%) n (%) 

Felony 918 (45.0) 844 (48.8) 539 (22.0) 2,301 (37.0) 

Misdemeanor 630 (30.9) 790 (45.7) 718 (29.3) 2,138 (34.4) 

Traffic/Civil 6 (0.3) 0 (0.0) 37 (1.5) 43 (0.7) 

Status/Unruly 69 (3.4) 96 (5.5) 390 (15.9) 555 (8.9) 

Missing 419 (20.5) 0 (0.0) 766 (31.3) 1,185 (19.0) 

Table 2.8 provides the descriptive statistics pertaining to the dispositions the youths 

received for their offenses. In the combined sample, approximately 28 percent of youths were 

committed to a state facility upon adjudication. Electronic monitoring and intensive supervision 

were used less frequently with 8.1% and 1.7% receiving these dispositions, respectively. Probation 

was used with some regularity; a little more than a quarter of the youths (26.4%) in the combined 

sample were placed on probation. Diversion was used in 12.8 percent of cases, and 6.2 percent of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

196 
 

 

 

 

 

 

 

cases were dismissed. The frequency with which each disposition is observed in the sample may 

not fully align with the prevalence that the dispositions are used in each state. For example, Table 

2.8 below shows that Diversion Tool assessments make up a lower proportion of cases than they 

typically do in the juvenile justice system (see, e.g., Furdella & Puzzanchera, 2015). The sampling 

strategies employed in this study were intended to ensure various stages of the juvenile justice 

system were adequately represented in the final sample, and some dispositions may be relied upon 

more or less frequently as individuals penetrate deeper into the juvenile justice system. 

Table 2.8. Descriptive Statistics for Comprehensive Assessment Sample – Dispositions 

State 1 State 2 State 3 Combined Variable (N = 2,042) (N = 1,730) (N = 2,450) (N = 6,222) 

State Commitment n (%) n (%) n (%) n (%) 

No 1,478 (72.4) 953 (55.1) 1,668 (68.1) 4,099 (65.9) 

Yes 564 (27.6) 688 (39.8) 466 (19.0) 1,718 (27.6) 

Missing 0 (0.0) 89 (5.1) 316 (12.9) 405 (6.5) 

Electronic Monitoring n (%) n (%) n (%) n (%) 

No 1,891 (92.6) 1,378 (79.7) 2,000 (81.6) 5,269 (84.7) 

Yes 151 (7.4) 353 (20.3) 0 (0.0) 503 (8.1) 

Missing 0 (0.0) 0 (0.0) 450 (18.4) 450 (7.2) 

Intensive Supervision n (%) n (%) n (%) n (%) 

No 2,034 (99.6) 1,640 (94.8) 1,585 (64.7) 5,259 (84.5) 

Yes 8 (0.4) 1 (0.1) 99 (4.0) 108 (1.7) 

Missing 0 (0.0) 89 (5.1) 766 (31.3) 855 (13.7) 

Probation n (%) n (%) n (%) n (%) 

No 1,452 (71.1) 923 (53.4) 1,348 (55.0) 3,723 (59.8) 

Yes 590 (28.9) 718 (41.5) 336 (13.7) 1,644 (26.4) 

Missing 0 (0.0) 89 (5.1) 766 (31.3) 855 (13.7) 

Diversion n (%) n (%) n (%) n (%) 

No 1,864 (91.3) 1,497 (86.5) 1,211 (49.4) 4,572 (73.5) 

Yes 178 (8.7) 144 (8.3) 473 (19.3) 795 (12.8) 

Missing 0 (0.0) 89 (5.1) 766 (31.3) 855 (13.7) 

Dismissed n (%) n (%) n (%) n (%) 

No 1,976 (96.8) 1,591 (92.0) 1,416 (57.8) 4,983 (80.1) 

Yes 66 (3.2) 50 (2.9) 268 (10.9) 384 (6.2) 

Missing 0 (0.0) 89 (5.1) 766 (31.3) 855 (13.7) 

Table 2.9 provides an overview of the treatment youths received during the processing of 

their cases. The values reported in the table reflect the number of youths who received the type of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

197 
 

 

 

 

 

 

 

 

 

 

 

 

treatment listed. It is important to note that treatment information was not provided for any youths 

in State 3. Among youths in State 1 and State 2, approximately 37 percent received some form of 

treatment intervention. The most frequently reported treatment type was mental health services 

(21.6%), and about 17 percent of youths received substance abuse treatment. Between 10 and 12 

percent of youths received cognitive behavioral treatment, other behavioral treatment, or family 

services. Less than five percent of youth in the sample had a record indicating that they received 

specific services for anger/aggression, education or employment, sexual offending, or some other 

forms of treatment. 

State 1 State 2 Combined Variable (N = 2,042) (N = 1,730) (N = 3,772) 

n (%) n (%) n (%) 

Cognitive Behavioral 379 (18.6) 72 (4.2) 451 (12.0) 

Anger/Aggression 134 (6.6) 29 (1.7) 163 (4.3) 

Education/Employment 52 (2.5) 89 (5.1) 141 (3.7) 

Family Services 266 (13.0) 150 (8.7) 416 (11.0) 

Mental Health Services 421 (20.6) 396 (22.9) 817 (21.6) 

Other Behavioral Treatment 410 (20.1) 0 (0.0) 410 (10.9) 

Residential Treatment 99 (4.8) 292 (16.9) 391 (10.4) 

Substance Abuse Treatment 413 (20.2) 219 (12.7) 632 (16.7) 

Sex Offender Treatment 70 (3.4) 58 (3.4) 128 (3.4) 

Other Treatment 32 (1.6) 6 (0.3) 38 (1.0) 

Any Treatment 841 (41.2) 552 (31.9) 1,393 (36.9) 

Risk and needs assessment information for the sample is displayed in Table 2.10. 

Assessment years spanned 2013 to 2017 in State 3, and are limited to 2014 and 2015 in State 1 

and State 2.

60 As a result, the bulk of assessments were carried out in 2014 (34.1%) and 2015 

(31.2%) for the combined sample. The Disposition Tool was used to assess the largest proportion 

of youths in the sample at 35.4 percent. The Residential Tool was used in slightly less than one 

quarter of cases (23.2%), and the Diversion Tool was used in approximately one fifth of cases 

60 The year range for that state reflects the fact that the corrections and court records were drawn from different systems 

by the agencies involved and record availability did not overlap precisely. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

198 
 

 

 

 

 

 

 

 

 

 

 

(19.5%). The Reentry Tool was used to assess risk and need in about 16 percent of cases, and the 

Detention Tool was used the least (5.9%). The most common classification was moderate risk 

(43.8%), followed closely by low risk (39.5%). A smaller portion of youths were classified as high 

risk (16.2%), and less than 1 percent were classified as being at a very high risk for recidivating.61 

Although these values are based on possible overrides, instances of practitioners overriding risk 

levels based on raw scores was low overall, with just 4.1 percent of youths’ risk levels being

adjusted. 

Table 2.10. Descriptive Statistics for Comprehensive Assessment Sample – Assessment Years, 

Numbers, and Results 

State 1 State 2 State 3 Combined Variable (N = 2,042) (N = 1,730) (N = 2,450) (N = 6,222) 

Assessment Year n (%) n (%) n (%) n (%) 

2013 0 (0.0) 0 (0.0) 159 (6.5) 159 (2.6) 

2014 896 (43.9) 1,008 (58.3) 219 (8.9) 2,123 (34.1) 

2015 1,146 (56.1) 722 (41.7) 72 (2.9) 1,940 (31.2) 

2016 0 (0.0) 0 (0.0) 1,040 (42.4) 1,040 (16.7) 

2017 0 (0.0) 0 (0.0) 960 (39.2) 960 (15.4) 

Assessment Type n (%) n (%) n (%) n (%) 

Detention 179 (8.8) 181 (10.5) 10 (0.4) 370 (5.9) 

Diversion 207 (10.1) 166 (9.6) 840 (34.3) 1,213 (19.5) 

Disposition 672 (32.9) 459 (26.5) 1,073 (43.8) 2,204 (35.4) 

Residential 479 (23.5) 459 (26.5) 508 (20.7) 1,446 (23.2) 

Reentry 505 (24.7) 465 (26.9) 19 (0.8) 989 (15.9) 

Risk Level n (%) n (%) n (%) n (%) 

Low 770 (37.7) 630 (36.4) 1,057 (43.1) 2,457 (39.5) 

Moderate 824 (40.4) 863 (49.9) 1,036 (42.3) 2,723 (43.8) 

High 446 (21.8) 237 (13.7) 322 (13.1) 1,005 (16.2) 

Very High 0 (0.0) 0 (0.0) 35 (1.4) 35 (0.6) 

Risk Level Override n (%) n (%) n (%) n (%) 

Yes 108 (5.3) 25 (1.4) 119 (4.9) 252 (4.1) 

Table 2.11 displays the average domain scores for youths who were assessed with the 

Disposition, Residential, and Reentry Tools in the comprehensive assessment sample. Each tool is 

comprised of some overlapping and some unique items. Although the domains are consistent 

61 This is a byproduct of one state adding a category of cases believed to be at especially high risk of recidivism. This 

is considered an override of the overall OYAS risk levels. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

199 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

across tools, the number of items within each domain varies based on the tool used. These 

differences in the ranges of scores should be considered when reviewing the averages. Among 

those who were assessed with the Disposition Tool, scores in the peers and social support domain 

tended to be the highest (x̅= 1.93, sd = 1.57). This was also observed in each of the three states. 

Average domain scores were similar for Education and Employment (x̅= 1.68, sd = 1.07) and 

Family and Living Arrangements (x̅= 1.70, sd = 1.51). Average scores were also very similar for 

the Substance Abuse, Mental Health, and Personality (x̅= 1.33, sd = 1.23) and Pro-Social Skills 

(x̅= 1.31, sd = 1.20) domains. Scores were lowest in the Juvenile Justice History (x̅= 0.96, sd = 

1.03) and Values, Beliefs, and Attitudes (x̅= 0.70, sd = 0.96) domains. Comparing scores across 

the three states, average scores tended to be highest in State 2 and lowest in State 3 with one 

exception—scores in the Juvenile Justice History domain were highest in State 1 and lowest in 

State 3. 

A slightly different pattern was observed among average domain scores for youths assessed 

with the Residential Tool. The highest domain scores were observed for the Substance Abuse, 

Mental Health, and Personality domain (x̅ = 4.44, sd = 1.98). Scores in the Peers and Social 

Support domain were also high (x̅= 3.16, sd = 1.68) relative to the other domains. The average 

score in the Pro-Social Skills domain was 2.80 (sd = 1.08), which is somewhat high considering 

the range of possible scores includes a maximum of 4. A similar pattern was observed for the 

Juvenile Justice History domain where the average score was 2.50 (sd = 1.31) on a scale of 0 to 3. 

Scores in the Values, Beliefs, and Attitudes domain were slightly lower at 2.22 (sd = 1.38), and 

lower still in the Education and Employment domain (x̅= 1.01, sd = 0.82). The lowest domain 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

200 
 

 

 

 

 

 

 

scores among youths assessed with the Residential Tool were observed in the Family and Living 

Arrangements domain (x̅= 0.51, sd = 0.85).62 

Table 2.11. Descriptive Statistics for Comprehensive Sample – Average Risk and Needs 

Assessment Domain Scores 

Variable State 1 State 2 State 3 Combined 

Disposition Tool Domain Scores N = 672 N = 459 N = 1,073 N = 2,204 

Range x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

1.22 0.91 0.83 0.96 Juvenile Justice History 0-3 (1.11) (0.99) (0.97) (1.03) 

1.90 2.05 1.41 1.68 Education and Employment 0-4 (1.09) (1.08) (0.98) (1.07) 

Peers and Social Support 0-6 1.95 

(1.59) 

2.57 

(1.67) 

1.64 

(1.43) 

1.93 

(1.57) 

Family and Living Arrangements 0-6 1.80 

(1.53) 

2.01 

(1.49) 

1.51 

(1.49) 

1.70 

(1.51) 

Values, Beliefs, and Attitudes 0-6 0.75 

(1.00) 

1.03 

(1.10) 

0.54 

(0.84) 

0.70 

(0.96) 

Substance Abuse, Mental Health, 

and Personality 0-6 1.47 (1.31) 

1.49 

(1.21) 

1.19 

(1.17) 

1.33 

(1.23) 

1.34 1.72 1.11 1.31 Pro-Social Skills 0-3 (1.18) (1.14) (1.18) (1.20) 

Residential Tool Domain Scores N = 479 N = 459 N = 508 N = 1,446 

Range x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

2.40 2.16 2.90 2.50 Juvenile Justice History 0-3 (1.30) (1.33) (1.21) (1.31) 

1.11 1.06 0.91 1.02 Education and Employment 0-3 (0.89) (0.76) (0.79) (0.82) 

3.25 2.92 3.30 3.16 Peers and Social Support 0-7 (1.70) (1.57) (1.74) (1.68) 

0.67 0.42 0.43 0.51 Family and Living Arrangements 0-3 (0.94) (0.75) (0.83) (0.85) 

2.36 1.91 2.39 2.22 Values, Beliefs, and Attitudes 0-5 (1.48) (1.23) (1.37) (1.38) 

Substance Abuse, Mental Health, 

and Personality 0-8 4.01 (1.92) 

4.29 

(2.01) 

4.97 

(1.88) 

4.44 

(1.98) 

3.13 2.71 2.59 2.80 Pro-Social Skills 0-4 (1.10) (1.03) (1.05) (1.08) 

62 Previous analyses with a portion of these data suggest that some domains may have differential pertinence to the 

population covered by the OYAS-Residential tool (McCafferty, Newsome, & Sullivan, 2018). Specifically, the 

Education and Employment and Family and Living Arrangements items may have differential salience for youths in 

custody as compared to those at other stages of the juvenile justice process. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

201 
 

 

 

 

 

 

 

 

 

 

Table 2.11. Descriptive Statistics for Comprehensive Sample – Average Risk and Needs 

Assessment Domain Scores 

Variable State 1 State 2 State 3 Combined 

Reentry Tool Domain Scores N = 505 N = 465 N = 19 N = 989 

Range x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

Juvenile Justice History 0-7 2.96 

(1.57) 

2.95 

(1.51) 

2.00 

(1.49) 

2.94 

(1.54) 

1.19 1.27 0.84 1.22 Education and Employment 0-4 (1.03) (0.98) (1.21) (1.01) 

Peers and Social Support 0-9 3.29 

(2.24) 

3.58 

(2.08) 

1.74 

(1.70) 

3.40 

(2.17) 

Family and Living Arrangements 0-4 1.21 

(0.91) 

1.20 

(0.75) 

1.26 

(1.15) 

1.21 

(0.84) 

2.27 1.80 1.05 2.02 Values, Beliefs, and Attitudes 0-7 (1.87) (1.65) (1.31) (1.77) 

Substance Abuse, Mental Health, 0-7 2.51 2.94 2.47 2.71 

and Personality (1.73) (1.65) (1.39) (1.70) 

1.60 1.40 1.37 1.50 Pro-Social Skills 0-4 (1.48) (1.38) (1.34) (1.43) 

As noted previously, the number of youths assessed with the Reentry Tool in the sample is 

lower relative to the Disposition and Reentry Tools. Of the 989 youths assessed with the Reentry 

Tool, only 19 were from State 3. Therefore, the average scores for the combined sample are largely 

a reflection of youths in State 1 and State 2 where scores were very similar between the two states. 

Like the Disposition Tool, average scores were highest in the peers and social support domain (x̅

= 3.40, sd = 2.17). Scores in the Juvenile Justice History (x̅= 2.94, sd = 1.54) and Substance Abuse, 

Mental Health, and Personality (x̅= 2.71, sd = 1.70) were also high relative to the other domains 

in the tool. The average domain score for Values, Beliefs, and Attitudes was 2.02 (sd = 1.77), which 

was slightly higher than scores in the Pro-Social Skills domain (x̅ = 1.50, sd = 1.43). In the 

Education and Employment and Family and Living Arrangements domains, scores were very 

similar at 1.22 (sd = 1.01) and 1.21 (sd = 0.84), respectively. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

202 
 

 

 

 

 

 

 

 

 

 

Descriptive Statistics for Youth Follow-Up Interviews. Of the 1,402 cases targeted for 

intensive phone interviews following the disposition of their focal case or facility release, 131 

youths completed the initial phone interviews. The descriptive statistics for the completed youth 

interviews are provided in Table 2.12. Males made up the majority of youths who completed an 

interview (n=95; 72.5%). A similar percentage of males completed a phone interview in State 1 

and State 2 (70.9% and 72%, respectively). A slightly higher percentage of males completed the 

interview in State 3 (77.8%). The final sample was racially mixed with 55% of respondents 

reported as White. Approximately 52% of the respondents in States 1 and 2 were White. The 

majority of non-white respondents from these states were African American (43% and 44%, 

respectively). In State 3 a greater percentage of the respondents were White compared to States 1 

and 2 (66.7%). The race of approximately 15% of the respondents from State 3 is missing. A 

relatively small proportion of the youths were identified as being from a Hispanic origin in States 

1 and 2 (8.9% and 8%, respectively). State 3 did not provide those data. The overall average age 

of youths in the final sample is 15.7 years old (sd=1.9). The average age of youths ranges from 

15.6 years old (sd=1.0) in State 2 to 15.9 years old (sd=1.8) in State 3. 

Table 2.12. Descriptive Statistics for Youth Follow-Up Sample – Youth and Case 

Characteristics 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

Gender n (%) n (%) n (%) n (%) 

Male 56 (70.9) 18 (72.0) 21 (77.8) 95 (72.5) 

Female 23 (29.1) 7 (28.0) 6 (22.2) 36 (27.5) 

Race n (%) n (%) n (%) n (%) 

Caucasian 41 (51.9) 13 (52.0) 18 (66.7) 72 (55.0) 

African American 34 (43) 11 (44.0) 3 (11.1) 48 (36.6) 

Native American 1 (1.3) 0 (0.0) 2 (7.4) 3 (2.3) 

Multi-racial 

(Caucasian and 1 (1.3) 0 (0.0) 0 (0.0) 1 (0.8) 

African American) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

203 
 

 

 

 

 

 

 

Table 2.12. Descriptive Statistics for Youth Follow-Up Sample – Youth and Case 

Characteristics 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

Other/Unlisted 2 (2.5) 1 (4.0) 0 (0.0) 3 (2.3) 

Missing 0 (0.0) 0 (0.0) 4 (14.8) 4 (3.1) 

Ethnicity n (%) n (%) n (%) n (%) 

Hispanic 7 (8.9) 2 (8.0) 0 (0.0) 9 (6.9) 

Non-Hispanic 68 (86.1) 23 (92.0) 0 (0.0) 91 (69.5) 

Missing 4 (5.1) 0 (0.0) 27 (100.0) 31 (23.7) 

Age x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

15.65 (2.16) 15.56 (1.04) 15.93 (1.77) 15.69 (1.91) 

Offense Type n (%) n (%) n (%) n (%) 

Violent/Sex 23 (29.1) 13 (52.0) 5 (18.5) 41 (31.3) 

Property 13 (16.5) 5 (20.0) 7 (25.9) 25 (19.1) 

Weapon 0 (0.0) 0 (0.0) 1 (3.7) 1 (0.8) 

Drug/Alcohol 3 (3.8) 4 (16.0) 10 (37.0) 17 (13.0) 

VOC, PV, or 

Probation 0 (0.0) 2 (8.0) 0 (0.0) 2 (1.5) 

Other 4 (5.1) 0 (0.0) 0 (0.0) 4 (3.1) 

Status/Unruly/DC 17 (21.5) 1 (4.0) 1 (3.7) 19 (14.5) 

Missing 19 (24.1) 0 (0.0) 3 (11.1) 22 (16.8) 

Offense Level n (%) n (%) n (%) n (%) 

Felony 22 (27.8) 12 (48.0) 6 (22.2) 40 (30.5) 

Misdemeanor 23 (29.1) 12 (48.0) 16 (59.3) 51 (38.9) 

Status/Unruly 7 (8.9) 1 (4.0) 2 (7.4) 10 (7.6) 

Missing 27 (34.2) 0 (0.0) 3 (11.1) 30 (22.9) 

Table 2.12 also displays the focal offense characteristics for the follow-up sample, which 

differs across the three states. Fifty-two percent of the respondents in State 2 were adjudicated for 

a violent or sex offense compared to only 29% and 18.5% in States 1 and 3, respectively. The 

majority of youths who completed the interview in State 3 were adjudicated for a drug or alcohol 

offense (37%). Smaller proportions of youths were adjudicated for drug or alcohol offense in State 

1 (3.8%) and State 2 (16%). A greater percentage of youths in State 1 were adjudicated for a status 

or disorderly conduct offense (21.5%) relative to State 2 (4%) and State 3 (3.7%). Across the three 

states, the offense level of the most serious focal offense is quite varied. The distribution of felonyThis resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

204 
 

 

 

 

 

 

 

 

 

 

 

 

and misdemeanor-level offenses is evenly split in State 1 (27.8% and 29.1%, respectively) and 

State 2 (48.0% and 48%, respectively). In State 3, however, the majority of youths in the final 

sample were adjudicated for a misdemeanor (59.3%) compared to a felony (22.2%). A relatively 

small proportion of the sample was adjudicated for a status offense in each state (ranging from 

4.0% in State 2 to 8.9% in State 1). 

Table 2.13 provides an overview of the most serious dispositions for youths’ focal cases. 

A large proportion of youths from State 2 were placed in a secure state commitment facility (28%) 

or a residential treatment facility (28%). A bigger percentage of youths from States 1 and 3 were 

sanctioned to probation (26.6% and 40.7%, respectively) relative to State 2 (16%). A greater 

percentage of the youths from State 1 were diverted (20.3%) compared to States 2 (8%) and 3 

(7.4%). A relatively large percentage of youths from State 3 had their cases dismissed or 

terminated (25.9%) in comparison to youths from States 1 and 2. 

Table 2.13. Descriptive Statistics for Youth Follow-Up Sample – Most Serious Disposition 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

Most Serious Disposition n (%) n (%) n (%) n (%) 

State Commitment 10 (12.7) 7 (28.0) 0 (0.0) 17 (13.0) 

CCF/Residential 3 (3.8) 7 (28.0) 0 (0.0) 10 (7.6) 

Electronic Monitoring1 3 (3.8) 1 (4.0) 0 (0.0) 4 (3.1) 

Intensive Probation 1 (1.3) 0 (0.0) 2 (7.4) 3 (2.3) 

Probation 20 (25.3) 4 (16.0) 9 (33.3) 33 (25.2) 

Diversion 16 

(20.3) 

2 

(8.0) 

2 

(7.4) 20 (15.3) 

Penalties2 0 

(0.0) 

1 

(4.0) 

0 

(0.0) 1 (0.8) 

Dismissed/Terminated 4 (5.1) 3 (12.0) 7 (25.9) 14 (10.7) 

Missing 22 (27.8) 0 (0.0) 7 (25.9) 29 (22.1) 1 Includes House Arrest 

2 Includes: Restitution; Court Costs; License Suspension; or Community Service 

Table 2.14 shows each disposition or placement across the three states. Probation was one 

of the most common dispositions across the three states (29.8%). In State 3, however, a slightly 

higher proportion of cases were more likely to be dismissed (22.2%) than receive probation 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

205 
 

 

 

 

 

(18.5%). Although the dispositions were quite varied between states, they are somewhat expected 

given the offense-level characteristics described above. Respondents in State 2, for example, were 

more likely to be adjudicated for a violent or sex offense in comparison to the youths from State 1 

and 3. Youths from State 2 were also more likely to receive a more serious disposition compared 

to youths from States 1 and 3. 

Table 2.14. Descriptive Statistics for Youth Follow-Up Sample – Dispositions and Placement 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

State Commitment n (%) n (%) n (%) n (%) 

No 65 (82.3) 17 (68.0) 24 (88.9) 106 (80.9) 

Yes 10 (12.7) 7 (28.0) 0 (0.0) 17 (13.0) 

Missing 4 (5.1) 1 (4.0) 3 (11.1) 8 (6.1) 

Electronic Monitoring1 n (%) n (%) n (%) n (%) 

No 70 (88.6) 21 (84.0) 27 (100.0) 118 (90.1) 

Yes 5 (6.3) 4 (16.0) 0 (0.0) 9 (6.9) 

Missing 4 (5.1) 0 (0.0) 0 (0.0) 4 (3.1) 

Intensive Supervision n (%) n (%) n (%) n (%) 

No 74 (93.7) 24 (96.0) 22 (81.5) 120 (91.6) 

Yes 1 (1.3) 0 (0.0) 2 (7.4) 3 (2.3) 

Missing 4 (5.1) 1 (4.0) 3 (11.1) 8 (6.1) 

Probation n (%) n (%) n (%) n (%) 

No 53 (67.1) 12 (48.0) 19 (70.4) 84 (64.1) 

Yes 22 (27.8) 12 (48.0) 5 (18.5) 39 (29.8) 

Missing 4 (5.1) 1 (4.0) 3 (11.1) 8 (6.1) 

Diversion n (%) n (%) n (%) n (%) 

No 59 (74.7) 23 (92.0) 22 (81.5) 104 (79.4) 

Yes 16 (20.3) 1 (4.0) 2 (7.4) 19 (14.5) 

Missing 4 (5.1) 1 (4.0) 3 (11.1) 8 (6.1) 

Dismissed n (%) n (%) n (%) n (%) 

No 72 (91.1) 23 (92.0) 18 (66.7) 113 (86.3) 

Yes 3 (3.8) 1 (4.0) 6 (22.2) 10 (7.6) 

Missing 4 (5.1) 1 (4.0) 3 (11.1) 8 (6.1) 

1 Includes House Arrest 

Table 2.15 breaks down the various treatment program types received by youths in the 

follow-up sample. State 3 did not provide treatment-related data and therefore those cases are not 

included in these analyses. Approximately 37 percent of youths from State 1 received a treatment 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

206 
 

 

 

 

 

 

 

 

 

referral compared to 28 percent of youths in State 2. Of the 28 youths referred to treatment services 

in State 1, larger proportions of them received family-related services (50%), treatment for mental 

health issues (39.2%), and substance abuse treatment (46.4%). Similar to State 1, a large 

percentage of youths from State 2 were referred to family treatment (e.g., Functional Family 

Therapy) (57.1%), mental health treatment (57.1%), and substance abuse treatment (57.1%). 

Table 2.15. Descriptive Statistics for Youth Follow-Up Sample – Treatment Received 

State 1 State 2 Combined Variable (N = 75) (N = 25) (N = 100) 

Type of Treatment n (%) n (%) n (%) 

Cognitive Behavioral 8 (10.7) 1 (4.0) 9 (9.0) 

Anger/Aggression 1 (1.3) 1 (4.0) 2 (2.0) 

Education/Employment 3 (4.0) 2 (8.0) 5 (3.8) 

Family Services 14 (18.7) 4 (16.0) 18 (18.0) 

Mental Health Services 11 (14.7) 4 (16.0) 15 (15.0) 

Other Behavioral Treatment 16 (21.3) 0 (0.0) 16 (16.0) 

Residential Treatment 4 (5.3) 7 (28.0) 11 (11.0) 

Substance Abuse Treatment 13 (17.3) 4 (16.0) 17 (17.0) 

Sex Offender Treatment 1 (1.3) 2 (8.0) 3 (3.0) 

Other Treatment 1 (1.3) 0 (0.0) 1 (1.0) 

Any Treatment 28 (37.3) 7 (28.0) 35 (35.0) 

Risk and need assessment information for the youth follow-up sample is displayed in Table 

2.16. Youths from State 1 and State 2 were assessed in 2014 and 2015. State 3 cases came from 

2016 and 2017. Relatively few cases from each state were assessed with the Detention Tool (6.9%). 

A relatively large percentage of youths from State 1 and State 3 were assessed with the Diversion 

Tool (24.1% and 25.9%, respectively). Conversely, a small proportion of youths from State 2 

received the Diversion Tool (8%). A greater percentage of youths from State 3 were assessed with 

the Disposition Tool (51.9%) relative to youths from States 1 (29.1%) and 2 (28%). The majority 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

207 
 

 

 

 

 

 

 

 

 

of youths from State 2 were assessed at later phases of the juvenile justice process (52%); whereas, 

approximately 39 percent of youths from State 1 and 22.2 percent of youths from State 3 were 

assessed with either the Residential or Reentry Tool. 

In regard to risk level, youths were more likely to be classified as low or moderate risk than 

high risk. This pattern holds true when examining risk level across the three states. However, a 

higher percentage of youths from State 1 were assessed as high risk (22.8%) compared to youths 

from State 2 (13.7%) and State 3 (13.1%). In comparison to States 1 and 3, youths from State 2 

were more likely to be classified as moderate risk. Additionally, the majority of youths from State 

3 were assessed as low risk (55.6%), which is somewhat higher than youths from States 1 (39.2%) 

and 2 (44%). The table also reports the domain scores by state. Inclusion of each domain varies by 

assessment type—meaning that the coverage is somewhat varied across the states based on the 

percentage of youths assessed with each tool. The most pronounced differences in domain scores 

involves the Values, Beliefs, and Attitudes score and the Pro-Social Skills score. In both instances, 

the average scores of youths from State 3 were much lower than youths from States 1 and 2. For 

example, the average score for the Values, Beliefs, and Attitudes domain was 0.40 for youths from 

State 3 compared to 1.67 for youths from State 1 and 1.35 for youths from State 2. 

Table 2.16. Descriptive Statistics for Youth Follow-Up Sample – Assessment Year, Type, 

Risk Level, and Domain Scores 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

Assessment Year n (%) n (%) n (%) n (%) 

2014 45 (57.0) 14 (56.0) 0 (0.0) 59 (45.0) 

2015 34 (43.0) 11 (44.0) 0 (0.0) 45 (34.4) 

2016 0 (0.0) 0 (0.0) 17 (63.0) 17 (13.0) 

2017 0 (0.0) 0 (0.0) 10 (37.0) 10 (7.6) 

Assessment Type n (%) n (%) n (%) n (%) 

Detention 6 (7.6) 3 (12.0) 0 (0.0) 9 (6.9) 

Diversion 19 (24.1) 2 (8.0) 7 (25.9) 28 (21.4) 

Disposition 23 (29.1) 7 (28.0) 14 (51.9) 44 (33.6) 

Residential 16 (20.3) 5 (20.0) 3 (11.1) 24 (18.3) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

208 
 

 

 

 

 

 

Table 2.16. Descriptive Statistics for Youth Follow-Up Sample – Assessment Year, Type, 

Risk Level, and Domain Scores 

State 1 State 2 State 3 Combined Variable (N = 79) (N = 25) (N = 27) (N = 131) 

Reentry 15 (19.0) 8 (32.0) 3 (11.1) 26 (19.8) 

Risk Level n (%) n (%) n (%) n (%) 

Low 31 (39.2) 11 (44.0) 15 (55.6) 57 (43.5) 

Moderate 29 (36.7) 11 (44.0) 8 (29.6) 48 (36.6) 

High 18 (22.8) 3 (13.7) 4 (13.1) 25 (19.1) 

Missing 1 (1.3) 0 (0.0) 0 (0.0) 1 (0.8) 

Risk Level Override n (%) n (%) n (%) n (%) 

Yes 2 (2.5) 0 (0.0) 3 (11.1) 5 (3.8) 

OYAS Domain Scores x̅(sd) x̅(sd) x̅(sd) x̅(sd) 

Juvenile Justice History 2.02 (1.33) 1.91 (1.93) 1.40 (1.35) 1.86(1.50) 

Education and Employment 1.46 (1.05) 1.25 (1.12) 1.05 (0.89) 1.32 (1.03) 

Peers and Social Support 2.63 (2.24) 3.05 (1.64) 1.85 (1.73) 2.55 (2.03) 

Family and Living 

Arrangements 1.16 (1.26) 1.05 (1.25) 1.05 (1.15) 1.11 (1.22) 

Values, Beliefs, and Attitudes 1.67 (1.58) 1.35 (1.46) 0.40 (0.75) 1.31 (1.48) 

Substance Abuse, Mental 

Health, and Personality 2.65 (2.13) 3.00 (2.05) 2.00 (1.69) 2.58 (2.03) 

Pro-Social Skills 1.90 (1.51) 2.20 (1.47) 1.20 (1.28) 1.81 (1.48) 

Alignment of Youth Follow-up and Comprehensive Assessment Samples. The 

representativeness of the youth follow-up sample is by nature tied into the degree to which it 

corresponds with key properties of the comprehensive assessment sample, which was randomly 

drawn from state record keeping systems. Very clearly, the state representation differed across the 

two samples in part due to the timing of data retrieval and number of potential cases provided for 

each of the respective states. Whereas State 1 cases make up approximately one-third of the 

comprehensive assessment sample, they make up 60 percent of the youth follow-up group. This 

is an important difference, but we also assessed whether other important measures described here 

differed significantly between the 131 youths in the follow-up sample and those that were not. 

Those analyses of pre-interview measures are summarized in Table 2.17. These 

comparisons included basic sociodemographics, OYAS risk score and level, and key case 

disposition and treatment indicators. With one exception, there were no significant differences on 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

209 
 

 

 

 

 

 

 

 

 

 

 

 

 

bivariate comparisons between the small sample of youths who participated in the youth followup interview and the cases in the larger comprehensive assessment sample that were not followed 

up. The youth follow-up sample comprised proportionally less than half as many youths with a 

state commitment disposition than the larger comprehensive assessment sample. A Cramer’s V 

statistic indicated that this relationship was relatively small—0.05 on a scale from 0 to 1.0—but 

that nevertheless should contextualize the results from the youth follow-up interview analyses. 

This likely stems in part from the fact that we did not obtain community contact information for 

youths in the State 3 corrections sample as well as the difficulty of interviewing those in custody 

in the other two states.63 Overall, the two groups look fairly similar across these different measures 

and therefore—with appropriate qualification for the state commitment difference—we analyze 

the youth follow-up sample as if it is a reasonably representative, but small, subgroup of the 

comprehensive assessment sample. 

Table 2.17. Comparison of Pre-Interview Variables for Youth Follow-Up and Comprehensive 

Assessment Samples 

Variable 

Youth 

Follow-Up 

(N = 131) 

Comprehensive 

Assessment Sample 

(N = 6,094) 

Bivariate 

Comparison 

Gender % % χ2 

(df) 

Male 73.4 76.0 χ2 

(1) = 0.46 

Race/Ethnicity % % χ2 

(df) 

Caucasian 56.5 57.3 χ2 

(9) = 6.80 

African American 37.9 32.0 

Hispanic 

Native American 

0.0 

2.4 

3.6 

3.3 

Multi-racial (Caucasian, 

African American) 

Other/Unlisted 

0.8 

2.4 

0.6 

2.9 

63 Hyland (2018) found that cases with such placements make up about 26 percent of those formally petitioned in the 

juvenile court. That prevalence drops to roughly nine percent when factoring in non-petitioned cases that have informal 

probation supervision or other sanctions. So, while this is a concern in its reflection of the comprehensive assessment 

sample, for the purposes of broader generalization this is not the largest proportion of cases processed in the juvenile 

justice system in terms of disposition and placement. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

210 
 

 

 

Table 2.17. Comparison of Pre-Interview Variables for Youth Follow-Up and Comprehensive 

Assessment Samples 

Variable 

Youth 

Follow-Up 

(N = 131) 

Comprehensive 

Assessment Sample 

(N = 6,094) 

Bivariate 

Comparison 

Age x̅

(sd) 

15.7 (1.93) 

x̅

(sd) 

15.8 (1.65) 

t(df) 

t(131) = 0.80 

Offense Type % % χ2 

(df) 

Violent/Sex 37.3 33.6 χ2 

(1) = 0.64 

Offense Level % % χ2 

(df) 

Felony 40.2 45.8 χ2 

(1) = 1.26 

OYAS Score x̅

(sd) 

9.6 (7.5) 

x̅

(sd) 

10.4 (7.3) 

t(df) 

t(6,218) = 1.15 

OYAS Risk Level % % χ2 

(df) 

Low 43.3 39.4 χ2 

(2) = 2.51 

Moderate 37.0 43.9 

High 19.7 16.7 

Case Disposition and 

Treatment 

State Commitment 

Probation 

Diversion 

Dismissal 

Any Treatment 

% 

13.7 

31.5 

15.3 

8.1 

34.7 

% 

29.9 

30.6 

14.8 

7.1 

37.0 

χ2 

(df) 

χ2 

(1) = 15.25*** 

χ2 

(1) = 0.04 

χ2 

(1) = 0.03 

χ2 

(1) = 0.16 

χ2 

(1) = 0.16 

Notes: *p<.05, **p<.01, ***p<.001 

Given the breadth of the youths’ case record data, the research team pursued several 

substudies to help shed light on important questions surrounding JRNA usage. Five different usage 

studies draw on these data and their results are presented below. These studies allow us to explore 

the contexts in which the OYAS is implemented and consider the impact that JRNA has on justice 

involved-youths and their cases. We first present these studies before transitioning to broader 

analysis of case-level processes and outcomes. 

Usage Study 2: Youth Appraisals by Race and Geography 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

211 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

This analysis considers potential variation in use of risk assessment that can affect how 

different groups of youths experience it and what it means for their case. A large body of research 

has examined racial inequities in the context of justice system contact and juvenile court 

processing. National-level estimates reveal that minority youths are vastly overrepresented at most 

phases of the juvenile justice system ranging from arrests through secure placements (Puzzanchera, 

2018; Davis & Sorensen, 2013). While there have been noticeable reduction in the number of 

juvenile arrests and placements nationally in recent decades, the disparity between Black and 

White ratios of those contacts are still pervasive in studies examining race and juvenile arrests and 

court processing (see, e.g., Pope & Leiber, 2005; Sullivan et al., 2016). Case factors, such as 

offense severity, have been identified as the strongest predictors of juvenile court outcomes and as 

important predictors in studies examining recidivism (Pope & Leiber, 2005; Cottle, Lee, & 

Heilbrun, 2001). Although these systematic reviews have pointed to the importance of offenselevel characteristics, race generally holds as a significant predictor of case processing decisions. 

Few studies, however, have attempted to examine whether race effects on juvenile court outcomes 

vary by geographic location or organizational features of juvenile courts (Bray, Sample, & KempfLeonard, 2005; Bishop, Leiber, & Johnson, 2010). 

JRNA has been identified as one possible avenue to reducing observed disparities. 

Schwalbe and colleagues (2006: 306) note that reducing race/ethnicity disparities are among the 

reasons for adopting these tools and that “in practice, the result should be an increase in the 

reliability or consistency of case decisions made by juvenile justice officials.” Research indicates

that assessment tools have varying degrees of success predicting recidivism among different 

populations and types of offenders, and across geographic regions (McCafferty, 2013). Research 

also demonstrates that a standardized risk instrument score partially mediates the relationship 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

212 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

between race/ethnicity and recidivism (Schwalbe et al., 2006). Ideally, the risk score would 

account for a large proportion of the variance in recidivism; however, the researchers found that 

race/ethnicity remained a statistically significant predictor of recidivism. Therefore, risk and need 

assessments have the potential to introduce systematic biases if assessment results and its 

relationship with recidivism varies by race subgroups. 

Risk assessment tools have been shown to have varying degrees of success in predicting 

recidivism in a number of different contexts (e.g., geographic regions and different populations of 

offenders). Research has suggested that it is important to study the performance of juvenile risk 

and need assessments across geographic locations (Johnson, Wagner, & Matthews, 2002; 

McCafferty, 2013; Vose, Cullen, & Smith, 2008). For example, Johnson et al. (2002) examined 

recidivism outcomes for a sample of 2,911 youths assessed with the Missouri Juvenile Risk 

Assessment. The researchers included an analysis to examine whether there were differences in 

risk-level classification among rural and urban youths. Johnson et al. (2002) found that a greater 

percentage of rural youths were classified as low risk compared to urban youth. Conversely, they 

found that urban youths were more likely to be assessed as high risk compared to rural youths. 

Although there is limited research examining court-level differences in applying risk and 

need assessments for decision making, there are several hypotheses as to why risk assessment tools 

can vary across settings. Researchers have raised concerns about the predictive validity of risk 

assessments on the basis that they may be specific to the data/sample used to create it. Therefore, 

the risk factors for recidivism may only be associated with the original sample. Even with the 

inclusion of risk factors identified as being the strongest predictors of recidivism (e.g., Cottle et 

al., 2001), there is the concern that differences in base rates of recidivism within the sample that 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

213 
 

 

 

 

 

 

 

 

 

 

 

 

was used to develop the instrument will heavily influence the predictive validity of the tool 

(Gottfredson & Moriarty, 2006). 

Method 

This substudy has two primary objectives. First, we assess the potential variation in the 

distribution of youths’ risk levels across juvenile court jurisdictions. Second, we investigate 

whether race accounts for any of the variation in risk-level classification between juvenile court 

jurisdictions. The data come from juvenile court case records obtained from three states. Cases 

were randomly selected from the assessment database sample frame. The final sample consisted 

of 4,663 juveniles across 55 juvenile court jurisdictions. There are two key dependent variables in 

this sub-study. First, we examine final risk level as a dependent variable (i.e., low-, moderate-, and 

high risk). We dummy coded the risk level categories and set low risk as the reference group. 

Additionally, the risk level variables take into account any overrides. Our main predictor variable 

is a dichotomous measure of race (1=Non-White). In the final model we account for the youths’

most serious offense type for their focal case. This is captured in a series of dummy coded variables 

(violent/sex; property; drug/alcohol; other [e.g., weapon, parole violations]; and status). We also 

control for gender (1 = female) and age at time of interview. 

Analytic Plan 

In order to answer the research questions, we used multinomial logistic regression analysis 

both in individual- and multi-level models (Raudenbush & Bryk, 2002). The analysis comprised 

three main stages for each outcome. First, we estimate an unconditional multilevel model. The 

purpose in this first step is to identify whether there is any variance in our outcome at the courtlevel (i.e., between court differences). Second, we assess whether our outcomes differ across race 

groups both in individual- and multi-level models. In the multi-level models, we assessed whether 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

214 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

the predictors varied randomly across counties. For example, the slope of race on our outcome was 

not statistically significant; therefore, we fixed the effects. This indicates that the slope of race on 

risk-level remains constant across counties. Additionally, we grand mean centered our predictors. 

The intercept now represents the mean for risk level adjusted for between-county differences in 

the distributions of the predictors. Lastly, we examine whether race accounts for any variation in 

our dependent variable across juvenile court jurisdictions and whether the strength of that effect 

differs when we account for additional control variables (i.e., gender, age, and focal offense type). 

We explored random effects for these variables – some of which had statistically significant slopes. 

Therefore, we used a mixed model. 

Results 

The analyses include a total of 4,663 youths from 55 counties across 3 states. State 1 has 

the most counties in the sample (n = 29). Both States 2 and 3 have the same number of counties 

represented in the sample (n = 13 each). On average, there are approximately 85 youths from each 

county. A similar percentage of youths were classified as either low or moderate risk (43% and 

44%, respectively). A relatively small percentage of youths were assessed as high risk. The 

majority of youths in the sample are White (63%). 

As discussed in the analytic plan, the first step to the analysis is to estimate the 

unconditional model. This allows us to examine the average likelihood of risk-level classification 

across juvenile court jurisdictions. The statistically significant variance component estimates 

indicate that there is county-level variation in the odds of being classified high risk relative to low 

risk and moderate risk to low risk. There is notably less between court variation in being assessed 

as moderate risk to low risk than there is in high risk to low risk. The average odds of being 

assessed as high risk relative to low risk is 0.32 and is statistically significant (Table 2.18). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

215 
 

 

 

 

 

 

 

 

 

 

Table 2.18. Unconditional Multilevel Models: Final Risk Level by County 

Coefficient Standard Error Exp(B) 

Mean log odds (High Risk) -1.15* 0.13 0.32 

Mean log odds (Mod. Risk) -0.02 0.08 0.98 

Variance Component X2(df) 

Mean log odds (High Risk) 0.58* 275.63(54) 

Mean log odds (Mod. Risk) 0.23* 239.53(54) 

* p <0.05 

Next, we assess whether risk-level varies by the race of youths at the individual level. The 

initial findings indicate that Non-White youths are significantly more likely to be assessed as high 

risk relative to low risk and moderate risk relative to low risk. Furthermore, the direction and 

magnitude of the relationship indicates that Non-White youths are much more likely to be 

classified at a higher risk-level category than their White counterparts. For example, the odds of 

being assessed as high risk compared to low risk is 2.12 times higher for Non-White youths 

compared to Whites. We also estimate whether risk-level varies by race accounting in a fixed 

effects multi-level model (Table 2.19 below). When comparing the variance component in this 

model to the unconditional model, there is a noticeable decrease in the estimates. This indicates 

that race accounts for a portion of the unexplained variation in risk-level classification between 

counties. The race coefficients are also statistically significant and in the positive direction. This 

indicates that Non-White youths are more likely to be classified as a higher risk level classification 

than White youths. For example, the odds of being assessed as high risk relative to low risk is 1.5 

times higher for Non-White youths compared to White youths. 

Table 2.19. Multilevel Model: Risk Level by Race 

Coefficient Standard Error Exp(B) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

216 
 

 

 

 

 

 

 

 

 

Mean log odds (High Risk) 

Race 

-1.11* 

0.40* 

0.13 

0.10 

0.33 

1.50 

Mean log odds (Mod. Risk) 

Race 

0.01 

0.33* 

0.08 

0.07 

1.02 

1.39 

Variance 

Component X2(df) 

Mean log odds (High Risk) 0.55* 235.98(54) 

Mean log odds (Mod. Risk) 0.19* 198.21(54) 

* p <0.05 

In the final step of the analysis, we introduce demographic controls and offense type 

category into the model (Table 2.20 below). The variance component estimates for the intercept 

indicate that there is a fair amount of unexplained variation left in risk-level classification between 

counties. Race remains a statistically significant predictor of risk-level classification (i.e., high to 

low; and moderate to low) when accounting for focal offense type and the demographic controls. 

The odds of being classified as high risk relative to low risk is 1.39 times higher for Non-White 

youths compared to White youths. Gender and age are also statistically significant. Females and 

youths that were older at the time of assessment were less likely to receive a higher risk level. In 

the main effects model that compares high risk relative to low risk, status offense is also 

statistically significant and in the negative direction. This indicates that the odds of being assessed 

as high risk relative to low risk is 0.55 for youths that were adjudicated for a status offense 

compared to those that were adjudicated for a violent or sex offense. 

Table 2.20 Multilevel Model: Main Effects of High Relative to Low Risk with Race 

Coefficient Standard Error Exp(B) 

Mean log odds (High Risk) -1.22* 0.15 0.30 

Race 0.33* 0.11 1.39 

Gender -0.58* 0.18 0.56 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

217 
 

 

 

 

 

 

 

 

 

 

 

 

Table 2.20 Multilevel Model: Main Effects of High Relative to Low Risk with Race 

Coefficient Standard Error Exp(B) 

Age -0.12* 0.04 0.88 

Offense Type1 

Property 0.14 0.18 1.15 

Drug -0.32 0.24 0.73 

Other -0.11 0.22 0.89 

Status -0.60* 0.18 0.55 

Variance Component X2(df) 

Mean log odds 0.78* 118.54(39) 

* p <0.05; 1 Reference category is violent/sex offense 

Summary of Youth Appraisals by Race and Geography. Several important findings 

emerged from this risk assessment substudy. First, risk assessment classifications varied 

significantly among sites in our sample of juvenile court jurisdictions. This suggests that the 

distribution of youths that fall into the risk classification categories differ across the courts. For 

example, one county may have a high proportion of high risk and moderate risk youths whereas 

another county might have a greater proportion of low risk youths. This finding runs somewhat 

counter to our expectations based on the design and structure of actuarial risk and needs 

instruments in that similar distributions across sites would generally be expected. Non-White 

youths were significantly more likely to receive a higher risk classification than their White 

counterparts. Although race helps to explain the variation in the distribution of risk-level across 

courts, we found no evidence that these effects are stronger or weaker between the different court 

jurisdictions. However, the effect of race on risk classification remained statistically significant 

when we accounted for the youths’ focal offense type and demographic characteristics. 

Taken together, the neutrality of risk and needs assessment tools needs further examination. 

It might be that items included in this instrument are correlated with race. For example, the Juvenile 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

218 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Justice History domain, which potentially captures differential offending, could be accounting for 

some differences in risk classification. Another potential explanation for the race differences, is 

that juvenile justice personnel that administer the tool may carry implicit biases that could impact 

the overall risk score/level. These findings have implications for case management, court 

processing, treatment-related decisions, and potential disparities in each as the risk assessment 

information may be impacting those decisions. Risk and needs assessment are valuable tools that 

help to guide a number of decision making points in the juvenile justice system, but researchers 

and practitioners should also carefully consider the degree to which practices may affect 

differential juvenile justice experiences among youths of different race and ethnic groups. While 

there have been a number of studies that examine the predictive validity of risk and need 

assessment instruments—even some across race—relatively few have examined their usage in the 

context of potential court-level differences in the juvenile justice system and also in terms of the 

context of disproportionate minority contact (DMC) trends that have been a subject of much 

discussion in recent decades (Pope & Leiber, 2005). In that way, it is an important lens into how 

JRNA usage may be tied into or help to ameliorate broader concerns in juvenile justice. 

Usage Study 3: Race, Ethnicity, and Validity of the Residential OYAS Tool 

The previous usage study shows that it is possible that the court location or place matters 

in the results of risk and needs assessment. Additionally, it is possible that elements of an 

assessment tool work differently for certain subgroups. The efficacy of evidence-based practice 

across place and population subgroups raise important questions for transporting knowledge from 

one setting to another (Welsh, Sullivan, & Olds, 2010), and implementation research must assess 

questions of transportability in order to balance the relationship between adherence and adaptation 

(Blakely et al., 1987). As the adoption of actuarial risk and need assessments has increasingly 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

219 
 

 

 

 

 

 

 

 

 

 

 

 

 

become standard practice across agencies in the juvenile justice system, there has also been a 

growing recognition of the importance of validating the tools for the populations on which they 

are used. As described above, assessing the predictive validity of juvenile risk and need 

assessments has been the subject of extensive research and the overall conclusion seems to be that 

modern risk and need assessments are capable of differentiating between offenders who will reoffend and those who will not. Concerns remain, however, regarding possible biases among 

offender sub-groups (Angwin, Larson, Mattu, & Kirchner, 2016; Dressel & Farid, 2018), and 

results from prior research examining the general performance of juvenile risk and need 

assessments among racial and ethnic subgroups are mixed. 

Several studies have found that the predictive validity of risk assessment scores do not 

differ across racial groups (Baglivio & Jackowski, 2013; Barnes et al., 2016; McCafferty, 2016; 

Perrault, Vincent, & Guy, 2017). For example, McCafferty (2016) found that effect sizes (e.g., 

correlations and area under the curve [AUC] estimates) of the OYAS were similar across a 

subsample of Caucasian youths (rpb = .31; AUC = .68) and African American youths (rpb = .33; 

AUC = .69) when predicting general recidivism. However, other studies have found differences 

in predictive validity across racial groups (Rembert, Henderson, & Pirtle, 2014; Schwalbe et al., 

2006; Vincent, Chapman, & Cook, 2011). For example, Rembert and colleagues (2014) examined 

the predictive validity of the Los Angeles County Needs Assessment Instrument (LAC) on 480 

African American and Hispanic youths. Scores from the tool were significant predictors of 

recidivism for Hispanic youths, but the same measures were not significant predictors of 

recidivism for African American youths. 

Research examining racial and ethnic differences in predictive validity across individual 

risk items or domains has also produced mixed findings. Baglivio and Jackowski (2013) noted 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

220 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

several differences across racial and gender subgroups in their analyses. For instance, alcohol and 

drug measures were significantly weaker predictors of recidivism for females than for males. 

Similarly, attitudes and behaviors were significantly weaker for females than for Caucasian and 

Hispanic males. Next, family history risk factors were significantly stronger predictors of 

recidivism for Hispanic females than for the rest of the sample. This finding is mirrored by another 

study conducted by Onifade and colleagues (2009), who also noted differences in prediction by 

domains across race and gender subgroups. In their study, they found that offense history was only 

significant for Caucasian males and not for Caucasian females nor males and females of AfricanAmerican descent. Risk factors related to peer groups were significant for both groups of males 

but neither group of females. If there are differences between key subgroups that influence risk for 

recidivism and these are not accounted for in risk and need assessment tools designed for general 

populations, the measurement of risk may be compromised. In turn, this can limit the predictive 

validity of the tools. Such questions of variance/invariance in measurement and tool performance 

across populations are essential in understanding effective usage in practice. 

Analysis of subgroup invariance in OYAS. Existing literature is currently limited in at 

least two respects. First, the measurement properties of tools are not routinely examined after their 

initial development. As a result, the insights gleaned from prior research rest on the assumption 

that the measurement of risk with a given tool holds across populations. This assumption applies 

both in general populations defined by geographic regions, as well as across subgroups within a 

given population (e.g., Caucasian and Hispanic youths). Second, research on the predictive validity 

of risk and need assessment tools also tends to focus on general populations, and less often 

examines potential differences across subgroups within them. Although some have investigated 

potential differences in predictive validity across subgroups based on characteristics such as 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

221 
 

 

 

 

 

 

 

 

 

 

 

 

 

gender or race, the available research is limited. The current study builds upon earlier research to 

conduct a preliminary analysis of whether there are differences in both the measurement properties 

and the predictive validity of the OYAS Residential Tool when used to assess youths of differing 

racial/ethnic backgrounds in State 3. 

Method 

Sample. Between 2015 and 2017, the state corrections agency in State 3 provided official 

data for 984 youth who had previously been released from state custody.64 The data included key 

demographic information, risk assessment scores, and recidivism information. Recidivism was 

tracked at 12-months for a subsample of 688 youths, which forms the main analytic sample for 

this substudy. 

Measures. Three sociodemographic variables were provided that are used to contextualize 

key findings, as well as consider potential differences in the capacity of the assessment to 

adequately capture risk and need and predict recidivism across subgroups. Age reflects the age of 

the youths at intake (x̅ = 16.05), gender was coded as male (89%) or female (11%), and 

race/ethnicity included Caucasian (29%), Hispanic (47%), African American (15%), and other 

racial groups (9%). Risk and need were assessed in the sample with the OYAS Residential Tool 

and scores for individual items, domains, and overall totals were examined. Consistent with the 

scoring guide, total scores were also collapsed into low risk (14%), moderate risk (42%), and high 

risk (44%) categories. Recidivism at each wave was measured dichotomously and is defined as 

any return to custody or any new commitment. Recidivism rates were 38% at the one-year mark. 

Analytic Plan. Confirmatory factor analysis (CFA) in Mplus was used in order to 

determine whether the presumed measurement of risk for recidivism based on the tool is observed 

64 The data for this usage study come from corrections agencies in State 3 and were obtained separately from court 

records in that state. Consequently, there are some differences in data and measure coverage across the two. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

222 
 

 

 

 

 

 

 

 

 

 

 

 

 

in the full sample of youth (Muthén & Muthén, 1998-2018). A second-order CFA permits the 

specification of a model that aligns with the structure of the tool in that individual items are loaded 

onto domains that are then loaded onto an overall measure of risk. After assessing the measurement 

properties of the tool, analyses were conducted for the full sample and racial subgroups to compare 

the extent to which the tool improved prediction among different groups. Predictive validity was 

examined through various statistical approaches including calculating correlations between risk 

scores and recidivism and conducting receiver operating characteristic (ROC) analyses. Finally, 

where significant differences in predictive validity were observed between racial subgroups, the 

measurement properties of the tool were examined separately for each group to investigate the 

extent to which the measurement model differed across subgroups. 

Results 

Measurement analysis. The results of the second order CFA for the full sample are shown 

in Table 2.21. The model that was fit to the data specified seven factors (i.e., each of the risk 

domains) to be extracted, which were loaded onto a measure of risk (i.e., the total risk score). A 

model is considered to provide an acceptable fit to the data when fit statistics produce a nonsignificant Chi-Square value, the comparative fit index (CFI) and the Tucker-Lewis index (TLI) 

are greater than .90, the root mean square estimation of approximation (RMSEA) is less than .05, 

and the weighted root mean square residual (WRMR) is less than 1 (Hu & Benter, 1999; Muthén 

& Muthén, 1998-2018). These fit statistics did not reach these thresholds in this sample. 

Specifically, the CFI and TLI both fell short of the .90 threshold at .87 and .86, respectively. 

Additionally, the RMSEA was .05 and the WRMR was 1.69. 

Table 2.21. Confirmatory Factor Analysis and Model Fit for OYAS Measurement Structure 

Chi-Square (χ2, df) 1421.39 (488)*** 

Comparative Fit Index (CFI) .87 

223 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.
 

 

 

 

 

 

 

 

 

 

 

 

Tucker-Lewis Index (TLI) .86 

Root Mean Square Error of Approximation (RMSEA) .05 

Weighted Root Mean Square Residual (WRMR) 1.69 

***p < .001 

Inspection of factor loadings is useful in determining whether particular items load onto 

the domains as expected, and whether each domain loads onto the overall measure of risk. 

Standardized loadings range from 0 to 1, and values that are low (< .30) or notably different from 

the pattern of loadings for a given measure can be used to identify potential threats to the validity 

of the measures. Of the 33 items included in the risk and need assessment, eight had standardized 

loadings that were lower than .30 including two of the three items in the Education and 

Employment domain. Beyond those weak item loadings, the Family and Living Arrangements 

domain score did not load onto the overall risk measure as evidenced by a standardized factor 

loading of .00. Together with the model fit statistics, these findings suggest room for improving 

the measurement of risk in this general population sample. 

Subgroup comparisons and predictive validity. Average total and domain scores are 

shown in Table 2.22 by racial group. The table also displays the results of tests that were employed 

to determine whether differences in scores across groups were statistically significant (i.e., analysis 

of variance). As the Caucasian and Hispanic subgroups are the largest and of primary interest for 

this study, key observations pertaining to these groups are highlighted here. Specifically, total risk 

scores were significantly higher (F = 11.41, p < .001) among Hispanic youths at 18.28 compared 

to Caucasians whose scores were 16.63 on average. Similarly, in the Peers and Social Support 

domain, Hispanic youths had significantly (F = 20.84, p < .001) higher average scores (x̅= 3.70) 

relative to Caucasian youths (x̅ = 2.76). Though average scores were somewhat low in the 

Education and Employment domain across all groups, the observed differences were significant 

(F = 5.66, p < .01) for Caucasian and Hispanic youths whose average scores were 0.80 and 1.07, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

224 
 

 

 

 

 

 

 

respectively. These two groups also differed in average scores on the Values, Beliefs, and Attitudes 

domain (F = 15.38, p < .001) where Hispanics had an average score of 2.67 compared to the 

average of 2.00 among Caucasian youths. 

Table 2.22. Comparison of Average OYAS Score and Domain Scores by Race/Ethnicity 

AfricanVariable American Caucasian Hispanic Other Key 

N = 199 N = 316 N = 69 Differences N = 96 

x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

x̅

(sd) F(DF) 

Overall Risk Score 19.51 

(4.5) 

16.63 

(5.1) 

18.28 

(4.9) 

15.63 

(6.1) 

11.41(3, 614)*** 

C, O< H, AA^ 

Juvenile Justice 2.94 3.07 2.98 2.21 10.51 (3, 677)*** 

History (1.0) (1.1) (1.1) (1.5) O<AA, H, C 

Family and Living 

Arrangements 

0.40 

(0.76) 0.44 (0.83) 0.43 (0.82) 0.31 (0.77) 0.46(3, 676) 

Peers and Social 4.10 2.76 3.70 2.97 20.84(3, 677)*** 

Support Network (1.4) (1.7) (1.7) (1.8) C, O <H, AA 

Education and 

Employment 

1.03 

(0.84) 0.80 (0.80) 1.07 (0.77) 0.84 (0.75) 

5.66(3, 677)** 

C< H 

Pro-Social Skills 2.84 2.43 2.68 2.61 4.15(3, 676)** 

(1.1) (1.0) (1.0) (0.94) C< AA 

Substance Use, Mental 

Health, and 

Personality 

5.43 

(1.5) 

4.79 

(2.1) 4.94 (1.7) 4.61 (2.1) 

3.30(3, 676)* 

C, O < AA 

Values, Beliefs, and 2.91 2.00 2.67 2.20 15.38(3, 676)*** 

Attitudes (1.4) (1.3) (1.3) (1.2) C< H, AA 

* p < .05 , ** p < .01, *** p < .001 

^AA = African American, C = Caucasian, H = Hispanic, O = Other 

As an initial step to assessing the predictive validity in this population, point-biserial 

correlations (rpb) were calculated for the overall sample, as well as the Caucasian and Hispanic 

subsamples for the overall risk score and each domain score and recidivism at 12-months. As 

shown in Table 2.23, a significant association was observed between the overall risk score and 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

225 
 

 

 

 

 

 

 

 

 

 

 

 

recidivism in the full sample (rpb = .26, p < .001), as well as for the Caucasian (rpb = .22, p < .01) 

and Hispanic (rpb = .28, p < .001) subsamples. The correlation was slightly stronger among 

Hispanic youths, however, which was the general pattern across the domain scores as well. In the 

full sample, correlations between Juvenile Justice History (rpb = .21, p < .001); Peers and Social 

Support (rpb = .23, p < .001); and Values, Beliefs, and Attitudes (rpb = .21, p < .001) domains and 

recidivism were significant, which also held for the Caucasian and Hispanic subsamples. Two 

domains did not appear to be related to recidivism in the full sample— Family and Living 

Arrangements (rpb = -.00) and Education and Employment (rpb = .02)—nor were they significant 

in the subsamples. Two notable inconsistencies emerged between the Caucasian and Hispanic 

groups. The relationships between recidivism and the domains assessing Pro-Social Skills and 

Substance Abuse, Mental Health, and Personality were significant in the Hispanic subgroup at .17 

and .16, respectively. Among Caucasian youths, however, the relationships were slightly weaker 

and non-significant. 

Table 2.23. Associations Between OYAS Score, Domain Scores, and Recidivism 

Variable Full Sample 

(N = 680) 

Caucasian 

(N = 199) 

Hispanic 

(N = 316) 

Overall Risk Score .26*** .22** .28*** 

Juvenile Justice History .21*** .20** .17** 

Family and Living Arrangements -.00 -.02 .05 

Peers and Social Support .23*** .19** .26*** 

Education and Employment .02 .03 -.00 

Pro-Social Skills .14*** .10 .17** 

Substance Use, Mental Health, and Personality .16*** .14 .16** 

Values, Beliefs, and Attitudes .21*** .17** .22*** 

**p < .01, ***p < .001 

ROC analyses were also carried out to estimate the improvement in predicting risk using 

scores from the risk and needs assessment tool relative to chance (Rice & Harris, 2005). The 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

226 
 

 

 

 

 

 

 

 

estimates presented in Table 2.24 quantify the area under the curve (AUC) and 95 percent 

confidence intervals. Scores over .50 reflect a relative improvement in predicting recidivism over 

random assignment of risk. The overall risk score provides some improvement in predicting 

recidivism (AUC = .65. Domain scores generally appear to enhance prediction as well, except for 

scores in the Family and Living Arrangements (AUC = .50) and Education and Employment (AUC 

= .51) domains. Similar patterns were observed in the Caucasian and Hispanic subgroups, though 

the estimates tended to be slightly higher or lower among Hispanic youths. 

Table 2.24. Area Under the Curve Estimates and 95% Confidence Intervals for OYAS Score, 

Domain Scores, and Recidivism at 12-Months 

Variable Full Sample 

(N = 680) 

Caucasian 

(N = 199) 

Hispanic 

(N = 316) 

AUC AUC AUC 

(95% CI) (95% CI) (95% CI) 

Overall Risk Score .65 .61 .66 

(.60 - .69) (.53 - .69) (.60 - .72) 

Juvenile Justice History .61 .61 .58 

(.57 - .65) (.54 - .69) (.52 - .64) 

Family and Living Arrangements .50 .50 .53 

(.47 - .54) (.43 - .56) (.47 - .58) 

Peers and Social Support .63 .62 .64 

(.59 - .67) (.54 - .70) (.58 - .70) 

Education and Employment .51 .51 .49 

(.47 - .54) (.44 - .59) (.43 - .55) 

Pro-Social Skills .59 .55 .60 

(.54 - .63) (.47 - .63) (.54 - .66) 

Substance Use, Mental Health, and Personality .58 .57 .59 

(.54 - .62) (.49 - .65) (.53 - .65) 

Values, Beliefs, and Attitudes .62 .58 .63 

(.58 - .66) (.50 - .67) (.57 - .69) 

To summarize, the results of the comparative and predictive validity analyses presented 

here suggest scores in the Family and Living Arrangements domain and the Education and 

Employment domain were low for all groups in the sample and do not appear to be related to 

recidivism in this context. The scores in other domains, and the overall risk score, generally are 

higher for Hispanic youths relative to Caucasian youths. Moreover, the relationships between risk 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

227 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

and need assessment scores and recidivism appear to be stronger for Hispanics. Understanding the 

source of these differences offers an opportunity to further improve the use of these assessment 

tools in the juvenile justice system. The next stage of the analysis investigates potential differences 

in the measurement properties of the Residential Tool across the Hispanic and Caucasian 

subgroups. 

Measurement invariance. Second-order CFAs were carried out for separately for 

Caucasian and Hispanic subgroups. Table 2.25 displays the model fit statistics for the two models. 

Among Caucasian youths, the theoretically specified measurement structure provides an adequate 

fit to the data. The CFI and TLI are both above the .90 threshold, and the RMSEA is .05. Among 

Hispanic youths, however, the model does not appear to fit the data as well. The CFI and TLI are 

below the .90 threshold and χ2 = 848.39, which is higher than that observed in the Caucasian 

subsample. 

Table 2.25. Confirmatory Factor Analysis Model Fit Statistics 

Caucasian Hispanic 

Chi-Square (χ2, df) 682.20 (488)*** 848.39 (488)*** 

Comparative Fit Index (CFI) .95 .84 

Tucker-Lewis Index (TLI) .94 .82 

Root Mean Square Error of Approximation (RMSEA) .05 .05 

***p < .001 

Similar to the process used in the full sample, the factor loadings were inspected to identify 

potential threats to the validity of the measures and differences that may exist between the two 

subgroups. Although there was variation in the factor loadings throughout different items on the 

tool, the most notable difference between the subgroups was observed in the Education and 

Employment domain. Factor loadings among the Caucasian youths ranged from .32 to .59, all of 

which are above the .30 threshold employed in the original measurement model with the full 

sample. In contrast, the loadings ranged from .02 to .05 on the three items in this domain in the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

228 
 

 

 

 

 

 

 

 

 

 

 

 

 

model estimated with the Hispanic subsample. These results provide some evidence to suggest that 

measurement invariance cannot be assumed within these subgroups. 

Summary of Race, Ethnicity, and Validity of the OYAS Residential Tool. The findings 

of this study highlight the importance of examining key subgroups within a given population, both 

in terms of the measurement properties of risk and need assessments and the extent to which they 

can be useful in enhancing the prediction of recidivism. The results suggest that the measurement 

of risk for recidivism with this tool could be improved in this population. In terms of the predictive 

validity of the tool in the full sample, the AUC estimate of .65 is consistent with prior research 

(Schwalbe, 2008; 2009); however, when racial subgroups were examined separately the tool 

appeared to be a stronger predictor of recidivism among Hispanic youths relative to Caucasians. 

This study inspected the measurement properties of the OYAS Residential tool by 

subgroup to investigate whether these differences could be explained in part by differences in the 

measurement properties of the tool across groups. The results suggested the measurement structure 

of the tool fit better among Caucasian youths relative to Hispanics. Patterns such as these can be 

indicative of individual items working differently in the two groups (e.g., more or less prevalent 

in one relative to the other), including items that do not contribute to the measurement of risk for 

one or both groups, or failing to include items that have a meaningful impact on risk. These 

complex relationships, which in some case reflect potential threats to the validity of the measures, 

warrant further investigation. 

Modern risk and need assessment tools are intended to be used beyond classifying youths 

in low, moderate, or high risk categories (Lovins & Latessa, 2013). Practitioners are instructed to 

use domain scores to manage cases more effectively by identifying high criminogenic need areas 

and making treatment decisions that align with the assessment results. Toward this end, more 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

229 
 

 

 

 

 

 

 

 

 

 

 

 

detailed investigations into the capacity of risk and need assessment tools to aid users in case 

management in practice is a worthy endeavor. Such detailed investigations of risk and need 

assessment tools require a high degree of planning and collaboration between researchers, 

practitioners, and policymakers. Validation studies are often designed with the general population 

in mind, and as a result rarely have adequate representation of minority groups to allow for analysis 

of some questions examined in this study. Such studies are essential, however, in better 

understanding how the theory of risk and needs assessment plays out in practice. 

Usage Study 4: Professional Override of Assessment Level 

Building from the question of whether there is measurement invariance across groups, the 

notion of systematic risk and needs assessment is to move toward uniformity – or at least 

consistency – in how youth are assessed and then classified. Actuarial risk and needs assessment 

tools still allow for a degree of professional discretion in the form of professional overrides so 

override decisions are a fruitful place to look more specifically at how tools are used by 

practitioners. Professional overrides occur when the administrators of a risk and needs assessment 

tool, or their supervisors, use their professional judgment to change a youth’s risk levels. In 

essence, if upon completion of an actuarial risk and needs assessment tool a youth gets classified 

as low risk, but the professional believes he or she is at a higher risk for recidivism, the professional 

can upward override the youth to either moderate or high risk. Conversely, if an actuarial tool 

classifies a youth as high risk but the administrator believes that the tool overstates his or her 

propensity for future delinquent behavior, the administrator can downward override the youth to 

either moderate or low risk. In practice, upward overrides are more prevalent than downward 

overrides (Wormith, Hogg, & Guzzo, 2012). 

Assessment tools consider youths in the aggregate, making it illogical to expect a risk and 

needs assessment to accurately predict the risk for each specific youth (Silver & Miller, 2002). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

230 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Therefore, a degree of professional discretion exists in the OYAS in the form of professional 

override, but it should be used sparingly (Latessa et al., 2009). According to the RNR framework, 

professional overrides should be used in less than 10 percent of assessments (Andrews et al., 1990). 

Proponents of professional override argue overrides are necessary to increase the accuracy of risk 

classifications because the tool does not account for all theoretical factors that can impact 

recidivism (i.e., history of abuse). However, there are concerns with the utility of professional 

overrides as they have the potential to be harmful. For example, if low risk youths are moved to 

high risk, they may receive more treatment dosage than they need and be in treatment with other 

high risk youths. Past scholarly work on treatment indicates that placing a low risk youth in 

treatment for/with high risk youths can increase his or her propensity for future criminal conduct 

(Latessa, Listwan, & Koetzle, 2014). Similarly, downward overrides have the ability to undermine 

public safety and deny youths the treatment they need. Recent research supports this caution. For 

example, Vaswani and Merone (2014) found the predictive validity of the YLS/CMI decreased in 

cases where professional override of a youth’s risk level occurred. 

Method and Analytic Plan. Little empirical research has been devoted to understanding 

professional overrides in risk and needs assessment. As part of this larger study, this substudy 

explores override usage in JRNA by answering two main questions. First, what is the prevalence 

of professional override in JRNA? Second, what is the nature of these override decisions? That is, 

in what direction (upward/downward) do these overridden primarily occur, what are some 

common characteristics of the cases that are overrides, and what are administrator attitudes toward 

their use? This substudy uses two sources of data: 1) youths’ case record data to examine the 

amount of professional overrides; and 2) data from the juvenile justice personnel interviews and 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

231 
 

 

 

 

 

 

 

 

 

 

 

web-based surveys. These data sources allow us to triangulate what personnel indicate about their 

use of professional overrides with what actually occurs. 

Results 

Case Record Data. As previously noted, the comprehensive assessment sample for this 

study includes case record data for 6,222 youths assessed by the OYAS across the three states 

studied. A professional override occurred in 213 of the assessments (3.4%). Of the 213 cases where 

an override occurred, 210 of them – roughly 98.5 percent – were upward overrides (see Table 2.26 

for the distribution by state). Across all upward override cases, 85 percent were male, 66 percent 

were white, and the average age of youth was approximately 16 years old (sd = 1.8). These profiles 

did not differ substantially from the overall sample, suggesting that there were relatively minor 

differences in sociodemographics between cases with/without override decisions. 

Table 2.26. Prevalence and Direction of Overrides – Case Record Data 

Variable State 1 State 2 State 3 Total 

n (%) n (%) n (%) n (%) 

No Override 1,803 (88.2) 1,705 (98.6) 1,929 (96.5) 5,437 (96.6) 

Downward Override 2 (0.1) 0 (0.0) 1 (0.1) 3 (0.1) 

Upward Override 

Missing 

Total 

106 (5.2) 

131 (6.4) 

2,042 (100.0) 

25 (1.5) 

0 (0.0) 

1,730 (100.0) 

79 (3.2) 

441 (18.0) 

2,450 (100.0) 

210 (3.4) 

572 (9.2) 

6,222 (100.0) 

The two cases with downward overrides in State 1 involved incarcerated youths. The 

downward override in State 3 involved a 10-year-old white male who was moved from moderate 

to low risk. This youth was charged with misdemeanor disorderly conduct and diverted from the 

system. State 1 was the only state that provided a field for a statement of reasons for overrides. 

Among the most common reasons for overrides were sex offenses, the need for more services, 

offense severity, and criminal history (see Figure 2.1). Sex offense, the most common reason for 

override, was noted in 34 percent of override cases. Services was the second most common reason 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

232 
 

 

 

 

 

 

 

 

 

 

 

for override (31%), followed by offense severity (29%), and criminal history (22%). State 2 did 

not provide statements of reason for override decisions; however, similar trends emerged. Among 

the 25 cases with overrides in State 2, 64 percent included a youth charged with a sex offense, 76 

percent were classified as a felony, and 92 percent received at least one treatment program. The 

most common disposition for override cases in State 2 was probation (76%). 

Though a statement of reason for override decisions was not provided in State 3, offense 

type information was available. Of the upward overrides in State 3 (n = 79), 42 percent had a focal 

offense categorized as a crime against a person and 55 percent had a focal offense categorized as 

a felony. No treatment information was available for State 3 cases; however, of the upward 

override cases with disposition information (n = 56), 16 percent (9 juveniles) were placed in a state 

facility, 34 percent (19 juveniles) received intensive probation, 25 percent (14 juveniles) received 

probation, and 20 percent (11 juveniles) had their cases dismissed. 

Figure 2.1. Reason for Overrides, State 1 Statement of Reasons (N = 107) Percent 

0 

20 

40 

60 

80 

100 

n = 36 n = 33 n = 31 n = 24 n = 19 n = 11 n = 9 n = 7 n = 4 n = 4 

In addition to the prevalence and reasons for overrides, we examined the point of the 

juvenile justice process at which overrides typically occur (See Table 2.27). They occurred almost 

exclusively in post-adjudication tools (i.e. Disposition, Residential, and Reentry). Only 5 of the 

213 overrides – or 2.3 percent – occurred in pre-adjudication tools (i.e. Detention and Diversion). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

233 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This is noteworthy, given that roughly 25 percent of the overall sample was assessed using preadjudication tools. It does, however, reflect the less expansive nature of those tools as well as the 

nature of the decision and expectations around the assessment at those stages. In interviews for a 

previous study, a detention facility superintendent reported that his/her staff rely more heavily on 

the seriousness of the current offense and likelihood to appear for court hearings in making 

decisions about youths’ risk.65 

Table 2.27. Override by Assessment Type 

Tool Overall % of Total Override % of Override 

Detention 370 5.95 3 1.41 

Disposition 2,204 35.42 102 47.89 

Diversion 1,213 19.51 2 0.94 

Reentry 989 15.90 39 18.31 

Residential 1,446 23.24 67 31.46 

Total 6,222 213 

In short, of the 6,222 cases, 213 cases were overrides. That is, across the three states only 

approximately three percent of cases were overrides. Of the 213 override cases, 210 overrides were 

upward overrides and three were downward overrides. The most common override was from lowto-moderate risk (n = 112 cases) (see Table 2.28). In an initial analysis of case outcomes, we found 

that the odds for recidivism (new adjudication) for those cases with override decisions about risk 

level were roughly 7 percent lower than those that were not overrides, but that relationship was 

not statistically significant.66 

Table 2.28. Cross-tabulation of Original Risk Level and Override Risk Level 

Assessed Risk Level 

n (%) 

Override Risk Level Low Mod High Total 

65 Citation omitted for confidentiality reasons, but results and raw transcript available from first author upon request. 66 A full assessment of the processing and outcomes of these cases is beyond the scope of the current report, but will 

be investigated in future research. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

234 
 

 

 

 

 

 

Low 

Mod 

High/Very High 

—

112 (71.3) 

45 (28.7) 

2 (3.6) 

—

53 (96.3) 

1 (100.0) 

--

—

3 

112 

98 

Total 157 55 1 213 

Personnel Interviews and Surveys. As outlined in Section I: Implementation of JRNA: 

Views from Juvenile Justice Personnel, juvenile justice personnel were asked a series of questions 

about the utility, process, and prevalence of professional override. Both the interviewees and 

survey respondents were asked to estimate the percent of assessments that result in an override 

(see Table 2.29). Interviewees indicated that about 10.5 percent (sd = 18.17) of assessments at 

their agencies result in professional override. However, there was some variation in that estimate 

as two interviewees, both with sex offender caseloads, reported that 100 percent of assessments 

are overrides in their respective agencies. In light of the two outliers, the median (6%) is a more 

accurate measure of central tendency. 

Table 2.29. Perceptions of Override Prevalence Among Juvenile Justice Personnel 

n Median x̅ sd Min.-Max. 

Interview 64 6% 10.5% 18.2 0-100 

Survey - General 99 5% 8.3% 11.6 0-95 

Survey - Self 496 2% 6.9% 15.1 0-100 

In the web-based survey, juvenile justice personnel who do not personally administer risk 

and needs assessments were asked what percent of assessments they perceived were overrides in 

their agencies. Like the interview data, there was some variation in response. In particular, one 

respondent signified his or her agency overrides 95 percent of assessments. However, this 

respondent works in a sex offender unit and specified that unit – not the agency as a whole. With 

this in mind, the median is the better measure of central tendency to describe the trends. The 

median response was 5 percent (29% of the sample gave this exact estimate). Meanwhile, if the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

235 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

juvenile justice personnel personally administer the assessment, they were asked what percentage 

of cases they personally override. On average, respondents reported they override 6.89 percent (sd 

= 15.05) of assessments. However, there was some variation in responses, as 76 percent of the 

sample indicated overriding five percent or less of their assessments. Likewise, with this in mind, 

the median is the better measure of central tendency, which was two percent. 

In short, we found evidence that risk and needs assessment administrators use professional 

overrides less often than non-administrators perceive overrides to occur in their agencies. Data 

from both of these self-report sources are roughly consistent with those identified in the case 

records above. Less than four percent of cases in the case record data were overrides, and across 

all three different self-report scenarios the median percent of cases perceived to be overrides 

ranged from two to six percent. 

Interviewees were asked to signal their level of agreement, on a five-point Likert scale, 

with the following statement: “Is there anything specific in the OYAS that you perceive as a 

limitation that lead to these overrides.” The majority of interviewees responded affirmatively,

saying the OYAS does have limitations that lead to overrides (Strongly Agree/Agree = 51.6%; 

Strongly Disagree/Disagree = 39.8%; Unsure = 8.6%). Aligning with the reasons for override 

identified in State 1 (i.e. sex offenses and offense severity), examples of open-ended responses to 

this statement include: “Question[s] could be a bit clearer. Better definitions;” “Family section not 

capturing reality;” and “Not taking into consideration the offense itself is the main thing (e.g. sex 

offenders).” Both interview and survey respondents were asked whether their agencies allow 

overrides. Among interviewees, 87.8 percent indicated their agencies allows them, 9.3 percent 

indicated their agencies did not (2.9% were unsure). Web-based survey respondents were not given 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

236 
 

 

 

 

 

 

 

 

 

 

 

 

 

the option to respond “Unsure.” Among web-based survey respondents, 87.4 percent said their 

agencies allow for overrides; 12.6 percent said their agencies did not. 

Interviewees were asked to provide common reasons for overrides (Figure 2.2). Their 

responses show that the most common reason for override was sex offender/offenses (60.4%), 

aligning with the statement of reasons provided in the State 1 case record data. The second most 

common reason for override recorded by interviewees was professional discretion (28.3%). For 

example, one interviewee said that relying on self-reporting by juveniles is insufficient to 

understanding their risk – in other words, the professional believed OYAS administrators should 

not rely on answers from the youths. Other common reasons for overrides listed by interviewees 

were discrepancy between the interview and collateral information, criminal history, and nature of 

the current charge. 

Figure 2.2. Juvenile Justice Personnel Interviews – Common Reasons for Overrides (N = 105) 

100 

90 

80 

70 n = 64 

n = 30 

n = 14 n = 13 n = 20 n = 18 

0 

10 

20 

30 

40 

50 

60 

% of Respondent Pool 

Sex Offender Professional Discrepancy Criminal History Nature of Other 

Discretion (Interview & Current Charge 

Collateral) 

Note: Interview respondents were able to indicate more than one common reason for overrides. Therefore, percent 

responding affirmatively is greater than 100%. 

Web-based survey respondents were asked to list the type of cases that are typically 

overridden (Figure 2.3). Of the 398 valid responses to this question, 305 respondents (77%) said 

that sex offenses were typically overridden, 119 respondents (30%) said cases with severe offenses 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

237 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

(e.g. violent, firearm) are typically overridden, and 47 respondents (12%) listed extensive criminal 

history as a typical reason for an override (see Figure 2.3). All three of these responses were also 

listed above in the case record and interview data as common reasons for overrides. 

The majority of interviewees said there were no offenses that carried a mandatory override 

decision in their agencies (48.2%). Meanwhile, 38.2 percent of interviewees said there were some 

cases which must be overridden because of the offense committed. Of these 38.2 percent, 30.9 

percent indicated that cases with a sex offense must be overridden, 3.6 percent said cases with a 

gun/violent/severe offense must be overridden,67 and 5.5 percent said some cases must be 

overridden but did not specify the types of cases. The remaining 12.7 percent of interviewees said 

they were unsure. 

Figure 2.3. Web-based Survey – Common Reasons for Overrides (N = 398) 

Sex Offenders 

n = 305 

Severe 

Offenses 

n = 119 

Criminal 

History 

n = 47 Mental Health 

n = 25 

Other 

n = 54 

0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

% of Respondent Pool 

Sex Offenders Violent/Gun/Severe Felony Lengthy Criminal History 

Mental Health Youth Dishonest High Service Needs 

Status Offenders Substance Use Other 

Note: Survey respondents were able to signify more than one common reason for overrides. Therefore, percent 

responding affirmative is greater than 100%. 

67 All four interviewees who said cases with a gun/violent/severe offense were from in State 1. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

238 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of Override Substudy. We draw four key findings from these analyses. First, 

the majority of cases are not overridden. Of the 6,220 assessments in the comprehensive 

assessment sample, 213 assessments (i.e., less than four percent) resulted in an override. Thus, 

across all three states, agencies tended to rely on the youth assessment tool for classification of 

youths and diverged from it relatively rarely, at least in terms of final risk level designations. 

Second, overrides were primarily used to increase youths’ risk levels. One potential 

explanation for this is that upward overrides are used when there is a perceived threat to community 

safety. In both the case record and interview/survey data, sex offenses (which are typically viewed 

as a threat to the community), were the most common reason for overrides. Another potential 

explanation is increasing risk levels results in more opportunity for services. Some programs have 

eligibility requirements that limit the pool of offenders who can participate. In essence, if an 

administrator thinks that a youth should enroll in a program, but the program has a risk-level 

eligibility requirement (e.g., youths must be high risk to participate), the administrator may use an 

override in order to render them eligible. This suggests the importance of understanding the 

implementation context, especially local political pressures and resource constraints, that can 

affect full adherence to assessment protocols (see Aarons et al., 2011; Fixsen et al., 2009). It also 

echoes findings from the interviews and survey results presented in Section I. A preliminary 

bivariate analysis found a small, nonsignificant negative relationship between override—which 

again was almost always upward—and new adjudication. 

Third, among the most common reasons for overrides were sex offenses, offense 

seriousness, and criminal history. Interviewees and survey respondents provided context to this 

finding. One respondent said about the OYAS, “It doesn’t capture sex offenders, [and] doesn’t 

address [the] youth who doesn’t attend school. This is a problem and should be scored better.”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

239 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Another respondent said, “It doesn’t pick up offense seriousness.” Thus, doubts about the validity

of the OYAS with certain juvenile justice personnel populations may be driving the rate of 

overrides. 

Last, the use of overrides tends to be heterogenous across agencies. For example, when we

asked interviewees about the process for administering an override, 48 percent said there was no

formal process or that they were unsure about the process. Meanwhile, 46 percent said that

supervisor approval must be obtained before an override can occur. In addition, 52 percent of

interviewees said that certain offense categories mandate an override at their agency, while 48

percent said that no offenses carry a mandatory override. 

Usage Study 5: Mental Health and Substance Use Information and Treatment Matching 

As noted in Usage Study #5, overrides were not often conducted. Now, we turn to 

examining the particular usage case of mental health and substance abuse screening and 

subsequent treatment decisions in order to further evaluate the ways in which OYAS was being 

used in our research sites. This is a useful substudy because it considers a very specific focus in 

matching interventions to potential needs, which is an intended use in the logic of JRNA. The 

number of agencies and facilities screening youths for service needs has increased from 2000 to 

2016. However, despite the prevalence of substance use and psychiatric disorders in youth 

populations, mental health and substance use needs continue to be screened for the least 

(Puzzanchera & Hockenberry, 2018). Several studies have found that between 60 percent and 70 

percent of youths in the juvenile justice system meet criteria for at least one psychiatric condition 

(Abram et. al, 2003; Atkins et. al, 1999; Shufelt & Cocozza, 2006; Wasserman et. al, 2002). 

McClelland, Teplin, & Abram (2004) utilized data from the Northwestern Juvenile Project to 

examine the prevalence of substance use in 1,829 juvenile detainees. Analysis of self-report data 

showed that 77 percent of the detained youths reported substance use in the past six months and 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

240 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

90 percent reported lifetime usage (McClelland, Teplin, & Abram, 2004). Rates of substance use 

and mental illness in juvenile justice populations are concerning as—in addition to their impact on 

youths—these needs may be barriers to effective treatment and successful completion of court 

requirements. 

Effective screening and assessment is a necessary first step in addressing those issues, but 

there are also outstanding questions about how to carry out such procedures (Grisso, Vincent, & 

Seagrave, 2005). For instance, Grisso and Vincent (2005) mention that time and resources may 

factor into the decision on the length of screening and assessment for mental health issues. 

Numerous interviewees and survey respondents note that mental health and substance use issues 

are concerns in screening and assessing youths (see e.g., Section I, pp. 54). As such, we study the 

linkage between substance use and mental health information included in the OYAS and 

subsequent treatment in one of the study states as a way to characterize potential uses of the tool. 

Specifically, we answer research questions about whether youths are appropriately matched with 

services based on specific domain and item information. In turn, this helps give insight as to 

whether the mental health and substance use indicators typically included on risk assessments are 

enough to actually inform treatment referral. 

Method and Analytic Plan. Case record data were obtained from four juvenile courts in 

State 2. These courts were utilized in this study because they had the most complete data for 

treatment received as well as item-level OYAS data necessary to precisely measure substance use 

and mental health needs. In order to address the above research questions, the current study had 

two objectives: (1) examine indicators of substance use and mental health problems based on the 

OYAS, and (2) assess the links between these indicators and the services received. 

Measures (Independent Variables) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

241 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mental health indicator. The mental health indicator found in the OYAS Disposition Tool 

was used to measure mental health need. Several questions are asked regarding mental health, but 

these responses are then condensed into a single indictor. Mental health need is indicated by 

affirmative response(s) to, or evidence of, any significant mental health issues that have not been 

stabilized; prescribed medication; or previously seen by a mental health professional (excluding 

ADHD and any victimization). This item is scored dichotomously as “0” (no mental health issues 

or is stabilized on medication) or “1” (youth demonstrated significant untreated mental health 

issues). 

Substance use indicators. Substance use need was measured using the items in the 

substance use domains of the OYAS Disposition, Residential, and Reentry Tools. Substance use 

indicators vary in assessment by type of question, scoring, and quantity of questions (see Appendix 

I for question variations). Because there are additional personality indicators in the substance use 

domain, we created a summed substance use index score (e.g., 0 = a youth did not indicate any 

substance use need, 1 = the youth indicated substance use need on 1 item). This step ensured that 

the personality indicator did not play a role in the total substance use score and potentially add 

error to measuring need on that domain. 

Measures (dependent variables) 

Mental health treatment and cognitive behavioral therapy (CBT). To measure whether a 

youth received any mental health treatment, we created a new dichotomous variable (0 = did not 

receive mental health treatment/psychological treatment/counseling, 1 = mental health 

treatment/psychological treatment/counseling received). To measure whether a youth received any 

CBT, we created a separate dichotomous variable (0 = did not receive CBT treatment, 1 = CBT 

treatment received). Because the provided treatment information was not always specific or 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

242 
 

 

 

 

 

 

 

 

 

 

 

detailed, we used any indication of treatments or programming often used to address these 

indicators such as counseling, therapy, cognitive behavioral therapy, or outpatient counseling. 

Substance use treatment. To measure whether a youth received any substance use 

treatment, we created a new dichotomous variable (0 = no substance use treatment, 1 = substance 

use treatment received). We used any indication of treatment or programming typically used to 

address substance use such as substance use counseling, addictions 101, or substance abuse 

assessment to measure treatment referrals in this area. 

Results 

Descriptive statistics. This sample was comprised of 1,107 (80%) male youths and 276 

(20%) female youths, with an average age of 15.8 (sd = 1.4). The majority (52.7%) of youths 

identified as Non-White and 47.2% of youths identified as White. About 21% of the youths who 

received the disposition assessment indicated a mental health need. Our sample had, on average, 

low to moderate substance use needs (see Table 2.30 for descriptive statistics). 

Table 2.30. Sample Descriptive Statistics (N = 459) 

Variable Descriptive Statistics 

x̅/% sd Min. Max. 

Age 

Race (1 = White) 

Gender (1 = Female) 

Juvenile Justice History Score (0-6) 

MH Indication Disposition (1 = Yes) 

Condensed SU Index (Disposition) 

Condensed SU Index (Reentry) 

Condensed SU Index (Residential) 

15.8 

53.0 

20.0 

2.01 

21.0 

0.79 

1.87 

2.79 

1.37 

--

--

1.54 

--

0.95 

1.31 

1.64 

11 

0 

0 

0 

0 

0 

0 

0 

20 

1 

1 

6 

1 

4 

5 

5 

Note: sd = Standard Deviation; Min. = Minimum; Max. = Maximum; MH = Mental Health; 

SU = Substance Use. 

Multivariate logistic regression models. We estimated multivariate logistic regression 

models of the relationship between mental health and substance use risk indicators and treatment 

received, breaking the analyses down by specific assessment type where necessary. Control 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

243 
 

 

 

 

 

 

 

 

 

 

 

 

variables included age, race, gender, whether the case had a violent focal offense (0 = No; 1 = 

Yes), and juvenile justice history score from the assessments (0 to 6). 

The results for the multivariate regression of youth referral to any mental health treatment 

suggest that—when controlling for other variables—those youth that were indicated on the OYAS 

were in fact significantly more likely to be referred to that programming. The odds ratio (1.74) 

suggest that they were 74 percent more likely to be referred to such treatment (see Table 2.31). No 

statistically significant relationships were identified for CBT and Substance Use Treatment and 

the OYAS mental health indicator. 

Table 2.31. Summary of Logistic Regression Analysis for Mental Health Indicator, Substance 

Use, and Treatment (N = 459) 

Variable Mental Health Indicator 

Treatment Type 

Any Mental Health Treatment 

Any Cognitive Behavioral Treatment 

Any Substance Use Treatment 

OR 

1.74* 

0.86 

0.99 

Notes: *p<.05, **p<.01; OR = odds ratio; all relationships estimated in models including controls for age, gender, 

race, violent focal offense, and juvenile justice history. 

The model results for the regression predicting treatment referral based on the substance 

use score are presented in Table 2.32. The relationship between substance use score and substance 

use treatment was statistically significant in all models, suggesting that a one unit increase on need 

score led to between 50 and 87 percent greater odds of referral to substance use treatment. In the 

residential tool model, youths also were more likely to be referred to CBT interventions. 

Table 2.32. Summary of Logistic Regression Analysis for Mental Health Indicator, Substance 

Use, and Treatment (N = 459) 

Variable Substance Use Need Score 

OR 

Disposition Tool 

Any Substance Use Treatment 1.71** 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

244 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table 2.32. Summary of Logistic Regression Analysis for Mental Health Indicator, Substance 

Use, and Treatment (N = 459) 

Variable Substance Use Need Score 

OR 

Any Cognitive Behavioral Treatment 1.33 

Any Mental Health Treatment 1.12 

Residential Tool 

Any Substance Use Treatment 1.87** 

Any Cognitive Behavioral Treatment 2.05* 

Any Mental Health Treatment 1.12 

Reentry Tool 

Any Substance Use Treatment 1.53* 

Any Mental Health Treatment 1.02 

Notes: *p<.05, **p<.01; OR = odds ratio; all relationships estimated in models including controls age, gender, race, 

violent focal offense, juvenile justice history, there were no cases with CBT treatment in the reentry sample.68 

Summary of Treatment Matching. Youths in the sample had relatively low rates of 

mental health need when identified with the OYAS marker. Of the youths who received the 

disposition assessment, only 20 percent had indicated for a mental health need, which is a low 

estimate compared to previous studies. On the other hand, the relationship between the mental 

health indicator and mental health treatment referral was statistically significant. This raises 

questions as to the thoroughness of the single item measure, but also suggests that it may be serving 

as a gateway to needed services. The youths in the current sample had low to moderate substance 

use risk. Overall, it appears that indication of substance use risk is informing receipt of necessary 

treatment—statistically significant result in most analyses, where higher scores lead to a greater 

likelihood of substance use treatment. They also were significantly more likely to receive cognitive 

behavioral interventions. 

68Violent focal offense and age were the only control variables that were statistically significant in multiple sets of 

regression models. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

245 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

The results of this study suggest that these items and scores are related to receipt of 

treatment in this state. These tools are intended to guide referrals to treatment, but indicators are 

limited and not necessarily specific. Still, the information does seem to be impacting decision 

making in these cases. With more information, assessments could potentially better inform 

treatment planning and case management for justice-involved youths. Mental health and substance 

use disorders can progress, sometimes at a rapid rate, and if not treated, they can have salient 

effects on youths’ ability to successfully complete treatment and supervision requirements and, in 

turn, may impact other life outcomes. 

Usage Study 6: OYAS Strengths and Barriers 

As mentioned in previously in this section of the report and in the usage studies specifically, 

case records can provide insight into juvenile risk and need assessments usage that extend insight 

drawn from interviews and surveys of juvenile justice personnel. Certain responses to usage 

questions suggest that perhaps the scope of use of risk and needs assessment is not as wide as 

researchers might recommend. For example, use of the assessment information seemed far more 

comprehensive on risk management concepts as opposed to case planning and treatment sides in 

the juvenile justice personnel responses.

69 To further this understanding, this section considers the 

possible extension of juvenile justice assessment to cover strengths and barriers that may be 

utilized in working with youths, including in fashioning treatment and supervision plans. 

The information gathered in juvenile risk and need assessments should be used to identify 

those who are at the greatest risk of recidivating, determine their most pressing criminogenic 

needs, and match the youths to the most appropriate treatments (Bonta & Andrews, 2017). In a 

juvenile justice context, Bonta and Andrew’s risk-needs-responsivity (RNR) model asserts that a 

69 See case record analyses in Section III of this report as well. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

246 
 

 

 

 

 

 

 

 

 

 

 

 

youth’s risk level should drive supervision and treatment intensity and duration. At an agency level 

resources should be prioritized for higher risk/need youths and lower risk/need youths should be 

given fewer services. The model also states that delinquency producing factors (i.e., criminogenic 

needs) should be targeted for change, especially if recidivism reduction is a goal. Lastly, the 

responsivity component of the theory purports that to be effective in reducing recidivism, proven 

methods should be used to deliver services and that these services need to be responsive to each 

individual youth in the juvenile justice system. 

While early assessments tended to focus primarily on risk and need factors, recent tools 

have expanded to include other factors that may impact an individual’s likelihood of success or

failure (Lovins & Latessa, 2013). Consistent with the responsivity principle, practitioners are 

encouraged to identify barriers to treatment and to be mindful of potential challenges when 

tailoring supervision and treatment. For example, common barriers faced among adolescents such 

as financial issues, problems with family and peer relationships, education, and access to health 

services and transportation (Arnold, Walsh, Oldham, & Rapp, 2007). Acknowledging these 

barriers is a fundamental step in effective case management and development of realistic case or 

treatment plans. 

In conjunction with the notion of responsivity, the growing focus on strength-based 

juvenile justice attempts to better engage with youths, provide opportunities to build upon 

available resources, and promote positive development (Nissen, 2006). A number of interventions 

have been developed under this strengths-based approach such as Strength-based Case 

Management (SBCM), Intensive Aftercare Program for youths reentering the community, and the 

Good Lives Model (GLM) of rehabilitation (Ward & Brown, 2004). Proponents of this general 

perspective claim that shifting from primarily focusing on problematic behaviors and labeling 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

247 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

youths as “difficult” will promote more successful outcomes among juvenile justice systeminvolved youths through a more comprehensive mode of case management and treatment. 

Given the potential benefits, more recent generations of risk and needs assessment tools—

including the OYAS—have incorporated items to identify strengths and barriers. While capturing 

these aspects of youths’ lives may be valuable, the extent to which strengths and barriers are

incorporated into case management may be limited by challenges in implementation. As an 

example, the strengths and barriers included on the OYAS are not required to be completed on the 

assessment nor are strengths and barriers incorporated into the total risk score or level. 

Practitioners whose agencies do not require them to fill out strengths and barriers, or practitioners 

who are facing time constraints during the risk and need assessment process, could decide to omit 

these items. Additionally, there may be difficulty in using the information effectively once it is 

collected. For example, agency resources may be too limited to fully address barriers to treatment 

(e.g., this may be particularly difficult in rural areas). 

Current substudy. The inclusion of strengths and barriers on actuarial risk assessments 

may be a promising step forward in providing services for justice-involved youths; however, this 

remains an under-researched area and adds steps to risk and needs assessment processes, which as 

highlighted above, can be challenging to implement at a basic level. Prior research has not 

examined the extent to which practitioners are capturing these aspects of youths’ life

circumstances, and whether doing so can enhance case management and our understanding of 

justice-involved youths. In light of these shortcomings, this study has two aims. First, we 

determine the extent to which practitioners are assessing strengths and barriers with the OYAS, 

and their perceptions of the tool’s utility in gathering this information. Second, we investigate

whether there are any differences in the composition of strengths and barriers across subgroups 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

248 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

based on the tool used to complete the assessment and risk classification. This allows us to better 

understand this special topic of JRNA and, in turn, consider the degree to which agencies and 

personnel utilize additional options for understanding youths’ cases when given the opportunity. 

Method 

Sample. The study was conducted using two sources of data. The first is a sample of 2,361 

assessments selected from the OYAS database that were completed between 2010 and 2015.70 

While some of these cases overlap with the comprehensive assessment sample described above, 

they are drawn more generally the OYAS database in State 1. These data include assessment and 

demographic information that is officially recorded in the database. We included every assessment 

that was completed with one of the three tools that incorporates strengths and barriers (i.e., 

Disposition, Residential, and Reentry).71 To gain insights into practitioners’ perceptions, data

gathered through surveys and interviews with 71 practitioners in State 1 in 2015 and 2016 is also 

examined (see Section I for juvenile justice personnel interview and survey methodology). 

Measures. The demographic measures captured in the youth data include whether the 

youths were white (48.1%) or non-white (50.3 %), and whether they were male (77.1%) or female 

(22.8%). OYAS assessment data includes a number of key variables including the assessment tool 

used, item scores, domain scores, total risk score, and total risk level. Overall, 57 percent of the 

sample was classified as low risk, 30.2 percent as moderate risk, and 12.9 percent as high risk. The 

tools included in this study are limited to the Disposition (74.4%), Residential (11.6%), and 

Reentry (14.1%) Tools, as the other tools in the OYAS do not include strengths and barriers. 

70 Though we did not formally assess time trends in recording of strengths and barriers during this time window, there 

were no concerted prevalence shifts in recording of strengths/barriers in the data on which these analyses were based. 

71 Fifty-three individuals were assessed more than once with different tools during this time period. Given the focus 

on the assessment process and that each type of tool is largely analyzed individually, all of these cases were retained 

in the sample. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

249 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Item scores are recorded for risk indicators, as well as strengths and barriers. With respect 

to the risk indicators the number of items varies, but the exact same 30 strength and barrier 

indicators are included across all of the tools. For every item on the assessments, practitioners are 

given scoring criteria to use to determine if the item applies to a particular case. Among the strength 

and barrier indicators, if the criteria specified in the scoring guide are met the appropriate box is 

checked. If a youth does not possess a particular strength or barrier based on the provided 

instructions, the OYAS administrator is instructed to leave the item blank. This aspect of 

implementation limits the capacity to distinguish between instances in which practitioners have 

skipped sections of the assessments and when an item was considered but determined to be neither 

a strength nor a barrier. In light of this concern, a measure was created to determine whether any 

item was recorded as a strength or barrier on a youth’s assessment.

In scoring the assessments, the total number of strengths and the total number of barriers 

in each domain and overall across the entire tool can be calculated. These values are included in 

the current study. One set of additional variables was created for the analyses to reflect the overall 

degree of strengths and barriers in each domain and overall. These measures were created by giving 

every item marked as a strength a value of 1 and every item marked as a barrier a value of -1, and 

then summing the values. Negative scores reflect a greater presence of barriers, positive scores are 

indicative of a greater concentration of strengths, and values close to 0 suggest that the number of 

strengths and barriers are nearly equivalent. Total scores for strengths and barriers can range from 

0 to 30, and the range of possible values for combined scores is -30 to 30. 

Analytic Plan. First, the general composition of the sample of youths is described to offer 

a sense of the strengths and barriers typically captured in the assessments. Importantly, these 

values will indicate how often practitioners assess these aspects of youths’ cases and provide a

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

250 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

general sense of common strengths and barriers observed in the sample. Chi-Square tests and 

analysis of variance are used to determine whether any observed differences between groups are 

statistically significant. These analyses provide insights into whether or not there are differences 

in assessment practices (e.g., are strengths and barriers recorded for girls more so than boys?), as 

well as whether or not there are differences in the extent to which youths in particular subgroups 

are exposed to strengths and barriers (e.g., do low risk youths encounter more strengths than 

moderate or high risk youths?). Where possible, we also incorporate information gathered through 

surveys and interviews with practitioners to contextualize the findings. 

Results. Overall, administrators recorded at least one strength or barrier for 65.7 percent 

of youths in the sample. Among youths assessed with the Disposition Tool, 62.3 percent had at 

least one strength or barrier noted in the assessment. About three quarters of youths who were 

assessed with the Residential and Reentry Tools had a strength or barrier recorded. Across all of 

the tools, 69 percent of non-white youths had at least one strength or barrier recorded, which was 

significantly more (χ2 = 11.48, p < .01) than the 62.3 percent of white youths. There was no 

significant differences observed in the recording of these items between males and females; 

however, the proportions of youths for which strengths and barriers were recorded did significantly 

differ by risk level (χ2 = 6.71, p < .05). Low risk youths had at least one recorded in 64.2 percent 

of cases, compared to 65.9 percent of moderate risk youths and 72 percent of high risk youths. 

Although practitioners appear to record strengths or barriers for the majority of youth who 

are assessed, it is important to further examine what is being captured. The average number of 

strengths and barriers overall and by domain for each tool are shown in Table 2.33. Recall that 

positive values on the combined scores reflect a greater number of strengths and negative scores 

reflect a greater number of barriers. For youths assessed with the Disposition Tool, the total 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

251 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

combined score of 5.38 indicates that there was a greater presence of strengths relative to barriers. 

This was generally observed across most domains, and the greatest number of strengths tended to 

be recorded in the Education and Employment domain (x̅ = 3.17), followed closely by the 

Substance Abuse, Mental Health, and Personality domain (x̅ = 3.00). The greatest number of 

barriers for youths assessed with the Disposition Tool were recorded in the Peers and Social 

Support Network domain (x̅= 1.87). Inspecting the frequencies across individual items further 

revealed that the least likely items to be recorded as a strength or barrier were whether the youth 

had a high school diploma or GED (missing = 76%), whether the youth had previous employment 

experience (missing = 72.5%), or whether the youth had an individualized education plan (missing 

= 70.7%).72 The most prevalent strengths were having a family that is supportive of change (41%) 

and parents who are supportive of education (38.8%). The items with the greatest prevalence as a 

barrier for youths were having prosocial peers (25.6%), managing antisocial peers effectively 

(26.5%), and engaging in prosocial leisure activities (25.4%). 

Table 2.33. Strengths and Barriers by Tool and Domain 

Variable Strengths Barriers Combined 

Disposition Tool x̅(sd) x̅(sd) x̅(sd) 

Family and Living Arrangements 2.90 (1.69) 1.08 (1.40) 1.82 (2.80) 

Peers and Social Support Network 1.47 (1.51) 1.87 (1.47) -.40 (2.79) 

Education and Employment 3.17 (1.94) 1.50 (1.63) 1.67 (2.84) 

Pro-Social Skills 2.07 (1.51) 1.25 (1.29) .82 (2.59) 

Substance Abuse, Mental Health, & 

Personality 3.00 (2.43) 1.62 (1.72) 1.38 (2.48) 

Values, Beliefs, & Attitudes 1.91 (1.14) .66 (.94) 1.24 (1.96) 

Total 11.95 (9.30) 6.57 (6.21) 5.38 (11.72) 

Residential Tool x̅(sd) x̅(sd) x̅(sd) 

Family and Living Arrangements 2.41 (1.77) 1.54 (1.55) .87 (3.06) 

72Frequencies for individual items were calculated for all individuals who were assessed with the Disposition tool, 

regardless of whether any strengths or barriers items were scored. This is true of the analysis of Residential and 

Reentry tools as well. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

252 
 

 

 

 

 

 

 

 

 

 

 

Table 2.33. Strengths and Barriers by Tool and Domain 

Variable Strengths Barriers Combined 

Peers and Social Support Network .86 (1.24) 2.69 (1.34) -1.84 (2.45) 

Education and Employment 2.89 (1.83) 2.16 (1.80) .72 (2.92) 

Pro-Social Skills 1.44 (1.29) 1.94 (1.25) -.50 (2.33) 

Substance Abuse, Mental Health, & 

Personality 2.43 (2.02) 2.99 (1.72) -.56 (3.27) 

Values, Beliefs, & Attitudes 1.28 (1.18) 1.37 (1.16) -.09 (2.24) 

Total 9.81 (7.32) 10.98 (6.75) -1.16 (10.41) 

Reentry Tool x̅(sd) x̅(sd) x̅(sd) 

Family and Living Arrangements 2.88 (1.79) 1.50 (1.74) 1.38 (3.36) 

Peers and Social Support Network 1.61 (1.57) 2.02 (1.51) -.41 (2.97) 

Education and Employment 3.99 (1.90) 1.69 (1.69) 2.31 (3.04) 

Pro-Social Skills 2.33 (1.57) 1.36 (1.47) .97 (2.94) 

Substance Abuse, Mental Health, & 

Personality 3.77 (2.21) 2.16 (1.83) 1.62 (3.68) 

Values, Beliefs, & Attitudes 1.99 (1.20) .88 (1.16) 1.10 (2.32) 

Total 15.11 (8.99) 8.76 (7.17) 6.34 (13.70) 

The average scores observed among youths assessed with the Residential Tool are also 

shown in Table 2.33 and reveal a somewhat different pattern. Overall, the total scores suggest that 

youths assessed with this tool may be facing more barriers (x̅= 10.98) than strengths (x̅= 9.81). 

The combined scores suggest that barriers are more common for these youths in the Peers and 

Social Support Network (x̅= -1.84); Pro-Social Skills (x̅= -.50); Substance Abuse, Mental Health, 

and Personality (x̅= -.56); and Values, Beliefs, and Attitudes (x̅= -.09) domains. Strengths tended 

to outweigh barriers in the Family and Living Arrangements (x̅ = .87) and Education and 

Employment (x̅ = .72) domains, however. Four items emerged as being the least likely to be 

recorded as a strength or barrier: whether the youth had a high school diploma or GED (missing = 

62.3%), whether the youth had previous employment experience (missing = 58.2%), whether the 

youth had an individualized education plan (missing = 57.1%) and the youth’s attitude towards

psychotropic medication (missing = 59.7%). Similar to the Disposition Tool, the most prevalent 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

253 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

strengths were having a family that is supportive of change (46.5%) and parents who are supportive 

of education (45.1%). The most frequently noted barriers included having prosocial peers (50.9%), 

managing antisocial peers effectively (51.6%). 

Among youths assessed with the Reentry Tool, strengths tended to outweigh barriers as 

indicated by the total combined average of 6.34. The average number of overall strengths was 

highest in these assessments at 15.11 compared to 11.95 on the Disposition Tool and 9.81 on the 

Residential Tool. Strengths were more prevalent than barriers in every domain except Peers and 

Social Support Network, where the average combined score was -.41. This was the only domain 

across all three tools in which youths tended to face more barriers. Relatedly, the Family and 

Living Arrangements domain was the only one in which youths tended to have a greater number 

of strengths, regardless of which tool was used to complete the assessment. Generally, the same 

item-level patterns were observed among the assessments completed with the Reentry Tool. In 

addition to the two items that were indicated as the most prevalent strengths in the other tools, a 

family that is supportive of change (56.9%) and parents who are supportive of education (53.9%), 

having parents who were supportive of employment was also a commonly noted strength (52.1%). 

Similarly, the same two barriers that were the most frequently recorded in the other tools were also 

among the most prevalent barriers recorded in the Reentry Tool assessments: having prosocial 

peers (43.1%), and managing antisocial peers effectively (41.0%). One other item, having a history 

of substance use, was also among the most prevalent barriers recorded (42.2%). 

Subgroup comparisons. The average number of strengths and barriers, as well as 

combined scores on each tool were also compared across groups based on risk levels. The results 

of the analyses for the disposition are shown in Table 2.34. Across each of the domain and total 

scores, the averages were consistent with the expectations for each risk level. For example, the low 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

254 
 

 

 

 

risk group consistently had the highest average number of strengths, the high risk group 

consistently had the lowest number of strengths, and the average number of strengths for the 

moderate risk group always fell between the low- and high risk groups. In each test, all three groups 

were significantly different from each other. Similarly, the opposite pattern was observed for the 

average number of barriers, with the low risk group encountering the fewest barriers and the high 

risk group encountering the greatest number of barriers. Average combined scores were also 

consistent with expectations, and all three groups significantly differed from one another. 

Table 2.34. Differences in the Average Number of Strengths and Barriers by Domain and Risk 

Level for the Disposition Tool 

Variable Low Moderate High ANOVA 

x̅ x̅ x̅ F Strengths (sd) (sd) (sd) (df) 

Family and Living 3.31ab 2.27 1.80 58.40*** 

Arrangements (1.55) (1.70) (1.53) (2,902) 

Peers and Social Support 2.05ab .55c .12 358.83***^ 

Network (1.48) (.94) (.36) (2,467) 

3.69ab 2.41c 1.76 81.85***^ Education and Employment (1.83) (1.84) (1.53) (2,242) 

2.62ab 

Pro-Social Skills 1.29c .52 219.60***^ 

(1.35) (1.32) (.83) (2, 265) 

Substance Abuse, Mental 3.67ab 2.06c 1.18 122.26***^ 

Health, & Personality (2.41) (2.07) (1.40) (2, 322) 

2.28ab 1.39c .60 163.31***^ Values, Beliefs, & Attitudes (.96) (1.10) (.83) (2, 209) 

14.59ab 

Total 8.05c 5.03 156.54***^ 

(9.48) (7.23) (4.56) (2, 378) 

x̅ Barriers x̅ x̅ F 

(sd) (sd) (sd) (df) 

Family and Living .78ab 1.44c 2.10 38.73***^ 

Arrangements (1.19) (1.52) (1.64) (2, 201) 

Peers and Social Support 1.29ab 2.73c 3.29 204.51***^ 

Network (1.31) (1.16) (1.02) (2, 257) 

1.19ab 1.91c 2.53 30.45***^ Education and Employment (1.38) (1.79) (2.02) (2, 206) 

.77ab 

Pro-Social Skills 1.89c 2.72 151.80***^ 

(.99) (1.26) (1.19) (2, 204) 

Substance Abuse, Mental 1.01ab 2.38c 3.46 116.18***^ 

Health, & Personality (1.22) (1.72) (2.19) (2, 222) 

Values, Beliefs, & Attitudes .33ab 1.02c 1.98 149.69***^ 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

255 
 

 

 

 

 

 

 

 

 

 

 

 

Table 2.34. Differences in the Average Number of Strengths and Barriers by Domain and Risk 

Level for the Disposition Tool 

Variable Low Moderate High ANOVA 

(.66) (1.02) (.91) (2, 181) 

Total 4.44ab 

(4.31) 

9.19c 

(6.42) 

13.62 

(8.10) 

121.97***^ 

(2, 236) 

Combined x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

F 

(df) 

Family and Living 2.53ab .83c -.30 60.26***^ 

Arrangements (2.46) (2.91) (2.84) (2, 209) 

Peers and Social Support .76ab -2.18c -3.17 329.88***^ 

Network (2.59) (1.85) (1.15) (2, 339) 

Education and Employment 2.51ab 

(2.49) 

.49c 

(2.75) 

-.77 

(2.76) 

95.05*** 

(2, 900) 

Pro-Social Skills 1.85ab 

(2.15) 

-.59c 

(2.31) 

-2.19 

(1.71) 

237.79***^ 

(2, 235) 

Substance Abuse, Mental 2.67ab -.32c -2.28 175.19*** 

Health, & Personality (2.99) (3.02) (2.83) (2, 970) 

Values, Beliefs, & Attitudes 1.95ab 

(1.50) 

.37c 

(1.97) 

-1.37 

(1.59) 

188.13***^ 

(2,194) 

Total 10.15ab 

(10.19) 

-1.14c 

(8.69) 

-8.59 

(8.05) 

308.98***^ 

(2, 298) 

*** p < .001; ANOVA=Analysis of Variance 

^ Welch F statistic reported and comparisons based on Games-Howell test a Low risk is significantly different from moderate risk; b Low risk is significantly different from high risk; c 

Moderate risk is significantly different from high risk. 

Scores on the Residential Tool were also examined for differences based on youths’ risk

levels. The results were largely consistent with those of the Disposition Tool. As shown in Table 

2.35, average scores are consistent with expectations based on risk level. Low risk youths 

consistently had the highest number of strengths and the lowest number of barriers, while the 

opposite was true for high risk youths. Significant differences were observed across groups for 

each score, though scores for low and moderate risk youths were not significantly different with 

respect to the average number of strengths overall and in the Family and Living Arrangements, 

Peers and Social Support Network, and Education and Employment domains. Similarly, there was 

not a significant difference between low and moderate risk youths in the average number of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

256 
 

 

 

barriers on the Family and Living Arrangements domain, nor the combined scores in the same 

domain and the Education and Employment domain. 

Table 2.35 Differences in the Average Number of Strengths and Barriers by Risk Level 

Variable Low Moderate High ANOVA 

x̅ x̅ x̅ F Strengths (sd) (sd) (sd) (df) 

3.00b 2.70c 1.72 9.16*** Family and Living Arrangements (1.75) (1.67) (1.70) (2, 190) 

1.69b 1.06c .29 19.28***^ Peers and Social Support Network (1.47) (1.30) (.72) (2, 95) 

3.59b 3.16c 2.19 8.66*** Education and Employment (1.86) (1.76) (1.70) (2, 174) 

2.30ab 

Pro-Social Skills 1.51c .94 12.93***^ 

(1.53) (1.25) (.95) (2, 79) 

Substance Abuse, Mental Health, & 3.78ab 2.61c 1.69 12.27*** 

Personality (2.17) (1.92) (1.76) (2, 171) 

2.03ab 1.36c .83 14.16***^ Values, Beliefs, & Attitudes (1.08) (1.22) (.99) (2, 82) 

13.58b 

Total 10.89c 6.66 15.11***^ 

(8.68) (6.95) (5.67) (2, 92) 

x̅ Barriers x̅ x̅ F 

(sd) (sd) (sd) (df) 

1.08b 1.24c 2.17 9.77*** Family and Living Arrangements (1.42) (1.40) (1.63) (2, 190) 

1.62ab 2.58c 3.25 17.82***^ Peers and Social Support Network (1.43) (1.32) (1.02) (2, 73) 

1.19ab 2.02c 2.83 12.87***^ Education and Employment (1.26) (1.74) (1.87) (96) 

.97ab 

Pro-Social Skills 1.88c 2.48 20.03*** 

(1.02) (1.18) (1.13) (2, 180) 

Substance Abuse, Mental Health, & 1.48ab 2.92c 3.68 19.14*** 

Personality (1.34) (1.69) (1.49) (2, 180) 

.58ab 1.27c 1.86 19.59***^ Values, Beliefs, & Attitudes (.85) (1.15) (1.08) (2, 91) 

5.66ab 

Total 10.35c 14.35 27.42*** 

(4.85) (5.96) (6.58) (2, 204) 

x̅ Combined x̅ x̅ F 

(sd) (sd) (sd) (df) 

1.92b 1.46c -.45 11.33*** Family and Living Arrangements (2.89) (2.80) (3.04) (2, 190) 

.07ab -1.52c -2.96 21.18***^ Peers and Social Support Network (2.69) (2.51) (1.58) (2, 70) 

2.41b 1.14c -.64 15.24*** Education and Employment (2.18) (2.77) (2.85) (2, 174) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

257 
 

 

 

 

 

 

 

 

Table 2.35 Differences in the Average Number of Strengths and Barriers by Risk Level 

Variable Low Moderate High ANOVA 

Pro-Social Skills 1.33ab 

(2.35) 

-.37c 

(2.18) 

-1.54 

(1.90) 

20.88*** 

(2, 180) 

Substance Abuse, Mental Health, & 2.30ab -.32c -1.99 20.92*** 

Personality (2.81) (3.15) (2.75) (2, 171) 

Values, Beliefs, & Attitudes 1.45ab 

(1.80) 

.08c 

(2.26) 

-1.03 

(1.96) 

18.74***^ 

(2, 87) 

Total 7.92ab 

(9.20) 

.54c 

(9.04) 

-7.69 

(8.19) 

43.52*** 

(2, 204) 

*** p < .001; ANOVA=Analysis of Variance ^ Welch F reported, comparisons based on Games-Howell test a Low risk is significantly different from moderate risk; b Low risk is significantly different from high risk; c 

Moderate risk is significantly different from high risk. 

The comparative analyses were repeated for the subsample of youths who were assessed 

with the Reentry Tool. The pattern of results was generally consistent with the other assessment 

tools and is displayed in Table 2.36. Low risk youths consistently had the greatest number of 

strengths and the least number of barriers, and high risk youths had the fewest strengths and most 

barriers. The average combined scores reflected the same pattern. In each of the comparisons, the 

average scores among low risk youths were significantly different from those of moderate and 

high risk youths. In the Family and Living Arrangements domain, however, the average number 

of strengths and barriers and combined scores did not significantly differ between the moderate 

and high risk groups. 

Table 2.36. Differences in the Average Number of Strengths and Barriers by Domain and Risk 

Level for the Reentry Tool 

Low Moderate High ANOVA 

Strengths x̅

(sd) 

x̅

(sd) 

x̅

(sd) 

F 

(df) 

Family and Living 3.56ab 2.38 1.82 20.78***^ 

Arrangements (1.51) (1.81) (1.74) (2, 86) 

Peers and Social Support 2.53ab .93c .24 78.91***^ 

Network (1.44) (1.18) (.74) (2, 120) 

Education and Employment 4.64ab 

(1.67) 

3.65c 

(1.92) 

2.59 

(1.64) 

20.27*** 

(2, 232) 

Pro-Social Skills 3.21ab 1.82c .52 99.51***^ 

(1.14) (1.43) (.94) (2, 95) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

258 
 

 

 

 

 

 

Table 2.36. Differences in the Average Number of Strengths and Barriers by Domain and Risk 

Level for the Reentry Tool 

Low Moderate High ANOVA 

Substance Abuse, Mental 4.94ab 2.94c 1.74 62.26***^ 

Health, & Personality (1.77) (2.09) (1.39) (2, 92) 

2.69ab 1.61c .30 186.13***^ Values, Beliefs, & Attitudes (.66) (1.17) (.60) (2, 82) 

19.99ab 11.67c 6.94 78.71***^ Total (7.82) (7.86) (4.63) (2, 118) 

Barriers x̅ x̅ x̅ F 

(sd) (sd) (sd) (df) 

Family and Living .98ab 1.80 2.52 12.37***^ 

Arrangements (1.49) (1.77) (1.94) (2, 83) 

Peers and Social Support 1.10ab 2.68c 3.44 89.32***^ 

Network (1.20) (1.27) (.86) (2, 105) 

1.12ab 1.82c 3.29 19.18***^ Education and Employment (1.25) (1.60) (2.11) (2, 79) 

.54ab 1.80c 3.18 90.16***^ Pro-Social Skills (.97) (1.33) (1.36) (2, 84) 

Substance Abuse, Mental 1.12ab 2.73c 4.39 69.99***^ 

Health, & Personality (1.14) (1.63) (1.65) (2, 74) 

.18ab 1.20c 2.70 238.73***^ Values, Beliefs, & Attitudes (.47) (1.13) (.60) (2, 71) 

4.68ab 10.53c 18.65 85.01***^ Total (4.00) (6.34) (6.75) (2, 80) 

Combined x̅ x̅ x̅ F 

(sd) (sd) (sd) (df) 

Family and Living 2.58ab .58 -.70 18.37***^ 

Arrangements (2.87) (3.35) (3.41) (2, 85) 

Peers and Social Support 1.43ab -1.75c -3.21 94.37***^ 

Network (2.51) (2.34) (1.49) (2, 114) 

3.52ab 1.83c -.71 29.24***^ Education and Employment (2.30) (2.90) (3.26) (2, 82) 

2.68ab .03c -2.67 108.14***^ Pro-Social Skills (2.01) (2.62) (1.87) (2, 90) 

Substance Abuse, Mental 3.82ab .21c -2.65 88.28***^ 

Health, & Personality (2.54) (3.21) (2.62) (2, 82.15) 

2.51ab .41c -2.40 217.38***^ Values, Beliefs, & Attitudes (1.07) (2.24) (1.19) (2, 75) 

15.31ab 1.14c -11.71 124.56*** Total (9.24) (10.57) (8.89) (2, 248) 

*** p < .001; ANOVA=Analysis of Variance 

^ Welch F statistic reported and comparisons based on Games-Howell test a c Low risk is significantly different from moderate risk; b Low risk is significantly different from high risk; 

Moderate risk is significantly different from high risk. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

259 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Juvenile justice personnel perceptions. In an effort to gain insights as to how and why 

practitioners use the strengths and barriers items in the OYAS, we examined some of the juvenile 

justice personnel survey data described above. Specifically, when asked whether the OYAS 

provides useful information regarding non-criminogenic needs, 55.6% of respondents answered 

affirmatively and 28.9% indicated that it did not. The remaining 15.6% were unsure. Additionally, 

81.8% of respondents indicated that they incorporate needs into case plans, 13.6% indicated that 

they do not, and the remaining 4.5% were unsure. Using QDA Miner, we also extracted some 

practitioner quotes on the non-criminogenic needs included in the OYAS, which provide some 

added insights on the strength and barrier portions of the tools. 

Generally, the open-ended responses suggest that users are receptive to using the 

information to enhance treatment, but there are some potential obstacles. For example, one 

respondent indicated that the non-criminogenic needs captured in the OYAS can be used “to match

services.” Another respondent (presumably one in a supervisory role) reported that his/her strategy 

was to “encourage probation officers to use strengths and barriers and work with strengths to 

enhance kid[s].” These sentiments align with the intended use of these portions of the tool and 

strength-based case management, but other responses suggest that this view is not consistently 

adopted by practitioners. One practitioner noted that the tools “only captures non-criminogenic 

needs if probation officers do it and it’s not a requirement.” Another expressed the concern that 

“we do not do a great job of integrating these factors since we are just checking boxes on the

tool[s].” These responses may be indicative of uncertainty among practitioners regarding the value 

of the information gained from these items or how to use the information once they take the time 

to collect it. When asked about the use of OYAS to gather information on non-criminogenic needs 

another respondent stated, “much of the focus is on criminogenic needs so no opinion on it.”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

260 
 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of OYAS Strengths and Barriers. Contemporary research in juvenile justice 

and correctional intervention has given greater attention to the potential benefits of examining 

strengths and barriers in enhancing case management and interventions. In line with these 

perspectives, the most recent generations of risk and needs assessments have incorporated these 

factors. Despite these developments in practice, little empirical attention has been given to 

assessing how these aspects of assessment tools are used in the juvenile justice system and the 

extent to which practical understanding of cases can be improved through the information gained. 

The results of this substudy indicate that practitioners who are using the OYAS elect to 

capture strengths and barriers with some regularity. About three fourths of youths who are assessed 

with the Residential and Reentry Tools have at least one item recorded. Though less frequent, the 

majority of those assessed with the Disposition Tool (62%) also had at least one item noted as 

strength or barrier. Moreover, the results suggest possible differences in the presence of strengths 

and barriers for youths at different stages of the juvenile justice system. Those assessed with the 

Residential Tool tended to have more barriers, and those assessed with the Disposition and Reentry 

Tools had more strengths. A closer inspection of each domain revealed that there were more 

barriers than strengths related to peers and social support, regardless of the tool that was used to 

complete the assessment. This may be a consistent problem for all justice-involved youth. There 

were also consistently more strengths related to the family and Education and Employment 

domains, which may reflect areas to build upon in interventions across different points in the 

system. In terms of risk level, the number of strengths and barriers for each category of risk was 

significantly different and in the expected directions. This is not surprising given the design of the 

tool, but does provide some indication that the tool is working as designed in this respect. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

261 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Overall, this analysis suggests that one state using the OYAS (State 1) has started to move 

toward a more comprehensive assessment process through use of strengths and barriers indicators 

to inform case planning. The findings simultaneously reveal some room for expansion in the use 

of the strengths and barriers portions of the OYAS tools as well, which may prove to be one avenue 

for identifying more options for clients and better tailoring their treatment and supervision plans. 

In order to achieve meaningful gains in this regard, however, it may be necessary to consider 

several enhancements for the strengths and barriers sections. For example, through demonstrating 

their potential usefulness to juvenile justice personnel, setting clearer guidelines for their use, and 

offering more training opportunities on this element of the process. Like several of these use 

example studies, this also helps to illustrate some places where understanding more about risk and 

needs assessment in practice can inform us about implementation and training needs. 

Summary of Section II 

Section II of this report examined individual youth case data and youth interviews to help 

explore case-level JRNA usage and its potential impact on youths. These results alert us to the fact 

that while JRNA may be a substantial element to working effectively with youths, there is room 

for improvement in better integrating the information into every-day decision making in order to 

maximize its potential to affect the cases of justice-involved youths and, in turn, the juvenile justice 

system. For example, while approximately 60 percent of sampled youths were assessed to be 

moderate or high risk to recidivate, only around 35 percent of youth records showed exposure to 

treatment. The usage studies reported here also show evidence of varying outcomes concerning 

race and geography and risk level and also room for improvement concerning race, ethnicity, and 

OYAS validity. Further, the usage case studies discussed patterns in OYAS overrides, showed that 

having substance abuse and mental health indicators did predict receiving services in these areas, 

and that juvenile justice personnel are using the strengths and barriers items (which are optional) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

262 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

at a higher level than anticipated. In Section III, presented next, we further examine the youth case 

level and interview data in an effort to help answer questions on how JRNA impacts youths’

outcomes. 

Section III. Youths’ Outcomes and Juvenile Risk and Needs Assessment 

Better understanding implementation and usage of risk and needs assessment is essential 

in creating a more comprehensive evidence base for their usage in juvenile justice practice and 

fills a clear need in the research in this area. At the same time, linking insight on implementation 

and usage to outcomes for youths helps to connect this study to previous research on JRNA. That 

research has focused heavily on the relationship between risk and needs assessment and justicebased outcomes (e.g., new referrals to juvenile court or placements in correctional facilities). We 

therefore consider the risk and needs assessment process in relation to record-based recidivism as 

in previous studies while also using a small subsample of self-report interviews with youths to 

measure and analyze other developmentally-relevant outcomes such as engagement in school 

and/or work as well as shifts in attitudes that may be conducive to crime and delinquency. 

Risk Assessment, Juvenile Justice Decisions, and Youths’ Outcomes 

As highlighted earlier, structured risk and need assessments has become increasingly 

prominent over the last two decades of juvenile justice practice (e.g., Wachter, 2015). JRNA are 

frequently part of the implementation of evidence-based practices, having been touted as 

potentially improving case decision-making, youths’ outcomes, and recidivism rates (Nelson & 

Vincent, 2018). The results of risk and need assessments can be viewed as a “cornerstone of 

treatment,”73 with the potential to exert great impact on youths’ paths through the juvenile justice 

system process due to their impact on decisions about their cases—ranging from intake to case 

73 This quote is drawn from a response to the juvenile justice personnel web-based survey described in Section I. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

263 
 

 

 

 

 

 

disposition and placement to treatment. Likewise, since youths’ justice-system experiences will 

impact their personal and case outcomes, assessment can be viewed as the first step towards 

generating more positive results for justice-involved youths. Building on the early background of 

the report, the applied logic of this framework as it manifests in juvenile justice case processing is 

briefly outlined below and illustrated in Figure 3.1. This model uses recidivism as an endpoint 

given its primacy in the existing research, practice, and the current study. Nevertheless, the JRNA 

process shown in the figure below can inform juvenile justice response in ways that impact other 

important youths’ outcomes as well (National Research Council, 2013; Sullivan, 2019). 

Figure 3.1. Overview of Risk and Needs Assessment, Juvenile Justice Decisions, and Youths’

Outcomes 

Administer assessment to 

gather information on 

youth risk and needs

Reduced Risk of 

Recidivism

Intake through 

Adjudication Disposition, Placement and 

Treatment

Use to assign appropriate

disposition, supervision, and 

treatment requirements

Recidivism (Y/N)

Assessment scores inform 

areas for placement, 

intervention, supervision level

Criminogenic needs 

targeted effectively

This logic model builds on the notion that risk and needs assessments should be 

administered to each youth to inform key decision points in the juvenile justice system. This starts 

at intake and detention decisions (Mears, 2012; Steinhart, 2006) and proceeds through later stages 

of the system (Baglivio, Wolff, Jackowski, & Greenwald, 2017). In addition to detailing 

background and personal characteristics (which could be useful in highlighting specific 

responsivity factors), results from each assessment should be used to gauge risk of re-offense and 

existing areas of criminogenic need (Duwe & Rocque, 2017; Latessa, Listwan, & Koetzle, 2014). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

264 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This information should then be used to match youths to appropriate services and levels of 

supervision, as well as inform dispositional and case-management decisions (Andrews & Bonta, 

2010; Viera, Skilling, & Peterson-Badali, 2009). Finally, assuming that each youth completes his 

or her assigned supervision requirements or is otherwise placed appropriately, we can expect to 

see reductions in recidivism and more positive developmental outcomes than if a systematic 

assessment was not used to inform the juvenile justice decision-making process. 

Despite the importance of ensuring that risk and need assessments are used completely and 

properly in decision-making, few studies have examined the use of risk assessment in practice in 

the juvenile justice system (Peterson-Badali, Skilling, & Haqanee, 2015; Vincent, Paiva-Salisbury, 

Cook, Guy, & Perrault, 2012). Existing studies have focused heavily on the utility and predictive 

validity of risk and need assessment – rather than examining the process in its entirety (Singh et 

al., 2014). While there is evidence that these tools are indeed useful when predicting recidivism, 

the observed effect sizes suggest considerable variation across systems and studies (Schwalbe, 

2007), and still reflect a good deal of prediction error. In the absence of strong implementation 

practices, risk assessments may be completed but not used to guide decision making (Shook & 

Saari, 2007) or be utilized correctly. In these cases, agencies and youths will fail to fully capitalize 

on the possible benefits of JRNA (Vincent et al., 2012; Vincent et al., 2016). 

Analysis of Case Decisions and Outcomes Based on JRNA 

To this point, the report has characterized the implementation process across three states—

and multiple agencies within those states—in part to set a foundation for other analyses of 

assessment use and youths’ outcomes and to assess some isolated examples of its use. The data 

discussed above are largely based or influenced on the perceptions of those in the field, however. 

Here we conduct a series of mediation models to better understand the degree to which the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

265 
 

 

 

 

 

 

 

 

information gathered and synthesized from risk and needs assessment affects juvenile justice case 

decision-making in the sites from which we gathered data. In turn, the results of those decisions 

theoretically should contribute to better outcomes among youths. To assess this, the data gathered 

in this study were used to formally test the impact of risk and need assessment results on juvenilejustice decision-making and subsequent recidivism outcomes. In particular, we use the data and 

sample described above to assess the statistical models outlined in Figure 3.2, which is a statistical 

distillation of the broader processes depicted in Figure 3.1 above. 

Figure 3.2. Statistical Model for Risk Assessment, Decision-Making, and Youths’ Outcomes

OYAS Risk Level

OYAS Risk Score

OYAS Domain Scores

Case Disposition/Placement

Type of Treatment Received

New Adjudication

(Yes/No)

We first consider whether some key disposition types mediate the relationship between the 

results of the OYAS risk assessment (i.e., overall risk level and score) and youths’ recidivism (i.e., 

new juvenile court adjudication). Secondarily, we determine whether the various domain scores in 

the OYAS are related to receipt of relevant treatment types (e.g., substance use, cognitive 

behavioral therapy). In turn, we identify whether that treatment affects youths’ recidivism. In doing 

so, we attempt to fill in the implicit gap in between risk assessment results and recidivism outcomes 

by modeling some relevant juvenile justice processes as potential mediators. This helps make a 

more precise determination of how JRNA usage works in practice. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

266 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We estimate these models using controlled causal mediation analyses with bootstrapped 

standard errors to assess the assumed risk and need assessment-decision-outcome relationship 

(Hayes, 2013; Vanderweele, 2015). These analyses were carried out using the paramed program 

created by Valeri & Vanderweele (2013) and available in Stata, as well as Mplus software for path 

modeling due to the fact that some of the domain and overall OYAS score variables are continuous. 

Both the mediation and outcome models are handled as multivariate logistic regressions due to the 

binary nature of the relevant variables in those parts of the model (e.g., received probation, any 

new adjudication). 

This strategy was used because it allowed us to effectively model indirect relationships 

between risk and recidivism while controlling for potential unmeasured effects on the risk and 

disposition/treatment relationships that could otherwise confound results. These modeling 

strategies also allow us to handle interaction assumptions in mediation analysis that cannot be dealt 

with in standard multivariate regression processes and other approaches to mediation (c.f., Baron 

& Kenny, 1986; Vanderweele, 2015). This interaction term is tested in each case in order to 

determine whether the effect of the mediator on the outcome is stronger or weaker depending on 

the initial risk level. The control variables used in each equation vary somewhat depending on the 

models and are described as we report the results for each. However, the core controls include: 

age, sex (1=Female), race (1=Non-White), violent focal offense (1=Yes), and a set of dummy 

variables for state site (Reference=State 1). 

Data from all three states were used in the disposition analyses and States 1 and 2 

comprised the analytic sample for the treatment mediation models. Relevant details for the 

independent variables (i.e., OYAS assessment results) and mediators (i.e., case 

disposition/placement and treatment indicators) are shown in Tables 2.8, 2.9, and 2.10 in the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

267 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

section on the comprehensive assessment sample above. Recidivism is the main dependent 

variable in this study. It was scored dichotomously, with “0” indicating no recidivism and “1”

indicating recidivism. Youths were considered to have recidivated if they had an adjudication on 

a new charge following their assessment for the focal charge. Overall, 26 percent of youths in the 

analytic sample for the disposition analysis, which mainly comprised cases assessed with 

disposition and diversion tools (~n=4,805), had a new adjudication following the disposition of 

their case or release from a facility. Roughly 24 percent of those in the smaller sample for the 

treatment mediation analysis (n=3,584) had a record of a new adjudication.74 

Disposition type captures the outcome of the youths’ disposition hearing for their focal

cases. “Disposition type” refers to control-oriented strategies. In the original study data, disposition 

types included secure commitment, non-secure residential placement, house arrest or electronic 

monitoring, intensive probation, standard probation, diversion, penalties (e.g., restitution, fines), 

community service, and dismissal or termination. In comparison, “treatment type” refers to 

therapeutic-oriented dispositions that include referrals to services (e.g. counseling, cognitive 

behavioral therapy, substance abuse treatment). In both cases, these serve as potential mediator 

variables. 

For the purposes of this analysis, disposition type and treatment type were collapsed into 

binary variables. The first dichotomous mediator compared youths who received the most severe 

disposition in our dataset, commitment to a state facility (1), with youths who received a less 

serious disposition (0). The second dichotomous mediator compared youths who received the most 

74 Reductions in the sample sizes were primarily due to the scope of information available in individual case records 

and the ability to effectively link across the assessment, case disposition and treatment referral, and recidivism data. 

The largest reductions to the analytic samples were due to available offense type (16.8%), treatment (55.5%), 

disposition (19.6%) data. Patterns of missingness did differ somewhat across OYAS risk levels with relatively greater 

proportions missing among low and high risk youths relative to moderate risk cases (χ2

(3) = 48.5, p<.001). 

Nevertheless, these differences were relatively modest in strength (V = 0.09). Still, it is possible that generalizability 

of some findings may be weighted toward moderate risk cases. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

268 
 

 

 

 

 

 

 

 

 

 

 

 

common disposition in our dataset, standard probation, or a less serious disposition (e.g. diversion, 

penalties, community service, or dismissal/termination) (1), with youths who received a more 

serious disposition (e.g., state commitment, residential placement, house arrest or electronic 

monitoring, or intensive probation) (0). Finally, the third dichotomous disposition type mediator 

compared youths who received intensive or standard probation (1) to youths who received a less 

serious case outcome (e.g., diversion, dismissal/transfer, penalties, or community service). 

Six binary categories of treatment referral were examined in this analysis: cognitive 

behavioral, anger/aggression, education/employment, family services, mental health services, and 

substance abuse treatment. Programs were classified into these categories based on the content or 

topic of the services they provided or the type of behavior/issue they claimed to target. A score of 

“0” indicated that the youth did not receive the type of treatment specified, while “1” meant s/he

did receive the type of treatment specified. For the mediation models estimating treatment effects, 

the Juvenile Justice History domain score was also included as a control. 

Results 

Disposition. Table 3.1 reports the results of the mediation process models for risk level, 

case dispositions, and new adjudication. In all estimated models age, gender, race/ethnicity, violent 

offense, and state category were included as control variables. In total, 6 models were estimated 

to examine the impact of disposition on the relationship between OYAS risk level and new 

adjudication. Two models were estimated for each disposition of interest. The first in each set 

compared moderate and high risk youth to low risk youth, while the second compared moderate 

youths to either high or low risk youths only. Although we do not devote much attention to 

controls, there are significant state effects in some models that tend to reflect differential patterns 

of disposition decisions and recidivism in particular analyses (e.g., State 3 is less likely and State 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

269 
 

 

 

 

 

 

 

 

 

 

2 more likely than State 1 to use state commitment). The violent focal offense indicator is generally 

statistically significant as well, which suggests that case characteristics play a significant role in 

understanding recidivism as well as in disposition decisions made during the juvenile justice 

process. In both cases, the results illustrate the importance of conditioning on these variables to 

obtain unbiased estimates of key relationships among risk assessment, juvenile justice decisions, 

and recidivism. 

Disposition: State Commitment. Results of the mediation process model for risk level and 

recidivism via state commitment are presented in Table 3.1 and Figure 3.3 below. We converted 

the estimates from the top panels of the tables into an Odds Ratio (OR) scale to ease interpretation 

and discussion of key results. Results indicated that compared to low risk youths, moderate and 

high risk youths had significantly higher chances of commitment to a state facility (OR = 1.39, p 

< 0.01). This finding suggests that the risk assessment information is associated with state 

commitment placement decisions in an expected way and one that is aligned with best practices in 

JRNA. Moderate and high risk youths also had significantly higher chances of recidivism (OR = 

1.39, p < 0.01) compared to low risk youths committed to a state facility. The relationship between 

state commitment and new adjudication suggests that youths with secure placement have one-third 

as high odds of recidivism as those who were not (OR = 0.33, p < 0.01). 

Table 3.1. Results of Mediation Process Model for State Commitment 

Mod/High vs. Low Risk and High vs. Mod Risk and 

State Commitment State Commitment 

b b 

(95% CI) (95% CI) 

New Adjudication on...

N=4750 N=2887 

OYAS Risk Level 0.40** 0.40** 

(0.25 - 0.54) (0.21 - 0.60) 

State Commitment -1.12** -1.22** 

(-1.32 - -0.92) (-1.46 - -0.98) 

Age -0.11** -0.09** 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

270 
 

 

Table 3.1. Results of Mediation Process Model for State Commitment 

Mod/High vs. Low Risk and High vs. Mod Risk and 

State Commitment State Commitment 

b b 

(95% CI) (95% CI) 

(-0.15 - -0.07) (-0.14 - -0.04) 

Female -0.22** -0.19 

(-0.38 - -0.06) (-0.40 - 0.02) 

Non-White 0.24** 0.23** 

(0.09 - 0.38) (0.05 - 0.41) 

Violent Offense -0.31** -0.37** 

(-0.47 - -0.16) (-0.55 - -0.19) 

State 2 -0.02 0.02 

(-0.19 - 0.15) (-0.18 - 0.23) 

State 3 0.18 0.21 

(0.001 - 0.36) (-0.01 - 0.44) 

Disposition on... 

OYAS Risk Level 0.33** 0.32** 

(0.17 - 0.48) (0.12 - 0.52) 

Age 0.45** 0.45** 

(0.40 - 0.51) (0.38 - 0.52) 

Female -0.92** -1.21** 

(-1.13 - -0.72) (-1.50 - -0.93) 

Non-White 0.31** 0.29** 

(0.15 - 0.46) (0.10 - 0.48) 

Violent Offense 0.61** 0.56** 

(0.46 - 0.76) (0.38 - 0.75) 

State 2 0.88** 0.78** 

(0.72 - 1.04) (0.57 - 0.98) 

State 3 -2.27** -1.74** 

(-2.60 - -1.95) (-2.09 - -1.39) 

Risk Level to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped CI) (95% Bootstrapped CI) 

Controlled Direct Effect 1.49** 1.50** 

(1.27 - 1.75) (1.22 - 1.81) 

Natural Direct Effect 1.49** 1.50** 

(1.27 - 1.74) (1.22 - 1.80) 

Natural Indirect Effect 0.97* 0.96* 

(Thru Disposition) (0.95 - 0.98) (0.92 - 0.98) 

Marginal Total Effect 1.44** 1.43** 

(1.23 - 1.70) (1.16 - 1.71) 

Notes: *p < .05; **p < .01; Mod=Moderate; CI=Confidence Interval 

The indirect effect of state commitment on the risk level and recidivism relationship was 

also significant (OR = 0.97, p < 0.05), indicating a small (~3%) decrease in odds of recidivism for 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

271 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

moderate/high risk youths based on commitment to a state facility—relative to those youths in the 

group that were not placed in secure facilities. These results indicate that an indirect pathway 

between risk level and recidivism does exist via state commitment. It does not negate the 

relationship between risk level and recidivism, as a significant direct relationship was still found 

between risk level and new adjudication for moderate and high risk youths (OR = 1.49, p < 0.01). 

It does, however, illustrate that appropriate dispositional placement based on risk level can affect 

the likelihood of recidivism. When this model was re-estimated comparing only moderate and high 

risk youths (i.e. excluding low risk youths), the results were very similar among all of the key 

comparisons made above (see Table 3.1 above).75 This suggests that appropriately matching higher 

risk youths to intensive services—even if it results in custody—does have some comparative 

advantages in reducing recidivism. This is likely conditioned by the quality of the programming 

in facilities, however (Lipsey, 1999), which was not measured in this study. 

Figure 3.3. Mediation Process Model for Risk Level, State Commitment, and New Adjudication 

1.39** 

Natural Indirect Effect 

0.97* 

OYAS Risk Level

1=Moderate/High Risk

State Commitment

1=Yes 

New Adjudication

1=Yes 

0.33** 

1.49** 

Notes: Estimates are in Odds-Ratio scale. Model includes control variables in all equations; **p<0.01 and *p<0.05 

with either t-test or bootstrapped 95% confidence interval. 

75 Additionally, models with risk level and recidivism via state commitment were also estimated using new 

commitment as the dependent variable instead of new adjudication. Results from this model were mainly consistent 

with the models used for the final analysis and indirect effects were non-significant. Only youths from States 1 and 3 

were used for this supplementary analysis. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

272 
 

 

 

 

 

 

 

Disposition: Standard Probation vs. Less Severe Sanctions. The mediation process model 

for risk level and recidivism via standard probation or less severe sanctions (compared to intensive 

probation or more severe sanctions) is shown in Table 3.2. The estimates suggest that, compared 

to low risk youths, moderate and high risk youths had significantly lower odds, by about 46 

percent, of assignment to standard probation or less severe sanctions (OR = 0.54, p < 0.01). 

Moderate and high risk youths also had significantly higher chances of recidivism (OR = 1.45, p 

< 0.01) compared to low risk youths. There was not, however, a statistically significant relationship 

between the probation or less severe sanctions indicator and recidivism. The results of the 

disposition model demonstrate that youths with moderate or high risk levels had a significantly 

lower likelihood of being assigned to sanctions like standard probation. 

Table 3.2. Results of Mediation Process Model for Standard Probation vs. Less Severe Sanctions 

Mod/High vs. Low Risk and High vs. Mod Risk and 

Probation or Less Severe Probation or Less Severe 

b b 

(95% CI) (95% CI) 

N=4836 N=2929 

New Adjudication on...

Risk Level x Less Severe 

Sanctions 

OYAS Risk Level 

Probation or Less Severe 

Sanctions 

Age 

Female 

Non-White 

Violent Offense 

State 2 

State 3 

Disposition on...

-

0.37** 

(0.22 - 0.51) 

0.12 

(-0.04 - 0.29) 

-0.15** 

(-0.19 - -0.11) 

-0.13 

(-0.29 - 0.025) 

0.19* 

(0.04 - 0.33) 

-0.39** 

(-0.54 - -0.25) 

-0.15 

(-0.31 - 0.03) 

0.33** 

(0.16 - 0.51) 

0.59** 

(0.21 - 0.97) 

0.07 

(-0.20 - 0.34) 

0.02 

(-0.21 - 0.25) 

-0.14* 

(-0.19 - -0.09) 

-0.08 

(-0.28 - 0.13) 

0.20* 

(0.02 - 0.38) 

-0.45** 

(-0.63 - -0.27) 

-0.09 

(-0.30 - 0.13) 

0.39** 

(0.17 - 0.62) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

273 
 

 

 

 

 

 

Table 3.2. Results of Mediation Process Model for Standard Probation vs. Less Severe Sanctions 

Mod/High vs. Low Risk and High vs. Mod Risk and 

Probation or Less Severe Probation or Less Severe 

b b 

(95% CI) (95% CI) 

OYAS Risk Level 

Age 

Female 

Non-White 

Violent Offense 

State 2 

State 3 

Risk Level to Recidivism 

Controlled Direct Effect 

Natural Direct Effect 

Natural Indirect Effect 

(Thru Disposition) 

Marginal Total Effect 

-0.62** 

(-0.76 - -0.47) 

-0.34** 

(-0.39 - -0.29) 

0.90** 

(0.73 - 1.07) 

-0.39** 

(-0.54 - -0.25) 

-0.47** 

(-0.61 - -.0.33) 

-1.61** 

(-1.77 - -1.45) 

1.44** 

(1.23 - 1.66) 

Odds Ratio 

(95% Bootstrapped CI) 

1.44** 

(1.24 - 1.65) 

1.44** 

(1.24 - 1.65) 

0.98 

(0.97 - 1.00) 

1.42** 

(1.23 - 1.63) 

-0.78** 

(-0.98 - -0.58) 

-0.35** 

(-0.41 - -0.29) 

0.94** 

(0.72 - 1.17) 

-0.43** 

(-0.61 - -0.25) 

-0.23* 

(-0.41 - -0.05) 

-1.70** 

(-1.91 - -1.50) 

0.98** 

(0.73 - 1.23) 

Odds Ratio 

(95% Bootstrapped CI) 

1.93** 

(1.47 - 2.66) 

1.63** 

(1.31 - 2.06) 

0.90** 

(0.83 - 0.95) 

1.46** 

(1.18 - 1.80) 

Notes: *p < .05; **p < .01; Mod=Moderate; CI=Confidence Interval 

The natural indirect effect of standard probation or less severe sanctions on the risk level 

and recidivism relationship was small and not significant (OR = 0.98) for moderate- or high risk 

youths as compared to low risk youths. Taken alone, these results indicate a lack of an indirect 

pathway between risk level and recidivism via standard probation or less severe sanctions. 

However, when the model was restricted to compare high vs. moderate risk youths only (i.e., low 

risk youths were excluded), a significant interaction between risk level and assignment to standard 

probation or less severe sanctions was found (OR = 1.80, p < 0.01). This suggests that the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

274 
 

 

 

 

 

 

relationship between probation and recidivism, which is small but positive (OR = 1.02), is stronger 

for high, as opposed to moderate, risk youths. 

Figure 3.4. Mediation Process Model for Risk Level, Standard Probation or Less Severe 

Sanctions, and New Adjudication 

-0.46** 

Natural Indirect Effect=0.90** 

Risk Level x Prob. or Less Severe Sanctions=1.80** 

OYAS Risk Level

1= High Risk

0=Moderate Risk

Disposition Assigned

1=Standard Probation or 

Less Severe Sanctions 

New Adjudication

1=Yes 

1.02 

1.07 

Notes: Estimates are in Odds-Ratio scale. Model includes control variables in all equations; **p<0.01 and *p<0.05 

with either t-test or bootstrapped 95% confidence interval 

The results of this modified model are presented in Figure 3.4. They suggest that the 

relationship of risk level and adjudication is mediated by disposition (at least among high risk 

youths assigned to standard probation or lower severity sanctions). When this interaction term was 

included in the model it rendered both the direct relationship between risk level and adjudication, 

and the impact of assignment to probation or less severe sanctions on adjudication, non-significant. 

A significant natural indirect effect via standard probation or a lower severity disposition for risk 

level and recidivism also emerged. This suggests potential for a recidivism reduction in high risk 

youths in terms of their new adjudication, but it is not materializing here due to the fact that the 

small relationship between probation and recidivism is stronger in the high risk group as compared 

to moderate risk youths. This is reflected in the controlled direct effect Odds Ratio of 1.93, which 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

275 
 

 

 

 

 

 

 

suggests a substantial relationship between high risk youths and recidivism that held when all of 

these relationships, processes, and controls were taken into account. 

Disposition: Intensive or Standard Probation. Estimations of the mediation process model 

for risk level and recidivism via standard or intensive probation (as compared to less severe 

sanctions) are shown in Table 3.3. Results indicated that compared to low risk youths, moderateand high risk youths had significantly lower chances of assignment to intensive or standard 

probation (OR=0.46, p<0.01). Youths on probation also tended to have greater odds of new 

adjudication compared to youths with a less serious disposition (OR=1.90, p<0.01). Neither risk 

level nor the interaction of risk level and intensive/standard probation were found to be have a 

significant effect on adjudication. The interaction effect was substantial enough to retain it in the 

model. The controlled direct effect of 1.84 means that high risk youths have a significantly higher 

likelihood of new recidivism when accounting for the probation disposition versus a lesser sanction 

as well as the other variables included in the model. 

Table 3.3. Results of Mediation Process Model for Intensive or Standard Probation 

Mod/High vs. Low Risk Mod vs. Low Risk and 

and Probation Probation 

b b 

(95% CI) (95% CI) 

N=2270 N=2044 

New Adjudication on... 

Risk Level x Probation 0.39 -0.50* 

OYAS Risk Level 

(-0.01 - 0.78) 

0.22 

(-0.92 - -0.09) 

-0.06 

Probation (Intensive or Standard) 

Age 

Female 

(-0.08 - 0.53) 

0.64** 

(0.33 - 0.94) 

-0.08* 

(-0.13 - -0.02) 

-0.10 

(-0.38 - 0.25) 

1.14** 

(0.85 - 1.42) 

-0.09** 

(-0.15 - -0.03) 

-0.11 

Non-White 

(-0.31 - 0.10) 

0.18 

(-0.33 - 0.10) 

0.20 

Violent Offense 

(-0.03 - 0.39) 

-0.25* 

(-0.02 - 0.43) 

-0.27* 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

276 
 

 

 

 

Table 3.3. Results of Mediation Process Model for Intensive or Standard Probation 

Mod/High vs. Low Risk Mod vs. Low Risk and 

and Probation Probation 

b b 

(95% CI) (95% CI) 

(-0.47 - -0.03) (-0.51 - -0.03) 

State 2 -0.39* -0.30 

(-0.67 - -0.11) (-0.59 - 0.00) 

State 3 0.11 0.11 

(-0.13 - 0.34) (-0.14 - 0.36) 

Disposition on... 

OYAS Risk Level -0.77** 0.89** 

(-0.95 - -0.59) (0.71 - 1.08) 

Age 0.11** 0.12** 

(0.06 - 0.16) (0.07 - 0.17) 

Female -0.55** -0.53** 

(-0.74 - -0.37) (-0.73 - -0.33) 

Non-White 0.14 0.14** 

(-0.05 - 0.34) (-0.07 - 0.35) 

Violent Offense 0.05 0.04** 

(-0.14 - 0.25) (-0.18 - 0.25) 

State 2 -0.52** -0.41** 

(-0.77 - -0.27) (-0.67 - -0.15) 

State 3 -0.93** -0.78** 

(-1.14 - -0.71) (-1.01 - -0.55) 

Risk Level to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped CI) (95% Bootstrapped CI) 

Controlled Direct Effect 1.84** 0.57** 

(1.46 - 2.37) (0.43 - 0.76) 

Natural Direct Effect 1.71** 0.68** 

(1.40 - 2.14) (0.53 - 0.86) 

Natural Indirect Effect 0.84** 1.14** 

(Thru Disposition) (0.80 - 0.88) (1.08 - 1.23) 

Marginal Total Effect 1.44** 0.77* 

(1.20 - 1.81) (0.62 - 0.97) 

Notes: *p < .05; **p < .01; Mod=Moderate; CI=Confidence Interval 

The model exploring the effect of intensive/standard probation on the risk level and 

recidivism relationship was then estimated comparing moderate and low risk youths (i.e., high risk 

cases were excluded). The results of this model are presented in Figure 3.5 below. A moderate, as 

opposed to low, risk level was found to significantly increase chances of youths’ assignment to

intensive or standard probation as opposed to a less serious sanction or case dismissal (OR=2.43, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

277 
 

 

 

 

 

 

 

 

 

p<0.01). Moderate risk youths on intensive or standard probation also had increased odds of 

recidivism. Furthermore, a significant interaction between risk level and assignment to intensive 

or standard probation was detected (OR = -0.61, p < 0.05), which decreased the odds of receiving 

a new adjudication. The natural indirect effect via standard or intensive probation also remained 

significant (OR = 1.14, p < 0.01), which reflects that moderate risk youths on probation will have 

greater likelihood of recidivism than those who are placed on less serious sanctions. This could 

potentially be a supervision effect associated with the probation process relative to other less 

formal sanctions. Risk level was not significantly associated with recidivism in the initial model 

estimated for this set of relationships, but the controlled direct effect of 0.57 indicates that 

moderate risk youths on probation or intensive probation (vs. something less formal) have lower 

odds of recidivism than low risk youths on probation versus lower sanctions. This further indicates 

that aligning risk and needs with particular case dispositions and placements can impact 

recidivism. In this case, moderate risk youths on probation perform better than low risk youths on 

probation, even though this is not necessarily their optimal placement. 

Figure 3.5. Mediation Process Model for Risk Level, Standard Probation or Less Severe Sanctions, 

and New Adjudication 

2.43** 

Natural Indirect Effect=1.14** 

OYAS Risk Level

1= Moderate Risk

0=Low Risk

Disposition Assigned

1=Intensive or Standard 

Probation 

New Adjudication

1=Yes 

3.13** 

0.94 

Risk Level x Intensive or Standard Probation=-0.61* 

Notes: Estimates are in Odds-Ratio scale. Model includes control variables in all equations; **p<0.01 and *p<0.05 

with either t-test or bootstrapped 95% confidence interval. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

278 
 

 

 

 

 

 

 

 

 

 

 

Treatment and Services Referral. While risk and needs assessment is meant to inform 

disposition decisions, it is also potentially valuable as a means of providing insight for treatment 

matching (e.g., Vieira et al., 2009). Given that expectation, we also estimated parallel multivariate 

mediation models to better understand potential linkages between risk and needs assessment, 

treatment, and youths’ recidivism. The estimates and interpretation of results is somewhat different 

than in the prior section due to the continuous nature of the variables used as key independent 

variables (i.e., risk scores as opposed to levels). Indirect estimates were also estimated to formally 

assess the impact of key independent variables on recidivism as they potentially operate through 

treatment and service referral. These referrals are made in different contexts and the OYAS 

assessment domains differ somewhat for risk/needs in the disposition and residential tools. This 

could lead to possible differential relationships in the models so we also divide the analyses into 

subgroups for community-based and residential samples.76 We present and interpret partially 

standardized estimates given the mixed levels of measurement involved in the predictor, mediator, 

and outcome variables (Muthѐn & Muthѐn, 1998-2018). 

Treatment and Services: Cognitive Behavioral (CBT). The results of the treatment 

mediation models examining the relationships between criminogenic needs, receiving CBT, and 

recidivism are shown in Table 3.4. The first column includes cases that were referred to community 

–based placements or diverted. The second column refers to cases in residential facilities. The 

Values, Beliefs, and Attitudes and Prosocial Skills domain scores were included as predictors of 

receiving CBT and recidivism, as higher scores in these domains suggest that treatment targeting 

cognition may be appropriate. The Juvenile Justice History domain score was included in the 

treatment models as a control variable as well. 

76 We also estimated the community-based disposition models without those cases that received informal resolutions 

(e.g., various types of dismissal) and reached similar conclusions to those presented here. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

279 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The results suggest that referral to CBT had did not have a significant relationship with 

recidivism. The juvenile justice history score had a statistically significant and moderate-sized 

relationship with recidivism in the community-based sample suggesting that there was a slight 

increase in the likelihood of recidivism for a one unit increase on that juvenile justice history 

score.

77 Two of the OYAS variables, Prosocial Skills and Juvenile Justice History, did have a 

relationship to whether or not youths were referred to CBT in the Residential sample. Those with 

higher prosocial skills deficits were slightly less likely to receive CBT (b = -0.10), but those with 

more extensive juvenile justice histories had a greater likelihood of referral to CBT (b = 0.10). 

The indirect effects estimates for the Values, Beliefs, and Attitudes and Prosocial Skills domain 

and the Prosocial Skills domain did not yield any significant effects on recidivism in either of these 

subgroups.

78 This is evident in the fact that the Bootstrapped Confidence Interval (CI) values 

overlap with 1.00 in each of the four cells capturing the “Natural Indirect Effect.” 

Table 3.4. Results of Mediation Process Model for Cognitive Behavioral Treatment 

Community-Based Residential Cases 

Cases (N = 1236) 

(N =1153) 

b b 

(se) (se) 

New Adjudication on...

Cognitive Behavioral Treatment 0.05 (0.09) 0.67 (0.39) 

Values, Beliefs, and Attitudes Domain Score 0.004 (0.04) 0.02 (0.04) 

Prosocial Skills Domain Score 0.04 (0.03) 0.06 (0.04) 

Juvenile Justice History Domain Score 0.20 (0.03)*** -0.06 (0.04) 

77 We sometimes see a negative estimate for “violent offense” in the models presented above and in this section. As

this is mainly a control in the current study, we intend to investigate this pattern of relationships in further research, 

There are three preliminary explanations based on the current analyses. First, this may be due in part to the collection 

of variables included in the models that may create some patterns of suppression and redundancy. Second, in this 

study “violent offense” as a categorical designation can include different levels (e.g., simple assault and aggravated

assault) and may be of mixed types across states. Therefore, it is not always the case that these are serious violent 

offenses. Finally, we utilize official records of recidivism in the analysis of records and therefore supervision effects 

may emerge during the follow up period where youths with such offenses may be monitored more closely—potentially 

creating artifactual patterns in its relationships with recidivism. 

78 Elements of the presentation of results differ slightly across the disposition and treatment/services models due to 

the nature of the variables involved. We present all relevant coefficient estimates and standard errors based on the 

method of estimation and software used. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

280 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.4. Results of Mediation Process Model for Cognitive Behavioral Treatment 

Community-Based Residential Cases 

Cases (N = 1236) 

(N =1153) 

b b 

(se) (se) 

Age -0.10 (0.03)*** -0.15 (0.04)*** 

Female -0.08 (0.09) 0.17 (0.72) 

Non-White -0.04 (0.08) -0.09 (0.12) 

Violent Offense -0.17 (0.08)* -0.46 (0.12)*** 

State 2 -0.26 (0.12)* 0.59 (0.89) 

Cognitive Behavioral Treatment on...

Values, Beliefs, and Attitudes Domain Score 0.06 (0.05) 0.01 (0.03) 

Prosocial Skills Domain Score 0.02 (0.05) -0.10 (0.05)* 

Juvenile Justice History Domain Score 0.03 (0.05) 0.10 (0.04)** 

Age -0.08 (0.04)* -0.03 (0.04) 

Female -0.26 (0.14) -0.42 (0.46) 

Non-White 0.06 (0.11) 0.19 (0.11) 

Violent Offense 0.05 (0.12) 0.19 (0.11) 

State 2 -0.72 (0.12)*** -0.74 (0.55)*** 

Values, Beliefs, and Attitudes to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped (95% Bootstrapped 

CI) CI) 

Natural Indirect Effect 1.004 1.01 

(0.99 – 1.02) (0.97 – 1.06) 

Natural Direct Effect 1.01 1.06 

(0.85 – 1.20) (0.92 – 1.26) 

Total Effect 1.02 1.07 

(0.86 – 1.21) (0.93 – 1.26) 

Prosocial Skills to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped (95% Bootstrapped 

CI) CI) 

Natural Indirect Effect 1.001 0.96 

(0.99 – 1.02) (0.85 – 1.00) 

Natural Direct Effect 1.09 1.03 

(0.95 – 1.27) (0.85 – 1.22) 

Total Effect 1.09 0.99 

(0.95 – 1.27) (0.82 – 1.14) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized Estimate; se=Standard 

Error of the Estimate 

Treatment and Services: Anger/Aggression. The mediation process model between 

criminogenic needs, receiving treatment for anger/aggression, and recidivism are displayed in 

Table 3.5. Two key domains were examined in relation to treatment for anger/aggression and 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

281 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

recidivism: Values, Beliefs, and Attitudes and Prosocial Skills. In the community sample, the 

results indicate that youths who received treatment for anger/aggression were significantly more 

likely to have a new adjudication (b = 0.40) relative to youths not referred to this type of 

intervention. The Juvenile Justice History domain score was significantly related to the likelihood 

of a new adjudication as well (b = 0.15). Neither of the relevant domain scores was significantly 

related to new adjudications after controlling for the other covariates included in this model. 

The only covariate that had a significant relationship with likelihood of anger/aggression 

treatment referral was violent focal offense (b = 0.47), suggesting that those youths were more 

likely to be referred to than those with a nonviolent or other focal offense. Similarly, there were 

no significant indirect effect estimates in the Community-placed youths’ model. Only two 

significant findings emerged among the key variables of interest in the Residential Case model; 

both related to the likelihood of anger/aggression treatment referral. Those with higher prosocial 

skills deficits were slightly less likely to receive such referrals (b = -0.17), but those with more 

extensive juvenile justice histories had a greater likelihood of referral to Anger/Aggression 

treatment (b = 0.18). A similar violent offense to treatment referral relationship (b = 0.32) emerged 

in the residential case model. None of the indirect effects was statistically significant, suggesting 

that the risk/needs information for these domains was not affecting recidivism through referral to 

Anger or Aggression treatment. 

Table 3.5. Results of Mediation Process Model for Anger/Aggression Treatment 

Community-Based Residential Cases 

Cases (N = 1236) 

(N =1153) 

b b 

(se) (se) 

New Adjudication on...

Anger/Aggression Treatment 0.40 (0.10)*** 0.16 (0.29) 

Values, Beliefs, and Attitudes Domain Score 0.06 (0.05) 0.02 (0.03) 

Prosocial Skills Domain Score -0.004 (0.05) 0.02 (0.05) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

282 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.5. Results of Mediation Process Model for Anger/Aggression Treatment 

Community-Based Residential Cases 

Cases (N = 1236) 

(N =1153) 

b b 

(se) (se) 

Juvenile Justice History Domain Score 0.15 (0.05)*** -0.02 (0.05) 

Age -0.11 (0.04)*** -0.17 (0.04)*** 

Female -0.21 (0.12) -0.04 (0.76) 

Non-White 0.05 (0.11) 0.00 (0.11) 

Violent Offense -0.36 (0.12)*** -0.38 (0.12)*** 

State 2 -0.35 (0.12)*** 0.11 (0.11) 

Anger/Aggression Treatment on...

Values, Beliefs, and Attitudes Domain Score -0.13 (0.08) 0.05 (0.04) 

Prosocial Skills Domain Score 0.11 (0.08) -0.17 (0.07)** 

Juvenile Justice History Domain Score 0.11 (0.07) 0.18 (0.06)** 

Age 0.02 (0.07) 0.02 (0.04) 

Female 0.30 (0.19) -0.40 (0.86) 

Non-White -0.21 (0.18) 0.25 (0.16) 

Violent Offense 0.47 (0.16)** 0.32 (0.16)* 

State 2 0.14 (0.19) -0.06 (0.06) 

Values, Beliefs, and Attitudes to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped (95% Bootstrapped 

CI) CI) 

Natural Indirect Effect 0.99 1.001 

(0.93 – 1.00) (0.99 – 1.02) 

Natural Direct Effect 1.02 1.07 

(0.86 – 1.22) (0.93 – 1.25) 

Total Effect 1.02 1.07 

(0.86 – 1.21) (0.93 – 1.26) 

Prosocial Skills to Recidivism Odds Ratio Odds Ratio 

(95% Bootstrapped (95% Bootstrapped 

CI) CI) 

Natural Indirect Effect 1.003 0.99 

(0.99 – 1.05) (0.96 – 1.02) 

Natural Direct Effect 1.09 0.99 

(0.94 – 1.27) (0.82 – 1.14) 

Total Effect 1.09 0.99 

(0.95 – 1.27) (0.81 – 1.14) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized 

Estimate; se=Standard Error of the Estimate 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

283 
 

 

 

 

 

 

 

 

 

 

 

 

 

Treatment and Services: Education/Employment. The mediation process model 

examining the relationships between criminogenic needs, participating in services targeting 

education or employment, and recidivism is shown in Table 3.6. The Education and Employment 

domain score was included as a key independent variable in these models, as these need areas are 

expected to be the most impacted by these types of interventions. The results indicate that receiving 

services aimed at education and employment is not a significant predictor of recidivism in either 

of the two models. The Education and Employment need/risk score is a statistically significant 

predictor of recidivism in both models. Focusing on the “Natural Direct Effect” in the bottom panel 

of Table 3.6, for instance, shows roughly 20 and 25 percent greater odds of new adjudication in 

the community and residential samples, respectively.79 

Controlling for the other variables in the model--the Juvenile Justice History domain score 

is significantly related to new adjudication in the community-based sample (b = 0.21), but not the 

sample of residential youths (b = 0.02). The Education and Employment score did not predict 

referral to those services in either of these models, however. The only significant coefficient among 

the key variables in that aspect of the model was the Juvenile Justice History domain score for the 

community-based cases (b = 0.14). This suggests that the likelihood that youths receive those 

services increases with each additional point on that domain score. Neither of the indirect effects 

was statistically significant, suggesting that the needs information for education and employment 

was not affecting recidivism through treatment referral (or not). 

79 The estimate for residential cases is more tenuous when judged by a statistical significance criteria, but is non trivial 

and roughly comparable to that observed in the other sample. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

284 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.6. Results of Mediation Process Model for Cognitive Education and Vocational Services 

Community-Based Cases Residential Cases 

(N =1155) (N = 1243) 

b b 

(se) (se) 

Education and Vocational Services 0.17 (0.11) 0.03 (0.33) 

Education and Employment Domain Score 0.10 (0.04)* 0.10 (0.05) 

Juvenile Justice History Domain Score 0.21 (0.04)*** 0.02 (0.04) 

Age -0.10 (0.03)*** -0.16 (0.04)*** 

Female -0.08 (0.09) -0.11 (0.42) 

Non-White -0.09 (0.10) 0.03 (0.41) 

Violent Offense -0.14 (0.09) -0.34 (0.13)** 

State 2 

Education and Vocational Services on...

Education and Employment Domain Score -0.005 (0.08) -0.02 (0.08) 

Juvenile Justice History Domain Score 0.14 (0.06)* -0.01 (0.08) 

Age -0.08 (0.05) -0.03 (0.07) 

Female 0.14 (0.19) 0.05 (1.04) 

Non-White 0.28 (0.18) 0.40 (0.65) 

Violent Offense -0.17 (0.17) 0.16 (0.23) 

State 2 0.48 (0.18)** 1.27 (0.25)*** 

Education and Employment Score to Odds Ratio Odds Ratio 

Recidivism (95% Bootstrapped CI) (95% 

Bootstrapped CI) 

Natural Indirect Effect 1.00 1.00 

(0.96 – 1.02) (0.96 – 1.03) 

Natural Direct Effect 1.20 1.25 

(1.05 – 1.41) (1.02 – 1.65) 

Total Effect 1.20 1.25 

(1.05 – 1.41) (1.02 – 1.64) 

New Adjudication on...

-0.42 (0.12)*** 0.03 (0.34) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized Estimate; se=Standard 

Error of the Estimate 

Treatment and Services: Family. The mediation process model examining the effects of 

four criminogenic needs areas on recidivism through treatment consisting of family services is 

shown in Table 3.7. The Family and Living Arrangements domain is included in the model as it 

most clearly maps onto the services of interest in this analysis. For residential cases, on average, 

youths who participated in family services had a significantly higher likelihood of new 

adjudication relative to youths who did not participate in these interventions (b = 0.56). That 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

285 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

relationship was not statically significant in the community-based sample (b = 0.01), though. For 

the community-based sample both OYAS domain scores were also found to increase the likelihood 

of a new adjudication: Family and Living Arrangements (b = 0.10) and Juvenile Justice History (b 

= 0.24). Interestingly, scores in the Family and Living Arrangements domain did not significantly 

influence whether or not the youths would receive family services in either of the samples, 

suggesting that youths may not be matched to these services based on assessment information. 

Coupling this with the findings for the “New Adjudication” equation suggests some potential

untreated need among these samples (especially the community-based youths). 

Table 3.7. Results of Mediation Process Model for Family Treatment and Services 

Community- Residential Cases 

Based Cases (N = 1247) 

(N =1302) 

b b 

(se) (se) 

New Adjudication on...

Family Treatment and Services 0.01 (0.06) 0.56 (0.11)*** 

Family and Living Situation Domain 0.10 (0.03)** -0.05 (0.06) 

Score 

Juvenile Justice History Domain Score 0.24 (0.04)*** 0.07 (0.04) 

Age -0.15 (0.03)*** -0.04 (0.05) 

Female -0.04 (0.04) -0.14 (0.29) 

Non-White 0.01 (0.04) -0.07 (0.13) 

Violent Offense -0.08 (0.04)* -0.28 (0.13)* 

State 2 -0.13 (0.04)*** -0.16 (0.17) 

Family Treatment and Services on...

Family and Living Situation Domain 0.02 (0.04) 0.12 (0.06) 

Score 

Juvenile Justice History Domain Score 0.02 (0.05) -0.10 (0.05)* 

Age -0.02 (0.03) -0.23 (0.06)*** 

Female 0.09 (0.12) 0.02 (0.38) 

Non-White 0.28 (0.11)** 0.22 (0.17) 

Violent Offense -0.16 (0.10) -0.11 (0.16) 

State 2 -0.35 (0.10)** -0.74 (0.55)*** 

Family and Living Situation Score to Recidivism Odds Ratio Odds Ratio 

(95% (95% 

Bootstrapped CI) Bootstrapped CI) 

Natural Indirect Effect 1.00 1.01 

(0.99 – 1.01) (1.00 – 1.14) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

286 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Natural Direct Effect 1.15 1.04 

(1.05 – 1.28) (0.80 – 1.32) 

Total Effect 1.15 1.04 

(1.05 – 1.28) (0.83 – 1.35) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized Estimate; se=Standard 

Error of the Estimate 

Treatment and Services: Mental Health. Treatment for mental health was the most 

commonly recorded form of treatment in the sample with 20 percent of youths referred to 

interventions in this category. The model estimates for the effects of three domains on recidivism 

through mental health treatment are shown in Table 3.8. The estimates suggest that this type of 

treatment is not associated with the likelihood of a new adjudication in either of the two 

subsamples. The Substance Use, Mental Health, and Personality domain was significantly and 

positively related to the likelihood of a new adjudication (b = 0.06 in both samples), however. 

That score was not significantly related to the odds of receiving this type of intervention in the 

model for either subgroup. As in some other models, this suggests that unaddressed need might be 

contributing to elevated levels of recidivism. The Juvenile Justice History (b = 0.11) and violent 

offense (b = 0.43) variables were significantly related to the likelihood of receiving this type of 

treatment, seemingly playing a more prominent role in that allocation than the information from 

the specific need domain. Neither of the indirect effects was statistically significant, suggesting 

that the risk domain information was not affecting recidivism through treatment referral (or not). 

Table 3.8. Results of Mediation Process Model for Mental Health Treatment and Services 

Community- Residential Cases 

Based Cases (N = 1248) 

(N =1156) 

b b 

(se) (se) 

New Adjudication on...

MH Treatment and Services 0.11 (0.07) 0.22 (0.13) 

Substance Abuse, Mental Health, and 0.06 (0.03)* 0.06 (0.03)* 

Personality Domain Score 

Juvenile Justice History Domain Score 0.19 (0.03)*** -0.01 (0.04) 

Age -0.11 (0.03)*** -0.16 (0.03)*** 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

287 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.8. Results of Mediation Process Model for Mental Health Treatment and Services 

Community- Residential Cases 

Based Cases (N = 1248) 

(N =1156) 

b b 

(se) (se) 

Female -0.10 (0.08) 0.06 (0.21) 

Non-White -0.04 (0.09) 0.04 (0.10) 

Violent Offense -0.15 (0.08) -0.40 (0.10)*** 

State 2 -0.31 (0.10)** -0.01 (0.13) 

MH Treatment and Services on...

Substance Abuse, Mental Health, and 0.02 (0.03) -0.05 (0.03) 

Personality Domain Score 

Juvenile Justice History Domain Score -0.01 (0.04) 0.11 (0.03)** 

Age -0.08 (0.04)* 0.01 (0.04) 

Female 0.03 (0.12) -0.69 (0.31)* 

Non-White -0.11 (0.11) 0.10 (0.12) 

Violent Offense -0.05 (0.11) 0.43 (0.11)*** 

State 2 0.34 (0.11)* 0.45 (0.24) 

Substance Abuse, Mental Health and Personality Odds Ratio Odds Ratio 

Score to Recidivism (95% (95% 

Bootstrapped CI) Bootstrapped CI) 

Natural Indirect Effect 1.003 0.99 

(0.99 – 1.02) (0.97 – 1.003) 

Natural Direct Effect 1.14 1.11 

(1.02 – 1.29) (1.00 – 1.25) 

Total Effect 1.15 1.04 

(1.03 – 1.29) (1.00 – 1.24) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized Estimate; se=Standard 

Error of the Estimate 

Treatment and Services: Substance Abuse. Interventions that target substance abuse were 

also commonly received by the youths in the sample. Table 3.9 provides the estimates for the 

mediation process model investigating the relationships between scores in Substance Use, Mental 

Health, and Personality, substance abuse treatment, and recidivism. We also consider the relative 

role of Juvenile Justice History. The estimates suggest that this type of treatment is not associated 

with the likelihood of a new adjudication in either of the two subsamples. As seen in the previous 

model, the Substance Use, Mental Health, and Personality score was significantly and positively 

related to the likelihood of a new adjudication (b = 0.06 in both samples). That score was also 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

288 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

significantly related to the likelihood of referral to substance abuse treatment (b = 0.08) in the 

community sample, but not among residential youth cases, suggesting that it is syncing up with 

decision-making in the former. This is one of the only places in the analyses where a relationship 

between the domain of interest and the treatment/service type has emerged. As seen in the previous 

analyses, there are no significant mediating effects from this OYAS domain to treatment and then 

recidivism. 

The OYAS Juvenile Justice History score has a significant, positive relationship with new 

adjudication (b = 0.19) in the community-based sample and is positively related to substance abuse 

treatment referral in the Residential case sample (b =0.16). This model therefore suggests some 

degree of untreated need that may be affecting likelihood of recidivism as well as mixed results on 

the degree to which youth receive treatment referrals in line with their substance use needs. 

Table 3.9. Results of Mediation Process Model for Substance Abuse Treatment and Services 

Community- Residential Cases 

Based Cases (N = 1248) 

(N =1156) 

b b 

(se) (se) 

New Adjudication on...

MH Treatment and Services 0.02 (0.08) 0.33 (0.13)* 

Substance Abuse, Mental Health, and 0.06 (0.03)* 0.06 (0.03)* 

Personality Domain Score 

Juvenile Justice History Domain Score 0.19 (0.03)*** -0.04 (0.04) 

Age -0.12 (0.03)*** -0.17 (0.04)*** 

Female -0.10 (0.08) 0.03 (0.23) 

Non-White -0.05 (0.09) 0.05 (0.10) 

Violent Offense -0.15 (0.09) -0.42 (0.11)*** 

State 2 -0.26 (0.11)* -0.11 (0.15) 

MH Treatment and Services on...

Substance Abuse, Mental Health, and 0.08 (0.03)* -0.04 (0.03) 

Personality Domain Score 

Juvenile Justice History Domain Score 0.02 (0.04) 0.16 (0.03)** 

Age 0.07 (0.03) 0.04 (0.04) 

Female -0.18 (0.12) -0.38 (0.37) 

Non-White -0.01 (0.11) 0.03 (0.13) 

Violent Offense -0.29 (0.10)** 0.34 (0.13)** 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

289 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.9. Results of Mediation Process Model for Substance Abuse Treatment and Services 

Community- Residential Cases 

Based Cases (N = 1248) 

(N =1156) 

b b 

State 2 

Substance Abuse, Mental Health and Personality 

Score to Recidivism 

Natural Indirect Effect 

(se) 

-0.59 (0.12)*** 

Odds Ratio 

(95% 

Bootstrapped CI) 

1.001 

(se) 

0.58 (0.24)* 

Odds Ratio 

(95% 

Bootstrapped CI) 

0.99 

Natural Direct Effect 

(0.99 – 1.01) 

1.15 

(0.97 – 1.00) 

1.10 

Total Effect 

(1.03 – 1.30) 

1.15 

(0.99 – 1.25) 

1.10 

(1.03 – 1.29) (0.99 – 1.24) 

Notes: *p < .05; **p < .01; ***p<.001 CI=Confidence Interval; b=Partially Standardized Estimate; se=Standard 

Error of the Estimate 

Summary of Case Decisions and Outcomes Based on JRNA 

While they require caveats about available detail, follow-up time, and potential artifacts in 

case record analysis, the results of these mediation analyses suggest that the relationships between 

risk and needs assessment, juvenile justice decisions, and recidivism are quite complex when they 

are broken down into the various pieces that operate within the implementation, usage, and 

outcome model on which the theorized process is predicated. The models focused on disposition 

show that risk level and juvenile court dispositions have an impact on youths’ recidivism 

(measured via new recorded adjudication), generally in ways anticipated by prior research. The 

addition of controlled mediation analysis (see Vanderweele, 2015) offers insight as to the match 

between disposition and risk level and how that may relate to later recidivism. This is true across 

various contrasts of OYAS risk. Here, we found, for instance, that when used appropriately, state 

commitment may have a slight negative effect on the likelihood of recidivism for higher risk 

youths. Similarly, there was a significant contrast in the fit of probation supervision (vs. less severe 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

290 
 

 

 

 

 

 

 

 

 

 

 

case dispositions) where moderate risk youths had increased likelihood of recidivism when placed 

on probation or intensive probation as opposed to a less formal supervision or case dismissal. 

In the second group of mediation models we examined processes that are expected to occur 

in which risk assessment information is used to strategically match youths to treatment, which in 

turn is intended to reduce recidivism. The results of the analysis generally suggest that matching 

criminogenic needs to treatment does not consistently occur in the juvenile justice systems as much 

as one would anticipate based on the underlying logic of JRNA. This is based on the absence of 

any indirect effects from OYAS subdomain scores through treatment/services to recidivism as well 

as the limited findings of relationships between the various domain score covariates and referral 

indicators (i.e., a single instance across several models and two subgroups). In fact, offense type, 

juvenile justice history, and the state dummy variable tended to have more consistent relationships 

with treatment referral than did the OYAS domain score that would seemingly be the best fit to 

that decision-making. Additionally, the relationships between participating in treatment and 

recidivism were largely null or—in three cases—in the opposite direction of what would be 

expected if the referrals made to treatment programming had their intended effect (i.e., engaging 

in treatment increased the likelihood of recidivism). This may be a marker of the (in)effectiveness 

of the programming to which youths—even those who have a particular area of need—are being 

exposed, availability of programming to fit with need, as well as some inefficiencies in how youth 

are assigned to particular types of treatments. This more formal modeling helps to better illustrate 

the manner in which different elements of measured risk and needs might impact eventual case 

outcomes. Lastly, in this section of the report, we also explore some additional analyses of the 

follow-up interview with youths below. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

291 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Analysis of Youth Follow-Up Interview Data 

Building on the background in the Section II above, we analyze data for the 131 youth 

follow-up cases in this section. As context, recall that these youths did not appear to differ much 

from the randomly-selected record sample with exception of the fact that there are relatively few 

youths who were placed in a state facility for their focal offense. Given the study priority of 

understanding the linkage between risk assessment, decisions about youths’ cases, and case

outcomes, we utilize that as an organizing variable in presenting much of the follow-up analysis 

and results. To reiterate, roughly 44 percent of these youths were originally identified as low risk 

with the OYAS, followed by 37 percent moderate risk, and 19 percent high risk. 

Table 3.10 reports descriptive statistics and risk group comparisons for the six scales that 

were developed based on interview items that are meant to capture some developmental outcomes 

of interest following study youths’ contact with the juvenile justice system (e.g., development of 

prosocial thinking, attitudes toward juvenile justice, attitudes toward family). Each of these is 

coded on an averaged scale from “1” to “5” with higher values reflecting stronger agreement. With 

the exception of procriminal attitudes, each reflects a prosocial response pattern to the relevant 

items in each summary measure. The F statistics shown in the far-right column of the table are all 

non-significant, indicating that the three groups of youths were similar statistically in their 

responses to these questions. 

Table 3.10. Descriptive Statistics for Youth Follow-Up – Self-Report Scales by Risk Level 

Variable Low 

(N = 57) 

x̅(sd) 

Moderate 

(N = 48) 

x̅(sd) 

High 

(N = 25) 

x̅(sd) F (df) 

Attitudes toward Family 4.74 (0.41) 4.65 (0.67) 4.70 (0.33) 0.412 (2, 127) 

Attitudes toward Peers 4.10 (0.62) 4.06 (0.96) 4.05 (0.84) 0.050 (2, 110) 

Attitudes-Juvenile Justice 

System/Services 4.0 (1.13) 4.3 (1.14) 3.9 (1.16) 1.38 (2, 127) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

292 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.10. Descriptive Statistics for Youth Follow-Up – Self-Report Scales by Risk Level 

Variable Low 

(N = 57) 

x̅(sd) 

Moderate 

(N = 48) 

x̅(sd) 

High 

(N = 25) 

x̅(sd) F (df) 

Attitudes-Work and School 4.2 (0.70) 4.3 (0.81) 4.3 (0.87) 0.806 (2, 106) 

TCU Adolescent Thinking 4.2 (0.82) 4.3 (0.79) 4.2 (0.90) 0.884 (2, 127) 

Procriminal Attitudes 2.30 (0.69) 2.19 (0.66) 2.55 (1.02) 1.85(2, 127) 

The responses noted in Table 3.10 are generally quite positive on average. The mean 

attitudes toward family are each above 4.6 (out of 5.0) for the three risk groups reflecting positive 

perceptions of and ties and support from family. The standard deviation values are generally 

smaller than for other scales suggesting a great deal of consensus in those responses. The groups 

are very similar, and report positive relationships with peers as well (4.1 out of 5.0). There is more 

variability between and within the groups on the attitudes toward juvenile justice and treatment 

score, which ranges between 3.9 and 4.3, but the responses still generally reflect favorable 

attitudes. Attitudes toward work or school are similarly positive across the three groups (4.2 to 4.4, 

on average), reflecting some level of investment and positive experience among the interviewees. 

The TCU Adolescent Thinking scale reflects youths’ confidence and efficacy in selfmanagement, problem solving, and avoidance of situations that can lead to substance use or other 

delinquency. The moderate risk youths reported the highest mean score, but also had considerably 

more variation than the other two groups (4.5, sd = 1.36). The low and high risk youths reported 

mean scores of 4.3 and 4.2 respectively; their standard deviations on the score were roughly 0.80 

on the five-point scale. Lastly, the procrime attitudes scale measured the degree to which youths 

identified with “criminal thinking.” Higher scores are more consistent with such attitudes and

thinking patterns. The mean scores ranged from 2.2 to 2.5 among the three OYAS risk groups. 

The moderate group had the lowest average score and the high risk group reported the average 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

293 
 

 

 

 

 

 

 

 

 

 

 

 

score of about 2.5. In contrast to some of the other results, although not exceptionally high, the 

youths still did tend to identify with some procrime and delinquency sentiments during the followup interview. 

To further assess relationships between OYAS subscales and the follow-up measures and 

potentially link the earlier risk assessment process that occurred in the juvenile justice system with 

these interview response scales, the results of bivariate Pearson correlation analyses are presented 

in Table 3.11. We use a more liberal p < .10 designation to reject the null hypothesis (ρ = 0) in

this case due to the low sample sizes in many of the domain scores, which is the result of the fact 

that not all OYAS tools include these domains. The table reveals few clear patterns except that 

overall risk is associated with higher scores on the TCU adolescent scale (r = 0.16), which runs 

counter to directional expectations. Similarly, the family and living arrangements risk score is 

positively correlated with positive attitudes toward work and school (r = 0.24). Additionally, the 

prosocial skills risk score is positively and significantly associated with the procrime attitudes 

measure (r = 0.21), which is also positively correlated with the earlier assessment of educational 

and employment risk (r = 0.18). Overall, these relationships suggest that while there is some 

carryover in particular attitudes and risks from the earlier assessment in the juvenile justice 

process, they are not all that highly tied together in youths’ responses on these particular measures, 

which are generally markers of positive attitudes at follow-up. This may in part reflect some natural 

“aging out” among these youths (see, e.g., Farrington, 1986), but also may be indicative of some

impact of their juvenile justice experiences. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

294 
 

 

 

 

 

 

 

 

 

 

 

Table 3.11. Summary of Correlations (r) Between Initial OYAS Scores and Follow-Up Scales 

Follow-Up Interview Scale 

Initial OYAS Risk Attitudes Attitudes Attitudes Attitudes TCU Procrime 

Toward Toward Toward Toward Adolescent Attitudes 

Family Peers Juvenile Work/ Thinking 

Justice School 

Overall Risk Score .058 -.105 -.095 .090 .157^ .120 

(N = 130) 

Peer and Social Support .082 .005 -.039 .150 .045 .149 

(N = 88) 

Pro-Social Skills -.087 -.030 .030 .004 .048 .210* 

(N = 88) 

Juvenile Justice History .015 -.082 -.011 .083 -.024 .019 

(N = 91) 

Family and Living -.052 .024 .077 .242* .152 .153 

Arrangements (N = 91) 

Education and -.034 -.221* .059 .080 .022 .178^ 

Employment (N = 88) 

Substance Abuse/Mental 

Health/Personality -.064 -.118 .031 .073 .114 .050 

(N = 88) 

Values/Beliefs/Attitudes .040 .101 -.041 .122 .089 .128 

(N = 88) 

Notes: All values are Pearson’s r correlations; ***p<.001, **p<.01, *p<.05, ^p<.10

Building on the analyses of youths’ outcomes above, interviewees were asked about receipt 

of treatment during their juvenile justice experience or in the follow-up period as well as their 

current engagement with work or school. Those results are summarized in Figure 3.6. The vast 

majority of youths across all three OYAS risk groups (>82%) were engaged in work or school 

when they were interviewed by UC researchers. At 86 percent, the high risk group was actually 

most likely to respond “yes” to being enrolled in school or working at the time of the interview.80 

80 Given the possibility of a supervision effect, we considered the degree to which this variable was associated with 

whether or not the youth was still on community supervision. The relationship was small in size and not statistically 

significant (χ2

(2)= 0.63, Cramer’s V = 0.07). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

295 
 

 

 

 

 

 

 

Figure 3.6. Youth Self-Reported Treatment and Current Work/School Involvement 

100.0 

90.0 86.4 

82.4 

31.6 33.3 

9.5 

4.8 

42.9 

82.2 

43.8 

47.8 

9.1 

17.4 

26.1 

48.0 

69.2 

20.0 20.0 

10.0 

0.0 

10.0 

20.0 

30.0 

40.0 

50.0 

60.0 

70.0 

80.0 

SR Family 

Treatment? 

SR 

Counseling 

Treatment? 

% Yes Responses 

SR Work or SR Any SR Substance SR Cognitive 

School at Treatment or Abuse Behavioral 

Follow Up? Programming? Treatment? Treatment? 

Low Risk (44%) Moderate Risk (37%) High Risk (19%) 

The self-reported treatment indicators suggest a clear contrast across the OYAS risk levels 

for “any treatment,” “substance abuse treatment,” and “CBT.” Forty-eight percent of interviewees 

from the OYAS high risk group responded that they received some treatment in the juvenile justice 

system or subsequently compared to 44 percent of the moderate and 32 percent of low risk group. 

A similar trend emerged with the substance abuse treatment question as well. High risk youths 

were more than twice as likely (20%) to report receiving CBT relative to those in the low (9.5%) 

and moderate (9.1%) groups. These results and relevant measures of association (Cramer’s V = 

0.13 to 0.27) suggest that there was some trend toward higher risk youths’ likelihood of reporting 

treatment. Still, the overall sample sizes warrant some caution in treating these trends too 

compellingly. The “family treatment” indicator shows that moderate and high risk youths report 

greater likelihood of receipt of that service at 17 and 20 percent respectively. Only five percent of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

296 
 

 

 

 

 

 

 

 

 

the OYAS low risk group reported some past or current family treatment at follow-up. The final 

panel of Figure 3.6 shows the comparative prevalence of involvement in “general counseling.” In

this case the reporting trend is reversed such that 42 percent of low risk youths state that they 

received that service compared to 26 and 10 percent of moderate and high risk youths, respectively. 

Chi-Square tests again found that these relationships were not statistically significant, but they 

may be indicative of patterns in treatment receipt that differ across risk groups with youths in 

moderate and high groups receiving more intensive services. 

Self-reported behavioral outcomes, including substance use, and updated contact with the 

juvenile or criminal justice system were assessed across risk level. We also report involvement in 

treatment and work and school during the follow-up period. Figure 3.7 reports the relationship 

between youths’ self-reported substance use and their initial risk levels. The alcohol use measure 

suggests that low and moderate risk youths are actually more likely to use alcohol during the 

follow-up, 61 and 52 percent used, respectively, compared to the high risk group (44%). This 

difference was not significant in a Chi-Square test; although, the Cramer’s V measure of 

association statistic was 0.13 suggesting a nontrivial relationship. The relationship differed for 

marijuana and harder drugs (e.g., cocaine, heroin) where high risk youths tended to have greater 

prevalence of self-reported use at 60 and 24 percent, respectively. Over half of youths in both the 

low and moderate risk groups used marijuana and thirteen to fifteen percent used harder drugs. 

Neither of these relationships were statistically significant in a Chi-Square test. The hard drug use 

by risk relationship had a Cramer’s V statistic of 0.12, however. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

297 
 

 

 

 

 

 

 

 

 

 

Figure 3.7. Youth Self-Reported Substance Use by Initial Risk Level 

100.0 

90.0 

80.0 

70.0 60.7 

53.6 

12.7 

52.1 54.2 

14.6 

44.0 

60.0 

24.0 

0.0 

10.0 

20.0 

30.0 

40.0 

50.0 

60.0 

Hard Drug Use During 

Follow Up? 

% Yes Responses 

High Risk (19%) 

Used Alcohol During Follow Used Marijuana During 

Up? Follow Up? 

Low Risk (44%) Moderate Risk (37%) 

The analysis of self-reported justice contact during the follow-up period followed as similar 

process. Figure 3.8 summarizes the results of comparisons of self-reported justice contact across 

the three risk levels identified in each youth’s assessment. Notably, the prevalence of juvenile

justice responses tended to follow the case record sample in terms of the overall magnitude 

reported earlier in the report (e.g., new adjudication in 26% of cases in the comprehensive 

assessment sample vs. 33% youth follow up interviewees). Overall, the levels of self-reported 

police stops (52% to 55%) and adjudications (32% to 34%) were quite similar across the three risk 

groups. The Chi-Square tests of association with those groups were not statistically significant. 

The other measures suggest that the higher risk groups reported somewhat greater levels of 

juvenile and criminal justice system contact during the follow-up window. The moderate (37.5%) 

and high risk (36.0%) groups had somewhat greater prevalence of self-reported arrest during the 

follow-up period while the low risk group was several percentage points lower in its prevalence 

(28.6%). The Chi-Square statistic was not statistically significant. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

298 
 

 

 

 

 

 

 

 

 

Figure 3.8. Youth Self-Reported Juvenile Justice Contact during Follow-Up by Initial Risk 

100.0 

90.0 

80.0 

70.0 

58.3 

28.6 

33.9 

30.4 

21.4 

37.5 

33.3 

41.7 

37.5 

52.0 

36.0 

32.0 

40.0 

36.0 

0.0 

10.0 

20.0 

30.0 

40.0 

50.0 

New SR Probation 

During Follow Up 

New SR Placement 

During Follow Up? 

% Yes Responses 60.0 55.4 

New SR Police Stop New SR Arrest New SR 

During Follow Up? During Follow Up? Adjudication During 

Follow Up? 

Low Risk (44%) Moderate Risk (37%) High Risk (19%) 

There was greater, but still not a statistically significant separation between the groups in 

self-reported probation referrals and new detention or residential placement during the follow-up 

period. In each case, the moderate and high risk youths groups reported higher levels of new system 

contact than low risk youths. For example, 38 and 36 percent of moderate and high risk youths 

reported a new placement during follow-up compared to the 21 percent of low risk youths who 

reported a new justice placement. The Cramer’s V statistics for probation and new placement are 

0.11 and 0.16 respectively, suggesting still limited, but non-trivial relationships between risk level 

and these outcomes. Notably, the Chi-Square statistic approaches statistical significance when the 

moderate and high risk groups are contrasted with low risk (χ2

(1) = 3.64). This suggests some 

differentiation in this outcome comparison—especially when coupled with the relatively low 

sample and cell sizes. 

The interview data convey relatively low statistical power and limited variability in some 

measures in the follow-up interview sample. Nevertheless, we utilized some of the findings 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

299 
 

 

 

 

 

 

 

 

 

 

 

 

 

described above to tentatively assess a key relationship that can help lend insight into how youths 

with different initial risk levels may have ended up responding fairly similarly on many of the 

interview questions. In turn, we considered what that implies for their other outcomes, justice 

contact in this case. Keeping in mind that there was a differential between risk and self-reported 

treatment receipt in most of the comparisons reported in the table above (e.g., CBT, any treatment), 

we began with the premise that higher risk youths were more apt to receive treatment. Mean 

comparisons were made between youths that reported receiving treatment and those that did not 

on each the six scales. All comparisons suggested that youths who reported treatment had higher 

scores on the positive outcome scales and a lower mean score on the procrime attitudes score. The 

difference between the treatment (x̅= 4.52, sd = 1.26) and non-treatment groups (x̅= 4.19, sd = 

0.89) on the TCU Adolescent Thinking Scale approached statistical significance (p = .07) and the 

effect size was modest in size (Hedges’ g = 0.38). 

Although nonsignificant, there is also a moderate relationship between the procrime 

attitudes and self-reported treatment engagement (Hedges’ g = 0.21). Given the relationships 

between procrime attitudes and OYAS risk scores described above, we in turn consider the degree 

to which that distinction in procrime attitudes might be associated with concurrent differences in 

justice and substance use related outcomes. In particular, the figure below summarizes the 

differences between those who self-reported arrest and substance use on procrime attitudes. With 

the exception of the comparison between those who report alcohol use and those who did not, each 

of the results is statistically significant using a t test. The Hedges’ g effect sizes range from 0.26 

(self-reported alcohol use) to 0.54 (self-reported arrest). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

300 
 

 

 

 

 

 

 

 

Figure 3.9. Interviewee Self-Reported Procrime Attitudes, Arrest, and Substance Use 

2.2 2.2 2.1 2.3 2.2 2.5 2.6 2.5 

0.0 

1.0 

2.0 

5.0 

SR Marijuana 

Use? 

SR Harder Drug 

Use? 

SR Alcohol Use? 

Average Procrime Attitudes 4.0 

3.0 2.6 2.5 

SR Arrest SR New 

Follow-Up? Placement 

Follow-Up? 

No Yes 

Summary of Analysis of Youth Follow-Up Interview Data 

In general, many of youths’ self-reported outcomes were positive when looking at the 

various attitudinal scales measured during the follow-up interviews. Most trended dramatically 

toward prosocial outcomes. Generally, those responses were not significantly related to youths’ 

scores from the initial assessment process or to the risk levels that emerged from the initial OYAS 

process. There are some important exceptions in that procriminal attitudes at follow-up was 

positively related to some of the earlier risk domain scores. This suggests in part that some change 

is occurring for these youths during the juvenile justice process and in the period thereafter. Also 

positive, youths frequently reported being in school or working at the follow-up—regardless of 

their initial risk level. 

Still, despite some positives in attitudes, interviewees also reported a good deal of later 

contact with the juvenile and criminal justice system and many self-reported drug or alcohol use. 

Youths frequently received treatment during the juvenile justice process or follow-up period. The 

moderate and higher risk groups tended to report greater involvement in treatment across most of 

the categories included in the interview protocol (with the exception of “counseling”). This 

suggests the potential that the juvenile justice process can utilize the assessment information 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

301 
 

 

 

 

 

 

 

 

 

 

collected early on to inform decisions that impact positive attitudes and other developmental 

outcomes later on—effectively breaking the relationship between risk and later poor outcomes. 

Although the analysis was limited by the overall and within group-sizes for the interview sample, 

this is illustrated in part based on the procrime attitudes example that was explored in order to 

illustrate some tentative relationships among JRNA, treatment, and different categories of youths’

outcomes. 

Section III of the report addresses the fourth objective of the study: evaluating justice-based 

and developmentally relevant youths’ outcomes based on variation in assessment-based decisions 

that reflect the usage, monitoring, and implementation of standardized risk and needs assessments. 

The results from this section alert us to the fact that risk level and juvenile court dispositions have 

an impact on youths’ recidivism and these are generally in ways anticipated by prior research. For 

example, when state commitment was used appropriately, it had a slight negative effect on the 

likelihood of recidivism for higher risk youths. Similarly, moderate risk youths had increased 

likelihood of recidivism when placed on probation or intensive probation as opposed to a less 

formal supervision or case dismissal. In the second group of mediation models, we found that 

matching criminogenic needs to treatment referral does not consistently occur in the juvenile 

justice agencies that we studied and that frequently factors beyond the assessment information 

tended to be better predictors of whether or not youths were assigned to particular services. 

Additionally, where there were relationships between participating in treatment and recidivism 

were largely found to be in the opposite direction of what would be expected (i.e., engaging in 

treatment increased the likelihood of recidivism). Finally in this section, we took a deeper look at 

the follow-up interviews with youths. These analyses suggest that youths are changing during the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

302 
 

 

 

 

 

 

 

 

 

 

 

 

 

juvenile justice process and in the period thereafter. Positively, youths frequently reported being 

in school or working at the follow-up—regardless of their initial risk level. 

The previous three sections of this report, set the stage for the Section IV: Summary and 

Conclusions. This section will elaborate on some of the key finding and provide recommendations 

for systematically improving the implementation and use of JRNA. 

IV. Summary and Conclusion 

The four project objectives described throughout the report focused on the implementation, 

usage, and outcomes of JRNA. We analyzed several sources of data in order to reach informed 

conclusions in each of those areas. This final section of the report first identifies some limitations 

of our data. It then summarizes key findings in each area of the study before considering relevant 

implications and offering recommendations in four areas pertinent to JRNA: Training, Usage, 

Monitoring, and Research and Evaluation. 

Limitations 

The project data collection efforts led to several useful sets of data pertinent to relevant 

research and practice questions in JRNA. Nevertheless, each source of data described in the report 

has its own limitations that contextualize the findings and recommendations made based on this 

study. Those have been mentioned in brief throughout the report as relevant, but we describe them 

in more detail here to contextualize the key findings and offer insight for the future research and 

evaluation section below. 

Sampling 

The research team approached the three states included in this study because they all were 

using the OYAS, but were at different stages in the implementation process. These states are in 

the Midwest and in the Southwest. Geographically, this means that these states may not be 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

303 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

representative of other locations and state contexts. Beyond that, the interview sites were hand 

selected by state actors and the research team. While selecting the sites, we attempted to include 

agencies and individuals who our state contacts believed might hold both positive and negative 

views of the OYAS and the implementation process. We also ensured that early and late adopters 

of the assessment system participated. Agencies varied in their size, setting, and geographic 

location. However, the sample was purposively selected and as a result, opinions of personnel in 

these agencies may not be representative of all personnel in these states, other personnel who use 

the OYAS, or a different JRNA tool. Related, while the web-based survey link was sent to all staff 

in state maintained lists, personnel who responded to the survey may have been those who were 

either more satisfied or dissatisfied with the assessment system. Again, this may have resulted in 

responses that are not illustrative of the views of all personnel. 

The sampling strategies used for the case record and youth interview data elements were 

intended to generate representative samples of the population of justice-involved youths based on 

geographic region (which may influence the size and operations of a given agency) and the various 

stages of the juvenile justice system where the OYAS might be employed to assess risk and needs. 

In States 1 and 2, however, some counties refused to participate. To illustrate, 71.4 percent of 

counties that were contacted in State 1 agreed to participate. In State 2, 66.7 percent of counties 

agreed to provide data. As a result, the samples may not have fully captured the range of youths in 

these states. 

In many instances, counties that responded to our requests and declined to participate 

expressed concerns about the agency’s capacity to complete the data extraction task due to a lack

of resources. Several indicated they were too understaffed to take on new obligations, and practical 

barriers may have amplified this concern. For example, the lack of sophisticated data management 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

304 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

systems in some counties necessitated that staff manually review each youth’s case for the 

requested information. In some cases, this required locating and reviewing paper files that may 

have been placed in a secure storage location that was not easily accessible, and the agency could 

not devote the staff time required to gather the information. These issues could have important 

implications for the study, particularly if agencies that are severely lacking staff and other 

resources also find it challenging to fully implement JRNA and the recommended case 

management strategies that come with using the tools. 

The subsample of youths selected for the follow-up interviews may also be limited in the 

extent to which it represents the youths in each state. In some instances, the states were unable to 

provide contact information for prospective interviewees, or the contact information that was 

received was outdated. Although multiple methods were employed to locate and contact those 

youths selected for this portion of the study, many were unreachable. Additionally, several youths 

who were found refused to participate. With an overall response rate was just below 10 percent 

and 20 percent among those youths with whom we had some contact, insights can be gathered 

from these data; however, the generalizability of the findings pertaining to the follow-up surveys 

is limited. 

Finally, contact information for youths who were selected for the follow-up interview 

portion of the study was received at varying points in time. Counties in State 1 and State 2 provided 

the requested data on a rolling basis, and State 3 only provided contact information in the final few 

months of data collection. This variation impacted the amount of time the research team had to 

make contact with potential interviewees (particularly in State 3), as well as the length of time 

between selection and follow-up interviews. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

305 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Measures and Data Collection 

The juvenile justice personnel interviews and surveys were designed to draw both breadth 

and depth into the data on implementation, usage, and perceptions of the OYAS. With that in mind, 

we used a dual-pronged approach to asking questions of juvenile justice personnel. Nevertheless, 

it is possible that the methods used for asking questions may have generated some artifacts in our 

data. For example, respondents may have been more apt to self-nominate particular types of 

strengths and weaknesses after having answered thematically similar questions earlier in the 

survey and/or interview. There was also the possibility of social desirability bias as reflected in 

the fact that interviewees, who were with a UC interviewer, tended to give more positive responses 

than web-based survey respondents on similar questions. In some sense, this helps to bolster the 

rationale for including both data collection modalities. 

The case record data collected from agencies presented some limitations with respect to 

measurement. First, data for the comprehensive assessment sample were gathered from numerous 

different agencies, each of which adopted their own norms for recordkeeping. Although each 

agency was given the same instructions for completing the data request, some items were not 

routinely collected by all agencies or there may have been variation in what and how data fields 

were documented. For example, agencies that listed a specific intervention such as Aggression 

Replacement Training may have classified it as cognitive behavioral therapy or anger management 

training. The research team discussed inconsistencies as they were identified and addressed them 

as each data file was cleaned and integrated. There were also limitations in the terms of the depth 

of data collected on each treatment type such that details on aspects of treatment such as dosage 

or completion status were not available (or not provided by the agency). Where the information 

provided was unclear or deviated from the instructions that were provided, efforts were made to 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

306 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

contact the agency for clarification. While these strategies likely enhanced the consistency and 

accuracy of the data, it is possible that some differences in interpretation or reporting exist. 

Although evaluation of specific treatment types and dosages was beyond the scope of this study, 

that may have impacted some conclusions about the relationships between treatment/services and 

recidivism. 

Similar issues came up in the officially recorded case details and justice outcome data. 

Agencies had different definitions for recording data about focal cases, dispositions, and 

recidivism and varied in the level of detail provided about each in response to data request. Most 

agencies provided dichotomous indicators that amounted to tallies of new referrals, adjudications, 

and commitments. This was particularly problematic in State 3 where no treatment information 

was provided for any cases. Further, only 35 percent of youths from State 1 received a treatment 

referral and only 28 percent of youths in State 2 received a treatment referral. As a result, it was 

not possible to examine differences in treatment effectiveness based on these factors. These 

measurement limitations are particularly relevant in the analyses focused on linkages between risk 

assessment information, treatment referrals, and youths’ recidivism. Overall, the data received 

across agencies varied in its completeness. In jurisdictions that provided large amounts of data that 

were easier to download from a larger database, only information contained in that database may 

have been submitted. 

Second, the accuracy of the data collected in the youth follow-up surveys is contingent upon 

the capacity of youths to recall events accurately as well as their willingness to respond honestly. 

Given that the interviews were completed by phone, it was not possible for the research team to 

ensure that the youths were in an environment that was conducive to sharing sensitive information. 

For example, when asked to report involvement in deviant or illegal behaviors, youths could have 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

307 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

been dishonest if a parent or guardian were present during the interview and they were fearful of 

receiving a consequence if their activities became known to that person. Prior research has 

generally found, however, that self-reported measures of delinquency are valid and reliable 

(Thornberry & Krohn, 2000). Additionally, the data obtained from that portion of the study seemed 

to reflect a reasonably high level of disclosure of substance use and new justice contact. 

A third limitation related to the measures included in the study is that many of the variables 

were ultimately dichotomized for the analyses that examined the use of JRNA in the juvenile 

justice process. As a result, the findings are more general in nature. For example, there was a great 

deal of variation in the information we received about the treatments the youths participated in. In 

order to examine patterns across the full sample, treatment information was collapsed into more 

generally defined categories (e.g., CBT) that did not allow for differences in the effects of 

particular types of treatment (e.g., Thinking for a Change) to be estimated individually. In turn, 

the findings do not provide any indication of whether some interventions are more or less effective 

at reducing recidivism than others. 

Analytic Limitations 

The limitations in our data analysis approaches generally follow from some of the 

measurement and sample shortcomings identified in the prior sections. The qualitative portion of 

the study relied on analysis toward grounded theory building. While we used a variety of checks 

and proceeded iteratively, emergent themes were nevertheless drawn from interpretive analysis of 

interview responses. In many cases, those responses comprised brief comments and interviewer 

notes as opposed to full narratives and are therefore limited in depth at times. In the quantitative 

analysis, at times the range of particular measures was limited and therefore we were required to 

use modeling strategies to follow that. In some cases, that meant that methods for dichotomous 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

308 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

measures were utilized. In other cases, the statistical power was relatively low for some 

comparisons. This was especially true in the youth follow-up interview analyses and the 

consideration of patterns of OYAS vignette scoring in the web-based survey. In general, we 

attempted to assess assumptions and sensitivity of particular estimates wherever possible in order 

to contextualize our main findings. 

Key Findings 

Key Findings and Discussion from Juvenile Justice Personnel Interviews and Surveys 

This portion of the study identified a solid foundation upon which juvenile justice agencies 

may build an effective and sustainable implementation process and many of the findings were used 

to draft the recommendations that are discussed at the end of this section of the report. Overall, 

most personnel were positive generally about risk and needs assessments and satisfied with the 

OYAS. However, they perceive their own satisfaction to be higher than that of their peers. 

Similarly, personnel who responded to the survey were generally less satisfied with the OYAS 

than interviewees. 

The key discussion points for the juvenile justice personnel interviews and surveys can be 

summarized into six main areas: (1) personnel were generally favorable towards the OYAS, 

however, there is a lot of room for improvement concerning their views; (2) personnel alluded to 

the fact that the full utility of the assessment system was not being realized; (3) personnel were 

able to identify numerous strengths and areas of concern (areas of concern were more consistent 

that the strengths); (4) personnel noted many ways to improve the implementation process; (5) 

there was a major lack of quality assurance processes taking place in the three states; and (6) only 

about half (52.2%) of survey respondents scored the vignettes correctly. 

Starting with the first point, interviewees and survey respondents were generally satisfied 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

309 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

with the OYAS. Across the board, interviewees reported higher levels of satisfaction than survey 

respondents. Concerning personal satisfaction with the OYAS (on a scale of 1 to 5 with 1 being 

‘not at all satisfied’ and 5 being ‘very satisfied), mean satisfaction for interviewees was 3.54.

Survey respondents’ mean satisfaction, separated by those who administer the tools and those who

do not, was 2.96 and 3.13, respectively. Both samples did agree at higher rate that the OYAS 

benefits youths, which is promising, and agencies may be able to capitalize and leverage that 

support. As a whole staff were positive, or at least neutral, towards to the assessment system. 

However, this indicates a lot of room for growth concerning increasing the favorable views of the 

OYAS. To illustrate, the OYAS was often pitched to staff as an “end all, be all” piece of

information, not as one piece of the puzzle in working effectively with justice involved youths. 

Many of the recommendations in the next area of this section relate to how to agencies can increase 

satisfaction with the tool before, during, and after implementation. 

The second clear finding was that agencies and personnel within the agencies are not using 

the OYAS to its full utility. The OYAS is typically used to (a) determine supervision level, (b) 

measure the progress of youths, (c) match youths to services, and (d) assist in placement decisions. 

However, the OYAS is not being used in the following ways: (a) assessment results are not 

consistently shared with those who may need the information (e.g., referral agencies, parents), (b) 

staff are not utilizing youths’ assessments results to inform case planning, (c) assessment results 

are not used to place youths with staff, and (d) agencies are not using aggregate data that are 

generated from use of the OYAS (i.e., agency resources are not being allocated based on OYAS 

findings and specialized caseloads have not been developed). 

Related, the mostly commonly reported uses of the OYAS matched the state and local 

policies. Policy in essence drove practice in this study and robust policies seemingly then would 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

310 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

translate to more comprehensive usage practices. Some interviewees specifically expressed the 

need for additional guidance on how to use the information gathered and produced by the OYAS 

in more meaningfully supervision of youths and in treatment/case planning. Further, the lack of 

using the OYAS to its full potential may be impacting the levels of satisfaction noted above. To 

illustrate, if staff are taking the time to complete the assessments, they may appreciate the 

assessment information being used to its full potential. 

Third, personnel were able to clearly articulate strengths and limitations. It is noteworthy 

that, despite the separate analysis processes, both the interview and survey samples generated 

many of the same themes, which helps to triangulate some key findings. Common strengths 

included: it helps decision-making and service provision; it provides overall system fairness, 

consistency, and objectivity; provides useful and comprehensive information; identifies risk level, 

needs, and aids in determining supervision level; establishes a baseline and allows for the 

monitoring of progress; is helpful in developing with case plans; and that it is easy to administer. 

Limitations were noted as: lack of validity/reliability (i.e., provides inaccurate or invalid 

results regarding risk level); it does not provide comprehensive information (i.e., fails to capture 

family and education issues, mental health needs, and/or substance abuse issues); concerns with 

the items on assessment; conducing the tools was time-consuming; that there is inconsistent 

administration and scoring of the tools; issues with using the information in case planning (e.g., 

staff were not trained how to do this, but are expected to do it); poor implementation and training 

(i.e., lack of policies and procedures); that the tool is inaccurate for sex offenders; issues with using 

the automated system; concerns with youths manipulating scores; lack of buy-in from personnel; 

and lack of utility of the results. In looking at one of these limitations in detail, the personnel were 

in agreement that the OYAS is too time-consuming. This speaks to the nature of their daily work, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

311 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

and lends itself to ‘shortcuts’ (e.g.,, cutting assessment questions) that can exacerbate issues with 

validity and reliability. As such, some perceived OYAS limitations and implementation challenges 

can become self-reinforcing over time without continuous quality improvement efforts. 

Taken together, while the tools have benefits, these may be overshadowed by the staffnoted limitations. In highlighting one of these limitations, there is concern regarding using OYAS 

information to determine placement and supervision level if the tool is not accurately predicting 

risk. It was suggested that inaccurate scoring has created a ‘net-widening’ effect in one state, as 

more youths at one particular agency were being classified as “high risk” than before use of the

OYAS. On the other hand, in a different state, it was suggested that the OYAS was routinely 

underscoring youths. These patterns of classification generally depended on the agency where the 

personnel was interviewed, but overall, there were concerns expressed about the manner in which 

risk level was tied to supervision and service levels. 

Fourth, personnel were able to provide numerous recommendations for improving buy-in, 

roll out, and implementation. Common suggestions included: making the automated system more 

user-friendly and integrated into other online systems prior to rollout; allowing more room for 

professional judgment to change the risk level of a youth in instances where that is necessary (i.e., 

many staff feel that the OYAS under or over classifies many youths); higher quality initial training 

and providing more frequent training and certification on the tools; designating certain staff to be 

OYAS administrators to ensure consistency and accuracy in administration; taking steps to foster 

consistency in implementation across counties of a given state (e.g., minimum usage guidelines 

and ensuring that treatment providers are in place to address the youths’ need areas); providing 

more convenient ways for staff to access aggregate OYAS data; and consistently training outside 

stakeholders on the OYAS (e.g., referral agencies attorneys, judges, prosecutors). In terms of 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

312 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

implementation and training, 17.6 percent of State 2 interviewees, 15.2 percent of State 3 

interviewees, and 12.7 percent of State 1 interviewees named this as a concern. 

Adding context to the recommendation regarding better training, one interviewee 

explained that, as a member of the detention staff, the trainings were too “probation-focused,” and 

not as applicable to his/her role as he/she would have liked. Another interviewee suggested the 

importance of asking staff, “What do you want the tool to do for you?” for establishing buy-in 

prior to conducting trainings. Though, the value in doing so is decidedly in whether trainers and/or 

agency leadership adjust their approach according to trainees' responses. Lastly, another 

interviewee suggested a post-training trial period for the tools, where staff could decide for 

themselves how it would be the most useful for a specific agency, department, or unit. In this way, 

while the tools may be mandated in a state, each agency, department, and/or unit could have the 

opportunity to determine how to best introduce and use each OYAS tool for their setting. State 3 

respondents agreed significantly more that formal OYAS training is routinely offered. Once more, 

this suggests the importance of not only providing regular training opportunities, but ensuring their 

quality or applicability to staff. This is evidenced by State 3’s generally low levels of satisfaction

for the OYAS and its uses, but there is a high level of agreement that training is available. This 

illustrates that satisfaction with the tool is not totally synonymous with a positive view of the 

implementation—and vice versa—again illustrating the multifaceted nature of implementation, 

use, and outcomes in JRNA. 

Fifth, there was a serious lack of quality assurance procedures. Only 37.1 percent of 

interviewees responded that any quality assurance processes were in place specific to the OYAS. 

Follow-up questions in this area evidence that these processes mostly relate to ensuring that the 

tools are conducted within a certain timeframe and are not related to quality of the information 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

313 
 

 

 

 

 

 

 

 

 

 

 

 

collected or that the tools are conducted and scored correctly. Two effective practice to showcase 

here is that (a) some agencies had identified a local point of contact for the agency to help answer 

questions and increase consistency in scoring and use of the OYAS and (b) one agency noted 

interrater reliability efforts where supervisors sit in interviews with newly trained staff and review 

the scoring of their assessments. In each of the three states, numerous staff noted that it was often 

difficult to get their questions answered related to the OYAS. This was a result of either supervisors 

who were not trained and could not support them or that there was someone knowledgeable about 

the OYAS, but that individual was not responsive to their questions. It would benefit agencies 

using risk and needs assessments to consider using dedicated staff to support the use of the tool. 

These individuals need to be knowledgeable about the tools and be provided enough time within 

their job responsibilities to properly support staff using these assessments. These same staff could 

then be used for general OYAS quality assurance processes as well. 

Finally, vignette results show that only half of OYAS administrators scored their assigned 

vignettes correctly (52.2%). Interviewees expressed concern that their coworkers were not scoring 

the OYAS correctly, and this seems to be supported by the vignettes. Incorrect scoring trended in 

the direction of underscoring the vignette, indicating that respondents were more likely to assess 

less risk, not more risk. Overall, the vignette scores suggest that, while there is some 

correspondence with the manual, there are also some discrepancies that require further attention 

and that might be addressed in future booster trainings. The vignette findings give more credence 

to the recursive nature of having a tool that may not be scored correctly—that staff become aware 

of this and lose faith in the results of the tool as a result. Similarly, survey respondents who reported 

higher level of support for the usefulness of the OYAS were more likely to accurately score the 

vignette. These two elements are interrelated in complex ways that need to be addressed early in 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

314 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

the implementation process in order to ensure that different components of the tool itself, 

implementation practices, and usage in case-level decisions do not undermine effective and fair 

usage. 

Key Findings and Discussion from Youth Case Records and Follow-Up Interviews 

Another important aim of the study was to determine how youths were impacted by the use 

of JRNA in the juvenile justice system. Tremendous emphasis has been placed on using these tools 

to classify youths into categories based on their likelihood of reoffending, and whether or not those 

classifications are valid and reliable. Far less attention has been given to understanding the use of 

risk and needs assessment to inform decision-making and case management, or the impact that 

may have on the youths who are involved in the juvenile justice system. Implementation and usage 

of JRNA are multifaceted and depend on a number of different sub-components to work effectively 

and efficiently. We therefore used the record data collected here to study questions about how 

OYAS was being used in practice in these sites, focusing on several key questions of interest. This 

in turn provides insight on implementation and youths’ outcomes. Key findings from each of these 

usage studies are presented in brief below followed by the youths’ case data and follow-up 

interviews. We also call attention to particularly important implications of each. 

Usage Study 1: Support for the OYAS and Integrity in the Risk and Needs Assessment Process 

This study explored the integrity in the risk and needs assessment process. The results from 

the study suggest that assessment administrators with higher levels of perceived support for the 

utility of the assessment system were more likely to score the domains accurately. Taking stock of 

what the OYAS is designed to do can influence integrity as captured by the degree to which 

administrators tended to score accurately. Those who underscore the tool may not see the utility 

of using it to its full potential and may believe they think they know what is best for a youth. This 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

315 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

in turn, may lead staff to not engage in proper information gathering during the interview to 

effectively score out the tools. 

Usage Study 2: Youth Appraisals by Race and Geography 

This study explored the potential variation in the distribution of youths’ risk levels across 

juvenile court jurisdictions and investigated whether race accounts for any of the variation in risklevel classification between juvenile court jurisdictions. The results indicate that risk assessment 

classifications varied significantly among sites in our sample of juvenile court jurisdictions (i.e., 

the distribution of youths that fall into the risk classification categories differ across the courts). 

Non-White youths were significantly more likely to receive a higher risk classification than their 

White counterparts, though these effects do not differ by court jurisdiction. This effect of race on 

risk classification remained statistically significant when we accounted for the youths’ focal

offense type and demographic characteristics. 

This substudy highlights the need to further examine the neutrality of JRNA. These 

findings have implications for case management, court processing, treatment-related decisions, 

and potential disparities in each as the risk assessment information may be impacting those 

decisions. Risk and needs assessments are valuable tools that can help to guide a number of 

decision making points in the juvenile justice system, but researchers and practitioners should also 

carefully consider the degree to which practices may affect differential juvenile justice experiences 

among youths of different race and ethnic groups. 

Usage Study 3: Race, Ethnicity, and Validity of the OYAS Residential Tool 

The third of these studies examined the measurement properties of the Residential Tool in 

State 3, and whether racial/ethnic differences in the predictive validity of the tool could be partially 

attributed to variance in the measurement of risk. The results indicated that it may be possible to 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

316 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

refine the measurement of risk for recidivism in this population. In the full sample, the Education 

and Employment and Family and Living Arrangements domains were the strongest areas of 

concern as evidenced by weak item and domain loadings. These findings highlight the importance 

of empirically testing whether JRNA tools developed in a specified population can be assumed to 

measure risk and needs equally well when adopted in a new population where the prevalence and 

nature of specific risk indicators may vary from that of the original population. As an example, a 

number of items had very low or very high prevalence in the State 3 population, which would limit 

the utility of those items in terms of distinguishing those who returned to custody from those who 

did not. 

Another important finding from this substudy is that the Residential Tool does aid in 

predicting recidivism relative to chance; however, the predictive validity varies across groups 

defined by race/ethnicity. Relationships between risk scores and recidivism appear to be stronger 

for Hispanic youths relative to Caucasians as evidenced by a general pattern of stronger 

correlations and higher AUC estimates. Similar to the measurement analysis, the analytic strategies 

used to examine the predictive validity revealed that the Education and Employment and Family 

and Living Arrangements domains did not aid in predicting returns to custody for the full sample 

or the Caucasian and Hispanic subgroups. 

Lastly, given the differences across racial/ethnic groups with respect to predictive validity, 

preliminary analyses were carried out to investigate whether those differences could be partially 

accounted for by differences in the constructs captured by the Residential Tool when used in 

Caucasian and Hispanic subgroups. The findings of this substudy suggest that the tool may not be 

measuring risk equally well among these two groups. Such patterns can be indicative of individual 

items working differently in the two groups (e.g., more or less prevalent in one relative to the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

317 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

other), including items that do not contribute to the measurement of risk for one or both groups, or 

failing to include items that have a meaningful impact on risk. These complex relationships, which 

in some case reflect potential threats to the validity of the measures, warrant further investigation. 

Usage Study 4: Professional Override of Risk Assessment 

This substudy includes case record data for 6,222 youths assessed by the OYAS across the 

three states studied and links it with the juvenile justice personnel interviews and the web-based 

survey. The overwhelming majority of interviewees noted that their agencies allow override of the 

OYAS risk level. Of the 213 cases where an override occurred in the analytic sample, 210 of them 

– roughly 98.5 percent – were upward overrides. This aligns with some of the concerns noted in 

the juvenile justice personnel interview and survey data as numerous staff mentioned concerns that 

the tool does not scores youths accurately. Another potential explanation for this is that upward 

override results in more opportunity for services. In essence, if an administrator thinks that a youth 

should enroll in a program, but the program has a risk-level eligibility requirement, the 

administrator may use an override in order to render the youth eligible. This suggests the 

importance of understanding the implementation context, especially local political pressures and 

resource constraints, that can affect full adherence to assessment protocols (see Aarons et al., 2011; 

Fixsen et al., 2009). Moreover, the most common reasons for overrides were sex offenses, offense 

seriousness, and criminal history. This is a potential area of concern that staff should receive

training in—what constitutes a reason for an override, what to do with special population youths,

and how to use the OYAS in conjunction with other risk and needs assessments. 

Usage Study 5: Use of Mental Health and Substance Use Information in Treatment Matching 

Case record data for four juvenile courts in State 2 were used in order to examine the OYAS 

indicators of substance use and mental health needs and asses the links between these indicators 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

318 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

and the services received. First, youths in the sample had relatively low rates of mental health need 

(~20%) and substance use (low mean scores on the substance use index score). Using multivariate 

logistic regression modeling, the results suggest that—when controlling for other variables—those 

youths that were indicated on the OYAS as having substance use and mental health needs were in 

fact significantly more likely to be referred to that programming. These tools are intended to guide 

referrals to treatment, but indicators are limited and not necessarily specific. Still, the mental health 

item and substance use score information seems to be impacting treatment referral decisions in 

these cases, and these findings are consistent with patterns observed in the treatment mediation 

models (described below). 

Usage Study 6: OYAS Strengths and Barriers Usage 

The final substudy investigated the extent to which the strengths and barriers portions of 

the OYAS Disposition, Residential, and Reentry tools are used in practice and whether exposure 

to strengths and barriers varies by risk level. The findings revealed that, though the strengths and 

barriers portions of the tools are optional, the majority of youths (~66%) have at least one strength 

or barrier noted in their assessments. Moreover, the relationship between risk levels and the 

number of strengths and barriers was consistent with expectations; low risk youths had the greatest 

number of strengths and the least number of barriers while high risk youths had the fewest strengths 

and the most barriers. Across the board, strengths were more prevalent than barriers, except in 

Peers and Social Support Networks—highlighting the need for agencies to anticipate treatment 

needs in this area. These findings are encouraging, as it suggests that staff are taking steps to gather 

more information that could be used to guide case management decisions, and it provides some 

preliminary evidence to suggest that these portions of the OYAS are working as intended. 

The results of this substudy also provided some important insights about justice-involved 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

319 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

youths at varying stages of the juvenile justice process. Acknowledging strengths and barriers may 

be a fundamental step in effective case management and the development of realistic treatment 

plans—especially in the context of juvenile justice where some barriers to effective treatment may 

lie outside of the youth’s control. Overall, these findings suggests that one state using the OYAS 

has started to move toward a more comprehensive assessment process through use of strengths 

and barriers indicators to inform case planning. The findings simultaneously reveal some room for 

expansion in the use of the strengths and barriers portions of the OYAS tools as well, which may 

prove to be one avenue for identifying more options for clients and better tailoring their treatment 

and supervision plans. In order to achieve meaningful gains in this regard, however, it may be 

necessary to consider several enhancements for the strengths and barriers sections and related 

training. 

Analysis of the Linkage Between Risk Assessment, Juvenile Justice Decisions, and Recidivism 

The large sample of case records from the three states studied was used to estimate 

statistical models to capture the key elements of the presumed process linking JRNA to youths’ 

recidivism. In the course of that analysis we found that higher risk youths tended to have 

significantly higher rates of recidivism—even with other controls in the models—which suggests 

that the OYAS is predicting risk for recidivism as intended. The main point of that analysis, 

however, was intended to link risk and needs assessment usage with outcomes in a way that more 

fully captured the system processing that occurs in between the two. The results indicate that the 

relationship between risk level and recidivism is at least partially mediated by dispositions. In 

particular, when high risk youths receive more intensive services—including commitment to state 

facilities—their likelihood of having a new adjudication decreases slightly (~3%). Further, 

moderate risk youths on probation had lower rates of recidivism relative to low risk youths on 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

320 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

probation—relative to less restrictive, or informal, sanctions. These findings align with prior 

research that stresses the importance of fitting the intensity of juvenile justice options to those at 

greater risk of reoffending. 

The theory on JRNA emphasizes the need to match the treatment youths receive to their 

most pressing criminogenic needs in order to reduce recidivism as well. That potential use of JRNA 

in this process was also examined in this study. Several notable patterns emerged. First, the most 

common treatment received by youths was for mental health, followed by substance use. 

Additionally, these seem to be supported by the fact that higher scores on these indicators drove 

these referrals. Second, generally, referral to the six types of treatment or services examined in this 

study did not significantly reduce the likelihood of a new delinquency adjudication. In every 

analysis the effect of treatment was either non-significant or was associated with an increase in the 

likelihood that youths would recidivate. Third, the domain scores generally were not significantly 

associated with their relevant treatment types as would be suggested by the underlying logic of 

JRNA. The multivariate models generally showed that other variables like juvenile justice history, 

focal offense type (i.e., violent/not), and State site generally were more relevant in whether a youth 

was referred to a particular treatment type or not. Not surprisingly given the previous finding, the 

relationships between criminogenic needs (i.e., domain scores) and recidivism were not mediated 

through treatment, as would be expected. Given that the relationships between criminogenic needs 

and recidivism were not mediated as would be expected by various modes of treatment, further 

research is needed to determine what may account for these unexpected patterns. 

Analysis of Youths’ Self-Reported Outcomes 

Youths in the study also reported on some of their behaviors, and the findings based on 

these data were positive overall. Many of the attitudinal scales included in the follow-up interviews 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

321 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

revealed favorable perspectives, and the majority of those interviewed reported being involved 

with school or work at the time of the interview. There was also a general trend suggesting that 

most justice-involved youths received some form of treatment, with higher rates of treatment 

reported among those at higher risk levels. While these findings are encouraging, the youths also 

reported using drugs or alcohol during the follow-up period, and many continued contact with the 

juvenile and criminal justice systems. The findings nevertheless provide some confirmation of 

patterns seen elsewhere in the study while also offering some additional insights that would not be 

possible based on the analysis of official records alone. 

Implications and Recommendations 

Section I of the current report outlines the trends in agency policies and practices around 

assessment, the facilitators to implementation and use of the OYAS, and the barriers to 

implementation and use. The second section of this report focuses on the results of usage, 

processing, and case decisions. Section III examines youths’ outcomes in relation to their risk and

needs assessments and their experiences while involved in the juvenile justice system. Across these 

three areas, these findings suggest that the OYAS is generally being used in line with what is 

intended. At the same time, there are areas of practice where improvements could be made to 

ensure optimal usage. These raise implications for both practice and research in JRNA. The start 

of this fourth and final section of the report summarizes key findings and highlights discussion 

points that inform the last piece of this section and report, implications and recommendations. 

With so many juvenile justice jurisdictions using JRNA, it is important to understand how they are 

being used and develop strategies to enhance their use. A huge element of this is the juvenile justice 

personnel who are interacting the youths. This study is unique in that it combined the actual 

practices and opinions of these personnel with both case records from youths assessed with JRNA 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

322 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

and follow-up interviews with a subset of these youths. The combination of these data allowed the 

research team a comprehensive view of JRNA so that recommendations could be developed to 

help achieve a next step with these assessments – usage optimization. 

In light of the findings presented above, the research team offers the following 

recommendations to cover the full spectrum of risk and needs assessment implementation 

including policies and practices, establishing buy-in, designing implementation, training, use of a 

comprehensive risk and needs assessment, promoting usage, and establishing quality assurance 

processes.81 The recommendations are a culmination of feedback from juvenile justice personnel 

and researcher insights into how to improve JRNA implementation and use. These 

recommendations are mindful of the fact that risk and needs assessment—and associated decisionmaking—involves a concerted change in how those in the field process and react to information 

on justice-involved youths. In parallel, we also contextualize the implementation and use of this 

assessment in the broader objectives of the juvenile justice system and the youths that it encounters. 

While risk and needs assessment prevalence and integration into the juvenile justice system 

has increased in recent years, relatively little research has been done on the implementation of risk 

and needs assessments and its associated processes. The available research informed this study 

and a lot of the recommendations provided in this section align with this body work. Of note in 

the research on JRNA is the work of Vincent and colleagues’ guidebook for risk assessment 

implementation in juvenile justice (2012). This work infused the results of past research and input 

from expert and stakeholder panels into an expansive how-to guide for implementing risk and 

needs assessments in juvenile justice settings (focusing on probation in particular). According to 

their research, Vincent et al. (2012) divided the implementation process into eight stages: (1) 

81 State-level reports are being provided to each state to consider efforts to improve JRNA training, usage, and 

monitoring within their own states. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

323 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Getting ready; (2) Establishing stakeholder and staff buy-in; (3) Selecting and preparing the risk 

and needs assessment tool; (4) Preparing policies and essential documents; (5) Training; (6) 

Implementing the pilot test; (7) Full implementation; and (8) On-going tasks for sustainability. In 

the guidebook, each step of the implementation process is accompanied with its own set of 

recommendations, many of which overlap with those resulting from the current study. 

In the same vein, OJJDP published a bulletin describing the main findings of recentlyfunded research (i.e., Vincent et al.’s work and preliminary results from this study), and their

implications for JRNA implementation (Vincent et al., 2018). Following the work of Fixsen et al. 

(2005) and the National Implementation Research Network (n.d.), these implications were divided 

into various sets of “implementation drivers” specific to staff competency, as well as 

organizational and leadership factors. While the current study’s recommendations cover these

three implementation drivers, they are organized more specifically by the objectives of the current 

study and provide more detailed suggestions for implementation based on the results previously 

described. 

While there is some degree of overlap with the goals, recurring themes, and 

recommendations set forth in the work of Vincent et al. (2012) and the studies described in the 

OJJDP bulletin (2018), the current study expands upon these works by including information from 

four unique data sources (i.e., in-person juvenile justice personnel interviews, a web-based juvenile 

justice personnel survey, a review of youths’ records, and results from a follow-up survey of 

youths). We also relied on the broader work on implementation of effective intervention and 

innovation in complex environments (e.g., Aarons et al., 2011; Durlak & DuPre, 2008; Fixsen et 

al., 2009; Greenhalgh et al., 2004). Additionally, the current study focused on the implementation 

of one particular already established risk and needs assessment across 22 agencies at various 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

324 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

juvenile justice decision-points (i.e., intake/diversion, detention, disposition, residential intake, 

and reentry), and across three states in unique phases of a statewide implementation process. The 

results include agency policies and practices surrounding the OYAS and identified facilitators and 

barriers to its implementation and use. 

As such, the current study allows us to outline recommendations with more specificity and 

in more topic areas. These recommendations, while based specifically on information gathered 

about the OYAS, are generalizable to the implementation of other JRNA (and moreover to the 

implementation of risk and needs assessments in the criminal justice system as well). However, as 

evidenced throughout the report, it is also important to note that each state (and each county or 

agency therein) presents its own unique implementation environment. Therefore, state, county, 

and agency leadership must be mindful of which recommendations have more importance in their 

context, and which may become obstacles. Striking a balance between consistency and adaptation 

is a challenging, yet key component of effective implementation. 

While certain recommendations are geared toward those implementing a statewide risk and 

needs assessment, others focus on the micro-level implementation concerns faced by counties and 

agencies, which may be part of a statewide risk and needs assessment implementation, or may 

result from implementing a risk and needs assessment independently. The recommendations are 

organized by three major elements of implementation (which reflect the objectives of the larger 

study): training, usage, and monitoring. Due to the dynamic nature of implementation, many 

recommendations apply to more than one of these areas. Therefore, recommendations are placed 

where they are considered most relevant, but may also apply to other pieces of the implementation 

process. For example, one recommendation under training is adding more detailed information 

about usage expectations into staff training, which also overlaps with usage recommendations 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

325 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

about policy requirements. 

Training 

For the purpose of the current report’s recommendations, ‘training’ recommendations are

divided into three unique phases: planning/pre-training, during training, and post-training. 

Training recommendations cover considerations that have a potentially large impact on buy-in 

among justice personnel prior to, during, and after training. 

Planning/Pre-Training 

1. A risk and needs assessment has the potential to impact not only the youths, but everyone 

who comes into contact with that youth. As such, it behooves agencies to include as many 

stakeholders as possible in the pre-implementation process. This may include judicial 

actors, detention center staff, court staff, internal and external service providers, institution 

staff, the youths, and their families/guardians. 

2. Elements of the implementation process (i.e., rollout timeframes, policies and procedures, 

tool usage practices, and quality assurance processes) should be clearly communicated to 

all staff impacted by the use of the assessment information (and not just those in leadership 

positions). More information on policies is located below under Usage. 

3. Agencies should decide between having staff who are generalists (i.e., those who conduct 

assessments and have other tasks), having some staff who are more specialized assessors, 

or having full-time staff who conduct only assessments, and accommodate and support 

those roles accordingly. 

4. States and local agencies should decide whether to develop and support local site trainers, 

state-level trainers, or if all trainings will be conducted by the developer of the tool (or 

some combination of the three options). State-level and local site trainers allow for more 

frequent and seamless ongoing training opportunities. State and local trainers may also 

increase buy-in from agency staff who are wary of outside trainers, or who have concerns 

about how a statewide assessment may apply to their specific agency or daily work with 

youths. 

5. States and local agencies should train staff who are both administrators and nonadministrators of the assessment tool, as this may harness natural buy-in from those who 

will not be responsible for administering the tool, but are eager to learn. Non-administrators 

may not need the full training, but rather an abbreviated version. The ‘champions’ of a risk 

and needs assessment need not be those who administer the tool, but those who 

acknowledge the importance and potential positive impact of a risk and needs assessment 

on the youths at their agency. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

326 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

6. All personnel who are responsible for administering the assessment should be formally 

trained and certified prior to doing so. Agencies should anticipate the resources needed to 

provide initial training as well as the onboarding of new staff and plan accordingly for their 

qualification to assess youths. It is appropriate for staff who are formally trained in the tool 

to conduct assessments for untrained staff until they are certified to do so. 

7. In order to gain a systems perspective of how the assessment process benefits the youths 

and staff throughout the entire process, states and agencies should consider the feasibility 

of cross training staff and stakeholders from outside agencies. This helps encourage a 

common language regarding risk and needs assessment across a county/state, and fosters 

the sharing and spread of best practices between agencies. For training that is more 

restricted (e.g., staff only need training on one element like treatment planning or one 

specific tool within a larger assessment system), shorter sessions are appropriate. 

8. If using a data entry and storage system, or management information system, this system 

should be adequately funded and tested for functionality prior to implementation. This 

includes entering information, mapping the workflow, and testing whether and how this 

system can be integrated into others already in use at the time of implementation. Issues 

and system bugs should be anticipated and resources reserved for fixes that can be 

completed quickly and with minimal interruption in the processes that are just taking root. 

9. Often a risk and needs assessment will be integrated into existing processes and other 

assessments will continue to be used. Therefore, agencies should consider how the 

assessment fits into existing assessment protocols and what other enhancements or changes 

are needed to provide a comprehensive assessment of each youth. For example, it is 

advantageous to have responsivity assessments (e.g., those that measure barriers) in place 

prior to the ‘roll out’ of a risk and needs assessment, as this information will bear directly

upon a youth’s success while on supervision or in custody. It is also important to begin to 

set a new process that integrates different information sources as early as possible and 

provide clear guidance to staff during training and support them after training. 

10. Systematic quality assurance (QA) processes should be planned and put in place prior to 

implementation (see ‘Monitoring’ section for more specific guidance on what quality 

assurance practices should entail). The necessary QA training should be planned and 

developed at the state and local level (e.g., training and certifying supervisors around 

interrater reliability). These practices should start concurrently with the assessment rollout, 

but be aligned with the stage of the implementation process. 

11. Risk and needs assessments provide useful information on the needs of youths. Services to 

address these need areas must be in place before the assessment is implemented. To 

promote assessment information usage and embeddedness, agency leadership should foster 

and formalize inter-agency collaborations (e.g., memoranda of understanding) prior to 

rolling out an assessment. More recommendations on service mapping can be found under 

‘Usage.’

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

327 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

During Training 

1. Information about why the assessment was chosen and how it fits with the agency’s

mission should be included in the training process. This may involve presenting 

information on other initiatives or routine practices that are informed by the assessment 

and any corresponding data or usage examples from other jurisdictions that can illustrate 

why the state or agency has chosen the tool (e.g., information on the validation of the tool 

in similar states/localities). This information should be used to address some of the 

anticipated issues of contention that are expected during the training and implementation 

process. 

2. Along with some other common risk and needs assessments, the OYAS has multiple 

screeners and assessments that are available for use. In cases where multiple tools are used 

within an agency, it is important during the training to map out the purpose and 

corresponding usage guidelines for the different screeners and assessment tools. Staff 

should receive clear and consistent messages about the purpose of each screening and 

assessment tool, their suggested timelines, and which tools should be used for 

reassessment. Should state-level training with multiple participating agencies be provided, 

staff should receive guidance at the local level immediately following training. 

3. In addition to requiring training and certification on the assessment, training should include 

information about how the assessment results should be used. It is helpful if agency policies 

and procedures are being provided to personnel at that time as well. This information 

should include expectations for supervision levels (i.e., frequency and types of contact 

based on risk and needs level), expectations for treatment referrals, and how to conduct 

effective case planning (including good example case plans that can serve as a point of 

reference post-training). For example, case plans should be seen as a living document and 

personnel should be trained on creating identifiable and measurable goals in order to avoid 

goal displacement (Klein, 1979). 

4. During the training process, staff should be trained specifically to use any management 

information systems that accompany the tool and case planning issues (if applicable). This 

may help alleviate some of the technological and case planning issues specific to such 

systems, which can spillover, creating potential discontent with other aspects of the 

assessment and associated processes. 

5. Personnel should be trained to systematically collect and review collateral information on 

each youth prior to the assessment. This training should also include guidance and methods 

for scoring discrepancies between collateral and youth-provided information. 

6. Relatedly, personnel should be explicitly trained on the ‘hierarchy of information’ for any

given assessment. For example, in the OYAS, official records (i.e., police, court) should 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

328 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

take precedence over information provided by the youths during interviews or in selfreports.82 

7. Personnel should be informed and/or trained on the systematic quality assurance processes 

that will roll out concurrently with the assessment. This information should include what 

the quality assurance processes will entail and how they will impact the daily work of staff 

who conduct the assessment. They should also convey how the QA process fits into the 

success of the risk and needs assessment process. 

8. Training should clarify the purpose and appropriate usage of other assessments (i.e., need 

and responsivity assessments) and how those intersect with the risk and needs assessment 

information, especially during the case planning process. Should training take place at the 

state level with multiple participating agencies, staff should receive guidance at the local 

level following training. 

9. For professional discretion overrides, there was some confusion about what should be 

overridden—individual items, the overall score, the overall risk level, the custody or 

supervision level, or any combination of the above. Training should be explicit on the 

differences and the process for conducting an override. For example, what constitutes a 

reason for an override, what to do with special population youths (e.g., youths with sex 

offenses), and how to use the OYAS in conjunction with other risk and needs assessments. 

10. Training should use plain and clear language (i.e., not overly research-oriented) about the 

validation of the chosen tool and any planned validation processes. This should include 

what “validation” means, the steps to conduct local validation, and validations of the tool 

that have taken place in other localities. This might also address any potential staff concerns 

about individual items on the tool, where an item may appear to have low “face validity”

(i.e., an item that at face value does not appear to be connected to recidivism risk). 

11. A portion of training should also explain the concept of reliability, and the important role 

reliability plays in the performance of the tools. This may help to further illustrate the 

importance of aligning the assessment process and scores with the guidelines provided for 

each tool. 

12. At the local site level, specific delinquent populations may have certain supervision 

expectations. For example, a youth with a sex offense may be supervised differently (i.e., 

more intensely) than another type of youthful offender. Expectations for how JRNA results 

(and results from other assessments) will be used to aid in these supervision strategies 

should be provided at the local level. For example, staff in a sex offender unit would be 

provided additional training and guidance by their unit supervisor about how to merge 

general risk to recidivate with other offense specific information. 

82 For example, if youths state that they were not arrested under the age of 14, but police or court records indicate that 

they were, then this should be scored as a risk factor despite the information provided by the youths. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

329 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

13. Trainings should be conducted as closely as possible to the actual rollout of the assessment 

to capitalize on not only enthusiasm and readiness for change, but also to ensure that the 

information learned in training can be used immediately post training. 

14. Agencies should be cognizant of the job roles of the various individuals participating in a 

training (e.g., probation officers, parole officers, treatment professionals), and 

adjust/provide context to the training accordingly. 

Post-Training Follow-up 

1. Once personnel start conducting assessments, they should be closely monitored initially to 

ensure they are being conducted properly and that staff have a venue to ask questions. For 

example, staff may struggle with asking some questions, scoring certain domains, or 

resolving discrepancies in scoring. Attention to these details shortly after training/rollout 

is needed in order to ensure that efficient and effective routines are established around the 

tools and personnel workflow. 

2. After initial training, ongoing training that continuously works toward reliability should be 

required at the state and/or local level. For example, at the state level, required boosters 

could be provided that would build toward success at planned recertification timeframes. 

It is possible to capitalize on available technology by providing specific refresher modules 

in an online framework in the automated system. At the local level, this should include 

regularly scoring a domain during a staff meeting, holding regular case conferences related 

to the JRNA, or holding communities of practice that come together regularly to discuss 

and practice assessment components. Each of these procedures might include an emphasis 

on more challenging cases/items (e.g., those that are more subjective to score) in order to 

account for the fact that there are some inherently subjective elements in aspects of 

information gathering and interpretation prior to scoring of the assessment. 

3. Initial and ongoing training should include updated training and messaging materials 

periodically that include new information about the risk and needs assessment as it is 

captured for the state/county/agency. For example, once aggregate data are available, these 

should be integrated into training initiatives including staff and unit meetings to solidify 

buy-in and foster continuous quality improvement efforts. 

4. If supervisors are tasked with QA processes after rollout and initial training, agencies 

should ensure that they are trained, certified (if required), and proficient with the 

assessment themselves. As such, supervisors and anyone with QA-related tasks should be 

provided their own QA trainings, booster trainings, and monitoring. This may include 

modules that focus on analysis of such information and linkage to aggregate agency 

objectives. For more information about QA processes, see ‘Monitoring.’

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

330 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Usage 

Optimal risk and needs assessment usage is of the utmost importance in its potential impact 

(or lack thereof) on the outcomes of justice-involved youths. This is evident from multiple aspects 

of the study in terms of the usage cases as well as consideration of possible indirect relationships 

between risk assessment information and recidivism. Practices across the three study states were 

good in some senses (e.g., using risk and needs assessment information to establish supervision 

level), mixed in others (e.g., use of risk and needs assessment information for treatment matching), 

or “next-tier” (e.g., looking at aggregate data or using assessment information to allocate resources 

within an agency). Across the states, practices generally matched written policies. So, increasing 

the breadth of the policies could lead to a direct impact in practice. Under the broad umbrella of 

assessment usage, the next set of recommendations first focuses on the policies and practices 

surrounding the assessment, moves into establishing buy-in among staff, and then the actual 

implementation and usage of assessment information. 

Policies and Practices 

1. State-level policies and best practices that provide a comprehensive view for how the state 

wishes the assessments and their results to be used should be established before initiating 

the adoption of the tool and training on it. The different screeners and assessments should 

be fully mapped out concerning when in the justice process they are recommended to be 

conducted, if and when reassessments should occur, how the assessment information 

should be used, and who should be privy to the assessment results. For example, the 

information from the assessment may be used to determine a youth’s placement at a facility

or in a local treatment program. Having policies that include this detailed mapping will 

ensure that personnel receive clear and consistent messages about the purpose of each 

screening and assessment tool, their suggested timelines, and which should be used for 

reassessment (particularly for residential youths). 

2. At the local level (e.g., court, probation, parole region, detention or institution), 

assessment-specific policies and procedures should be developed for each specific 

locality/facility that is tasked with conducting or utilizing the assessments. These should 

also align with best practices in juvenile justice intervention (i.e., assigning supervision 

level and treatment dosage based on risk level), and fall within the bounds of the broader, 

state-level policies mentioned previously. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

331 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3. Within all of these policies and procedures, specifics regarding timelines for initial 

assessment, reassessment, expectations for developing and updating case plans, and 

expectations for how the assessment information should be used and shared should be 

included based on the audience. For example, these policies and procedures should 

provide: (a) specific guidance for staff and supervisors regarding overrides; (b) suggestions 

for case planning and service referrals (according to each domain and the referral process), 

which may take the form of service mapping; and (c) expectations and suggestions for 

sharing information with outside stakeholders judiciously and effectively. 

4. Leadership should ensure that usage policies are logical, and are tailored to the 

environment in which they are to be monitored. If staff are expected to use assessment 

information to drive treatment and service referral decisions it is important that this is 

clearly laid out in policy and practice manuals and that appropriate options are available. 

5. At the state and local level, the assessment policies and procedures should be readily 

available to staff and the public. This is particularly important in light of recent public 

discussion about risk and needs assessment (and recent controversies about fairness, see 

e.g., Schwalbe, Fraser, Day, & Cooley, 2006). For the public, it would be helpful that 

families and guardians are also provided information about the assessment process and 

how the assessment information is gathered and used throughout a youth’s involvement 

with the juvenile justice system. 

6. Agencies adopting a risk and needs assessment need to plan out in detail how the results 

are expected to be used prior to rollout and training and included in policies and procedures 

(e.g., placement, supervision standards, running of local data reports, case planning, 

treatment referrals, reassessment, sharing of assessment results). This usage policy should 

be specific to the assessment tool being used, in order for the guidelines to be as clear and 

relevant as possible. It is essential that the agencies introducing and utilizing the tools 

recognize the series of interrelated parts in information gathering, decision-making, and 

case management that are a part of this JRNA process. 

7. Usage expectations, and examples of what they look like in practice (e.g., review of an 

actual case), should be included in the assessment policies and procedures, and be 

presented during initial training, ongoing training, and quality assurance checks and 

discussions. This will assist in promoting the full potential of the tool in a well-planned 

and purposeful way. For example, the case plan policy should require those for all youths 

with at least two high or moderate need areas from the tool, and require that they be updated 

at least once a month. 

8. Alternatively, should agencies wish to provide latitude in the use of tool, they should still 

offer minimum usage guidelines. These may include any minimum requirements for 

entering data in the management information system, and placement requirements for state 

facilities. It is important to ensure that this not undermine the integrity of the assessment. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

332 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9. For agencies using multiple assessments, a “decision matrix” may need to be developed to

assist staff in deciding when to use certain assessments (e.g., responsivity assessments, or 

specialized risk and needs assessments). 

10. As referenced in the ‘Planning/Pre-training’ recommendations, it is possible that, as the 

treatment resources in the community increase, so will the perceived usefulness and overall 

staff satisfaction with the tool. Therefore, promoting inter-agency collaboration (through 

the use of MOUs and other inter-agency agreements) is also essential for optimized usage 

practices that are mutually beneficial to youths and agencies. 

Establishing Buy-In 

1. States and local entities should bear in mind key implementation strategies regarding 

rolling out a new initiative (Fixsen, Blase, Naoom, & Wallace, 2009; National 

Implementation Research Network, n.d.). The current study revealed that training was the 

main strategy used to garner buy-in the assessment system. However, training alone is not 

enough to establish buy-in and effectively implement any new initiative. Preimplementation decision-making and communication (noted above in ‘Planning/PreTraining’) is essential in establishing support. 

2. Agency staff may have endured changes in the juvenile justice field, at their agencies, and 

over the course of their tenure in the field. Leadership should anticipate that the assessment 

tool is not being implemented to a “blank slate,” and brainstorm how this may impact the

current implementation effort in their state, county, or agency. 

3. Agencies should be deliberate and strategic in garnering personnel and stakeholder buy-in. 

This is important in the selection and adoption phases (more on the tool selection phase is 

available in Vincent et al. [2012], which, unlike the current study, included implementation 

sites that had yet to select an assessment tool). Further, agencies should prioritize the 

necessary resources for this purpose. This includes personnel (e.g., which staff are tasked 

with this role and providing them time to build relationships and enthusiasm), technology, 

quality assurance, and research/evaluation. 

4. State and/or agency leadership should gather and present to staff the anticipated positive 

consequences of a systematic assessment process, such as allowing staff to build rapport 

with youths and establish that early in the juvenile justice process. In turn, they should be 

clear on how that turns into positive outcomes for youths and for the agency itself. This 

can happen in numerous contexts—introducing each training, in-person site visits to local 

agencies, webinars, pre-recorded videos on state or county websites, and written 

communications through emails and memos. They should also be upfront about changes to 

routine practice that will come with implementation of a tool and be prepared to effectively 

answer questions from staff who might be skeptical. 

5. States and agencies should work collaboratively with the developers of the risk and needs 

assessment tool to ensure that they can provide evidence and data from counties or states 

that previously adopted the tool. This information should be used to demonstrate how the 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

333 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

tool(s) are likely to perform and to help staff anticipate their potential benefits. They should 

be specific in terms of how the results will impact supervision and service delivery. States 

and agencies can show how aggregate data have been used in other counties or states, and 

how it could potentially be used in their state or county. 

6. Relatedly, many of the obstacles to fully embedding risk and needs assessments come after 

initial implementation with application of the risk and needs assessment information to 

case disposition, supervision, and treatment decisions. “Optimal use” cases from peers may

be more effective than statistical data in convincing skeptical agency personnel or even 

reassuring those who are more enthusiastic but would like to improve practice. Video 

testimonials, eventually featuring their own staff, could be used as well. 

7. Researchers and champions for assessment systems should be forthright about the limits of 

actuarial prediction (e.g., risk and needs assessments are not an ‘end-all-be-all’ solution,

which is why overrides are allowed and expected). During this process, staff should able 

to voice their concerns and administrators should be responsive to those concerns. 

8. Agencies should capitalize on enthusiastic staff members who are available to be 

champions of the initiative. These champions will act as influencers for other staff who 

have not bought in to the assessment process, are new to the agency, or in an agency that 

has not yet adopted the tool and process. They should be encouraged to become trainers, 

and/or part of ongoing quality assurance processes. These staff can then present at unit and 

staff meetings to continue to build staff support. 

9. Allow staff and stakeholders who will not be responsible for conducting the assessments 

to attend trainings if they so desire. This will capitalize on their natural buy-in and increase 

institutional knowledge about the assessment process. Consideration for whom these staff 

and stakeholders may be was also mentioned in the ‘Planning/Pre-training’ 

recommendations section, as this may require additional resources and foresight. This falls 

in line with Vincent et al.’s (2012) finding that buy-in from judges was instrumental for 

the success of assessment implementation. 

10. The individuals who conduct and use the assessments will inevitably have valuable insight 

into the process and identify questions that perhaps were not considered in planning and 

initial roll-out. As such, agencies should be open to staff feedback and be willing to adapt 

policies, procedures, and practices according to feedback as appropriate, while maintaining 

general adherence to evidence-based practices around the assessment. 

11. Leadership should be mindful of the unique professional roles within an agency and plan 

for how this may impact staff perceptions of the tool. While some staff are supervisors, and 

therefore may have more ‘ownership’ over the assessment tool and its associated outcomes, 

others (e.g., frontline, or non-supervisory treatment staff) may not have this sense of 

ownership and may need time to adjust to the assessment. Agencies should develop a 

strategy to foster ‘ownership’ of the risk and needs assessment with those who are not in 

supervisory roles. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

334 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

12. Demonstrating an ongoing focus on ensuring the tool is accurate should help alleviate some 

staff concerns about validity and reliability. Continued training and QA are essential to 

these efforts. 

Implementation and Usage Practices 

1. Those in leadership positions should identify key elements of the state/county juvenile 

justice context and be prepared to adjust implementation strategies accordingly (Sullivan, 

2019). For example, some states have a centralized juvenile justice system, while others do 

not. Those states with a decentralized system that want to adopt a statewide risk and needs 

assessment may have to be more persuasive, or somehow incentivize, the case for the 

assessment, providing training, and implementing JRNA in juvenile justice processes. 

2. A data entry and management system should not serve as a reason to impede buy-in for the 

assessment. As such, before the roll-out, the system should be tested by likely users of the 

tool to ensure a user-friendly interface. This should include consideration of the interaction 

of the assessment system and existing case management and information storage tools. 

3. Agencies should strive to adopt and use an information-management system around the 

tool as that will ease information sharing across agencies and collect case-level and 

aggregate data. It is important to determine who should have access to the system and to 

ensure that data are being entered, examined, and results shared with personnel and 

stakeholders according to policy and practice guidelines. 

4. The longer (i.e., non-screener) OYAS assessment tools studied here have responsivity 

areas that staff can mark as either a strength or a barrier. Current training practices often 

treat these as optional. However, these should not be treated as optional and staff should 

be trained to fill out responsivity information in the assessment to optimize use of 

assessments. Further, as needed, they should routinely use additional responsivity 

assessments and work to match youths to staff and/or services based on these factors. 

5. Assessment results should be shared with treatment providers and all appropriate 

stakeholders. Sharing this information helps ensure that stakeholders see themselves as 

partners who are operating from the same base information. Information sharing should be 

systematic, such as allowing these partners shared access to the automated system. 

Alternatively, agencies should be provided hard copies of the relevant assessment results 

at the time of referral, which may come in the form of the assessment itself, pre-disposition 

reports that include the assessment information, or treatment referral packets. Given the 

nature of juvenile justice populations, this should occur in a thoughtful way so as not to 

compromise the youths’ ability to change due to labels or stigma that might be attached to 

certain assessment information. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

335 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

6. Another extremely beneficial practice that was largely missing from the current report was 

using the assessment data to make data driven decisions. As such, planning for this and 

assisting local agencies with aggregating their own data would help ensure full use of and 

risk and needs assessment process. For example, providing aggregate data results by county 

with examples for its potential uses (e.g., specialized units or caseloads, justifying the 

reallocation of resources to fit agency strategy, or developing referral sources in a particular 

area based on the needs of youths). 

7. OYAS information may be used for reasons other than those specifically outlined in past 

research, provided that they align with evidence-based practices. This may help to increase 

the embeddedness of the assessment within an agency. For example, assessment 

information can be used to interface with the youths’ guardians or in determining rewards 

and sanctions based on progress in treatment or specialized programming. 

Monitoring 

Once staff are trained on the risk and needs assessment and usage practices are in place, it 

is necessary to monitor the daily activities of staff via QA processes which were decidedly missing 

in the current study. Many interviewees and survey respondents across the states noted concerns 

about validity and reliability and this may be a direct result of that state of affairs. The quality 

assurance processes that were named were more reminiscent of supervisorial tasks related to 

meeting deadlines, and not the quality of the information gathering process. While agencies 

provided clear examples of buy-in tactics and usage practices at their agencies, there QA practices 

were less common. Systematic QA processes are essential for effective implementation and 

sustainability thereafter. Agencies implementing JRNA should maintain a focus on tangible youth 

and agency-level goals and related quality controls rather than succumbing to bureaucratic drift in 

monitoring (Warner & Havens, 1968). 

Quality Assurance Processes and Local Validation 

1. State-, county-, and agency-level administrators who monitor the rollout of the assessment 

process should remain close to the frontline work and be reachable and responsive to staff 

concerns. For example, at the county level, Chief POs, directors, managers, and supervisors 

will be central in establishing and maintaining these connections. Additionally, focus 

groups or meetings with state-, county-, and facility-level administrators, line staff, and 

stakeholders will facilitate this type of open communication. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

336 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2. QA tasks should go beyond the managerial tasks of a supervisor (e.g., making sure 

reassessments are completed by a certain date). These tasks should include checks on the 

integrity of the assessment process, such as whether it is being conducted properly (i.e., is 

it a valid assessment?), and whether the assessment information is being used to make 

informed decisions about youth cases (e.g., supervision level, case planning, treatment 

referrals). These should reflect considerations of practice at both the individual staff and 

agency levels. 

3. Formal quality assurance efforts should take place to ensure the JRNA is being carried out 

correctly, that items are scored accurately, and the assessments are a reliable indicator of a 

youth’s risk level. In order to boost interrater reliability, computerized learning modules 

could be developed and required for each certified assessor. These modules should use 

scoring vignettes and provide explanations for each item not scored correctly, before 

prompting the user to score it correctly. 

4. After acceptable interrater reliability levels have been achieved, validation (both at the state 

and local level) should be planned for and conducted. Planning should involve realistic 

time frames for completion. As validation occurs, the results should be clearly 

communicated to staff. This should include areas for further development and concrete 

steps for continuous quality improvement. 

5. Agencies should create a mechanism for receiving and answering staff questions in a timely 

and consistent fashion. This could be accomplished through the establishment of an OYAS 

task force that includes members from agencies at various decision-points in the juvenile 

justice process, and creating a frequently asked questions document that is updated and 

available to all staff to consult as needed. Should agencies have an automated system, these 

questions can be added to the code book (which would ideally “pop up” as a staff member 

scrolls over the question). 

6. Agencies should consider their ability to provide one “point-person” for all QA questions 

or concerns, though QA processes can and should be carried out by a team of individuals. 

Having one point person may streamline the process and help ensure consistency in 

responses. 

7. States and local agencies should anticipate and plan for a potential drop in enthusiasm for 

the assessment after its initial implementation. Turning a new process into a daily practice 

requires time and effort. Administrators should recognize that the positive impact or 

usefulness of the tool may not be initially felt by frontline staff, and continuing with 

consistent messages and identifying and highlighting success stories will be key to 

furthering buy-in. 

8. QA practices should be clearly delineated to ensure the related policies and procedures 

regarding information usage are being met. For example, supervisors may be tasked with 

reviewing a certain percentage of case plans for each staff member for content and quality. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

337 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9. Training on basic data interpretation and mapping to court and corrections inputs/outputs 

should be developed in order to optimize agency-level usage and planning using aggregate 

risk and needs assessment and case outcome information. 

Future Research and Evaluation 

Many of the recommendations above are predicated on a strong and comprehensive 

evidence base for the implementation and use of JRNA. Research on the implementation of risk 

and needs assessment tools is still minimal relative to the amount of attention focused on the 

association between risk classification and recidivism (i.e., “predictive validity”). This leaves an 

important gap in the evidence-base as understanding implementation is a major element of 

successfully translating such research into practice—especially when the objective is to alter 

decision-making practice in a complex juvenile justice eco-system. In that context, we also make 

a series of recommendations for the research and evaluation community who can indirectly inform 

the effective implementation and use of risk and needs assessment in the juvenile justice system. 

Building on the recommendations for Training, Usage, and Monitoring above, we add several 

recommendations for Future Research and Evaluation: 

1. Researchers should use a variety of methods to both objectively identify markers of 

good practice and areas that requires improvement and to understand the perspectives 

of those in the field who must administer and make decisions based on the information 

from these tools. Full population data collected from the assessment system can be 

useful in providing a global view of an agency’s caseload to help identify patterns of

need and predict outcomes of strategic shifts. At the same time, narrative data on cases 

and perspectives of juvenile justice personnel can be beneficial in identifying how that 

information impacts case decisions and the day-to-day work in juvenile justice. 

2. Researchers must better balance answering global and more localized questions about 

JRNA. While studies show that structured assessment tools work in a general sense, 

this investigation and another recent OJJDP-funded study (see Baird et al., 2013) 

suggest a need to look more deeply at questions beyond simply whether risk assessment 

results are correlated with recidivism. A 360° evaluation agenda for JRNA would 

include a host of different types of validity and reliability studies (e.g., assessment of 

domain-level integrity, optimal scoring protocols). The next generation in applied risk 

and needs assessment in the juvenile justice system seems to be focused on use of this 

information for treatment matching and monitoring of change. Those areas have not 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

338 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

been covered all that thoroughly in the research to date and therefore more studies using 

both case and variable based methods is necessary to develop a viable evidence base to 

inform effective practice at that level. 

3. It is essential to study JRNA as a series of interrelated parts as they are embedded in 

the juvenile justice system. In this way the study of risk and needs assessment must be 

more aware of the implementation context and understand the mechanisms and 

outcomes associated with JRNA. In particular, we recommend more studies using 

varied methods to assess the underlying logic model of JRNA, which assumes a link 

between the risk assessment, decisions about disposition and treatment, and justice and 

developmental outcomes. It is also important to determine how that process plays out 

in different agency environments—especially relative to caseload size and resource 

restrictions. Further investigations of that sort would also be beneficial in potentially 

confirming (or not) our findings regarding risk-treatment matching processes in the 

current data. 

4. Practice-informed research should evolve to ensure that organizations have the tools to 

take next steps in implementation and use of JRNA. This might include more attention 

to examining case studies to better understand the (in)effective application of the 

information gleaned from the assessment process. Presently, much of the information 

on risk and needs assessment is based on the average case and therefore may not hold 

as much use to practitioners as researchers might hope. Additionally, data driven 

decision making has increasingly taken hold in justice organizations. With that in mind, 

it would be useful to develop research focused on effective agency-level use of risk and 

needs assessment data for organizational planning and continuous quality 

improvement. 

5. Validation research should more thoroughly consider usage questions associated with 

the subpopulations that are involved in the juvenile justice system and be sure to 

conduct both predictive and measurement validity research on relevant groups. Some 

of this work is now underway, but—in light of other juvenile justice initiatives such as 

trauma-informed intervention—it is important to include females, minority youths who 

are disproportionately represented in the system, and youths at different developmental 

stages and ages. Researchers should also undertake more intensive study of prediction 

errors in their samples to fine-tune risk models and scoring systems. This more focused 

analysis becomes increasingly important as agencies attempt to map risk and needs 

information from particular domains to treatment planning and monitoring. This is not 

simply an academic question as data collected in this study show that the strength of 

evidence in these areas may have a downstream impact on practitioners’ perceptions 

and use of the tools. Some respondents used “researcher” language in raising concerns

about validity and reliability. This was surprising, but perhaps should not have been 

given the degree to which recent juvenile justice initiatives have been informed—at 

least nominally—by research. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

339 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

6. Building on the previous point and our findings in the treatment matching mediation 

analyses, validation studies of juvenile risk and needs assessments have been heavily 

focused on recidivism as opposed to holistically considering the assessment process in 

evaluating the validity, reliability, and utility of tools. If the latest generation of tools 

is meant to inform case management via treatment and service allocation, it is essential 

that research on these tools considers questions that often sit in the periphery right now. 

For example, at least in research and development, subdomain scores on general risk 

tools should be considered against relevant criteria that can speak to their ability to 

inform an understanding of degree of need in that specific area (e.g., a substance use 

risk domain could be checked against a full substance abuse inventory to check for at 

least partial recovery of information to establish criterion validity). From there this 

could be checked against the degree to which that process and related treatment can 

effect change in that level of need. 

7. The results from this study also highlight three other more specific areas for future 

research. First, JRNA tools are intended in part to guide referrals to treatment. 

However, indicators in general risk and need assessment tools are limited in scope and 

not always specific to key need areas. Research should examine whether specialized 

need assessments should be conducted by court actors, at the time the JRNA flags the 

need area, or if these are better assessed by referring a youth to a treatment provider. 

Essentially, is there a preferred practice in terms of youth benefits and system 

effectiveness and efficiency? A second question pertains to the comprehensive written 

policies for JRNA in terms of how those may drive practice. In the vein of previous 

implementation science, researchers should study the degree to which such formal 

factors impact buy-in and sustainability relative to more process-oriented approaches. 

Third, elements of the juvenile justice personnel data raise questions about the need 

for more widespread sharing of assessment information across different points of the 

juvenile justice system (and its partners). More research is needed about the possible 

benefits and costs of more widespread sharing of assessment information about youths. 

Greater attention to these and related questions would help to create a more comprehensive 

and rigorous evidence base for the next generation of applied risk and needs assessment in the 

juvenile justice system. Many of the methods are in place to conduct these studies effectively, 

which means there is a great deal more that can be learned about the implementation, use, and 

outcomes of JRNA in the near future. 

Conclusion 

Gaining a more comprehensive view of JRNA as it is occurring in the field and its impact 

on youths’ cases was the overarching goal of this study. In turn, we endeavored to provide

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

340 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

practitioners, policymakers, and researchers with some useful insights about that process—which 

has been the subject of a lot of research but surprisingly narrow in the scope of questions asked. 

As a state or agency is adopting new tools, or looking to improve their current risk and needs 

assessment processes, they should consider key implementation facilitators: careful planning that 

includes establishing support for the risk and needs assessment amongst a variety of stakeholders, 

creating realistic but detailed usage and implementation guidelines and policies and procedures, 

sharing information with all those who will be impacted by the use of the assessment, sufficient 

training and post-training support, and beneficial quality assurance practices that will help ensure 

the risk and needs assessment is completed correctly and used to its full potential. In usage and 

process studies, we found that use of JRNA often reflected what was intended, but at other times 

the picture was more complicated and suggested areas for further consideration and improvement. 

This is only natural as a field adopts and takes an approach to scale. It also reflects the tremendous 

variability that is likely to occur across States and local agencies. 

The ubiquity of this practice in the current juvenile justice system—and the fact that it is a 

platform for other decisions and initiatives—underscores its possible impact and suggests that it is 

important to continually evaluate and improve these practices as they are applied to youths. 

Collectively, the findings from the study—both supportive and critical—offer some insight on how 

risk and needs assessment can be used as an engine to help generate better outcomes for youths 

and the juvenile justice agencies whom they encounter. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

341 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

References 

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidencebased practice implementation in public service sectors. Administration and Policy in 

Mental Health and Mental Health Services Research, 38(1), 4-23. 

Abram, K. M., Teplin, L. A., McClelland, G. M., & Dulcan, M. K. (2003). Comorbid psychiatric 

disorders in youth in juvenile detention. Archives of general psychiatry, 60(11), 1097-

1108. 

Andrews, D. A., Zinger, I., Hoge, R. D., Bonta, J., Gendreau, P., & Cullen, F. T. (1990). Does 

correctional treatment work? A clinically relevant and psychologically informed meta‐

analysis. Criminology, 28(3), 369-404. 

Andrews, D. A., Bonta, J., & Wormith, J. S. (2006). The recent past and near future of risk and/or 

need assessment. Crime & Delinquency, 52(1), 7-27. 

Andrews, D. A., & Bonta, J. (2010). The psychology of criminal conduct (5th ed.). New Providence, 

NJ: Matthew Bender & Company, INC. 

Andrews, D. A., Bonta, J., Wormith, J. S., Guzzo, L., Brews, A., Rettinger, J., & Rowe, R. (2011). 

Sources of variability in estimates of predictive validity: A specification with level of 

service general risk and need. Criminal Justice and Behavior, 38(5), 413-432. 

Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias: There’s software used

across the county to predict future criminals and it’s biased against blacks. ProPublica. 

Retrieved from: https://www.propublica.org/article/machine-bias risk-assessments-incriminal-sentencing 

Arnold, E. M., Walsh, A. K., Oldham, M. S., & Rapp, C. A. (2007). Strengths-based case 

management: Implementation with high risk youth. Families in Society, 88, 86-94. 

Atkins, D. L., Pumariega, A. J., Rogers, K., Montgomery, L., Nybro, C., Jeffers, G., & Sease, F. 

(1999). Mental health and incarcerated youth. I: Prevalence and nature of 

psychopathology. Journal of Child and Family Studies, 8(2), 193-204. 

Backer, T.E. (1995). Assessing and enhancing readiness for change: Implications for technology 

transfer. In T.E. Backer, S.L. David, & G. Soucy (Eds.) Reviewing the behavioral science 

knowledge base on technology transfer. Rockville, MD: National Institute on Drug Abuse. 

Baglivio, M.T., Wolff, K.T., Jackowski, K., & Greenwald, M.A. (2017). A Multilevel Examination 

of Risk/Need Change Scores, Community Context, and Successful Reentry of Committed 

Juvenile Offenders. Youth Violence and Juvenile Justice, 15: 38-61. 

Baglivio, M. T., & Jackowski, K. (2013). Examining the validity of a juvenile offending risk 

assessment instrument across gender and Race/Ethnicity. Youth Violence and Juvenile 

Justice, 11(1), 26-43. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

342 
 

 

 

 

 

 

 

Baird, C., Healy, T., Johnson, K., Bogie, A., Dankert, E. W., & Scharenbroch, C. (2013). A 

comparison of risk assessment instruments in juvenile justice. Madison, WI: National 

Council on Crime and Delinquency. 

Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social 

psychological research: Conceptual, strategic, and statistical considerations. Journal of 

personality and social psychology, 51(6), 1173. 

Barnes, A. R., Campbell, N. A., Anderson, V. R., Campbell, C. A., Onifade, E., & Davidson, W. 

S. (2016). Validity of initial, exit, and dynamic juvenile risk assessment: An examination 

across gender and race/ethnicity. Journal of Offender Rehabilitation, 55(1), 21-38. 

Barnoski, R. (2004). Washington State juvenile court assessment manual, version 2.1. Olympia, 

WA: Washington State Institute for Public Policy. 

Bell, D. E., Raiffa, H, & Tversky, A. (1988). Decision making: Descriptive, normative, and 

prescriptive interactions. Cambridge, MA: Cambridge University Press. 

Berman, G., & Fox, A. (2010). Trial and error in criminal justice reform: Learning from failure. 

Washington, DC: Urban Institute. 

Bishop, D. M., Leiber, M., & Johnson, J. (2010). Contexts of decision making in the juvenile 

justice system: An organizational approach to understanding minority overrepresentation. 

Youth Violence and Juvenile Justice, 8, 213-233. 

Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., & 

Emshoff, J. G. (1987). The fidelity-adaptation debate: Implications for the implementation 

of public sector social programs. American Journal of Community Psychology, 15(3), 253-

268. 

Bland J. M., & Altman D. G. (1997). Statistics notes: Cronbach's alpha. British Medical Journal, 

314, 572-572. 

Bonta, J. (1996). Risk-needs assessment and treatment. In A. T. Harland (Ed.), Choosing 

correctional options that work: Defining the demand and evaluating the supply (pp. 18-

32). Thousand Oaks, CA: Sage Publications, Inc. 

Bonta, J. (2002). Offender risk assessment: Guidelines for selection and use. Criminal Justice and 

Behavior, 29(4), 355-379. 

Bonta, J. & Andrews, D. A. (2017). The psychology of criminal conduct. New York, NY. 

Routledge. 

Bray, T. M., Sample, L. L., and Kempf-Leonard, K. (2005). “Justice by geography”: Racial

disparity and juvenile courts. In D. Hawkins & K. Kempf-Leonard (Eds.), Our Children, 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

343 
 

 

 

 

 

 

 

 

 

 

 

Their Children, edited by Darnell Hawkins and Kimberly Kempf-Leonard (pp. 270-299). 

Chicago: The University of Chicago Press. 

Brown, T. (2006) Confirmatory factor analysis for applied research. New York, NY: The 

Guildford Press. 

Cohen, J. (1992). A power primer. Psychological bulletin, 112(1), 155-159. 

Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for 

developing grounded theory (3rd ed.). Los Angeles, CA: Sage Publications, Inc. 

Corrigan, P. W., Williams, O. B., Mccracken, S. G., Kommana, S., Edwards, M., & Brunner, J. 

(1998). Staff attitudes that impede implementation of behavioral treatment programs. 

Behavior Modification, 22, 548-562. 

Cottle, C. C., Lee, R. J., & Heilbrun, K. (2001). The prediction of criminal recidivism in juveniles: 

A meta-analysis. Criminal justice and behavior, 28(3), 367-394. 

Cronbach L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 

297-334. 

Childs, K. K., Ryals Jr, J., Frick, P. J., Lawing, K., Phillippi, S. W., & Deprato, D. K. (2013). 

Examining the validity of the Structured Assessment of Violence Risk in Youth (SAVRY) 

for predicting probation outcomes among adjudicated juvenile offenders. Behavioral 

Sciences & The Law, 31(2), 256-270. 

Davis, J. & Sorensen, J. R. (2013). Disproportionate minority confinement of juveniles: A national 

examination of Black-White disparity in placements, 1997-2006. Crime & Delinquency, 

59(1), 115-139. 

Dowden, C., & Andrews, D. A. (2004). The importance of staff practice in delivering correctional 

treatment: A meta-analytic review of core correctional practice. International Journal of 

Offender Therapy and Comparative Criminology, 48, 203-214. 

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the 

influence of implementation on program outcomes and the factors affecting 

implementation. American journal of community psychology, 41(3-4), 327-350. 

Duwe, G., & Rocque, M. (2017). Effects of automating recidivism risk assessment on reliability, 

predictive validity, and return on investment (ROI). Criminology & Public Policy, 16(1), 

235-269. 

Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. 

Research on Social Work Practice, 19(5), 531-540. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

344 
 

 

 

 

 

 

 

 

 

 

 

 

 

Fixsen, D., Blase, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidencebased programs. Exceptional Children, 79(2), 213-230. 

Flores, A. W., & Travis III, L. F. (2004). Case classification for juvenile corrections: An 

assessment of the Youth Level of Service/Case Management Inventory (YLS/CMI), final 

report. 

Flores, A. W., Lowenkamp, C. T., Holsinger, A. M., & Latessa, E. J. (2006). Predicting outcome 

with the Level of Service Inventory-Revised: The importance of implementation integrity. 

Journal of Criminal Justice, 34(5), 523-529. 

Gendreau, P., Little, T., & Goggin, C. (1996). A meta‐analysis of the predictors of adult offender 

recidivism: What works! Criminology, 34(4), 575-608. 

Gottfredson, S. D., & Moriarty, L. J. (2006). Statistical risk assessment: Old problems and new 

applications. Crime & Delinquency, 52(1), 178-200. 

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of 

innovations in service organizations: Systematic review and recommendations. The 

Milbank Quarterly, 82(4), 581-629. 

Grisso, T., Vincent, G., & Seagrave, D. (Eds.). (2005). Mental health screening and assessment in 

juvenile justice. Guilford Press. 

Hayes, A. F.(2013). Introduction to mediation, moderation, and conditional process analysis: A 

regression‐based approach. New York, NY: The Guilford Press. 

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: 

Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. 

Hubbard, D. J., Travis, L. F., & Latessa, E. J. (2001). Final Report: Case Classification in 

Community Corrections: A National Survey of the State of the Art. Washington, DC: 

National Institutes of Justice, NIJ 98-IJ-CX-0008. 

Hyland, N. (2018). Delinquency Cases in Juvenile Court, 2014. Washington, DC: Office of 

Juvenile Justice and Delinquency Prevention. 

Johnson, K., Wagner, D., & Matthews, T. (2002). Missouri juvenile risk assessment revalidation 

report. Madison, WI: National Council on Crime and Delinquency. 

Klein, M. W. (1979). Deinstitutionalization and diversion of juvenile offenders: A litany of 

impediments. Crime and Justice, 1, 145-20 

Knight, D. K., Becan, J. E., Landrum, B., Joe, G. W., & Flynn, P. M. (2014). Screening and 

assessment tools for measuring adolescent client needs and functioning in substance abuse 

treatment. Substance use & misuse, 49(7), 902-918. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

345 
 

 

 

 

 

 

 

 

 

 

 

 

Labreque, R. M. & Schweitzer, M. (2014). Targeted RECLAIM 2012 University of Cincinnati 

Outcome Study January 2012 to December 31, 2012. Cincinnati, OH: Center for Criminal 

Justice Research, University of Cincinnati. 

Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. 

Biometrics, 33, 159-174. 

Latessa, E., Lovins, B., & Ostrowski, K. (2009). The Ohio Youth Assessment System. Cincinnati, 

OH: Center for Criminal Justice Research, University of Cincinnati. 

Latessa, E. J., Listwan, S. J., & Koetzle, D. (2014). What works (and doesn’t) in reducing 

recidivism. Cincinnati, OH: Anderson. 

Laub, J.H. (2016). Life course research and the shaping of public policy. In Shanahan, M., 

Mortimer, J.T. & Johnson, M.K. (Eds.), Handbook of the Life Course, Volume 2. New 

York: Springer. 

Lipsey, M. W. (1999). Can intervention rehabilitate serious delinquents? The Annals of the 

American Academy of Political and Social Science, 564(1), 142-166. 

Lovins, B., & Latessa, E. (2013). Creation and validation of the Ohio Youth Assessment System 

(OYAS) and strategies for successful implementation. Justice Research and Policy, 15(1), 

67-93. 

Luong, D., & Wormith, J. S. (2011). Applying risk/need assessment to probation practice and its 

impact on the recidivism of young offenders. Criminal Justice and Behavior, 38(12), 1177-

1199. 

MacCallum, R.C., Browne, M.W., & Sugawara, H., M. (1996). Power analysis and 

determination of sample size for covariance structure modeling. Psychological Methods, 

1, 130-149. 

Maloney, D., Romig, D., & Armstrong, T. (1988). Juvenile probation: The balanced approach. 

Reno, NV: National Council of Juvenile and Family Court Judges. 

McCafferty, J. T. (2013). The predictive validity of the Ohio Youth Assessment System-Disposition 

Instrument: A revalidation study (Doctoral dissertation). Retrieved from Retrieved from 

https://etd.ohiolink.edu/. 

McCafferty, J.T. (2016). Unjust disparities? The impact of race on juvenile risk assessment 

outcomes. Criminal Justice Policy Review, 17, 330-342. 

McCafferty, J., Newsome, J., & Sullivan, C.J. (2017). Study of State 3 Youth Assessment System 

Residential Tool. Cincinnati, OH: University of Cincinnati Center for Criminal Justice 

Research. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

346 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

McClelland, G. M., Teplin, L. A., & Abram, K. M. (2004). Detection and prevalence of substance 

use among juvenile detainees. US Department of Justice, Office of Justice Programs, 

Office of Juvenile Justice and Delinquency Prevention. 

McCuller, W. J., Sussman, S., Holiday, K., Craig, S., & Dent, C. W. (2002). Tracking procedures 

for locating high risk youth. Evaluation & The Health Professions, 25(3), 345-362. 

McGrath, A., & Thompson, A. P. (2012). The relative predictive validity of the static and dynamic 

domain scores in risk-need assessment of juvenile offenders. Criminal Justice and 

Behavior, 39(3), 250-263. 

Mears, D. P. (2012). Prison experience. Journal of Criminal Justice, 40(5), 345-347. 

Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: 

A synthesis of critical steps in the implementation process. American Journal of 

Community Psychology, 50(3-4), 462-480. 

Muthѐn, L., & Muthѐn, B. (1998-2018). MPlus User’s Guide. Los Angeles, CA: Muthѐn &

Muthѐn.

National Implementation Research Network (n.d.). Implementation drivers. Web document 

available at https://nirn.fpg.unc.edu/learn-implementation/implementation-drivers 

National Research Council [NRC] (2013). Reforming juvenile justice: A developmental approach. 

Committee on Assessing Juvenile Justice Reform, Bonnie, R. J., Johnson, R. L., Chemers, 

B. M., & Schuck, J. A. (Eds.). Committee on Law and Justice, Division of Behavioral and 

Social Sciences and Education. Washington, DC: The National Academies Press. 

Nelson, R. J., & Vincent, G. M. (2018). Matching services to criminogenic needs following 

comprehensive risk assessment implementation in juvenile probation. Criminal Justice and 

Behavior, 45(8), 1136-1153. 

Newman, J. C., Des Jarlais, D. C., Turner, C. F., Gribble, J., Cooley, P., & Paone, D. (2002). The 

differential effects of face-to-face and computer interview modes. American Journal of 

Public Health, 92(2), 294-297. 

Nissen, L. (2006). Bringing strength-based philosophy to life in juvenile justice. Reclaiming 

Children and Youth, 15, 40-46. 

Olver, M. E., Stockdale, K. C., & Wormith, J. S. (2009). Risk assessment with young offenders: 

A meta-analysis of three assessment measures. Criminal Justice and Behavior, 36(4), 329-

353. 

Onifade, E., Davidson, W., & Campbell, C. (2009). Risk assessment: The predictive validity of 

the youth level of service case management inventory with African Americans and girls. 

Journal of Ethnicity in Criminal Justice, 7(3), 205-221. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

347 
 

 

 

 

 

 

 

 

 

Orbis Partners Inc. (2007). Long-term validation of the Youth Assessment and Screening 

Instrument (YASI) in New York State juvenile probation. 

Patton, M.Q. (2001). Qualitative Evaluation Research Methods, 3rd Edition. Thousand Oaks, CA: 

Sage. 

Perrault, R., Vincent, G., & Guy, L. (2017). Are risk assessments racially biased? Field study of 

the SAVRY and YLS/CMI in probation. Psychological Assessment, 29(6), 664-678. 

Peterson-Badali, M., Skilling, T., & Haqanee, Z. (2015). Examining implementation of risk 

assessment in case management for youth in the justice system. Criminal Justice and 

Behavior, 42(3), 304-320. 

Pope, C. E. & Leiber, M. J. (2005). Disproportionate minority confinement/contact (DMC): The 

federal initiative. In D. Hawkins & K. Kempf-Leonard (Eds.), Our Children, Their 

Children, edited by Darnell Hawkins and Kimberly Kempf-Leonard (pp. 300-347). 

Chicago: The University of Chicago Press. 

Pratt, C. C., & Hernandez, R. (2003). Building Results Through Community Mobilization: From 

Wellness Goals to Community Outcomes for Oregon's Children, Youth and Families. 

Salem, OR: Oregon Commission on Children and Families. 

Puzzanchera, C. (2018). Juvenile arrests, 2016. Washington, DC: U.S. Department of Justice: 

Office of Juvenile Justice and Delinquency Prevention. 

Puzzanchera, C., & Hockenberry, S. (2018). Characteristics of Delinquency Cases Handled in 

Juvenile Court, 2015. Washington, DC: U.S. Department of Justice: Office of Juvenile 

Justice and Delinquency Prevention. 

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data 

analysis methods (Second ed.). Thousand Oaks, CA: Sage. 

Rembert, D. A., Henderson, H., & Pirtle, D. (2014). Differential racial/ethnic predictive validity. 

Youth Violence and Juvenile Justice, 12(2), 152-166. 

Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, 

Cohen’s d, and r. Law and Human Behavior, 29, 615–620. 

Rocque, M. & Plummer-Beale, J. (2014). In the eye of the beholder? An examination of the interrater reliability of the LSI-R and YLS/CMI in a correctional agency. Journal of Criminal 

Justice, 42, 568-578. 

Rossegger, A., Gerth, J., Seewald, K., Urbaniok, F., Singh, J. P., & Endrass, J. (2013). Current 

obstacles in replicating risk assessment findings: A systematic review of commonly used 

actuarial instruments. Behavioral Sciences & The Law, 31(1), 154-164. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

348 
 

 

 

 

 

 

 

 

 

 

 

 

Schneider, A .L., Ervin, L., & Snyder-Joy, Z. (1996). Further exploration of the flight from 

discretion: The role of risk/need instruments in probation supervision decision. Journal of 

Criminal Justice, 24, 109-121. 

Shufelt, J. L., & Cocozza, J. J. (2006). Youth with mental health disorders in the juvenile justice 

system: Results from a multi-state prevalence study (pp. 1-6). Delmar, NY: National Center 

for Mental Health and Juvenile Justice. 

Schwalbe, C. S., Fraser, M. W., Day, S. H., & Cooley, V. (2006). Classifying juvenile offenders 

according to risk of recidivism: Predictive validity, Race/Ethnicity, and gender. Criminal 

Justice and Behavior, 33(3), 305-324. 

Schwalbe, C. S. (2007). Risk assessment for juvenile justice: A meta-analysis. Law and Human 

Behavior, 31(5), 449. 

Schwalbe, C. S., Fraser, M. W., & Day, S. H. (2007). Predictive validity of the Joint Risk Matrix 

with juvenile offenders: A focus on gender and race/ethnicity. Criminal Justice and 

Behavior, 34(3), 348-361. 

Scott, E. S., & Steinberg, L. D. (2009). Rethinking juvenile justice. Cambridge, MA: Harvard 

University Press. 

Seave, P. L. (2011). Evidence‐based practices reduce juvenile recidivism: Can state government 

effectively promote implementation among probation departments? American Journal of 

Community Psychology, 48(1-2), 138-140. 

Shlonsky, A., & Wagner, D. (2005). The next step: Integrating actuarial risk assessment and 

clinical judgment into an evidence-based practice framework in CPS case management. 

Children and Youth Services Review, 27(4), 409-427. 

Shook, J. J. & Sarri, R. C. (2007). Structured decision making in juvenile justice: Judges' and 

probation officers' perceptions and use. Children and Youth Services Review, 29, 1335-

1351. 

Singh, J. P., Desmarais, S. L., Sellers, B. G., Hylton, T., Tirotti, M., & Van Dorn, R. A. (2014). 

From risk assessment to risk management: Matching interventions to adolescent offenders' 

strengths and vulnerabilities. Children and Youth Services Review, 47, 1-9. 

Silver, E., & Miller, L. L. (2002). A cautionary note on the use of actuarial risk assessment tools 

for social control. Crime & Delinquency, 48(1), 138-161. 

Sperber, K., & Lowenkamp, C. T. (2017). Dosage is more than just counting program hours: The 

importance of role-playing in treatment outcomes. Journal of Offender Rehabilitation, 56, 

433-451. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

349 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Steinhart, D. (2006). A practice guide to juvenile detention reform: Juvenile detention risk 

assessment. Baltimore: Annie E. Casey Foundation. 

Sullivan, C.J. (2019). Taking juvenile justice seriously: Developmental insights and system 

challenges. Philadelphia, PA: Temple University Press 

Sullivan, C. J., Mueller, D. J., Gann, S. M., Spiegel, S. N., & McManus, H. D. (2016). Weapon 

and drug offenses and juvenile disproportionate minority contact: An impact assessment 

and practical discussion. Journal of Crime and Justice, 39(1), 107-130. 

Taxman, F. S., Young, D. W., Wiersema, B., Rhodes, A. & Mitchell, S. (2007). The national 

criminal justice treatment practices survey: Multilevel survey methods and procedures. 

Journal of Substance Abuse Treatment, 32(3), 225-238. 

Thornberry, T. P., & Krohn, M. D. (2000). The self-report method for delinquency and crime. 

Criminal Justice, 4, 33-83. 

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 

185(4157), 1124-1131. 

Valeri, L. & VanderWeele, T. J. (2013). Mediation analysis allowing for exposure-mediator 

interactions and causal interpretation: Theoretical assumptions and implementation with 

SAS and SPSS macros. Psychological Methods, 18(2), 137-150. 

VanderWeele, T. (2015). Explanation in causal inference: methods for mediation and interaction. 

Oxford, UK: Oxford University Press. 

Van der Knaap, L. M., Leenarts, L. E. W., Born, M. P., & Oosterveld, P. (2012). Reevaluating 

interrater reliability in offender risk assessment. Crime & Delinquency, 58, 147-163. 

Van Voorhis, P. & Salisbury, E. (2014). Correctional counseling and rehabilitation. Cincinnati, 

OH: Anderson. 

Vaswani, N., & Merone, L. (2013). Are there risks with risk assessment? A study of the predictive 

accuracy of the Youth Level of Service–Case Management Inventory with young offenders 

in Scotland. The British Journal of Social Work, 44(8), 2163-2181. 

Vieira, T. A., Skilling, T. A., & Peterson-Badali, M. (2009). Matching court-ordered services with 

treatment needs: Predicting treatment success with young offenders. Criminal Justice & 

Behavior, 36(4), 385-401. 

Vincent, G. M., Chapman, J., & Cook, N. E. (2011). Risk-needs assessment in juvenile justice: 

Predictive validity of the SAVRY, racial differences, and the contribution of needs factors. 

Criminal Justice and Behavior, 38(1), 42-62. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

350 
 

 

 

 

 

 

 

 

 

Vincent, G. M., Guy, L. S., & Grisso, T. (2012). Risk assessment in juvenile justice: A guidebook 

for implementation. MacArthur Foundation. 

Vincent, G. M., Guy, L. S., Gershenson, B. G., & McCabe, P. (2012). Does risk assessment make 

a difference? Results of implementing the SAVRY in juvenile probation. Behavioral 

Sciences & The Law, 30(4), 384-405. 

Vincent, G. M., Paiva-Salisbury, M. L., Cook, N. E., Guy, L. S., & Perrault, R. T. (2012). Impact 

of risk/needs assessment on juvenile probation officers’ decision making: Importance of

implementation. Psychology, Public Policy, and Law, 18, 549–576. 

Vincent, G. M., Guy, L. S., Perrault, R. T., & Gershenson, B. (2018). Risk assessment matters, but 

only when implemented well: a multisite study in juvenile probation. Law & Human 

Behavior, 40(6), 683-696). 

Vincent, G. M., Sullivan, C.J., Sullivan, C.C., Guy, L., Latessa, E., Tyson, J., & Adams, B. (2018). 

Studying drivers of risk and needs assessment instrument implementation in juvenile 

justice. Washington, DC: Office of Juvenile Justice and Delinquency Prevention. 

Vose, B., Cullen, F. T., & Smith, P. (2008). The empirical status of the Level of Service Inventory. 

Federal probation, 72(3), 22-29. 

Wachter, A. (2015). Statewide risk assessment in juvenile probation. JJGPS StateScan. Pittsburgh, 

PA: National Center for Juvenile Justice. 

Ward, T., & Brown, M. (2004). The good lives model and conceptual issues in offender 

rehabilitation. Psychology, Crime & Law, 10, 243-257. 

Warner, W. K., & Havens, A. E. (1968). Goal Displacement and the Intangibility of Organizational 

Goals. Administrative Science Quarterly, 12(4). 539-555. 

Wasserman, G. A., McReynolds, L. S., Lucas, C. P., Fisher, P., & Santos, L. (2002). The voice 

DISC-IV with incarcerated male youths: Prevalence of disorder. Journal of the American 

Academy of Child & Adolescent Psychiatry, 41(3), 314-321. 

Welsh, B. C., Sullivan, C. J., & Olds, D. L. (2010). When early crime prevention goes to scale: A 

new look at the evidence. Prevention Science, 11(2), 115-125. 

Wormith, S. J., Hogg, S., Guzzo, L. (2012). Predictive validity of a general risk/needs assessment 

inventory on sexual offender recidivism and an exploration of the professional override. 

Criminal Justice and Behavior, 39(12), 1511-1538. 

Wright, J. D., Allen, T. L., & Devine, J. A. (1995). Tracking non-traditional populations in 

longitudinal studies. Evaluation and Program Planning, 18(3), 267-277. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

351 
 

 

 

 

Appendices 

Appendix A. Juvenile Justice Personnel Semi-Structured Interview Guide 

Agency & Staff Characteristics 

To start, we want to get a sense of your role in the agency/facility and the practices of the 

agency/facility. 

1. What is your job title? 

2. How long have you worked for this agency/facility? 

3. Could you tell me about the types of tasks you regularly perform in your role or describe your 

daily responsibilities as ____________ [INSERT JOB TITLE HERE]? 

4. Please describe your involvement with the OYAS used by your agency/facility. 

5. In your opinion, how would you characterize the agency’s/facility’s current status with respect 

to the assessment process? On a scale from 1 to 5, with 1 being at a "planning" stage and 5 

being "full implementation," how would you rate the agency's/facility's current status with 

respect to the assessment process? 

a. Are there particular benchmarks that you have achieved or things that you have 

observed that led you to this response? 

6. Please tell me a little bit about your agency/facility. 

7. Can you describe the type(s) of youths served by this agency/facility? 

8. What do you view as the most challenging cases for your agency/facility? 

9. What specific programs does your agency/facility offer? 

10. Does your agency use referral agencies/facilities to provide treatment services to youths? If 

so, please describe the type(s) of treatment services these agencies/facilities provide. 

11. What is the approximate number of youths served by this agency/facility per year? 

12. Does your agency/facility have specialized caseloads/programs? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

352 
 

 

☐ No ☐ Yes ☐ Don’t Know

a. If yes, what types of specialized caseloads/programs do you have? [CHECK ALL THAT 

APPLY] 

☐ Drug/Substance Abuse Offender ☐ High Risk 

☐ Sex Offender ☐ Truancy 

☐ Family ☐ Electronic Monitoring 

☐ Mentally Disordered ☐ Violent Offender 

☐ Diversion ☐ Other (please specify) 

☐ Low Risk 

13. What is the process by which juveniles reach your agency/facility (i.e., how are youths referred 

to your agency/facility or how do they get to your agency/facility)? 

14. Do you think this process runs smoothly? 

**Note other concerns raised in this section: 

General Agency/Facility Approach to Youths Assessment 

The first few questions in this section will focus on your general thoughts regarding risk and 

needs assessment. The remaining items will focus on your agency's/facility's broader risk and 

needs assessment practices specific to the OYAS. 

15. On a scale from 1 to 5, with 1 being you do not at all agree and 5 being you completely agree, 

rate your level of agreement with the following statement: risk and needs assessments enhance 

fairness in decision-making in the juvenile justice process (i.e., reduces bias)? 

a. What are some reasons that you rated this question at this level? 

16. On a scale from 1 to 5, with 1 being you do not at all agree and 5 being you completely agree, 

rate your level of agreement with the following statement: the OYAS benefits the youths in 

your agency/facility? 

a. What are some reasons that you give that value? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

353 
 

 

 

 

 

 

17. On a scale from 1 to 5, with 1 being you do not at all agree and 5 being you completely agree, 

rate your level of agreement with the following statement: the OYAS benefits your 

agency/facility? 

a. What are some reasons for your level of agreement (disagreement) with that statement? 

18. Please describe your agency’s/facility’s history using risk and needs assessment tools prior to 

their implementation of the OYAS. 

19. From your standpoint, how did the agency/facility first get involved with the OYAS 

assessment process? [MONTH AND YEAR IMPLEMENTED] 

20. Did anyone tell you the reasons for using the OYAS? If so, what are some of those reasons? 

21. Do you or other members of the agency/facility do anything specific to facilitate staff buy-in? 

If so, please explain what was done? 

a. In what ways did the buy-in process work/not work? 

22. On a scale from 1 to 5, with 1 being not at all supportive and 5 being very supportive, how 

would you rate overall level of staff support for use of the OYAS in processing, supervising, 

managing, and/or treating youths? 

a. What are some reasons that you would rate the support as a ____? 

23. How much attention is given to the OYAS assessment at formal staff meetings? 

☐ Every Meeting ☐ Occasionally ☐ Seldom ☐ Never 

24. How frequently does the OYAS assessment come up in informal meetings? 

☐ All of the time ☐ Occasionally ☐ Seldom ☐ Never 

25. What are some of the concerns raised by staff about the OYAS? 

a. How are such concerns typically handled (i.e., how do you address these concerns 

raised by staff about the OYAS)? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

354 
 

26. Generally, what are the strengths of the OYAS and where is there room for improvement? 

**Note other concerns raised in this section: 

Implementing the OYAS 

The following questions are related to implementation. By implementation, we are talking about 

a variety of concepts related to how the OYAS was rolled out and is being maintained including 

training, quality assurance, and challenges you have faced during implementation. 

27. Were there particular resources introduced to help ease or promote the rollout of the OYAS? 

28. What challenges has your agency/facility faced concerning implementation of the OYAS (e.g., 

technological, training, reassessment processes, override processes)? 

a. How could the implementation of the OYAS be improved? 

29. If an agency had to roll out a new assessment system in the future, what would you recommend 

they do differently? 

30. Have there been any modifications to usage or implementation practices within the 

agency/facility since the OYAS was first introduced? 

☐ No ☐ Yes 

a. If so, could you give me some examples? 

31. Are staff members formally trained on how to use the OYAS? 

☐ No ☐ Yes 

a. If yes, could you briefly describe the process? 

b. Is certification required as part of this process? 

32. Is this training required of everyone, even if they do not personally administer the tools? 

33. Is re-certification required? 

☐ No ☐ Yes 

a. If so, how frequently must staff members be re-certified? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

355 
 

 

 

 

b. How does someone get recertified as an OYAS assessor? 

34. Are there any quality assurance processes that your agency/facility engages in concerning the 

OYAS? 

35. Has the agency/facility looked at all of the OYAS data and used that to make any 

agency/facility wide decisions? (i.e., aggregate data) 

a. If so, please describe this process. 

36. Have you had any experience working with the automated OYAS system (i.e., the online webbased OYAS system)? 

☐ No ☐ Yes 

a. If so, what do you like most about the automated system? 

b. What do you like the least about the automated system? 

c. Do you have other databases/automated systems (e.g., case management systems) that 

are integrated with the OYAS automated system? If so, has this “all-in-one” system been 

beneficial? Why or why not? 

**Note other concerns for this area: 

Youth Assessment Practices 

This section is related to which tools you use and how you use the assessment information. 

37. Does the agency/facility have written documentation regarding the policies and practices of 

using the youths assessment tools? 

☐ No ☐ Yes ☐ Don’t Know ☐ N/A 

a. If yes, how closely do you follow this written documentation? 

38. Which OYAS assessment tool(s) does your agency/facility use (please check all that apply): 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

356 
 

 

 

 

 

 

☐ Diversion Tool ☐ Residential Tool ☐ Disposition Tool 

☐ Detention Tool ☐ Reentry Tool ☐ Don’t Know

☐ N/A 

a. How and when are the tool(s) used? 

39. Which OYAS assessment tool(s) does you personally administer (please check all that apply)? 

☐ Diversion Tool ☐ Residential Tool ☐ Disposition Tool 

☐ Detention Tool ☐ Reentry Tool ☐ Don’t Know

☐ N/A 

40. What is the average time it takes to complete the interview? 

41. In addition to what the youth tells you in the interview, what other sources of information do 

you consult when scoring the assessment? 

a. What priority do you give these sources of information? 

42. How is the OYAS used in your agency/facility/court? 

a. Do you believe that the information from the OYAS is a valuable part of the decisionmaking process? Why or why not?83 

43. Do you or does your agency/facility use the OYAS assessment information to... 

a. Match youths to 

staff ? 

No 

☐ No 

Yes 

☐ Yes 

Don’t 

Know Explanation: 

b. Allocate resources to 

particular parts of 

the agency/facility? 

☐ No ☐ Yes ☐ DK 

83 The interview guide was revised after the State 1 interviews had already begun. Therefore, eight State 1 interviewees 

answered the following instead: “Do you think the OYAS provides valuable information?”

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

357 
 

c. Develop specialized 

caseloads? 

☐ No ☐ Yes ☐ DK 

d. Determine 

supervision level? 

☐ No ☐ Yes ☐ DK 

e. Measure youths 

progress in reducing 

risk and needs? 

☐ No ☐ Yes ☐ DK 

f. Match youths to 

appropriate 

treatment services? 

☐ No ☐ Yes ☐ DK 

g. Assist in diversion / 

disposition / release 

decisions? 

☐ No ☐ Yes ☐ DK 

h. Does your 

agency/facility use 

the assessment 

☐ No ☐ Yes ☐ DK 

information for any 

other purposes? If 

so, please elaborate: 

44. Does the agency/facility allow for overrides? 

☐ No ☐ Yes 

a. If yes, what does the agency/facility allow to be overridden? 

b. If yes, what are common reasons for overrides? 

c. If yes, what is the process for overriding an OYAS? 

d. Are there particular types of juvenile offenders or specific offenses that must be 

overridden regardless of the results of the OYAS (e.g., sex offenders, gun spec. cases, 

first time offenders, etc.)? 

☐ No ☐ Yes 

1. If so, what process is used for determining that? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

358 
 

 

 

e. Based on your understanding, roughly what percentage of cases are overridden? 

f. Is there anything specific in the OYAS that you perceive as a limitation that leads to 

these overrides? 

45. Are there particular types of juvenile offenders or specific offenses that you do not conduct an 

OYAS on at all? 

☐ No ☐ Yes 

a. If so, please explain this process further. 

46. Do you use any additional assessments aside from the OYAS? 

☐ No ☐ Yes 

a. If yes, what assessment(s)? 

b. How do you use the results of these additional assessments? 

47. Does the OYAS assessment provide useful information regarding a juvenile’s criminogenic 

needs? 

a. In what ways is the information about specific criminogenic need areas used? 

48. Does the OYAS assessment provide useful information regarding a juvenile’s noncriminogenic needs? 

a. In what ways is the information about specific non-criminogenic need areas used? 

49. Do you incorporate both the youth’s criminogenic and non-criminogenic needs into their case 

plan? 

☐ No ☐ Yes 

a. If so, how is this done? 

50. Does the agency/facility have a policy in place for youths reassessment? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

359 
 

 

 

 

 

 

 

 

☐ No ☐ Yes 

a. If so, what is the policy? 

b. Is this reassessment policy consistently followed throughout the agency/facility (i.e., is 

there a process in place to ensure that staff follow the reassessment process and an 

assessment is completed for every youth)? 

c. Are there any instances when a youth would leave the agency/facility without a 

reassessment? 

51. Does your agency/facility share OYAS assessment information/results with other stakeholders, 

agencies, and/or departments? 

☐ No ☐ Yes 

a. If so, who receives this information and what specific information is shared? 

b. Are there any issues with sharing this information? If so, explain. 

c. If the agency/facility does not share assessment information/results with other 

stakeholders, agencies, and/or departments, do you know why they do not do this? 

52. Based on how your agency/facility/court currently use the OYAS, what do you see as the 

strengths? 

53. Based on how your agency/facility/court currently use the OYAS, what do you see as 

weaknesses? 

54. Given everything we have discussed, I’d like you to rate your overall satisfaction with the 

OYAS. On a scale from 1 to 5, with 1 being you are not satisfied and 5 being you are 

completely satisfied, please rate your overall satisfaction with the assessment system. 

a. What are some reasons that you would rate your overall satisfaction a____? 

55. Is there anything else that you want to tell me about the OYAS that I did not ask you? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

360 
 

 

 

 

 

 

 

 

 

 

 

 

Appendix B. Juvenile Justice Personnel Web-Based Survey84 

1. Which of the following best describes your current job title? (please select only one) 

☐ State Level Administrator ☐ Probation Officer 

☐ Judge/Magistrate ☐ Parole Officer 

☐ Supervisor ☐ Detention Officer 

☐ Intake Officer/Intake Staff ☐ Case Manager 

☐ Unit Manager ☐ Court Administrator 

☐ Other (specify) ________________ 

2. Do you administer any of the OYAS instruments? (Yes/No) 

If No, continue to question 7. If Yes, answer questions 3-6. 

3. Which of the following OYAS assessment tool(s) do you administer regularly? (check all that 

apply) 

☐ Diversion Instrument ☐ Residential Instrument 

☐ Detention Instrument ☐ Reentry Instrument 

☐ Disposition Instrument 

4. Have you personally conducted reassessments of youths using any of these tools? (Yes/No) 

5. If yes, approximately how often do you administer these assessments? 

☐ Once a day or more frequently ☐ Once a week 

☐ Once every two weeks ☐ Once a month 

☐ Less than once a month 

6. On average, how long does it take to administer the OYAS tool(s)? 

Assessment Tool Minutes N/A 

Diversion 

Detention 

Disposition 

84 This version of the web-based survey most closely reflects that which was distributed to most of the survey 

respondents. Judges/magistrates received a more streamlined version of the survey that accounted for their unique 

relationship to the OYAS (i.e., they do not administer the tool or carry a youth caseload, but they likely use the OYAS 

information in case decision-making). Questions pertaining only to judges/magistrates are noted. Those identifying as 

Court Administrators also received a number of skip patterns, based on the relevancy of a given question to their job 

role. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

361 
 

 

Assessment Tool Minutes N/A 

Residential 

Reentry 

7. Do you or your agency use the OYAS assessment information to...

Explanation: 

i. Assign staff to cases? (Yes/No) 

j. Allocate resources to particular parts of the 

agency/facility? (Yes/No) 

k. Develop specialized caseloads? (Yes/No) 

l. Determine supervision level? (Yes/No) 

m. Measure youths progress in reducing risk and needs? 

(Yes/No) 

n. Match youths to appropriate treatment services? 

(Yes/No) 

o. Assist in diversion / disposition / release decisions? 

(Yes/No) 

p. There is a quality assurance process concerning the 

tool(s)? (Yes/No) 

Does your agency/facility use the assessment information for 

any other purposes? (Yes/No) 

8. Have you received formal training on how to use the tool(s)? (Yes/No) 

9. Were you formally certified on the tool(s) (i.e., were you required to pass a certification test in 

order to conduct the OYAS assessment)? (Yes/No) 

10. Have you received any retraining on the tool(s) (i.e. attended a recertification or booster 

session)? (Yes/No) 

a. If yes, approximately how often do these trainings occur? 

☐ Never ☐ Once per year 

☐ More than once per year ☐ Once every two years 

☐ Once every three years ☐ Once every four years 

11. On a scale from 1 to 5, with 1 being you are not satisfied and 5 being you are completely 

satisfied, please rate your overall satisfaction with the assessment tool(s). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

362 
 

1 2 3 4 5 

Not Satisfied Completely 

Satisfied 

Optional Comments: 

12. Do you or your agency/facility conduct additional assessments besides the OYAS? (Yes/No) 

a. If yes, please list the additional assessments used by you and your agency/facility: 

13. Does your agency/facility allow for overrides concerning supervision level and/or risk level 

(e.g. professional discretion when raising/lowering youth’s risk levels)? (Yes/No) 

14. Is there a policy concerning overrides (i.e., written documentation)? (Yes/No) 

15. Approximately what percentage of cases are overridden in your agency/facility? 

16. Approximately what percentage of cases have you overridden? 

17. Please list the types of juveniles where professional discretion or override is most often used 

(e.g. sex offenders): 

Section I: Implementing the OYAS 

18. Please indicate your level of agreement with the following statements related to the OYAS 

assessment results (choose one): 

The Youth Assessment Tool: Disagree Agree Disagree Uncertain Agree Strongly Strongly 

a. Helps to determine appropriate 

interventions based on youths’ 

risk levels 

b. Helps identify youths’ 

criminogenic needs 

c. Helps to make treatment planning 

decisions for youths 

d. Helps establish a common 

language regarding youths’ risks

and needs for our agency/facility 

e. Reduces our agency's/facility's 

use of overrides by considering 

youths' risks, needs, and barriers 

to supervision and/or treatment 

services 

f. Increases consistency in rules, 

policies, and procedures regarding 

assessment practices 

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

363 
 

 

The Youth Assessment Tool: Disagree 

Strongly Disagree Uncertain Agree Agree Strongly 

g. Reduces variation in decisionmaking across individual staff and ☐ ☐ ☐ ☐ ☐

juvenile justice agencies/facilities 

h. Helps our agency/facility make 

proper decisions regarding ☐ ☐ ☐ ☐ ☐

youths’ supervision levels

i. Reduces bias in juvenile justice 

decision-making ☐ ☐ ☐ ☐ ☐

j. Helps our agency/facility 

determine whether a youth is 

appropriate for community ☐ ☐ ☐ ☐ ☐

supervision versus residential 

placement 

19. Please indicate your level of agreement with the following statements related to the usefulness 

of the OYAS (choose one): 

Disagree Agree The OYAS is useful for... Disagree Uncertain Agree Strongly Strongly 

a. Assigning staff to cases ☐ ☐ ☐ ☐ ☐

b. Allocating resources to particular ☐ ☐ ☐ ☐ ☐ parts of the agency/facility 

c. Developing specialized caseloads ☐ ☐ ☐ ☐ ☐

d. Determining youths’ supervision ☐ ☐ ☐ ☐ ☐ levels 

e. Measuring progress in addressing ☐ ☐ ☐ ☐ ☐ youths' risks and needs 

f. Matching youths to appropriate ☐ ☐ ☐ ☐ ☐ treatment services 

g. Assisting in diversion, 

disposition, placement, or release ☐ ☐ ☐ ☐ ☐

decisions 

h. Providing information to develop ☐ ☐ ☐ ☐ ☐ comprehensive case plans 

19a. Does your agency/facility use the OYAS for any other purpose? If so, please describe. 

20. Please indicate your level of agreement with the following statements related to implementing 

the OYAS in your agency (choose one): 

Disagree Agree Disagree Uncertain Agree Strongly Strongly 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

364 
 

 

 

 

 

 

 

a. The use of the OYAS has made 

my job easier 

b. The OYAS benefits youths 

c. There is staff support for 

implementing the OYAS 

d. There are clear guidelines for 

when to use the OYAS 

e. There is a protocol for how to use 

the OYAS 

f. Staff buy-in for the OYAS is hard 

to achieve 

g. Formal training in the OYAS is 

not routinely offered 

h. Implementation of the OYAS is 

not consistent 

i. Finding resources to properly 

address the needs identified from 

the OYAS is difficult 

j. The OYAS instruments are easy 

to read, interpret, and use 

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐

☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐

21. On a scale from 1 to 5, with 1 being you are not confident and 5 being you are completely 

confident, please rate how confident you are that the scores produced by the OYAS tools 

accurately represent the likelihood of a juvenile engaging in delinquent behavior again. 

1 2 3 4 5 

Not Confident Completely 

Confident 

Optional Comments: 

22. On a scale from 1 to 5, with 1 being you are not confident and 5 being you are completely 

confident, please rate how confident you are that an offender will receive the same score no 

matter which staff member conducted the OYAS assessment. 

1 2 3 4 5 

Not Confident Completely 

Confident 

Optional Comments: 

23. Based on how your agency/facility currently uses the OYAS, what do you see as the strengths 

in the process? (Please list up to 3 strengths) 

24. Based on how your agency/facility currently uses the OYAS, what do you see as the limitations 

in the process? (Please list up to 3 limitations) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

365 
 

 

 

 

25. Please provide any specific suggestions for improving the OYAS and/or process in your 

agency/facility: 

26. Do you use the state’s automated system for the OYAS? (Yes/No) 

If No, continue to question 28. If Yes, answer question 27. 

27. On a scale from 1 to 5, with 1 being you are not satisfied and 5 being you are completely 

satisfied, please rate your overall satisfaction with the OYAS automated system. 

1 2 3 4 5 

Not Satisfied Completely 

Satisfied 

Optional Comments: 

Section II: Applying the OYAS 

28. The following depicts a hypothetical case. Please score and comment on the vignette using the 

provided OYAS tool. Click on the following link to access the scoring guide: scoring guide 

[NOTE: ONLY THOSE WHO ADMINISTER THE OYAS ARE ASSIGNED A VIGNETTE. 

RESPONDENTS ARE GIVEN ONE VIGNETTE AT RANDOM.] 

Vignette Option 1: 

Shelly is a 16-year-old female coming before the court on her 3rd adjudication. Her previous 

adjudications are for theft of means of transportation and shoplifting. She was successfully 

released from probation 4 months ago. Her first contact with the police was at age 13, but the 

police warned her and let her go. Her first referral to the juvenile court was at age 14, for 

shoplifting. She was placed on probation which she successfully completed. At age 15, she 

reoffended and was referred for the theft and continued probation. 

1.1) Documented Contact with the Juvenile Justice System 

0 = 14 or older 

1 = 13 or younger 

1.2) Previous Adjudications 

0 = No priors 

1 = 1 prior adjudication 

2 = 2 or more priors 

Total:______ 

Vignette Option 2: 

Anthony and his peers engage in physical fights with other teens when they feel bullied by 

others. Engaging in these fights have lead Anthony and his peers to get into trouble. During 

your interview with Anthony, he reports that he has 10 close friends, all of which have been in 

trouble with the law. When you inquire further about his troubled friends, you determine that 9 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

366 
of them have been officially arrested. Anthony further reports that 3 out of 10 of these friends 

have been arrested and detained with him. Anthony explains the altercations are with other teens 

they go to school with. He feels that these other boys disrespect him and his friends and he 

wants to make sure they stand up for themselves. Anthony reports he and his friends also work 

together to steal cars. They have stolen about 20 cars to sell them for parts. He also reports that 

6 of his friends have been suspended from school for 3 days for fighting. He says that his friends 

are important but that he sees himself as more important. 

3.1) Friends Fight 

0 = Friends do not fight 

1 = Friends fight 

3.2) Friends Arrested 

0 = Less than 50% 

1 = 50% or more 

3.3) Friends/Family Associated with Gang Activity 

0 = No 

1 = Yes 

3.4) Arrested with Friends 

0 = No 

1 = Yes 

3.5) Friends Susp./Exp. from School 

0 = Less than 50% 

1 = 50% or more 

3.6) Friends are Important 

0 = Friends are very important 

1 = Friends are not very important 

Total:______ 

Vignette Option 3: 

Hank is currently on probation for possession of marijuana. Hank indicates that he can stop 

using any time. He is asked by his friends to go to the park after school. These are the friends 

Hank hung around with when he used to smoke pot. He doesn’t really think about it and agrees 

to go. Hank walks to the park with his friends, but once he gets there, he thinks about the fact 

that he is on probation and that he has a drug test in a couple days. He decides that he does not 

want to look bad in front of his friends so he goes ahead and smokes the weed but tries to make 

an excuse and leaves as soon as he can. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

367 
 

 

 

 

5.1) Can identify triggers/high risk situations 

0 = Identifies high risk situations 

1 = Does not identify high risk situations 

5.2) Weighs pros/cons of a situation 

0 = Weighs the pros/cons of a situation 

1 = Does not weight the pros and cons of a situation 

5.3) Pro-social decision making 

0 = Demonstrates pro-social decision making 

1 = Does not demonstrate pro-social decision making 

Total:______ 

Section III: Staff Information 

Tell us more about yourself. This last section is strictly for descriptive purposes. 

29. Gender: (Male/Female) 

30. Race: 

☐ White ☐ Black or African American 

☐ Asian ☐ Pacific Islander 

☐ American Indian/Alaska Native ☐ Other (specify)________ 

31. Ethnicity: (Hispanic/Non-Hispanic) 

32. What is your highest level of education completed? (choose one) 

☐ GED ☐ Bachelor’s degree

☐ High school diploma ☐ Graduate degree 

☐ Associate’s degree

33. If you have a college degree, in what field did you receive your highest degree? 

☐ Criminal Justice ☐ Social Work 

☐ Psychology ☐ Sociology 

☐ Law ☐ Other (please specify)______________ 

34. How long have you been working in the field? 

35. How long have you been working at your current agency/facility? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

368 
 

 

36. Do you carry a caseload? (Yes/No) 

If no, continue to question 40. If yes, answer question 37. 

37. How many youths are currently on your caseload? 

38. Do you see juvenile delinquency cases? (Judges/magistrates only) 

If no, continue to question 41. If yes, answer question 39. 

39. On average, how many delinquency cases do you see a month? (Judges/magistrates only) 

40. The juvenile offender population you currently work with includes: (check one) 

☐ Males ☐ Females ☐ Both ☐ N/A 

41. Please provide any additional comments or concerns regarding the OYAS. (optional) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

369 
Appendix C. Code Definitions for Personnel-Generated Strengths of the OYAS 

(Interviews and Surveys) 

Code Definition 

1. Helps decision- Any mention of support for the assessment 

making or as helpful for guiding/directing the 

service provision supervision/case management decisionmaking process (including placement 

decisions, which needs to prioritize, 

diversion, treatment referrals, provider 

collaboration, etc.). 

2. Fair, consistent, Any mention of support that characterizes 

or objective the assessment as encouraging fairness, 

credibility, objectivity, and/or consistency 

in the supervision decision-making process 

or quality of service provision. 

3. Identifies needs Any mention of support that characterizes 

the assessment as helpful in identifying a 

youth’s needs. 

4. Useful or Any mention of support that characterizes 

comprehensive the assessment as useful and/or 

information comprehensive in gathering information on 

a youth. 

5. Identifies risk Any mention of support that characterizes 

level the assessment as useful for identifying risk 

score or level. 

6. Establishes Any mention of support that characterizes 

baseline and/or the assessment as useful in establishing a 

monitors youth's baseline level of functioning or risk, 

progress and/or monitoring youths progress through 

supervision, treatment, or otherwise. 

7. Easy/clear/quick Any mention of support for the assessment 

to administer in that it is widely available, easy to 

use/access, and/or well-organized or quick 

to administer. 

8. Helpful with Any mention of support that characterizes 

case the assessment as helpful in creating case 

plans/treatment plans. These segments mention case plans 

planning or case planning specifically, not just 

service provision. Any mention of guiding 

case management priorities is captured 

instead under the ‘helps decisionmaking/service provision’ code.

9. Evidence-based, Any mention of support for the assessment 

validated, or due to its base in 

reliable results 

Present in Present in 

Interviews? Surveys? 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

X x 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

370 
10. Good 

implementation, 

training, or 

support 

11. Helpful (NOS) 

12. Dictates level of 

supervision 

13. Builds rapport 

14. Automated 

system useful 

15. Reduces 

workload 

research/evidence/validation/validity/reliabi 

lity. 

Any mention of support for the assessment 

that characterizes the implementation x process, administrative support, training, or 

supervision as good or valuable. 

Any mention of support that characterizes 

the assessment, assessment process, or its 

results as generally helpful, but does not x indicate a specific reason for why, or in 

what capacity (i.e., is ‘not otherwise 

specified’ [NOS]).

Any mention of support that characterizes 

the assessment as useful in identifying the 

appropriate level of supervision (frequency) x 

or length of supervision (time). 

Any mention of support that characterizes 

the assessment as helpful in establishing x practitioner-youth rapport, or involving 

youths in their own treatment process. 

Any mention of support for the assessment 

that categorizes its automated system as 

useful. Any mention of the automated case x 

plans are captured instead under the ‘helpful 

with case plans’ code.

Any mention of support for the assessment 

that characterizes it as helpful in reducing x 

the workload. 

x 

x 

Combined 

with 

‘dictates 

risk level’

code. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

371 
 

 

Appendix D. Code Definitions for Personnel-Generated Limitations of the OYAS 

(Interviews and Surveys) 

Code Definition Present in Present in 

Interviews Survey 

1. Lack of validity or Any mention of concern regarding x Validity 

reliability the validity or reliability of the and 

assessment. This includes any reliability 

mention of uncertainty or concerns 

subjectivity in scoring, but does not are both 

include any mention of concern present, 

about sex offenders, which are but coded 

captured in the ‘inaccurate for sex separately 

offenders’ code. in the 

survey 

data. 

2. 

3. 

4. 

Items on the 

assessment 

Poor 

implementation or 

training 

Difficulty in case 

planning 

Any mention of concern about 

specific items on the assessment. 

This includes the lack of certain 

items on the assessment. 

Any mention of concern about the 

implementation or training process 

associated with the assessment. This 

includes a lack of clarity or 

communication on policies and 

procedures regarding the OYAS. 

Training concerns regarding case 

planning should be coded as 

“Difficulty in case planning.”

Any mention of concern about case 

planning in reference to the 

assessment. This also includes any 

concerns with automated system's 

case plan mechanism, as well as 

concerns about training specific to 

case planning. 

x 

x 

x 

x 

x 

x 

5. Time-consuming Any mention of the concern about 

the time necessary to complete the 

assessment interview or scoring 

process. This does NOT include any 

concerns about the automated 

system, which are captured in the 

“Automated system issues” code. 

x x 

6. Inaccurate for sex 

offenders 

Any mention of concern for the 

accuracy, usefulness, or 

applicability of the assessment 

results for youths with sex offenses. 

x x 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

372 
 

7. Utilization Any mention of concern about the x Similar, 

lack of utility for the information but unique 

gathered from the assessment. This code used 

includes assessment information not for 

being used by an agency/facility surveys, 

advantageously. see “Poor 

usage 

practices”

8. Automated system 

issues 

Any mention of concern regarding x 

the assessment's automated system 

(e.g., technical issues, userfriendliness issues, etc.). This does 

NOT include concerns regarding the 

automated system's case planning 

mechanism, which are captured in 

the “Difficulty in case planning”

code. 

9. Lack of buy-in Any mention of concern regarding 

buy-in to the assessment. This can 

be on behalf of staff members, or 

x x 

parents of the youths. 

10. Youths/family 

manipulate scores 

Any mention of concern regarding 

the youths or their family lying or 

purposefully manipulating their 

score on the assessment. 

x x 

11. Other limitation 

(NOS) 

Any mention of concern regarding 

the OYAS that does not fit in to one 

of the aforementioned codes. 

x 

12. Poor usage 

practices 

Any mention of concern that the 

OYAS information is not used as 

designed. 

Similar, 

but unique 

code used 

for 

x 

interviews 

, see 

“Utilizatio

n” above.

13. Lack of resources 

available for 

treatment 

Any mention of concern for a lack 

of community resources to treat 

youths (as suggested by OYAS 

information) 

x 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

373 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix E. Standardized Results for OYAS Usefulness Confirmatory Factor Analysis 

Estimate S.E. Est./S.E. 

OYAS Usefulness BY 

Identify criminogenic needs .79*** .02 36.9 

Determining supervision levels .77*** .02 36.6 

Develop comprehensive case plans .83*** .02 50.9 

Determine appropriate interventions .83*** .02 45.4 

Matching youth to treatment .86*** .01 60.4 

Measuring youth progress .83*** .02 51.9 

Diversion/disposition/placement/release 

decisions .78*** .02 40.8 

OYAS has made my job easier .76*** .02 34.5 

Fit Statistics 

Chi-Square Test of Model Fit 

Chi-Square Test for Baseline Model 

Root Mean Square Error of Approximation 

Confirmatory Fit Index 

Tucker Lewis Index 

Weighted Root Mean Square Residual 

Kaiser-Meyer-Olkin Measure 

Residual Variances 

97.79(20)*** 

7464.70(28)*** 

.08 

.99 

.99 

.86 

.93 

Identify criminogenic needs .62 .03 18.5 

Determining supervision levels .60 .03 18.3 

Develop comprehensive case plans .69 .03 25.4 

Determine appropriate interventions .68 .03 22.7 

Matching youth to treatment .73 .02 30.2 

Measuring youth progress 

Diversion/disposition/placement/release 

decisions 

OYAS has made my job easier 

.69 

.61 

.58 

.03 

.03 

.03 

26.0 

20.4 

17.2 

*p < .05, ** p < .01, ***p < .001 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

374 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix F. OYAS Administrator Sample Characteristics for Vignette and Usefulness 

Analysis 

Variable 

Gender n % 

Male 175 28.0 

Female 279 44.6 

Race n % 

White 381 60.9 

Non-White 71 11.3 

Ethnicity n % 

Hispanic 38 6.1 

Non-Hispanic 385 61.5 

Job Title n % 

Supervisor 64 10 

Probation or Parole Officer 456 73 

Intake Officer/Intake Staff 38 6 

Other Job Title 101 16 

Education Level n % 

GED 1 .2 

High School Diploma 10 1.6 

Associate’s Degree 11 1.8 

Bachelor’s Degree 315 50.3 

Graduate Degree 119 19.0 

Work Experience n % 

Years (x̅and sd) 14.9 8.4 

Retraining 270 43 

State n % 

State 1 249 40 

State 2 195 31 

State 3 182 29 

Due to missing data, totals are less than reported sample size (N = 626). 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

375 
 

 

 

Appendix G. Overview of Requested Record Data Fields 

Data 

Category Variable Variable Description Youth Information Provided 

Youth ID 

Date of Birth 

First Name 

Middle Name 

Last Name 

DYS/DOC ID Number 

SSN 

Assessment Type 

County of Commitment 

Assessment Date 

Assessment ID 

Youth ID, DOB, First Name, Middle Name, 

Last Name, DYS#, SSN, Assessment Type, 

County, Date Updated, and Assessment ID 

have all been provided as supplemental 

identifying information. 

Focal Offense Information Requested 

Arrest or Referral Date Arrest or Referral Date is the date of arrest 

or referral associated with the assessment date 

listed in the current spreadsheet. 

First Court Date First Court Date is the date of the youth's first 

court appearance associated with the offense(s) 

and the OYAS assessment date listed in the 

current spreadsheet. 

If a youth's case was handled informally by the 

court, the referral date can be used here 

instead. 

Case Number Case Number is the case number associated 

with the offense and the OYAS assessment 

date listed in the current spreadsheet. 

Note, this variable may not apply if the case 

has not been filed formally with the court. 

Youth Status Youth Status is whether the youth is active, 

inactive, successfully completed 

supervision/program, or was terminated from 

supervision/ program for highest level offense 

and OYAS assessment date listed in the 

current spreadsheet. 

Adjudication Status Adjudication Status concerns whether the 

youth was adjudicated for the highest level 

offense associated with the OYAS assessment 

date in the current spreadsheet. 

Adjudication Date Adjudication Date is the date of adjudication 

(if applicable) associated with the highest level 

offense and the OYAS assessment date listed 

in the current spreadsheet. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

376 
 

 

 

 

Data 

Category Variable Variable Description 

If a case did not result in an adjudication or 

was handled informally, please note the date in 

which the case was disposed of, regardless of 

the disposition. 

Case Disposition Case Disposition is the initial placement or 

referral type (e.g., probation, residential 

placement, diversion, etc.) that was made for 

the youth associated with the offense and 

OYAS assessment date listed in the current 

spreadsheet. 

Name of Treatment/Program 

Provider 

Name of Treatment/Program Provider is the 

name of the specific provider or agency (e.g., 

Horizons Recovery Center) that was 

responsible for providing treatment to the 

youth associated with the offense and OYAS 

assessment date listed in the current 

spreadsheet. 

If there are multiple treatment/program 

agencies, please list them all here. 

Name of Treatment/Program Name of Treatment/ Program is the specific 

treatment or program name (e.g., Thinking for 

a Change, Aggression Replacement Training) 

that the youth participated in associated with 

the offense and date of OYAS assessment 

listed in the current spreadsheet. 

If there are multiple treatment/program names, 

please list them all here. 

Type of Treatment/Program 

Received 

Type of Treatment/Program Received is the 

type of treatment or program a youth received 

(e.g., cognitive behavioral, education, 

substance abuse, etc.) that is associated with 

the offense and OYAS assessment date listed 

in the current spreadsheet 

If there are multiple treatment/ program types, 

please list them all here. 

Treatment/Program Start Date Treatment Start Date is the date the youth 

started treatment or was admitted to a program 

that is associated with the offense and OYAS 

assessment date listed in the current 

spreadsheet 

If there are multiple treatment/program start 

dates, please list them all here. 

Treatment/Program End Date Treatment End Date is the date the youth 

ended treatment or was terminated from a 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

377 
Data 

Category Variable Variable Description 

program that is associated with the offense and 

OYAS assessment date listed in the current 

spreadsheet 

If there are multiple treatment/program end 

dates, please list them all here. 

Other Referrals/Sanctions Other Referrals/Sanctions is any other nontreatment referrals or sanctions (e.g., 

restitution, community service, etc.) associated 

with the offense and OYAS assessment date 

listed in the current spreadsheet. 

Recidivism Information Requested 

Was the Youth Subsequently 

Arrested for a New 

Offense(s)? 

Was the Youth Subsequently Arrested for a 

New Offense(s)? This question pertains to 

whether the youth recidivated (i.e., was 

arrested) after his/her OYAS assessment date 

listed in the current spreadsheet. 

If the youth was arrested, please list the date of 

arrest for the highest level of offense in the 

cell. If the youth was not arrested, please 

simply put "no" in the cell. 

Was the Youth Subsequently 

Adjudicated for a New 

Offense(s)? 

Was the Youth Subsequently Adjudicated 

for a New Offense(s)? This question pertains 

to whether the youth recidivated (i.e., was 

adjudicated after his/her OYAS assessment 

date listed in the current spreadsheet. 

If the youth was adjudicated, please list the 

date of adjudication for the highest level of 

offense in the cell. If the youth was not 

adjudicated, please simply put "no" in the cell. 

Was the Youth Subsequently 

Incarcerated for a New 

Offense(s)? 

Was the Youth Subsequently Incarcerated 

for a New Offense(s)? This question pertains 

to whether the youth recidivated (i.e., was 

incarcerated after his/her OYAS assessment 

date listed in the current spreadsheet. 

If the youth was incarcerated, please list the 

date of incarceration for the highest level of 

offense in the cell. If the youth was not 

incarcerated, please simply put "no" in the cell. 

Was the Youth Subsequently Was the Youth Subsequently Charged with 

Charged with a Probation a Probation Violation(s) or Technical 

Violation or Technical Violation(s)? This question pertains to 

Violation? whether the youth recidivated (i.e., received a 

probation violation or violation of a court 

order) after his/her OYAS assessment date 

listed in the current spreadsheet. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

378 
Data 

Category Variable Variable Description 

If the youth was charged with a violation, 

please list the date of charge for the highest 

level of violation in the cell. 

If the youth was not charged with a violation, 

please simply put "no" in the cell. 

Offense Code Offense Code is the state criminal code 

associated with the highest level of offense for 

which the youth recidivated. 

Offense Code Description Offense Code Description is the description 

of the state criminal code associated with the 

highest level of offense for which the youth 

recidivated. 

Offense Level Offense Level is related to the level of the 

offense (e.g., felony, misdemeanor, or status 

offense). Please note the highest level of 

offense for which the youth recidivated. 

Offense Type Offense Type is whether the offense is 

conspiracy related, deemed as complicity, or 

attempt for the highest level of offense for 

which the youth recidivated. 

Degree Degree is the degree of the felony (e.g., F1, 

F2, F3, etc.) or misdemeanor (e.g., Class A, 

Class B, etc.) for the highest level of offense 

for which the youth recidivated. 

Offense Count Offense Count is the number of counts 

associated with the highest level of offense for 

which the youth recidivated. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

379 
 

 

 

 

Appendix H. Youth Follow-Up Interview 

Name of Interviewer: _____________________ Date of Interview: _________________ 

ID Number #: _____________________ Month (circle one): 12 month / 24 month 

Section I: Education/Employment 

Interview prompt: Hello. My name is ________ and I am with the University of Cincinnati. 

Today we are going to talk about your experience with the juvenile justice system and things that 

have happened in your life since you [either they were involved with the system for the particular 

instant offense of focus or since their last interview]. 

I would like to start by asking you a few questions about your education and employment 

background. 

1. Highest grade completed (check only one response): 

☐ 8th grade or below 

☐ 9th grade 

☐ 10th grade 

☐ 11th grade 

☐ HS diploma 

☐ GED 

☐ Some college 

Comments: 

2. Are you currently in school? (If no, continue to question 9. If yes, ask questions 3-8.) 

☐ No ☐ Yes 

Comments: 

3. If yes, how many days have you skipped school/classes in the past month? 

Comments: 

4. How many times have you been suspended from school in the past 12 months? (To be 

counted, the youth must have been suspended for at least 1 full day.) 

Comments: 

5. How many times have you been expelled from school in the past 12 months? 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

380 
Comments: 

6. Would you say that you get along with your teachers or other school staff: 

☐ Always 

☐ Sometimes 

☐ Never 

Comments: 

7. What is your average letter grade in school? 

☐ A 

☐ B 

☐ C 

☐ D or below 

Comments: 

8. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements (choose one response): 

8a. My schoolwork is worthwhile. 

1 2 3 4 5 

8b. I feel committed to school. 

1 2 3 4 5 

8c. I put a lot of effort into school. 

1 2 3 4 5 

8d. I am satisfied with my schoolwork. 

1 2 3 4 5 

8e. Even though school can be difficult, it’s worth it in the long run. 

1 2 3 4 5 

Comments: 

9. Are you currently employed? (If No, continue to question 16. If Yes, answer questions 10-

15.) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

381 
 

 

☐ No ☐ Yes 

Comments: 

10. On average, how many hours per week do you work? 

Comments: 

11. Do you get along with your coworkers? 

☐ Always 

☐ Sometimes 

☐ Never 

Comments: 

12. Do you get along with your boss? 

☐ Always 

☐ Sometimes 

☐ Never 

Comments: 

13. Have any of your coworkers been in trouble with the law? (If no or don’t know, continue to 

question 14. If yes, answer question 13a.) 

☐ No ☐ Yes ☐ Don’t Know

Comments: 

13a. Do you consider those coworkers who have been in trouble with the law friends? 

☐ No ☐ Yes 

Comments: 

14. Have you ever been in trouble at any job? (If no, continue to question 15. If yes, answer 

question 14a.) 

☐ No ☐ Yes 

Comments: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

382 
 

 

 

 

14a. What happened to you as a result? 

☐ Fired 

☐ Reprimanded 

☐ No Consequences 

Comments: 

15. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements (choose one response): 

15a. My work is worthwhile. 

1 2 3 4 5 

15b. I feel committed to my work. 

1 2 3 4 5 

15c. I put a lot of effort into my work. 

1 2 3 4 5 

15d. I am satisfied with my job. 

1 2 3 4 5 

15e. Even though work can be difficult, it’s worth it in the end. 

1 2 3 4 5 

Comments: 

Additional Comments Regarding Education/Employment 

Section II: Family, Living Arrangements, and Neighborhood 

Interview prompt: Now I would like to talk to you about your family and living arrangements. 

16. Who do you currently live with? 

☐ Biological or adoptive parent(s) 

☐ Other family member(s) 

☐ State residential facility 

☐ Community residential facility 

☐ Friend(s) 

☐ Other ______________ (specify) 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

383 
 

 

 

Comments: 

17. Do you have contact with your biological/adoptive parent(s)? 

☐ No ☐ Yes 

Comments: 

18. How long have you lived in your current home? 

☐ 1 year or less 

☐ 2-3 years 

☐ Over 3 years 

Comments: 

19. How many times have you moved in the last 12 months, not including moves to or from a 

juvenile or adult justice facility? 

Comments: 

20. How many of your immediate family members have been arrested in the past 12 months? 

Comments: 

21. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

21a. My family is important to me. 

1 2 3 4 5 

21b. My family wants me to stay out of trouble. 

1 2 3 4 5 

21c. My family would do anything for me. 

1 2 3 4 5 

21d. I feel safe with my family. 

1 2 3 4 5 

21e. I feel safe in my neighborhood. 

1 2 3 4 5 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

384 
 

 

 

21f. It is easy to get drugs in my neighborhood. 

1 2 3 4 5 

21g. If I need something, I have at least one person I can turn to. 

1 2 3 4 5 

Comments: 

Additional Comments Regarding Family, Living Arrangements, and Neighborhood: 

Section III: Peer Associations 

Interview prompt: Now I am going to ask you a few questions about your friends. 

22. How many close friends do you have? (This can include family members considered close 

friends). 

22a. Of those friends, how many use alcohol? 

22b. How many use drugs (note, this can be common illicit drugs or prescription drugs used 

illegally)? 

22c. How many have been arrested? 

Comments: 

23. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

23a. My friends are important to me. 

1 2 3 4 5 

23b. My friend wants me to stay out of trouble. 

1 2 3 4 5 

23c. My friends stay out of trouble. 

1 2 3 4 5 

23d. I spent most of my time with friends. 

1 2 3 4 5 

23e. My friends are good role models. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

385 
 

 

1 2 3 4 5 

Comments: 

Additional Comments Regarding Peer Associations: 

Section IV: Situational Awareness 

Interview prompt: I am going to provide you with a set of scenarios for the next three questions 

Please answer as you would normally act in each situation. 

24. A friend asks you to go to a party. Would you go to the party knowing there are drugs and/or 

alcohol there? 

☐ No ☐ Yes 

Comments: 

25. If you found a wallet with $100, would you try to return the wallet and all its contents to the 

rightful owner? 

☐ No ☐ Yes 

Comments: 

26. If you knew your friend was driving a car that was not his/hers, would you get in the car? 

☐ No ☐ Yes 

Comments: 

27. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

27a. I am confident that I can walk away from a fight. 

1 2 3 4 5 

27b. I am confident that I can find ways to reduce stress that do not involved alcohol or 

drugs. 

1 2 3 4 5 

27c. I am confident that I can remain calm when things get heated. 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

386 
 

 

 

1 2 3 4 5 

27d. I am confident that I can resist pressure from friends to do things that may get me in 

trouble. 

1 2 3 4 5 

27e. I am confident that I can avoid situations and people where alcohol or drugs are present. 

1 2 3 4 5 

27f. I am confident that I can resist the urge to give up easily when I run into problems. 

1 2 3 4 5 

27g. I am confident that I can handle tough situations in ways that will keep me out of 

trouble. 

1 2 3 4 5 

Comments: 

Additional Comments Regarding Situational Awareness: 

Section V: Beliefs 

Interview prompt: Now I am going to ask you a series of questions about your feelings and 

attitudes toward the illegal behavior and the juvenile justice system. 

28. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

28a. The juvenile justice system is fair. 

1 2 3 4 5 

28b. If you get in trouble, it’s usually because you are with the wrong crowd.

1 2 3 4 5 

28c. People should be allowed to use illegal drugs without any legal consequences. 

1 2 3 4 5 

28d. There are some good things about gangs. 

1 2 3 4 5 

28e. You have to get even with people who do no respect you. 

1 2 3 4 5 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

387 
 

28f. If you really want something, it does not matter how you get it. 

1 2 3 4 5 

28g. It is okay to hit someone else if they hit you first. 

1 2 3 4 5 

28h. Sometimes you have to lie to get what you want. 

1 2 3 4 5 

28i. Fighting is the best way to solve problems. 

1 2 3 4 5 

28j. A kid in a gang should not be blamed for the trouble the gang causes. 

1 2 3 4 5 

28k. If people are careless where they leave their things, it is their own fault if they get 

robbed. 

1 2 3 4 5 

28l. It is alright to fight when someone you love or care for is threatened. 

1 2 3 4 5 

Comments: 

Interview prompt: You have been in trouble with the past, now I am going to ask you a series of 

questions about your feelings and attitudes about trying to change. 

29. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

29a. I am worried I may go back to my old ways, so I am seeking help with things. 

1 2 3 4 5 

29b. I am really working hard to change. 

1 2 3 4 5 

29c. I can stop breaking the law. 

1 2 3 4 5 

Comments: 

Additional Comments Regarding Beliefs: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

388 
Section VI: Substance Use 

Interview prompt: Now we are going to talk about any alcohol or drugs you may have used. 

This includes beer and liquor, cigarettes, illegal drugs, prescription drugs not prescribed to you, 

and synthetic drugs, such as salvia or K-2. 

30. Tell me about the drugs and/or alcohol you have used. If no drug or alcohol use, continue to 

question 34. If drug or alcohol use, ask questions 31-33. 

How Often 

Used Sold have you Used How Last Use (date 

Type of Drug Yes or No? Yes or in the Past 12 Much? or about how 

No? Months (daily, long ago)? 

weekly, etc.)? 

Cigarettes 

Alcohol 

(beer or liquor) 

Marijuana 

Cocaine 

(crack or powder) 

Heroin 

Other (specify) 

Other (specify) 

Other (specify) 

Comments: 

31. In the past 12 months, has someone complained about your use of alcohol or drugs? 

☐ No ☐ Yes 

Comments: 

32. Has your use of alcohol/drugs caused you any problems at school, work, home, or in the 

community in the last 12 months? 

☐ No ☐ Yes 

Comments: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

389 
 

 

 

 

 

33. How likely are you to quit using drugs? 

Never used Highly unlikely Unlikely Likely Highly Likely 

Comments: 

Additional Comments Regarding Substance Abuse: 

Section VII: Contact with the Criminal Justice System and Assessment 

Interview prompt: Now I’m going to ask you a few questions about your contact with the 

criminal justice system. 

34. In the past 12 months (or since the time of your termination from probation or release from 

an institution/facility) how many times have you been: 

Type of Involvement Number of Times 

Stopped by police? 

Arrested? 

Adjudicated on a criminal charge? 

Placed on probation? 

Sent to detention/an institution? 

Comments: 

35. Tell me about these most recent offenses: 

Comments: 

36. Were you with your friends at the time of the offense? 

☐ No ☐ Yes 

Comments: 

37. Were your friends arrested too? 

☐ No ☐ Yes 

Comments: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

390 
 

 

 

38. Have you done anything else in the past 12 months that could have gotten you into trouble 

besides what we have already talked about? List activities and circumstances. 

Comments: 

Additional Comments Regarding Contact with the Criminal Justice System and Assessment: 

Section VIII: Treatment Services 

Interview prompt: Now I am going to ask you a few questions about any treatment you may 

have received as a result of your contact with the criminal justice system. 

39. Have you been on probation in the past 12 months? If no, continue to question 41, if yes ask 

questions 40a-40d. 

☐ No ☐ Yes 

Comments: 

40. If yes, have you: 

40a. Received a technical violation? 

☐ No ☐ Yes 

40b. Received a positive drug screen? 

☐ No ☐ Yes 

40c. Met with your probation officer as required? 

☐ No ☐ Yes 

40d. Been required to wear an electronic monitoring device? 

☐ No ☐ Yes 

Comments: 

41. Have you been in a facility/institution at any point during the past 12 months? If no, continue 

to question 43, if yes ask questions 42a-42d. 

☐ No ☐ Yes 

Comments: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

391 
 

42. If yes, have you: 

42a. Been written up? 

☐ No ☐ Yes 

42b. Received a positive drug screen? 

☐ No ☐ Yes 

42c. Been in physical fights with staff? 

☐ No ☐ Yes 

42d. Been in trouble for trying to leave a facility without permission? 

Comments: 

43. Were you required to attend treatment after your most recent contact with the system? If no, 

continue question 46. If yes, ask question 44-45. 

☐ No ☐ Yes 

Comments: 

44. Tell me about any treatment you’ve received and/or participated in during the last 12 months: 

Interview prompt: For example, have you participated in Thinking 4 a Change, skill-building, 

substance abuse, or an anger management class or received any other service? 

Receive Length of Practice new Engage in Treatment type/Program name certificate of treatment skills role play? completion? 

Comments: 

45. Tell me about what you learned during treatment. Describe any skills you learned that you 

can use in real life or any ways to avoid trouble. 

Comments: 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

392 
 

46. On a scale of 1-5, with 1 being strongly disagree and 5 being strongly agree, please indicate 

your level of agreement for each of the following statements: (choose one response): 

46a. I am receiving (or received) the kind of help I need. 

1 2 3 4 5 N/A 

46b. The services I received have helped me turn my life around. 

1 2 3 4 5 N/A 

46c. The services I have received helped me to stay out of trouble with the law. 

1 2 3 4 5 N/A 

46d. Juvenile justice personnel have helped me get the services I need. 

1 2 3 4 5 N/A 

46e. I am required to get services that I do not need. 

1 2 3 4 5 N/A 

46f. I thought once I received treatment I would stay out of trouble, but sometimes I still find 

myself struggling. 

1 2 3 4 5 N/A 

Comments: 

Additional Comments Regarding Treatment Services 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

393 
Appendix I. Variation of Substance Use Indicators on the Disposition, Residential, and 

Reentry Tools 

Assessment 

Disposition 

Age of Onset 

Used Drugs Recently 

Reentry 

Age of Onset 

Other’s Complaints

Residential 

Age of Onset 

Most Recent Use 

Used Alcohol Recently Positive Test Last 6 Months Other’s Complaints

(Un)Likely to Quit Caused Problems in Major Areas Positive Test Last 6 Months 

Used While in Residential Caused Problems in Major 

Areas 

This resource was prepared by the author(s) using Federal funds provided by the U.S. 

Department of Justice. Opinions or points of view expressed are those of the author(s) and do not 

necessarily reflect the official position or policies of the U.S. Department of Justice.

394