DOI: 10.3886/icpsr37201.v1
title: Probation officers' Compliance with the Youth Level of Service/Case Management Inventory (YLS/CMI): A Multi-level Study of Post-implementation Practice across Pennsylvania counties, 2015-2018
title: Version 1

description: This research uses in-depth cases studies in five counties, combining interviews and observations, interviews with state reform leaders, and a statewide survey, to examine juvenile probation officers' use of the Youth Level of Service/Case Management Inventory (YLS/CMI) in local practice. In particular, the study assessed the form YLS/CMI policies take in local Pennsylvania counties and their consistency with the Risk-Need-Responsivity (RNR) model, the extent to which officers' routine practices align with policies and RNR principles, and how state and county factors have shaped local policies and practices. In doing so, it sought to highlight promising strategies for effective RNA implementation. Study findings describe an example of an apparently well-planned statewide effort to implement and support the YLS/CMI in local counties, involving a sustained attention to attention to training, quality assurance, and problem-solving. Local county policies, while showing some variation, focused on strategies for conducting the YLS/CMI assessment and applying its results to a variety of decisions and activities--supporting, in particular, risk and need principles. Study results further suggest a substantial statewide level of practitioner adherence to a general policy model, though this pattern was stronger for some activities than others, and varied substantially by local county. Variation in implementation across counties seemed to relate, in particular, to the extent of local quality assurance processes, leadership enthusiasm for the YLS/CMI, staff enthusiasm for evidence-based practices (with implications for training and recruitment), and organizational climate.
description: The purpose of this study is to provide research examining juvenile probation officer adherence to the Youth Level of Service/Case Management Inventory (YLS/CMI) juvenile risk/need assessment (RNA) tool, several years after its implementation in Pennsylvania.
description: Presence of Common Scales: The survey used an adaptation of the "organizational climate" scale, based on work by Taxman et al. 2007
description: Response Rates: Response rates are described below according to the data collection method: Core interviews: 86 interviews were conducted across the five case study counties, out of a target sample of 93 probation staff. This figure includes two field substitutions of the original target list, made with suitable alternatives for unavailable target subjects--one of whom had left the organization and one who was unavailable for interview. This amounts to a response rate of 92 percent (disregarding substitutions). Across individual counties, the response rate varied between 88 and 100 percent. Of the seven target subjects not interviewed (and not substituted), one declined, one was unavailable because of military service, one left the organization during the fieldwork period, and four were not interviewed because of other scheduling challenges. YLS/CMI Example Interviews: A total of 38 probation officers were interviewed from a target list of 45. This list was based on YLS/CMI-using officers identified in the core interviews (though one was recruited from an observation episode to boost numbers in County C). This was an effective response rate of 84 percent, though the rate across counties between 70 and 100 percent. One eligible subject from the core interviews was overlooked and not included in the target list of 45. Of those on the target list who were not interviewed, two declined and five were ultimately not responsive to follow-up scheduling for interview. Through the 38 interviews conducted, a total of 81 separate YLS/CMI events were discussed (typically two per officer). Observational Data: Observations were based on convenience sampling principles that did not formally target a list of subjects. 74 subjects were observed. Only one directly declined (to the researcher) to be observed. This produced approximately 198 hours of direct observations of probation officers at work, evenly split across counties (i.e. approximately 40 hours in each). State Reform Leader Interviews: All 12 subjects targeted were interviewed (i.e. 100 percent response rate). Survey: Of 1322 juvenile probation officers in Pennsylvania targeted for the survey, a total of 492 officers completed the survey and indicated their county affiliation (15 additional excluded cases completed the survey without indicating their county affiliation, preventing their inclusion in analysis). This represents a response rate of 37 percent of officers across all 67 PA counties. If we count officers in only the 56 counties (1223 officers) that ultimately participated, this would represent a response rate of 40 percent. Responses for case study counties A to D ranged from 55% to 88%. Responses from County E were not sufficient for separate analysis without risking deductive disclosure.
description: Datasets: DS0: Study-Level Files DS1: Case Study Subject Summary Data DS2: Client Supervision and Assessment Contacts Observed Data DS3: Court Hearings Observed Data DS4: Decision Making Events Data DS5: Survey Response Data
description: face-to-face interview
description: telephone interview
description: web-based survey
description: Juvenile probation officers working within PA. Data collected also relates to some of their clients. State reform leaders. Smallest Geographic Unit: Counties (anonymized)
description: Core interviews: The study team targeted officers from a range of roles and ranks in each county for interviews, using systematic sampling strategies. For smaller counties (D and E), the study team targeted all officers. In larger counties (A, B and C) the study team purposively selected officers from relevant units, to ensure a representation of role types (e.g. line officers, supervisors, and chief level staff). The study team also selected randomly where they need to choose among multiple eligible officers within an operational unite/role type. While in Counties A and B the study team were able to select from all relevant operational subunits, in County C the study team could only focus on a subset of operational units because of the size and complexity of the office. This made their sampling in this county less representative. YLS/CMI Example Interviews: The study team sought to target all staff from our core interviews who were direct users of the YLS/CMI (i.e. they had caseloads, and routinely completed the YLS/CMI and/or made decisions based on it). As noted, The study team recruited one observation subject to our target list also, to supplement numbers in County C. Once The study team spoke them, they asked them to talk through a couple of recent examples of YLS/CMIs. This was a convenience sampling approach, but emphasized a focus on recent cases. For juvenile probation officers with supervision caseloads The study team typically sought an example of a case they had recently begun supervising (where they may not have personally conducted the YLS/CMI, but had the completion information available), and an example of a YLS/CMI that they had recently completed themselves (usually an update or case-closing YLS/CMI). For intake or investigations probation officers (who didn't supervise probationers), the study team usually asked for two recently completed YLS/CMIs, which tended to be the initial YLS/CMIs conducted on youth at the beginning of their case. Observational Data: Observations were targeted at officers when they were meeting with clients, for example through assessment meetings (e.g. typically at intake), supervision contacts or court appearances. The study team also sought to steer our observations to achieve a variety of activities and probation officers. Te tended to work with managers and administrators to help schedule time with officers at times when researchers were visiting or available, though this process evolved differently in each county, and could also involve direct contacts with probation officers to schedule interviews. In one county, (County B) the study team provided administrators with target lists of officers to help guide them, to ensure a range of roles. Inevitably, given our reliance on convenience and the involvement of office managers and administrators the sampling of observations was influenced by the decisions of office staff, and could have been shaped to present a particular appearance to researchers. State Reform Leader Interviews: The study team began by targeting key individuals who the study team knew had been influential leaders in YLS/CMI implementation and related reforms. Our target list was further expanded based on the suggestions of research subjects, effectively deploying a snowballing strategy. Subjects included administrators, probation officers of varied ranks, and consultants. Survey: The survey targeted all juvenile probation officers (of all ranks) in all 67 county officers of Pennsylvania. This was 1322 officers.