text: Background Job Corps Eligibility Criteria and Program Services To be eligible for the Job Corps program, an individual must generally be 16 to 24 years old at the time of enrollment; be low income; and have an additional barrier to education and employment, such as being homeless, a high school dropout, or in foster care. See table 1 for characteristics of students served by Job Corps during program year 2016. Once enrolled in the program, youth are assigned to a specific Job Corps center, usually one located nearest their home and which offers a job training program of interest. The vast majority of students live at Job Corps centers in a residential setting, while the remaining students commute daily from their homes to their respective centers. This residential structure is unique among federal youth programs and enables Job Corps to provide a comprehensive array of services, including housing, meals, clothing, academic instruction, and job training. In program year 2016, about 16,000 students received a high school equivalency and about 28,000 students completed a career technical training program, according to ETA officials. Job Corps Structure and Operations ETA administers Job Corps’ 123 centers through its national Office of Job Corps under the leadership of a national director and a field network of six regional offices located in Atlanta, Boston, Chicago, Dallas, Philadelphia, and San Francisco (see fig. 1). Job Corps is operated primarily through contracts, which according to ETA officials, is unique among ETA’s employment and training programs (other such programs are generally operated through grants to states). Among the 123 centers, 98 are operated under contracts with large and small businesses, nonprofit organizations, and Native American tribes. The remaining 25 centers (called Civilian Conservation Centers) are operated by the U.S. Department of Agriculture’s (USDA) Forest Service through an interagency agreement with DOL. Job Corps center contractors and the USDA Forest Service employ center staff who provide program services to students. The President’s fiscal year 2019 budget seeks to end USDA’s role in the program, thereby unifying responsibility under DOL. The Administration reported that it was proposing this action because workforce development is not a core mission of USDA, and the 25 centers it operates are overrepresented in the lowest performing cohort of centers. According to ETA officials, the Office of Job Corps has oversight and monitoring responsibility to ensure that center operators follow Job Corps’ Policy and Requirements Handbook, including the safety and security provisions. Job Corps regional office staff are largely responsible for these duties. Requirements for Job Corps Centers Related to Incident Reporting Job Corps’ Policy and Requirements Handbook requires centers to report certain significant incidents to the national Office of Job Corps and to regional offices using SIRS. Centers are required to report numerous categories of incidents, including assaults, alcohol and drug-related incidents, and serious illnesses and injuries (see appendix II for definitions of these categories of incidents). Within the Policy and Requirements Handbook, ETA establishes student standards of conduct that specify actions centers must take in response to certain incidents. In some cases, the incident categories in SIRS are related to the specific infractions defined in the Policy and Requirements Handbook, which are classified according to their level of severity. Level I infractions are the most serious, and includes infractions such as arrest for a felony or violent misdemeanor or possession of a weapon, and are required to be reported in SIRS. Level II includes infractions such as possession of a potentially dangerous item like a box cutter, or arrest for a non-violent misdemeanor. The majority of these infractions are required to be reported in SIRS. Minor infractions—the lowest level—include failure to follow center rules, and are not required to be reported in SIRS. Centers must report incidents involving both Job Corps students and staff, and incidents that occur onsite at centers as well as those that occur at offsite locations. According to ETA officials, the agency and its center operators must take steps to protect the safety and security of Job Corps students when students are under Job Corps supervision. Students are under Job Corps supervision when they are onsite at Job Corps centers and when they are offsite and engaged in center-sponsored activities, such as work-based learning or community service. According to ETA officials, the agency and its contractors are not responsible for protecting the safety and security of Job Corps students when students are offsite and not under Job Corps supervision, such as when students are at home on leave. However, when offsite safety and security incidents of any type occur, Job Corps center operators are responsible for enforcing the student conduct policy. For example, if a student is arrested for a felony offsite while not under Job Corps supervision, the arrest may result in a Level I infraction and dismissal from the program. Job Corps Student Satisfaction Survey Since 2002, ETA used its student satisfaction survey to periodically obtain views from enrolled Job Corps students on various aspects of the program, including career development services, interactions between students and staff, access to alcohol and drugs, and overall satisfaction with the program. The survey of 49 questions has remained the same over time and included 12 questions on students’ perceptions of safety and security at centers. ETA used the responses to the 12 safety-related survey questions to calculate a center safety rating, which represented the percentage of Job Corps students who reported feeling safe at each center, as well as a national safety rating, which represented the percentage of Job Corps students who reported feeling safe nationwide. ETA officials said they used these ratings to assess students’ perceptions of safety at individual centers and nationwide, to monitor and evaluate center operators, and to determine whether ETA needed to take action to better address students’ safety and security concerns. In 2018, ETA will pilot a stand-alone survey for safety related topics and remove the safety questions from the student satisfaction survey. Job Corps Centers Reported Nearly 14,000 Incidents of Various Types during Program Year 2016, Which Mainly Occurred Onsite and Involved Recently Enrolled Males under Age 20 Almost Half of the Reported Onsite and Offsite Incidents Involved Drugs or Assaults Our analysis of ETA’s data from the Significant Incident Reporting System (SIRS) showed that Job Corps centers reported 13,673 safety and security incidents involving students, including those that occurred both onsite and offsite, in program year 2016. During this time period (July 1, 2016, through June 30, 2017), approximately 79,000 students were served by the program, according to ETA officials. Drug-related incidents (29 percent) and assaults (19 percent) accounted for 48 percent of all reported incidents involving students. The remaining 52 percent of reported incidents involving students included breaches of security and safety (12 percent), alcohol-related incidents (6 percent), serious illness and injury (6 percent), theft or damage to property (5 percent), danger to self or others (5 percent), and all other types of incidents (18 percent) (see fig. 2). According to ETA officials, about half of the 3,926 drug- related incidents are due to positive drug test results among students that are administered drug tests about 40 days after entering the program. We found that about 20 percent of reported onsite and offsite incidents in program year 2016 were of a violent nature, which we define as homicides, sexual assaults, and assaults. There were two reported homicide incidents in program year 2016 and both occurred while students were offsite and not under Job Corps supervision. Also, centers reported 177 sexual assaults and 2,593 assaults involving students during program year 2016. For each reported sexual assault and assault, SIRS provides an additional description of the incident (see table 2). In our June 2017 testimony, we stated that 49,836 onsite and offsite safety and security incidents of various types were reported by Job Corps centers between January 1, 2007, and June 30, 2016, based on our preliminary analysis of ETA’s SIRS data. We cannot compare our analysis of safety and security incidents in our June 2017 testimony to the analysis contained in this report for program year 2016 due to a policy change by ETA beginning July 1, 2016, which affected the categorization and number of reportable incidents. Specifically, ETA changed the way some incidents are defined, and required that some incidents be reported in SIRS that previously had no such requirement. Anecdotally, officials from one ETA regional office and two Job Corps centers that we visited said that the number of reported incidents has increased since July 1, 2016, due to these changes. In its December 2017 report, the DOL OIG compared the number of safety and security incidents reported to the OIG for the same 8-month periods in 2016 and 2017 and found an increase of 134 percent. According to the DOL OIG, this increase is likely due to more accurate incident reporting as a result of the recent policy change. In addition, the DOL OIG said an actual increase in incidents is also possible. Most Reported Incidents Occurred Onsite, but Arrests and Deaths Most Frequently Occurred Offsite While Students Were Not Under Job Corps Supervision Our analysis of SIRS data found that in program year 2016, 90 percent of the 13,673 reported safety and security incidents involving students occurred onsite at Job Corps centers, and 10 percent occurred at offsite locations (see fig. 3). For example, 99 percent of drug-related incidents, 96 percent of assault incidents, and 84 percent of alcohol-related incidents occurred onsite. While most reported incidents occurred onsite, our analysis showed that the majority of reported arrests, deaths, and motor vehicle accidents occurred offsite. For example, of the 21 student deaths,18 occurred at offsite locations and 3 occurred onsite. In our June 2017 testimony, we reported that from January 1, 2007, through June 30, 2016, 76 percent of the reported safety and security incidents occurred onsite at Job Corps centers, and 24 percent occurred at offsite locations based on our preliminary analysis of ETA’s SIRS data. However, as previously noted, that analysis is not comparable to the analysis in this report for program year 2016 due to ETA’s July 1, 2016, policy change that impacted the categorization and number of reportable incidents. We analyzed the 1,406 incidents of 13,673 total reported incidents that were reported to have taken place offsite in program year 2016 to determine if the students involved were on duty (i.e., under Job Corps supervision) or off duty (i.e., not under Job Corps supervision). We found that for offsite incidents, similar percentages of student victims and perpetrators were on duty and off duty. Specifically, we found that 50 percent of student victims were on duty, 44 percent were off duty, and we were unable to determine the duty status of 6 percent. For student perpetrators, we found that 45 percent of students were on duty, 45 percent were off duty, and we were unable to determine the duty status of 10 percent. Some types of reported incidents occurred more frequently when students were offsite and off duty. For example, of the reported arrest incidents that occurred offsite, 76 percent of student perpetrators were off duty. Of the reported death-related incidents that occurred offsite, student duty status was reported as off duty for 16 of 18 incidents. We were unable to determine the duty status for all students involved in offsite incidents due to inconsistencies in ETA’s data. Of the 1,406 offsite incidents reported in SIRS, there were 178 instances in which a student’s duty status location conflicted with the incident location. For example, the student’s duty status was listed as onsite and on duty, but the incident location was listed as offsite. We asked ETA officials why these inconsistencies existed and they were unable to explain all instances in which these inconsistencies occurred. ETA officials did state, however, that these inconsistences can sometimes occur when centers enter information in SIRS based on the student’s duty status at the time the incident report is completed instead of the student’s duty status at the time the incident occurred. Due to this data limitation, we were unable to determine if the 178 students involved in those incidents were on duty or off duty. Student Victims and Perpetrators Most Often Were Recently Enrolled Males under Age 20, Reflective of the Job Corps Population We analyzed SIRS data to determine the characteristics of students involved in reported safety and security incidents and found that about 17,000 students were reported as victims or perpetrators of all onsite and offsite incidents in program year 2016. The total number of students reported as victims or perpetrators is 22 percent of the students served in program year 2016. The number of student victims and perpetrators varied across incident types (see fig. 4). In program year 2016, we found that about 5,000 students (6 percent of students served) were reported as victims of various types of onsite and offsite incidents. We separately examined the gender, age, and enrollment time of reported student victims and found that for all reported incidents the majority of student victims were male, under age 20, and enrolled in Job Corps for less than 4 months (see fig. 5). These characteristics are somewhat similar to the overall Job Corps student population, which is primarily male and under age 20, as previously noted. For example, 65 percent of reported assault victims and 73 percent of reported theft victims were male. However, the number of female victims exceeded the number of male victims within some reported incident categories, such as sexual assault, inappropriate sexual behavior, and missing persons. Students under age 20 were victims of 67 percent of reported assault incidents and 63 percent of danger to self or others incidents. According to ETA officials, 18 percent of students served in program year 2016 were enrolled for less than 4 months; however, across all reported incidents 56 percent of student victims were enrolled for less than 4 months. For example, about 60 percent of student victims of reported assault and danger to self or other incidents were enrolled in Job Corps for less than 4 months. Our analysis of SIRS data shows that about 13,000 students (17 percent of students served) were reported as perpetrators of various types of onsite and offsite incidents in program year 2016. The most commonly reported incidents—drug-related and assaults—also had the highest numbers of student perpetrators. We found that 6 percent and 5 percent of students served in program year 2016 were perpetrators of reported drug-related and assault incidents, respectively. Similar to our analysis of student victims, we separately examined student characteristics and found that the majority of reported student perpetrators of all reported incidents were male, under age 20, and enrolled in Job Corps for less than 4 months (see fig. 6). Students Generally Reported Feeling Safe; ETA Plans to Create a New, Expanded Survey Most Students Reported Feeling Safe, but Fewer Reported Feeling Safe on Selected Questions Our analysis of ETA’s student satisfaction survey data from program year 2016 showed that while students generally reported feeling safe at Job Corps centers, a smaller proportion reported feeling safe in certain situations. ETA considers students to feel safe if they provide certain responses to each of the 12 safety-related survey questions, some of which are phrased as statements. For example, if a student provided a response of “mostly false” or “very false” to the statement “I thought about leaving Job Corps because of a personal safety concern,” that student would be counted as feeling safe on that survey question. On 6 of the 12 safety-related survey questions in program year 2016, at least 70 percent of responding students indicated that they felt safe (see table 3). For example, 74 percent of students responded that they did not ever or in the last month carry a weapon, and 83 percent of students responded that it was very or mostly true that a student would be terminated from Job Corps for having a weapon at the center. These are responses that ETA considered to indicate feeling safe. At the two centers we visited, students that we interviewed said that they felt safe onsite at their center. For example, students at one center said that they felt safe because absolutely no weapons, fighting, or drugs were allowed at the center. A smaller number of students reported feeling safe on questions that dealt with hearing threats or hearing things from other students that made them feel unimportant. For example, 36 percent of students reported they had not ever or in the last month heard a student threaten another student at the center, which is considered safe according to ETA policy. Meanwhile, 49 percent reported that they had heard a student threaten another student at least once in the last month, and ETA considered these responses to indicate that students felt unsafe. Another 15 percent chose “don’t know / does not apply.” On another question, 53 percent of students reported that other students had not ever or in the last month said things that made them feel like they were not important, which ETA considered as feeling safe. Yet 30 percent reported that others made them feel unimportant at least once in the last month—which ETA considered as feeling unsafe—and 17 percent chose “don’t know / does not apply.” In response to a question about the student conduct policy, 35 percent of students indicated that the policy was not applied equally to all students. At the two centers we visited, students that we interviewed had varying views on applying the student conduct policy. Students from one center said that staff have applied the policy in a fair way. Yet at another center, students told us that they have occasionally perceived that staff have not applied the student conduct policy fairly. They mentioned that they were aware of favoritism in a few recent incidents when staff applied the policy’s disciplinary consequences for certain students but not others. For example, they said that a student they perceived as the perpetrator remained in Job Corps while a student they perceived as innocent was dismissed. Our June 2017 testimony contained similar observations about students’ perceptions of their safety, with students generally reporting that they felt safe at their Job Corps centers. For example, most students reported feeling safe because a student found with a weapon at the center would be terminated. In that testimony, we also noted that students reported feeling less safe on such questions as hearing threats or applying the student conduct policy. In addition to the 12 safety-related questions, we examined data on the 2 questions about access to alcohol or drugs, and found that almost two- thirds of survey respondents said that it was mostly or very false that they could access alcohol or drugs at their Job Corps center. Although a large number of reported incidents in program year 2016 involved drugs or alcohol, less than 15 percent of survey respondents said that it was mostly or very true that they could access alcohol or drugs at their Job Corps center. National Measures of Safety and Security Have Been Developed Based on students’ responses to the 12 safety-related questions, ETA determined that 88 percent of students indicated that they felt safe in program year 2016. ETA calculated its national measure of safety— referred to as a safety rating—to summarize and track students’ perceptions of their safety and to determine the need for additional action, as noted previously. Similarly, it calculated a safety measure for each center. However, we calculated a national measure differently and found that an average of 73 percent of students reported feeling safe in program year 2016. Our national measure reflected the average of how safe each student felt on the 12 safety-related survey questions. We estimated that one key difference accounted for about 11 of the 15 percentage points between our and ETA’s measure. (See table 7 in appendix I.) Specifically, we calculated our measure based on a numeric average for each student without rounding. For example, if a student answered all 12 safety questions with 6 responses that he felt safe and another 6 that he felt unsafe, we counted this student as half safe (0.5). Meanwhile, ETA rounded the average to either safe or unsafe, so that ETA counted a student with 6 safe responses and 6 unsafe responses as feeling safe. In addition to differences in calculations, we developed our own national measure of safety because it is important to assess and track students’ perceptions for the program as a whole, as ETA has noted. Also, a national measure facilitates analysis of groups of students, such as male or female students or younger or older students, as described below. We examined whether our national measure differed by age, gender, time in program, center size, or operator type and found statistically significant and meaningful differences in our national measure by students’ length of time in the program. In particular, an average of 78 percent of students in the program for less than 4 months responded that they felt safe, compared to an average of 71 percent for students in the program for at least 4 months. According to ETA officials, differences in responses based on length of time in the program may relate to new students being less aware about life at the center because they begin the program with other newly arrived students for up to 2 months. For example, ETA officials said that new students may live in a dormitory specifically for new students. Thus, they are not yet fully integrated into the larger student body. Although differences were also statistically significant between age groups, center size, and operator type, such differences were not meaningful in a practical manner (i.e., around 3 percentage points or less). Differences in our national measure by gender were not statistically significant. When we analyzed the survey’s separate question about overall satisfaction with Job Corps, we found that students who reported they were satisfied with the Job Corps program responded that they felt safer than students who were not satisfied. In program year 2016, about two- thirds of students said it was very or mostly true that they would recommend Job Corps to a friend, which ETA uses to gauge overall satisfaction with the program. Of the 65 percent of students who would recommend Job Corps to a friend, 79 percent said they felt safe. Of the 11 percent of students who would not recommend Job Corps to a friend, 52 percent felt safe. ETA’s New Web-based Survey Is Designed to Be More Timely and Detailed ETA officials said that the agency is creating a new expanded safety survey to improve upon the prior survey. With Job Corps’ heightened attention to safety and security, the new survey—the Student Safety Assessment—is focused solely on safety and security issues and is designed to provide more timely and more detailed information. More timely information. ETA plans to administer the new safety survey monthly to a random sample of students rather than twice per year to all enrolled students. Also, it will be web-based, rather than the current paper-based survey. As a result, ETA officials said that they will receive more timely information from students because it will take less time to administer the survey and analyze the responses. More detailed information. The number of questions about center safety will increase from 12 to about 50—pending finalization of the survey—which is about the same number of questions on the current student satisfaction survey. For example, the new questions will ask about sexual assaults and harassment or the types of drugs bought or used at the center, which were not topics covered by the prior survey. ETA continues to work with its contractor with survey expertise to develop, test, and administer the new survey in 2018, according to ETA officials. To develop the new survey, ETA and its contractor have considered, incorporated, and revised questions from other existing surveys. For example, they have drawn from safety surveys of teenage students and postsecondary students. ETA plans to continue developing and refining the survey and its administration in 2018, including conducting monthly pilots from January to June 2018, assessing response rates, and developing a new way to calculate national and center-level safety measures. Additionally, ETA officials said that, in 2018, they will seek to obtain comments and approval on the survey from the Office of Management and Budget. ETA officials told us that they plan to administer the new survey nationally by January 2019. As ETA refines and administers this new survey, officials told us they plan to develop a new way to measure student safety based on the more detailed survey. ETA Initiated Multiple Actions to Improve Center Safety and Security, but the New Monitoring Strategy Was Implemented Inconsistently and ETA Lacks a Comprehensive Plan ETA Initiated Multiple Actions to Improve Center Safety and Security In 2014, ETA launched multiple actions to improve safety and security at Job Corps centers in response to DOL OIG recommendations (see table 4). For example, in 2015 the DOL OIG found ETA’s oversight of Job Corps centers ineffective, in part, because ETA’s student conduct policy excluded some violent offenses. As a result, ETA revised its student conduct policy by elevating several infractions previously classified as Level II to Level I (the most severe) and by adding several new categories of reportable incidents. Under the revised student conduct policy, assault, a Level I infraction, now includes fighting, which was previously a Level II infraction. In addition, the DOL OIG found that ETA did not monitor centers regularly enough to ensure center consistency in administering Job Corps disciplinary policies. In response, ETA implemented a risk- based monitoring strategy that identifies potential safety and security issues before they occur. Staff from five ETA regional offices and at one Job Corps center we visited said that ETA’s actions overall helped to improve center safety and security. For example, staff from five regional offices said that the changes to the student conduct policy that were implemented in July 2016 clearly describe the penalties for infractions and eliminate grey areas that previously allowed center staff to use their professional judgement. Staff from four regional offices also said these changes resulted in tradeoffs that reduced center staff discretion in imposing penalties. In addition, at one center we visited, the Director of Safety and Security told us he updated the center’s security-related standard operating procedures in response to ETA’s guidance. ETA’s guidance was part of the 2017 updates to the Policy and Requirements Handbook in response to DOL OIG concerns about reporting potentially serious criminal misconduct to law enforcement. ETA Officials Reported That Some New Actions Improved Center Monitoring, but That Actions Were Inconsistently Implemented and May Create Reporting Overlaps ETA national officials said that the new risk-based monitoring strategy has improved center monitoring because it has allowed them to more effectively direct resources to areas of greatest need. Officials in five ETA regional offices agreed that the new strategy improved their ability to monitor centers. The new monitoring strategy shifted the focus from addressing problems after they have occurred to a data-driven strategy that tracks center performance and identifies emerging problems. This strategy provides ETA and center operators an opportunity to address problems before they occur, according to ETA national officials. For example, the new monitoring strategy features new tools, including the Risk Management Dashboard. The dashboard is a summary analysis tool that conducts trend analysis using center data and allows regional staff to engage in targeted interventions at centers with potential safety and security concerns. In addition, under the new monitoring strategy, instead of only conducting scheduled monitoring visits to a center at set times, regional staff conduct unannounced visits based on data indicating a decline in center performance or other triggers. See appendix VI for additional information on the new monitoring strategy. Although the new risk-based monitoring strategy has improved center monitoring, it is not consistently implemented across regional offices, according to ETA national officials. They told us that similar problems identified at centers may be treated with different levels of focus or intensity from one region to another. In addition, national and regional officials told us that regional office staff have relied on professional judgment to determine the appropriate response to centers that may be at risk of noncompliance with safety and security policies, which could lead to inconsistencies. For example, when problems are identified at centers, the type of assessment to conduct is left to regional office staff discretion. As a result, staff in one region may decide that the most comprehensive assessment, the Regional Office Center Assessment, is needed, while another region’s staff would select a targeted assessment, which is more limited in scope. ETA national officials said that although each determination could be justified based on resource constraints and competing priorities, they would like to increase implementation consistency in this area. To address regional inconsistencies, ETA national and regional office staff said that guidance in the form of standard operating procedures (SOP) would be helpful. These procedures would promote consistency in how policies are interpreted and applied and would help ensure that centers are held to the same standards, according to ETA national officials. For example, SOPs could specify which type of assessment to conduct in response to specific problems identified at centers. Internal control standards state that managers should document in policies each unit’s responsibility for an operational process. Regional office staff said that they previously had a helpful tool, the Program Assessment Guide, that linked policies in the Policy and Requirements Handbook to the monitoring assessment process. Regional office staff said they used the Program Assessment Guide to prepare for center monitoring visits and it was a helpful training tool for new staff. Our review of ETA documentation found that the Program Assessment Guide included specific questions to ask center staff about how they meet safety and security requirements and suggested where to look for information to determine center compliance with policies. However, the Program Assessment Guide, which has not been updated since 2013, does not include recent changes to the Policy and Requirements Handbook, such as the updated student conduct policy. ETA national officials told us that limited staffing has made it difficult to update the Program Assessment Guide as frequently as changes are made to the Policy and Requirements Handbook. In February 2018, ETA national officials told us they plan to issue a variety of SOPs related to monitoring center safety and security issues (see table 5). ETA officials initially said these SOPs would be completed in August or November 2018 and later revised its plans with a goal of completing all SOPs by August 2018. However, in August 2017, ETA officials had told the DOL OIG that these SOPs would be completed in the March to July 2018 timeframe. ETA officials said that a staffing shortage in the Office of Job Corps’ Division of Regional Operations and Program Integrity delayed development of the SOPs. This Division— established in 2015 to coordinate regional operations and strengthen communications and quality assurance—includes eight staff positions; however, as of January 2018, the Division has two staff members on board. ETA officials said that they have not yet received departmental approval to fill the six vacant positions in the Division. Given this uncertainty, it is questionable whether ETA’s revised timeframes will be met. Without SOPs or other relevant guidance, ETA cannot ensure that monitoring for center safety and security will be carried out uniformly across the program. As a result, centers may be held to different standards, and the program may not achieve its center safety and security goals. In addition to inconsistencies in monitoring and a lack of sufficient guidance, staff in all six regional offices told us that components of ETA’s risk-based monitoring strategy created reporting overlaps. As part of the new monitoring strategy, regional staff have additional reports that they complete—such as the Risk Management Dashboard Action report and Corrective Action Tracker—about potential safety and security problems or actual violations found at centers. Some regional staff said the desk monitoring report includes similar information to the Risk Management Dashboard and Corrective Action Tracker reports, which regional offices submit to the ETA national office. Staff in one regional office said that they enter the same information about the status of center safety and security violations multiple times on the Corrective Action Tracker because the time between reporting periods is too short to allow for meaningful action to be taken. Staff from four regional offices said completing duplicative reports reduces time that could be used to conduct additional center monitoring, such as onsite visits, or to perform other key duties. ETA national officials disagreed that overlap exists among monitoring reports. They said that although reports may appear to overlap, the reports are complementary and not duplicative, and are used at different points in the monitoring process (see fig. 7 for an overview of ETA’s monitoring process). For example, ETA national staff told us that desk monitoring reports are primarily used by regional staff at the beginning of the monitoring process to identify potential problems and are not substantially reviewed by the national office. ETA national officials also said that the Risk Management Dashboard report is used at the beginning of the monitoring process to identify problems, whereas the Corrective Action Tracker is used later in the process after violations have been identified and corrective actions have been planned to bring the center back into compliance. In addition, ETA national officials also noted that regional staff are not asked to complete all reports every month. For example, regional staff complete a Risk Management Dashboard Action report only for those centers with potential safety and security concerns. We compared the information included in five monitoring reports—the Center Culture and Safety Assessment, Corrective Action Tracker, Desk Audit, Regional Office Center Assessment, and Risk Management Dashboard Action report—and found opportunities for streamlining. For example, we found that the Center Culture and Safety Assessment, Corrective Action Tracker, and Regional Office Center Assessment, all include a narrative description of the violations identified by regional staff categorized according to the corresponding requirement in the Policy and Requirements Handbook. In addition, ETA regional office staff said the Corrective Action Tracker, a Microsoft Excel spreadsheet, is cumbersome to use and within the spreadsheet they attach and submit additional documentation. ETA national officials agreed that streamlining or automating monitoring tools would be helpful for its regional staff, along with additional training to help staff understand the different reports and how to write the required narratives. ETA national officials also told us that they did not systematically review existing reports before creating additional ones for the new risk-based monitoring process. Officials said they have lacked the resources to make some improvements that could reduce the time regional office staff spend on reporting. Standards for internal control state that managers should identify the organizational level at which the information is needed, the degree of specificity needed, and state that managers should review information needs as an on-going process. Streamlining or automating reporting requirements can help centralize documentation relevant to monitoring center safety and security, possibly eliminate seemingly duplicative reporting requirements, and help regional staff manage their workloads. ETA Lacks a Comprehensive Plan to Link Its Various Efforts to Improve Center Safety and Security While ETA initiated multiple actions to address various safety and security issues, the agency does not have a comprehensive plan to improve center safety and security. A comprehensive plan describes the organization’s long-term goals, its strategy and timelines for achieving those goals, and the measures that will be used to assess its performance in relationship to its goals. It can also guide decision-making to achieve desired outcomes, including the priority with which to implement these efforts. ETA officials told us that although they do not have a single document that reflects a formal comprehensive plan, they have employed a comprehensive approach to improve center safety and security. However, in prior work, GAO established the importance of comprehensive planning to ensure agencies effectively execute their missions and are accountable for results. GAO has also identified leading practices that help ensure organizations achieve their objectives. These leading practices include developing goals, strategies to achieve goals, plans to assess progress toward goals, and leadership and stakeholder involvement in plan development (see table 6). ETA officials agreed that a comprehensive plan is needed, but told us that limited staff capacity and lack of expertise have hindered their ability to produce a comprehensive plan. In particular, the Division of Regional Operations and Program Integrity would have a role in developing the agency’s comprehensive plan. As previously mentioned, ETA officials told us that they did not have approval to fill the six vacant positions in the Division. With only two of the eight positions filled, ETA officials said that they prioritized correcting the deficiencies identified by the DOL OIG and responding to immediate safety and security concerns. ETA officials told us they plan to produce a comprehensive plan when they have secured the staff to do so. However, at this time, ETA does not have a specific timeframe for producing such a plan. When the agency begins developing a comprehensive plan, it could consider using the leading practices outlined above and drawing on the expertise of the government-wide Performance Improvement Council. In the absence of a comprehensive plan for safety and security, ETA risks the success of its new initiatives because they are not linked in an overall framework that demonstrates how they are aligned or contribute to goals for improving center safety and security. Conclusions It is important that Job Corps students be provided with a safe and secure learning environment. For the last several years, however, numerous incidents have threatened the safety and security of students. ETA has taken steps to improve center safety and security, but its efforts could be strengthened by ensuring regional office staff responsible for monitoring Job Corps centers are better supported with additional guidance and streamlined reporting requirements. Without providing regional staff with this additional support, the full potential of the new monitoring strategy may not be realized. While ETA has implemented several actions to address safety and security concerns, it does not have a comprehensive plan to guide all of its efforts. Without a comprehensive plan, ETA will not be able to assess its overall effectiveness in addressing center safety and security. Recommendations for Executive Action We are making the following three recommendations to ETA: The Assistant Secretary of ETA should ensure the Office of Job Corps expeditiously develops additional guidance, such as SOPs or updates to the Program Assessment Guide, to ensure regional offices consistently implement the risk-based monitoring strategy. (Recommendation 1) The Assistant Secretary of ETA should ensure the Office of Job Corps streamlines the monitoring reports completed by regional office staff. This streamlining could include automating monitoring tools, consolidating monitoring reports, or taking other appropriate action. (Recommendation 2) The Assistant Secretary of ETA should ensure the Office of Job Corps commits to a deadline for developing a comprehensive plan for Job Corps center safety and security that aligns with leading planning practices, such as including a mission statement with goals, timelines, and performance measures. This could also include developing the planning expertise within the Office of Job Corps, leveraging planning experts within other agencies in DOL, or seeking out external experts, such as the government-wide Performance Improvement Council. (Recommendation 3) Agency Comments and Our Evaluation We provided a draft of this report to DOL for review and comment. We received written comments from DOL, which are reprinted in appendix VII. DOL concurred with our three recommendations. The department stated that it will move forward to develop standard operating procedures for its risk-based monitoring strategy, review and streamline existing monitoring reports, and provide additional training for its regional office staff. The department also plans to develop a formal written comprehensive plan for Job Corps safety and security. DOL also provided technical comments that we have incorporated in the report as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Secretary of Labor. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. Appendix I: Additional Information about Our Methodology The objectives of this review were to examine (1) what is known about the number and types of reported incidents involving the safety and security of Job Corps students in program year 2016; (2) what is known about student perceptions of safety and security at Job Corps centers, and what steps, if any, is the Employment and Training Administration (ETA) taking to improve the survey used to collect this information; and (3) the extent to which ETA has taken steps to address safety and security at Job Corps centers. To address all three objectives, we reviewed agency policies and procedures, such as the Job Corps Policy and Requirements Handbook and guidance issued to center operators and ETA staff. In addition, we interviewed ETA officials, including Office of Job Corps national staff, Office of Job Corps regional directors, and staff in all six regional offices. We also conducted site visits at the Woodstock Job Corps Center in Woodstock, Maryland, and the Potomac Job Corps Center in Washington, D.C. We selected these two centers because they were within geographical proximity to Washington, D.C., operated by different contractors, and had over 100 reported safety and security incidents each in program year 2016. At each center, we interviewed the Center Director, Head of Safety and Security, a group of staff members, and a group of students. The staff and students we spoke with were selected by the centers. While these two site visits are not generalizable to all Job Corps centers, they provide examples of student and staff experiences with safety and security. Analysis of Safety and Security Incidents at Job Corps Centers To determine the number and types of safety and security incidents reported by Job Corps centers, we analyzed ETA’s incident data for program year 2016 (July 1, 2016 to June 30, 2017). This was the most recent year of Job Corps data available at the time of our review. ETA captures these data in its Significant Incident Reporting System (SIRS). Centers must report incidents involving both Job Corps students and staff, and incidents that occur at onsite and offsite locations. ETA has 20 categories of incidents in SIRS. See appendix II for incident category definitions. The incident categories and definitions in this report are taken directly from ETA documents and represent how ETA categorizes these incidents. We did not assess these categories and definitions. In this report, we present information on reported safety and security incidents in program year 2016 involving at least one student victim or perpetrator. There were 13,673 reported incidents involving students; additional incidents are reported in SIRS that did not involve students. When these additional incidents are included, a total of 14,704 safety and security incidents were reported in program year 2016. See appendix III for further information on the total number of incidents reported. To calculate the number and types of reported incidents, we analyzed the primary incident type that was assigned to each incident reported in SIRS. To provide additional information on reported assaults and sexual assaults, we also analyzed the secondary incident type that was assigned to each reported assault and sexual assault in SIRS. To calculate the total number and types of reported deaths, we analyzed both primary incident types and secondary incident types. In SIRS, deaths can be reported under three different primary incident types (“death”, “assault”, and “danger to self or others”). When an incident is assigned to any of these primary incident types, it may also be assigned a secondary incident type of “homicide,” among other secondary incident types. In addition, we analyzed the duty status for student victims and perpetrators of offsite incidents. In SIRS, students are described as being either (1) on duty, which means that they are onsite at a center or in a Job Corps supervised offsite activity; or (2) off duty, which means they are offsite and not under Job Corps supervision. For the 1,406 offsite incidents, we were unable to determine student duty status in 178 instances due to inconsistencies in ETA’s data. This report focuses on reported safety and security incidents in program year 2016, which was from July 1, 2016, to June 30, 2017. On July 1, 2016, ETA implemented policy changes that impacted the categorization and number of reportable safety and security incidents. Accordingly, incident data after July 1, 2016, are not comparable with earlier incident data, including incident data we reported in a June 2017 testimony. We assessed the reliability of SIRS data by reviewing relevant agency documentation about the data and the system that produced them and interviewing ETA and Department of Labor Office of Inspector General (DOL OIG) officials knowledgeable about the data. We determined the data were sufficiently reliable to report the minimum number of incidents that occurred in program year 2016. It is likely that the actual number of incidents was greater than the number reported in SIRS because the information is reported by Job Corps centers and the DOL OIG previously found instances of underreporting by a non-generalizable sample of center operators. In its March 2017 report, DOL OIG found that 12 of 125 Job Corps centers did not report 34 percent of significant incidents in SIRS from January 1, 2014, through June 30, 2015. ETA has recently taken steps to improve center reporting of significant incidents, such as revising the student conduct policy to more clearly define behavior infractions and conducting system-wide training to ensure uniform understanding and enforcement of student conduct policies. However, DOL OIG officials told us in January 2018 that it is too early to determine if these steps have resolved the DOL OIG’s concerns regarding center underreporting. Analysis of Student Perceptions of Safety Survey Response Rate and Reliability To examine what is known about student perceptions of their safety and security at Job Corps centers, we analyzed students’ responses to the student satisfaction survey administered during program year 2016: September 2016 and March 2017. We analyzed responses from both of these surveys in program year 2016, which was the most recent year for which data were available. ETA provided centers with the standardized paper-based survey to administer to students in-person on designated weeks. The survey of 49 close-ended questions contained 12 questions that ETA used to assess students’ safety. In addition to questions on student safety, the survey includes questions on other topics, including student demographics, overall satisfaction with Job Corps, and access to drugs and alcohol on center. According to data from ETA, the response rate for each survey was approximately 90 percent of all enrolled students. ETA calculated the response rate by dividing the number of students who responded to the survey by the number of enrolled students during the week of survey administration. Students responded anonymously to the survey. Because about 90 percent of students provided responses and about 10 percent did not, we analyzed the potential for non-response biases based on several student characteristics. If the responses of those who did not respond would have differed from the responses of those who did on relevant safety questions, the results calculated solely from those who responded may be biased from excluding parts of the population with different characteristics or views. We compared age, time in program, race, and gender—key characteristics available for the population of enrollees and respondents—to determine areas for potential bias. We determined that the potential for non-response biases existed for particular groups of students: younger students and those enrolled in the program for at least 6 months. For race, the potential for non-response bias was unclear. We found no potential bias for gender. Specifically, we found the following: Age. Younger students were under-represented, and older students were over-represented among survey respondents. Thus, to the extent that non-responding younger students would have answered safety questions differently than responding younger students, the potential for bias existed in the survey results we analyzed. When we asked ETA officials about such a potential bias, they responded that they did not have evidence or documentation suggesting that age is a predictor of students’ level of perceived safety in the program. Length of time in the program. Students in the program less than 6 months were over-represented among survey respondents, and students enrolled in the program over 6 months were under- represented in the survey. To the extent that non-responding students would have answered safety questions differently based on length of time enrolled, the potential for bias existed in the survey results we analyzed. When we asked ETA officials about such a potential bias, they noted that new students may be less aware about life at the center because they begin the program with other newly arrived students for up to 2 months. Thus, they are not yet fully integrated into the larger student body. Otherwise, they did not have evidence or documentation suggesting that length of time in the program correlates with students’ level of perceived safety. Race. It is unclear whether the distribution of race for respondents differs from that in the population. Specifically, ignoring item non- response, about 7 percent of respondents selected “Other,” and if those respondents were Black/African American, the distributions between the respondents and sample would be similar since this would result in the respondent race percentage being close to 50 percent, like the population of enrollees. If respondents who selected “Other” were actually distributed across the race categories, this would result in a difference between the respondent and population race/ethnicity characteristics, and to the extent that students’ responses to safety questions differ by race, this could result in a potential bias of respondent survey results we analyzed. We analyzed race for purposes of potential non-response bias, and not as part of statistical tests of survey results described below. Gender. We found no potential non-response bias for gender because the distribution of gender for respondents was similar to that in the population of students enrolled in the program. In addition to our non-response bias analysis, we assessed the reliability of the survey data by reviewing relevant agency documentation about the data and the system that produced them, testing data electronically, and interviewing ETA officials knowledgeable about the data. We determined that the student survey data were sufficiently reliable for our purposes. Calculations of Safety for Individual Survey Questions and for National Measures For the 12 safety-related survey questions, Job Corps policy specified responses that the agency counted as safe or unsafe, which we followed. As noted previously, ETA considers students to feel safe if they provided certain responses to each of the 12 safety-related survey questions, some of which are phrased as statements. For example, if a student provided a response of “mostly false” or “very false” to the statement “I thought about leaving Job Corps because of a personal safety concern,” that student would be counted as feeling safe on that survey question (see table 3). The percentages that we calculated are not comparable to prior publications, including ETA reports, because, for example, ETA revised (i.e., recoded) students’ responses in certain circumstances, as explained below in table 7. Meanwhile, we used the original responses that students provided and did not revise them. Also, ETA excluded responses of “don’t know / does not apply” from its percentages. As a result, our percentages are not comparable with those reported by ETA. We also calculated national measures of safety for the program and for particular demographic groups of students (e.g., male, female). Our calculation was similar to ETA’s national safety rating in certain respects. For example, as ETA did, we determined how safe each individual student felt as the unit of analysis. Therefore, the national measures of GAO and ETA may not equal the average of the 12 questions because, for example, not all students answered every safety question. However, in other respects, we produced our national measure differently than ETA. Table 7 explains the three ways that our calculation differed from ETA’s. Although the student safety surveys were an attempt to survey a census of the population of participants, we treated the survey as a sample in certain respects due to the non-response of about 10 percent of students as well as the ongoing nature of the regularly repeated survey. Therefore, we considered these data as a random sample from a theoretical population of students in this program and used statistical tests to assess any differences. Treating the data as a statistical sample, we carried out statistical tests of differences in safety measures for student characteristics (e.g., age, gender, length of time in the program). Because of the large sample size, smaller differences may be detected as statistically significant. This is because statistical significance is a function of the magnitude of the true difference (statistical tests are more likely to detect differences when the true values are very different) as well as the sample size (larger samples can detect statistical significance of smaller magnitudes, when compared to smaller sample sizes, when all else is equal). However, we used statistical significance in conjunction with whether the detected differences are meaningful or important, in a practical sense. In particular, we used a series of f-tests to statistically test, at the alpha = 0.05 level, for difference in average safety measure, across categories of age, gender, time in program, center size, and operator type. Appendix II: Categories of Incidents in the Significant Incident Reporting System (SIRS) Appendix II: Categories of Incidents in the Significant Incident Reporting System (SIRS) Appendix III: All Significant Incidents Reported by Job Corps Centers in Program Year 2016 Our analysis of the Employment and Training Administration’s (ETA) Significant Incident Reporting System (SIRS) data showed that there were 14,704 reported safety and security incidents at Job Corps centers in program year 2016, which include incidents involving students, staff, and non-Job Corps individuals. See table 9. Appendix IV: Reported Safety and Security Incidents Involving Students by Job Corps Center, Program Year 2016 Job Corps centers reported 13,673 safety and security incidents involving students, including those that occurred both onsite and offsite, in program year 2016. See table 10 for information on each Job Corps center, including the number of incidents involving students reported in program year 2016. Appendix V: GAO Safety Measure for Job Corps Centers, March 2017 We calculated safety measures for each Job Corps center, based on student responses to the safety-related questions on the student satisfaction survey (see table 11). We used the methodology described in appendix I to calculate safety measures for the centers. Results in table 11 are from the March 2017 survey, the most recent for program year 2016. The percentages in this table are not comparable and should not be analyzed with the numbers of reported incidents at each center because they are distinct measures that cover different periods of time. Appendix VI: ETA’s Monitoring of Job Corps Centers The Employment and Training Administration’s (ETA) risk-based monitoring strategy is designed to identify emerging problems that place a Job Corps center at-risk for safety and security problems. The strategy is largely implemented by regional office staff, which work with the Office of Job Corps’ newly formed Division of Regional Operations and Program Integrity and use a variety of tools to assess, track, and report on center performance (see table 12). Appendix VII: Comments from the Department of Labor Appendix VIII: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Mary Crenshaw (Assistant Director), Andrea Dawson (Analyst-in-Charge), Sandra Baxter, and Matthew Saradjian made key contributions to this report. Additional assistance was provided by Alex Galuten, Gretta Goodwin, Benjamin Licht, Grant Mallie, Mimi Nguyen, Nhi Nguyen, Monica Savoy, Almeta Spencer, Manuel Valverde, Kathleen van Gelder, and Sonya Vartivarian.
target: Deficiencies identified in multiple DOL Inspector General audits since 2009 and two student deaths in 2015 have raised concerns regarding the safety and security of Job Corps students. GAO was asked to review safety and security of students in the Job Corps program. GAO's June 2017 testimony summarized preliminary observations. This report further examines (1) the number and types of reported safety and security incidents involving Job Corps students; (2) student perceptions of their safety at Job Corps centers; and (3) the extent to which ETA has taken steps to address safety and security at Job Corps centers. GAO analyzed ETA's reported incident data for Job Corps centers from July 1, 2016, through June 30, 2017. GAO also analyzed ETA's student survey data from the same period, reviewed relevant documentation, and interviewed ETA officials at its national office and all six regions. GAO also visited two Job Corps centers that had different operators and at least 100 recent incidents. These two centers are not generalizable to all centers. Job Corps centers reported 13,673 safety and security incidents involving students from July 2016 to June 2017, according to GAO's analysis of the Department of Labor's (DOL) Employment and Training Administration's (ETA) data. Most reported incidents occurred onsite and involved recently enrolled male students under age 20. During that time, the program served about 79,000 students at 125 Job Corps centers, according to ETA officials. ETA's Office of Job Corps administers the program, which is the nation's largest residential, educational, and career and technical training program for low-income youth generally between the ages of 16 and 24. Drug-related incidents and assaults accounted for 48 percent of all reported incidents (see fig.). Students generally felt safe at Job Corps centers, yet fewer felt safe in some situations, based on GAO's analysis of ETA's September 2016 and March 2017 Job Corps student satisfaction surveys. At least 70 percent of students reported that they felt safe on half of the 12 safety-related questions in the 49 question survey about their experiences in the Job Corps program; but fewer students reported feeling safe when asked if they were made to feel unimportant or if they heard students threaten each other. ETA plans to administer a new survey nationally by January 2019 that focuses solely on safety and security issues. ETA has initiated several actions to improve safety and security at Job Corps centers, but insufficient guidance for its monitoring staff and absence of a comprehensive plan for safety and security may put the success of these actions at risk. Among its actions, ETA adopted a new risk-based monitoring strategy to identify emerging problems at the centers. Officials GAO spoke with in five of ETA's regional offices said that the new strategy has improved monitoring, but that more guidance on how to interpret and apply safety and security policies is needed to promote consistency across centers. Also, ETA lacks a comprehensive plan linking its new efforts to an overall safety and security framework. ETA officials told GAO that limited staff capacity and lack of expertise have hindered their efforts in developing such a plan. Without a comprehensive plan, ETA runs the risk that its new efforts will not be successful.