Youth Opportunity Grants: Lessons Can Be Learned from Program, but Labor Needs to Make Data Available
Published: 2005-12-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Youth Opportunity Grant program (YO) represented an innovative approach to improving education and employment opportunities for at-risk youth by targeting resources in high poverty areas and incorporating strategies that experts have identified as effective for serving this population. The Department of Labor (the Department) awarded 36 grants in 2000, and the program continued for 5 years. The Department had used a similar approach on a smaller scale in previous programs, but little information is available on the impact of these other programs. In order to understand what can be learned from the Youth Opportunity Grant program, GAO examined the grantees' implementation of the program, challenges they faced, and what is known about the program's outcomes and impact. To view selected results from GAO's Web-based survey of the Program Directors, go to GAO-06-56SP (http://www.gao.gov/cgi-bin/getrpt?GAO-06-56SP).
section_title: What GAO Found
paragraphs: Grantees used a variety of approaches to build the infrastructure of the YO program, provide services to at-risk youth, and conduct outreach efforts. While grantees set up centers and trained core staff to deliver services, they differed somewhat in their approaches, depending on circumstances within their communities. In addition, grantees employed a combination of strategies to provide youth services, including collaborating with other providers and inventing unique programming. To recruit this hard-to-serve target population, grantees used a range of techniques, from partnering with juvenile justice agencies, to combing malls for eligible youth. Fast program start up, conditions in their communities, and the characteristics and needs of the youth challenged the grantees;however they used features of the program design to address some of these difficulties. Many grantees struggled to set up the program, especially within the Department's time frame. In addition, grantees felt encumbered by the drugs, violence, and a lack of jobs in their communities as well as the obstacles facing their clients, such as low academic achievement and lack of family support. Grantees used the discretion and other components built into the program design to address many of these challenges. For example, in response to safety concerns, an urban grantee elected to provide transportation for youth attending evening events. However, grantees and others said more planning time would have been beneficial. Grantees and others reported that the youth and their communities made progress toward the YO program goals, but the program's impact is still under study. Grantees reported that they had enrolled about 91,000 youth nationwide, many of whom completed high school, entered college, or found employment after enrolling in the program. In addition, grantees and others said that the grants had benefited their communities. However, without an impact analysis, it is not known whether these events would have occurred in the absence of the program. The Department contracted for a $24 million evaluation of the program that included plans for an impact analysis; however, agency officials are unsure if the analysis will be completed.

section_title: Letter
paragraphs: More than 5 million youth between the ages of 16 and 24 were jobless and  More than 5 million youth between the ages of 16 and 24 were jobless and  out of school in 2001, according to a recent study, and young people living  out of school in 2001, according to a recent study, This was a 5-year, $1 billion program aimed at increasing the  Act of 1998.
Andrew Sum, Ishwar Khatiwada, Nathan Pond, and Mykhaylo Trub’skyy, Left Behind in  the Labor Market: Labor Market Problems of the Nation’s Out of School, Young Adult  Populations. Center for Labor Market Studies (Northeastern University:  November 2002).
Pub. Law. No. 105-220. and once enrolled, could usually remain in the program as long as needed.  The program design included many of the strategies experts have  suggested are effective in serving at-risk youth, including providing a safe  place for young people to go, assigning a key adult to monitor their  progress, and offering a wide array of services. The program also provided  a system for grantees to report performance information and included  plans for an extensive evaluation study. The program began in 2000 and  was funded for 5 years.
In order to understand what can be learned from the Youth Opportunity  Grant program, you asked us to examine the following: (1) How did  grantees implement the Youth Opportunity Grant program? (2) What  challenges did they face, and what can be learned from their experiences  in addressing them? (3) What is known about the outcomes and impact of  this program, both on the participants and on their communities?
To answer these questions, we surveyed the 36 Youth Opportunity Grant  program directors, visited 7 grantees, and analyzed program data provided  by the Department of Labor. All of the program directors completed the  survey, which we conducted between March and May 2005. We conducted  five in-depth site visits to grantees, during which we interviewed program  administrators, staff, youth, and other members of the community. We also  toured youth opportunity centers, in-school facilities, and observed  program activities. We selected these five sites according to the  proportionate representation of urban, rural, and Native American  grantees as well as for their geographic distribution. We briefly visited two  additional grantees while pretesting the survey, and at these sites, we  toured grantees’ centers and spoke with program administrators, staff, and  youth. In addition, we reviewed agency documents and interviewed  agency officials and relevant experts. To address the third question  regarding outcomes and impact, we also analyzed electronic data the  Department of Labor collected from the grantees in a management  information system developed specifically for the program.
We conducted our work from September 2004 through November 2005 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The Workforce Investment Act of 1998 authorized Youth Opportunity  Grants (YO), a $1 billion, 5-year program aimed at increasing the  educational attainment and long-term employment of youth residing in  high poverty areas. The program was designed to assist at-risk, hard-to- serve youth and the communities in which they live by concentrating  resources into a targeted geographic area. YO expanded on earlier  demonstration programs that were based on a similar design, the most  recent of which were known as Kulick grants. The Department of Labor  (the Department) announced the 36 YO grantees—24 urban, 6 rural, and 6  Native American—in February 2000 and set a goal for the grantees to have  their programs operational by September of that year. Although the  Department originally planned to continue to add grantees, funding for the  program was eliminated in the budget for fiscal year 2004. Figure 1 shows  the 36 grantees by location.
The Department of Labor’s stated goals for the YO program were to  increase educational attainment and promote long-term employment of  youth in the target areas, as well as to improve the youth service delivery  systems in these communities. Specifically, grantees were expected to  effect increases in the rates of high school completion, college enrollment,  and employment, both for youth participating in the program and, by  extension, the overall target area. Grantees were also expected to facilitate  the delivery of services by partnering with other institutions in the  community, especially local schools and colleges, the juvenile justice  system, and private employers. The program was designed to allow  grantees to enhance the local infrastructure of youth services by filling in  critical service gaps, coordinating existing services, and, in the case of  some rural and tribal areas, developing an infrastructure where services  were limited.
Many of the YO program components were based on practices that experts  have identified as effective for working with at-risk youth, and the design  of the program supported flexibility and invention at the local level. Based  on a youth development framework that emphasizes a comprehensive  approach to meeting a young person’s needs, the program required YO  grantees to offer a full range of education, employment, and leadership  development services as well as provide other supports to the youth.  Advocates and experts have also stressed that youth need a place to gather  where they feel comfortable and a sense of belonging. In this vein, YO  grantees were required to have at least one youth center in the target  community that would provide a focal point for services and activities as  well as a safe place for the youth to go. In addition, grantees were  expected to maintain a core staff of trained youth workers and to provide  follow up services to participants for at least 2 years after they completed  participation in program activities, which is in keeping with the identified  effective practice of providing youth access to a caring, trusted adult for  an extended period of time. Additionally, experts have suggested youth  benefit from the opportunity to continue a relationship with a program for  as long as they need. Similarly, the program had flexible enrollment rules  that allowed participants to remain enrolled even through periods of  inactivity. However, the program design allowed grantees latitude in  deciding many of the particulars of the program. For example, grantees  could choose how and by whom services would be delivered, the number  and location of the YO centers, and the institutions with which they  partnered. Figure 2 displays the key components of Youth Opportunity  Grant program.
Like some prior demonstration programs that the Department of Labor has  administered, eligibility for YO was not based on income, but rather on  geographic residence. Youth between the ages of 14 and 21 living in the  target area were eligible for the program, regardless of income. In most  cases, the target areas were federally designated empowerment zones or  enterprise communities, which are, by definition, areas with high rates of  poverty.
The Department of Labor provided extensive, ongoing technical assistance  to grantees. The Department assigned grantees coaches with considerable  experience working in youth programs who helped grantees with a range  of issues, such as developing services and building partnerships. The  Department also collaborated with a private foundation to establish a  training institute that provided courses in youth development to more than  2,000 youth workers. In addition, the Department sponsored opportunities  for grantees to share best practices and strategies in regular directors’  meetings and conference calls and through peer-to-peer training sessions.
The Department laid the foundation for collecting performance data on  the program by setting up a management information system (MIS). The  Department collected performance data on the YO program in a  centralized, electronic system. Grantees submitted information to fulfill  requirements of the Workforce Investment Act, which mandated that  grantees report on seven performance measures related to employment,  retention, earnings, attainment of skills, and attainment of credentials. In  addition to these, grantees reported on interim measures developed by the  Department to gauge the progress of participants as they moved through  the program. These measures were designed to document youth  participation in activities and other intermediate milestones.
The Department had funded several prior programs premised on a  geographic and community concept similar to that of the YO, although on  a much smaller scale, but little information is available on the impact of  these demonstration programs. Although evaluations were conducted of  these earlier programs, the impact studies were either incomplete or not  released. The first of these demonstrations, Youth Opportunities  Unlimited, was evaluated and the results published, but the study did not  include a systematic analysis of impact based on comparison groups. The  funding for the second demonstration, Youth Fair Chance, was cut 2 years  into the program, and although the evaluation study was published, it was  based on only 2 of the planned 5 years of the program. As a result, the  authors of this study advised that the findings should be interpreted  cautiously. The most recent of the demonstration programs was known as  the Kulick grants. The Department has prepared an evaluation of the  Kulick grants, but has not yet released the results publicly.
For the Youth Opportunity Grants, the Department set up an extensive  effort in 2000 to collect and analyze information to assess the program’s  impact. The Department contracted for a $24 million evaluation study that  included plans to estimate the impact of the program by comparing key  characteristics in the YO communities and comparable areas that did not  receive YO grants. It was to begin in 2000, and the final report was  originally scheduled to be completed in July 2005. The study was designed  to differentiate between those observed changes in participants and the  communities that resulted from the program and those that would have  occurred even without the program’s intervention.
section_title: Grantees Carried Out the Program Using a Variety of Approaches
paragraphs: Grantees adapted key elements of the program’s design to their particular  circumstances and used a variety of approaches in providing services and  conducting outreach. Grantees set up centers that varied widely in terms  of number and other characteristics where youth could receive services  and participate in activities. The centers housed a core staff who provided  case management and helped plan individualized services for youth. Staff  numbers and duties differed between grantees. Most grantees used the  management information system (MIS) provided by the Department of  Labor to submit program data to the Department, but some used their own  systems. Most grantees used a combination of approaches to provide  youth services, including collaborating with other organizations and, when  they deemed it necessary, developing services of their own. To reach hard- to-serve target population, grantees used a variety of recruiting techniques,  ranging from the conventional to the innovative.
subsections: section_title: Grantees Adapted Program’s Design to Their Particular Circumstances
paragraphs: The centers set up under the terms of the grant varied widely in number  and character. Because the target communities varied in size, some  grantees had 1 center while other, more geographically dispersed grantees  had as many as 40. The centers were intended to serve as the hub of the  local programs, and the types of activities offered in them differed  considerably. One center we visited in rural Louisiana had a variety of  recreational facilities, such as a basketball court and recreation room, in  addition to a classroom, computer lab, and staff offices. Youth  participating in this program told us nothing like these facilities had  existed for them before the YO program. On the other hand, a center we  visited in Houston resembled a traditional career center, with computer  kiosks set up near staff offices where the youth could undertake job  searches and skills assessments. Houston officials said they had formerly  offered some recreational activities on site, but found them to be  duplicative of other services available in the neighborhood. Some grantees  provided still other facilities at their centers. For example, Baltimore had  an on-site health clinic and music recording studio. Some grantees also  had centers located within schools. The District of Columbia had staff  sited in a local high school, where they offered training and support for  youth. This in-school center was located in a large classroom, with a staff  office off to one side and furnished with a computer lab with equipment  purchased by the grantee. Figure 3 depicts photos of two youth  opportunity grant centers we visited.
The centers housed staff who varied substantially in number and makeup.  Houston divided staff duties between as many as 115 people working in 4  centers. In this program, “personal service representatives” provided case  management and identified youths’ goals and training needs, and  “employment counselors” assisted youth with job searches and conducted  follow-up services to youth who had been placed in employment. Other  grantees, however, had sites with only one person performing all of the  major staff duties. At the California Indian Manpower Consortium (CIMC)  Manchester Point Arena site, one youth worker handled recruitment, case  management, and job placement. Because of the remoteness of the site,  this youth worker also frequently drove a van, weekly transporting youth  to services as far as 4 hours away.
Most grantees used the management information system (MIS) provided  by the Department of Labor to record program data, but some used their  own systems and then submitted the data to the Department. Those who  did not use the Department MIS used a variety of systems. For example,  Milwaukee developed its own system with the aim of using it in  conjunction with other youth programs offered by its local Workforce  Investment Board. Similarly, Hartford developed customized software  that enabled it to share information with other provider agencies and the  schools in the community.
section_title: Grantees Used Collaboration and Invention to Varying Degrees in Delivering Services
paragraphs: Grantees used a combination of approaches to establish a network of  educational, occupational, and other services for youth, but varied the  extent to which they relied on other providers. Grantees availed  themselves of existing services, either through formal arrangements or by  referring participants to other organizations in the community. Kansas City  arranged for youth to attend classes at other local agencies to help prepare  for the GED exam. Similarly, Baltimore referred young women who  needed interview clothing to a local charity organization. Most grantees  also collaborated with other institutions to provide some services. San  Antonio partnered with a local community college to establish academies  in which youth could spend half a day on their home campuses and half a  day at the community college, allowing them to earn a certificate or  associate degree in areas such as aviation or biotechnology. Houston  purchased credit retrieval software for local high schools that students  could use to complete their academic requirements for graduation. In  addition, grantees originated programs to fill perceived gaps in services.  Memphis established its own charter school with an emphasis on the arts  in order to provide alternative education opportunities for the youth, one  of the emphasized youth activities in the program. To provide leadership  development opportunities, one of the CIMC sites sponsored a cemetery  care project. In this project, participants worked with the tribal elders to  make a map of the local cemetery with the names of the deceased because  cemeteries on this reservation did not have headstones. Table 1 shows  youth activities and examples of how grantees implemented the activities.
Grantees also used an array of strategies to help youth find long-term  educational or occupational opportunities and to follow up with  participants after they exited the program. Some grantees had staff  specifically devoted to finding appropriate employment opportunities and  connecting youth with these opportunities. For example, Milwaukee  created the position of job developer, who identified potential employers  and spoke to them directly about the youth in the program. Grantees also  used a number of methods for helping youth enroll in post-secondary  institutions, such as conducting college tours, assisting with financial aid  forms, and placing support staff on college campuses. The statutory  provision authorizing the YO program required grantees to provide follow- up services for 2 years after the youth completed participation in program  activities. Grantees recorded on a quarterly basis the employment and  education status of the youth who had completed participation in the  program, for example, if they were in school or working at the time of  contact. Some grantees contacted youth more frequently. Staff in  Baltimore told us they varied the frequency depending on the degree to  which they regarded the young person to be at some risk. One might be  contacted monthly, while another would be checked on nearly every day.
section_title: Grantees Actively Recruited Youth, Including the Hard to Serve, and Used Incentives to Draw and Maintain Involvement
paragraphs: Grantees recruited youth, including traditionally difficult to serve  populations, by creating connections with other youth-serving agencies.  Milwaukee operated a juvenile justice project to help youth coming out of  the corrections system to reintegrate into the community. Personnel at a  local correctional facility worked with program staff to identify those due  for release who were expecting to live within the geographic target area of  the program. Program staff traveled to the facility to conduct an  orientation session and then worked with the youth, corrections officials,  and parents to develop a reintegration plan. The services offered to them  were similar to those offered to all of the YO participants.
Grantees also used a variety of innovative methods to recruit participants.  Since many of the youth were disconnected from both school and work, as  one grantee told us, they could be not only hard to serve, but hard to find.  Some grantees went beyond conventional outreach activities of mailers  and radio advertisements and conducted community walking campaigns  using staff to saturate shopping malls and other areas where youth  congregate. Others used youth to lead recruitment. In some cases, a  grantee used employment as an inducement to link youth to other  activities the program offered. For example, Louisiana’s work program  allowed participants to work more hours in subsidized employment the  further they advanced toward their educational goals. We were told many  of the youth lived in unstable economic conditions, and these needs had to  be addressed before the youth would focus on other areas, such as  education. In other cases, grantees used cash or non-monetary rewards to  encourage participation. In Houston, youth could attend special events,  such as a basketball game, if they participated in the program a minimum  number of hours.
section_title: Grantees Used Program Design to Address Major Challenges, but Found Start-up Time to be Short
paragraphs: Grantees were able to address a variety of challenges in setting up the  program and delivering services to youth using local discretion, flexible  enrollment rules, and other aspects of the program’s design, but they and  others said a longer start-up period would have been beneficial. Most  grantees found it difficult to establish centers and retain participants in  their programs. They felt that these challenges were compounded by  expectations for a quick start-up, and subsequently they and Department  officials felt more planning time would have been beneficial. Grantees also  had difficulty establishing an information-reporting system, but once in  place, found it was helpful for program management purposes. Conditions  in the communities such as violence and lack of jobs presented a  challenge to most grantees, but they took advantage of the local discretion  built into the program to develop strategies to address them. Grantees also  cited as an obstacle the vast service needs of many of the youth the  program was intended to serve, but case management, individualized  services, and flexible enrollment rules were useful in dealing with them.
subsections: section_title: Grantees Felt Challenged to Set Up Program in Time Allotted, and More Planning Time Would Have Been Beneficial
paragraphs: The majority of grantees reported that finding or renovating centers was a  challenge. Many grantees did not have suitable sites in their communities  for use as centers, and some put considerable effort into renovating  existing structures. In the extreme, the grantee in Alaska renovated 40  centers in remote communities not accessible by road and shipped  equipment and material by air. In a few cases, grantees did not have a  permanent building until after the first year of the program. In Boston, the  program was housed in temporary facilities for a period of time because a  permanent center could not be completed for the program’s opening.
Another challenge identified by most grantees was retaining participants,  which grantees linked in part to expectations from the Department to start  to enroll youth quickly. The Department set a goal to recruit 3,000  participants nationwide within the first few months of the program, or a  little less than 100 youth per grantee. Some grantees said they felt rushed  to meet recruitment goals and told us they recruited many youth who were  not committed to the program. In addition, several directors mentioned  that they felt they were asked to serve participants before their  programming was fully developed, a situation that as one expert  commented, was akin to asking grantees to “fly the plane before they were  finished building it.” We were told that, in these situations, some youth  became disenchanted with the program and left because of the limited  offerings the center presented early on.
Grantees, agency officials, and experts said a longer planning period  would have been beneficial. Grantees had an 7 month window between  the announcement of the grant recipients and the Department’s target date  for having the programs operational. For example, two grantees with  whom we spoke said about three additional months would have been  necessary for them to meet the goals set out by the Department. Agency  officials also told us that grantees were pushed to start-up too quickly and  could have benefited from more time to plan. Agency officials said it  would have been appropriate to give grantees more time to meet the  Department’s initial goals.
section_title: Grantees Had Difficulties Establishing an Information Reporting System, but Found It Useful Once in Place
paragraphs: Most grantees reported that establishing an information reporting system  was challenging, and grantees told us it would have been useful for the  Department to have had a workable system in place at the outset of the  program. The management information system (MIS) initially provided by  the Department was fraught with problems. Department officials told us it  took about 18 months to iron out the kinks. In addition, the Office of  Inspector General identified inconsistencies in the way grantees were  recording data early on. Although the Department eventually developed  an improved system, grantees said it would have been better had the MIS  been in place from the beginning. Not having a system in place created  additional burden for the grantees. For example, Louisiana developed its  own system to collect information until an updated system was available  and then manually transferred 11 months of data into the new MIS.
Once in place, grantees additionally used the information system for their  own program management purposes such as to improve performance and  reinforce accountability. The Hartford grantee was able to integrate its  system with the school district, allowing case managers to track  attendance and grades to allow better monitoring of participant  achievements. The Hartford system also enabled program management to  see which case managers were most successful at engaging youth in the  activities required for the completion of their goals. Regarding  accountability, Portland used its information system to monitor the  performance of its contractors on a weekly basis.
section_title: Grantees Used Their Discretion to Address Community-Based Challenges
paragraphs: Some of the major challenges also identified by grantees were problems  external to the program, yet affected their ability to deliver services.  Twenty-eight of the grantees reported on the survey that a lack of jobs in  the community was a challenge. In some areas, jobs were on the decline  because of shifts in the local economy or relocation of major employers.  For example, the grantee in San Francisco related that due to the dot-com  bust, youth in the program were competing with a skilled workforce  willing to fill entry level positions that would otherwise be available to  youth. In other areas, especially rural ones, there were few employers, let  alone large ones. In addition, 29 of 36 grantees reported on the survey that  risk factors such as violence, drugs, and gangs were challenges in  implementing the program. In some cases, these factors made it difficult  for participants to receive services. In one urban community, we were told  there were safety concerns with youth participating in evening activities  sponsored by the YO because they would have to return home after dark.
Local discretion built into the program design helped grantees respond to  external challenges. Within the structure of the program, grantees were  allowed some amount of latitude to develop responses to circumstances in  their communities, such as a scarcity of jobs. For example, the California  Indian Manpower Consortium created opportunities for youth to have  work experiences, such as subsidizing summer jobs with local business  including a campground and senior center or working with the tribe to  place youth in clerical positions. Similarly, grantees were able to develop  services to address specific risk factors in their communities. Imperial  County offered a curriculum series to help address interpersonal violence  among adolescents, an issue they had identified as particularly  problematic in their area. The series was designed to reduce impulsive and  aggressive behavior through empathy training, interpersonal problem  solving, behavior skill training, and anger management. In Milwaukee, the  grantee addressed safety concerns by renting vans to transport the youth  to nighttime events.
section_title: Service Needs of Participants Presented Challenges, but Tailored Services, Individualized Planning, and Flexible Enrollment Rules Were Useful in Addressing Them
paragraphs: Grantees identified as a major challenge the obstacles faced by their  clients, such as homelessness, lack of family support, mental health  problems, and low levels of academic attainment. Staff in Baltimore told  us homelessness was a frequent issue faced by youth in their area and  finding a place to sleep can preoccupy the youth and disrupt the learning  process. YO staff and others said that participants may lack support in  other areas of their lives, such as from their families. For example, one  grantee told us of a participant who was awarded a full scholarship to  college, but the parents would not sign the paperwork to receive the  money. Some youth also faced mental health issues, which prevented them  from moving forward in their lives. All of these factors, we were told,  could mean a slow start for youth who enrolled in the program. The  director of an alternative high school in Milwaukee told us that because of  the disorder in their lives, many of these youth may take a year just to  become comfortable in the program before they can even begin making  any forward progress.
Grantees found aspects of the program design such as case management,  individualized services, and flexible enrollment rules useful in addressing  the service challenges of participants. For example, grantees used  assessments to help determine the academic needs of clients and provide  each client with the appropriate individualized services. Philadelphia  designed a program in which each participant had a personalized  remediation plan based on the results of an evaluation test and interview  with an education coach. This plan was intended to build on clients’  strengths and accounted for their particular learning styles. After the  initial assessment, participants were reevaluated at regular intervals to  monitor their academic progress and goal attainment. The curriculum was  designed so that youth could increase at least one grade level after 90  hours of instruction. In addition, both youth and community members told  us that YO staff were key in helping youth stay motivated and guiding  them through difficult situations. Grantees also found the enrollment rules  useful in working toward program goals. The program’s enrollment policy  allowed most participants to continue to receive services even if they  experienced periods of inactivity. Staff in Houston told us that the policy  was a benefit to the youth because, unlike other employment programs,  YO did not end their relationship with youth who had not been  participating for a while.
section_title: Grantees and Others Reported That Participants and Their Communities Made Progress, but the Program’s Impact Is Still under Study
paragraphs: Grantees and others reported that the participants and their communities  made advancements in education and employment; however, a formal  assessment of the program’s impact is still under study. Data reported by  the grantees showed that a number of youth advanced their education  while they were participating in the program, such as completing high  school. Grantees also reported data showing a portion of the participants  entered unsubsidized employment after enrolling in the program.  Similarly, grantees and others believe their communities made  advancements toward the Department’s stated education and employment  goals of the program. In addition, the majority of grantees reported that  they made improvements in the youth service delivery systems in their  communities. The Department funded an evaluation to assess the impact  of the program, which was designed to shed light on the extent to which  observed changes can be attributed to the program. However, the study  has not yet been completed.
subsections: section_title: Grantees Reported Data Showing Some Participants Advanced Their Education and Found Employment
paragraphs: Data reported by the grantees showed that a number of youth advanced  their education while they were participating in the program. Two of the  Department’s primary goals of program were to increase the rates of high  school completion and college enrollment for the youth. Data that  grantees entered into the management information system (MIS) showed  that of the approximately 91,000 youth who signed up for in the program  nationwide, about 18,700 either completed high school or attained a GED  after enrolling in the program. In addition, about 11,700 youth entered  college, 37 percent of whom were reported to be out of school when they  initially enrolled in the program. Youth with whom we spoke credited the  program with giving them a second chance to increase their education and  to better their employment opportunities. One Louisiana youth’s  comments typified what we heard elsewhere from participants during our  site visits, “YO is the best thing that has happened to me. YO has given me  a job and put a little money in my pocket.” He added, “YO has also been  instrumental in keeping me out of jail. Above all, YO has helped me realize  that education is important.” Figure 4 summarizes enrollment,  participation, education, and employment data for youth in the program,  as of June 2005.
Grantees reported data showing that a portion of the youth entered  unsubsidized employment after completing YO activities. Data from the  MIS show that about 17,300 youth were placed in employment that was  not subsidized by the program, 62 percent of whom were out of school  when they initially enrolled in the program. Grantees and others told us  the job readiness training that some youth received as part of the program  was particularly important in helping them get jobs. Employers we spoke  with told us they knew the youth that were sent to them by YO would be  well trained and ready to work. A local human resources manager for a  national chain of home improvement stores told us that he thought YO  participants were better prepared for employment than other residents of  the area, adding that they show up with better job skills and “soft” skills.
section_title: Grantees and Others Believe Their Communities Moved toward Program Goals
paragraphs: Grantees and others expressed the view that their communities had made  advancements in concert with the Department’s education and  employment goals for the program. Almost all of the grantees reported  that high school completion, college enrollment, and youth employment  rates improved in their communities as a result of the program. Several  grantees also asserted that affecting the youth led to changes in the  community. In addition, community members with whom we spoke said  the program helped their communities make progress toward greater  education and employment. For example, a tribal leader in California told  us that YO motivated the youth in his tribe to stay in school. Similarly, one  expert pointed to some of the rural grantees who had many youth enroll in  college, which she said was especially important because of a lack of jobs  in these areas.
The majority of grantees reported making improvements in the youth  service delivery systems in their communities. Some of the described  changes were to the service infrastructure in these communities. For  example, San Diego pointed out that it had created a multiservice youth  center in a neighborhood where none had existed before. Grantees cited  other improvements related to the mode of service delivery. Tucson  reported that prior to YO, it had funded three stand-alone youth programs,  but now it has a one-stop system with multiple entry points for delivering  youth services. Other grantees noted that their efforts to foster better  communication and collaboration among service providers had benefited  the youth. As the Portland grantee commented, by partnering with their  local employment department and community college system, they were  able to leverage staff to provide intensive job search and college  preparation services to youth. In other cases, local leaders credited the YO  as a catalyst for change in their community. For example, a school  superintendent in California told us that YO staff had helped to forge an  agreement between the school district and five Indian tribes in the area to  develop an education curriculum that would be more sensitive to the  tribes’ cultures.
section_title: Department of Labor’s Study of Program Impact Has Not Been Completed
paragraphs: While the Department of Labor funded an evaluation designed to assess  the impact of the program, it has not yet been completed. This was  planned as a 5-year, $24 million evaluation, which began in 2000. As part of  the evaluation, the Department conducted baseline and follow-up surveys  of the target areas, and gathered extensive descriptions of the 36 programs  and communities. The evaluation also included plans for an impact study  that was designed to compare YO participants and communities with  other, similar youth and communities that did not participate in YO. This  type of comparative analysis would be necessary to determine if the  events reported by grantees and others would have occurred in the  absence of the program. However, the evaluation is not finished.  Moreover, agency officials are unsure if the impact study that was to be  part of the evaluation will be completed. Due to departmental allocations  being lower than expected, the Department spent $1.9 million less than the  full amount of the original contract on the evaluation. Agency officials told  us that the impact study was likely to be scaled back because of the lower  allocation. The evaluation was originally scheduled to be completed in  July 2005. However, agency officials told us they do not expect the study  to be finished until June 2006.
section_title: Conclusions
paragraphs: The Youth Opportunity Grant program was designed to help at-risk youth  and their communities by concentrating resources geographically and by  incorporating components that experts have suggested are effective in  assisting this population. To continue to improve the ability to serve these  youth, researchers and practitioners must be able to learn from the most  promising and innovative approaches in serving these youth, including  those used in the Youth Opportunity Grant program. The Department of  Labor has begun an evaluation of the program that includes many pieces  of a potentially useful study. Although informative, these pieces will not by  themselves answer the question of impact, in other words, whether the  described events would have occurred in the absence of the program. In  order to understand what effect, if any, the program had on these  observed events, it is necessary to have a systematic comparison with  other, similar communities that did not receive grants. The Department  planned, but has not yet completed, such an analysis as part of the  evaluation. The Department has an opportunity to contribute to the  research on programs for serving at-risk youth, if the evaluation study is  completed and the results made public. However, the Department has not  taken full advantage of past opportunities to release information on other  programs based on a similar model, such as the Kulick grants. Given the $1  billion investment in this program—including almost $24 million for an  evaluation effort—and the need for rigorous data on these types of  programs, the study should be completed and the results should be made  available. Unless the Department completes the evaluation of the Youth  Opportunity Grant program and releases the results, researchers and  practitioners will not be able to fully realize the potential to learn from the  program.
section_title: Recommendation for Executive Action
paragraphs: To continue to improve efforts to serve at-risk youth and in order that  researchers can evaluate the quality of information and determine possible  impact of the program, we recommend that the Secretary of Labor take  the actions necessary to complete the impact analysis of the Youth  Opportunity Grant program and release the data and all related research  reports from the program’s evaluation.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Labor for its  review and comment.  In its response, the Department agreed with our  conclusions and recommendation and indicated that it intends to complete  the impact analysis and publish all related reports from the Youth  Opportunity Grant program evaluation.  A copy of the Department’s  response is in appendix II.  The Department also provided us with  technical comments, which we incorporated into the report where  appropriate.
We will send copies of this report to the Secretary of Labor, relevant  congressional committees, and other interested parties. We will also make  copies available to others upon request. In addition, the report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
Please contact me at (415) 904-2272 if you or your staff have any questions  about this report. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To accomplish our research objectives, we surveyed Youth Opportunity  Grant Program directors on the implementation of the Youth Opportunity  Grants. To augment information from our survey, we conducted five site  visits of programs. We chose sites for our visits to represent  proportionately the three grantee types (urban, rural, and Native  American) as well as their geographic distribution. The grantees we visited  were the District of Columbia, Houston, Milwaukee, rural Louisiana, and  the California Indian Manpower Consortium. During our visits we met  with program administrators, staff, and participants to learn their  perspectives on program implementation, challenges, outcomes, impact,  and lessons learned. Also, we interviewed local workforce board  members, school officials, community based service providers, private  employers, and local government officials to discuss their perspectives on  the impact and other aspects of the program. In addition, we toured Youth  Opportunity Centers, in-school facilities, and observed program activities.  We briefly visited an additional two grantees— Baltimore and  Philadelphia—in the course of pretesting the survey, and while at their  sites, toured centers and spoke with program administrators, staff, and  participants. We gathered additional information by reviewing agency  documents including site assessments and grant applications. We also  interviewed agency officials and other relevant experts, including  researchers, representatives of advocacy organizations, YO coaches, and  former Department of Labor officials. We performed our work from  September 2004 to November 2005 in accordance with generally accepted  government auditing standards.
We also investigated several possible sources of external data to  quantitatively measure outcomes and impact, but determined none of  them were feasible for our purposes. The data sets we reviewed were:  Current Population Survey, Common Core of Data, National Longitudinal  Survey of Youth, High School and Beyond, and American Community  Survey.
subsections: section_title: Survey of Youth Opportunity Grant Program Directors
paragraphs: To learn about the implementation of the Youth Opportunity Grants, we  conducted a Web-based survey of all Youth Opportunity Grant Program  Directors. We asked directors about the usefulness of program  components, challenges they faced implementing the program and  strategies used to deal with them, and their opinion on the program’s  impact on participants and communities. Additionally, we asked directors  about the size and structure of their programs, the manner in which they  delivered services, and the types of organizations with which their  programs partnered. The survey also included a series of questions about  how programs maintained information to help us determine the reliability  of data in the Department of Labor’s management information system. We  pretested the survey with several program directors and modified the  survey to take their comments into account. All 36 program directors  completed the survey, for a response rate of 100 percent. We administered  the survey between March 17 and May 31, 2005. To view selected results of  the survey, go to GAO-06-56SP.
section_title: Department of Labor Management Information System (MIS)
paragraphs: We used electronic data collected on the program by the Department of  Labor in a management information system (MIS) to describe the number  of youth in the program who achieved the Department’s goals for the  program, such as high school completion, college enrollment, and long- term education and employment placements. We have determined the MIS  data are sufficiently reliable for the purposes of this study through  discussions with officials at the Department of Labor, in-depth interviews  with MIS staff during site visits, and responses to a comprehensive array  of survey items in which each grantee described their procedures for  editing and auditing the data they entered into the MIS. We analyzed the  MIS data using guidance provided to us by the Department.
section_title: Appendix II: Comments from the Department of Labor
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: David Lehrer, Assistant Director, and Anne Welch, Analyst-in-Charge,  managed this assignment and made significant contributions to all aspects  of the work. Dan Concepcion and Eleanor Johnson also made significant  contributions to this report. In addition, Kevin Jackson assisted in all  aspects of the survey of Program Directors and in reviewing external data  sources. Jerry Sandau and Cathy Hurley assisted in the analysis of the MIS  data and assessment of the MIS data reliability. Jessica Botsford provided  legal support, and Susan Bernstein provided writing assistance.
section_title: Related GAO Products
paragraphs: Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington, D.C.: February 23, 2004.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing Is Needed. GAO-03-725. Washington, D.C.:   June 18, 2003  Workforce Investment Act: Youth Provisions Promote New Service  Strategies, but Additional Guidance Would Enhance Program  Development. GAO-02-413. Washington, D.C.: April 5, 2002.