Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services
Published: 2004-02-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Workforce Investment Act (WIA) has been in effect for several years and is currently undergoing reauthorization. In order to provide the Congress with information on the implementation and effectiveness of the WIA youth program, GAO was asked to explore how services have been delivered at the local level, whether the Department of Labor's guidance has addressed challenges faced by local areas, and how effective the program has been.
section_title: What GAO Found
paragraphs: Local areas primarily used the WIA program for dropout prevention and other efforts to improve academic achievement for in-school youth. Nationally, about 70 percent of youth served were in school, but percentages ranged from 38 to 86 percent by state. Officials in the five states GAO visited said that they focused on in-school youth because serving out-of-school youth was much more difficult and expensive, and less effective. Local areas emphasized learning-related summer employment for in-school youth and occupational skills training and supportive services for out-of-school youth. Over half of local boards nationwide used providers that had subcontracting arrangements with others to deliver youth services. The majority of youth were served primarily from educational institutions and community organizations. Despite Labor's guidance, local areas continue to face implementation challenges in identifying and retaining out-of-school youth, providing youth with mentoring and follow-up services, and using interim measures for ongoing program assessment. While Labor supports information exchange forums, a promising practices Web site, and technical assistance, some local areas may have difficulties gaining access to and using these resources. Little is known about the effectiveness of the WIA youth program because Labor has not yet conducted an impact evaluation. In addition, while the youth program exceeded most of its performance goals, these data were questionable because of problems with state information systems and inadequate oversight of data quality. While states will be required to verify data, concerns remain about their ability to fully implement the requirement and Labor's ability to monitor implementation consistently.

section_title: Letter
paragraphs: More than 5 million young people—15 percent of the nation’s youth  population between ages 16 and 24—are out of school and out of work,  according to a recent study. Further, many teenagers and young adults  who are in school are at risk of dropping out. According to some experts,  this indicates that a considerable portion of the country’s emerging  workforce may face significant difficulty making the transition to  productive adulthood. At the same time, the Department of Labor projects  that some labor demands will go unmet because there will be too few  workers in the labor market with the necessary skills. Enacted in 1998, the  Workforce Investment Act (WIA) aims to address these issues by assisting  the nation’s emerging workforce in realizing its full potential. As the  administering agency, Labor has budgeted about $1 billion annually on  WIA Title I-B youth employment and training programs to serve an  estimated 721,000 of the nation’s most at-risk young people.
WIA services are based on promising practices in the fields of youth  development and employment. Research suggests, for example, that youth  can achieve positive outcomes when they interact with caring adults,  engage in hands-on education and training activities, and receive support  for personal growth. The WIA program has sought to make these sorts of  experiences available to both in-school and out-of-school youth  participants. Under WIA, local areas can tailor their approach to the types  of youth served, the services provided, and how they are delivered. To  ensure that youth programs are tailored to local areas, WIA requires the  participation of a wide variety of people—youth policy experts,  representatives from youth-serving agencies, parents, and others with a  vested interest in the local youth programs. These participants serve on  local workforce investment boards created by WIA to establish workforce  development policies and oversee implementation at the local level, or on  youth councils, subgroups of the local board that plan and coordinate the  local youth program.
We previously reported on the implementation challenges local areas  faced during the first few years of implementation. Now that the program  has been ongoing for several years and WIA is undergoing reauthorization,  you asked us to review (1) what approaches local areas have taken to  serve at-risk youth, (2) whether Labor’s youth program guidance has  addressed ongoing implementation challenges, and (3) what is known  about the effectiveness of the WIA youth program.
To obtain information on what approaches local areas have taken to  deliver youth services, we administered a survey to the directors of all   604 local workforce investment boards across the nation, including those  in the 50 states, the District of Columbia, Puerto Rico, and U.S. territories.  We received responses from 496 local workforce investment boards   (82 percent) and relied on self-reported data. To further understand local  area approaches to service delivery, we visited nine local workforce  investment areas in five states: California, Louisiana, New Hampshire,  Ohio, and Virginia. We selected these states to obtain a mix based on their  differences in geographic location, number of workforce investment areas,  amount of youth funding, and presence of a state youth council. In  addition, our selection was informed by recommendations from Labor,  youth policy experts, and state workforce officials. We interviewed  officials representing state and local boards, youth councils, one-stop  centers, youth-service providers, business representatives, and state and  local education agencies. We reviewed Labor’s program evaluation agenda  and published guidance letters from program year 2000 to the present. We  also assessed the reliability of the performance data submitted by states in  their annual reports and compiled in Labor’s WIA database known as the  Workforce Investment Act Standardized Record Data (WIASRD) by  performing checks for internal consistency, reviewing Office of Inspector  General reports and other relevant documents, and speaking with the  contractor for WIASRD. We determined that the data were not sufficiently  reliable to use for the purposes of this report. We also interviewed officials  from Labor, including each of its six regional offices, as well as from the  Department of Education and national associations. Our work was  conducted between January and December 2003 in accordance with  generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Enacted in 1998, WIA replaced the fragmented and overlapping programs  under the Job Training Partnership Act (JTPA) with a system that sought  to connect employment, education, and training services to better match  job seekers to labor market needs. The youth program falls under Title I   of WIA along with programs for adults and dislocated workers; Title II  deals with adult and family literacy; Title III pertains to employment  services under the Wagner-Peyser Act; and Title IV addresses vocational  rehabilitation. To better prepare low-income youth who face barriers to  employment and education, WIA requires youth programs to focus on  long-term, comprehensive services delivered year-round through a  coordinated network of service providers. In addition to meeting the low- income requirement, to be eligible for WIA services youth must be  between the ages of 14 and 21 and faced with one or more of six barriers  to employment. WIA serves both in-school and out-of-school youth. A  significant portion of out-of-school youth are high school dropouts and  unemployed youth who are struggling to succeed in the public education  system and lack financial, family, or social support. Thus, at least   30 percent of local WIA youth funds must be spent on out-of-school youth.
subsections: section_title: WIA Youth Funding
paragraphs: Since WIA’s full implementation in 2000, funding for the youth program  has ranged between approximately $1 billion and $1.4 billion a year. Labor  follows a formula to allocate WIA funds to states, which in turn distribute  money to their local workforce investment areas. At the state level, the  governor can reserve up to 15 percent of the annual WIA allotment for  such statewide activities as developing the WIA strategic plan for the state.  States allocate the remaining 85 percent of funds to local boards that use  the funds to develop the local strategic plan, establish a youth council, and  award competitive contracts to youth service providers that have been  recommended by the youth council. In addition to recommending eligible  providers, the youth council coordinates youth activities in the local area,  oversees providers, and carries out other duties authorized by the local  board such as forging linkages with educational agencies. Youth councils  may also leverage additional public and private funds to supplement their  WIA funding in order to provide comprehensive youth services.
section_title: Youth Services
paragraphs: Once they are determined to be WIA eligible, youth receive an objective  assessment of their academic level, skills, and service needs. Local youth  programs then use the assessment to create each participant’s individual  service strategy, which lays out employment goals, educational objectives,  and necessary services. Every local youth program must offer the  following 10 services, known as program elements, to eligible youth,  though participants may receive different combinations of these elements  depending on their service strategy. Labor groups the 10 required program  elements around four major themes:  1.  tutoring, study skills training, and instruction leading to completion   of secondary school, including dropout prevention,  2.  alternative school services,  Preparing for and succeeding in employment  3.  summer employment linked to academic and occupational learning,  4.  paid and unpaid work experiences, including internships and job  5.  occupational skills training,  Developing the potential of young people as citizens and leaders  6. leadership development, which may include community service and  peer-centered activities encouraging responsibility,  7.  supportive services (such as child care and housing assistance),  8.  adult mentoring for at least 12 months that may occur both during   and after program participation,  9.  follow-up upon program completion for at least 12 months, and  10.  comprehensive guidance and counseling.
Eligible youth may obtain these services directly from approved youth  service providers or through WIA’s one-stop system, which serves as a  gateway to a variety of employment and training services. In addition to  helping WIA youth gain access to the 10 program elements, the one-stop  system also provides all youth with basic services, whether or not they are  eligible for WIA. Any young person may walk into a one-stop center and  make use of the center’s career resources and obtain information on and  referrals to other providers. While one-stop centers are designed primarily  to serve those 18 and older, some states have established one-stops that  serve only youth, as we reported previously.
section_title: Youth Program Guidance
paragraphs: Labor provides guidance to help states and local areas implement WIA.  Labor’s guidance includes issuing annual Training and Employment  Guidance Letters on the youth program in general as well as on specific  topics, sponsoring WIA Learning Exchanges in every region, and  maintaining a Web site for promising and effective practices. Labor  responds to requests for clarification or additional information by phone  or e-mail or at conferences. Labor’s guidance often includes promising and  effective practices to reinforce state and local flexibility under WIA to  tailor programs to best meet youths’ needs.
section_title: Performance Measures
paragraphs: The law requires that states and local areas collect performance  information on seven youth measures, which are separated for younger  and older youth (see table 1). All seven youth measures apply to both  statewide and local performance.
WIA performance information is collected from service providers and  flows upward to the local boards, then to state boards or agencies, and  finally to Labor. Local performance data are aggregated and entered into a  state’s own automated data system that tracks the activities of individual  WIA participants. States use two mechanisms to report performance to  Labor: WIASRD, a database of individual records containing activity and  outcome information for each registered participant that has exited WIA,  and state annual reports. Labor uses the state annual reports to track  states’ progress in meeting negotiated performance goals. Labor then  awards monetary incentives to states that meet or exceed their  performance goals and sanctions states that fail to meet at least 80 percent  of each goal in two consecutive years.
section_title: WIA Program Used Primarily for Dropout Prevention and Emphasized Summer Services and Skills Training Coordinated among Multiple Service Providers
paragraphs: Most local areas used WIA youth program services for a range of  approaches to prevent academic failure and school dropouts, emphasizing  learning-related summer services that were coordinated through case  managers and multiple service providers. Nationally, about 70 percent of  youth served were in school, according to local boards responding to our  survey. For in-school youth, local boards provided summer employment  services linked to classroom learning more often than for out-of-school  youth, while services for this latter population more often included  occupational skills training and supportive services. Local boards reported  that most youth received services primarily from community organizations  and educational institutions. These providers were most likely to  subcontract or make informal arrangements with other organizations to  deliver the full range of WIA services to youth participants.
subsections: section_title: Local Areas Focused on Dropout Prevention for In- School Youth
paragraphs: Overall, local areas’ approach to serving youth was to supplement schools’  dropout prevention efforts to keep youth connected to an educational  system, according to state and local workforce and education officials.  WIA allows states and local areas to determine the proportion of in-school  and out-of school youth to be served, requiring only that they spend at  least 30 percent of funds for out-of-school youth. Labor reported that with  some exceptions, states chose to focus the majority of resources on in- school youth, and our survey showed that about 81 percent of local areas  served more in-school youth than out-of-school youth. Nationally, about   70 percent of youth served were in school, according to local boards we  surveyed. However, this percentage varied across the nation, as shown in  figure 1. For example, 38 percent of youth served in South Dakota were in  school, compared with 86 percent in Nebraska.
According to officials in four states that we visited, WIA youth programs  primarily targeted in-school youth because recruiting and retaining out-of- school youth for the WIA program was much more difficult and expensive.  For example, officials from a rural area in Ohio had difficulty identifying  and retaining out-of-school youth whose contact information changes  frequently, and Labor reports that the average cost of serving an out-of- school youth under the WIA program is about $4,000 a year, twice as much  as for an in-school youth. Officials from other local areas we visited  considered other factors. A local official in Louisiana said that in-school  youth are interested in the WIA program, unlike out-of-school youth, who  are difficult to engage. In contrast, a rural area in Virginia chose to focus  primarily on out-of-school youth because so few services were available  for this population.
The Administration has proposed amending the WIA youth program to  focus more resources on out-of-school youth. Department of Education  officials said that WIA’s services to in-school youth were not unique, since  schools already offer various services to their students to deter them from  dropping out. While Education officials said that the department’s grants  geared exclusively toward dropout prevention would be insufficient to  address the national dropout problem, they also stated that funding is  available under many Education programs that could be used for dropout  prevention activities. However, local workforce and education officials  we spoke with in three states we visited said that they were either  unaware of or unable to gain access to other available federal resources  that could be used to provide intensive services to at-risk youth. In New  Hampshire, for example, officials told us that WIA provides the only  dropout prevention program, and that they were unaware of other  available education dropout prevention resources.
State and local workforce and education officials we spoke with in the   five states we visited said that WIA funding complemented rather than  duplicated education services and was critical in preventing in-school  youth from dropping out of school. For example, in a rural area in Ohio,  workforce officials stated that the WIA program was the only dropout  prevention program and that WIA provided students with their only  chance at academic and career success. In addition, they said that WIA’s  services were more intensive and comprehensive, and were delivered in a  one-on-one setting where each student received individualized attention.  In one urban school in Virginia, there was 1 WIA counselor for 50 eligible  in-school youth, compared to 1 school counselor for as many as   300 students.
section_title: In-School Youth Received More Summer Employment; Out-of- School Youth Received More Occupational Skills Training
paragraphs: Local areas tailored services provided to youth based on their needs,  emphasizing different services, depending on whether youth were in or   out of school. To meet youth’s many developmental needs, the WIA youth  program requires that local areas offer the same menu of 10 academic,  employment, and support services to all eligible youth, which WIA  providers choose from when tailoring services to an individual’s service  strategy based on an assessment of needs. As shown in figure 2, services  such as work experience and leadership development were provided fairly  equally to both youth populations, but there were differences in other  areas. For example, in-school youth were more likely to receive tutoring  services and summer employment linked to classroom learning, while out- of-school youth were more likely to receive occupational skills training  and supportive services to help prepare them for employment.
WIA requires that summer employment programs be linked to academic  learning, and Labor guidance promotes meaningful summer enrichment  experiences to help youth attain skills throughout the school year. Local  youth programs in the five states we visited were providing youth with  various types of summer experiences. For example, a service provider in a  rural area of California enrolled in-school youth in a 6-week summer  enrichment program where students worked part-time while learning  reading skills. Another service provider in a rural area in Louisiana offered  in-school youth summer services that included academic enrichment,  community service, and exposure to different career options.
Local youth programs provided out-of-school youth training in  occupational skills and job readiness, as well as offered them supportive  services. Labor’s guidance states that enrolling youth in occupational skills  training and retaining them until program completion leads to better  outcomes. Local areas provided several different types of occupational  skills training, depending on local labor market needs. For example, one  urban local area in Ohio trained out-of-school youth in nursing, welding,  and computer repair, while another urban local area in Virginia offered  technology certification training. Local areas also provided job readiness  skills cited by employers as lacking in many youth seeking employment.  For example, a program for out-of-school youth in an urban area of Ohio  taught participants such job readiness skills as customer service, conflict  management, and other interpersonal skills. Supportive services were also  important for out-of-school youth who needed additional assistance to  help them overcome their multiple barriers to employment. A service  provider in New Hampshire provided out-of-school participants with  support services such as child care, transportation, and housing  assistance.
section_title: Case Managers Developed Service Strategies and Providers Coordinated to Deliver Services
paragraphs: Almost all local boards we surveyed reported using case managers to  coordinate services for youth in and out of school. Because at-risk youth  often require services from a host of departments, including Education,  Health and Human Services, Housing and Urban Development, Justice,  and others, Labor’s guidance promotes the use of a case manager to  coordinate services among all these youth-serving partners to provide  seamless access to and delivery of services. Nationwide, all but 3 percent  of local boards responding to our survey reported that their youth program  uses case managers. Local boards reported that, in addition to performing  other duties, case managers assessed youth upon enrollment in the WIA  program. These assessments typically included a review of educational  attainment, work readiness, work experience, and career interests. The  assessments are then used to develop an individual service strategy for  each youth participant. Labor’s guidance requires that each strategy  identify employment goals and educational objectives, and prescribe  appropriate services for each participant. About three-fourths of local  boards reported that youth were greatly or very greatly involved in the  development of their individual service strategies.
Most local areas used multiple service providers to deliver the youth  services spelled out in each participant’s individual service strategy. WIA  does not specify how services must be provided to youth, allowing local  areas to determine how many providers they will fund and hold  responsible for delivering the services outlined in each youth’s individual  service strategy. Over half of local boards responding to our survey  reported using 4 or fewer service providers, but 15 percent used more than  10 providers, as shown in figure 3.
Over half of local boards nationwide used providers that did not deliver all  services themselves, using formal or informal subcontracting  arrangements to provide the range of services needed. The extent that  providers coordinated with others to deliver services was related to the  number of youth served in the local area. In local areas with 250 or fewer  youth, providers delivered all services themselves more often than in areas  with more than 250 youth, as indicated in table 2.
Regardless of the level of coordination used to provide youth services,  workforce officials told us that providers often informally collaborated  with one another through regular meetings to discuss problems, train one  another, share best practices, or share other resources.
Local boards responding to our survey reported using a range of public  and private entities to deliver youth services. Over half of all youth  received services primarily from educational institutions or community  organizations, while less than 1 percent of youth received services  primarily from faith-based organizations, as shown in figure 4.
Local boards reported that about one-fourth of youth received services  primarily from WIA one-stops—whether the one-stops focused on serving  adults or youth. However, among the one-stops we visited, these entities  were frequently used for supplemental services such as information and  referrals. Officials we spoke with in New Hampshire and Ohio said that  service providers or schools usually took youth on a field trip to the local  one-stop for basic orientation to expose them to employment services they  could use in the future. Although most youth were served by the WIA  program through other providers, one-stop usage varied considerably by  local area. For example, the majority of WIA youth living in an urban area  of California were served by youth-exclusive one-stops, while youth in a  rural area of the state rarely interacted with the local adult one-stop.
In-school and out-of-school youth usually did not participate in WIA  programs together. Nationwide, 43 percent of local boards reported that  these two types of youth received services from different providers. In  another 27 percent of cases, the two groups received services from the  same providers but participated separately. Another 24 percent of local  boards reported that all youth were served by the same service providers  and participated together.
section_title: Local Areas Developed Partnerships with Business and Education to Deliver Youth Services
paragraphs: Local areas developed partnerships with the local business and education  communities to identify employer needs and provide comprehensive youth  services related to academic and employment preparation. In many cases,  youth councils helped facilitate these partnerships.
Local areas partnered with businesses to identify and provide employment  and training services for WIA youth needed to fill high-demand, high- growth occupations. Over 85 percent of local boards reported using each  of the following methods to develop and maintain relationships with local  businesses: consulting with businesses about their job needs, training  participants in skills needed by local businesses, training program  participants to understand the values and attitudes local businesses look  for in employees, and providing employment experience to participants  that suits the jobs available in local businesses. Officials we spoke with at  the local youth programs in all five states we visited provided such work  readiness training as punctuality, teamwork, respect for others, and  appropriate dress. In addition to identifying needed skills, businesses also  provided employment opportunities. Nationally, 34 percent of local boards  reported that businesses subsidized work experience for WIA youth. In  New Hampshire, for example, a financial services firm employed out-of- school youth and taught them about personal financial management. In  addition, an appliance-store owner, once an at-risk youth himself, hired  WIA youth and hoped to groom one of them to take over the business.  Finally, officials in three states we visited noted that businesses donated  in-kind assistance such as building materials, work clothes, work  readiness training workshops, and a financial management curriculum.  For example, in an urban area in Virginia, businesses donated and  remodeled the space for the one-stop center.
Local areas also partnered with schools to provide academic preparation  services to WIA youth. In a local area of Louisiana, for example, a service  provider helped eighth graders explore classes leading to high school  industry-based certifications. In all five states we visited, representatives  from local postsecondary institutions made presentations to WIA youth,  informing them about higher education opportunities. In some instances,  WIA youth were allowed to take advanced level courses at their  institutions and even earn college credit. In addition, a community college  in California provided a pathway to facilitate the transition from high  school to higher education by hosting youth for a one-day college  experience of classes and conducting workshops on financial aid. Officials  in four states we visited also said that schools provided in-kind assistance  such as office space and tutoring.
Local boards reported that youth councils performed a number of  important functions that facilitated partnerships between local boards and  the community. Over 70 percent of local boards reported that the youth  council served as a forum to bring together key community partners who  may never have collaborated with each other. Almost two-thirds of local  boards reported that they would keep their youth council even if it became  optional. In addition, over half of local boards reported that their youth  councils elevated the importance and visibility of local youth issues and  programs and added value to the youth program.
section_title: Labor’s Guidance Has Not Fully Addressed Implementation Challenges Faced by Local Areas
paragraphs: Local areas continue to face challenges in implementing aspects of the  WIA youth program despite guidance issued by Labor. Some of these  ongoing challenges include recruiting and retaining out-of-school youth,  providing mentoring and follow-up services, and designing interim  performance measures that can be used to continually improve aspects of  program performance before youth exit the program. Labor supplements  guidance on these issues through information forums, a Web site, and a  technical assistance program. However, regional Labor officials said that  state and local areas’ access to these resources had been limited.
subsections: section_title: Out-of-School Youth Were Hard to Recruit and Retain
paragraphs: Labor has issued guidance on recruiting and retaining out-of-school youth  for the WIA program during the last two years, but local areas continue to  face challenges in serving this population. Labor issued a guidance letter  to states in April 2001 outlining strategies for recruiting youth to the  program, suggesting such methods as engaging youth to recruit their  peers, collaborating with community organizations that already work with  disadvantaged youth, and offering incentives for recruiting new  participants. In subsequent guidance issued in May 2002, Labor described  ways to keep out-of-school youth engaged in the program, such as by  helping youth make the link between career potential and education and  skills training as well as by providing support services like child care.  Despite this guidance, local officials from most of the states we visited  said that recruiting out-of-school youth continues to be a problem.  According to data from Labor, the District of Columbia, Delaware, and  New Mexico had not met WIA’s requirement to spend at least 30 percent of  WIA funds on out-of-school youth for program year 2001 as of September  30, 2003.
Some local workforce officials indicated that closer coordination with  schools to immediately connect dropouts to the WIA program would help  them identify and recruit more out-of-school youth. Some local workforce  and regional Labor officials said that one reason schools may not share  information on dropouts directly with the WIA program was because of  concerns about student privacy restrictions. However, schools we visited  in four states shared dropout information by referring students who had  been expelled or had dropped out directly to the WIA service provider,  notified or provided contact information to the WIA provider when a  student had dropped out, or worked with the WIA program to develop a  list of dropouts. None of the schools, however, had procedures in place to  routinely connect all dropouts with local WIA programs. Labor and  Education officials agreed that schools could do more to work closely  with local workforce officials to help connect dropouts to local WIA youth  programs.
Local officials from most of the states we visited said that retaining out-of- school youth in the WIA program was also difficult. According to some  local officials, one reason was that out-of-school youth tend to prefer  immediate employment instead of training and academic learning. Other  officials said that both in-school and out-of-school youth face difficulties  with transportation—particularly in rural areas—-that limit their  involvement in WIA programs. For example, local officials in a rural area  in Ohio said that transportation was their biggest issue. To mitigate this  challenge, the local area offered services through a mobile one-stop unit  that traveled to WIA clients throughout the county. Some state workforce  officials said that to be more effective in serving this population, they  needed guidance and technical assistance that was more focused on the  specific service delivery issues within their local areas.
section_title: Mentoring and Follow-up Were among the Most Difficult Services to Provide
paragraphs: Nearly all local areas we visited indicated that providing mentoring  services continued to be a problem for the WIA youth program, but Labor  has not addressed mentoring in its annual youth program guidance or  shared best practices on the provision of high-quality mentoring services.  Local areas identified several reasons why mentoring has been difficult. In  one instance, a Virginia official in a rural area said that it was difficult to  identify sufficient numbers of qualified mentors to work with eligible  youth. In another instance, a local area official in California noted that  geographical distances within the county resulted in long commutes and  discouraged adults from mentoring youth. A service provider in New  Hampshire said that finding mentors was especially difficult for out-of- school youth, as some adults are uncomfortable working with school  dropouts. Labor youth program officials said that mentoring is an  important way of connecting youth to caring adults and said they would  take action to provide guidance on this topic.
Another service element that remains challenging for local areas is  providing complete and thorough follow-up services to help youth succeed  after they have exited the program. WIA regulations require that follow-up  services last for at least 12 months, and Labor’s guidance states that  follow-up may include regular contact with a youth’s employer to track  progress made, assistance in addressing work-related problems, and help  in securing better-paying jobs and further education. Labor’s policy  guidance for program year 2001 provided some principles from best  practices in the field of youth development to help local areas develop  strategies for follow-up. The principles included developing a systematic  approach for maintaining contact and interaction with the young person;  evening and weekend social activities for informal support; helping youth  access services to fulfill physical, emotional, and vocational needs; and  visiting the job site soon after the youth has started employment.  Nevertheless, several local officials cited continued difficulties in  sustaining a connection with youth and identifying outcomes for them  once they exited the WIA program.
section_title: Few Local Areas Used Interim Measures to Gauge Program Success
paragraphs: Labor’s guidance underscored the importance of establishing and using  optional interim performance measures to monitor the success of  delivering WIA youth services, but some regional Labor officials said that  such measures were not widely used. Six of the seven required WIA youth  measures are collected only after youth exit the program. However,  because youth may remain in the program for many years, local areas with  long-term youth retention strategies may have limited means of gauging  progress without interim measures. Labor issued guidance in May 2002,  encouraging the use of interim measures, which may be tracked and  reported separately from the required annual performance measures, to  provide ongoing feedback on which aspects of the youth program were  working well or needed modification. The guidance identified the  following interim measures that local boards could use to monitor  progress as youth move through the WIA program:  intake, including development of a comprehensive individual service  strategy with short-term and long-term goals;   participation in program elements to help ensure youth meet short-term  skill attainment goals;   skill attainment to help ensure successful completion of short- and long-   exit data, including number of participants exiting the program; and  follow-up services received to help measure youth performance outcomes.
Some local areas we visited were using interim measures to hold providers  accountable for delivering services. For example, New Hampshire  workforce officials said that they tracked enrollment and expenditure  levels to monitor activity levels across their contracted service providers.  In a local area in California, workforce officials monitored the progress of  youth by tracking enrollment, participation, work readiness skills, and the  rate at which youth made a successful transition to other activities once  they exited the WIA program.
Despite the potential usefulness of these data, interim measures were not  widely used, according to Labor officials in two regions. These officials  said that states lacked the resources to properly track them. Additionally,  the guidance did not explain how states and local areas could collect,  analyze, and use the data to assess progress and make needed  adjustments. For example, officials in one state we visited said that they  were unclear about how to apply the interim measures.
section_title: Access to and Awareness of Labor’s Assistance Has Been Limited
paragraphs: Labor has disseminated information related to these and other issues  through forums that allow local areas to exchange information with one  another, but access to these forums has been limited. From December  2002 to April 2003, for example, Labor sponsored peer-to-peer WIA  Learning Exchanges with two national youth organizations that provided  venues around the country for local areas to share information and  observe an exemplary program firsthand. However, state budgetary cuts  prevented many local areas from attending some of these forums in  person, according to an official from the sponsoring organization.  Although information and ideas shared at the Learning Exchanges were  later summarized and made available to all local areas, those that were  unable to attend missed the opportunity to network with their colleagues  and develop an action plan to take back and apply to their program.
Labor has also supported the development of an online resource to  facilitate information sharing among local areas. The agency’s Promising  Practices Web site was intended to provide a mechanism for local areas to  post and share promising workforce development practices, including  those pertaining to WIA youth programs. However, some state workforce  and regional Labor officials said that states and local areas may not even  have been aware of the site, that it has been difficult to enter practices for  inclusion, and that some users found it difficult to access the documents  described in the narrative. We also had difficulty accessing and using the  Web site to find information. For example, when we conducted a search  on the key word “mentoring,” the five results did not satisfactorily address  the topic.
Labor guidance encourages local areas to tap into other resources that can  help them develop quality youth programs. The Promising and Effective  Practices Network, for example, provides a useful listing of promising  practices categorized by the specific strategy and the 10 required WIA  youth services that local areas can use for improving their WIA youth  programs.
Labor has provided technical assistance to states through its Performance  Enhancement Project, initiated in program year 2002. Under this initiative,  Labor grouped into three tiers states that failed or were at risk of failing to  meet their performance goals, based on their reported outcomes,  according to Labor officials. These officials also said that states in the  first tier received priority for targeted technical assistance to improve their  youth programs, with the goal of improving performance outcomes. Labor  officials said that states could use this technical assistance to address any  of the challenges they faced in implementing their WIA youth program.  However, unless the state falls into one of these three tiers, most local  areas may not receive such assistance to help them increase the  proportion of out-of-school youth served, improve mentoring and follow- up, and use interim measures.
section_title: Lack of Program Evaluations and Questionable Performance Data Prevent Assessments of WIA Youth Program
paragraphs: Little is known about the effectiveness of the WIA youth program because  Labor has not yet initiated an impact evaluation, and results from a  planned evaluation will likely not be available until 2009, according to  Labor officials. While Labor’s performance data for program year   2001 indicate that five of seven youth measures were exceeded, these  results cannot be used to infer program outcomes because of insufficient  monitoring of state data quality and inadequacies of some state  management information systems. Labor officials said that they will  require states to validate local area data beginning with program year   2002 data, but some implementation concerns remain.
subsections: section_title: Labor Plans to Initiate First Impact Evaluation in 2006, and Results Are At Least Five Years Away
paragraphs: According to department officials, Labor intends to initiate an impact  evaluation of the WIA youth program in 2006, and preliminary results will  not be available until about 2009. While Labor officials said that the youth  program’s 3 years of full implementation was sufficient time to initiate a  comprehensive evaluation, they were anticipating significant changes to  the WIA youth program as a result of reauthorization. They did not plan to  begin an impact evaluation of the program until after these changes had  been implemented.
If reauthorization is completed by spring 2004, Labor officials said they  anticipate that the process of awarding the contract for the study will be  completed by 2006 when the impact evaluation is scheduled to begin. The  evaluation will proceed with 5 or 6 years of data collection with an  additional 3 or 4 years of follow-up activity. Labor officials said they  expect to issue a series of interim reports before the final product. If the  project begins in 2006, Labor expects to issue the first report on short-term  impacts in 2009, with a final report on long-term impacts available in   2011. According to officials, this evaluation is part of Labor’s proposed  research plan to study all of its major employment and training programs,  including the WIA youth program, on a regular cycle.
section_title: State Performance Outcome Data Were Questionable, Partly because of Inconsistent Data Monitoring
paragraphs: Performance data submitted by states to Labor in quarterly and annual  reports were not sufficiently reliable to determine outcomes for the WIA  youth program. Labor’s national aggregation of WIA performance data for  program year 2001 indicated that the program exceeded its goals for five  of the seven youth measures. According to Labor’s Office of Inspector  General (OIG), however, there is little assurance that the states’  performance data for all WIA programs, including the youth program, are  either accurate or complete because of inadequate oversight of data  collection and management at the local, state, and federal levels. The OIG  also found that just 2 of 12 local areas and none of four states it reviewed  had formal policies on documentation requirements for participant  activities and outcomes. At the local level, for example, another OIG  report found that WIA youth program outcomes were adequately  documented only 37 percent of the time for a sample of 420 participants  across 14 local areas. An official in one of Labor’s regional offices added  that documentation requirements are inconsistent among states and local  areas.
At the state level, the OIG reported that two of four states it reviewed had  not monitored local areas’ performance data at the case file level, and that  none of the four states had adequate procedures in place to ensure the  accuracy of their performance data. According to regional Labor officials,  some states had insufficient procedures in place for verifying the accuracy  of the data collected by their local areas. For example, officials in one  region said that state monitoring rarely addressed data accuracy or  included the verification of a sample of data items against original records.  In addition, some state information systems had significant flaws that  caused them to produce incorrect information, casting further doubt on  the quality of WIA performance data. For example, Ohio state officials said  they were in the process of replacing their old information system that had  caused such errors as data changing or disappearing entirely after it had  been entered in the system.
At the federal level, Labor did not have a standard data-monitoring guide  in place, and officials in some regional offices—who, according to agency  officials, are responsible for overseeing the quality of states’ reported  data—-said they followed their own oversight procedures. These  procedures did not usually include verifying the accuracy of a sample of  the data submitted by states. For example, an OIG report stated that while  regional offices conducted some data accuracy reviews, such as computer  edit checks, they did not verify the data’s accuracy with such tests as  comparing the data with participant case files. Consistent with the OIG’s  findings, officials in all six of Labor’s regional offices said that they  examined state data submissions through desk reviews, which included  checking for errors such as incorrect calculation of performance  measures, extreme outliers, and miscoding of data. However, only the  Atlanta regional office checked a sample of data records against source  documentation. In its review of data records from six of the states it  oversees, the Atlanta office examined a sample of participant records  across all WIA programs from each state and found errors that affected the  calculation of one or more performance measures. In one state, for  example, 17 percent of participant records had at least one error,  compared with 83 percent of participant records in another state. The  regional office also found that two of six states it reviewed computed the  younger youth skill attainment measure incorrectly.
Labor recognizes these problems with data reliability, but in accordance  with WIA regulations, uses states’ annual performance reports—-the only  reports that depict states’ progress in achieving or exceeding negotiated  performance levels—-to make incentive grant and sanction decisions.  States that exceed the WIA performance goals negotiated between Labor  and the states are eligible to receive incentive grants that generally range  from $750,000 to $3 million. States that fail to meet 80 percent of their  WIA performance goals for 2 years in a row are subject to sanction of up  to a 5 percent reduction in their annual WIA formula grant. However,  Labor’s use of questionable performance data reported by the states may  hinder its ability to negotiate realistic performance goals and make  appropriate incentive grant and sanction decisions. Ohio state officials  questioned the appropriateness of being sanctioned 2 percent of its  program year 2002 WIA youth allotment. They believed that poor  performance levels were due to problems with its information system,   not its workforce development system.
section_title: Data Validation Initiative May Improve Reliability but Concerns Remain
paragraphs: To address the data issues described above, Labor is implementing a new  data validation policy requiring states to ensure the accuracy of their  annual reports and verify a sample of the data they submit. According to  Labor’s policy, these requirements will be phased in over a 3-year period.  In the first year, states will be required to validate their annual reports and  data submissions for the program year 2002 period. According to agency  officials, Labor does not plan to publicly release these initial reports but  will use their findings to work with the states to correct their data  accuracy problems. In the second year, Labor will use validation reports  covering program year 2003 data to establish acceptable error rates. In the  third year, Labor will require states to meet the acceptable error rates for  their program year 2004 data submissions. Labor will consider failure to  meet the standard a violation equivalent to failing to submit a report, for  which states may be subject to corrective action or financial sanction, as  appropriate. In addition, Labor’s data validation guidance indicates that  states that do not meet data accuracy standards will receive technical  assistance.
While the data validation initiative may improve the reliability of WIA  performance data, several implementation concerns remain. First, officials  in some of Labor’s regional offices said that the states they oversee will  have difficulty implementing the data element verification requirement  because of limited staff resources. Further, these officials said that in  states where local providers keep the original documentation on-site,  retrieving the documentation to check it against records in their  information system will be time-consuming and expensive. Second, a  Labor official said that the agency does not plan to issue a program- monitoring guide to standardize procedures across regional offices for  several more years.
section_title: Conclusions
paragraphs: Many youth are struggling to be successful in the public education system  and often face substantial obstacles to obtaining a high school diploma  and going on to college or getting jobs with career advancement  possibilities. Many of these youth may lack meaningful social and family  supports and may require comprehensive, intensive services to remain  engaged in society and avoid risky behaviors that can lead to chronic  unemployment, criminal activity, and other adverse outcomes. WIA  currently provides both the education and workforce systems with  strategies and resources to engage youth in academic and job training.   The opinion held by some federal education officials that WIA in-school  services overlap with existing education programs is not necessarily  seconded by officials in local areas. State and local workforce and  education officials believe WIA’s educational, occupational, and support  services provide critical support and services to at-risk youth and that  without WIA’s comprehensive services, schools may face an increasing  burden to keep these youth in school and ensure their academic success.  The connection between WIA youth services and schools could be made  more effective if Labor worked with Education to find ways to connect  school dropouts with local WIA youth programs.
Since WIA’s passage, Labor has provided general guidance and promising  practices in addressing implementation issues, but increased availability   of technical assistance may be necessary to overcome some of the more  difficult challenges some states and local areas face in providing youth  services. This will be especially critical for states and local areas that are  reportedly meeting performance goals but still need assistance in  improving delivery of youth services and shifting program resources to  target different populations such as out-of-school youth. In addition, local  areas will need guidance, including specific strategies, to help them  provide effective mentoring and follow-up services and use interim  measures to track program performance.
Robust research and reliable performance data are needed to obtain a  complete picture of the WIA youth program’s effectiveness and outcomes  and to make quality decisions about managing the program. However,  none currently exist. Labor has not yet initiated an impact evaluation and  does not expect to report on program impacts from a planned evaluation  until 2009. In addition, states and local areas continue to struggle to collect  and document accurate and complete participant and performance data  and maintain data systems that can yield reasonably reliable outcome  information. For its part, Labor is taking action to improve data integrity  by requiring states to validate WIA performance data, beginning with data  from program year 2002. However, Labor’s inconsistent monitoring  processes across regions will continue to challenge Labor’s capability to  ensure that all states validate and report data consistently and effectively.  In the short term, the lack of accurate outcome data will continue to  hinder Labor’s ability to negotiate realistic state performance goals and  use data to make sound decisions about financial incentives or sanctions.  In the long term, lack of accurate and complete information will keep  Labor from obtaining a true picture of how effectively the youth program  is working.
section_title: Recommendations for Executive Action
paragraphs: To promote information sharing that improves local WIA youth programs’  ability to identify and serve out-of-school youth, we recommend that the  Secretaries of Labor and Education coordinate efforts to clarify how  schools can work with workforce officials to connect school dropouts  with local WIA youth programs.
To assist state and local WIA youth programs address ongoing  implementation challenges, we recommend that the Secretary of Labor  increase availability of guidance and technical assistance to local areas  that continue to face challenges in serving out-of-school youth;   disseminate guidance, including specific strategies, to help local areas  provide effective mentoring services; and   develop additional guidance on providing follow-up services and using  interim measures to track program performance.
To obtain an accurate and complete gauge of WIA outcomes and  determine whether local programs are operating successfully, we  recommend that the Secretary of Labor establish standard monitoring  procedures that Labor’s regional offices could use to oversee state data  validation efforts.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to Labor and Education officials for  their review and comment. Education’s comments are reprinted in  appendix I. In its formal comments, Education concurred with our  recommendation to work with Labor to connect out-of-school youth to  local WIA youth programs. Labor responded informally, and said that  many of the findings corroborated its own observations and that the  recommendations are consistent with Labor’s current program direction.  Both agencies also provided technical comments, which we incorporated  as appropriate.
We will send copies of this report to the Secretaries of Labor and  Education, relevant congressional committees, and other interested  parties. Copies will be made available to others upon request. In addition,  the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
Please contact me at (415) 904-2272 if you or your staff have any questions  about this report. Other major contributors to this report are listed in  appendix II.
section_title: Appendix I: Comments from the Department of Education
section_title: Appendix II: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals mentioned above, the following staff made  key contributions to this report: Karyn Angulo, Susan Baker, Andrew  Bauck, Paula Bonin, Maya Chakko, Keira Dembowski, Joel Grossman,  Corinna Nicolaou, Rebecca Woiwode, and Monica Wolford.
section_title: Related GAO Products
paragraphs: Transportation-Disadvantaged Populations: Some Coordination Efforts  Among Programs Providing Transportation Services, but Obstacles  Persist. GAO-03-697. Washington, D.C.: June 30, 2003.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing is Needed. GAO-03-725. Washington D.C.: June 18,  2003.
Workforce Investment Act: Issues Related to Allocation Formulas for  Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.:  April 25, 2003.
Workforce Investment Act: States’ Spending Is on Track, but Better  Guidance Would Improve Financial Reporting. GAO-03-239. Washington,  D.C.: November 22, 2002.
Workforce Investment Act: Youth Provisions Promote New Service  Strategies, but Additional Guidance Would Enhance Program  Development. GAO-02-413. Washington, D.C.: April 5, 2002.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.
School Dropouts: Education Could Play a Stronger Role in Identifying  and Disseminating Promising Prevention Strategies. GAO-02-240.  Washington, D.C.: February 1, 2002.