Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services
Released: 2004-02-23T00:00:00

Letter

Background

Enacted in 1998, WIA replaced the fragmented and overlapping programs under the Job Training Partnership Act (JTPA) with a system that sought to connect employment, education, and training services to better match job seekers to labor market needs. The youth program falls under Title I of WIA along with programs for adults and dislocated workers; Title II deals with adult and family literacy; Title III pertains to employment services under the Wagner-Peyser Act; and Title IV addresses vocational rehabilitation. To better prepare low-income youth who face barriers to employment and education, WIA requires youth programs to focus on long-term, comprehensive services delivered year-round through a coordinated network of service providers. In addition to meeting the low- income requirement, to be eligible for WIA services youth must be between the ages of 14 and 21 and faced with one or more of six barriers to employment. WIA serves both in-school and out-of-school youth. A significant portion of out-of-school youth are high school dropouts and unemployed youth who are struggling to succeed in the public education system and lack financial, family, or social support. Thus, at least 30 percent of local WIA youth funds must be spent on out-of-school youth.

WIA Youth Funding

Since WIA’s full implementation in 2000, funding for the youth program has ranged between approximately $1 billion and $1.4 billion a year. Labor follows a formula to allocate WIA funds to states, which in turn distribute money to their local workforce investment areas. At the state level, the governor can reserve up to 15 percent of the annual WIA allotment for such statewide activities as developing the WIA strategic plan for the state. States allocate the remaining 85 percent of funds to local boards that use the funds to develop the local strategic plan, establish a youth council, and award competitive contracts to youth service providers that have been recommended by the youth council. In addition to recommending eligible providers, the youth council coordinates youth activities in the local area, oversees providers, and carries out other duties authorized by the local board such as forging linkages with educational agencies. Youth councils may also leverage additional public and private funds to supplement their WIA funding in order to provide comprehensive youth services.

Youth Services

Once they are determined to be WIA eligible, youth receive an objective assessment of their academic level, skills, and service needs. Local youth programs then use the assessment to create each participant’s individual service strategy, which lays out employment goals, educational objectives, and necessary services. Every local youth program must offer the following 10 services, known as program elements, to eligible youth, though participants may receive different combinations of these elements depending on their service strategy. Labor groups the 10 required program elements around four major themes: 1. tutoring, study skills training, and instruction leading to completion of secondary school, including dropout prevention, 2. alternative school services, Preparing for and succeeding in employment 3. summer employment linked to academic and occupational learning, 4. paid and unpaid work experiences, including internships and job 5. occupational skills training, Developing the potential of young people as citizens and leaders 6. leadership development, which may include community service and peer-centered activities encouraging responsibility, 7. supportive services (such as child care and housing assistance), 8. adult mentoring for at least 12 months that may occur both during and after program participation, 9. follow-up upon program completion for at least 12 months, and 10. comprehensive guidance and counseling.

Eligible youth may obtain these services directly from approved youth service providers or through WIA’s one-stop system, which serves as a gateway to a variety of employment and training services. In addition to helping WIA youth gain access to the 10 program elements, the one-stop system also provides all youth with basic services, whether or not they are eligible for WIA. Any young person may walk into a one-stop center and make use of the center’s career resources and obtain information on and referrals to other providers. While one-stop centers are designed primarily to serve those 18 and older, some states have established one-stops that serve only youth, as we reported previously.

Youth Program Guidance

Labor provides guidance to help states and local areas implement WIA. Labor’s guidance includes issuing annual Training and Employment Guidance Letters on the youth program in general as well as on specific topics, sponsoring WIA Learning Exchanges in every region, and maintaining a Web site for promising and effective practices. Labor responds to requests for clarification or additional information by phone or e-mail or at conferences. Labor’s guidance often includes promising and effective practices to reinforce state and local flexibility under WIA to tailor programs to best meet youths’ needs.

Performance Measures

The law requires that states and local areas collect performance information on seven youth measures, which are separated for younger and older youth (see table 1). All seven youth measures apply to both statewide and local performance.

WIA performance information is collected from service providers and flows upward to the local boards, then to state boards or agencies, and finally to Labor. Local performance data are aggregated and entered into a state’s own automated data system that tracks the activities of individual WIA participants. States use two mechanisms to report performance to Labor: WIASRD, a database of individual records containing activity and outcome information for each registered participant that has exited WIA, and state annual reports. Labor uses the state annual reports to track states’ progress in meeting negotiated performance goals. Labor then awards monetary incentives to states that meet or exceed their performance goals and sanctions states that fail to meet at least 80 percent of each goal in two consecutive years.

WIA Program Used Primarily for Dropout Prevention and Emphasized Summer Services and Skills Training Coordinated among Multiple Service Providers

Most local areas used WIA youth program services for a range of approaches to prevent academic failure and school dropouts, emphasizing learning-related summer services that were coordinated through case managers and multiple service providers. Nationally, about 70 percent of youth served were in school, according to local boards responding to our survey. For in-school youth, local boards provided summer employment services linked to classroom learning more often than for out-of-school youth, while services for this latter population more often included occupational skills training and supportive services. Local boards reported that most youth received services primarily from community organizations and educational institutions. These providers were most likely to subcontract or make informal arrangements with other organizations to deliver the full range of WIA services to youth participants.

Local Areas Focused on Dropout Prevention for In- School Youth

Overall, local areas’ approach to serving youth was to supplement schools’ dropout prevention efforts to keep youth connected to an educational system, according to state and local workforce and education officials. WIA allows states and local areas to determine the proportion of in-school and out-of school youth to be served, requiring only that they spend at least 30 percent of funds for out-of-school youth. Labor reported that with some exceptions, states chose to focus the majority of resources on in- school youth, and our survey showed that about 81 percent of local areas served more in-school youth than out-of-school youth. Nationally, about 70 percent of youth served were in school, according to local boards we surveyed. However, this percentage varied across the nation, as shown in figure 1. For example, 38 percent of youth served in South Dakota were in school, compared with 86 percent in Nebraska.

According to officials in four states that we visited, WIA youth programs primarily targeted in-school youth because recruiting and retaining out-of- school youth for the WIA program was much more difficult and expensive. For example, officials from a rural area in Ohio had difficulty identifying and retaining out-of-school youth whose contact information changes frequently, and Labor reports that the average cost of serving an out-of- school youth under the WIA program is about $4,000 a year, twice as much as for an in-school youth. Officials from other local areas we visited considered other factors. A local official in Louisiana said that in-school youth are interested in the WIA program, unlike out-of-school youth, who are difficult to engage. In contrast, a rural area in Virginia chose to focus primarily on out-of-school youth because so few services were available for this population.

The Administration has proposed amending the WIA youth program to focus more resources on out-of-school youth. Department of Education officials said that WIA’s services to in-school youth were not unique, since schools already offer various services to their students to deter them from dropping out. While Education officials said that the department’s grants geared exclusively toward dropout prevention would be insufficient to address the national dropout problem, they also stated that funding is available under many Education programs that could be used for dropout prevention activities. However, local workforce and education officials we spoke with in three states we visited said that they were either unaware of or unable to gain access to other available federal resources that could be used to provide intensive services to at-risk youth. In New Hampshire, for example, officials told us that WIA provides the only dropout prevention program, and that they were unaware of other available education dropout prevention resources.

State and local workforce and education officials we spoke with in the five states we visited said that WIA funding complemented rather than duplicated education services and was critical in preventing in-school youth from dropping out of school. For example, in a rural area in Ohio, workforce officials stated that the WIA program was the only dropout prevention program and that WIA provided students with their only chance at academic and career success. In addition, they said that WIA’s services were more intensive and comprehensive, and were delivered in a one-on-one setting where each student received individualized attention. In one urban school in Virginia, there was 1 WIA counselor for 50 eligible in-school youth, compared to 1 school counselor for as many as 300 students.

In-School Youth Received More Summer Employment; Out-of- School Youth Received More Occupational Skills Training

Local areas tailored services provided to youth based on their needs, emphasizing different services, depending on whether youth were in or out of school. To meet youth’s many developmental needs, the WIA youth program requires that local areas offer the same menu of 10 academic, employment, and support services to all eligible youth, which WIA providers choose from when tailoring services to an individual’s service strategy based on an assessment of needs. As shown in figure 2, services such as work experience and leadership development were provided fairly equally to both youth populations, but there were differences in other areas. For example, in-school youth were more likely to receive tutoring services and summer employment linked to classroom learning, while out- of-school youth were more likely to receive occupational skills training and supportive services to help prepare them for employment.

WIA requires that summer employment programs be linked to academic learning, and Labor guidance promotes meaningful summer enrichment experiences to help youth attain skills throughout the school year. Local youth programs in the five states we visited were providing youth with various types of summer experiences. For example, a service provider in a rural area of California enrolled in-school youth in a 6-week summer enrichment program where students worked part-time while learning reading skills. Another service provider in a rural area in Louisiana offered in-school youth summer services that included academic enrichment, community service, and exposure to different career options.

Local youth programs provided out-of-school youth training in occupational skills and job readiness, as well as offered them supportive services. Labor’s guidance states that enrolling youth in occupational skills training and retaining them until program completion leads to better outcomes. Local areas provided several different types of occupational skills training, depending on local labor market needs. For example, one urban local area in Ohio trained out-of-school youth in nursing, welding, and computer repair, while another urban local area in Virginia offered technology certification training. Local areas also provided job readiness skills cited by employers as lacking in many youth seeking employment. For example, a program for out-of-school youth in an urban area of Ohio taught participants such job readiness skills as customer service, conflict management, and other interpersonal skills. Supportive services were also important for out-of-school youth who needed additional assistance to help them overcome their multiple barriers to employment. A service provider in New Hampshire provided out-of-school participants with support services such as child care, transportation, and housing assistance.

Case Managers Developed Service Strategies and Providers Coordinated to Deliver Services

Almost all local boards we surveyed reported using case managers to coordinate services for youth in and out of school. Because at-risk youth often require services from a host of departments, including Education, Health and Human Services, Housing and Urban Development, Justice, and others, Labor’s guidance promotes the use of a case manager to coordinate services among all these youth-serving partners to provide seamless access to and delivery of services. Nationwide, all but 3 percent of local boards responding to our survey reported that their youth program uses case managers. Local boards reported that, in addition to performing other duties, case managers assessed youth upon enrollment in the WIA program. These assessments typically included a review of educational attainment, work readiness, work experience, and career interests. The assessments are then used to develop an individual service strategy for each youth participant. Labor’s guidance requires that each strategy identify employment goals and educational objectives, and prescribe appropriate services for each participant. About three-fourths of local boards reported that youth were greatly or very greatly involved in the development of their individual service strategies.

Most local areas used multiple service providers to deliver the youth services spelled out in each participant’s individual service strategy. WIA does not specify how services must be provided to youth, allowing local areas to determine how many providers they will fund and hold responsible for delivering the services outlined in each youth’s individual service strategy. Over half of local boards responding to our survey reported using 4 or fewer service providers, but 15 percent used more than 10 providers, as shown in figure 3.

Over half of local boards nationwide used providers that did not deliver all services themselves, using formal or informal subcontracting arrangements to provide the range of services needed. The extent that providers coordinated with others to deliver services was related to the number of youth served in the local area. In local areas with 250 or fewer youth, providers delivered all services themselves more often than in areas with more than 250 youth, as indicated in table 2.

Regardless of the level of coordination used to provide youth services, workforce officials told us that providers often informally collaborated with one another through regular meetings to discuss problems, train one another, share best practices, or share other resources.

Local boards responding to our survey reported using a range of public and private entities to deliver youth services. Over half of all youth received services primarily from educational institutions or community organizations, while less than 1 percent of youth received services primarily from faith-based organizations, as shown in figure 4.

Local boards reported that about one-fourth of youth received services primarily from WIA one-stops—whether the one-stops focused on serving adults or youth. However, among the one-stops we visited, these entities were frequently used for supplemental services such as information and referrals. Officials we spoke with in New Hampshire and Ohio said that service providers or schools usually took youth on a field trip to the local one-stop for basic orientation to expose them to employment services they could use in the future. Although most youth were served by the WIA program through other providers, one-stop usage varied considerably by local area. For example, the majority of WIA youth living in an urban area of California were served by youth-exclusive one-stops, while youth in a rural area of the state rarely interacted with the local adult one-stop.

In-school and out-of-school youth usually did not participate in WIA programs together. Nationwide, 43 percent of local boards reported that these two types of youth received services from different providers. In another 27 percent of cases, the two groups received services from the same providers but participated separately. Another 24 percent of local boards reported that all youth were served by the same service providers and participated together.

Local Areas Developed Partnerships with Business and Education to Deliver Youth Services

Local areas developed partnerships with the local business and education communities to identify employer needs and provide comprehensive youth services related to academic and employment preparation. In many cases, youth councils helped facilitate these partnerships.

Local areas partnered with businesses to identify and provide employment and training services for WIA youth needed to fill high-demand, high- growth occupations. Over 85 percent of local boards reported using each of the following methods to develop and maintain relationships with local businesses: consulting with businesses about their job needs, training participants in skills needed by local businesses, training program participants to understand the values and attitudes local businesses look for in employees, and providing employment experience to participants that suits the jobs available in local businesses. Officials we spoke with at the local youth programs in all five states we visited provided such work readiness training as punctuality, teamwork, respect for others, and appropriate dress. In addition to identifying needed skills, businesses also provided employment opportunities. Nationally, 34 percent of local boards reported that businesses subsidized work experience for WIA youth. In New Hampshire, for example, a financial services firm employed out-of- school youth and taught them about personal financial management. In addition, an appliance-store owner, once an at-risk youth himself, hired WIA youth and hoped to groom one of them to take over the business. Finally, officials in three states we visited noted that businesses donated in-kind assistance such as building materials, work clothes, work readiness training workshops, and a financial management curriculum. For example, in an urban area in Virginia, businesses donated and remodeled the space for the one-stop center.

Local areas also partnered with schools to provide academic preparation services to WIA youth. In a local area of Louisiana, for example, a service provider helped eighth graders explore classes leading to high school industry-based certifications. In all five states we visited, representatives from local postsecondary institutions made presentations to WIA youth, informing them about higher education opportunities. In some instances, WIA youth were allowed to take advanced level courses at their institutions and even earn college credit. In addition, a community college in California provided a pathway to facilitate the transition from high school to higher education by hosting youth for a one-day college experience of classes and conducting workshops on financial aid. Officials in four states we visited also said that schools provided in-kind assistance such as office space and tutoring.

Local boards reported that youth councils performed a number of important functions that facilitated partnerships between local boards and the community. Over 70 percent of local boards reported that the youth council served as a forum to bring together key community partners who may never have collaborated with each other. Almost two-thirds of local boards reported that they would keep their youth council even if it became optional. In addition, over half of local boards reported that their youth councils elevated the importance and visibility of local youth issues and programs and added value to the youth program.

Labor’s Guidance Has Not Fully Addressed Implementation Challenges Faced by Local Areas

Local areas continue to face challenges in implementing aspects of the WIA youth program despite guidance issued by Labor. Some of these ongoing challenges include recruiting and retaining out-of-school youth, providing mentoring and follow-up services, and designing interim performance measures that can be used to continually improve aspects of program performance before youth exit the program. Labor supplements guidance on these issues through information forums, a Web site, and a technical assistance program. However, regional Labor officials said that state and local areas’ access to these resources had been limited.

Out-of-School Youth Were Hard to Recruit and Retain

Labor has issued guidance on recruiting and retaining out-of-school youth for the WIA program during the last two years, but local areas continue to face challenges in serving this population. Labor issued a guidance letter to states in April 2001 outlining strategies for recruiting youth to the program, suggesting such methods as engaging youth to recruit their peers, collaborating with community organizations that already work with disadvantaged youth, and offering incentives for recruiting new participants. In subsequent guidance issued in May 2002, Labor described ways to keep out-of-school youth engaged in the program, such as by helping youth make the link between career potential and education and skills training as well as by providing support services like child care. Despite this guidance, local officials from most of the states we visited said that recruiting out-of-school youth continues to be a problem. According to data from Labor, the District of Columbia, Delaware, and New Mexico had not met WIA’s requirement to spend at least 30 percent of WIA funds on out-of-school youth for program year 2001 as of September 30, 2003.

Some local workforce officials indicated that closer coordination with schools to immediately connect dropouts to the WIA program would help them identify and recruit more out-of-school youth. Some local workforce and regional Labor officials said that one reason schools may not share information on dropouts directly with the WIA program was because of concerns about student privacy restrictions. However, schools we visited in four states shared dropout information by referring students who had been expelled or had dropped out directly to the WIA service provider, notified or provided contact information to the WIA provider when a student had dropped out, or worked with the WIA program to develop a list of dropouts. None of the schools, however, had procedures in place to routinely connect all dropouts with local WIA programs. Labor and Education officials agreed that schools could do more to work closely with local workforce officials to help connect dropouts to local WIA youth programs.

Local officials from most of the states we visited said that retaining out-of- school youth in the WIA program was also difficult. According to some local officials, one reason was that out-of-school youth tend to prefer immediate employment instead of training and academic learning. Other officials said that both in-school and out-of-school youth face difficulties with transportation—particularly in rural areas—-that limit their involvement in WIA programs. For example, local officials in a rural area in Ohio said that transportation was their biggest issue. To mitigate this challenge, the local area offered services through a mobile one-stop unit that traveled to WIA clients throughout the county. Some state workforce officials said that to be more effective in serving this population, they needed guidance and technical assistance that was more focused on the specific service delivery issues within their local areas.

Mentoring and Follow-up Were among the Most Difficult Services to Provide

Nearly all local areas we visited indicated that providing mentoring services continued to be a problem for the WIA youth program, but Labor has not addressed mentoring in its annual youth program guidance or shared best practices on the provision of high-quality mentoring services. Local areas identified several reasons why mentoring has been difficult. In one instance, a Virginia official in a rural area said that it was difficult to identify sufficient numbers of qualified mentors to work with eligible youth. In another instance, a local area official in California noted that geographical distances within the county resulted in long commutes and discouraged adults from mentoring youth. A service provider in New Hampshire said that finding mentors was especially difficult for out-of- school youth, as some adults are uncomfortable working with school dropouts. Labor youth program officials said that mentoring is an important way of connecting youth to caring adults and said they would take action to provide guidance on this topic.

Another service element that remains challenging for local areas is providing complete and thorough follow-up services to help youth succeed after they have exited the program. WIA regulations require that follow-up services last for at least 12 months, and Labor’s guidance states that follow-up may include regular contact with a youth’s employer to track progress made, assistance in addressing work-related problems, and help in securing better-paying jobs and further education. Labor’s policy guidance for program year 2001 provided some principles from best practices in the field of youth development to help local areas develop strategies for follow-up. The principles included developing a systematic approach for maintaining contact and interaction with the young person; evening and weekend social activities for informal support; helping youth access services to fulfill physical, emotional, and vocational needs; and visiting the job site soon after the youth has started employment. Nevertheless, several local officials cited continued difficulties in sustaining a connection with youth and identifying outcomes for them once they exited the WIA program.

Few Local Areas Used Interim Measures to Gauge Program Success

Labor’s guidance underscored the importance of establishing and using optional interim performance measures to monitor the success of delivering WIA youth services, but some regional Labor officials said that such measures were not widely used. Six of the seven required WIA youth measures are collected only after youth exit the program. However, because youth may remain in the program for many years, local areas with long-term youth retention strategies may have limited means of gauging progress without interim measures. Labor issued guidance in May 2002, encouraging the use of interim measures, which may be tracked and reported separately from the required annual performance measures, to provide ongoing feedback on which aspects of the youth program were working well or needed modification. The guidance identified the following interim measures that local boards could use to monitor progress as youth move through the WIA program: intake, including development of a comprehensive individual service strategy with short-term and long-term goals; participation in program elements to help ensure youth meet short-term skill attainment goals; skill attainment to help ensure successful completion of short- and long- exit data, including number of participants exiting the program; and follow-up services received to help measure youth performance outcomes.

Some local areas we visited were using interim measures to hold providers accountable for delivering services. For example, New Hampshire workforce officials said that they tracked enrollment and expenditure levels to monitor activity levels across their contracted service providers. In a local area in California, workforce officials monitored the progress of youth by tracking enrollment, participation, work readiness skills, and the rate at which youth made a successful transition to other activities once they exited the WIA program.

Despite the potential usefulness of these data, interim measures were not widely used, according to Labor officials in two regions. These officials said that states lacked the resources to properly track them. Additionally, the guidance did not explain how states and local areas could collect, analyze, and use the data to assess progress and make needed adjustments. For example, officials in one state we visited said that they were unclear about how to apply the interim measures.

Access to and Awareness of Labor’s Assistance Has Been Limited

Labor has disseminated information related to these and other issues through forums that allow local areas to exchange information with one another, but access to these forums has been limited. From December 2002 to April 2003, for example, Labor sponsored peer-to-peer WIA Learning Exchanges with two national youth organizations that provided venues around the country for local areas to share information and observe an exemplary program firsthand. However, state budgetary cuts prevented many local areas from attending some of these forums in person, according to an official from the sponsoring organization. Although information and ideas shared at the Learning Exchanges were later summarized and made available to all local areas, those that were unable to attend missed the opportunity to network with their colleagues and develop an action plan to take back and apply to their program.

Labor has also supported the development of an online resource to facilitate information sharing among local areas. The agency’s Promising Practices Web site was intended to provide a mechanism for local areas to post and share promising workforce development practices, including those pertaining to WIA youth programs. However, some state workforce and regional Labor officials said that states and local areas may not even have been aware of the site, that it has been difficult to enter practices for inclusion, and that some users found it difficult to access the documents described in the narrative. We also had difficulty accessing and using the Web site to find information. For example, when we conducted a search on the key word “mentoring,” the five results did not satisfactorily address the topic.

Labor guidance encourages local areas to tap into other resources that can help them develop quality youth programs. The Promising and Effective Practices Network, for example, provides a useful listing of promising practices categorized by the specific strategy and the 10 required WIA youth services that local areas can use for improving their WIA youth programs.

Labor has provided technical assistance to states through its Performance Enhancement Project, initiated in program year 2002. Under this initiative, Labor grouped into three tiers states that failed or were at risk of failing to meet their performance goals, based on their reported outcomes, according to Labor officials. These officials also said that states in the first tier received priority for targeted technical assistance to improve their youth programs, with the goal of improving performance outcomes. Labor officials said that states could use this technical assistance to address any of the challenges they faced in implementing their WIA youth program. However, unless the state falls into one of these three tiers, most local areas may not receive such assistance to help them increase the proportion of out-of-school youth served, improve mentoring and follow- up, and use interim measures.

Lack of Program Evaluations and Questionable Performance Data Prevent Assessments of WIA Youth Program

Little is known about the effectiveness of the WIA youth program because Labor has not yet initiated an impact evaluation, and results from a planned evaluation will likely not be available until 2009, according to Labor officials. While Labor’s performance data for program year 2001 indicate that five of seven youth measures were exceeded, these results cannot be used to infer program outcomes because of insufficient monitoring of state data quality and inadequacies of some state management information systems. Labor officials said that they will require states to validate local area data beginning with program year 2002 data, but some implementation concerns remain.

Labor Plans to Initiate First Impact Evaluation in 2006, and Results Are At Least Five Years Away

According to department officials, Labor intends to initiate an impact evaluation of the WIA youth program in 2006, and preliminary results will not be available until about 2009. While Labor officials said that the youth program’s 3 years of full implementation was sufficient time to initiate a comprehensive evaluation, they were anticipating significant changes to the WIA youth program as a result of reauthorization. They did not plan to begin an impact evaluation of the program until after these changes had been implemented.

If reauthorization is completed by spring 2004, Labor officials said they anticipate that the process of awarding the contract for the study will be completed by 2006 when the impact evaluation is scheduled to begin. The evaluation will proceed with 5 or 6 years of data collection with an additional 3 or 4 years of follow-up activity. Labor officials said they expect to issue a series of interim reports before the final product. If the project begins in 2006, Labor expects to issue the first report on short-term impacts in 2009, with a final report on long-term impacts available in 2011. According to officials, this evaluation is part of Labor’s proposed research plan to study all of its major employment and training programs, including the WIA youth program, on a regular cycle.

State Performance Outcome Data Were Questionable, Partly because of Inconsistent Data Monitoring

Performance data submitted by states to Labor in quarterly and annual reports were not sufficiently reliable to determine outcomes for the WIA youth program. Labor’s national aggregation of WIA performance data for program year 2001 indicated that the program exceeded its goals for five of the seven youth measures. According to Labor’s Office of Inspector General (OIG), however, there is little assurance that the states’ performance data for all WIA programs, including the youth program, are either accurate or complete because of inadequate oversight of data collection and management at the local, state, and federal levels. The OIG also found that just 2 of 12 local areas and none of four states it reviewed had formal policies on documentation requirements for participant activities and outcomes. At the local level, for example, another OIG report found that WIA youth program outcomes were adequately documented only 37 percent of the time for a sample of 420 participants across 14 local areas. An official in one of Labor’s regional offices added that documentation requirements are inconsistent among states and local areas.

At the state level, the OIG reported that two of four states it reviewed had not monitored local areas’ performance data at the case file level, and that none of the four states had adequate procedures in place to ensure the accuracy of their performance data. According to regional Labor officials, some states had insufficient procedures in place for verifying the accuracy of the data collected by their local areas. For example, officials in one region said that state monitoring rarely addressed data accuracy or included the verification of a sample of data items against original records. In addition, some state information systems had significant flaws that caused them to produce incorrect information, casting further doubt on the quality of WIA performance data. For example, Ohio state officials said they were in the process of replacing their old information system that had caused such errors as data changing or disappearing entirely after it had been entered in the system.

At the federal level, Labor did not have a standard data-monitoring guide in place, and officials in some regional offices—who, according to agency officials, are responsible for overseeing the quality of states’ reported data—-said they followed their own oversight procedures. These procedures did not usually include verifying the accuracy of a sample of the data submitted by states. For example, an OIG report stated that while regional offices conducted some data accuracy reviews, such as computer edit checks, they did not verify the data’s accuracy with such tests as comparing the data with participant case files. Consistent with the OIG’s findings, officials in all six of Labor’s regional offices said that they examined state data submissions through desk reviews, which included checking for errors such as incorrect calculation of performance measures, extreme outliers, and miscoding of data. However, only the Atlanta regional office checked a sample of data records against source documentation. In its review of data records from six of the states it oversees, the Atlanta office examined a sample of participant records across all WIA programs from each state and found errors that affected the calculation of one or more performance measures. In one state, for example, 17 percent of participant records had at least one error, compared with 83 percent of participant records in another state. The regional office also found that two of six states it reviewed computed the younger youth skill attainment measure incorrectly.

Labor recognizes these problems with data reliability, but in accordance with WIA regulations, uses states’ annual performance reports—-the only reports that depict states’ progress in achieving or exceeding negotiated performance levels—-to make incentive grant and sanction decisions. States that exceed the WIA performance goals negotiated between Labor and the states are eligible to receive incentive grants that generally range from $750,000 to $3 million. States that fail to meet 80 percent of their WIA performance goals for 2 years in a row are subject to sanction of up to a 5 percent reduction in their annual WIA formula grant. However, Labor’s use of questionable performance data reported by the states may hinder its ability to negotiate realistic performance goals and make appropriate incentive grant and sanction decisions. Ohio state officials questioned the appropriateness of being sanctioned 2 percent of its program year 2002 WIA youth allotment. They believed that poor performance levels were due to problems with its information system, not its workforce development system.

Data Validation Initiative May Improve Reliability but Concerns Remain

To address the data issues described above, Labor is implementing a new data validation policy requiring states to ensure the accuracy of their annual reports and verify a sample of the data they submit. According to Labor’s policy, these requirements will be phased in over a 3-year period. In the first year, states will be required to validate their annual reports and data submissions for the program year 2002 period. According to agency officials, Labor does not plan to publicly release these initial reports but will use their findings to work with the states to correct their data accuracy problems. In the second year, Labor will use validation reports covering program year 2003 data to establish acceptable error rates. In the third year, Labor will require states to meet the acceptable error rates for their program year 2004 data submissions. Labor will consider failure to meet the standard a violation equivalent to failing to submit a report, for which states may be subject to corrective action or financial sanction, as appropriate. In addition, Labor’s data validation guidance indicates that states that do not meet data accuracy standards will receive technical assistance.

While the data validation initiative may improve the reliability of WIA performance data, several implementation concerns remain. First, officials in some of Labor’s regional offices said that the states they oversee will have difficulty implementing the data element verification requirement because of limited staff resources. Further, these officials said that in states where local providers keep the original documentation on-site, retrieving the documentation to check it against records in their information system will be time-consuming and expensive. Second, a Labor official said that the agency does not plan to issue a program- monitoring guide to standardize procedures across regional offices for several more years.

Conclusions

Many youth are struggling to be successful in the public education system and often face substantial obstacles to obtaining a high school diploma and going on to college or getting jobs with career advancement possibilities. Many of these youth may lack meaningful social and family supports and may require comprehensive, intensive services to remain engaged in society and avoid risky behaviors that can lead to chronic unemployment, criminal activity, and other adverse outcomes. WIA currently provides both the education and workforce systems with strategies and resources to engage youth in academic and job training. The opinion held by some federal education officials that WIA in-school services overlap with existing education programs is not necessarily seconded by officials in local areas. State and local workforce and education officials believe WIA’s educational, occupational, and support services provide critical support and services to at-risk youth and that without WIA’s comprehensive services, schools may face an increasing burden to keep these youth in school and ensure their academic success. The connection between WIA youth services and schools could be made more effective if Labor worked with Education to find ways to connect school dropouts with local WIA youth programs.

Since WIA’s passage, Labor has provided general guidance and promising practices in addressing implementation issues, but increased availability of technical assistance may be necessary to overcome some of the more difficult challenges some states and local areas face in providing youth services. This will be especially critical for states and local areas that are reportedly meeting performance goals but still need assistance in improving delivery of youth services and shifting program resources to target different populations such as out-of-school youth. In addition, local areas will need guidance, including specific strategies, to help them provide effective mentoring and follow-up services and use interim measures to track program performance.

Robust research and reliable performance data are needed to obtain a complete picture of the WIA youth program’s effectiveness and outcomes and to make quality decisions about managing the program. However, none currently exist. Labor has not yet initiated an impact evaluation and does not expect to report on program impacts from a planned evaluation until 2009. In addition, states and local areas continue to struggle to collect and document accurate and complete participant and performance data and maintain data systems that can yield reasonably reliable outcome information. For its part, Labor is taking action to improve data integrity by requiring states to validate WIA performance data, beginning with data from program year 2002. However, Labor’s inconsistent monitoring processes across regions will continue to challenge Labor’s capability to ensure that all states validate and report data consistently and effectively. In the short term, the lack of accurate outcome data will continue to hinder Labor’s ability to negotiate realistic state performance goals and use data to make sound decisions about financial incentives or sanctions. In the long term, lack of accurate and complete information will keep Labor from obtaining a true picture of how effectively the youth program is working.

Recommendations for Executive Action

To promote information sharing that improves local WIA youth programs’ ability to identify and serve out-of-school youth, we recommend that the Secretaries of Labor and Education coordinate efforts to clarify how schools can work with workforce officials to connect school dropouts with local WIA youth programs.

To assist state and local WIA youth programs address ongoing implementation challenges, we recommend that the Secretary of Labor increase availability of guidance and technical assistance to local areas that continue to face challenges in serving out-of-school youth; disseminate guidance, including specific strategies, to help local areas provide effective mentoring services; and develop additional guidance on providing follow-up services and using interim measures to track program performance.

To obtain an accurate and complete gauge of WIA outcomes and determine whether local programs are operating successfully, we recommend that the Secretary of Labor establish standard monitoring procedures that Labor’s regional offices could use to oversee state data validation efforts.

Agency Comments and Our Evaluation

We provided a draft of this report to Labor and Education officials for their review and comment. Education’s comments are reprinted in appendix I. In its formal comments, Education concurred with our recommendation to work with Labor to connect out-of-school youth to local WIA youth programs. Labor responded informally, and said that many of the findings corroborated its own observations and that the recommendations are consistent with Labor’s current program direction. Both agencies also provided technical comments, which we incorporated as appropriate.

We will send copies of this report to the Secretaries of Labor and Education, relevant congressional committees, and other interested parties. Copies will be made available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

Please contact me at (415) 904-2272 if you or your staff have any questions about this report. Other major contributors to this report are listed in appendix II.

Appendix I: Comments from the Department of Education

Appendix II: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the individuals mentioned above, the following staff made key contributions to this report: Karyn Angulo, Susan Baker, Andrew Bauck, Paula Bonin, Maya Chakko, Keira Dembowski, Joel Grossman, Corinna Nicolaou, Rebecca Woiwode, and Monica Wolford.

Related GAO Products

Transportation-Disadvantaged Populations: Some Coordination Efforts Among Programs Providing Transportation Services, but Obstacles Persist. GAO-03-697. Washington, D.C.: June 30, 2003.

Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing is Needed. GAO-03-725. Washington D.C.: June 18, 2003.

Workforce Investment Act: Issues Related to Allocation Formulas for Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.: April 25, 2003.

Workforce Investment Act: States’ Spending Is on Track, but Better Guidance Would Improve Financial Reporting. GAO-03-239. Washington, D.C.: November 22, 2002.

Workforce Investment Act: Youth Provisions Promote New Service Strategies, but Additional Guidance Would Enhance Program Development. GAO-02-413. Washington, D.C.: April 5, 2002.

Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002.

School Dropouts: Education Could Play a Stronger Role in Identifying and Disseminating Promising Prevention Strategies. GAO-02-240. Washington, D.C.: February 1, 2002.