Employment and Training Program Grants: Evaluating Impact and Enhancing Monitoring Would Improve Accountability
Published: 2008-05-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2001, Labor has spent nearly $900 million on three workforce employment and training grant initiatives: High Growth Job Training Initiative (High Growth), Community-Based Job Training Initiative (Community Based), and the Workforce Innovation in Regional Economic Development (WIRED). GAO was asked to examine (1) the intent of the grant initiatives and the extent to which Labor will be able to assess their effects, (2) the extent to which the process used competition, was adequately documented, and included key players, and (3) what Labor is doing to monitor individual grantee compliance with grant requirements. To answer these questions, GAO obtained from Labor a list of grants for fiscal years 2001 through 2007, and reviewed relevant laws and Labor's internal grant award procedures. GAO interviewed grantees, and state and local workforce officials in seven states where grantees were located, Labor officials, and subject matter experts.
section_title: What GAO Found
paragraphs: According to Labor officials, the grant initiatives were designed to shift the focus of the public workforce system toward the training and employment needs of high-growth, in-demand industries, but Labor will be challenged to assess their impact. Under the initiatives, Labor awarded 349 grants totaling almost $900 million to foster this change. However, the grant initiatives were not fully integrated into Labor's strategic plan or overall research agenda; therefore, it is unclear what criteria Labor will use to evaluate their effectiveness. Labor lacks data that will allow it to compare outcomes for grant-funded services to those of other federally funded employment and training services. While grants under all three initiatives are now awarded competitively, the initial noncompetitive process for High Growth grants was not adequately documented and did not include key players. Community Based and WIRED grants have always been awarded competitively, but more than 80 percent of High Growth grants were awarded without competition. Labor began awarding some High Growth grants competitively in 2005 and Congress required Labor to award certain grants competitively in fiscal years 2007 and 2008. This requirement applies only to those years. Labor could not document criteria used to select the noncompetitive High Growth grants or whether these grants met internal or statutory requirements. Labor has taken steps to strengthen the noncompetitive process, but these procedures do not explicitly require documentation of compliance with statutory program requirements. Labor's process for identifying solutions for industry workforce challenges did not include the vast majority of local workforce investment boards, which oversee local employment and training services. Labor provides some monitoring for grantees under all three initiatives and uses a risk-based monitoring approach for the High Growth and Community Based grants, but not for WIRED. Labor has a process to address findings from single audit reports, but its inspector general found problems with Labor's follow up on the completion of these audits for its grantees in general and recommended that Labor put procedures in place to do so. Labor provides technical assistance for the three initiatives and spent $16 million on contracts to help offer this support.

section_title: Letter
paragraphs: Federally funded training and employment services are delivered through  Federally funded training and employment services are delivered through  what is known as the one-stop system, which was developed under the  what is known as the one-stop system, which was developed under the  Workforce Investment Act (WIA) of 1998 and is governed by state and  Workforce Investment Act (WIA) of 1998 and is governed by state and  local workforce investment boards. Sixteen categories of programs,  local workforce investment boards. Sixteen categories of programs,  funded by four federal agencies, deliver their services through this system.  funded by four federal agencies, deliver their services through this system.  In large part, program performance is measured by collecting standard  In large part, program performance is measured by collecting standard  measures on the outcomes of individuals who use these services. Under  measures on the outcomes of individuals who use these services. Under  WIA, the Department of Labor (Labor) has general responsibility and  WIA, the Department of Labor (Labor) has general responsibility and  oversight of the one-stop system.  oversight of the one-stop system.
Since 2001, Labor has spent almost $900 million on three employment and  Since 2001, Labor has spent almost $900 million on three employment and  training grant initiatives to address what Labor perceived as shortcomings  training grant initiatives to address what Labor perceived as shortcomings  in the one-stop service delivery system: the High Growth Job Training  in the one-stop service delivery system: the High Growth Job Training  Initiative (High Growth) beginning in 2001, the Community Based Job  Initiative (High Growth) beginning in 2001, the Community Based Job  Training Initiative (Community Based) beginning in 2005, and the  Training Initiative (Community Based) beginning in 2005, and the  Workforce Innovation in Regional Economic Development (WIRED)  Workforce Innovation in Regional Economic Development (WIRED)  initiative beginning in 2006. In 2005 Congress expressed interest in how  Labor was awarding the High Growth grants and instituted a number of  reporting requirements, including a justification for sole source awards.
Given the size of the investment and the interest in noncompetitive  awards, we were asked to examine: (1) the intent of the grant initiatives  and the extent to which Labor will be able to assess their effects; (2) the  extent to which the process used competition, was adequately  documented, and included key players; and (3) what Labor is doing to  monitor individual grantee compliance with grant requirements.
To determine the intent of the grant initiatives and the extent to which  Labor will be able to assess their effects, we reviewed grant applications,  grantee quarterly reports, Labor’s strategic plan, and documents related to  evaluations of the initiatives and their purpose. We obtained a list of all  High Growth, Community Based, and WIRED grants from Labor for fiscal  year 2001, the first year High Growth grants were awarded, through fiscal  year 2007. We assessed the reliability of these data by (1) reviewing the  data for obvious errors and completeness, (2) reviewing related  documentation, and (3) interviewing agency officials knowledgeable about  the data. We determined that the data were sufficiently reliable for the  purposes of this report. We also interviewed relevant Labor officials and  persons with recognized workforce and training expertise. To determine  the extent to which the process used competition, was adequately  documented, and included key players, we reviewed relevant laws, Labor’s  internal procedures for awarding grants, and Labor’s inspector general’s  report on High Growth grants. Out of a universe of about 340 grantees  provided by Labor at the time of our review, we selected 8 for in-depth  study. These were selected because each grant exceeded $1 million, and  they represented the range of grant types. These grantees were located in  seven states: Colorado, Florida, Illinois, New Hampshire, Pennsylvania,  New York, and Washington. For each grantee, we conducted semi- structured interviews and reviewed each grantee’s application and recent  quarterly reports. We also interviewed state and local workforce officials  in these states. To address what Labor is doing to monitor the  performance of grantees, we reviewed Labor’s monitoring procedures, six  grantee monitoring reports provided by Labor as examples, and the single  audit of a grantee we contacted. We also reviewed a list of High Growth  and Community Based grants that had been monitored by Labor. See  appendix I for a more detailed discussion of the scope and methodology.  We conducted our work from May 2007 to May 2008 in accordance with  generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: When it was enacted in 1998, WIA created a new, comprehensive  workforce investment system designed to change the way employment  and training services are delivered. Under WIA, each state establishes a  state workforce investment board that, among other duties, determines  strategic priorities, current and projected employment opportunities, and  job skills necessary to obtain such employment.  It also designates local  workforce investment areas across the state and develops an allocation  formula for distributing funds to those local areas. Each local area is  governed by local workforce investment boards that make decisions about  the number and location of one-stop career centers, where partner  programs make their services and activities available. Local boards are  required to promote employers’ participation in the workforce investment  system and assist them in meeting hiring needs. Training services provided  must be directly linked to occupations in demand in the local area. To  further the involvement of employers, WIA requires that more than half the  members of each state and local workforce investment board be  representatives of business. The purpose of the one-stop system is to  increase the employment, retention, and earnings of job seekers and, by  increasing their occupational skills, enhance the productivity and  competitiveness of the national economy. WIA requires states and  localities to track the performance of WIA-funded activities and Labor to  hold states accountable for their performance in the areas of job  placement, employment retention, and earnings change.
The Employment and Training Administration (ETA) oversees the High  Growth, Community Based, and Wired grant initiatives. The vast majority  of these grants are awarded under a provision of WIA, which provides  authority for demonstration, pilot, multi-service, research, and multi-state  projects, and a provision of the American Competitiveness and Workforce  Improvement Act (ACWIA), which provides authority for job training  grants funded by the H-1B visa program.
Labor is required to conduct impact evaluations of its programs and  activities carried out under WIA, including pilot and demonstration  projects. While impact evaluations make it possible to isolate a program’s  effect on participants’ outcomes, there are several ways to conduct them,  including experimental and quasi-experimental methods. In 2004 and  2007, GAO recommended that Labor comply with WIA requirements to  conduct an impact evaluation of WIA services to determine what services  are most effective for improving employment-related outcomes. Labor  agreed with our recommendation. In December 2007, the agency  announced it had begun a quasi-experimental evaluation⎯an impact  evaluation that does not use a control group—of the WIA Adult and  Dislocated Worker programs, with a final report expected in November  2008.
Federal law recommends, but does not require that all grants be awarded  through competition. The Federal Grant and Cooperative Agreement Act  encourages competition in grant programs, where appropriate, to ensure  that the best possible projects are funded. In addition, Labor’s own  guidance governing procurement and grant operations—the Department  of Labor Manual Series—states that competition is recommended, unless  one or more of eight exceptions apply. Further, a guide on improving  grant accountability developed by the Domestic Working Group Grant  Accountability Project recommends grants be awarded competitively  because competition facilitates accountability, promotes fairness and  openness, and increases assurance that grantees have the systems in place  to efficiently and effectively use funds to meet grant goals.
Effective monitoring is also a critical component of grant management.  The Domestic Working Group’s suggested grant practices states that  financial and performance monitoring is important to ensure  accountability and the attainment of performance goals. Labor monitors  most grants through a risk-based strategy based on its “Core Monitoring  Guide.” A key goal is to determine compliance with specific program  requirements. The guide includes five monitoring areas: program design  and governance, program and grant management systems, financial  management systems, service, and product delivery, and performance  accountability. In addition, entities receiving Labor grants are subject to  the provisions of the Single Audit Act if certain conditions are met.  The  act established the concept of the single audit to replace multiple grant  audits with one audit of a recipient as a whole. As such, a single audit is an  organization wide audit that covers, among other things, the recipient’s  internal controls and its compliance with applicable provisions of laws,  regulations, contracts, and grants.
section_title: Grants Are Intended to Change the Workforce System, but Labor Will Be Challenged to Evaluate Their Impact
paragraphs: According to Labor officials, the grant initiatives are designed to change  the focus of the public workforce system to emphasize the employment  and training needs of high-growth, high-demand industries, but Labor will  be challenged in assessing their impact. For the three grant initiatives,  Labor awarded 349 grants totaling almost $900 million that were intended  to bring about this change by identifying the workforce and training needs  of growing, high-demand industries; engaging workforce, industry, and  educational partners to develop innovative solutions to workforce  challenges, such as worker shortages; leveraging a wide array of resources  to fund the solutions; and integrating workforce and economic  development to transform regional economies by creating good jobs.  However, 7 years after awarding the first grant, Labor will be challenged to  evaluate the effect of the grants.
subsections: section_title: Labor Said the Grants Are Designed to Make the Workforce System More Focused on High-Growth, High-Demand Industries
paragraphs: According to Labor officials, the High Growth, Community Based, and  WIRED initiatives are designed to collectively change the focus of the  workforce investment system by giving greater emphasis to the  employment and training needs of high-growth, high-demand industries.  They characterized High Growth as a systematic change initiative designed  to make the system more demand-driven (i.e., focused on the needs of  growing and high-demand industries) and to make the system’s approach  to workforce development more strategic by engaging business, industry,  and education partners to identify workforce challenges and solutions.   The Computing Technology Industry Association provides one example of  the kinds of activities funded under these grants. It used its High Growth  grant to expand an apprenticeship program in a growing industry— information technology. As a related effort, the Community Based grants  were designed to build the training capacity of community colleges for  high-growth, high-demand occupations. In expanding existing training  programs and developing new programs aimed at addressing worker  shortages in three high-demand health care occupations—nursing, dental  hygiene, and health unit coordinators, the Seattle Central Community  College in Washington provides an example of efforts funded under these  grants. The goal of third grant initiative, WIRED, was to “catalyze” the  creation of high-skill and high-wage opportunities for workers within the  context of regional economies, to test models for integrating workforce  and economic development, and to demonstrate that workforce  development is a key driver in transforming regional economies. The  northeast region of Pennsylvania provides one example of the types of  projects funded through WIRED. Pennsylvania is using WIRED grant funds  to develop Wall Street West, a nine county regional project intended to  provide backup and disaster recovery operations and facilities for the New  York City financial markets.  According to Labor officials, these funds  support workforce education and training to support this effort. From  2001 through 2007, Labor awarded 349 grants totaling almost $900 million  for these initiatives (see table 1). For a list of these grants, see appendix II.
While Labor said the three grant initiatives were necessary to shift the  focus of the workforce investment system, experts held varying opinions  on the extent to which state and local workforce boards were pursuing  demand-driven strategies on their own. Of the experts we spoke with, one  said that many states were already focusing on high-growth sectors,  reaching out to industry, and developing strategic partnerships. Another  characterized innovation among workforce boards as a normal curve that  ranged from highly functional to less than functional. One said that some  boards had always been pursuing these types of strategies. Another said  that the system had been lacking in this regard.
In addition, when considering how extensive demand-driven practices  might have been, experts identified a number of systematic factors that  might explain why workforce boards would face challenges in pursuing  them. These included insufficient funding, limited flexibility in how funds  can be used, statutory requirements to target services to certain groups of  workers, and the need to respond to local economic conditions.  Commenting on workforce boards’ ability to form strategic partnerships,  one expert noted that there are no funds to support such endeavors and no  performance standards to measure them. With regard to regional  economic development, experts said boards are structured around local  areas, not regions, regional economies are highly variable, regional  governance structures can make achieving buy-in difficult and that rural  areas can be particularly challenged in pursuing regional approaches.
Beyond the question of how extensive demand-driven practices were,  Labor officials said a number of indicators show that the initiatives are  changing the system. According to Labor, they have seen a “system-shift”  in the approach to implementing workforce solutions through an increase  in demand-driven topics at conferences since the roll out of the initiatives.  Labor said this shift has been driven by partnerships between the  workforce investment system, business, industry, and educators using the  High Growth framework. Labor also said it is seeing demand-driven  strategies in state and local strategic plans and in states using their own  money to fund High Growth-like projects. Labor pointed out that the  system has evolved to the point where high performing local workforce  boards with demand-driven practices are mentoring lesser performers.  Lastly, Labor said the content on its Web site, Workforce3 One, was also  evidence of change. For example, Labor held an interactive seminar  broadcast on this site to train participants to use an on-line tool to share  curricula developed through the initiatives.
section_title: Seven Years after Awarding the First Grant, Labor Will Be Challenged to Evaluate Their Impact
paragraphs: Despite the money invested and emphasis placed on these initiatives,  Labor did not fully integrate them into its strategic plan or ETA’s research  plan from the start. The Government Performance and Results Act states  that strategic plans shall contain strategic goals and objectives, including  outcome-related, or performance goals, and objectives for an agency’s  major functions and operations.  However, the strategic plan includes  performance goals only for the Community Based initiative. High Growth  and WIRED—the two initiatives where Labor spent the most money⎯are  mentioned in the strategic plan, but not specifically linked to a  performance goal; therefore, it is unclear what criteria Labor will use to  evaluate their effectiveness. Moreover, the data needed to assess the  performance of these initiatives are not specified. Labor officials said the  strategic plan did not address the initiatives because it focuses on budget  issues. Just as the initiatives are not fully integrated into the strategic plan,  neither are they fully integrated into ETA’s research plan, which cites  plans for future evaluations, but it does not specify an assessment of their  impact.
Not fully incorporating the initiatives into its strategic or research plans  may have limited Labor’s ability to collect consistent outcome data. Labor  said that prior to 2005, it consistently collected data from grantees on the  number of participants enrolled in and completing training funded under  High Growth—the only one of the three grant initiatives operating at that  time. However, it did not collect performance outcomes similar to those  being collected for its other training and employment services. In 2005,  Labor instituted what were called common measures to assess the  effectiveness of one-stop programs and services. The common measures  include participant employment outcomes, earnings, and job retention  after receiving services. Currently, Labor cannot require High Growth and  Community Based grantees to provide data on the common measures until  it receives OMB approval. According to Labor, it can collect such  measures for WIRED grantees, but it has not yet done so. As a result,  Labor may not have consistent data for individuals participating in the  programs funded under the grant initiatives. In addition, it may lack data  that will allow it to compare outcomes for individuals served by grant- funded programs with those served by employment and training programs  offered through the one-stop system. Having comparable outcome data is  important because the goal of an impact evaluation is to determine if  outcomes are attributable to a program or can be explained by other  factors.
Labor will face challenges in obtaining the data necessary to make  meaningful comparisons. Starting in 2006, Labor included information on  the common measures in all new solicitations for High Growth and  Community Based grants, notified grantees of its goal for standardizing  performance reporting, and provided technical assistance to help grantees  prepare for it. Labor also encouraged grantees to work with local  workforce system partners to leverage their experience in tracking and  reporting performance outcomes. Labor anticipates having an OMB  approved reporting format in place by July 1, 2008.  However, because  some of the first grantees have already completed their projects, obtaining  information about workers that have left the program may prove difficult  and costly.
Labor has some plans underway to evaluate the initiatives, but may face  challenges drawing strong conclusions from them. Labor has conducted  an evaluation of the implementation and sustainability of 20 early High  Growth grantees. It is now evaluating the impact of the training provided  by High Growth grantees. Labor anticipates the final report in December  2008. However, Labor has already experienced a number of challenges in  evaluating the initiatives. These include having to limit its evaluation to  only 6 grantees of 166, because only 6 had sufficient participants to ensure  a statistically significant evaluation.  They also include problems gaining  access to workers’ earnings data, and inconsistent outcome data from  grantees.
Labor officials said they plan to conduct a comprehensive evaluation of  the Community Based initiative. The first phase of the evaluation will  examine the extent to which the Community Based grants addressed the  stated workforce objectives and challenges funded projects were intended  to address, as well as document the role of business and the workforce  investment system in the overall success of the grants, according to Labor.  This phase will also include an examination of the feasibility of performing  an impact evaluation and will be completed in late 2008.  Depending on the  results of this phase, Labor officials said an impact evaluation will begin in  2009.
For its evaluation of the WIRED initiatives, Labor says it is examining the  implementation and cumulative effects of WIRED strategies, including  change in the number and size of companies in targeted high-growth  industries and whether new training led to job placement in the targeted  industries. It contracted with the Berkeley Policy Associates to conduct  the evaluation for the first 13 grantees, and a final report is expected by  June 2010.  It also contracted with Public Policy Associates to similarly  evaluate the 28 remaining WIRED grantees.  Labor officials said these  initiatives are not included in the agency’s broader WIA impact study.
section_title: The Initial Noncompetitive Process Was Not Adequately Documented and Did Not Include Key Players
paragraphs: While Labor now awards grants under all three grant initiatives  competitively, initially almost all High Growth grants were awarded  without competition. Labor also did not document the criteria for selecting  noncompetitive High Growth grants or whether they met Labor’s internal  requirements or the requirements of the laws under which the grants are  authorized. In addition, Labor’s process for identifying workforce  solutions did not include most of the state and local workforce investment  boards.
subsections: section_title: All Three Types of Grants Are Now Awarded Competitively, but the Vast Majority of High Growth Grants Were Awarded Without Competition
paragraphs: The Community Based and the WIRED grants have always been awarded  through a competitive process, but until 2005, Labor did not award High  Growth grants competitively even though federal law and Labor’s internal  procedures recommend competition. While Labor had discretion in  awarding High Growth grants without competition, the extent to which it  did so raises questions about how Labor used this method of awarding  grants. Competition facilitates accountability, promotes fairness and  openness, and increases assurance that grantees have systems in place to  meet grant goals. Yet Labor chose to award 83 percent of the High Growth  grants, which represented almost 90 percent of the funds, without  competition between fiscal years 2001 and 2007 (see table 2). Congress  required that High Growth grants funded by H-1B fees be awarded  competitively for fiscal years 2007 and 2008. Prior to that time, there were  no provisions requiring Labor to award High Growth grants competitively.
Labor said that it used a noncompetitive process to promote innovation.  They also said that they awarded grants without competition to save the  time it would have taken to solicit grants through competition. In  hindsight, they said they could have offered the High Growth grants  competitively earlier because they recognized that the number of  noncompetitive awards created a perception that the process was unfair.  They said, however, that they always intended to award later grants  competitively.
In contrast to the High Growth grants, the Community Based and WIRED  initiatives have always been awarded through competition. These funding  opportunities were announced to potential applicants through a  solicitation for grant application that listed the information that an  application must include to compete for funding. These applications were  then reviewed and scored by a knowledgeable technical panel. These  solicitations were also reviewed by Labor attorneys for compliance with  procurement and statutory program requirements for awarding grants,  according to officials.
section_title: Labor Did Not Document the Criteria for Selecting Noncompetitive High Growth Grants or Whether They Met Labor’s Internal Requirements or Requirements of the Law
paragraphs: Because the initial High Growth process was noncompetitive,  documenting the decision steps was all the more important to ensure  transparency. However, Labor was unable to provide documentation of  the initial criteria for selecting grantees. As a result, it did not meet federal  internal control standards, which state that all transactions and other  significant events need to be clearly documented and that the  documentation should be readily available for examination. In addition, it  was unable to document that it met the statutory requirements for the laws  authorizing the grants. Lastly, according to Labor’s inspector general, it did  not adequately document that it had followed its own procedures for  awarding grants without competition.
Labor did not document the criteria used to select the early  noncompetitive High Growth projects. Labor officials told us there were  no official published guidelines specific to High Growth grants, only draft  guidelines, which were no longer available. In addition, Labor officials told  us that generally they were looking for grantees that pursued partnerships  and leveraged resources, but that attributes they sought changed over  time. Labor published general requirements for noncompetitive grants in  2005 and updated them in 2007. Officials said these were not requirements,  only guidelines for the kinds of information Labor would find valuable in  evaluating proposals.
In addition, while Labor said noncompetitive grants could be awarded  without competition under the WIA provision authorizing demonstrations  and pilot projects and under ACWIA, they could not document that the  grants fully complied with the requirements of these provisions. For  example, WIA requirements include providing direct services to  individuals, including an evaluative component, and being awarded to  private entities with recognized expertise or to state and local entities with  expertise in operating or overseeing workforce investment programs.  Officials said that they were certain they had ensured that the projects met  all statutory requirements, but acknowledged they did not document that  the requirements were met.
Labor’s inspector general found the agency did not always document that  it followed its own procedures or always obtained required review and  approval before awarding grants noncompetitively. Labor officials said  most of the noncompetitive grant proposals were presented to Labor’s  Procurement Review Board for review and approval allowed under  exceptions for proposals that were unique or innovative, highly cost- effective, or available from only one source. However, in 2007, Labor’s  inspector general reviewed a sample of the noncompetitive High Growth  grants awarded between July 2001 and March 2007 and found that 6 of the  26 grants, which should have undergone review, were awarded without  prior approval from the review board. Furthermore, they found that Labor  could not demonstrate that proper procedures were followed in awarding  the High Growth grants without competition.
Although they were unable to provide documentation, Labor officials said  they used considerable rigor in selecting grant recipients under the  noncompetitive process. Similar to a competitive process, the  noncompetitive grant proposals were highly scrutinized and reviewed to  ensure they made best use of scarce resources. They said that in most  cases, staff created abstracts to highlight strengths and weaknesses and  multiple staff and managers participated in reviews and decision-making.   In addition, Labor officials strongly disagreed with the majority of the  inspector general’s findings. They said they followed established  procurement practices as required but agreed that additional  documentation would be valuable.
In response to the inspector general’s report, Labor has taken steps to  strengthen the noncompetitive process.  These include developing  procedures to review noncompetitive grant proposals for criteria including  support of at least one of ETA’s strategic goals and investment priorities.  The procedures also require ETA to document that required procedures  are followed and that required review and approval is obtained before  awarding grants noncompetitively. However, the newly developed  procedures do not explicitly identify the statutory program requirements  for which compliance should be documented.
section_title: Labor’s Process for Identifying Industry Workforce Challenges Did Not Include the Majority of Workforce Investment Boards
paragraphs: The vast majority of workforce boards—which oversee the workforce  investment system—were not included in the meetings that served as  incubators for grant proposals. After identifying 13 high-growth/high- demand sectors, Labor held a series of meetings between 2002 and 2005  with industry executives and other stakeholders to identify workforce  challenges and to develop solutions to them. According to Labor, they  first held meetings with industry executives—executive forums—for 13  sectors to hear directly from industry leaders about the growth potential  for their industries and to understand the workforce challenges they faced.  Second, they hosted a series of workforce solutions forums for 11 of the  sectors, which brought together industry executives (often those engaged  in human resources and training activities) with representatives from  education, state and local workforce boards, or other workforce-related  agencies. However, a review of Labor’s rosters for the solutions forums  shows that while there were more than 800 participants, 26 of the almost  650 local workforce boards nationwide were represented, and these came  from 15 states. (See fig. 1.)
Further only 20 of the 50 states had their state workforce investment  board or other agency represented (see table 3).
Labor officials said they went to great lengths to include workforce system  participants in solutions forums. Officials said they asked state workforce  agencies to identify a state coordinator to interface with Labor, work  collaboratively with industry partners, and identify potential attendees for  executive and solutions forums. Further, the state coordinators were to  help Labor communicate with the workforce system about High Growth  activities and were kept updated through routine conference calls and  periodic in-person meetings, according to Labor. Labor officials also said  the Assistant Secretary and other senior officials traveled frequently,  speaking to workforce system partners at conferences to gather  information about innovative practices. Labor officials said, even with  these efforts, they found only a few workforce boards operating unique or  innovative demand-driven programs.
However, most workforce board officials we spoke to in our site visits  reported becoming aware of the meetings and the grant opportunities after  the fact, even though they were pursuing the kinds of innovative practices  the meeting was supposed to promote. Some state board officials said that  they were often unaware that grants had been awarded, and at least one  local workforce board said it became aware of a grant only when the  community college grantee approached it for assistance in getting enough  students for their program. In addition, officials in states we visited said  they had been developing and using the types of practices that Labor was  seeking to promote at the meetings.
Being present at the meetings could have been beneficial to workforce  boards. Labor officials acknowledged that when meeting participants  suggested a solution to an employment challenge that they deemed  innovative and had merit, they encouraged them to submit a proposal for a  grant to model the solution. In addition, officials said that in some cases,  they provided applicants additional assistance to increase the chances that  the proposal would be funded.
section_title: Labor Uses a Risk- Based Monitoring Approach for High Growth and Community Based Grants but Not for WIRED Grants
paragraphs: For all three grant initiatives, Labor has a process to resolve findings  found in single audits, collects quarterly performance information, and  provides technical assistance as a part of monitoring. In addition, it has a  risk-based monitoring approach for High Growth and Community Based  grants, but has not implemented such an approach for WIRED grants.  However, Labor officials told us that they are developing a risk-based  approach for monitoring WIRED grants that they plan to implement in the  spring of 2008. According to Labor’s grants monitoring guide, improving  grant administration is crucial to Labor’s management improvement plans  and risk-based on-site monitoring of grantees is an important part of this  effort.
subsections: section_title: Labor Has a Process to Ensure Grantees Resolve Findings in Single Audits, Collects Quarterly Performance Information, and Provides Technical Assistance
paragraphs: Labor said it has a process to work with grantees, including High Growth,  Community Based, and WIRED to resolve findings in single audits.  However, Labor’s inspector general reported that Labor does not have  procedures in place for grant officers to follow up with grantees with past  due audit reports to ensure timely submission and thus proper oversight  and correction of audit findings. The inspector general recommended that  Labor implement such procedures and Labor has done so, but the finding  remains open because Labor’s inspector general has not yet determined if  the procedures adequately address the recommendation.
As part of its monitoring, Labor requires High Growth, Community Based,  and WIRED grantees to submit quarterly financial and performance  reports. Financial reports contain information, such as total amount of  grant funds spent and amount of matching funds provided by the grantee.  Performance reports focus on activities leading to performance goals,  such as grantee accomplishments and challenges to meeting grant goals.  Labor officials said they review these reports and follow up with grantees  if there are questions.  Labor officials acknowledge, however, that they are  still working to ensure the consistency of performance reports provided  by High Growth and Community Based grantees and are working with  OMB to establish consistent reporting requirements. In addition, while the  finding was not specific to these three grants, Labor’s inspector general  cited high error rates in grantee performance data as a management  challenge. Labor is taking steps to improve grant accountability, such as  providing grantee and grant officer training.
All grantees receive technical assistance from Labor on how to comply  with laws and regulations, program guidance, and grant conditions. For  example, Labor issued guides for High Growth and Community Based  grantees which include information on allowable costs and reporting  requirements. In addition, Labor officials said they trained national and  regional office staff to address grantees’ questions and help High Growth  and Community Based grantees obtain assistance from experts at Labor  and other grantees. Labor officials said they hold national and regional  High Growth and Community Based grantee orientation sessions for new  grantees, present technical assistance webinars and training sessions  focused on specific high-growth industries, assist grantees with  disseminating grant results and products, such as curricula, and set up  virtual networking groups of High Growth grantees to encourage  collaboration.
Labor officials told us they have teams who provide technical assistance to  each WIRED grantee including weekly contact. During these sessions,  Labor staff work with WIRED grantees on grant management issues, such  as costs that are allowed using grant funds. Labor staff provide additional  assistance through conference calls, site visits, and documentation  reviews. In addition, Labor officials said they have held five webinars on  allowable costs and provided grantees with a paper on allowable costs in  July 2006, which was updated in July 2007. Finally, Labor officials  explained that they made annual site visits for the first 13 WIRED grantees  in spring and summer of 2007 to discuss implementation plans and  progress toward plan goals. In addition, Labor staff said they have  reviewed the implementation of the remaining WIRED grants to ensure  that planned activities comply with requirements of the law. However,  none of these reviews resulted in written reports with findings and  corrective action plans.
Labor has spent $16 million on contracts to provide technical assistance,  improve grant management, administration, and monitoring, and to assist  Labor with tasks such as holding grantee training conferences. The larger  of these contracts focus on providing technical assistance to WIRED  grantees. For example, one contract valued at over $2 million provides  WIRED grantees assistance with assessing regional strength and  weaknesses and developing regional economic strategies and  implementation plans. Another grant, valued at almost $4 million, provides  a database and geographic information system that WIRED grantees can  use to facilitate data analysis and reporting, among other things. For a list  of these contracts, see appendix III.
While these monitoring and technical assistance efforts are useful to help  grantees manage their grants, they do not provide a risk-based monitoring  process to identify and resolve problems, such as compliance issues, in a  consistent and timely manner.
section_title: Labor Provides Risk-Based Monitoring for High Growth and Community Based Grants
paragraphs: Labor uses a risk-based strategy to monitor grants under two of the three  grant initiatives: High Growth and Community Based.  For these initiatives  it selects grantees to monitor based on indications of problems that may  affect grant performance. Labor’s risk-based approach to monitoring most  grants reflects suggested grant practices. Suggested grant practices  recognize that it is important to identify, prioritize, and manage potential  at-risk grant recipients for monitoring given the large number of grants  awarded by federal agencies. Through this process, Labor staff determine  if grantee administration and program delivery systems operate, the  grantee is in compliance with program requirements, and information  reported is accurate.
Labor’s risk-based monitoring strategy involves conducting site visits  based on grantees’ assessed risk-levels and availability of resources,  among other things. These site visits include written assessments of  grantee’s management and performance and compliance findings and  requirements for corrective action. For example, Labor’s site visit guide  includes questions about financial and performance data reporting  systems, such as how well the grantee maintains files on program  participants.
Labor has monitored about half of the High Growth grants and over one- quarter of the Community Based grants. Labor officials said these  monitoring efforts have resulted in a number of significant findings which  have generally been resolved in a timely manner. (See table 4.) For  example, during a November 2006 site visit of a Community Based grantee  Labor identified three findings: incomplete participant files, failure to  follow internal procurement procedures, and missing grant partnership  agreements. Similarly, during a site visit in spring 2006 to a High Growth  grantee, Labor found that the grantee did not accurately track participant  information and reported incorrect information on expenditures, among  other things. As of September 2007 Labor said these findings had been  resolved (see table 4).
As another part of Labor’s risk-based monitoring strategy, Labor’s internal  requirements specify that Labor staff are to make site visits to all new  grantees, including High Growth, Community Based, and WIRED, within  12 months of beginning grant activity and to new grantees rated as “at  risk” within 3 months. Labor officials said they consider “new grantee” site  visits to be orientation visits and had not made visits to most new  grantees.   They said they broadly interpret this requirement to include a  variety of methods of contact and generally use teleconference and video  conference training sessions rather than site visits, based on the  availability of resources. For example, Labor calls each new Community  Based grantee to schedule new grantee training. Labor is taking steps to  update its internal requirements to better reflect the purpose of the new  grantee monitoring.
section_title: Labor Is Planning to Develop a Risk-Based Approach for Monitoring WIRED Grants, but It Has Not Yet Done So
paragraphs: Labor officials said they are developing a WIRED grant assessment tool,  but it is not finalized. They plan to implement it in spring 2008; however,  the department has not developed a schedule for its use. Labor officials  explained that a specific tool is needed for monitoring WIRED grants  because the site visit guide used for most other Labor grants would not be  appropriate for WIRED grants. One reason is that WIRED grants are  awarded to states but are generally administered by other entities. For  example, Labor awarded a $15 million WIRED grant to the state of  Colorado, but programs funded by the grant are primarily administered by  the not-for-profit Metro Denver Economic Development Corporation.
section_title: Conclusions
paragraphs: Seven years after awarding the first grants, Labor is in the process of  evaluating the High Growth and WIRED initiatives and has plans to  evaluate the Community Based initiative but will be challenged to evaluate  the effect of the almost $900 million spent on the grant initiatives. Because  of poor planning, the agency will not be able to perform the kind of  comprehensive impact evaluation that would have been possible if it had  been collecting data consistent with that collected for other WIA programs  all along. Because Labor lacks such data it will be challenged in drawing  strong conclusions from its planned evaluations. Until Labor collects  consistent outcome data, such as the common measures, that would allow  comparison of impact of each of the three grant initiatives to those of  other Labor programs, Labor cannot know the extent to which the  initiatives improve employment outcomes, if at all.
The vast majority of High Growth grants—more than $263 million— were  awarded noncompetitively using inconsistent selection criteria under a  process that was not adequately documented. As a result there is little  assurance that the grants selected were the best possible projects.  Although Labor currently awards grants under all three grant initiatives  competitively, the fact that Labor used inconsistent selection criteria that  changed over time and did not adequately document the process when  awarding High Growth grants noncompetitively raises questions about  how Labor uses this method of awarding grants. This is a critical concern  given that Labor could again award millions of dollars of noncompetitive  grants. While it has taken steps to strengthen the noncompetitive process,  the newly developed procedures do not explicitly identify the statutory  program requirements for which compliance should be documented.
Finally, the absence of a risk-based monitoring approach for WIRED puts  Labor at risk of not knowing if the millions of dollars awarded to WIRED  grantees have been used for the purposes they are intended.
section_title: Recommendations for Executive Action
paragraphs: To determine the impact of the three grant initiatives, ensure that the best  possible projects are selected, and improve accountability of grant funds,  we recommend that the Secretary of Labor take the following three  actions:    Take steps to ensure that the department can evaluate the impact of the  initiatives so that it can draw strong conclusions based on its evaluations,  such as following through with plans to collect consistent data, integrating  the initiatives into its overall research agenda with relevant performance  goals and indicators, and including these initiatives in its assessment of the  impact of WIA services.
Direct the Employment and Training Administration to identify the  statutory program requirements for which compliance must be  documented when awarding noncompetitive grants.
Develop and implement a risk-based monitoring approach for WIRED and  a schedule for its use.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of the report to Labor for review and comment. In  response to our first recommendation, Labor said it has taken initial steps  to evaluate the impact of the initiatives and disagreed with our assertion  that the initiatives are not integrated into its strategic plan.  We agree that  the agency has taken initial steps, but these do not ensure that they will be  able to assess the initiatives’ impact.  We do not dispute that the strategic  plan mentions the initiatives; however, we found that both the strategic  and research plans do not tie the High Growth or WIRED initiatives to  performance goals, making it unclear what criteria Labor will use to  evaluate their effectiveness.  In response to the first recommendation,  Labor also disputed a claim that we did not make, namely that the  initiatives belong in the assessment of WIA services.
In response to our second recommendation, Labor said it recently  developed procedures to document noncompetitive grant proposals’  compliance with statutory requirements.  In its comments and in its  corrective action plan to address Labor’s inspector general’s findings,  Labor noted that its new procedures for reviewing noncompetitive grant  proposals included determining if they met these requirements and  requires documenting this.  Labor’s inspector general is in the process of  reviewing these corrective actions to determine their adequacy.  Our  review of the documents supporting these new procedures found no  specific reference to statutory program requirements against which  proposals would be checked for compliance. We revised our draft  recommendation to clarify that we are asking Labor to identify the  statutory program requirements for which compliance should be  documented.
In response to our third recommendation, Labor said it plans to implement  a risk-based monitoring approach for the WIRED initiative.   Given that  Labor has not yet done so, we stand by our recommendation that they  follow through with this effort.
In its comments Labor disagreed with several of our findings and stated  that  the initiatives were not considered research projects and were not  designed to compare participant outcomes with the participant  outcomes achieved under the WIA formula program; workforce boards face no challenges that cannot be overcome in  pursuing demand driven practices that are encouraged under the  three initiatives;   it is not improper to modify criteria for selecting grantees when the  agency assessment of workforce needs changes; and  there are no specific rules for documenting the decision-making  process for noncompetitive grants, but agreed that additional  documentation would be valuable and indicated that it has taken  steps to require such documentation   While we recognize that the grants awarded under these initiatives were  not research projects per se, many were funded as pilots and  demonstrations and by statute were required to have an evaluative  component.  Although Labor says that it did not intend to compare the  initiative participants’ outcomes with those of participants in WIA formula  programs, its own documents suggest that these types of comparisons are  warranted.  We stand by our findings that workforce boards face  challenges implementing demand-driven approaches.  Labor  acknowledged their efforts to provide workforce boards with assistance in  this area. Regarding selection of noncompetitive, High Growth grantees,  the fact remains that Labor could not provide specific criteria used for  selecting these grantees and told us that these criteria changed over time.   Although Labor’s statement that no rules exist for documenting decision  making runs counter to federal principles for internal controls, it indicated  that it is taking steps to address this issue.
Labor officials also provided technical comments that we incorporated  into the report where appropriate.  Labor’s written comments are  reproduced in appendix IV.
We are sending copies of this report to the Secretary of Labor, relevant  congressional committees, and other interested parties. We also will make  copies available to others upon request. In addition, the report will be  made available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7215 or scottg@gao.gov. Contact points for our Office of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Our objectives were to examine (1) the intent of the grant initiatives and  the extent to which Labor will be able to assess their effects; (2) the extent  to which the process used competition, was adequately documented, and  included key players; and (3) what Labor is doing to monitor individual  grantee compliance with grant requirements.
To determine the intent of the grant initiatives and the extent to which  Labor was able to assess their effects, we reviewed documents, including  grant proposals and quarterly reports, Labor’s strategic plan to identify  plans for research, and a study on the implementation and sustainability of  the early High Growth grants. We reviewed available solicitations for  grant applications to understand the purpose and goals of the grants and  to identify the kinds of outcome data that Labor expected from grantees— three for competitive High Growth grants, three Community Based grants,   and two WIRED grants.  In addition, we interviewed the appropriate Labor  officials and also public workforce experts from the Upjohn Institute, the  Council on Competitiveness, the National Governors Association, the  National Network of Sector Partners, the Urban Institute, the National  Association of State Workforce Agencies, and the National Association of  Workforce Boards to obtain their views on the grant initiatives and  workforce system activities. We also obtained a list of all High Growth,  Community Based, and WIRED grants awarded between fiscal year 2001  and 2007 to determine the amounts, the source of funding, and the timing  of the awards. We assessed the reliability of the individual grant data  Labor provided by (1) reviewing the data for obvious errors and  completeness, (2) reviewing related documentation, and (3) interviewing  agency officials knowledgeable about the data. We determined that the  data were sufficiently reliable for the purposes of this report. In addition,  we reviewed the Workforce Investment Act (WIA) and the American  Competitiveness Act (ACWIA), the two primary laws under which the  funding for these grants was authorized.
We also conducted site visits. Out of a universe of about 340 grantees  provided by Labor at the time of our review, we selected eight for more in- depth study, conducted semi-structured interviews of grantees and state  and local workforce officials, and reviewed grantees’ proposals and recent  quarterly reports. We visited six grantees in five states—Colorado, Florida,  New Hampshire, New York, and Washington and contacted two additional  grantees by telephone in Illinois and Pennsylvania. Four criteria were used  to select these grantees: (1) grant type—we selected an industry/business  representative, a community college, and a representative of the public  workforce system for High Growth grantees;  three community colleges  for the Community Based grantees; and two regional grantees for WIRED;  (2) geographic location—the sites represented different regions of the  country; (3) time period—for the High Growth grantees, we selected two  that received a grant through the noncompetitive process and had been  operating 3 or more years and one grantee that competed for its award and  was 2 years old; all of the Community Based were 2-year-old grants; and  the WIRED grantees represented the first generation, which were only 1  year old; and (4) award amount—we selected grantees with awards that  exceeded $1 million.
In examining the extent to which the process used competition, was  adequately documented, and included key players, we reviewed  documents regarding the processes used to award both competitive and  noncompetitive grants, and interviewed officials. We also analyzed data on  the timing, numbers, and amounts of the High Growth grants and whether  they were awarded competitively and noncompetitively. To understand  Labor’s internal procedures and requirements for awarding grants, we  reviewed the Department of Labor’s Manual Series guidance for awarding  noncompetitive grants, its guidelines for submitting a noncompetitive  grant proposal, and several solicitations for grant applications, which are  used to announce competitive grant proposals. In addition, we reviewed  recommendations from the Domestic Working Group Grant Accountability  Project’s report on awarding federal grants and the Standards for Internal  Control.  We also reviewed the Federal Grant and Cooperative Agreement  Act to determine the legal requirements for awarding noncompetitive  grants. Finally, we reviewed Labor’s inspector general’s report on High  Growth grants for findings concerning the procedures for awarding these  grants noncompetitively. Labor also provided rosters of invitees and  participants to its executive and solutions forums, which we analyzed to  determine the numbers of state and local workforce participants at each of  the forums.
To address what Labor is doing to monitor the performance of grantees,  we interviewed Labor officials regarding the monitoring process and  reviewed a list of High Growth and Community Based grants that had been  monitored. We reviewed examples of six grantee monitoring reports and  the single audit report of one grantee we contacted. We also reviewed  Labor’s monitoring procedures, but we did not assess their effectiveness.
In addition, we reviewed Labor’s inspector general’s reports on grantee  performance data and on the single audit process. We conducted our work  from May 2007 to May 2008 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: High Growth, Community Based, and WIRED Grantees through September 30, 2007, by Initiative and State
paragraphs: Berrien Cass Van Buren Michigan Works!
section_title: Appendix III: Department of Labor Technical Assistance Contracts
paragraphs: Appendix III: Department of Labor Technical  Assistance Contracts DAH Consulting, Inc., and Aspen Institute  M.H. West & Co., Inc.
Berkeley Policy Associates Old H-1B Technical Skill Training Grant  Program, recast for HG  Mathematica Policy Research, Inc.
DAH Consulting, Inc.
M.H. West & Co., Inc.
Technology & Management Services, Inc.
section_title: Appendix IV: Comments from the Department of Labor
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Patrick di Battista, Assistant Director, and Julianne Hartman Cutts  managed this engagement.
Karen A. Brown and Nancy Purvine, Senior Analysts, and Stephanie Toby,  Analyst, made significant contributions to this report. Jean McSween  provided methodological assistance, and Jessica Botsford provided legal  assistance. The team also benefited from key technical assistance from  Susan Aschoff, Pat L. Bohan, Paul Caban, Jessica Orr, Michael Springer,  and Charles Willson.
section_title: Related GAO Products
paragraphs: Workforce Investment Act: Additional Actions Would Improve the  Workforce System. GAO-07-1061T. Washington, D.C.: June 28, 2007.
Veterans’ Employment and Training Service: Labor Could Improve  Information on Reemployment Services, Outcomes, and Program  Impact.  GAO-07-594. Washington, D.C.: May 24, 2007.
Workforce Investment Act: Employers Found One-Stop Centers Useful in  Hiring Low-Skilled Workers; Performance Information Could Help  Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December  22, 2006.
National Emergency Grants: Labor Has Improved Its Grant Award  Timeliness and Data Collection, but Further Steps Can Improve Process.  GAO-06-870. Washington, D.C.: September 5, 2006.
Discretionary Grants: Further Tightening of Education’s Procedures for  Making Awards Could Improve Transparency and Accountability.   GAO-06-268. Washington, D.C.: February 21, 2006.
Workforce Investment Act:  Labor and States Have Taken Actions to  Improve Data Quality, but Additional Steps Are Needed.  GAO-06-82.   Washington, D.C.: November 14, 2005  Workforce Investment Act: Substantial Funds Are Used for Training, but  Little Is Known Nationally about Training Outcomes. GAO-05-650.  Washington, D.C.: June 29, 2005.
Workforce Investment Act: Labor Should Consider Alternative  Approaches to Implement New Performance and Reporting  Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.
Workforce Investment Act: Employers Are Aware of, Using, and Satisfied  with One-Stop Services, but More Data Could Help Labor Better Address  Employers’ Needs. GAO-05-259. Washington, D.C.: February 18, 2005.
Public Community Colleges and Technical Schools: Most Schools Use  Both Credit and Noncredit Programs for Workforce Development.   GAO-05-4. Washington, D.C.: October 18, 2004.
Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
National Emergency Grants: Labor Is Instituting Changes to Assess  Performance, but Labor Could Do More to Help. GAO-04-496. Washington,  D.C.: April 16, 2004.
Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington, D.C.: February 23, 2004.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing Is Needed. GAO-03-725. Washington, D.C.:   June 18, 2003.
Workforce Training: Employed Worker Programs Focus on Business  Needs, but Revised Performance Measures Could Improve Access for  Some Workers. GAO-03-353. Washington, D.C.: February 14, 2003.