Employment and Training Program Grants: Evaluating Impact and Enhancing Monitoring Would Improve Accountability
Released: 2008-05-07T00:00:00

Letter

Background

When it was enacted in 1998, WIA created a new, comprehensive workforce investment system designed to change the way employment and training services are delivered. Under WIA, each state establishes a state workforce investment board that, among other duties, determines strategic priorities, current and projected employment opportunities, and job skills necessary to obtain such employment. It also designates local workforce investment areas across the state and develops an allocation formula for distributing funds to those local areas. Each local area is governed by local workforce investment boards that make decisions about the number and location of one-stop career centers, where partner programs make their services and activities available. Local boards are required to promote employers’ participation in the workforce investment system and assist them in meeting hiring needs. Training services provided must be directly linked to occupations in demand in the local area. To further the involvement of employers, WIA requires that more than half the members of each state and local workforce investment board be representatives of business. The purpose of the one-stop system is to increase the employment, retention, and earnings of job seekers and, by increasing their occupational skills, enhance the productivity and competitiveness of the national economy. WIA requires states and localities to track the performance of WIA-funded activities and Labor to hold states accountable for their performance in the areas of job placement, employment retention, and earnings change.

The Employment and Training Administration (ETA) oversees the High Growth, Community Based, and Wired grant initiatives. The vast majority of these grants are awarded under a provision of WIA, which provides authority for demonstration, pilot, multi-service, research, and multi-state projects, and a provision of the American Competitiveness and Workforce Improvement Act (ACWIA), which provides authority for job training grants funded by the H-1B visa program.

Labor is required to conduct impact evaluations of its programs and activities carried out under WIA, including pilot and demonstration projects. While impact evaluations make it possible to isolate a program’s effect on participants’ outcomes, there are several ways to conduct them, including experimental and quasi-experimental methods. In 2004 and 2007, GAO recommended that Labor comply with WIA requirements to conduct an impact evaluation of WIA services to determine what services are most effective for improving employment-related outcomes. Labor agreed with our recommendation. In December 2007, the agency announced it had begun a quasi-experimental evaluation⎯an impact evaluation that does not use a control group—of the WIA Adult and Dislocated Worker programs, with a final report expected in November 2008.

Federal law recommends, but does not require that all grants be awarded through competition. The Federal Grant and Cooperative Agreement Act encourages competition in grant programs, where appropriate, to ensure that the best possible projects are funded. In addition, Labor’s own guidance governing procurement and grant operations—the Department of Labor Manual Series—states that competition is recommended, unless one or more of eight exceptions apply. Further, a guide on improving grant accountability developed by the Domestic Working Group Grant Accountability Project recommends grants be awarded competitively because competition facilitates accountability, promotes fairness and openness, and increases assurance that grantees have the systems in place to efficiently and effectively use funds to meet grant goals.

Effective monitoring is also a critical component of grant management. The Domestic Working Group’s suggested grant practices states that financial and performance monitoring is important to ensure accountability and the attainment of performance goals. Labor monitors most grants through a risk-based strategy based on its “Core Monitoring Guide.” A key goal is to determine compliance with specific program requirements. The guide includes five monitoring areas: program design and governance, program and grant management systems, financial management systems, service, and product delivery, and performance accountability. In addition, entities receiving Labor grants are subject to the provisions of the Single Audit Act if certain conditions are met. The act established the concept of the single audit to replace multiple grant audits with one audit of a recipient as a whole. As such, a single audit is an organization wide audit that covers, among other things, the recipient’s internal controls and its compliance with applicable provisions of laws, regulations, contracts, and grants.

Grants Are Intended to Change the Workforce System, but Labor Will Be Challenged to Evaluate Their Impact

According to Labor officials, the grant initiatives are designed to change the focus of the public workforce system to emphasize the employment and training needs of high-growth, high-demand industries, but Labor will be challenged in assessing their impact. For the three grant initiatives, Labor awarded 349 grants totaling almost $900 million that were intended to bring about this change by identifying the workforce and training needs of growing, high-demand industries; engaging workforce, industry, and educational partners to develop innovative solutions to workforce challenges, such as worker shortages; leveraging a wide array of resources to fund the solutions; and integrating workforce and economic development to transform regional economies by creating good jobs. However, 7 years after awarding the first grant, Labor will be challenged to evaluate the effect of the grants.

Labor Said the Grants Are Designed to Make the Workforce System More Focused on High-Growth, High-Demand Industries

According to Labor officials, the High Growth, Community Based, and WIRED initiatives are designed to collectively change the focus of the workforce investment system by giving greater emphasis to the employment and training needs of high-growth, high-demand industries. They characterized High Growth as a systematic change initiative designed to make the system more demand-driven (i.e., focused on the needs of growing and high-demand industries) and to make the system’s approach to workforce development more strategic by engaging business, industry, and education partners to identify workforce challenges and solutions. The Computing Technology Industry Association provides one example of the kinds of activities funded under these grants. It used its High Growth grant to expand an apprenticeship program in a growing industry— information technology. As a related effort, the Community Based grants were designed to build the training capacity of community colleges for high-growth, high-demand occupations. In expanding existing training programs and developing new programs aimed at addressing worker shortages in three high-demand health care occupations—nursing, dental hygiene, and health unit coordinators, the Seattle Central Community College in Washington provides an example of efforts funded under these grants. The goal of third grant initiative, WIRED, was to “catalyze” the creation of high-skill and high-wage opportunities for workers within the context of regional economies, to test models for integrating workforce and economic development, and to demonstrate that workforce development is a key driver in transforming regional economies. The northeast region of Pennsylvania provides one example of the types of projects funded through WIRED. Pennsylvania is using WIRED grant funds to develop Wall Street West, a nine county regional project intended to provide backup and disaster recovery operations and facilities for the New York City financial markets. According to Labor officials, these funds support workforce education and training to support this effort. From 2001 through 2007, Labor awarded 349 grants totaling almost $900 million for these initiatives (see table 1). For a list of these grants, see appendix II.

While Labor said the three grant initiatives were necessary to shift the focus of the workforce investment system, experts held varying opinions on the extent to which state and local workforce boards were pursuing demand-driven strategies on their own. Of the experts we spoke with, one said that many states were already focusing on high-growth sectors, reaching out to industry, and developing strategic partnerships. Another characterized innovation among workforce boards as a normal curve that ranged from highly functional to less than functional. One said that some boards had always been pursuing these types of strategies. Another said that the system had been lacking in this regard.

In addition, when considering how extensive demand-driven practices might have been, experts identified a number of systematic factors that might explain why workforce boards would face challenges in pursuing them. These included insufficient funding, limited flexibility in how funds can be used, statutory requirements to target services to certain groups of workers, and the need to respond to local economic conditions. Commenting on workforce boards’ ability to form strategic partnerships, one expert noted that there are no funds to support such endeavors and no performance standards to measure them. With regard to regional economic development, experts said boards are structured around local areas, not regions, regional economies are highly variable, regional governance structures can make achieving buy-in difficult and that rural areas can be particularly challenged in pursuing regional approaches.

Beyond the question of how extensive demand-driven practices were, Labor officials said a number of indicators show that the initiatives are changing the system. According to Labor, they have seen a “system-shift” in the approach to implementing workforce solutions through an increase in demand-driven topics at conferences since the roll out of the initiatives. Labor said this shift has been driven by partnerships between the workforce investment system, business, industry, and educators using the High Growth framework. Labor also said it is seeing demand-driven strategies in state and local strategic plans and in states using their own money to fund High Growth-like projects. Labor pointed out that the system has evolved to the point where high performing local workforce boards with demand-driven practices are mentoring lesser performers. Lastly, Labor said the content on its Web site, Workforce3 One, was also evidence of change. For example, Labor held an interactive seminar broadcast on this site to train participants to use an on-line tool to share curricula developed through the initiatives.

Seven Years after Awarding the First Grant, Labor Will Be Challenged to Evaluate Their Impact

Despite the money invested and emphasis placed on these initiatives, Labor did not fully integrate them into its strategic plan or ETA’s research plan from the start. The Government Performance and Results Act states that strategic plans shall contain strategic goals and objectives, including outcome-related, or performance goals, and objectives for an agency’s major functions and operations. However, the strategic plan includes performance goals only for the Community Based initiative. High Growth and WIRED—the two initiatives where Labor spent the most money⎯are mentioned in the strategic plan, but not specifically linked to a performance goal; therefore, it is unclear what criteria Labor will use to evaluate their effectiveness. Moreover, the data needed to assess the performance of these initiatives are not specified. Labor officials said the strategic plan did not address the initiatives because it focuses on budget issues. Just as the initiatives are not fully integrated into the strategic plan, neither are they fully integrated into ETA’s research plan, which cites plans for future evaluations, but it does not specify an assessment of their impact.

Not fully incorporating the initiatives into its strategic or research plans may have limited Labor’s ability to collect consistent outcome data. Labor said that prior to 2005, it consistently collected data from grantees on the number of participants enrolled in and completing training funded under High Growth—the only one of the three grant initiatives operating at that time. However, it did not collect performance outcomes similar to those being collected for its other training and employment services. In 2005, Labor instituted what were called common measures to assess the effectiveness of one-stop programs and services. The common measures include participant employment outcomes, earnings, and job retention after receiving services. Currently, Labor cannot require High Growth and Community Based grantees to provide data on the common measures until it receives OMB approval. According to Labor, it can collect such measures for WIRED grantees, but it has not yet done so. As a result, Labor may not have consistent data for individuals participating in the programs funded under the grant initiatives. In addition, it may lack data that will allow it to compare outcomes for individuals served by grant- funded programs with those served by employment and training programs offered through the one-stop system. Having comparable outcome data is important because the goal of an impact evaluation is to determine if outcomes are attributable to a program or can be explained by other factors.

Labor will face challenges in obtaining the data necessary to make meaningful comparisons. Starting in 2006, Labor included information on the common measures in all new solicitations for High Growth and Community Based grants, notified grantees of its goal for standardizing performance reporting, and provided technical assistance to help grantees prepare for it. Labor also encouraged grantees to work with local workforce system partners to leverage their experience in tracking and reporting performance outcomes. Labor anticipates having an OMB approved reporting format in place by July 1, 2008. However, because some of the first grantees have already completed their projects, obtaining information about workers that have left the program may prove difficult and costly.

Labor has some plans underway to evaluate the initiatives, but may face challenges drawing strong conclusions from them. Labor has conducted an evaluation of the implementation and sustainability of 20 early High Growth grantees. It is now evaluating the impact of the training provided by High Growth grantees. Labor anticipates the final report in December 2008. However, Labor has already experienced a number of challenges in evaluating the initiatives. These include having to limit its evaluation to only 6 grantees of 166, because only 6 had sufficient participants to ensure a statistically significant evaluation. They also include problems gaining access to workers’ earnings data, and inconsistent outcome data from grantees.

Labor officials said they plan to conduct a comprehensive evaluation of the Community Based initiative. The first phase of the evaluation will examine the extent to which the Community Based grants addressed the stated workforce objectives and challenges funded projects were intended to address, as well as document the role of business and the workforce investment system in the overall success of the grants, according to Labor. This phase will also include an examination of the feasibility of performing an impact evaluation and will be completed in late 2008. Depending on the results of this phase, Labor officials said an impact evaluation will begin in 2009.

For its evaluation of the WIRED initiatives, Labor says it is examining the implementation and cumulative effects of WIRED strategies, including change in the number and size of companies in targeted high-growth industries and whether new training led to job placement in the targeted industries. It contracted with the Berkeley Policy Associates to conduct the evaluation for the first 13 grantees, and a final report is expected by June 2010. It also contracted with Public Policy Associates to similarly evaluate the 28 remaining WIRED grantees. Labor officials said these initiatives are not included in the agency’s broader WIA impact study.

The Initial Noncompetitive Process Was Not Adequately Documented and Did Not Include Key Players

While Labor now awards grants under all three grant initiatives competitively, initially almost all High Growth grants were awarded without competition. Labor also did not document the criteria for selecting noncompetitive High Growth grants or whether they met Labor’s internal requirements or the requirements of the laws under which the grants are authorized. In addition, Labor’s process for identifying workforce solutions did not include most of the state and local workforce investment boards.

All Three Types of Grants Are Now Awarded Competitively, but the Vast Majority of High Growth Grants Were Awarded Without Competition

The Community Based and the WIRED grants have always been awarded through a competitive process, but until 2005, Labor did not award High Growth grants competitively even though federal law and Labor’s internal procedures recommend competition. While Labor had discretion in awarding High Growth grants without competition, the extent to which it did so raises questions about how Labor used this method of awarding grants. Competition facilitates accountability, promotes fairness and openness, and increases assurance that grantees have systems in place to meet grant goals. Yet Labor chose to award 83 percent of the High Growth grants, which represented almost 90 percent of the funds, without competition between fiscal years 2001 and 2007 (see table 2). Congress required that High Growth grants funded by H-1B fees be awarded competitively for fiscal years 2007 and 2008. Prior to that time, there were no provisions requiring Labor to award High Growth grants competitively.

Labor said that it used a noncompetitive process to promote innovation. They also said that they awarded grants without competition to save the time it would have taken to solicit grants through competition. In hindsight, they said they could have offered the High Growth grants competitively earlier because they recognized that the number of noncompetitive awards created a perception that the process was unfair. They said, however, that they always intended to award later grants competitively.

In contrast to the High Growth grants, the Community Based and WIRED initiatives have always been awarded through competition. These funding opportunities were announced to potential applicants through a solicitation for grant application that listed the information that an application must include to compete for funding. These applications were then reviewed and scored by a knowledgeable technical panel. These solicitations were also reviewed by Labor attorneys for compliance with procurement and statutory program requirements for awarding grants, according to officials.

Labor Did Not Document the Criteria for Selecting Noncompetitive High Growth Grants or Whether They Met Labor’s Internal Requirements or Requirements of the Law

Because the initial High Growth process was noncompetitive, documenting the decision steps was all the more important to ensure transparency. However, Labor was unable to provide documentation of the initial criteria for selecting grantees. As a result, it did not meet federal internal control standards, which state that all transactions and other significant events need to be clearly documented and that the documentation should be readily available for examination. In addition, it was unable to document that it met the statutory requirements for the laws authorizing the grants. Lastly, according to Labor’s inspector general, it did not adequately document that it had followed its own procedures for awarding grants without competition.

Labor did not document the criteria used to select the early noncompetitive High Growth projects. Labor officials told us there were no official published guidelines specific to High Growth grants, only draft guidelines, which were no longer available. In addition, Labor officials told us that generally they were looking for grantees that pursued partnerships and leveraged resources, but that attributes they sought changed over time. Labor published general requirements for noncompetitive grants in 2005 and updated them in 2007. Officials said these were not requirements, only guidelines for the kinds of information Labor would find valuable in evaluating proposals.

In addition, while Labor said noncompetitive grants could be awarded without competition under the WIA provision authorizing demonstrations and pilot projects and under ACWIA, they could not document that the grants fully complied with the requirements of these provisions. For example, WIA requirements include providing direct services to individuals, including an evaluative component, and being awarded to private entities with recognized expertise or to state and local entities with expertise in operating or overseeing workforce investment programs. Officials said that they were certain they had ensured that the projects met all statutory requirements, but acknowledged they did not document that the requirements were met.

Labor’s inspector general found the agency did not always document that it followed its own procedures or always obtained required review and approval before awarding grants noncompetitively. Labor officials said most of the noncompetitive grant proposals were presented to Labor’s Procurement Review Board for review and approval allowed under exceptions for proposals that were unique or innovative, highly cost- effective, or available from only one source. However, in 2007, Labor’s inspector general reviewed a sample of the noncompetitive High Growth grants awarded between July 2001 and March 2007 and found that 6 of the 26 grants, which should have undergone review, were awarded without prior approval from the review board. Furthermore, they found that Labor could not demonstrate that proper procedures were followed in awarding the High Growth grants without competition.

Although they were unable to provide documentation, Labor officials said they used considerable rigor in selecting grant recipients under the noncompetitive process. Similar to a competitive process, the noncompetitive grant proposals were highly scrutinized and reviewed to ensure they made best use of scarce resources. They said that in most cases, staff created abstracts to highlight strengths and weaknesses and multiple staff and managers participated in reviews and decision-making. In addition, Labor officials strongly disagreed with the majority of the inspector general’s findings. They said they followed established procurement practices as required but agreed that additional documentation would be valuable.

In response to the inspector general’s report, Labor has taken steps to strengthen the noncompetitive process. These include developing procedures to review noncompetitive grant proposals for criteria including support of at least one of ETA’s strategic goals and investment priorities. The procedures also require ETA to document that required procedures are followed and that required review and approval is obtained before awarding grants noncompetitively. However, the newly developed procedures do not explicitly identify the statutory program requirements for which compliance should be documented.

Labor’s Process for Identifying Industry Workforce Challenges Did Not Include the Majority of Workforce Investment Boards

The vast majority of workforce boards—which oversee the workforce investment system—were not included in the meetings that served as incubators for grant proposals. After identifying 13 high-growth/high- demand sectors, Labor held a series of meetings between 2002 and 2005 with industry executives and other stakeholders to identify workforce challenges and to develop solutions to them. According to Labor, they first held meetings with industry executives—executive forums—for 13 sectors to hear directly from industry leaders about the growth potential for their industries and to understand the workforce challenges they faced. Second, they hosted a series of workforce solutions forums for 11 of the sectors, which brought together industry executives (often those engaged in human resources and training activities) with representatives from education, state and local workforce boards, or other workforce-related agencies. However, a review of Labor’s rosters for the solutions forums shows that while there were more than 800 participants, 26 of the almost 650 local workforce boards nationwide were represented, and these came from 15 states. (See fig. 1.)

Further only 20 of the 50 states had their state workforce investment board or other agency represented (see table 3).

Labor officials said they went to great lengths to include workforce system participants in solutions forums. Officials said they asked state workforce agencies to identify a state coordinator to interface with Labor, work collaboratively with industry partners, and identify potential attendees for executive and solutions forums. Further, the state coordinators were to help Labor communicate with the workforce system about High Growth activities and were kept updated through routine conference calls and periodic in-person meetings, according to Labor. Labor officials also said the Assistant Secretary and other senior officials traveled frequently, speaking to workforce system partners at conferences to gather information about innovative practices. Labor officials said, even with these efforts, they found only a few workforce boards operating unique or innovative demand-driven programs.

However, most workforce board officials we spoke to in our site visits reported becoming aware of the meetings and the grant opportunities after the fact, even though they were pursuing the kinds of innovative practices the meeting was supposed to promote. Some state board officials said that they were often unaware that grants had been awarded, and at least one local workforce board said it became aware of a grant only when the community college grantee approached it for assistance in getting enough students for their program. In addition, officials in states we visited said they had been developing and using the types of practices that Labor was seeking to promote at the meetings.

Being present at the meetings could have been beneficial to workforce boards. Labor officials acknowledged that when meeting participants suggested a solution to an employment challenge that they deemed innovative and had merit, they encouraged them to submit a proposal for a grant to model the solution. In addition, officials said that in some cases, they provided applicants additional assistance to increase the chances that the proposal would be funded.

Labor Uses a Risk- Based Monitoring Approach for High Growth and Community Based Grants but Not for WIRED Grants

For all three grant initiatives, Labor has a process to resolve findings found in single audits, collects quarterly performance information, and provides technical assistance as a part of monitoring. In addition, it has a risk-based monitoring approach for High Growth and Community Based grants, but has not implemented such an approach for WIRED grants. However, Labor officials told us that they are developing a risk-based approach for monitoring WIRED grants that they plan to implement in the spring of 2008. According to Labor’s grants monitoring guide, improving grant administration is crucial to Labor’s management improvement plans and risk-based on-site monitoring of grantees is an important part of this effort.

Labor Has a Process to Ensure Grantees Resolve Findings in Single Audits, Collects Quarterly Performance Information, and Provides Technical Assistance

Labor said it has a process to work with grantees, including High Growth, Community Based, and WIRED to resolve findings in single audits. However, Labor’s inspector general reported that Labor does not have procedures in place for grant officers to follow up with grantees with past due audit reports to ensure timely submission and thus proper oversight and correction of audit findings. The inspector general recommended that Labor implement such procedures and Labor has done so, but the finding remains open because Labor’s inspector general has not yet determined if the procedures adequately address the recommendation.

As part of its monitoring, Labor requires High Growth, Community Based, and WIRED grantees to submit quarterly financial and performance reports. Financial reports contain information, such as total amount of grant funds spent and amount of matching funds provided by the grantee. Performance reports focus on activities leading to performance goals, such as grantee accomplishments and challenges to meeting grant goals. Labor officials said they review these reports and follow up with grantees if there are questions. Labor officials acknowledge, however, that they are still working to ensure the consistency of performance reports provided by High Growth and Community Based grantees and are working with OMB to establish consistent reporting requirements. In addition, while the finding was not specific to these three grants, Labor’s inspector general cited high error rates in grantee performance data as a management challenge. Labor is taking steps to improve grant accountability, such as providing grantee and grant officer training.

All grantees receive technical assistance from Labor on how to comply with laws and regulations, program guidance, and grant conditions. For example, Labor issued guides for High Growth and Community Based grantees which include information on allowable costs and reporting requirements. In addition, Labor officials said they trained national and regional office staff to address grantees’ questions and help High Growth and Community Based grantees obtain assistance from experts at Labor and other grantees. Labor officials said they hold national and regional High Growth and Community Based grantee orientation sessions for new grantees, present technical assistance webinars and training sessions focused on specific high-growth industries, assist grantees with disseminating grant results and products, such as curricula, and set up virtual networking groups of High Growth grantees to encourage collaboration.

Labor officials told us they have teams who provide technical assistance to each WIRED grantee including weekly contact. During these sessions, Labor staff work with WIRED grantees on grant management issues, such as costs that are allowed using grant funds. Labor staff provide additional assistance through conference calls, site visits, and documentation reviews. In addition, Labor officials said they have held five webinars on allowable costs and provided grantees with a paper on allowable costs in July 2006, which was updated in July 2007. Finally, Labor officials explained that they made annual site visits for the first 13 WIRED grantees in spring and summer of 2007 to discuss implementation plans and progress toward plan goals. In addition, Labor staff said they have reviewed the implementation of the remaining WIRED grants to ensure that planned activities comply with requirements of the law. However, none of these reviews resulted in written reports with findings and corrective action plans.

Labor has spent $16 million on contracts to provide technical assistance, improve grant management, administration, and monitoring, and to assist Labor with tasks such as holding grantee training conferences. The larger of these contracts focus on providing technical assistance to WIRED grantees. For example, one contract valued at over $2 million provides WIRED grantees assistance with assessing regional strength and weaknesses and developing regional economic strategies and implementation plans. Another grant, valued at almost $4 million, provides a database and geographic information system that WIRED grantees can use to facilitate data analysis and reporting, among other things. For a list of these contracts, see appendix III.

While these monitoring and technical assistance efforts are useful to help grantees manage their grants, they do not provide a risk-based monitoring process to identify and resolve problems, such as compliance issues, in a consistent and timely manner.

Labor Provides Risk-Based Monitoring for High Growth and Community Based Grants

Labor uses a risk-based strategy to monitor grants under two of the three grant initiatives: High Growth and Community Based. For these initiatives it selects grantees to monitor based on indications of problems that may affect grant performance. Labor’s risk-based approach to monitoring most grants reflects suggested grant practices. Suggested grant practices recognize that it is important to identify, prioritize, and manage potential at-risk grant recipients for monitoring given the large number of grants awarded by federal agencies. Through this process, Labor staff determine if grantee administration and program delivery systems operate, the grantee is in compliance with program requirements, and information reported is accurate.

Labor’s risk-based monitoring strategy involves conducting site visits based on grantees’ assessed risk-levels and availability of resources, among other things. These site visits include written assessments of grantee’s management and performance and compliance findings and requirements for corrective action. For example, Labor’s site visit guide includes questions about financial and performance data reporting systems, such as how well the grantee maintains files on program participants.

Labor has monitored about half of the High Growth grants and over one- quarter of the Community Based grants. Labor officials said these monitoring efforts have resulted in a number of significant findings which have generally been resolved in a timely manner. (See table 4.) For example, during a November 2006 site visit of a Community Based grantee Labor identified three findings: incomplete participant files, failure to follow internal procurement procedures, and missing grant partnership agreements. Similarly, during a site visit in spring 2006 to a High Growth grantee, Labor found that the grantee did not accurately track participant information and reported incorrect information on expenditures, among other things. As of September 2007 Labor said these findings had been resolved (see table 4).

As another part of Labor’s risk-based monitoring strategy, Labor’s internal requirements specify that Labor staff are to make site visits to all new grantees, including High Growth, Community Based, and WIRED, within 12 months of beginning grant activity and to new grantees rated as “at risk” within 3 months. Labor officials said they consider “new grantee” site visits to be orientation visits and had not made visits to most new grantees. They said they broadly interpret this requirement to include a variety of methods of contact and generally use teleconference and video conference training sessions rather than site visits, based on the availability of resources. For example, Labor calls each new Community Based grantee to schedule new grantee training. Labor is taking steps to update its internal requirements to better reflect the purpose of the new grantee monitoring.

Labor Is Planning to Develop a Risk-Based Approach for Monitoring WIRED Grants, but It Has Not Yet Done So

Labor officials said they are developing a WIRED grant assessment tool, but it is not finalized. They plan to implement it in spring 2008; however, the department has not developed a schedule for its use. Labor officials explained that a specific tool is needed for monitoring WIRED grants because the site visit guide used for most other Labor grants would not be appropriate for WIRED grants. One reason is that WIRED grants are awarded to states but are generally administered by other entities. For example, Labor awarded a $15 million WIRED grant to the state of Colorado, but programs funded by the grant are primarily administered by the not-for-profit Metro Denver Economic Development Corporation.

Conclusions

Seven years after awarding the first grants, Labor is in the process of evaluating the High Growth and WIRED initiatives and has plans to evaluate the Community Based initiative but will be challenged to evaluate the effect of the almost $900 million spent on the grant initiatives. Because of poor planning, the agency will not be able to perform the kind of comprehensive impact evaluation that would have been possible if it had been collecting data consistent with that collected for other WIA programs all along. Because Labor lacks such data it will be challenged in drawing strong conclusions from its planned evaluations. Until Labor collects consistent outcome data, such as the common measures, that would allow comparison of impact of each of the three grant initiatives to those of other Labor programs, Labor cannot know the extent to which the initiatives improve employment outcomes, if at all.

The vast majority of High Growth grants—more than $263 million— were awarded noncompetitively using inconsistent selection criteria under a process that was not adequately documented. As a result there is little assurance that the grants selected were the best possible projects. Although Labor currently awards grants under all three grant initiatives competitively, the fact that Labor used inconsistent selection criteria that changed over time and did not adequately document the process when awarding High Growth grants noncompetitively raises questions about how Labor uses this method of awarding grants. This is a critical concern given that Labor could again award millions of dollars of noncompetitive grants. While it has taken steps to strengthen the noncompetitive process, the newly developed procedures do not explicitly identify the statutory program requirements for which compliance should be documented.

Finally, the absence of a risk-based monitoring approach for WIRED puts Labor at risk of not knowing if the millions of dollars awarded to WIRED grantees have been used for the purposes they are intended.

Recommendations for Executive Action

To determine the impact of the three grant initiatives, ensure that the best possible projects are selected, and improve accountability of grant funds, we recommend that the Secretary of Labor take the following three actions: Take steps to ensure that the department can evaluate the impact of the initiatives so that it can draw strong conclusions based on its evaluations, such as following through with plans to collect consistent data, integrating the initiatives into its overall research agenda with relevant performance goals and indicators, and including these initiatives in its assessment of the impact of WIA services.

Direct the Employment and Training Administration to identify the statutory program requirements for which compliance must be documented when awarding noncompetitive grants.

Develop and implement a risk-based monitoring approach for WIRED and a schedule for its use.

Agency Comments and Our Evaluation

We provided a draft of the report to Labor for review and comment. In response to our first recommendation, Labor said it has taken initial steps to evaluate the impact of the initiatives and disagreed with our assertion that the initiatives are not integrated into its strategic plan. We agree that the agency has taken initial steps, but these do not ensure that they will be able to assess the initiatives’ impact. We do not dispute that the strategic plan mentions the initiatives; however, we found that both the strategic and research plans do not tie the High Growth or WIRED initiatives to performance goals, making it unclear what criteria Labor will use to evaluate their effectiveness. In response to the first recommendation, Labor also disputed a claim that we did not make, namely that the initiatives belong in the assessment of WIA services.

In response to our second recommendation, Labor said it recently developed procedures to document noncompetitive grant proposals’ compliance with statutory requirements. In its comments and in its corrective action plan to address Labor’s inspector general’s findings, Labor noted that its new procedures for reviewing noncompetitive grant proposals included determining if they met these requirements and requires documenting this. Labor’s inspector general is in the process of reviewing these corrective actions to determine their adequacy. Our review of the documents supporting these new procedures found no specific reference to statutory program requirements against which proposals would be checked for compliance. We revised our draft recommendation to clarify that we are asking Labor to identify the statutory program requirements for which compliance should be documented.

In response to our third recommendation, Labor said it plans to implement a risk-based monitoring approach for the WIRED initiative. Given that Labor has not yet done so, we stand by our recommendation that they follow through with this effort.

In its comments Labor disagreed with several of our findings and stated that the initiatives were not considered research projects and were not designed to compare participant outcomes with the participant outcomes achieved under the WIA formula program; workforce boards face no challenges that cannot be overcome in pursuing demand driven practices that are encouraged under the three initiatives; it is not improper to modify criteria for selecting grantees when the agency assessment of workforce needs changes; and there are no specific rules for documenting the decision-making process for noncompetitive grants, but agreed that additional documentation would be valuable and indicated that it has taken steps to require such documentation While we recognize that the grants awarded under these initiatives were not research projects per se, many were funded as pilots and demonstrations and by statute were required to have an evaluative component. Although Labor says that it did not intend to compare the initiative participants’ outcomes with those of participants in WIA formula programs, its own documents suggest that these types of comparisons are warranted. We stand by our findings that workforce boards face challenges implementing demand-driven approaches. Labor acknowledged their efforts to provide workforce boards with assistance in this area. Regarding selection of noncompetitive, High Growth grantees, the fact remains that Labor could not provide specific criteria used for selecting these grantees and told us that these criteria changed over time. Although Labor’s statement that no rules exist for documenting decision making runs counter to federal principles for internal controls, it indicated that it is taking steps to address this issue.

Labor officials also provided technical comments that we incorporated into the report where appropriate. Labor’s written comments are reproduced in appendix IV.

We are sending copies of this report to the Secretary of Labor, relevant congressional committees, and other interested parties. We also will make copies available to others upon request. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our objectives were to examine (1) the intent of the grant initiatives and the extent to which Labor will be able to assess their effects; (2) the extent to which the process used competition, was adequately documented, and included key players; and (3) what Labor is doing to monitor individual grantee compliance with grant requirements.

To determine the intent of the grant initiatives and the extent to which Labor was able to assess their effects, we reviewed documents, including grant proposals and quarterly reports, Labor’s strategic plan to identify plans for research, and a study on the implementation and sustainability of the early High Growth grants. We reviewed available solicitations for grant applications to understand the purpose and goals of the grants and to identify the kinds of outcome data that Labor expected from grantees— three for competitive High Growth grants, three Community Based grants, and two WIRED grants. In addition, we interviewed the appropriate Labor officials and also public workforce experts from the Upjohn Institute, the Council on Competitiveness, the National Governors Association, the National Network of Sector Partners, the Urban Institute, the National Association of State Workforce Agencies, and the National Association of Workforce Boards to obtain their views on the grant initiatives and workforce system activities. We also obtained a list of all High Growth, Community Based, and WIRED grants awarded between fiscal year 2001 and 2007 to determine the amounts, the source of funding, and the timing of the awards. We assessed the reliability of the individual grant data Labor provided by (1) reviewing the data for obvious errors and completeness, (2) reviewing related documentation, and (3) interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. In addition, we reviewed the Workforce Investment Act (WIA) and the American Competitiveness Act (ACWIA), the two primary laws under which the funding for these grants was authorized.

We also conducted site visits. Out of a universe of about 340 grantees provided by Labor at the time of our review, we selected eight for more in- depth study, conducted semi-structured interviews of grantees and state and local workforce officials, and reviewed grantees’ proposals and recent quarterly reports. We visited six grantees in five states—Colorado, Florida, New Hampshire, New York, and Washington and contacted two additional grantees by telephone in Illinois and Pennsylvania. Four criteria were used to select these grantees: (1) grant type—we selected an industry/business representative, a community college, and a representative of the public workforce system for High Growth grantees; three community colleges for the Community Based grantees; and two regional grantees for WIRED; (2) geographic location—the sites represented different regions of the country; (3) time period—for the High Growth grantees, we selected two that received a grant through the noncompetitive process and had been operating 3 or more years and one grantee that competed for its award and was 2 years old; all of the Community Based were 2-year-old grants; and the WIRED grantees represented the first generation, which were only 1 year old; and (4) award amount—we selected grantees with awards that exceeded $1 million.

In examining the extent to which the process used competition, was adequately documented, and included key players, we reviewed documents regarding the processes used to award both competitive and noncompetitive grants, and interviewed officials. We also analyzed data on the timing, numbers, and amounts of the High Growth grants and whether they were awarded competitively and noncompetitively. To understand Labor’s internal procedures and requirements for awarding grants, we reviewed the Department of Labor’s Manual Series guidance for awarding noncompetitive grants, its guidelines for submitting a noncompetitive grant proposal, and several solicitations for grant applications, which are used to announce competitive grant proposals. In addition, we reviewed recommendations from the Domestic Working Group Grant Accountability Project’s report on awarding federal grants and the Standards for Internal Control. We also reviewed the Federal Grant and Cooperative Agreement Act to determine the legal requirements for awarding noncompetitive grants. Finally, we reviewed Labor’s inspector general’s report on High Growth grants for findings concerning the procedures for awarding these grants noncompetitively. Labor also provided rosters of invitees and participants to its executive and solutions forums, which we analyzed to determine the numbers of state and local workforce participants at each of the forums.

To address what Labor is doing to monitor the performance of grantees, we interviewed Labor officials regarding the monitoring process and reviewed a list of High Growth and Community Based grants that had been monitored. We reviewed examples of six grantee monitoring reports and the single audit report of one grantee we contacted. We also reviewed Labor’s monitoring procedures, but we did not assess their effectiveness.

In addition, we reviewed Labor’s inspector general’s reports on grantee performance data and on the single audit process. We conducted our work from May 2007 to May 2008 in accordance with generally accepted government auditing standards.

Appendix II: High Growth, Community Based, and WIRED Grantees through September 30, 2007, by Initiative and State

Berrien Cass Van Buren Michigan Works!

Appendix III: Department of Labor Technical Assistance Contracts

Appendix III: Department of Labor Technical Assistance Contracts DAH Consulting, Inc., and Aspen Institute M.H. West & Co., Inc.

Berkeley Policy Associates Old H-1B Technical Skill Training Grant Program, recast for HG Mathematica Policy Research, Inc.

DAH Consulting, Inc.

M.H. West & Co., Inc.

Technology & Management Services, Inc.

Appendix IV: Comments from the Department of Labor

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Patrick di Battista, Assistant Director, and Julianne Hartman Cutts managed this engagement.

Karen A. Brown and Nancy Purvine, Senior Analysts, and Stephanie Toby, Analyst, made significant contributions to this report. Jean McSween provided methodological assistance, and Jessica Botsford provided legal assistance. The team also benefited from key technical assistance from Susan Aschoff, Pat L. Bohan, Paul Caban, Jessica Orr, Michael Springer, and Charles Willson.

Related GAO Products

Workforce Investment Act: Additional Actions Would Improve the Workforce System. GAO-07-1061T. Washington, D.C.: June 28, 2007.

Veterans’ Employment and Training Service: Labor Could Improve Information on Reemployment Services, Outcomes, and Program Impact. GAO-07-594. Washington, D.C.: May 24, 2007.

Workforce Investment Act: Employers Found One-Stop Centers Useful in Hiring Low-Skilled Workers; Performance Information Could Help Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December 22, 2006.

National Emergency Grants: Labor Has Improved Its Grant Award Timeliness and Data Collection, but Further Steps Can Improve Process. GAO-06-870. Washington, D.C.: September 5, 2006.

Discretionary Grants: Further Tightening of Education’s Procedures for Making Awards Could Improve Transparency and Accountability. GAO-06-268. Washington, D.C.: February 21, 2006.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed. GAO-06-82. Washington, D.C.: November 14, 2005 Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes. GAO-05-650. Washington, D.C.: June 29, 2005.

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.

Workforce Investment Act: Employers Are Aware of, Using, and Satisfied with One-Stop Services, but More Data Could Help Labor Better Address Employers’ Needs. GAO-05-259. Washington, D.C.: February 18, 2005.

Public Community Colleges and Technical Schools: Most Schools Use Both Credit and Noncredit Programs for Workforce Development. GAO-05-4. Washington, D.C.: October 18, 2004.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

National Emergency Grants: Labor Is Instituting Changes to Assess Performance, but Labor Could Do More to Help. GAO-04-496. Washington, D.C.: April 16, 2004.

Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services. GAO-04-308. Washington, D.C.: February 23, 2004.

Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18, 2003.

Workforce Training: Employed Worker Programs Focus on Business Needs, but Revised Performance Measures Could Improve Access for Some Workers. GAO-03-353. Washington, D.C.: February 14, 2003.