Youthbuild Program: Analysis of Outcome Data Needed to Determine Long-Term Benefits
Released: 2007-02-28T00:00:00

Letter

Background

CPD manages the Youthbuild program. CPD programs generally fall into three categories: (1) formula grants, of which the largest are the Community Development Block Grant and the HOME Investment Partnership Program, which account for the majority of CPD funding; (2) competitive homeless programs; and (3) competitive nonhomeless programs, including Youthbuild. While no staff are dedicated solely to Youthbuild oversight, CPD monitors the program as part of its oversight responsibilities for 22 programs and initiatives, which 829 staff (233 in headquarters and 596 in field offices) administered as of the beginning of fiscal year 2007. In fiscal year 2006, Congress appropriated $19.1 billion for the CPD programs and initiatives, of which Youthbuild represents a small portion. The first program appropriation for Youthbuild was $40 million for fiscal year 1993. Appropriations reached a high of $65 million in fiscal year 2002, but dropped to $49.5 million in fiscal year 2006, the last year in which HUD was authorized to award grants (see fig. 1).

HUD has provided program funding for Youthbuild through an annual competitive grant application process. Any applicant can receive up to a $700,000 grant; however, new applicants and rural/underserved applicants primarily receive grants up to $400,000. Grant awards can last up to 30 months (or longer if the grantee is given an extension), but some grantees use the funds at a much quicker rate and usually reapply every year, while others may take the full 30 months.

Prospective grantees are state, private, nonprofit, and local entities that implement the Youthbuild program by serving low-income youth, ages 16 to 24, who demonstrate educational need—either by virtue of being high- school dropouts or by justifying their need for inclusion in the program. According to program regulations, at least 75 percent of the participants must come from very low-income families and have dropped out of high school. The program allows up to 25 percent of the youth to have a high- school diploma or have a slightly higher income; however, data developed by YouthBuild USA show that most organizations report that less than 25 percent of participants have high-school diplomas or General Equivalency Diplomas (GED) upon entering their programs. Also, according to data from YouthBuild USA, a number of the youths have been adjudicated (judicially determined to be delinquent), were on welfare upon entering the program, or lived in public housing (see fig. 2). Other studies we reviewed identified similar participant characteristics. Program regulations require participants to spend 50 percent of their program time receiving job training on construction sites, building affordable housing, and 50 percent receiving academic training. Recently, some of the Youthbuild grantees have added nonconstruction programs to their offerings, such as in the areas of computer repair and health care. Funding for the additional programs has come from AmeriCorps.

In 1994, YouthBuild USA, located in Somerville, Massachusetts, became the primary technical assistance contractor for the Youthbuild program. YouthBuild USA is a national support center and intermediary for the program. As such, it provides training, develops materials and handbooks to help grantees replicate the program, conducts peer-to-peer seminars, and disseminates materials on best practices. YouthBuild USA also has established an affiliated network that grantees can choose to join. As of September 2006, 107 of 173 Youthbuild grantees with active grants were affiliates. Since 2002, YouthBuild USA has been using a Web-based system, WebSTA, to collect participant characteristic and performance data from its affiliates.

Among the purposes of transfering the YouthBuild program to Labor was better alignment of existing federal workforce and youth training programs. Labor’s Employment and Training Administration offers programs that assist disadvantaged youths, similar to the programs that Youthbuild grantees offer. The Workforce Investment Act (WIA) of 1998 established a variety of programs to serve low-income youths who face barriers to employment. Labor allocates funds for the youth service programs to state and local areas based on a distribution formula. The grants are made to states, and the states in turn allocate funds to approximately 600 local workforce investment boards. The amount of funding available to states is determined by the distribution of unemployed individuals and disadvantaged youths by state. From 2000 through 2005, Labor administered the Youth Opportunity Grant program, which also was authorized by WIA. This grant program focused on improving education and employment opportunities for at-risk youths, ages 14 to 21, in high-poverty areas. Funding for the program was eliminated beginning in fiscal year 2004. In addition, Labor administers the Job Corps program, which began in 1964, and provides education, training and support services to economically challenged youths, ages 16 through 24, who face multiple barriers to employment. Private companies—chosen through competitive contracting processes—and other federal agencies operating under interagency agreements with Labor, operate the Job Corps centers (located throughout the United States and Puerto Rico) on behalf of Labor.

HUD’s Grant-Making Procedures Incorporated Legislative Criteria

Each year, HUD staff rated and ranked grant applications for serving economically distressed areas based on factors outlined in the enacting legislation and detailed in an annual NOFA. The Housing and Community Development Act of 1992 required HUD to establish selection criteria such as the qualifications or potential capabilities of applicants, the feasibility of the proposed Youthbuild program, the potential for developing a successful Youthbuild program, and the applicants’ commitment to obtain outside resources.

Accordingly, HUD annually issued a NOFA that described the grant application process, outlined criteria for grant awards, and included scoring factors (see fig. 3). The major factors considered were the capacity and relevant experience of the organization (performance), need and extent of the problem, applicant’s soundness of approach, leveraging of nonhousing funds, and ability to show how it would achieve and measure results. Under these factors, grant applicants had to consider area poverty and high-school dropout rates, describe their management and training capabilities, and offer proof of access to property and the existence of partners to build low-income housing. HUD issued its last NOFA for the Youthbuild program on March 8, 2006, and announced the winning grantees in November 2006.

Since fiscal year 2001, HUD has grouped applicants into three separate scoring categories—new, general (that are typically awarded to established Youthbuild organizations), and rural/underserved area grantees. Figure 4 shows the awards by category. The total numbers of grants as well as the grants by the individual categories have varied by year.

The awards process also has been competitive. HUD historically funded less than one-third of all the applicants (see fig. 5). Since 1996, the number of applicants funded ranged from about 7 to 44 percent.

HUD Oversight of Grantees’ Performance Has Been Limited

Although HUD requires grantees to submit closeout reports, which are intended to help evaluate the programs, it has not verified, aggregated, or analyzed data from those reports. Further, the closeout reports do not include other data that could help determine participant characteristics upon entry or the types of jobs they received after completing the program. YouthBuild USA, the technical services provider for HUD, collects quarterly information, but it has not aggregated the information to obtain performance information over the span of a grant. HUD also has encountered significant delays in developing a database intended to gather additional performance information from grantees. Finally, citing resource constraints, HUD has conducted limited on-site oversight of grantees. As a result of a lack of verified and aggregated data or analysis and limited oversight, HUD largely was unable to tell how well the individual Youthbuild grantees performed.

HUD Has Not Collected or Analyzed Data That Could Help It Assess Grantees

HUD requires grantees to submit performance reports, including closeout reports, but HUD has not developed or fully utilized information— particularly from the closeout reports—that could help it assess Youthbuild grantee programs and has lacked resources to conduct comprehensive oversight of grantees. Such information serves as the basis for management and oversight of programs. For instance, HUD states in the instructions for filling out the closeout reports that the reports will be used to monitor and evaluate progress of grantees and programs. Further, the Government Performance and Results Act of 1993 (GPRA) shifted the focus of federal managers from the number of tasks completed or services provided to a more direct consideration of the outcomes or results of the program. GPRA provides a performance-based management framework for agencies to set goals; measure progress toward those goals; deploy strategies and resources to achieve them; and ultimately, use performance information to make the programmatic decisions necessary to improve performance. Accurate and complete performance data that document outcomes are needed to inform Congress and the public about the effectiveness of programs and help agencies conduct effective oversight. The combination of baseline information and intermediate outputs or outcomes can be used to show progress or contribution toward final outcomes, that is, the intended results of carrying out the program. For example, in Youthbuild, baseline information such as the number of participants entering a program with and without GEDs or high-school diplomas could be combined with intermediate outputs and outcomes such as graduation rates and jobs attained, which could then be linked to desired program outcomes such as the number of Youthbuild participants attaining economic self-sufficiency.

However, HUD has not set up a system to verify, aggregate, or analyze the data from closeout reports—information and analyses that would form the basis for assessments of grantee performance. Citing resource constraints, HUD filed the reports in the individual grant files in headquarters and field offices and did not further review them. In addition, some basic baseline data are missing from the closeout reports, including the number of participants who enter the programs with a GED or high-school diploma. Such information is necessary to get an accurate count of the number of participants receiving a GED degree or diploma through Youthbuild programs and also provides a basis for comparison with which performance can be measured. Furthermore, while closeout reports require that some performance outputs be collected, including the number of participants receiving GEDs or high-school diplomas, placed in jobs, or entering higher education programs, the reports do not collect other output information such as how many participants were placed in construction-related employment.

Citing resource constraints and a need to comply with the Office of Management and Budget (OMB) performance reporting requirements, HUD contracted with YouthBuild USA in 2004 to collect quarterly grantee performance information for all Youthbuild grantees. Programs administered by six federal agencies, including HUD and Labor, are subject to “common measures” reporting which OMB established to provide agencies with the ability to describe, in a similar manner, their core performance outputs across different grantee programs. The common measures include employment or education placements, degrees or certificates attained, and literacy and numeracy skills increased (increases in educational functioning). Consequently, YouthBuild USA provides HUD with program information including participant gender and ethnicity data, GED attainment statistics, grade level improvements, job or school placements, and the number of new and rehabilitated housing units that were completed. HUD uses the reports to provide performance information to Congress and meet OMB reporting requirements. However, because the information is captured on a quarterly basis, it only represents the performance of a program at a particular point in time and YouthBuild USA has not aggregated the information to obtain data for the term of individual grants. As a result, HUD has been largely unable to track program outcomes, but its data will be needed for ongoing program assessments.

Moreover, HUD has encountered significant delays in developing a database intended to provide additional performance outcomes on grantee programs. Recognizing this deficiency, HUD officials said that they have been planning for about 3 years to implement an electronic data collection system to help HUD measure the success of Youthbuild programs, along with its other nonhomeless competitive grant programs. However, the system’s implementation has been delayed because of funding and technical problems. As a result of not systematically collecting or analyzing needed information from closeout reports, relying predominately on limited point-in-time program information, and not being able to utilize a planned database to aggregate performance outcomes, HUD is limiting its ability to assess grantees. Further, although the YouthBuild Transfer Act shifted oversight of grants issued in fiscal year 2007 to Labor, HUD will be required to continue monitoring existing Youthbuild grants for approximately the next 3 years. Therefore, the data HUD has at its disposal or could develop will continue to be valuable for ongoing assessments of the program.

Citing Resource Constraints, HUD Performed Limited Oversight of Grantees

In addition to not having or utilizing available performance data, HUD has exercised limited on-site review and oversight of grantee performance. According to HUD’s Office of Inspector General, on-site monitoring is an essential tool for HUD (through CPD) to assess program performance and identify and address potential program problems. Such monitoring is also useful in motivating grantees to exercise sound judgment in carrying out their grant activities. GAO’s Standards for Internal Control in the Federal Government note that controls generally should be in place to ensure that ongoing monitoring occurs during normal operations. Monitoring should be performed continually, be ingrained in an entity’s operations, be used to assess the quality of performance over time, and support the prompt resolution of any identified problems.

CPD goals require on-site monitoring of only a small percentage of grants. For fiscal year 2003, the plan called for the field offices to monitor 35 percent of formula grants; 20 percent of competitive homeless grants; and 7 percent of competitive nonhomeless grants, including Youthbuild. HUD has since changed its baseline monitoring goals (requiring monitoring of 20 percent of grantees with active, open grants established since 2004) to give its field offices more flexibility to meet monitoring priorities based on local grantee knowledge. As a result, goals are no longer set by program. Field offices decide on the number of formula versus competitive grantees to be monitored based on risk analyses that evaluate each grantee on factors such as the size of the grant, perceived project management capacity, grantee responsiveness, and the extent of citizen complaints. HUD officials cited a lack of resources as the primary reason for not requiring the monitoring of a larger percentage of grantees. In 2006, according to HUD officials, field offices typically had one person working part time responsible for overseeing the individual Youthbuild grantees in the state, visiting no more than one or two grantees per year. As a result, field staff have not been able to comprehensively monitor grantees and verify the accuracy of the output data submitted by the grantees.

In general, we found that the monitoring reports that we reviewed (which HUD field offices produced) on grantees primarily addressed various compliance issues, rather than reporting on the overall performance of a particular grant. For example, several reports identified grantees’ failure to receive prior approval from HUD for changing proposed job training construction sites. Other reports identified inadequate record keeping, insufficient documentation, and lack of sufficient financial internal controls on the part of grantees. However, although a few reports did comment on some individual performance outputs, such as job placements or GEDs attained, most did not.

The lack of analysis of performance information of grantees also has implications for assessing the Youthbuild program overall, particularly because HUD has devoted few staff to overall program management and oversight. Since 2000, in headquarters, Youthbuild had a program director with one or two staff managing the program each year. During fiscal year 2006, with the expected program transfer to Labor, the HUD program manager was assigned to other duties, and a part-time program manager was assigned to oversee the 2006 grant process, in which HUD received 308 applications and awarded 74 grants. Although they are aware of the limited oversight of Youthbuild, CPD division officials said it has provided appropriate staff resources for program oversight of Youthbuild grantees, commensurate with the program’s size—the annual appropriation for which represents less than 1 percent of CPD’s usual annual budget of about $8 billion. However, without sufficient oversight HUD was unable to tell how the individual Youthbuild grantee programs performed.

Incomplete Data on Participants’ Outcomes Precludes an Overall Assessment of Program’s Success

HUD has not performed the programwide analysis needed to determine overall program success, and weaknesses in other assessments of the programs and data preclude an overall assessment of Youthbuild success. While limited in scope, three studies suggest Youthbuild helped youths achieve economic self-sufficiency. And, although YouthBuild USA tracks performance data, grantees do not have the follow-up information needed to measure performance outcomes. To augment limited existing data, we analyzed information in 245 closeout reports, but the results cannot be generalized to the entire program. We found a wide range of results in outputs such as job placements; however, because grantees generally did not follow up on participants, we were unable to determine long-term outcomes such as percentages of participants retaining employment. Moreover, while closeout reports include impediments to success and “best practices,” HUD did not systematically review this information or share it with YouthBuild USA. As a result, the lack of follow-up data and programwide evaluations make it very difficult to determine which programs and strategies have worked best and assess the performance of Youthbuild over time. Labor officials said they would consider that the closeout reports include outcomes such as jobs obtained and retained and said that they believe impediments and best practices should be evaluated and disseminated to grantees.

A Few Studies Document Individual Youthbuild Program Successes but Are Limited in Scope

A few studies have provided some evidence of the success of individual Youthbuild programs. Although subject to many limitations, the studies suggest that identified programs have achieved certain Youthbuild objectives. However, without comparable information for the entire population of Youthbuild grantees, it is not possible to determine the extent to which these outcomes and reported successes are reflective of what was achieved by the grantees not included in the studies. More specifically: In June 2004, researchers from Brandeis and Temple Universities published a study entitled, “Life after YouthBuild - 900 YouthBuild Graduates Reflect on Their Lives, Dreams, and Experiences,” in collaboration with YouthBuild USA. The study was based on a survey of graduates from 11 Youthbuild grant sites and included in-depth interviews with 57 graduates from eight of these sites. The researchers determined that, for those 882 participants (for a 23 percent response rate) who responded to the survey, many are doing well and were helped considerably by the Youthbuild program. Many participants found jobs or furthered their education and have volunteered in community activities. In addition, a number of participants surveyed indicated they also obtained intangible benefits from the program, such as increased self-esteem and had developed positive attitudes about themselves and their community. A limited nonresponse analysis was conducted by YouthBuild USA to compare the 882 survey responses and the 57 interviews with one site that had a 98 percent response rate. However, using one site that had a high response rate is not representative and cannot be generalized to the whole sample. As such, it is not known if the 77 percent who did not respond were doing more or less well than those who responded to the survey.

In August 2003, HUD issued a study entitled, “Evaluation of the Youthbuild Program,” in which a HUD contractor compared program results and costs per participant from the Youthbuild Program and four other federal employment and training programs—Job Corps, JOBSTART, both designed to serve disadvantage youth, and the youth components of the Job Training Partnership Act and Supported Work Demonstration programs. The researchers conducted their work at 20 Youthbuild sites, selected in part because they had received funding for at least two grants, obtained a variety of organizational sponsorships, built new or rehabilitated housing, and covered a variety of funding category sizes. The study found that 36 percent of the Youthbuild participants at these sites obtained jobs after enrolling and 29 percent obtained either a GED or high- school diploma. In addition, 12 percent of the Youthbuild graduates in the 20 sites reportedly went onto higher education. However, results at the five federal programs were not directly comparable because researchers obtained data from Youthbuild program participants at the time they exited the program while data from the other programs’ were obtained from tracking participants up to 3 years after leaving the program. In addition, the limited scope of this study precludes an overall assessment of the program’s success.

In February 2003, the state of Minnesota issued a study entitled, “Minnesota Youthbuild Program: A Measurement of Costs and Benefits to the State of Minnesota.” This study analyzed the costs and benefits of the state-funded Youthbuild program in Minnesota. The researchers measured post-program participant performance outcomes by tracking program year participants beyond graduation. They found that Youthbuild graduates earned more than double the minimum wage, whereas youth with similar characteristics who had not been through the program typically earned only the minimum wage. In addition, the graduates who had been adjudicated had a considerably lower rate of recidivism (rearrest, reconviction, or return to prison) than other youth involved with the correctional system. Using state administrative data to estimate the amount of revenues and savings generated through increased tax revenues and reduced prison costs, the study estimated first-year benefits to the state of about $1.5 million dollars versus costs of about $877,000. Over a 4- year period, the study estimated the total net benefit to the state at about $7.3 million (by 2006). The study was limited to costs and benefits directly related to state programs and did not include potential benefits such as reductions to welfare rolls, state-sponsored medical care, or other reductions in human services costs.

YouthBuild USA WebSTA Data Have Limitations, Particularly for Assessing Outcomes

Since 2002, YouthBuild USA has been using a Web-based system— WebSTA—to collect participant characteristics and performance data from its affiliates. According to YouthBuild USA officials, as of December 2006, about 90 percent of its affiliates were entering data into WebSTA to some extent. However, the value of YouthBuild USA’s WebSTA data is limited, largely because grantees self-report the data, not all grantees are meeting the requirement to send in information, and the information is not reported at the grant level. That is, if affiliates have more than one Youthbuild grant, the data for each grant are not reported separately. As a result, the data are not verified, comprehensive, or comparable. Also, because the grantees submit the information quarterly, the data provide only a point-in-time snapshot of the program’s performance and cannot be summed to provide annual data. Recognizing these limitations, according to YouthBuild USA’s President, upgrades are planned to be implemented in early 2007 that will allow grantees to enter data in WebSTA by individual grant, which in turn will allow YouthBuild USA to analyze grant- level performance.

We also found that the limited depth of the information collected restricts the value of WebSTA information for performance assessments. For example, WebSTA collects information on participant attendance, retention and graduation rates, and initial job and college placements and participant characteristics (such as welfare, public housing, and adjudication status) when participants enter Youthbuild programs. However, many grantees do not follow up on participants once they leave the programs. As a result, they often do not know if students placed in jobs were still working at those jobs weeks or months later. Also, several program directors we visited said that most of their program graduates who go on to higher education sign up for free remedial-level courses at nearby colleges. Although they have not had the funding to systematically follow the students’ progress, some said they generally have found that the students have had varying success in advancing to college-level credit courses.

Without grantee-level follow-up systems, WebSTA cannot track program outcomes that would reflect whether the graduates are achieving economic self-sufficiency—such as the percentage of program graduates who remove themselves from the welfare rolls, move out of public housing, or stay out of the court system. The system also does not collect follow-up information on the type of jobs attained (including construction), pay raises, benefits received, and college or trade degrees or certifications received. Analyzing such performance outcomes could provide, HUD, Labor, and other interested parties with the measures needed to evaluate overall program performance and success.

Closeout Reports We Analyzed Provided Little Detail on Long-Term Outcomes Because Grantees Are Not Required to Follow up on Graduates

Because HUD had not aggregated or analyzed data for Youthbuild, we obtained and analyzed 245 closeout reports that grantees submitted to HUD. Figure 6 provides selected performance information that we summarized from the closeout reports. However, because HUD has not established objective performance goals for the program, which would serve as baseline criteria against which to judge performance, the information related to the experience of the 12,863 participants who entered the program under the 245 grants gave us limited perspectives about how individual programs were working. In addition, because we had information on only 245 (or 46 percent) of the grants, the results cannot be generalized to the entire program.

As shown in figure 6 the percentages for participants graduating, receiving a GED, or being placed in employment varied widely. As previously discussed, HUD has not set benchmarks, and critical data elements (baseline data) are missing in the closeout reports. For instance, it is not possible to obtain an accurate number of diplomas and GEDs attained from the closeout reports because the reports do not contain data on the number of participants that enter the program with a high-school diploma or GED. As result, we could not determine how well the programs (except for those with the very highest percentages) performed individually or in comparison with each other.

The closeout reports also do not provide consistent and reliable data on the number of units rehabilitated or built by Youthbuild participants. A basic purpose for which Youthbuild was authorized was to expand the supply of affordable housing. According to the closeout reports, 158 grantees reported that their participants built a total of 1,483 new units, while 139 grantees reported that their participants rehabilitated a total of 2,238 units. Some of the Youthbuild program directors with whom we met said that during a typical reporting period they would complete from one to four new or gut-rehabilitated housing units. Other directors said that when a program reports about 200 units rehabilitated, it generally means the students are performing minor repetitive tasks such as changing locks or painting a limited number of interior walls. As a result, the number of units rehabilitated likely is overstated in the closeout reports. Although construction training is a major part of the Youthbuild program, HUD’s closeout reports also do not track the number of graduates obtaining construction-related jobs or specify the types of jobs obtained (construction or otherwise).

In addition, HUD officials told us that grantees are not required to follow up on participants after they leave the program. While some grantee officials with whom we met recently had attempted to establish follow-up systems, most grantees cited funding constraints as the reason for not following up on graduates. Several officials further noted that it was difficult to locate many participants after they had left the program. However, as a result, it is not possible to obtain comprehensive post- program results.

Finally, some program directors with whom we spoke cautioned that any follow-up system should not be strictly focused on students obtaining higher education degrees or obtaining high-paying jobs. They emphasized that such goals were beyond the abilities of some of their participants, but that moving from a fifth-grade to a ninth-grade reading level or obtaining a service sector job sometimes was a real victory and should be counted as a success.

Limited Information We Developed from Site Visits Pointed to Varied Results for Programs, and Also Provided Little Detail on Long-Term Outcomes

The information we obtained from our visits also demonstrates varied results for continued employment and academic success, and little detail on long-term success. As with our closeout report analysis, the information cannot be generalized to the universe of Youthbuild grantees—although YouthBuild USA officials told us that they believe we visited some of the more innovative grantees. For example, although not required, some of the nine grantees we visited used other funding sources, such as United Way or AmeriCorps funds, to establish a participant follow- up system. As a result, some of the grantees were able to provide us with varying amounts of information on the status of program graduates. For example: One grantee, which was able to track individuals who completed the program since 2001, provided us with data that showed 70 of the 82 program participants were either employed or pursuing higher education. Seven of the 70 participants were working and going to school. The hourly wages of those employed ranged from $7 to $13.75. However, the grantee did not have information on the types of employment.

Another grantee that recently established a follow-up system showed that 11 of the 26 program graduates who started in 2004 had obtained employment. The positions ranged from pizza delivery person to union carpenter. However, only the union carpenter and a graduate hired by the program as a construction trainer were working in the construction field. The hourly wages of the graduates ranged from $5.50 to more than $16 per hour for a graduate hired by the grantee.

Another grantee used WEBSTA to track continuing education and work history, including each job held. The program director told us that such a follow-up system allows the program to spot trends, which in turn can be used to fine-tune the program. Of 60 graduates from 2002 to 2005, the program continually has tracked 53 and reported that 8 are receiving higher education, 24 are working in nonconstruction jobs, and 16 are working in construction jobs. The pay for the graduates ranged from $6.50 to $15 per hour. The grantee continues to keep in touch with the graduates by helping them with housing, legal aid, school advice, tutoring, and help toward achieving a GED.

Grantees are not required to collect such post-program follow-up information. Yet, both grantees and program participants emphasized that the Youthbuild program created a very supportive atmosphere that many participants had never experienced prior to joining a program. During our visits, we met with students who credited the program with improving their lives by either helping get them off drugs, get out of the gang lifestyle, or disengage from other activities that likely could have resulted in incarceration or worse. The studies we mentioned previously had similar findings. However, it is difficult to gain insight into how many program participants remain drug free, crime free, in advanced degree programs, or working in construction or other trades without post-program performance data. It is also difficult to gain insight into which program strategies are most effective for serving these youth.

HUD Does Not Review or Disseminate Impediments and Best Practices Identified in Closeout Reports That Could Help Improve Program Performance

Citing resource constraints, HUD did not systematically review the impediments and best practices identified in closeout reports or share the information in them with YouthBuild USA. In particular, identifying best practices (or the converse, impediments) can provide a model for improvement for other organizations with similar missions. The closeout reports contain a section addressing what the grantees considered to be best practices. For example: Several grantees set-up educational opportunities, such as GED preparation, at local colleges for participants. Two grantees noted that being on a college campus provided a positive experience for the participants. Two grantees noted that they made an arrangement with a local technical college to offer free enrollment to all participants that successfully completed the Youthbuild program. Another grantee noted that working through community college GED programs allowed them to lower their per-student cost and serve more students.

One grantee established an alumni council consisting of elected representatives from each of its seven graduating classes. All graduates receive quarterly mailings that include an alumni newsletter, new job and educational opportunities, and a quarterly calendar of events.

Another grantee has maintained a program serving graduates for up to 12 months to provide on-going employment support, including assistance in securing a second job in the event that the initial placement did not work out.

One grantee started a “drug free” club and found that students really wanted to participate and were anxious to become drug free and would publicly announce their success.

Another grantee worked extensively recruiting area college students to volunteer to help with tutoring and program community service activities.

Closeout reports also contain information on impediments that grantees encountered in implementing and operating the programs. Impediments to program success varied in scope and severity. Some of the more significant impediments mentioned by numerous grantees included the following.

Turnover of key staff, especially on-site construction managers; Negative and apathetic attitudes among participants; Alcohol and drug abuse issues among participants; Difficulties in dealing with municipal organizations to obtain building permits and contractual difficulties, which delayed construction training activities; and Acceptance of students with very low educational levels, which made obtaining high GED levels very difficult.

HUD’s contract with YouthBuild USA specifically calls for the contractor to assist HUD in the management, supervision, and coordination of Youthbuild programs by strengthening Youthbuild program design and disseminating information on best program practices. Although analyzing the impediment and best practices falls within the scope of HUD’s contract with YouthBuild USA, two CPD program officials said that HUD never considered sharing the reports with its technical services provider. YouthBuild USA’s chief operating officer said that the organization would welcome access to the closeout reports. The officer said his staff already spent many hours each month trying to identify best practices from Youthbuild grantees and would be anxious to analyze the closeout reports for both the problems and successes identified and forward the information to interested grantees. YouthBuild USA’s President told us that she not only agreed that reviewing the closeout reports for problems and “best practices” would be beneficial, but that she believes the closeout reports needed to “be analyzed to get to the bottom line because the total impact (outcomes) of a grant could not be determined from data provided on a rolling quarterly basis.” As a result of not sharing the closeout reports with YouthBuild USA, HUD may have missed opportunities to improve the program as a whole and help grantees improve performance.

Labor Officials Indicated They Will Consider Using Information from Closeout Reports to Assess and Improve YouthBuild Performance

According to Labor officials, they will consider analyzing information from the closeout reports to assess and improve the performance of the YouthBuild program. Until 2006, Labor assessed past performance of its youth employment and training grantees through measures required under WIA. For example, under the WIA Youth and Job Corps programs, Labor tracked and analyzed 6-month retention and earnings wage change information. Labor officials told us that they would use OMB’s common measures to track and analyze the performance of YouthBuild grantees but said that they have not finalized the information they would require YouthBuild grantees to collect. The officials indicated to us that they would consider developing post-program performance outcomes such as the types of employment graduates attained and retained (including construction), wage rates, and college or trade degrees or certifications received. Although Labor may incur some additional costs in developing such information, the ability to measure program success would improve. Labor officials also noted that the information currently contained in the closeout reports would be considered with other factors they deem as pertinent to assessing grantees’ performance. However officials said they would also likely consider modifying the closeout reports to include other program participant data, such as degrees obtained prior to program entry and details on the types of construction jobs obtained upon program completion.

Labor officials also acknowledged that information on program impediments and best practices should be evaluated and passed to grantees and that performance data needed to be aggregated, verified, and analyzed to determine which programs and strategies worked best. As of October 2006, Labor had not determined whether its Employment and Training Administration or an outside contractor would have primary responsibility for collecting, aggregating, and disseminating the information.

Youthbuild Grantees Have Experienced Varied Success in Obtaining Outside Funding Sources, and Many Cited Continued HUD Funding as Crucial

Our analysis of available closeout reports showed that the grantees have had varying success in obtaining additional funding sources. In assessing grant applications, HUD gave consideration to program applicants that were able to secure outside funding. Applicants received points for leveraging nonhousing resources compared with the amount of Youthbuild funds requested in the application. Overall, the median amount spent by a HUD Youthbuild grantee was $400,000 in HUD funds; the HUD grant represented about 48 percent of the average total spending per grantee, which included additional funding sources of about $430,000. However, success in obtaining additional funds varied widely, from 21 grants reporting no additional funding sources to 40 reporting more than $1 million (see fig. 7).

According to information in the closeout reports and the officials from the grantees that we visited, the additional funding they obtained largely came from federal, state, and local agencies, businesses, and charitable organizations. According to a YouthBuild USA official, states have provided funding to local Youthbuild grantees over the past several years. For instance, the Massachusetts State Department of Education has provided funding in its budget for the state’s Youthbuild program. The Massachusetts funds are provided to existing Youthbuild programs, based on a formula the state developed in conjunction with a state coalition of Youthbuild grantees. In Minnesota, 10 Youthbuild programs in 2005 shared $754,000 in state Youthbuild program funding, which the state supplemented with more than $3 million in matching funds from local partners. In Minnesota as elsewhere, urban grantees have had greater opportunities to obtain funding or in-kind support from local businesses and charitable organizations than rural grantees that typically have had access to fewer businesses and might not be a focus of charitable activity.

Regardless of their success in obtaining outside funds, several grantees we visited stated that their Youthbuild program could not be sustained without the HUD grant because the HUD funding allowed them to attract the leveraged funds. Also, several grant officials said that many existing and potential contributors would not be willing to continue funding if they perceived that the program would not be able to continue without the HUD grant. One exception (which we did not visit) was in Minnesota where the state Youthbuild program director said that only 4 of the state’s 10 programs have had HUD funds in the past, including 2 programs which received HUD funding in 2004 through a HUD grant administered by the state. The program director added that the structure of the Minnesota state Youthbuild program is similar to that of HUD’s, although the programs are typically smaller, having from 10 to 25 students.

Information from YouthBuild USA generally agrees with grantees’ contention about the need for continuous HUD funding. Overall, YouthBuild USA found that 60 percent of programs that HUD did not fund for 2 consecutive years ceased operations, and 90 percent of programs not funded for 3 consecutive years ceased operations. According to YouthBuild USA, 173 of 462 grantees funded since 1994 currently are operating with active HUD grants. YouthBuild USA, also found that grantees with continuous HUD funding achieved higher outputs, such as GED attainments and job placements, than the ones that did not receive one or more follow-on grants. According to some grantee officials, that success is partly due to continuous funding that allows grantees to develop infrastructure and maintain experienced staff to establish a better program. In its fiscal year 2005 Performance and Accountability Report, HUD noted the benefits of making grants to previous awardees: “Having established programs and experience running a Youthbuild program, these grantees are more efficient in enrolling students, resulting in a greater number of youth trained.” Furthermore, Youthbuild program directors with whom we met emphasized the intangible benefit that such established programs provide such as a family atmosphere that allows graduates to return for advice and support as needed—a benefit that goes away when programs close.

HUD has not kept statistics to find out why some grantees did not reapply for additional grants or why grantees did not qualify for follow-up grants. Through discussions with a nonprobability selection of 23 of the 100 grantees that closed programs, we found that some did not reapply because they were no longer interested in dealing with at-risk youth and others closed due to poor management. But most of the closed grantees with whom we spoke said they closed because they lost HUD funding for one or more years and were unable to obtain sufficient private funds to make up for the lost funding. Of the sites we contacted, two were still in operation after ceasing to receive HUD funds—but they appeared to be the exception. Both were already operating youth technical training programs prior to their first HUD Youthbuild grant.

In addition, officials from several grantees that also received funding from AmeriCorps stated that they preferred the AmeriCorps concept of providing dedicated funds for 3 years. These officials stated that knowing that funding levels were longer-term provided more continuity to their program, enabled better planning, and allowed them to build a cushion through leveraged funding in the event that they did not receive a HUD grant for a year or two. Finally, when the question of multiyear funding was raised at a Web-based listening session in December 2006, designed to help Labor identify Youthbuild program administration issues and concerns, most participating grantees spoke out in favor of multiyear funding.

Conclusions

HUD has not taken advantage of several opportunities to use existing information for purposes of program assessment and oversight. Such opportunities should not be ignored, particularly because HUD has limited resources with which to assess and oversee current grantees. Specifically, HUD has not verified, aggregated or analyzed closeout reports, which grantees are required to submit. HUD also has conducted limited oversight of grantees, citing resource constraints; the impending transfer of oversight to Labor bodes no change to this specific situation. However, given these conditions, analysis of existing closeout reports would help provide crucial information on the performance of the Youthbuild programs, particularly in the absence of a centralized HUD database that could be used to track program performance. At a minimum, the closeout reports form a logical basis for providing useful information that HUD, Labor, and other interested parties can use to judge the performance of Youthbuild. HUD still will maintain an active involvement for several more years, so coordination between Labor and HUD will be critical to producing reliable assessments of how successful the program has been and will be. HUD also does not share reported constraints and best practices with the technical services contractor it pays for performing such work and disseminating best practices. By not utilizing performance data and sharing information, HUD has missed opportunities to determine whether the overall Youthbuild program has achieved its intended results, among them enabling disadvantaged young adults to gain economic self- sufficiency. Therefore, Congress and the public also lack the information needed to make such determinations.

Labor has experience in managing youth training and education programs and collecting performance information, and Congress transferred Youthbuild to Labor in part to better align the program with existing federal workforce development and youth training programs. In offering perspectives for Labor to consider, we stress the value that closeout reports have in the absence of more comprehensive and programwide data, although the existing reports lack certain elements that could help program managers determine and report on outcomes. The reports do contain valuable information on impediments and best practices, which could provide both program managers and grantees with much broader perspectives on how successful the program has been and suggestions for improvements. Labor has indicated that it would consider developing post- program performance outcomes such as the types of jobs graduates attained and retained (including construction), wage rates, and college or trade degrees or certifications received. As a result, while there may be some additional costs involved, the ability to measure program success would improve. And, because alternatives to the HUD data are limited, the grant-level assessment discussed above could act as the building blocks of an overall program assessment. Labor also has agreed that information on impediments and best practices would be valuable and ought to be passed on to grantees, but has not yet determined how this might be done.

Finally, data on grantee success in attracting other funding, although limited, offer more perspectives for Labor to consider because outside funding helps leverage limited federal dollars. While many grantees have obtained extensive outside funding and support, YouthBuild USA has found that very few operated for more than a few years without continued HUD funding. Tellingly, their data show that 90 percent of programs not funded by HUD for 3 consecutive years ceased operations. According to YouthBuild USA officials, their data indicate that grantees with repeat awards also produced better outcomes such as a higher level of job placements. Further, when HUD funding stopped, potential contributors became concerned that grantees would no longer stay in business and were reluctant to jeopardize their own funds. These data, while not definitive, do suggest that multiyear funding has the potential to produce dividends for the program. Many grantees expressed similar belief in the benefits of longer term funding at a Department of Labor YouthBuild program listening session.

Recommendations for Executive Action

To improve the reporting and assessment of performance for Youthbuild grantee programs and develop the bases for an overall assessment of the program, we are making the following three recommendations: that the Secretary of HUD analyze closeout reports by grant and share information on identified problems and “best practices” with its technical services contractor and Labor; that the Secretary of Labor develop and monitor post-program performance outcome measures for the YouthBuild program, such as the types of employment graduates attained and retained, wage rates, and degrees or certifications received, and share the data with the grantees; and that the Secretary of Labor consider whether multiyear funding could be useful in helping YouthBuild grantees attract additional outside funding.

Agency Comments and Our Evaluation

We provided HUD and Labor with a draft of this report for review and comment. HUD’s comments are summarized below. Labor’s Assistant Secretary for Employment and Training provided written comments that are presented in appendix I.

Officials from HUD’s Office of Community and Planning and Development provided comments. They stated that they did not disagree with GAO’s overall findings, but added that HUD’s oversight responsibility was limited by resource constraints. They cited the fact that the Youthbuild program represents 0.63 percent of the Community Planning and Development Division’s entire portfolio. Nevertheless, because HUD will have monitoring responsibilities for existing grants for 3 years or more, we believe it is important that HUD analyze the closeout reports in order to provide Labor with information about how current grantees are performing.

Labor’s Assistant Secretary for Employment and Training wrote that she believes that the report will be very useful to Labor as it assumes responsibility for administering the YouthBuild program. She noted that Labor agreed with our recommendation to develop and monitor post program performance outcome measures for the YouthBuild program and share these data with grantees. She added that Labor was building a Web- based Management Information System to collect and report on participant information. Finally, she wrote that Labor agreed that there could be value on multiyear funding and that the Department was considering such an approach for existing grantees.

We are sending copies of this report to other interested congressional committees and the Secretaries of Housing and Urban Development and Labor. We will also make copies available to others upon request. The report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have questions regarding this report, please contact me at (202) 512-4325 or shearw@gao.gov. Contact points for our Offices of Public and Congressional Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the U.S. Department of Labor

Appendix II: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Andy Finkel, Assistant Director; Rich LaMore, John McGrail, Marc Molino, Luann Moy, Barbara Roesmann, and Tom Taydus made key contributions to this report.