Youthbuild Program: Analysis of Outcome Data Needed to Determine Long-Term Benefits
Published: 2007-02-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1993, the Department of Housing and Urban Development (HUD) has provided funding for Youthbuild, a competitive grant program that trains and educates disadvantaged youth and helps build low-income housing. In 2006, Youthbuild was transferred to the Department of Labor (Labor) to better align the Youthbuild program with existing youth workforce and training programs. In response to concerns about the overall quality of Youthbuild, a Senate report directed GAO to assess the program. GAO's objectives included (1) evaluating how HUD assessed and oversaw the program, (2) determining what results the program achieved, and (3) assessing how successful grantees were in obtaining outside funding. GAO analyzed Youthbuild performance data, visited Youthbuild sites, and interviewed agency officials.
section_title: What GAO Found
paragraphs: While HUD requires grantees to report basic performance data, such as the number of program participants and graduates and job placements, HUD has not aggregated or analyzed the data and conducted limited oversight of grantees. According to HUD officials, they did not have staff available to analyze the closeout reports that grantees must submit, and a lack of resources also limited oversight of grantees. The monitoring HUD did primarily focused on compliance with program requirements such as documentation rather than on performance. As a result, HUD largely was unable to tell how the individual Youthbuild grantee programs performed. Limited outcome data preclude any overall assessment of the performance of the Youthbuild program; further, the few other analyses available such as the one GAO did in this study to augment limited existing data cannot be generalized programwide. GAO analyzed 245 closeout reports, representing 46 percent of the grantees or 12,863 participants. While GAO could determine percentages of participants who received high school diplomas or were placed in jobs, GAO could not determine outcomes over time, partly because the reports lacked baseline information and grantees were not required to and generally did not follow participants after graduation. Further, while closeout reports include information about impediments to program success and "best practices," HUD did not systematically review this information or share it with its primary technical assistance contractor. Consequently, the lack of programwide evaluations, follow-up data, and dissemination of best practices make it very difficult to assess the performance of Youthbuild over time and determine which programs and strategies have worked best. Reporting on post-program performance outcomes, such as the number of participants placed and retained in construction-related employment, could increase the value of the closeout reports and better measure program results. Labor officials indicated that they would consider including such measures for program reporting. Grantees had varying success in obtaining funds from outside sources, but YouthBuild USA data suggest that continued (multiyear) HUD funding was critical to sustaining grantee operations and attracting leveraged funds. Grantees' success in obtaining additional funds varied widely, from 21 grants reporting no additional funding sources to 40 reporting more than $1 million. While most grantees have generated outside funding, YouthBuild USA reported that most grantees have had difficulty continuing operations without continued HUD funding. Their data show that 90 percent of grantees ceased operations if not funded for 3 consecutive years by HUD. Further, YouthBuild USA also noted that grantees with follow-on HUD funding achieved better performance outcomes, such as higher rates of job placements, than grantees that did not receive subsequent HUD funding.

section_title: Letter
paragraphs: Since 1993, the Department of Housing and Urban Development (HUD)  has provided funding for Youthbuild, a “second chance” program that  trains and educates disadvantaged youth and helps build housing for low-  and very low-income families across the United States. As stated in the  authorizing legislation, the purposes of the Youthbuild program are (1)  expanding the supply of affordable housing by utilizing economically  disadvantaged young adults, (2) providing economically disadvantaged  young adults with opportunities for meaningful work and service to their  communities, (3) enabling economically disadvantaged young adults to  obtain the education and employment skills necessary to achieve  economic self-sufficiency, and (4) fostering the development of leadership  skills and commitment to community development among young adults in  low-income communities. Although a few studies have shown instances  where Youthbuild graduates credit the program for helping them  overcome difficult circumstances, gain critical job skills, and continue  their education, concerns have been expressed about whether HUD’s  program oversight has been sufficient to adequately measure program  outcomes.
The Housing and Community Development Act of 1992 authorized the  Youthbuild program. The program was based on the Youth Action  Program developed in East Harlem, New York, in the late 1970s in which  high school dropouts received academic training and learned job skills by  renovating abandoned buildings. The YouthBuild Transfer Act shifted  program oversight of grants issued beginning in fiscal year 2007 to the  Department of Labor (Labor)—in part, to better align the program with  existing federal workforce development and youth training programs. The  Transfer Act also expanded the program’s scope to allow construction and  rehabilitation of community facilities in addition to housing. HUD will  continue to oversee Youthbuild grants funded through fiscal year 2006.  That is, HUD’s oversight responsibility could continue for several more  years since grants can remain active for 3 years or more.
Noting concerns regarding the overall quality of the Youthbuild program,  the Senate report accompanying the fiscal year 2006 Transportation,  Treasury, Housing and Urban Development, the Judiciary, the District of  Columbia, and Independent Agencies Appropriations Act directed us to  comprehensively assess the Youthbuild program’s overall success,  including its ability to attract local support and funding. Specifically, the  objectives of this report are to (1) describe how HUD makes Youthbuild  program grants; (2) evaluate how HUD assesses and oversees the  Youthbuild program; (3) determine how successful the Youthbuild  program has been in achieving desired results, such as providing the  education and job training needed to achieve economic self-sufficiency  and supplying affordable housing; and (4) assess how successful local  Youthbuild organizations have been in obtaining private funding and  support.
We focused our analysis on reviewing HUD’s grant making and oversight  processes and evaluating available data for measuring program success.
To determine how HUD makes annual program grants, we reviewed the  application and review process HUD uses to identify and select  organizations for grant awards. To evaluate how HUD assesses and  oversees the program, we interviewed HUD officials at the national and  field offices. We also obtained information on the extent to which grantees  are monitored and the results of the monitoring. Further, we obtained  information on the type of performance data grantees submit to HUD and  HUD’s role in reviewing the information. We also obtained funding and  grant award information from HUD. We assessed the reliability of these  data by comparing the information with other data sources to identify  obvious problems with completeness or accuracy, and interviewing  knowledgeable agency officials about the data. We determined that the  data were sufficiently reliable for the purposes of this report. To develop  information on how successful the Youthbuild program has been, we  consulted relevant literature, including three studies of the Youthbuild  program’s effect on participants’ lives. However, the studies were limited  in scope and methodology and constrained by the absence of a database  containing performance information from all the grantees in the  Youthbuild program. We also interviewed officials and received  information from HUD’s primary technical assistance contractor,  YouthBuild USA. Further, because HUD lacked a centralized database on  grantees’ program performance results, we analyzed 245 grant closeout  reports submitted by grantees to HUD from April 2000 to August 2006. We  also visited nine Youthbuild programs in California, Illinois,  Massachusetts, Pennsylvania, Rhode Island, and Washington, D.C. The  sites were chosen in part because the programs varied in size, leadership  style, and teaching approach, including the charter school format. The  closeout reports we reviewed represented about 46 percent of the closed  grants awarded from 1997 through 2003. We were unable to review the  remainder of the reports either because HUD was not able to provide a  copy of the closeout reports, the grantee had not submitted a closeout  report, or the report did not have sufficient information for us to use. In  addition, we reviewed monitoring reports covering 16 grantees prepared  by HUD field office staff as part of their program oversight duties. To  assess how successful local Youthbuild organizations have been in  obtaining private funding and support, we analyzed the closeout reports to  determine the extent to which the individual grantees obtained additional  funding. From a list of 100 grantees from YouthBuild USA that closed their  operations, we obtained information on a nonprobability selection of 23  grantees by directly contacting them or from other knowledgeable sources  to determine how HUD funding decisions contributed to their closure. We  selected and contacted grantees primarily based on those that had  received more than one grant and which we were able to locate because  they were still providing other social services. We also reviewed literature  and reports on Labor’s youth employment and training programs and met  with officials from Labor, where oversight of the Youthbuild program will  be transferred. We conducted our work in Boston and Somerville,  Massachusetts; Washington, D.C.; and various Youthbuild locations in the  states noted above from March 2006 through February 2007 in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: CPD manages the Youthbuild program. CPD programs generally fall into  three categories: (1) formula grants, of which the largest are the  Community Development Block Grant and the HOME Investment  Partnership Program, which account for the majority of CPD funding; (2)  competitive homeless programs; and (3) competitive nonhomeless  programs, including Youthbuild. While no staff are dedicated solely to  Youthbuild oversight, CPD monitors the program as part of its oversight  responsibilities for 22 programs and initiatives, which 829 staff (233 in  headquarters and 596 in field offices) administered as of the beginning of  fiscal year 2007. In fiscal year 2006, Congress appropriated $19.1 billion for  the CPD programs and initiatives, of which Youthbuild represents a small  portion. The first program appropriation for Youthbuild was $40 million  for fiscal year 1993. Appropriations reached a high of $65 million in fiscal  year 2002, but dropped to $49.5 million in fiscal year 2006, the last year in  which HUD was authorized to award grants (see fig. 1).
HUD has provided program funding for Youthbuild through an annual  competitive grant application process. Any applicant can receive up to a  $700,000 grant; however, new applicants and rural/underserved applicants  primarily receive grants up to $400,000. Grant awards can last up to 30  months (or longer if the grantee is given an extension), but some grantees  use the funds at a much quicker rate and usually reapply every year, while  others may take the full 30 months.
Prospective grantees are state, private, nonprofit, and local entities that  implement the Youthbuild program by serving low-income youth, ages 16  to 24, who demonstrate educational need—either by virtue of being high- school dropouts or by justifying their need for inclusion in the program.  According to program regulations, at least 75 percent of the participants  must come from very low-income families and have dropped out of high  school. The program allows up to 25 percent of the youth to have a high- school diploma or have a slightly higher income; however, data developed  by YouthBuild USA show that most organizations report that less than 25  percent of participants have high-school diplomas or General Equivalency  Diplomas (GED) upon entering their programs. Also, according to data  from YouthBuild USA, a number of the youths have been adjudicated  (judicially determined to be delinquent), were on welfare upon entering  the program, or lived in public housing (see fig. 2). Other studies we  reviewed identified similar participant characteristics. Program  regulations require participants to spend 50 percent of their program time  receiving job training on construction sites, building affordable housing,  and 50 percent receiving academic training. Recently, some of the  Youthbuild grantees have added nonconstruction programs to their  offerings, such as in the areas of computer repair and health care. Funding  for the additional programs has come from AmeriCorps.
In 1994, YouthBuild USA, located in Somerville, Massachusetts, became  the primary technical assistance contractor for the Youthbuild program.  YouthBuild USA is a national support center and intermediary for the  program. As such, it provides training, develops materials and handbooks  to help grantees replicate the program, conducts peer-to-peer seminars,  and disseminates materials on best practices. YouthBuild USA also has  established an affiliated network that grantees can choose to join. As of  September 2006, 107 of 173 Youthbuild grantees with active grants were  affiliates. Since 2002, YouthBuild USA has been using a Web-based system,  WebSTA, to collect participant characteristic and performance data from  its affiliates.
Among the purposes of transfering the YouthBuild program to Labor was  better alignment of existing federal workforce and youth training  programs. Labor’s Employment and Training Administration offers  programs that assist disadvantaged youths, similar to the programs that  Youthbuild grantees offer. The Workforce Investment Act (WIA) of 1998  established a variety of programs to serve low-income youths who face  barriers to employment. Labor allocates funds for the youth service  programs to state and local areas based on a distribution formula. The  grants are made to states, and the states in turn allocate funds to  approximately 600 local workforce investment boards. The amount of  funding available to states is determined by the distribution of  unemployed individuals and disadvantaged youths by state. From 2000  through 2005, Labor administered the Youth Opportunity Grant program,  which also was authorized by WIA. This grant program focused on  improving education and employment opportunities for at-risk youths,  ages 14 to 21, in high-poverty areas. Funding for the program was  eliminated beginning in fiscal year 2004. In addition, Labor administers the  Job Corps program, which began in 1964, and provides education, training  and support services to economically challenged youths, ages 16 through  24, who face multiple barriers to employment. Private companies—chosen  through competitive contracting processes—and other federal agencies  operating under interagency agreements with Labor, operate the Job  Corps centers (located throughout the United States and Puerto Rico) on  behalf of Labor.
section_title: HUD’s Grant-Making Procedures Incorporated Legislative Criteria
paragraphs: Each year, HUD staff rated and ranked grant applications for serving  economically distressed areas based on factors outlined in the enacting  legislation and detailed in an annual NOFA. The Housing and Community  Development Act of 1992 required HUD to establish selection criteria such  as the qualifications or potential capabilities of applicants, the feasibility  of the proposed Youthbuild program, the potential for developing a  successful Youthbuild program, and the applicants’ commitment to obtain  outside resources.
Accordingly, HUD annually issued a NOFA that described the grant  application process, outlined criteria for grant awards, and included  scoring factors (see fig. 3). The major factors considered were the capacity  and relevant experience of the organization (performance), need and  extent of the problem, applicant’s soundness of approach, leveraging of  nonhousing funds, and ability to show how it would achieve and measure  results. Under these factors, grant applicants had to consider area poverty  and high-school dropout rates, describe their management and training  capabilities, and offer proof of access to property and the existence of  partners to build low-income housing. HUD issued its last NOFA for the  Youthbuild program on March 8, 2006, and announced the winning  grantees in November 2006.
Since fiscal year 2001, HUD has grouped applicants into three separate  scoring categories—new, general (that are typically awarded to  established Youthbuild organizations), and rural/underserved area  grantees. Figure 4 shows the awards by category. The total numbers of  grants as well as the grants by the individual categories have varied by  year.
The awards process also has been competitive. HUD historically funded  less than one-third of all the applicants (see fig. 5). Since 1996, the number  of applicants funded ranged from about 7 to 44 percent.
section_title: HUD Oversight of Grantees’ Performance Has Been Limited
paragraphs: Although HUD requires grantees to submit closeout reports, which are  intended to help evaluate the programs, it has not verified, aggregated, or  analyzed data from those reports. Further, the closeout reports do not  include other data that could help determine participant characteristics  upon entry or the types of jobs they received after completing the  program. YouthBuild USA, the technical services provider for HUD,  collects quarterly information, but it has not aggregated the information to  obtain performance information over the span of a grant. HUD also has  encountered significant delays in developing a database intended to gather  additional performance information from grantees. Finally, citing resource  constraints, HUD has conducted limited on-site oversight of grantees. As a  result of a lack of verified and aggregated data or analysis and limited  oversight, HUD largely was unable to tell how well the individual  Youthbuild grantees performed.
subsections: section_title: HUD Has Not Collected or Analyzed Data That Could Help It Assess Grantees
paragraphs: HUD requires grantees to submit performance reports, including closeout  reports, but HUD has not developed or fully utilized information— particularly from the closeout reports—that could help it assess  Youthbuild grantee programs and has lacked resources to conduct  comprehensive oversight of grantees. Such information serves as the basis  for management and oversight of programs. For instance, HUD states in  the instructions for filling out the closeout reports that the reports will be  used to monitor and evaluate progress of grantees and programs. Further,  the Government Performance and Results Act of 1993 (GPRA) shifted the  focus of federal managers from the number of tasks completed or services  provided to a more direct consideration of the outcomes or results of the  program. GPRA provides a performance-based management framework  for agencies to set goals; measure progress toward those goals; deploy  strategies and resources to achieve them; and ultimately, use performance  information to make the programmatic decisions necessary to improve  performance. Accurate and complete performance data that document  outcomes are needed to inform Congress and the public about the  effectiveness of programs and help agencies conduct effective oversight.  The combination of baseline information and intermediate outputs or  outcomes can be used to show progress or contribution toward final  outcomes, that is, the intended results of carrying out the program. For  example, in Youthbuild, baseline information such as the number of  participants entering a program with and without GEDs or high-school  diplomas could be combined with intermediate outputs and outcomes  such as graduation rates and jobs attained, which could then be linked to  desired program outcomes such as the number of Youthbuild participants  attaining economic self-sufficiency.
However, HUD has not set up a system to verify, aggregate, or analyze the  data from closeout reports—information and analyses that would form the  basis for assessments of grantee performance. Citing resource constraints,  HUD filed the reports in the individual grant files in headquarters and field  offices and did not further review them. In addition, some basic baseline  data are missing from the closeout reports, including the number of  participants who enter the programs with a GED or high-school diploma.  Such information is necessary to get an accurate count of the number of  participants receiving a GED degree or diploma through Youthbuild  programs and also provides a basis for comparison with which  performance can be measured. Furthermore, while closeout reports  require that some performance outputs be collected, including the number  of participants receiving GEDs or high-school diplomas, placed in jobs, or  entering higher education programs, the reports do not collect other  output information such as how many participants were placed in  construction-related employment.
Citing resource constraints and a need to comply with the Office of  Management and Budget (OMB) performance reporting requirements,  HUD contracted with YouthBuild USA in 2004 to collect quarterly grantee  performance information for all Youthbuild grantees. Programs  administered by six federal agencies, including HUD and Labor, are  subject to “common measures” reporting which OMB established to  provide agencies with the ability to describe, in a similar manner, their  core performance outputs across different grantee programs. The  common measures include employment or education placements, degrees  or certificates attained, and literacy and numeracy skills increased  (increases in educational functioning). Consequently, YouthBuild USA  provides HUD with program information including participant gender and  ethnicity data, GED attainment statistics, grade level improvements, job or  school placements, and the number of new and rehabilitated housing units  that were completed. HUD uses the reports to provide performance  information to Congress and meet OMB reporting requirements. However,  because the information is captured on a quarterly basis, it only represents  the performance of a program at a particular point in time and YouthBuild  USA has not aggregated the information to obtain data for the term of  individual grants. As a result, HUD has been largely unable to track  program outcomes, but its data will be needed for ongoing program  assessments.
Moreover, HUD has encountered significant delays in developing a  database intended to provide additional performance outcomes on grantee  programs. Recognizing this deficiency, HUD officials said that they have  been planning for about 3 years to implement an electronic data collection  system to help HUD measure the success of Youthbuild programs, along  with its other nonhomeless competitive grant programs. However, the  system’s implementation has been delayed because of funding and  technical problems. As a result of not systematically collecting or  analyzing needed information from closeout reports, relying  predominately on limited point-in-time program information, and not being  able to utilize a planned database to aggregate performance outcomes,  HUD is limiting its ability to assess grantees. Further, although the  YouthBuild Transfer Act shifted oversight of grants issued in fiscal year  2007 to Labor, HUD will be required to continue monitoring existing  Youthbuild grants for approximately the next 3 years. Therefore, the data  HUD has at its disposal or could develop will continue to be valuable for  ongoing assessments of the program.
section_title: Citing Resource Constraints, HUD Performed Limited Oversight of Grantees
paragraphs: In addition to not having or utilizing available performance data, HUD has  exercised limited on-site review and oversight of grantee performance.  According to HUD’s Office of Inspector General, on-site monitoring is an  essential tool for HUD (through CPD) to assess program performance and  identify and address potential program problems. Such monitoring is also  useful in motivating grantees to exercise sound judgment in carrying out  their grant activities. GAO’s Standards for Internal Control in the Federal  Government note that controls generally should be in place to ensure that  ongoing monitoring occurs during normal operations. Monitoring should  be performed continually, be ingrained in an entity’s operations, be used to  assess the quality of performance over time, and support the prompt  resolution of any identified problems.
CPD goals require on-site monitoring of only a small percentage of grants.  For fiscal year 2003, the plan called for the field offices to monitor 35  percent of formula grants; 20 percent of competitive homeless grants; and  7 percent of competitive nonhomeless grants, including Youthbuild. HUD  has since changed its baseline monitoring goals (requiring monitoring of  20 percent of grantees with active, open grants established since 2004) to  give its field offices more flexibility to meet monitoring priorities based on  local grantee knowledge. As a result, goals are no longer set by program.  Field offices decide on the number of formula versus competitive grantees  to be monitored based on risk analyses that evaluate each grantee on  factors such as the size of the grant, perceived project management  capacity, grantee responsiveness, and the extent of citizen complaints.  HUD officials cited a lack of resources as the primary reason for not  requiring the monitoring of a larger percentage of grantees. In 2006,  according to HUD officials, field offices typically had one person working  part time responsible for overseeing the individual Youthbuild grantees in  the state, visiting no more than one or two grantees per year. As a result,  field staff have not been able to comprehensively monitor grantees and  verify the accuracy of the output data submitted by the grantees.
In general, we found that the monitoring reports that we reviewed (which  HUD field offices produced) on grantees primarily addressed various  compliance issues, rather than reporting on the overall performance of a  particular grant. For example, several reports identified grantees’ failure to  receive prior approval from HUD for changing proposed job training  construction sites. Other reports identified inadequate record keeping,  insufficient documentation, and lack of sufficient financial internal  controls on the part of grantees. However, although a few reports did  comment on some individual performance outputs, such as job  placements or GEDs attained, most did not.
The lack of analysis of performance information of grantees also has  implications for assessing the Youthbuild program overall, particularly  because HUD has devoted few staff to overall program management and  oversight. Since 2000, in headquarters, Youthbuild had a program director  with one or two staff managing the program each year. During fiscal year  2006, with the expected program transfer to Labor, the HUD program  manager was assigned to other duties, and a part-time program manager  was assigned to oversee the 2006 grant process, in which HUD received  308 applications and awarded 74 grants. Although they are aware of the  limited oversight of Youthbuild, CPD division officials said it has provided  appropriate staff resources for program oversight of Youthbuild grantees,  commensurate with the program’s size—the annual appropriation for  which represents less than 1 percent of CPD’s usual annual budget of  about $8 billion. However, without sufficient oversight HUD was unable to  tell how the individual Youthbuild grantee programs performed.
section_title: Incomplete Data on Participants’ Outcomes Precludes an Overall Assessment of Program’s Success
paragraphs: HUD has not performed the programwide analysis needed to determine  overall program success, and weaknesses in other assessments of the  programs and data preclude an overall assessment of Youthbuild success.  While limited in scope, three studies suggest Youthbuild helped youths  achieve economic self-sufficiency. And, although YouthBuild USA tracks  performance data, grantees do not have the follow-up information needed  to measure performance outcomes. To augment limited existing data, we  analyzed information in 245 closeout reports, but the results cannot be  generalized to the entire program. We found a wide range of results in  outputs such as job placements; however, because grantees generally did  not follow up on participants, we were unable to determine long-term  outcomes such as percentages of participants retaining employment.  Moreover, while closeout reports include impediments to success and  “best practices,” HUD did not systematically review this information or  share it with YouthBuild USA. As a result, the lack of follow-up data and  programwide evaluations make it very difficult to determine which  programs and strategies have worked best and assess the performance of  Youthbuild over time. Labor officials said they would consider that the  closeout reports include outcomes such as jobs obtained and retained and  said that they believe impediments and best practices should be evaluated  and disseminated to grantees.
subsections: section_title: A Few Studies Document Individual Youthbuild Program Successes but Are Limited in Scope
paragraphs: A few studies have provided some evidence of the success of individual  Youthbuild programs. Although subject to many limitations, the studies  suggest that identified programs have achieved certain Youthbuild  objectives. However, without comparable information for the entire  population of Youthbuild grantees, it is not possible to determine the  extent to which these outcomes and reported successes are reflective of  what was achieved by the grantees not included in the studies. More  specifically:  In June 2004, researchers from Brandeis and Temple Universities  published a study entitled, “Life after YouthBuild - 900 YouthBuild  Graduates Reflect on Their Lives, Dreams, and Experiences,” in  collaboration with YouthBuild USA. The study was based on a survey of  graduates from 11 Youthbuild grant sites and included in-depth interviews  with 57 graduates from eight of these sites. The researchers determined  that, for those 882 participants (for a 23 percent response rate) who  responded to the survey, many are doing well and were helped  considerably by the Youthbuild program. Many participants found jobs or  furthered their education and have volunteered in community activities. In  addition, a number of participants surveyed indicated they also obtained  intangible benefits from the program, such as increased self-esteem and  had developed positive attitudes about themselves and their community. A  limited nonresponse analysis was conducted by YouthBuild USA to  compare the 882 survey responses and the 57 interviews with one site that  had a 98 percent response rate. However, using one site that had a high  response rate is not representative and cannot be generalized to the whole  sample. As such, it is not known if the 77 percent who did not respond  were doing more or less well than those who responded to the survey.
In August 2003, HUD issued a study entitled, “Evaluation of the Youthbuild  Program,” in which a HUD contractor compared program results and costs  per participant from the Youthbuild Program and four other federal  employment and training programs—Job Corps, JOBSTART, both  designed to serve disadvantage youth, and the youth components of the  Job Training Partnership Act and Supported Work Demonstration  programs. The researchers conducted their work at 20 Youthbuild sites,  selected in part because they had received funding for at least two grants,  obtained a variety of organizational sponsorships, built new or  rehabilitated housing, and covered a variety of funding category sizes. The  study found that 36 percent of the Youthbuild participants at these sites  obtained jobs after enrolling and 29 percent obtained either a GED or high- school diploma. In addition, 12 percent of the Youthbuild graduates in the  20 sites reportedly went onto higher education. However, results at the  five federal programs were not directly comparable because researchers  obtained data from Youthbuild program participants at the time they  exited the program while data from the other programs’ were obtained  from tracking participants up to 3 years after leaving the program. In  addition, the limited scope of this study precludes an overall assessment of  the program’s success.
In February 2003, the state of Minnesota issued a study entitled,  “Minnesota Youthbuild Program: A Measurement of Costs and Benefits to  the State of Minnesota.” This study analyzed the costs and benefits of the  state-funded Youthbuild program in Minnesota. The researchers measured  post-program participant performance outcomes by tracking program year  participants beyond graduation. They found that Youthbuild graduates  earned more than double the minimum wage, whereas youth with similar  characteristics who had not been through the program typically earned  only the minimum wage. In addition, the graduates who had been  adjudicated had a considerably lower rate of recidivism (rearrest,  reconviction, or return to prison) than other youth involved with the  correctional system. Using state administrative data to estimate the  amount of revenues and savings generated through increased tax revenues  and reduced prison costs, the study estimated first-year benefits to the  state of about $1.5 million dollars versus costs of about $877,000. Over a 4-  year period, the study estimated the total net benefit to the state at about  $7.3 million (by 2006). The study was limited to costs and benefits directly  related to state programs and did not include potential benefits such as  reductions to welfare rolls, state-sponsored medical care, or other  reductions in human services costs.
section_title: YouthBuild USA WebSTA Data Have Limitations, Particularly for Assessing Outcomes
paragraphs: Since 2002, YouthBuild USA has been using a Web-based system— WebSTA—to collect participant characteristics and performance data  from its affiliates. According to YouthBuild USA officials, as of December  2006, about 90 percent of its affiliates were entering data into WebSTA to  some extent. However, the value of YouthBuild USA’s WebSTA data is  limited, largely because grantees self-report the data, not all grantees are  meeting the requirement to send in information, and the information is not  reported at the grant level. That is, if affiliates have more than one  Youthbuild grant, the data for each grant are not reported separately. As a  result, the data are not verified, comprehensive, or comparable. Also,  because the grantees submit the information quarterly, the data provide  only a point-in-time snapshot of the program’s performance and cannot be  summed to provide annual data. Recognizing these limitations, according  to YouthBuild USA’s President, upgrades are planned to be implemented  in early 2007 that will allow grantees to enter data in WebSTA by  individual grant, which in turn will allow YouthBuild USA to analyze grant- level performance.
We also found that the limited depth of the information collected restricts  the value of WebSTA information for performance assessments. For  example, WebSTA collects information on participant attendance,  retention and graduation rates, and initial job and college placements and  participant characteristics (such as welfare, public housing, and  adjudication status) when participants enter Youthbuild programs.  However, many grantees do not follow up on participants once they leave  the programs. As a result, they often do not know if students placed in  jobs were still working at those jobs weeks or months later. Also, several  program directors we visited said that most of their program graduates  who go on to higher education sign up for free remedial-level courses at  nearby colleges. Although they have not had the funding to systematically  follow the students’ progress, some said they generally have found that the  students have had varying success in advancing to college-level credit  courses.
Without grantee-level follow-up systems, WebSTA cannot track program  outcomes that would reflect whether the graduates are achieving  economic self-sufficiency—such as the percentage of program graduates  who remove themselves from the welfare rolls, move out of public  housing, or stay out of the court system. The system also does not collect  follow-up information on the type of jobs attained (including  construction), pay raises, benefits received, and college or trade degrees  or certifications received. Analyzing such performance outcomes could  provide, HUD, Labor, and other interested parties with the measures  needed to evaluate overall program performance and success.
section_title: Closeout Reports We Analyzed Provided Little Detail on Long-Term Outcomes Because Grantees Are Not Required to Follow up on Graduates
paragraphs: Because HUD had not aggregated or analyzed data for Youthbuild, we  obtained and analyzed 245 closeout reports that grantees submitted to  HUD. Figure 6 provides selected performance information that we  summarized from the closeout reports. However, because HUD has not  established objective performance goals for the program, which would  serve as baseline criteria against which to judge performance, the  information related to the experience of the 12,863 participants who  entered the program under the 245 grants gave us limited perspectives  about how individual programs were working. In addition, because we had  information on only 245 (or 46 percent) of the grants, the results cannot be  generalized to the entire program.
As shown in figure 6 the percentages for participants graduating, receiving  a GED, or being placed in employment varied widely. As previously  discussed, HUD has not set benchmarks, and critical data elements  (baseline data) are missing in the closeout reports. For instance, it is not  possible to obtain an accurate number of diplomas and GEDs attained  from the closeout reports because the reports do not contain data on the  number of participants that enter the program with a high-school diploma  or GED. As result, we could not determine how well the programs (except  for those with the very highest percentages) performed individually or in  comparison with each other.
The closeout reports also do not provide consistent and reliable data on  the number of units rehabilitated or built by Youthbuild participants. A  basic purpose for which Youthbuild was authorized was to expand the  supply of affordable housing. According to the closeout reports, 158  grantees reported that their participants built a total of 1,483 new units,  while 139 grantees reported that their participants rehabilitated a total of   2,238 units. Some of the Youthbuild program directors with whom we  met said that during a typical reporting period they would complete from  one to four new or gut-rehabilitated housing units. Other directors said  that when a program reports about 200 units rehabilitated, it generally  means the students are performing minor repetitive tasks such as changing  locks or painting a limited number of interior walls. As a result, the  number of units rehabilitated likely is overstated in the closeout reports.  Although construction training is a major part of the Youthbuild program,  HUD’s closeout reports also do not track the number of graduates  obtaining construction-related jobs or specify the types of jobs obtained  (construction or otherwise).
In addition, HUD officials told us that grantees are not required to follow  up on participants after they leave the program. While some grantee  officials with whom we met recently had attempted to establish follow-up  systems, most grantees cited funding constraints as the reason for not  following up on graduates. Several officials further noted that it was  difficult to locate many participants after they had left the program.  However, as a result, it is not possible to obtain comprehensive post- program results.
Finally, some program directors with whom we spoke cautioned that any  follow-up system should not be strictly focused on students obtaining  higher education degrees or obtaining high-paying jobs. They emphasized  that such goals were beyond the abilities of some of their participants, but  that moving from a fifth-grade to a ninth-grade reading level or obtaining a  service sector job sometimes was a real victory and should be counted as  a success.
section_title: Limited Information We Developed from Site Visits Pointed to Varied Results for Programs, and Also Provided Little Detail on Long-Term Outcomes
paragraphs: The information we obtained from our visits also demonstrates varied  results for continued employment and academic success, and little detail  on long-term success. As with our closeout report analysis, the  information cannot be generalized to the universe of Youthbuild  grantees—although YouthBuild USA officials told us that they believe we  visited some of the more innovative grantees. For example, although not  required, some of the nine grantees we visited used other funding sources,  such as United Way or AmeriCorps funds, to establish a participant follow- up system. As a result, some of the grantees were able to provide us with  varying amounts of information on the status of program graduates. For  example:    One grantee, which was able to track individuals who completed the  program since 2001, provided us with data that showed 70 of the 82  program participants were either employed or pursuing higher education.  Seven of the 70 participants were working and going to school. The hourly  wages of those employed ranged from $7 to $13.75. However, the grantee  did not have information on the types of employment.
Another grantee that recently established a follow-up system showed that  11 of the 26 program graduates who started in 2004 had obtained  employment. The positions ranged from pizza delivery person to union  carpenter. However, only the union carpenter and a graduate hired by the  program as a construction trainer were working in the construction field.  The hourly wages of the graduates ranged from $5.50 to more than $16 per  hour for a graduate hired by the grantee.
Another grantee used WEBSTA to track continuing education and work  history, including each job held. The program director told us that such a  follow-up system allows the program to spot trends, which in turn can be  used to fine-tune the program. Of 60 graduates from 2002 to 2005, the  program continually has tracked 53 and reported that 8 are receiving  higher education, 24 are working in nonconstruction jobs, and 16 are  working in construction jobs. The pay for the graduates ranged from $6.50  to $15 per hour. The grantee continues to keep in touch with the graduates  by helping them with housing, legal aid, school advice, tutoring, and help  toward achieving a GED.
Grantees are not required to collect such post-program follow-up  information. Yet, both grantees and program participants emphasized that  the Youthbuild program created a very supportive atmosphere that many  participants had never experienced prior to joining a program. During our  visits, we met with students who credited the program with improving  their lives by either helping get them off drugs, get out of the gang lifestyle,  or disengage from other activities that likely could have resulted in  incarceration or worse. The studies we mentioned previously had similar  findings. However, it is difficult to gain insight into how many program  participants remain drug free, crime free, in advanced degree programs, or  working in construction or other trades without post-program  performance data. It is also difficult to gain insight into which program  strategies are most effective for serving these youth.
section_title: HUD Does Not Review or Disseminate Impediments and Best Practices Identified in Closeout Reports That Could Help Improve Program Performance
paragraphs: Citing resource constraints, HUD did not systematically review the  impediments and best practices identified in closeout reports or share the  information in them with YouthBuild USA. In particular, identifying best  practices (or the converse, impediments) can provide a model for  improvement for other organizations with similar missions. The closeout  reports contain a section addressing what the grantees considered to be  best practices. For example:    Several grantees set-up educational opportunities, such as GED  preparation, at local colleges for participants. Two grantees noted that  being on a college campus provided a positive experience for the  participants. Two grantees noted that they made an arrangement with a  local technical college to offer free enrollment to all participants that  successfully completed the Youthbuild program. Another grantee noted  that working through community college GED programs allowed them to  lower their per-student cost and serve more students.
One grantee established an alumni council consisting of elected  representatives from each of its seven graduating classes. All graduates  receive quarterly mailings that include an alumni newsletter, new job and  educational opportunities, and a quarterly calendar of events.
Another grantee has maintained a program serving graduates for up to 12  months to provide on-going employment support, including assistance in  securing a second job in the event that the initial placement did not work  out.
One grantee started a “drug free” club and found that students really  wanted to participate and were anxious to become drug free and would  publicly announce their success.
Another grantee worked extensively recruiting area college students to  volunteer to help with tutoring and program community service activities.
Closeout reports also contain information on impediments that grantees  encountered in implementing and operating the programs. Impediments to  program success varied in scope and severity. Some of the more  significant impediments mentioned by numerous grantees included the  following.
Turnover of key staff, especially on-site construction managers;    Negative and apathetic attitudes among participants;    Alcohol and drug abuse issues among participants;    Difficulties in dealing with municipal organizations to obtain building  permits and contractual difficulties, which delayed construction training  activities; and   Acceptance of students with very low educational levels, which made  obtaining high GED levels very difficult.
HUD’s contract with YouthBuild USA specifically calls for the contractor  to assist HUD in the management, supervision, and coordination of  Youthbuild programs by strengthening Youthbuild program design and  disseminating information on best program practices. Although analyzing  the impediment and best practices falls within the scope of HUD’s  contract with YouthBuild USA, two CPD program officials said that HUD  never considered sharing the reports with its technical services provider.  YouthBuild USA’s chief operating officer said that the organization would  welcome access to the closeout reports. The officer said his staff already  spent many hours each month trying to identify best practices from  Youthbuild grantees and would be anxious to analyze the closeout reports  for both the problems and successes identified and forward the  information to interested grantees. YouthBuild USA’s President told us  that she not only agreed that reviewing the closeout reports for problems  and “best practices” would be beneficial, but that she believes the closeout  reports needed to “be analyzed to get to the bottom line because the total  impact (outcomes) of a grant could not be determined from data provided  on a rolling quarterly basis.” As a result of not sharing the closeout  reports with YouthBuild USA, HUD may have missed opportunities to  improve the program as a whole and help grantees improve performance.
section_title: Labor Officials Indicated They Will Consider Using Information from Closeout Reports to Assess and Improve YouthBuild Performance
paragraphs: According to Labor officials, they will consider analyzing information from  the closeout reports to assess and improve the performance of the  YouthBuild program. Until 2006, Labor assessed past performance of its  youth employment and training grantees through measures required under  WIA. For example, under the WIA Youth and Job Corps programs, Labor  tracked and analyzed 6-month retention and earnings wage change  information. Labor officials told us that they would use OMB’s common  measures to track and analyze the performance of YouthBuild grantees but  said that they have not finalized the information they would require  YouthBuild grantees to collect. The officials indicated to us that they  would consider developing post-program performance outcomes such as  the types of employment graduates attained and retained (including  construction), wage rates, and college or trade degrees or certifications  received. Although Labor may incur some additional costs in developing  such information, the ability to measure program success would improve.  Labor officials also noted that the information currently contained in the  closeout reports would be considered with other factors they deem as  pertinent to assessing grantees’ performance. However officials said they  would also likely consider modifying the closeout reports to include other  program participant data, such as degrees obtained prior to program entry  and details on the types of construction jobs obtained upon program  completion.
Labor officials also acknowledged that information on program  impediments and best practices should be evaluated and passed to  grantees and that performance data needed to be aggregated, verified, and  analyzed to determine which programs and strategies worked best. As of  October 2006, Labor had not determined whether its Employment and  Training Administration or an outside contractor would have primary  responsibility for collecting, aggregating, and disseminating the  information.
section_title: Youthbuild Grantees Have Experienced Varied Success in Obtaining Outside Funding Sources, and Many Cited Continued HUD Funding as Crucial
paragraphs: Our analysis of available closeout reports showed that the grantees have  had varying success in obtaining additional funding sources. In assessing  grant applications, HUD gave consideration to program applicants that  were able to secure outside funding. Applicants received points for  leveraging nonhousing resources compared with the amount of Youthbuild  funds requested in the application. Overall, the median amount spent by a  HUD Youthbuild grantee was $400,000 in HUD funds; the HUD grant  represented about 48 percent of the average total spending per grantee,  which included additional funding sources of about $430,000. However,  success in obtaining additional funds varied widely, from 21 grants  reporting no additional funding sources to 40 reporting more than $1  million (see fig. 7).
According to information in the closeout reports and the officials from the  grantees that we visited, the additional funding they obtained largely came  from federal, state, and local agencies, businesses, and charitable  organizations. According to a YouthBuild USA official, states have  provided funding to local Youthbuild grantees over the past several years.  For instance, the Massachusetts State Department of Education has  provided funding in its budget for the state’s Youthbuild program. The  Massachusetts funds are provided to existing Youthbuild programs, based  on a formula the state developed in conjunction with a state coalition of  Youthbuild grantees. In Minnesota, 10 Youthbuild programs in 2005 shared  $754,000 in state Youthbuild program funding, which the state  supplemented with more than $3 million in matching funds from local  partners. In Minnesota as elsewhere, urban grantees have had greater  opportunities to obtain funding or in-kind support from local businesses  and charitable organizations than rural grantees that typically have had  access to fewer businesses and might not be a focus of charitable activity.
Regardless of their success in obtaining outside funds, several grantees we  visited stated that their Youthbuild program could not be sustained  without the HUD grant because the HUD funding allowed them to attract  the leveraged funds. Also, several grant officials said that many existing  and potential contributors would not be willing to continue funding if they  perceived that the program would not be able to continue without the  HUD grant. One exception (which we did not visit) was in Minnesota  where the state Youthbuild program director said that only 4 of the state’s  10 programs have had HUD funds in the past, including 2 programs which  received HUD funding in 2004 through a HUD grant administered by the  state. The program director added that the structure of the Minnesota state  Youthbuild program is similar to that of HUD’s, although the programs are  typically smaller, having from 10 to 25 students.
Information from YouthBuild USA generally agrees with grantees’  contention about the need for continuous HUD funding. Overall,  YouthBuild USA found that 60 percent of programs that HUD did not fund  for 2 consecutive years ceased operations, and 90 percent of programs not  funded for 3 consecutive years ceased operations. According to  YouthBuild USA, 173 of 462 grantees funded since 1994 currently are  operating with active HUD grants. YouthBuild USA, also found that  grantees with continuous HUD funding achieved higher outputs, such as  GED attainments and job placements, than the ones that did not receive  one or more follow-on grants. According to some grantee officials, that  success is partly due to continuous funding that allows grantees to  develop infrastructure and maintain experienced staff to establish a better  program. In its fiscal year 2005 Performance and Accountability Report,  HUD noted the benefits of making grants to previous awardees: “Having  established programs and experience running a Youthbuild program, these  grantees are more efficient in enrolling students, resulting in a greater  number of youth trained.” Furthermore, Youthbuild program directors  with whom we met emphasized the intangible benefit that such  established programs provide such as a family atmosphere that allows  graduates to return for advice and support as needed—a benefit that goes  away when programs close.
HUD has not kept statistics to find out why some grantees did not reapply  for additional grants or why grantees did not qualify for follow-up grants.  Through discussions with a nonprobability selection of 23 of the 100  grantees that closed programs, we found that some did not reapply  because they were no longer interested in dealing with at-risk youth and  others closed due to poor management. But most of the closed grantees  with whom we spoke said they closed because they lost HUD funding for  one or more years and were unable to obtain sufficient private funds to  make up for the lost funding. Of the sites we contacted, two were still in  operation after ceasing to receive HUD funds—but they appeared to be the  exception. Both were already operating youth technical training programs  prior to their first HUD Youthbuild grant.
In addition, officials from several grantees that also received funding from  AmeriCorps stated that they preferred the AmeriCorps concept of  providing dedicated funds for 3 years. These officials stated that knowing  that funding levels were longer-term provided more continuity to their  program, enabled better planning, and allowed them to build a cushion  through leveraged funding in the event that they did not receive a HUD  grant for a year or two. Finally, when the question of multiyear funding  was raised at a Web-based listening session in December 2006, designed to  help Labor identify Youthbuild program administration issues and  concerns, most participating grantees spoke out in favor of multiyear  funding.
section_title: Conclusions
paragraphs: HUD has not taken advantage of several opportunities to use existing  information for purposes of program assessment and oversight. Such  opportunities should not be ignored, particularly because HUD has limited  resources with which to assess and oversee current grantees. Specifically,  HUD has not verified, aggregated or analyzed closeout reports, which  grantees are required to submit. HUD also has conducted limited oversight  of grantees, citing resource constraints; the impending transfer of  oversight to Labor bodes no change to this specific situation. However,  given these conditions, analysis of existing closeout reports would help  provide crucial information on the performance of the Youthbuild  programs, particularly in the absence of a centralized HUD database that  could be used to track program performance. At a minimum, the closeout  reports form a logical basis for providing useful information that HUD,  Labor, and other interested parties can use to judge the performance of  Youthbuild. HUD still will maintain an active involvement for several more  years, so coordination between Labor and HUD will be critical to  producing reliable assessments of how successful the program has been  and will be. HUD also does not share reported constraints and best  practices with the technical services contractor it pays for performing  such work and disseminating best practices. By not utilizing performance  data and sharing information, HUD has missed opportunities to determine  whether the overall Youthbuild program has achieved its intended results,  among them enabling disadvantaged young adults to gain economic self- sufficiency. Therefore, Congress and the public also lack the information  needed to make such determinations.
Labor has experience in managing youth training and education programs  and collecting performance information, and Congress transferred  Youthbuild to Labor in part to better align the program with existing  federal workforce development and youth training programs. In offering  perspectives for Labor to consider, we stress the value that closeout  reports have in the absence of more comprehensive and programwide  data, although the existing reports lack certain elements that could help  program managers determine and report on outcomes. The reports do  contain valuable information on impediments and best practices, which  could provide both program managers and grantees with much broader  perspectives on how successful the program has been and suggestions for  improvements. Labor has indicated that it would consider developing post- program performance outcomes such as the types of jobs graduates  attained and retained (including construction), wage rates, and college or  trade degrees or certifications received. As a result, while there may be  some additional costs involved, the ability to measure program success  would improve. And, because alternatives to the HUD data are limited, the  grant-level assessment discussed above could act as the building blocks of  an overall program assessment. Labor also has agreed that information on  impediments and best practices would be valuable and ought to be passed  on to grantees, but has not yet determined how this might be done.
Finally, data on grantee success in attracting other funding, although  limited, offer more perspectives for Labor to consider because outside  funding helps leverage limited federal dollars. While many grantees have  obtained extensive outside funding and support, YouthBuild USA has  found that very few operated for more than a few years without continued  HUD funding. Tellingly, their data show that 90 percent of programs not  funded by HUD for 3 consecutive years ceased operations. According to  YouthBuild USA officials, their data indicate that grantees with repeat  awards also produced better outcomes such as a higher level of job  placements. Further, when HUD funding stopped, potential contributors  became concerned that grantees would no longer stay in business and  were reluctant to jeopardize their own funds. These data, while not  definitive, do suggest that multiyear funding has the potential to produce  dividends for the program. Many grantees expressed similar belief in the  benefits of longer term funding at a Department of Labor YouthBuild  program listening session.
section_title: Recommendations for Executive Action
paragraphs: To improve the reporting and assessment of performance for Youthbuild  grantee programs and develop the bases for an overall assessment of the  program, we are making the following three recommendations:  that the Secretary of HUD analyze closeout reports by grant and share  information on identified problems and “best practices” with its technical  services contractor and Labor; that the Secretary of Labor develop and monitor post-program  performance outcome measures for the YouthBuild program, such as the  types of employment graduates attained and retained, wage rates, and  degrees or certifications received, and share the data with the grantees;  and that the Secretary of Labor consider whether multiyear funding could be  useful in helping YouthBuild grantees attract additional outside funding.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided HUD and Labor with a draft of this report for review and  comment. HUD’s comments are summarized below. Labor’s Assistant  Secretary for Employment and Training provided written comments that  are presented in appendix I.
Officials from HUD’s Office of Community and Planning and Development  provided comments.  They stated that they did not disagree with GAO’s  overall findings, but added that HUD’s oversight responsibility was limited  by resource constraints. They cited the fact that the Youthbuild program  represents 0.63 percent of the Community Planning and Development  Division’s entire portfolio. Nevertheless, because HUD will have  monitoring responsibilities for existing grants for 3 years or more, we  believe it is important that HUD analyze the closeout reports in order to  provide Labor with information about how current grantees are  performing.
Labor’s Assistant Secretary for Employment and Training wrote that she  believes that the report will be very useful to Labor as it assumes  responsibility for administering the YouthBuild program. She noted that  Labor agreed with our recommendation to develop and monitor post  program performance outcome measures for the YouthBuild program and  share these data with grantees. She added that Labor was building a Web- based Management Information System to collect and report on  participant information. Finally, she wrote that Labor agreed that there  could be value on multiyear funding and that the Department was  considering such an approach for existing grantees.
We are sending copies of this report to other interested congressional  committees and the Secretaries of Housing and Urban Development and  Labor. We will also make copies available to others upon request. The  report will also be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have questions regarding this report, please contact me  at (202) 512-4325 or shearw@gao.gov. Contact points for our Offices of  Public and Congressional Affairs may be found on the last page of this  report. GAO staff who made major contributions to this report are listed in  appendix II.
section_title: Appendix I: Comments from the U.S. Department of Labor
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Andy Finkel, Assistant Director;  Rich LaMore, John McGrail, Marc Molino, Luann Moy, Barbara Roesmann,  and Tom Taydus made key contributions to this report.