Defense Management: Actions Are Needed to Improve the Management and Oversight of the National Guard Youth Challenge Program
Published: 2005-11-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The fiscal year 1993 National Defense Authorization Act established the National Guard Youth Challenge Program as a pilot program to evaluate the effectiveness of providing military based training to improve the life skills of high school dropouts. The Assistant Secretary of Defense for Reserve Affairs, under the authority of the Under Secretary of Defense for Personnel and Readiness, is responsible for overall policy for the program. The National Guard Bureau (NGB) provides direct management and oversight. In 1998, Congress permanently authorized the program and began decreasing the federal cost share until it reached its current level of 60 percent in 2001. Conference Report 108-767 directed GAO to review the program. Specifically, GAO reviewed (1) historical trends of the program; (2) the extent of analyses performed to determine program costs and the need to adjust the federal and state cost share; and (3) NGB oversight of the program. GAO is also providing information on Reserve Affairs' and states' efforts to obtain funding from alternative sources.
section_title: What GAO Found
paragraphs: Between fiscal years 1998 and 2004, total expenditures for the Challenge Program, including funds spent to cover the federal and state cost shares and federal management expenses, have increased from about $63 million to $107 million. During this same period, participation in the program has grown from 10 sites in 10 states to 29 sites in 24 states and Puerto Rico. Since the program's inception, NGB has reported positive performance outcomes in academic performance, community service activities, and post-residential placements. For example, in 2004, NGB reported graduating 7,003 students, or 79 percent of those enrolled, with 70 percent of those graduates earning a high school equivalent diploma. While Reserve Affairs and NGB have expressed concern about the current program funding level and have suggested increasing both the cost basis used to determine funding needs and the federal cost share, neither has performed analyses to support the need for such changes. Federal financial standards state that reliable cost information is crucial for effective management of government operations. Since 1993, NGB has used $14,000 per student as the basis for determining the amount of funds needed to cover program operating costs, and applied the federal-state cost share to this amount. To keep pace with inflation, NGB has suggested increasing the per student cost to $18,000. Reserve Affairs has reported some states are having difficulty meeting their share and, in 2004, recommended the federal share be increased from 60 percent to 75 percent. However, neither Reserve Affairs nor NGB has compiled or analyzed data on actual program costs, states' financial situations, or the impact of adjusting the federal and state cost-share. Without better cost and financial information, the Department of Defense (DOD) cannot justify future funding requests or a change in the cost-share ratio. Although NGB uses various oversight mechanisms, it lacks a complete oversight framework, making it difficult to measure program effectiveness and to adequately address audit and review findings. Also, some audits have not been performed as required. The Government Performance and Results Act suggests a complete oversight framework including goals and measures against which to objectively evaluate performance. While NGB requires states to report certain performance outcomes, it does not require states to establish performance goals in these areas, and therefore does not have a firm basis for evaluating program outcomes and DOD's return on investment. Existing agreements require state programs to be audited at least every three years. However these audits have not been conducted as required and no provisions exist for submitting audit results to NGB. Without regular audits and access to results, NGB cannot be assured that programs are using federal funds appropriately and that audit findings are addressed.

section_title: Letter
paragraphs: After Operation Desert Storm and the end of the Cold War, Congress  After Operation Desert Storm and the end of the Cold War, Congress  began to reevaluate the focus of U.S. military activities and proposed using  began to reevaluate the focus of U.S. military activities and proposed using  some military assets and training to help address critical domestic  some military assets and training to help address critical domestic  challenges such as drugs, poverty, and unemployment.  In particular, some  challenges such as drugs, poverty, and unemployment.  In particular, some  policymakers saw an opportunity to use the military’s capabilities to  policymakers saw an opportunity to use the military’s capabilities to  address domestic needs by providing role models for youth and training  address domestic needs by providing role models for youth and training  and educational opportunities for the disadvantaged.  Section 1091 of the  and educational opportunities for the disadvantaged.  Section 1091 of the  National Defense Authorization Act for Fiscal Year 1993 established the  National Defense Authorization Act for Fiscal Year 1993 established the  National Guard Youth Challenge Program (Challenge Program) as a pilot  National Guard Youth Challenge Program (Challenge Program) as a pilot  program to evaluate the feasibility and effectiveness of military based  program to evaluate the feasibility and effectiveness of military based  training to improve the life skills and employability of high school  training to improve the life skills and employability of high school  dropouts.  After the program was permanently authorized in 1998,  dropouts.  After the program was permanently authorized in 1998,  Congress began limiting the share of state operating costs covered by the  Congress began limiting the share of state operating costs covered by the  federal government through the Department of Defense (DOD), which  federal government through the Department of Defense (DOD), which  initially paid for 100 percent of these costs.  DOD now pays 60 percent of  initially paid for 100 percent of these costs.  DOD now pays 60 percent of  state operating costs, and states provide 40 percent.  In addition, the  state operating costs, and states provide 40 percent.  In addition, the  authorizing legislation for the Challenge Program allows the Secretary of  authorizing legislation for the Challenge Program allows the Secretary of  Defense to use nondefense funding sources in support of the program.    Defense to use nondefense funding sources in support of the program.
The Assistant Secretary of Defense for Reserve Affairs, under the authority  The Assistant Secretary of Defense for Reserve Affairs, under the authority  of the Under Secretary of Defense for Personnel and Readiness, is  of the Under Secretary of Defense for Personnel and Readiness, is  responsible for management oversight of the Challenge Program, including  responsible for management oversight of the Challenge Program, including  policy and funding issues, while the National Guard Bureau (NGB) is  policy and funding issues, while the National Guard Bureau (NGB) is  responsible for the administration of the program, including daily  responsible for the administration of the program, including daily  management and oversight.  For example, NGB maintains frequent  management and oversight.  For example, NGB maintains frequent  contacts with state programs and conducts program evaluations through a  contacts with state programs and conducts program evaluations through a  contractor, while Reserve Affairs prepares the program budget based on  input from NGB and ensures that states are able to provide their required  share.  To participate in the program, states must sign a cooperative  agreement with NGB that lays out roles and responsibilities among the  states and NGB, and also provides guidance on the content of the  program.  Since 1993, NGB has used a cost of $14,000 per student as a  basis for providing funds to state programs.
Conference Report 108-767 directed the Comptroller General to conduct a  study of the Challenge Program.  Specifically, we examined: (1) historical  trends of the Challenge Program, including program expenditures,  participation, and performance; (2) the extent to which Reserve Affairs  and NGB have determined actual program costs and the consequent need  to adjust the federal and state cost share; and (3) the extent to which NGB  has provided oversight of the program.  We also determined the extent to  which Reserve Affairs and participating states have made an effort to  obtain alternative funding support for the Challenge Program.
In performing our work, we interviewed officials from the Office of the  Assistant Secretary of Defense for Reserve Affairs; NGB; the Departments  of Education, Labor, and Justice; the contractor that monitors and  evaluates state programs; the National Guard Youth Foundation; and  selected states.  We also reviewed the program’s annual reports and other  documents provided by Reserve Affairs and NGB as well as the data  management system used by states to collect Challenge Program student  information and track individual and program activities.  We concluded  that the data were sufficiently reliable for our purposes.  We analyzed  program evaluations, resource management reviews, and audits, in  addition to reviewing existing policies and procedures for managing the  program.  We conducted a survey of all 29 established programs to collect  information on their budgets and expenditures, federal and state funding  levels, each state’s ability to fund their individual program, and the  availability of funding from other sources.  We also visited or contacted  eight Challenge Program sites in seven states and asked them about any  alternative sources of funding for their programs.  A detailed description  of our scope and methodology is presented in appendix I.  We conducted  our work from January 2005 to October 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: The National Guard Youth Challenge Program is a 17-month program that  serves at-risk youth at 29 sites in 24 states and Puerto Rico.  The purpose  of the program is to improve the education, life skills, and employment  potential of students by providing military-based training, supervised work  experience, and knowledge in eight core program components.  Students  must be 16 to 18 years old, drug-free, unemployed, high school dropouts,  and not in trouble with the law.  NGB reports that more than 59,000  students have been graduated from the Challenge Program since it began  as a pilot in 1993. The program was authorized by 32 U.S.C. §509 on a  permanent basis in fiscal year 1998, at which time states were to begin  paying a share of operating costs.
subsections: section_title: Challenge Program Structure
paragraphs: Each Challenge Program site operates two residential classes per year,  one of which begins in January and the other around July.  A typical  graduation goal is 100 students per class, or 200 per year, although several  programs graduate more students.  In 2004, for example, Illinois graduated  almost 800 students in 2004, and Louisiana’s three sites combined  graduated more than 950 students.  The residential phase of the program  runs 22 weeks and includes a 2-week Pre-Challenge phase. During Pre- Challenge, applicants are assessed for their ability and motivation to  complete the remaining 20 weeks of the residential program.  Those who  successfully complete Pre-Challenge are then formally enrolled in the  Challenge Program in numbers that equal each program’s graduation  target plus normal program attrition rate.  In the residential phase,  students receive military-based training and supervised work experience.  Additionally, each state develops a curriculum that incorporates the eight  core components and the tasks, conditions, and standards that students  must complete to demonstrate progress in those components.  Each  student must receive a score of at least 80 percent on each core  component to be graduated from the program.
During the 12-month post-residential phase, individuals who have  successfully completed the residential phase are involved in placement  activities, which include employment, education, volunteer activities, or  any combination of the three or military service.  The graduates work with  adult mentors who were matched with them during the residential phase.   These mentors provide guidance and support to the graduates and are  required to contact the youths twice each month at a minimum.  Program  staff use the written, post-residential action plan that each student  prepares and updates during the residential phase to monitor placement  activities.  Mentors also use this plan during their interactions with  graduates.  The Challenge Program reports youth placement activities at  the end of the 12-month follow-up period.  To further assess the long-term  impact of the program, NGB has contracted with AOC Solutions to  conduct a retrospective longitudinal study of program graduates as well as  students who did not complete the program.
section_title: Program Funding
paragraphs: Prior to 1998, the federal government, through DOD, completely funded  the Challenge Program.  In fiscal year 1998, Congress began requiring  states to provide a minimum of 25 percent of their programs’ operating  costs.  The state cost share increased 5 percent each year until fiscal year  2001, when it reached the current funding requirement of 40 percent.  Although some states had provided more funds than required in the past,  program funding each year is now determined by the 40 percent share,  which is based on $14,000 for each youth targeted for graduation.  In  addition to the federal and state funds used to operate the program sites,  DOD also provides funds for NGB management expenses such as program  evaluations, contractor-provided training, and travel for training and  workshops.  These NGB program management costs are not subject to the  federal/state 60/40 cost share requirement.
Each state submits a budget to NGB that is based on that state's target for  number of graduates.  Since the program’s inception, the funding provided  by NGB has been based on a cost per student of $14,000.  For example, if  a state has a target of 100 students per class (200 per year) to graduate, the  estimated program costs would be $2.8 million. The federal contribution,  or 60 percent of the total, would be $1.68 million, while the state  contribution would be $1.12 million.
To receive federal funding, a state must certify that it has sufficient funds  to provide its 40 percent share. State funds can be composed of cash,  noncash supplies, services, or a combination of these sources.  States are  allowed to provide additional funding (over and above the 40 percent  share) to the program from sources such as individual and corporate  donations, additional moneys from the state general fund or other state  revenue sources, or other federal funding.  Some Challenge Program sites,  for example, operate as alternative schools and are reimbursed by their  state education agencies for portions of their program costs.
section_title: Management, Oversight, and Support
paragraphs: Reserve Affairs, under the authority of the Under Secretary of Defense for  Personnel and Readiness, is responsible for preparing the annual budget  and reviewing state budgets and funding certifications.  Reserve Affairs is  to monitor program compliance with DOD policy, issue supplemental  policy guidance, and submit the Challenge Program annual report to  Congress.  NGB provides day-to-day administration and oversight of the  Challenge Program, issuing regulations, and submitting budgets and  annual report drafts to Reserve Affairs.  NGB has contracted with AOC  Solutions to assist with the oversight of the Challenge Program.  AOC  Solutions performs the annual operational evaluations and the biennial  resource management reviews.  This contractor also pulls together the  program information for the annual report and maintains and oversees the  Data Management and Reporting System (DMARS), which is used to  collect student data and report on individual and program activities.  NGB  has also contracted with Dare Mighty Things to provide training and  technical assistance to Challenge Program staff.  Finally, NGB has United  States property and fiscal officers in each state who are responsible for  receiving and accounting for all federal Challenge Program funds and  property under control of the National Guard in that state.  The property  and fiscal officers are also responsible for ensuring that federal funds are  properly obligated and expended.
NGB enters into cooperative agreements with governors of states  approved to participate in the Challenge Program.  The cooperative  agreements describe the responsibilities of the states and NGB as well as  the funding, costs, and regulations for operating National Guard Youth  Programs.  The cooperative agreements also define the eight core  components and provide guidance on how to run the residential and post- residential phases and other aspects of the Challenge Program.  Each  Challenge Program state is also required to submit state plans and budget  estimates for their state.  These state plans must include details on the  state’s procedures and be consistent with overall program guidance  provided by DOD.  For example, state plans include information on  application and selection procedures, staffing and staff training, and a  detailed budget.
section_title: Challenge Program Expenditures and Participation Have Increased and NGB Has Reported Positive Performance Outcomes
paragraphs: According to NGB, Challenge Program expenditures and state  participation have increased since the program began, and the program  has achieved positive program performance outcomes over time.  Since  the program’s inception, total expenditures have increased from about $63  million to about $107 million per year.  The number of states participating  in the Challenge Program has also increased, and several states have  expressed interest in adding a program or expanding existing ones.   Challenge sites must account for their activities throughout the year, and  NGB has reported positive performance outcomes over time.
subsections: section_title: Program Expenditures Have Increased Over Time
paragraphs: NGB reports that overall federal expenditures for the Challenge Program  have increased over time, but states have also increased their  expenditures since the program was permanently authorized in fiscal year  1998.  Between fiscal years 1998 and 2004, total expenditures for the  Challenge Program, including funds spent to cover the federal and state  cost shares and federal management expenses, have increased from about  $63 million to about $107 million.  For fiscal year 2004, for example, NGB  expenses included $61.6 million for the federal cost share and $5.8 million  for NGB management costs, while states contributed approximately $40.5  million.  In addition, in 2000 and 2001, the Challenge Program received $5  million and $7,483,500 respectively from the Department of Justice.   Reserve Affairs stated that the primary use of these funds has been to start  new Challenge Program sites.  Since 2001, four programs were established  using these funds and three programs’ operations were maintained in 2002.   In total, approximately $5.97 million remain unspent in a nonexpiring  account.  Officials at Reserve Affairs and NGB told us that these funds  remain unspent because no new Challenge Programs have started.  According to these officials, new programs have not been established  because state governments have not committed the required 40 percent  match. (See fig. 1 for total program expenditures from fiscal year 1998 to  2004, broken down by federal and state cost share and NGB management  expenses.)
section_title: The Number of States Participating in the Challenge Program Has Increased
paragraphs: When the pilot program began in 1993, there were 10 Challenge Program  sites in 10 states.  The program has now grown to 29 sites in 24 states and  Puerto Rico.  In fiscal year 2005, Wyoming received funds to start up a  program site.  According to NGB, Wyoming will begin its first class in  January 2006. In addition to those states currently operating Challenge  Program sites, there are also nine states that have expressed interest in  establishing new programs.  For example, according to NGB officials,  representatives from Washington and Indiana National Guard units have  visited some existing program sites and are in the process of developing  state programs.  Other states are interested in expanding their programs to  serve more youth at existing sites or to open new locations.  On the other  hand, for various reasons including difficulty meeting the state match  requirement, lack of state support, and substandard facilities, four states  have discontinued their Challenge Programs.  Connecticut, a pilot program  state, dropped its program in 1994 after two classes.  Colorado  discontinued its program after 1999, Missouri after 2002, and New York  after 2003.  Figure 2 describes the number of Challenge Program sites for  each year since the program began. Appendix II identifies the individual  states with Challenge Program sites.
Student participation in and graduation from the Challenge Program have  also increased over time.  States are required to track the number of youth  who have applied to the Challenge Program, enrolled in the third week of  the program (after the 2-week Pre-Challenge phase), and were graduated  from the residential phase.  According to NGB, the target graduation rate  for 2004 was 6,961; the actual number of enrollees was 8,920; and 7,003  students were graduated, or 79 percent of those enrolled, from the  Challenge Program. Figure 3 shows the target numbers, the actual number  of students who were enrolled in the residential phase at week 3, and the  number that graduated from the program from 2000 through 2004.
Although all Challenge Programs graduate two classes per year, the  number of graduates per class varies.  In addition, some states have  multiple programs.  For example, Louisiana has three Challenge Programs  and, in 2004, graduated a total of 952 students.  Figure 4 identifies those  states currently participating in the Challenge Program by the number of  graduates they reported for 2004.
section_title: NGB Has Reported Positive Performance Outcomes
paragraphs: NGB has reported positive performance outcomes in academic  performance, community service activities, and post-residential  placements. Program performance information is tracked by each  Challenge site and submitted to NGB.  Each year, the Challenge Program  reports on outcomes for the two classes completing the 22-week  residential phase during that reporting year and for the two preceding  classes as they complete their 1-year post-residential follow-up phase. The  Challenge Program sites use the same automated system, DMARS, to  collect information on students and report on their progress and activities.   The information collected in DMARS is reviewed by the contractor  through weekly and monthly reports and during random checks of source  documents during operational evaluation site visits.
Some residential phase outcomes of the Challenge Program, such as the  number of graduates earning a general educational development (GED)  credential or high school degree and changes in scores on standardized  math and reading tests, are tied to the core component of Academic  Excellence.  For example, NGB reported that 70 percent of graduates in  2004 earned a GED.  Figure 5 illustrates the outcomes of GED attainment  for the past 5 reporting years.  Students also take the Tests of Adult Basic  Education, a series of tests that identify individual education levels in  various academic subject areas. Each state program tests its students early  in the residential phase and then toward the end of the 22-week period,  and it reports the changes in test scores. In 2004, for example, NGB  reported that graduating students improved 1.7 grade levels in reading and  1.8 grade levels in math during the residential phase. Another core  component, Service to the Community, requires each student to perform a  minimum of 40 hours of service to the community or conservation project  activities. The number of community service hours performed by each  student is tracked, and the total number of hours for each site is another  outcome that the Challenge Program reports annually.  For example, in  2004, NGB reported that Challenge Program students performed more  than 590,000 hours of community service, such as maintaining historical  cemeteries and parks and supporting organizations such as Special  Olympics and Habitat for Humanity.
Each month of the post-residential phase, each Challenge Program  graduate, or that individual’s mentor, reports on the graduate’s post- residential activities. Following the 12-month post-residential phase, each  Challenge Program site reports graduate placements in continuing  education, the military, or the labor force.  These placements are verified  with schools, the military, and employers and are documented.  Program  representatives are not always able to contact all graduates for placement  information and therefore placement data reflect only the students  contacted, not all graduates.  Education placements include returning to  high school or going to a post-secondary or vocational-training institution,  which students may be attending full- or part-time.  Some Challenge  Program graduates also enter the military, into either the active or the  reserve forces.  Post-residential employment placements can be full- or  part-time, and they include those graduates who are self-employed.   Graduates can have placements in more than one of these categories.  For  example, an individual might be attending school and working part-time.   Challenge Program sites continue to update their placement records after  the 12-month follow-up period when they come in contact with former  students.  During the longitudinal study, for example, the contractor has  been able to update placement data based on information received from  state program officials and graduates.  Figure 6 shows post-residential  placement trends for the past 3 reporting years. The total numbers of  graduates placed in these 3 years are 2,407 in 2003; 3,698 in 2004; and 4,086  in 2005.
section_title: Reserve Affairs and NGB Have Not Analyzed Data to Determine Actual Program Costs or the Need to Adjust the Federal and State Cost Share
paragraphs: Although Reserve Affairs and NGB have expressed concern about the  current program funding level and have suggested increasing both the cost  basis used to determine funding needs and the federal cost share, we  found that neither Reserve Affairs nor NGB has performed analyses to  support the need for such changes.  Good budget practices, included in the  Office of Management and Budget’s Federal Financial Accounting  Standards, state that agencies should determine actual costs of their  activities on a regular basis and that reliable cost information is crucial for  effective management of government operations.  Without better cost and  financial information, DOD cannot justify future funding requests or a  change in the cost-share ratio.
Other than calculating how inflation has affected program costs, NGB has  not analyzed data on actual program costs.  Since 1993, NGB has used a  cost of $14,000 per student as the basis for determining the amount of  funds needed to cover program operating costs.  In 2003, NGB calculated  that if that amount were adjusted for inflation, it would be $18,000.  The  results of our survey of all Challenge Programs showed that in 2004, states  actually spent between $9,300 and $31,031 per graduate with an average of  $15,898 per graduate.  In addition, our survey showed that, on average,  states estimated that the program should be funded at approximately  $16,900 per target graduate to cover all of the services in the cooperative  agreements, although the estimates ranged from $14,000 to $31,800.  Most  Challenge Program officials also told us that increasing the cost per  student funding level for the program without increasing the federal cost  share would negatively impact their programs because their states would  be unlikely to come up with the additional state match money.  Because  costs vary between states due to regional differences in salary levels, staff  benefits, and facility costs, Reserve Affairs has asked NGB to determine a  new funding formula for the program based on individual state needs.  At  the time of our review, NGB had not yet done this and Reserve Affairs has  not given NGB a deadline for completion.
In addition to expressing a desire to change the amount of funding per  student, Reserve Affairs, NGB, and participating states have suggested that  the cost-share ratio be changed from its current 60 percent federal share to  a 75 percent federal share because they believe that the current 40 percent  state share is sometimes difficult for states to meet; however, neither  Reserve Affairs nor NGB has analyzed states’ financial situations or the  impact of adjusting the federal and state cost share.  Challenge Program  officials told us that increasing the federal cost share of the program  would be beneficial because it would enable states to expand their  existing programs; give states more flexibility in funding their programs;  and allow programs to restore to students some services that had been  eliminated due to budgetary constraints.  Although we did not analyze how  changing the cost basis or the cost-share ratio would affect specific states,  we prepared hypothetical examples for illustrative purposes.  Table 1  shows how changing the cost basis and the cost-share ratio would affect  federal and state required funding levels.
In our survey, states reported varying views on whether they were  experiencing difficulty in meeting their share of program costs.  We did  not verify the basis for their responses.  Our survey showed that some  states are able to provide funds above the required match; some states  provide only the required match; and some states are unable to provide a  match based on $14,000 per student and therefore fund the program at a  lower level and receive less money from NGB.  For example, California is  able to provide additional support beyond the required match through  additional money provided from the state general fund and funding from  the program’s local school district.  In 2004, California spent  approximately $20,200 per graduate.  Oregon, on the other hand, funds its  program primarily through state education money.  Due to recent state  budget difficulties, Oregon cannot fund the program at $14,000 per  student.  In 2004, Oregon spent approximately $12,600 per graduate.   According to the National Guard Bureau, all 29 programs are providing the  services required by the cooperative agreements, and several states have  added program enhancements such as field trips or vocational classes.   However, some states reported that they reduced nonrequired services to  stay within their budgets.  For example, they implemented pay and hiring  freezes; eliminated the student stipend; and eliminated program  enrichment activities, such as field trips and vocational classes.  Some  states told us that additional funding, provided by a change in the cost- share ratio or an increase in the per student funding amount, would allow  them to restore some of these services.
section_title: The National Guard Bureau Lacks a Complete Framework for Oversight of the Challenge Program, Making It Difficult to Measure Program Effectiveness
paragraphs: Although NGB has several mechanisms in place for overseeing the  Challenge Program, it lacks a complete oversight framework, making it  difficult to measure the effectiveness of the program.  A complete  oversight framework, as suggested by the Government Performance and  Results Act of 1993 (the Results Act) and Standards for Internal Control  in the Federal Government, includes performance goals and measures  against which to objectively measure performance as well as a mechanism  for tracking findings of audits or reviews and responding to those findings.   Currently, NGB does not require participating states to establish  performance goals for individual programs and therefore does not have a  firm basis for evaluating program outcomes and DOD’s return on  investment.  In accordance with the cooperative agreements, U.S. property  and fiscal officers in each state are required to conduct full audits of state  Challenge Programs at least every 3 years.  However, these audits have not  been conducted as required; and, when audits are conducted, copies of the  results are not provided to NGB for review.  Without regular audits and  access to results, NGB cannot be assured that programs are using federal  funds appropriately and that audit findings are addressed.
subsections: section_title: The National Guard Bureau Conducts Several Oversight Activities and Recently Implemented a Mechanism for Tracking Audit Findings
paragraphs: NGB conducts several oversight activities for the Challenge Program.  In  accordance with the DOD Instruction and cooperative agreements  governing the Challenge Program, NGB is responsible for the overall  administration of the program, including program oversight.  NGB uses  both informal and formal mechanisms to oversee the program.  On an  informal basis, NGB frequently communicates with state program  directors via e-mail and telephone calls.  In addition, according to NGB, if  a program director has a problem or an issue that he or she feels NGB  needs to be involved with, he or she will initiate contact.  Formal oversight  of the program is conducted by NGB through AOC Solutions with yearly  operational evaluations and biennial resource management reviews of all  29 Challenge Programs.  The purpose of the operational evaluations is to  assess the programs’ compliance with the cooperative agreements and the  implementation of the eight core components.  The resource management  reviews focus on assessing programs’ financial accountability and  reviewing resources including staffing levels and salaries; food service  costs; and physical inventory of property.
Both the operational evaluations and the resource management reviews  have identified areas for improvement and, according to NGB, changes  were made to the program.  For example, the operational evaluation of  one state program conducted in fiscal year 2004 reported that program  staff was calling the students inappropriate names.  According to NGB, a  staff member was dismissed as a result of this finding.  Another  operational evaluation of a different program conducted in fiscal year 2005  found that over 90 percent of the community service hours accumulated  by the students were for kitchen patrol.  The cooperative agreements state  that work in the dining facility may not be counted towards community  service hours.  According to NGB, this program has completely revamped  its community service program and students now participate in such  activities as visiting with residents of the local veterans’ home and caring  for a historic cemetery.
Until recently, NGB did not have a formal mechanism for tracking the  findings of the reports conducted by AOC Solutions.  During our review,  we discussed with NGB the importance of keeping track of review findings  in order to adequately respond to these findings, in accordance with  Standards for Internal Control in the Federal Government.  In response  to our review, in October 2005, NGB provided new guidance to AOC  Solutions regarding the operational evaluations and resource management  reviews, which required, among other things, AOC Solutions to review  findings from previous evaluation reports to determine whether corrective  actions have been taken where warranted.  In addition, officials at NGB  told us that they currently monitor responses to audit and review findings  informally with individual program directors.
section_title: The National Guard Bureau Does Not Require Participating States to Establish Performance Goals for the Challenge Program
paragraphs: Although NGB requires state Challenge Programs to report on certain  outcome measures, such as GED attainment and number of graduates,  NGB does not require states to establish any performance goals in these  areas to measure the effectiveness of the program.  The establishment of  performance goals is consistent with the principles of effective  management as set forth in the Results Act and would allow NGB to better  evaluate the overall performance of the program and to assess DOD’s  return on its investment.  Without clear and agreed upon performance  goals, there is no objective yardstick against which to fully measure  program performance and thereby assess DOD’s return on investment.
Although it may not be reasonable to have the same performance goal for  all state programs, it would be appropriate for each state program to  negotiate a performance goal for defined performance areas such as  increases in standardized test scores or physical fitness levels.  Similar  state programs, overseen by the Department of Labor, set individual  negotiated levels of performance for specified core performance  measures.  These measures are used to provide information for  systemwide reporting and evaluation for program improvement.  For  example, state Workforce Investment Act programs negotiate  performance measures for youth ages 14 to 18 in three areas: attainment of  basic skills; attainment of high school diplomas or their equivalents; and  placement in education, the workforce, or the military.  For the area of  diploma attainment, the goals range from 42.8 percent of participants in  Louisiana to 68 percent of participants in New Hampshire.
Currently, Challenge Program states are required to submit state plans  annually.  These state plans are required to contain long-term and annual  performance goals and are to be updated annually.  However, NGB has not  provided guidance on specific performance areas where states should  focus their goals; therefore, states may not have goals in the same  performance areas, making it difficult for NGB to compare performance  across programs.  For example, California’s state plan contains a goal to  acquire additional sources of funding through grants and charitable  contributions.  Oregon’s state plan, on the other hand, contains a goal  stating that 80 percent of graduates from the residential portion of the  program will be placed in education, the military, or employment but does  not contain any goal related to acquiring additional sources of funding.  In  addition, states are not currently held accountable to the goals that they  do set since the evaluation process does not measure the states’  performance against their goals.  NGB told us that beginning in January  2006, states would be held accountable to the goals outlined in their state  plans.
section_title: U.S. Property and Fiscal Officers Have Not Conducted Required Audits and Audit Results Are Not Provided to NGB
paragraphs: Although the cooperative agreements governing the Challenge Program  currently require U.S. property and fiscal officers to perform full audits of  state Challenge Programs at least every 3 years, and prior to January 2005,  the cooperative agreements required full audits every year, these audits  have not been conducted as required.  For example, according to NGB, out  of a required 29 audits, only 14 were conducted in 2003, and only 7 audits  were conducted in 2004.  According to property and fiscal officers that we  spoke with, audits were not conducted due to the lower priority placed on  these audits compared to other audits that needed to be conducted within  the state and a lack of staff to conduct the audits.  Because the property  and fiscal officers are responsible for ensuring that federal dollars are  appropriately spent, if these audits are not conducted, it may be difficult to  ensure that federal interests are adequately protected.
When the property and fiscal officers do conduct audits of the Challenge  Program, they are not currently providing copies of the audit results to  NGB because, according to the Chief of Property and Fiscal Affairs at  NGB, there is no specific requirement to do so.  Standards for Internal  Control in the Federal Government states that agencies need to ensure  that the findings of audits and reviews are promptly resolved.  If NGB does  not review these audits, it cannot ensure that audit findings are resolved or  identify trends across programs that may require action at a programwide  level.  According to officials at NGB, the audits remain internal to the state  and the property and fiscal officer works directly with the state Challenge  Program site to resolve any issues.  In addition, if any audit findings  require action from NGB, the state property and fiscal officer will contact  NGB and ask for assistance.  NGB needs to be aware of all audit findings,  including those reported by the property and fiscal officers to effectively  manage the Challenge Program.
We reviewed the most recent property and fiscal officer audits for each  Challenge Program participating state and found that they did identify  areas that needed improvement.  For example, an audit conducted of one  program in fiscal year 2004 found unspent program dollars totaling  approximately $180,000 that needed to be returned to the NGB.  According  to a follow-up report by the property and fiscal officer, this money was  returned to NGB.  Another audit conducted during fiscal year 2003 found  that oversight of budget expenditures for one program was not adequate  to ensure accurate, timely, and complete accounting of expenditures and  recommended that key management controls over the program be  identified, documented, published and tested.  A follow-up audit of the  program found that this recommendation was implemented.
section_title: Reserve Affairs Lacks a Formal Approach for Securing Alternative Funding Support for the Challenge Program, While Some Participating States Have Obtained Alternative Funding Support as a Result of Their Efforts
paragraphs: Reserve Affairs has not adopted a formal strategy for pursuing nondefense  funding, while some participating states have obtained alternative funding  support for their programs as a result of their own efforts.  Because  Reserve Affairs has not made a formal business case to request funds from  nondefense agencies, these agencies are unable to determine whether or  not they are able to fund the Challenge Program.  As a result, Reserve  Affairs is potentially missing out on additional sources of funding that  could enhance the program.  Some participating states have obtained  alternative funding support to enhance or maintain services provided to  Challenge Program students.
subsections: section_title: Nondefense Agencies Are Unable to Determine If They Can Fund the Challenge Program without a More Formal Request for Funding
paragraphs: Reserve Affairs has not made a formal business case to request funds from  nondefense agencies and therefore these agencies are unable to determine  whether or not they are able to fund the Challenge Program.  Although the  authorizing legislation for the Challenge Program allows the Secretary of  Defense to use nondefense funding sources in support of the program,  Reserve Affairs has not adopted a formal strategy for pursuing nondefense  funding.  Rather, Reserve Affairs has primarily adopted informal strategies  to contact agencies outside DOD to inform them about the Challenge  Program and seek opportunities for partnerships.  For example, Reserve  Affairs officials have sent information via e-mail to officials at federal  agencies with programs targeting at-risk youth (i.e. Department of Justice  (DOJ), Department of Labor (DOL), and Department of Education (ED))  to inform them about the Challenge Program.  These e-mails did not  include specific requests for program funding.  In addition, in July 2003,  Reserve Affairs presented senior DOL officials with a proposal for forming  an interagency partnership with the Challenge Program, but did not ask  for a commitment of funding from DOL in a specified amount.   Additionally, officials from these agencies have been invited to events  sponsored by the National Guard Youth Foundation in support of the  Challenge Program.  Lastly, Reserve Affairs officials told us that they  participate or have participated on a number of interagency councils and  working groups that represent at-risk youth such as the President’s Crime  Prevention Council, the White House Federal Interagency Working Group  on Service, the Corporation for National and Community Service, the  National Civilian Community Corps, the Math Science Initiative, and the  Juvenile Justice and Delinquency Prevention Council.
Officials from DOJ, DOL, and ED noted that their agencies have general  authority to provide funds to other programs if the transferred funds were  used to support a program consistent with their agency’s interests.  It is  not unprecedented for an agency to transfer funds to the Challenge  Program.  In fiscal years 2000 and 2001, DOD and DOJ signed an  interagency agreement under which DOJ agreed to provide $5 million in  2000 and $7,483,500 in 2001 to the Challenge Program.  As DOJ concluded  in this situation, and as officials at DOJ, DOL and ED stated, the transfer of  funds to another agency must be consistent with the purpose and legal  requirements of the program from which funds would be transferred.   Moreover, these officials stated that sufficient funding must be available in  the program from which the funds would be transferred, and making such  a transfer must be in the interest of the agency.
Officials at DOJ, DOL, and ED stated that Reserve Affairs needed to  present more specific information to them before they could make a  determination as to whether funds could be provided to the Challenge  Program.  These officials stated that an executive branch agency that  wishes to make a request for funding from another executive branch  agency could make such a request in any number of ways, either formally  or informally.  At a minimum, they noted, any such request should contain  the amount of funding sought and a sufficiently detailed description of the  program to allow the agency receiving the request to determine whether it  would be an appropriate use of funds.  At the time of our review, none of  the agency officials we met with were aware of any specific request from  Reserve Affairs concerning this matter.  Until Reserve Affairs makes a  more formal request for funding, other agencies will be unable to  determine if they are able to provide funds for the Challenge Program.
section_title: Some Participating States Have Obtained Alternative Funding Support to Enhance or Maintain Services Provided to Challenge Program Students
paragraphs: Under cooperative agreements with NGB, states are not required to seek  funding support beyond the required federal and state contributions;  however, some states have made efforts to obtain alternative funding  support for their programs to enhance or maintain the services provided to  Challenge Program students.  Under authorizing legislation for the  Challenge Program, states may accept, use, and dispose of gifts or  donations of money, other property, or services for the Challenge  Program.  Reserve Affairs and NGB officials said that they encouraged  states to seek out additional funding sources for their programs if they  want to enhance the services provided to program students.
To assist state programs in their efforts to obtain that additional support,  NGB, through its contractor Dare Mighty Things, shares information  among state programs on strategies for successfully organizing a 501 (c)  (3) corporation, developing a fundraising policy that focuses on the long- term vision of the Challenge Program, accessing and sharing a grant writer  with other programs, applying for National School Lunch Program funds,  and educating state legislators to secure funding support.  By providing  information and examples of programs that are currently implementing  these strategies, other state Challenge Programs have the information  available to them on how to obtain additional funding support for their  programs.  In our review of the program, we found that some states had  identified strategies for soliciting additional funding support from  nonprofit organizations.  For example, in the annual plans submitted to  NGB, some states developed specific strategies for obtaining additional  funding support such as setting a goal for the number of grants they would  apply for and establishing a 501 (c) (3) corporation to raise funds on  behalf of the program.
We saw further evidence of states’ efforts to obtain additional funding  support through our site visits and in our survey of the 29 Challenge  Program sites.  For example, in one state we visited, the Challenge site  was also a charter school, which allowed the program to receive  additional funding from the local school district.  Moreover, in four states  we visited, the state legislatures provided additional funding beyond the  required state match of 40 percent to pay for additional staffing costs or  facilities.  According to our survey of all Challenge Programs, 28 out of 29  programs identified receiving some type of funding support beyond the  required federal and state contributions to pay for program expenditures  incurred during 1999 through 2004.  These programs identified receiving  additional funding from the states’ general fund beyond the required state  match as well as support from other state agencies such as the state  Department of Education.  Additionally, the programs we surveyed relied  on funding assistance from other federal agencies such as the Department  of Agriculture, which provides funding under the National School Lunch  Program; nonprofit organization grants; the programs’ private 501 (c) (3)  corporation; and donations from private individuals.
Results from our site visits and national survey also showed that state  Challenge Programs rely on donations of goods and services to support  the program.  For example, states we visited relied on donations from  private citizens, corporations, schools, and the states’ National Guard units  for such items as computers and software, exercise equipment, books,  uniforms, and shoes.  Some programs also received support from the  surrounding community in the form of donated services such as  transportation assistance and medical services from local doctors and  nurses.  Our survey further showed that 21 out of 29 programs received  some form of donated goods and services.  Program officials we met  stated that donations of goods and services were vital to programs  because many times they do not have funds available to acquire equipment  or services for the program.  Donations of goods and services can enhance  the basic program beyond the military-based training and academics  required under the cooperative agreements adding little or no cost to the  program.
Despite the success that many states had in obtaining additional funding  support and donations of goods and services for their programs, some  states reported obstacles in securing supplemental funding.  For example,  the cooperative agreements governing the Challenge Program do not allow  states to use Challenge Program funds to hire full-time grant writers.   Lastly, some programs we visited expressed concern that if they received  funding support outside the federal and state required contributions, then  their state governments might reduce their allocations.
section_title: Conclusions
paragraphs: Although Reserve Affairs, NGB, and participating states have suggested  the current cost basis of $14,000 per student is not sufficient to sustain the  program and that the cost-share ratio should be changed, Reserve Affairs  and NGB have done little analysis to show what the actual costs of the  program are and how changing the cost-share ratio for the program would  impact participating states.  Without better cost and financial information,  DOD cannot justify future funding requests or a change in the cost-share  ratio.
Until NGB establishes clear and agreed upon performance goals, there is  no objective yardstick against which to fully measure program  performance.  Without these performance goals in place, NGB does not  have a firm basis for evaluating program outcomes and DOD’s return on  investment.
Although property and fiscal officers are required to conduct full audits of  the Challenge Program at least every 3 years, these audits have not been  conducted due to competing priorities at the state level and a lack of staff.   Without these audits, it may be difficult to ensure that federal interests are  adequately protected.   Also, because NGB does not receive copies of  audits conducted by property and fiscal officers, it cannot know what the  findings of these audits were and if changes were made to the programs  based on these findings.
Until Reserve Affairs makes a more formal request for funding, other  agencies will be unable to determine if they can provide funds for the  Challenge Program.  Without a more formal process that outlines the  amount of funding needed and a detailed description of the Challenge  Program and its specific funding needs, Reserve Affairs is potentially  missing out on additional sources of funding that could enhance the  program.
section_title: Recommendations for Executive Action
paragraphs: To improve the management and oversight of the National Guard Youth  Challenge Program, we recommend that the Secretary of Defense, direct  the Under Secretary of Defense for Personnel and Readiness, in  consultation with the Assistant Secretary of Defense for Reserve Affairs  and the Chief of the National Guard Bureau, to take the following three  actions:     Determine the actual costs of the Challenge Program, including states’  ability to fund their share of the program, and use this information, as  appropriate, to support funding requests or a request to change the cost- share ratio.
Establish performance goals to measure the effectiveness of the Challenge  Program.
Direct U.S. property and fiscal officers to conduct audits as required and  require that copies of audit results are provided to the appropriate office  at the National Guard Bureau in order to ensure that the results of audits  are promptly reviewed and resolved.
To strengthen efforts at obtaining alternative funding in support of the  National Guard Youth Challenge Program, we recommend that the  Secretary of Defense direct the Undersecretary of Defense for Personnel  and Readiness, in consultation with the Assistant Secretary of Defense for  Reserve Affairs, to develop more formal strategies for requesting  alternative funding support for the Challenge Program.  Such strategies  may include submitting requests for funding that include the amount of  funding requested and a sufficiently detailed description of the proposed  program to allow potential providers of funds, such as nondefense  agencies, to determine whether it would be an appropriate use of their  funds.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with our  recommendations.   In its overall comments, DOD asserts that our report  reaffirms that the program met its congressional mandate to improve the  life skills and employment potential of participants by providing military- based training and supervised work experience.  Further, DOD’s  comments state that our report validates the core program components of  assisting participants to receive a high school diploma or its equivalent;  developing leadership skills; promoting fellowship and community service;  developing life coping and job skills; and improving physical fitness, health  and hygiene.  DOD also states that it appreciates GAO’s confirmation that  it is properly executing the program.
We do not agree with DOD’s characterization of our report.  Whereas our  report provides background data on program performance and mentions  that DOD has reported positive outcomes from the Challenge Program, we  do not assess whether or not DOD met its congressional mandate; we do  not validate the program’s core components, nor do we confirm that DOD  is properly executing the program, as DOD’s comments suggest.  Rather  our objectives were to examine: (1) historical trends of the National Guard  Youth Challenge Program, including program expenditures, participation,  and performance; (2) the extent to which Reserve Affairs and the NGB  have determined actual program costs and the need to adjust the federal  and state cost-share; and (3) the extent to which the NGB has provided  oversight of the program. We also determined the extent to which Reserve  Affairs and participating states have made an effort to obtain alternative  funding support for the Challenge Program.
In response to our recommendation to determine the actual costs of the  Challenge Program, including the ability of states to fund their share, DOD  concurred, but claims that the matching fund requirement makes the  program vulnerable to state budget cuts.  DOD also contends that until  recently, sluggish state revenues made the cost share requirement more  burdensome and inhibited the ability of both the NGB and the states to  focus on determining the actual costs of the Challenge Program. DOD  claims that the budget shortfalls required the states to reduce and/or  eliminate program services and that, with state revenues slowly closing  their budget gaps, restoring funding for needed program services is being  considered.  DOD also notes that it is currently working on a new funding  formula for the program.
Although DOD’s comments suggest that adequate funding was not  available for needed services, through our survey and other work, we  found that all 29 programs were able to provide the services required by  the cooperative agreements, and several states reported they added  program enhancements such as field trips or vocational classes. However,  some states reported that they reduced nonrequired services to stay within  their budgets.  Furthermore, the basis for DOD’s comment regarding the  effect of the matching fund requirement and the level of state revenues on  the program is unclear.  As we reported, neither Reserve Affairs nor NGB  has compiled or analyzed data on actual program costs, states’ financial  situations, or the impact of adjusting the federal and state cost share.  We  continue to believe that any future funding request, or a request to change  the cost share requirement, should be based on an analysis of the actual  costs of the program.
DOD’s comments are printed in their entirety in appendix III.
We are sending copies of this report to the Secretary of Defense and  interested congressional committees.  We will also make copies available  to others upon request.  This report will be available at no charge on  GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-9619 or pickups@gao.gov.   Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To examine historical trends of the National Guard Youth Challenge  Program, including program expenditures, participation, and performance,  we interviewed officials from the Office of the Assistant Secretary of  Defense for Reserve Affairs; the National Guard Bureau (NGB); and the  contractor that monitors and evaluates state programs, AOC Solutions.   We reviewed other documentation provided by Reserve Affairs and NGB,  such as program funding summaries.  We also reviewed the Challenge  Program’s annual reports submitted to Congress from 1994 through the  present, excluding 1997, which provided us background information on  the program.  For data on participation and performance outcomes,  including general education development credential attainment,  community service hours, and post-residential placements, we relied on  the Data Management and Reporting System (DMARS) that is used by  NGB to collect participant information and track individual and program  activities.  DMARS was implemented for the 2003 reporting year, and the  contractor has performed procedures to clean up the data as far back as  2000. We conducted data reliability tests on the program’s annual reports  and data management system to conclude that the data are sufficiently  reliable for our purposes.  We did not compare the costs or outcomes of  this program to other similar youth programs currently funded by the  federal government since that was not within the scope of this  engagement.
To assess the extent to which Reserve Affairs and NGB determined actual  program costs and analyzed states’ ability to fund their share of the  program, we interviewed officials at Reserve Affairs and NGB as well as  officials at state programs.  We reviewed and analyzed internal budget  documents prepared by state programs and submitted to NGB annually as  well as documents prepared by individual program directors that  discussed the funding situation in their state.  We also reviewed and  analyzed good budget practices as described in federal financial  accounting standards.  Using a semi-structured questionnaire, we  interviewed eight state Challenge Program directors and their budget  officers in seven states and discussed sources of funding for their  programs, actual costs of operating their programs, and the types of  services that are provided to program participants given the available  funding.  States selected for site visits or telephone interviews are shown  in table 2. There were a number of factors which affected the judgmental  sampling of these state programs.  States were chosen because of their  length of time in the program, size, geographic location, experience of the  program’s director, and whether or not there were multiple programs in  the state.  Additionally, in July 2005, we surveyed all 29 participating  Challenge Programs and asked them about the actual costs of their  programs and their assessment of their states’ ability to fund the programs.   This survey had a response rate of 100 percent.
To assess the extent to which NGB has provided oversight of the  Challenge Program, we interviewed officials at Reserve Affairs and NGB  as well as program officials at the state level.  To determine the oversight  responsibilities of federal and state officials involved in the program, we  reviewed and analyzed pertinent laws, the Department of Defense’s (DOD)  Instruction governing the program, and the NGB Master Cooperative  Agreement, which defines the terms and conditions of the program in each  state.  We also interviewed the independent contractor hired by NGB to  conduct yearly and biennial on-site program reviews, AOC Solutions.  We  obtained access to its data management system and analyzed the program  evaluations and resource management reviews it completed on behalf of  NGB since 2003.  We also reviewed and analyzed individual state  program’s annual plans and NGB’s strategic plan to identify the types of  goals set by the Challenge program. We consulted previous GAO work  regarding performance measurement and evaluations, identified best  practices for establishing and measuring performance goals, and reviewed  Standards for Internal Control in the Federal Government and the  Government Performance and Results Act of 1993. We interviewed  representatives of the U.S. Property and Fiscal Officer’s Office in six states  as well as the NGB Chief of Property and Fiscal Affairs to determine the  time frame for completing audits and mechanisms for reporting, tracking,  and resolving issues that arise out of the PFO audits.
To determine the extent to which Reserve Affairs and participating states  have obtained alternative funding support for the program, we reviewed  relevant laws, policies, and reports to determine the relevant authorities  for receiving and transferring funds between federal agencies.  We also  interviewed federal officials at Reserve Affairs and the Departments of  Labor, Education, and Justice to determine the extent to which these  agencies discussed the possibility of transferring funds to DOD in support  of the Challenge program.   To determine states’ efforts at obtaining  alternative funding support for their program’s operations, we conducted a  survey of all 29 participating programs to collect information about their  sources of funding for the program, how funding is distributed across  different program operational functions, the sources and types of donated  goods and services, and descriptions of strategies programs use to obtain  alternative funding support.  Using a semi-structured questionnaire, we  interviewed eight state Challenge Program directors and members of their  staff in seven states and discussed sources of funding for their programs,  the strategies these specific programs use for obtaining alternative funding  support and donations, and the difficulties these states face in seeking out  other funding.
We provided a draft of this report to officials at DOD for their review and  incorporated their comments where appropriate.  We conducted our work  from January 2005 to October 2005 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: National Guard Youth Challenge Program Sites
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgment
subsections: section_title: Acknowledgments
paragraphs: In addition to the individual named above, Harold Reich, Karyn Angulo,  Leslie Bharadwaja, Susan Ditto, K. Nicole Harms, Wilfred Holloway,  Jessica Kaczmarek, Stanley Kostyla, Julia Matta, Renee McElveen, and  John Van Schaik made key contributions to this report.