Runaway and Homeless Youth Grants: Improvements Needed in the Grant Award Process
Published: 2010-05-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Health and Human Services (HHS) awards grants to provide shelter and services to runaway and homeless youth through the Basic Center, Transitional Living and Street Outreach Programs. In response to a mandate for a review of the grant award process for these programs in the Reconnecting Homeless Youth Act of 2008 (Pub. L. No. 110-378), GAO examined (1) grant announcements and application requirements, (2) technical assistance for grant applicants, (3) how grant award decisions are made, and (4) notification of grant award decisions. GAO reviewed requirements, documents, and records associated with this process for fiscal years 2007 and 2008, observed the grant evaluation portion of this process, and interviewed applicants, peer reviewers, and agency officials.
section_title: What GAO Found
paragraphs: Based on GAO's review of past grant announcements for these programs, GAO found that the criteria upon which grant applications were evaluated were not clearly identified or presented in a single location in the announcement. Rather, GAO found that criteria were scattered throughout various sections of the announcement, had multiple labels, and were not presented in an orderly manner. As a result, applications that did not address the criteria from all sections were likely to receive lower evaluation scores, decreasing their chances of receiving a grant. HHS provides technical assistance to potential applicants for runaway and homeless youth grants, as required by statute. Of the 20 applicants GAO interviewed who sought technical assistance, 17 were satisfied with the help they received. Grant award decisions are primarily based on the results of the peer review process, and internal controls in place to ensure that applications are evaluated consistently were not always adequate. GAO found weaknesses in four out of the six procedures the agency relies on to ensure consistent evaluation of applications. For example, although HHS policy requires peer reviewers to be experts in the field of runaway and homeless youth programs, about one- quarter of the reviewers who evaluated applications for 2009 Street Outreach grants had little or no experience in this area. With regard to notification of grant award decisions, GAO found that they have not always been communicated to applicants in a timely manner, which can delay the start of new programs and present planning challenges for existing ones. GAO also found that the information in notification letters to applicants who were not awarded grants was not always clear or complete.

section_title: Letter
paragraphs: According to the Department of Health and Human Services (HHS),  thousands of U.S. youth run away from home, are asked to leave their  homes, or become homeless each year. Without shelter and guidance,  these youth are vulnerable to exploitation and involvement in illicit  activities, such as selling drugs and prostitution. In 2008 and 2009, HHS  awarded more than $100 million each year in discretionary grants to  providers of shelter and services for this vulnerable population through  three programs for runaway and homeless youth. The Basic Center and  Transitional Living Programs fund short-term and longer-term shelter for  youth, respectively, and the Street Outreach Program funds services to  prevent sexual abuse and exploitation of these youth.
Because only about one out of three homeless and runaway youth grant  applications is approved, it is particularly important that the agency have  systems in place to help ensure consistent and transparent grant award  decisions. The Reconnecting Homeless Youth Act of 2008 mandated a  GAO review of HHS’s process for awarding certain Runaway and  Homeless Youth Act grants. This report responds to that mandate by  addressing the following questions: (1) How clear are grant  announcements and application documents and requirements? (2) How  useful do applicants find the technical assistance they receive to assist  them with applying for grants? (3) How are grant award decisions made  and to what extent does this process comply with federal requirements?  (4) To what extent are grant award decisions communicated to applicants  in a timely and clear manner in accordance with federal requirements?
To answer these questions, we reviewed relevant laws, regulations, grant  announcements, applications and other agency documents, and we  interviewed key agency officials and contractors. To determine applicants’  views on the usefulness of technical assistance, we randomly selected and  interviewed 24 out of the 590 applicants that competed for fiscal year 2008  grants. This sample included applicants who were awarded grants and  applicants who were not. We also randomly selected and interviewed 6  peer reviewers out of approximately 170 who evaluated applications for at  least one of the three grant programs for fiscal year 2008. Additionally, we  interviewed representatives from the National Network for Youth, an  organization that represents providers of services to youth and families.  To determine how grant award decisions are made, we analyzed agency  data and documents related to peer review evaluations and final award  decisions for the three programs for fiscal years 2007 and 2008. We also  observed the peer review process for the Street Outreach Program in June  2009. To determine the extent to which the grant award process complies  with federal requirements, we compared the grant award process to  relevant requirements in the law, HHS regulations and written guidance,  and internal control standards for the federal government. To determine  the extent to which grant award decisions are communicated in a timely  and clear manner in accordance with federal requirements, we reviewed a  random sample of 69 notifications that were sent to applicants for at least  one of the three programs for fiscal year 2008 grants. Additionally, we  established a timeline for each grant program’s award process for fiscal  years 2007 and 2008.
We conducted this performance audit from March 2009 through April 2010  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Program Descriptions
paragraphs: The Runaway and Homeless Youth Act, as amended, authorizes federal  funding in the form of discretionary grants for three programs to assist  runaway and homeless youth. These programs are administered by the  Family and Youth Services Bureau (FYSB) within HHS’s Administration  for Children and Families (ACF).  The three programs—the Basic Center  Program, Transitional Living Program, and Street Outreach Program— enable local public and private organizations and shelters in all 50 states  and the U.S. territories to compete for grants that allow them to serve  runaway, homeless, and sexually exploited youth who may be on the  streets and in need of shelter or longer-term support.
The Basic Center Program provides temporary shelter, counseling, and  other services to runaway and homeless youth under the age of 18. Basic  Center grants are awarded competitively to providers and may be awarded  for a period of up to 3 years.  The Transitional Living Program provides  homeless youth ages 16 through 21 with longer-term residential services  for up to 18 months.  These services include such things as counseling  and education in basic life skills, interpersonal skills, educational  advancement, job attainment skills, and physical and mental health care.  Transitional Living grants are awarded competitively to providers and may  be awarded for up to 5 years. The Street Outreach Program provides  education, treatment, counseling, and referrals for runaway, homeless  youth under the age of 18 who have been subjected to or are at risk of  being sexually abused and exploited. Street Outreach grants may be  awarded for up to 3 years. See figure 1 for key aspects of these three  programs.
For fiscal years 2002 through 2009, funding for these programs has been  just over $100 million in total, with Basic Center funding representing the  largest dollar amount authorized of the three grant programs. Funding for  these programs over the past several years is shown in table 1.
section_title: Grant Award Process
paragraphs: HHS’s grant award process for Runaway and Homeless Youth grants is  comprised of several major steps. HHS’s Family and Youth Services  Bureau, which is referred to as the Program Office, and the Grants  Management Office are responsible for carrying out and overseeing this  process. Some of the steps in the grant award process are performed by  contractors on behalf of the agency, and one step in the process is  performed by panels of peer reviewers selected by the agency to evaluate  grant applications.
The grant award process consists of the major steps as illustrated in figure 2.
Grant Announcement: Each fiscal year, the agency develops and  publishes a grant announcement for each grant program. Announcements  provide the information potential applicants need to determine if they are  eligible to apply and the instructions on how to complete and submit their  application. In addition, they include the criteria used to evaluate  applications.
Technical Assistance: Each announcement lists the technical service  providers responsible for providing technical assistance to potential  applicants to help them understand the announcement requirements.  Technical assistance can be provided through a webinar, seminar,  information on the Web, or upon request.
Application Submission: Applications may be submitted electronically  via Grants.gov or by hard copy via mail or other delivery service.  Applicants may also hand deliver their application to the agency’s  contractor responsible for receiving the applications. The deadline for  submitting applications is usually 45 to 60 days after an announcement is  published.
Application Pre-Screening: Applications are prescreened to determine  whether they meet two requirements. Applications are eliminated from  review if they are received after the deadline or if they request more  funding than the maximum amount specified in the announcement.
Peer Review of Applications: Applications remaining after pre- screening are submitted to peer review panels. A panel generally consists  of three peer reviewers, who apply the evaluation criteria contained in the  announcement to applications and score the applications, and a panel  chair responsible for facilitating consensus of the peer review panel.
Peer reviewers assign points to each application, based upon specific  criteria that are outlined in the announcement. The points are added up  and the applicant’s average score is derived. This score, ranging from 0 to  100, becomes the basis for the ranked listing of applicants which the  agency uses in its award decisions. Because applicants whose score places  them below the total available funding limit may be denied a grant, a single  point can make a difference between awarding a grant and denying a  grant.
Final Grant Award Decisions: Taking into account peer review panel  scores and comments for each application and, in some cases, other  factors, the Program Office and the Grants Management Office make the  final award decisions. These decisions are documented in the final funding  award decision memos, which contain the listing of all applicants, ranked  by their scores, and the final award decisions.
Notification of Award Decisions: Each applicant is sent a letter that  communicates the grant award decision. Successful applicants are notified  before letters are sent to unsuccessful applicants. application, including those for Runaway and Homeless Youth Grants,  must receive an objective, “…advisory review...by a minimum of three  unbiased reviewers with expertise in the programmatic area for which  applications are submitted.” To meet this requirement, the agency relies on  the peer review process, in which three reviewers convene to evaluate and  score applications based on the criteria outlined in the announcement. The  peer reviewers are defined by the agency as experts in the field of runaway  and homeless youth programs. Figure 3 provides an overview of the peer  review process.
HHS awarded grants to about one-quarter of the applicants that applied in  2007 and 2008, as shown in table 2.
section_title: Grant Announcements Have Not Always Provided Clear and Concise Information
paragraphs: Based on the grant announcements we reviewed and our observation of  the peer review process, the criteria upon which grant applications would  be evaluated were not clearly defined in a single location in the  announcement. Rather, we found that criteria were scattered throughout  various sections of the announcement, had multiple labels, and were not  presented in an orderly manner in a single location. For example, for the  2009 Street Outreach Program grant competition, grant applications were  evaluated and scored based on how well they addressed criteria contained  in three different sections of the announcement. First, applicants must  address the “Program Requirements,” found in Section 1 of the  announcement. Second, applicants must address the “Project  Description,” found in Section 4. Third, applicants must address the  “Evaluation Criteria” found in Section 5. However, only the “Evaluation  Criteria” section of the announcement explicitly described how their  responses would be evaluated and scored. Because the applicant must  address criteria contained in different sections of the announcement, if  applicants focused primarily on responding to the “Evaluation Criteria”  they may not have adequately addressed information in the other sections.  If the applicants focused only on the “Evaluation Criteria,” these  applicants could have received lower scores, which would have decreased  their likelihood of being awarded grants. Figure 4 represents the various  locations where we found descriptions of criteria.
Additionally, peer reviewers we interviewed noted that consolidation of  criteria in the announcement into a single location would aid them in their  evaluation of applications by reducing the time it would take to review the  application because they would not need to look in multiple places in the  application for information.
During our observation of the 2009 Street Outreach Program grant  competition, we found that the agency provided detailed guidance to peer  reviewers to help them evaluate and score applications. This guidance,  which was not available to applicants, consolidated information from  various sections of the announcement. The federal officials instructed  reviewers to focus on specific information when evaluating and scoring  applications. Because applicants did not have this detailed guidance,  which combined information from various parts of the announcement,  applicants may not have had full knowledge of what information was  critical to receiving a high score. Table 3 shows examples of the guidance  provided to peer reviewers during the 2009 Street Outreach Program grant  competition.
section_title: ACF Provides Technical Assistance That Applicants Found Helpful
paragraphs: ACF provides technical assistance to potential applicants for runaway and  homeless youth grants, as required by statute. Technical assistance is  generally defined as providing expertise or support to applicants and  grantees for the purpose of strengthening their capabilities for providing  shelter and support services for runaway and homeless youth. In fiscal  years 2006 and 2007, the agency provided technical assistance to potential  applicants through its regional network of 10 providers, and listed these  providers in its announcements.
However, beginning in September 2007, ACF centralized its technical  assistance in order to provide more consistent technical assistance for all  applicants, regardless of where they were located. At that time, the agency  entered into cooperative agreements with the University of Oklahoma to  provide technical assistance nationwide.
Through its providers, the agency coordinates technical assistance, which  generally consists of a pre-application conference (webinar) covering the  application requirements such as the project description, eligibility, and  the evaluation criteria, among other things. After the conference, a  recording and transcript is posted on the agency Web site.
Potential applicants may also ask specific questions of the contacts listed  on the announcement. These contacts include the Program Office officials  and technical assistance providers. If the technical assistance providers  cannot answer the questions, they coordinate with agency staff to obtain  responses that are then posted to the Web site. The technical assistance  providers also arrange seminars on broader topics related to runaway  youth, such as mental health, crisis intervention, and skills training.
Most of the applicants we interviewed who received technical assistance  under both systems reported that they found it helpful. For example, 17 of  the 20 applicants who sought technical assistance were satisfied with the  help they received. However, three of these applicants said they prefer the  technical assistance provided by their regional providers because of such  things as the regional assistance being more “hands-on,” the regional staff  being more responsive and accessible, and the regional staff being more  knowledgeable of local programs. Agency officials noted that the agency  has moved toward centralized approach to gain a more consistent  approach to the technical assistance it provides.
section_title: ACF’s Process for Determining Grant Awards May Not Ensure Consistent Decisions
subsections: section_title: ACF’s Peer Review Process Has Weak Internal Controls
paragraphs: ACF’s process for determining which grant applicants will be awarded  grants is primarily based on the results of the peer review process, which  has weak internal controls to ensure that applications are evaluated  consistently. According to GAO standards, internal controls should  provide reasonable assurance that the agency’s objectives, such as  providing grants to the most qualified providers, are being achieved.  Ideally, internal controls should be continuous, built-in components of the  agency’s processes, and should provide reasonable assurance that the  grant award process works as it is designed to work. Our review of ACF’s  grant award process found that, while the agency has a number of internal  controls in place to help ensure consistent application of evaluation  criteria across reviewers and across panels, some of these controls are  limited in their effectiveness. For example, we found weaknesses in four  out of six internal controls related to the grant award process, as shown in  table 4.
First, ACF does not always select peer reviewers whose qualifications  comply with the standards outlined in HHS policy. The policy states that  each application for runaway and homeless youth grants must receive an  objective, advisory review by a minimum of three unbiased reviewers with  expertise in the programmatic area for which applications are submitted.  Furthermore, the announcements we reviewed stated that grant  application reviewers should be experts in the field of runaway and  homeless youth programs. However, we found that HHS considered  students, school teachers, business consultants, and television and media  workers as qualified peer reviewers. Our review of resumes of all the peer  reviewers and chairs for 2009 Street Outreach Program grants found that  many had professional and volunteer experiences that were not always  directly related to runaway and homeless youth programs. Based on the  resumes of 76 peer reviewers, we found that 26 peer reviewers had direct  experience with runaway and homeless youth programs listed on their  resume, and another 31 had indirect experience, such as social work,  teaching, or grant reviewing. However, 19 did not appear to have any of  the relevant knowledge and expertise in runaway and homeless youth  programs required by HHS policy. Three of these 19 reviewers were  identified as “youth reviewers” in their resumes. One agency official  responsible for the grant review process during 2009 explained that HHS  interprets its policies governing peer reviewer qualifications broadly and  accepts all related experience. He also noted that HHS encourages the use  of “youth reviewers” for its peer review panels.
Second, during our observation of the 2009 Street Outreach Program grant  competition, we found that the meetings for peer reviewers and chairs  were not mandatory. These meetings included an orientation session,  panel chair meetings, and new reviewer meetings. Meetings for panel  chairs are particularly important for helping to ensure consistent  evaluations across panels because in these meetings, all panel chairs agree  on how to apply the evaluation criteria. However, we observed that some  panel chairs did not attend these meetings, and, therefore, their panels  may not have applied the evaluation criteria in the same manner as panels  whose chairs had attended the meetings. Similarly, new reviewers were  permitted to miss the new reviewers’ meetings and still participate in the  reviews, which could also increase the risk of inconsistent application of  evaluation criteria.
Third, we observed that the agency provided detailed guidance to peer  reviewers to aid them in evaluating applications. The detailed guidance  provided to reviewers explaining the evaluation criteria has led to  variation in application of criteria by review panels. For example, when we  observed peer review panel deliberations for the 2009 Street Outreach  grants, we found that peer review panels varied in the way they applied  the criterion for evaluation of emergency evacuation plans. The  announcement’s “Evaluation Criteria” section contained the following  evaluation criterion related to emergency plans:  The application “describes the emergency preparedness and management  plan by addressing steps to be taken in case of a local or national situation  that poses risk to the health and safety of program staff and youth.”
At the panel session, agency officials told panels that they should also  apply all of the information in the detailed guidance they were given,  which included information in the “Program Requirements” section of the  announcement. Federal officials advised peer reviewers that they should  score the application on the following information:  “Grantees must immediately provide notification to FYSB when  evacuation plans are executed.”
As a result, peer review panels that followed the guidance gave lower  scores to applicants that did not specifically indicate that they would  notify the agency when an evacuation occurred. One peer review panel we  observed, however, did not give lower scores when this was not specified  in an application. These peer reviewers said that they did not think it was  fair to assign lower scores in these cases because the more detailed  information about evacuation requirements was not listed in the  “Evaluation Criteria” section of the announcement.
Additionally, we interviewed peer reviewers who participated in panels for  2008 runaway and homeless youth grants. Three of the six peer reviewers  we interviewed told us they observed variations in the way panels applied  the criteria. Reviewers said that the 2008 Transitional Living Program  announcements contained evaluation criteria requiring applicants to  provide background checks for all staff members who would be working  with youth. However, the peer reviewers told us that the guidance  provided to peer reviewers by the agency during that review process  further specified that these background checks must be conducted in  accordance with local, state, and national requirements. According to the  peer reviewers we interviewed, this could have led to variation in how this  aspect of the application was evaluated by different panels. Given that the  peer review score is the key factor in determining grant awards,  inconsistent evaluation criteria across panels can have a significant impact  on whether an applicant is awarded a grant or not.
The fourth control weakness we observed during our review of the 2009  Street Outreach Program grant competition was that agency officials did  not keep a permanent record of their comments and feedback to peer  review panels during their oversight of the peer review process, which  introduced further potential for inconsistent application of evaluation  criteria. Agency officials review the panel’s scores and narrative comments  for each application during the peer review process before they are  finalized. The officials visit panels as peer reviewers deliberate and  respond to their questions, and provide feedback to chairpersons on their  panel’s evaluations. Agency officials told us their review and feedback is  meant to ensure that all panels apply the evaluation criteria in the same  way. However, the federal officials we observed did not record this  information in a permanent record. Instead, the officials provided their  feedback to the chair via comments written on post-it notes. This lack of  permanent documentation of federal official feedback to peer review  panels makes it difficult for the agency to ensure that it is providing  consistent guidance to panels and responding to problems across panels in  the same way during the peer review process.
No weaknesses were apparent in two of the six internal controls--(1) the  provision of standard training materials to peer reviewers prior to panel  sessions, and (2) the presence of federal officials on site during panel  sessions to respond to questions from, and communicate on a daily basis  with panels.
section_title: The Basis for Denying Grants Is Not Always Documented
paragraphs: Final funding decision memos used to internally document grant award  decisions for 2007 and 2008 did not contain supporting information  regarding why applications with high scores were not funded. Final  decisions regarding grant awards are determined by HHS’s Program Office  and Grants Management Office, taking into account the review panels’  scores and narrative comments for each application. According to HHS  policy and guidance, the agency has the discretion to deny a grant to an  applicant who would otherwise receive one based on the results of the  peer review score alone. The agency is permitted to use its discretion to  deny grants based on other reasons, such as the agency’s concerns about  the applicant’s program or about the concentration of service providers in  the applicant’s location, which is referred to as concerns about  “geographic distribution” of services. However, the agency does not  always clearly document the rationale for its decision to deny a grant  based on “geographic distribution” of services. When grants were denied  for geographic reasons in 2007 and 2008, we found that the final funding  decision memos did not clearly describe the details surrounding such  denials, such as the number of other programs that exist in the same  locale, the services they provide, or the numbers of youth they serve. Such  details could support or justify a denial for geographic reasons. Without  fully documenting and permanently recording its rationale for exercising  its discretion to deny grants to highly scored applicants, the agency  decision-making process is not transparent.
section_title: Notification of Grant Award Decisions Have Not Always Been Timely, and Notices Are Not Always Clear
subsections: section_title: Notification Time Frames Can Present Planning Challenges for Some Applicants
paragraphs: Grant award decisions are not always communicated in a timely manner,  which may present planning challenges for some applicants. According to  one ACF official, successful applicants are generally notified at the end of  the federal fiscal year. Based on our review of grant documents for fiscal  years 2007 and 2008, we found that for all but the 2008 Transitional Living  Program grants, this was true, regardless of when the announcement  closed or when the funding decisions were made. For example,  applications for the 2008 Basic Center Program were due in February 2008  and were evaluated and scored in March; however, applicants were not  notified of their award status until September, 6 months later. HHS policy  does not indicate when notification letters should be distributed to  applicants, but according to an ACF official, awards to successful  applicants are made by September 30 because most new programs are  expected to start on or before October 1. Given the proximity of the  notification date to program start date, some successful applicants with  new programs we spoke with told us that the September notification  timeframe did not allow enough preparation time to hire staff and secure  the resources needed to provide services. See figure 5 for the timeline of  dates for key events for the fiscal year 2007 and 2008 grant award process.
Notification delays also create planning issues for ongoing programs that  are not awarded new grants and, as a result, need to develop contingency  plans for continuing or discontinuing services. Since unsuccessful  applicants are notified of their grant award status after successful  applicants have been notified, an applicant whose previous grant is about  to expire may experience planning problems if notifications are delayed.  Delays in notifying unsuccessful applicants may not give applicants  adequate time to react to not being awarded a new grant. In the event that  funding is denied or discontinued, earlier notification of award decisions  could help providers properly plan.
According to an ACF official, there is nothing in policy that prohibits  notifying an applicant as soon as award decisions have been made. The  official told us that delays in sending out notification letters are linked to  the timeliness of writing and issuing the announcement. According to this  official, announcements must be reviewed by many departments within  the agency, and, therefore, the turnaround time is not as timely as it could  be. However we found that even after the announcements were published  and closed, applicants were still not notified of their award status for  several months. For example, for the 2008 Transitional Living grant,  regardless of when the announcement was published, applicants were not  notified of their award, until close to 4 months after the panels had  completed evaluating the applications. Similarly, notifications of decisions  related to 2008 Basic Center grants were not sent out until about 7 months  after the panels.
In addition to the challenges applicants experienced due to notification  delays, the agency created additional planning challenges for applicants  when it unexpectedly changed the timing of the funding cycle for the  Transitional Living Program in fiscal year 2008 without notifying  applicants of this change in a timely manner. The announcement stated  that ACF anticipated making grant awards in the first quarter of fiscal year  2008, which would have been from October through December of 2007.  However, the grant award start date was changed to March 2008 after this  announcement was published. According to an agency official, the original  start date was moved in an effort to spread out the timing of peer review  panels for each of the three runaway and homeless youth programs and  other activities that were scheduled to occur around the same time during  the summer months. As a result of moving the cycle start date—from  October to March—some successful applicants were without federal  funding for several months between the end of the previous grant cycle  and the new grant award start date.
Runaway and homeless youth service providers have also raised concerns  to their congressional representatives about the timeliness of notifications.  Specifically, we reviewed nine complaint letters that were sent to  congressional representatives regarding runaway and homeless youth  grants applications in 2007 and 2008. One letter, representing six  providers, stated that notification delays created planning problems for  service providers who were not able to develop contingency planning for  either the continuation or discontinuation of their programs. ACF  responded to the complaint by noting that it offers funding for successful  applicants to recoup some of the costs that programs incurred due to the  delay.
In addition, the National Network for Youth, an organization that  represents providers of services to youth and families also noted that the  timeliness of notifications has been an issue of concern for its  membership. In particular, some service providers have raised issues  about the difficulties receiving timely communications from ACF  concerning grant awards.
section_title: Notification Letters to Applicants Are Not Always Clear
paragraphs: All of the successful applicants we spoke with felt that their notification  letters were clear and contained sufficient information; however,  unsuccessful applicants were not all satisfied with the clarity and  completeness of the information presented in their letters. The standard  letter to unsuccessful applicants may list several possible “other factors”  for the denial, beyond their peer review panel score, without any  indication of which of the reasons listed in the standard notice applied to  their application. See appendix I for a standard letter. The “other factors”  include:  “comments of reviewers and government officials,” “staff evaluation and input,” “geographic distribution,” and  “audit reports and previous program performance.”
Some unsuccessful applicants told us the letter did not contain enough  information for them to understand why their application was denied. In  particular, some applicants told us that they did not understand what the  agency meant by geographic distribution, which was the basis for denying  grants to at least eight applicants during fiscal years 2007 and 2008.
Officials told us that “geographic distribution” means that an applicant was  denied because the geographical area their program would serve is already  served by another runaway and homeless youth service provider. The  agency does not keep a record to document detailed information that  would support or justify a denial for geographic reasons, such as the  number or names of programs that exist in the same locale, the services  these programs provide, or the numbers of youth they serve. As a result, it  is not possible to verify that denying a grant based on “geographic  distribution” was justified.
Applicants who want further explanation of their award decisions may  request additional information along with their scores from ACF through a  Freedom of Information Act request. An ACF official told us that it would  be difficult to provide all unsuccessful applicants more information  supporting the denial decision based on other factors such as “geographic  distribution” in notification letters because of limited resources. However,  the resources needed to provide such information may be small, given that  “geographic distribution” was the basis for denying grants to only a small  number of applicants during fiscal years 2007 and 2008.
Moreover, based upon our review of decision notices sent to applicants  who were screened out of the competition due to late submissions or  improper funding requests, we found contradictory language that may  confuse applicants. Specifically, the letter states that “the limited  availability of funds permitted us to select only the highest scoring  applications that also met all of the eligibility requirements,” leaving the  impression that the application, in these cases, had been evaluated and  scored by a peer review panel. However, applications that are screened- out of the process before the peer review session are not evaluated or  scored. When we pointed out this statement to the agency, officials agreed  the language could be confusing to applicants.
section_title: Conclusions
paragraphs: The runaway and homeless youth grant programs provide much needed  services to a vulnerable population and the number of applications far  exceed the number of grants that can be awarded with available funding.
To ensure that ACF awards these grants to the most capable applicants, its  award process must be fair and transparent.
Without clearly organized evaluation criteria in grant announcements,  applicants can have difficulty determining what their applications will be  evaluated on. Furthermore, without consistent evaluation of applications  in the process, there cannot be a level playing field for all applicants. All  peer reviewers must have the required programmatic expertise, or not all  applicants are evaluated by their peers. Additionally, unless all peer  reviewers attend meetings at panel review sessions; these meetings cannot  help ensure consistent evaluation of applications. Without documentation  of ACF comments to peer review panels during the review process there is  also a risk that the evaluation process will not be consistent. Moreover,  without fully documenting the rationale for denying grants to highly  scored applicants, agency grant award decisions are not transparent.
Once the grant award decisions are made, it is incumbent on ACF to notify  applicants of decisions in a timely manner and provide them with clear  and specific information about, in particular, decision not to grant awards.  Without such notification, applicants may experience planning challenges  and not fully understand the reasons they were denied grants.
section_title: Recommendations
paragraphs: To enhance transparency and fairness in the grant award process, and  improve grantees ability to plan for services, we recommend that the  Secretary of Health and Human Services direct the Assistant Secretary for  the Administration for Children and Families to take the following seven  actions:    Clearly identify in grant announcements all the criteria that peer reviewers  will use to evaluate and score applications, and ensure that peer reviewers  use only those criteria during the peer review process.
Select peer reviewers with expertise in the programmatic area for which  they are evaluating grant applications.
Make all meetings for peer reviewers, including those for new reviewers  and chairs, mandatory.
Document and maintain records of ACF comments to peer review panels  during the review process.
Document the specific reasons for denying grants to high-scoring  applicants in favor of other applicants for the agency record.
Provide clear information to applicants about the specific reasons their  applications were not approved.
Notify applicants about the outcome of their applications as soon as grant  award decisions are made.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Health and Human  Services for review and comment; these appear in appendix II. In its  comments, HHS disagreed with our recommendation to review and revise  announcements to ensure that all evaluation criteria listed be clearly  labeled as evaluation criteria and be contained in a single section of the  announcement.  HHS maintains that all of the criteria used to evaluate and  score applications are contained in section 5 of the announcement.   However during the peer review process we observed, in addition to  evaluating and scoring applications based on criteria specified in the  “Evaluation Criteria” section of the announcement (section 5), some of the  panels evaluated and scored applications based on criteria from two other  sections of the announcement. Given the difference between the agency’s  response to our recommendation and what we observed, we are revising  our recommendation to highlight the need to ensure that the all criteria  used to evaluate and score applications are clearly identified to applicants  and peer reviewers, and that peer reviewers use only those criteria when  evaluating and scoring applications.
With regard to our recommendation to select peer reviewers with  expertise in the program for which they are evaluating grant applications,   HHS commented that the agency has elected to accept reviewers who are  knowledgeable of the risk factors faced by runaway and homeless youth,  and that many professional disciplines often intersect with runaway and  homeless youth. However, we found that in the past the agency has used  individuals that would not be expected to have relevant expertise, such as  television and media workers.  Noting our concern in this area, the agency  indicated that they plan to take steps to ensure that all reviewers possess  the knowledge and expertise in the particular program for which they are  reviewing grant applications.  In the event of a shortage of reviewers, the  agency intends to staff panels with at least one peer reviewer with  extensive relevant knowledge, which would continue to differ from the  current policy that grants must receive an objective, advisory review by a  minimum of “three” unbiased reviewers with expertise in the  programmatic area for which applications are submitted. We agree that  professionals in varied disciplines could have sufficient expertise to serve  as reviewers and recognize that it may be difficult for the agency to find  enough reviewers with expertise in a particular program.  As a result, we  are clarifying our recommendation to include those that have expertise in  the programmatic area for which they are evaluating grant applications,  and not a specific program.
Regarding our recommendations to make peer review meetings  mandatory, HHS indicated that all meetings for peer reviewers and chairs  are already mandatory but due to unforeseen factors, it is not always  possible for all reviewers to attend.  Indeed, during our observation of a  peer review session, not all reviewers and chairs attended the meetings.   Moreover, at the time, agency officials told us that attendance at these  meetings was not explicitly mandatory, but highly encouraged.  They also  indicated that attendance was not enforced and attendance records were  not maintained.  In response to this recommendation, the agency indicated  that they plan to officially notify all reviewers and chairpersons  participating in future reviews that all training is mandatory. In the event  some reviewers and chairpersons are not able to attend the mandatory  training sessions due to unforeseen circumstances, the agency intends to  offer “make up” sessions.
HHS did not provide comments on our recommendation to maintain  records of ACF comments to peer reviewer panels during the review  process. However they agreed with our recommendation to document the  specific reasons for denying grants to high-scoring applicants in favor of  other applicants.  HHS commented that the agency plans to include more  details concerning geographic distribution in the letters to applicants who  are denied grants for this reason.  While these efforts would be in line with  our recommendation; the details supporting such decisions must be  consistently documented in the agency’s records to support the  information provided to applicants in their letters.
In response to our recommendation to provide clear information to  applicants about specific reasons their applications were not approved,  HHS stated that in accordance with ACF policy and procedures, every  unsuccessful applicant is entitled to an explanation of why their  application was not funded.  In addition, the agency noted that, upon  request, the Program Office will provide a debriefing to applicants.  However, letters sent to unsuccessful applicants should clearly note that  applicants may request a debriefing by the Program Office regarding  specific reasons why their application was not funded. Currently, letters to  unsuccessful applicants do not include this information.  In addition, it is  important to revise the language in letters to applicants that are screened  out of the grant competition that implies their application was evaluated  and scored.
Finally, HHS agrees with our recommendation to notify applicants about  the outcome of their application as soon as grant award decisions are  made.  As part of the grant application process, the agency plans to  explain to applicants that final grant decisions depend on the results of the  grant award negotiations between ACF and the prospective grantees. We  recognize that these grants are discretionary and that final award  decisions involve negotiations that may take time.  However, every effort  should be made to complete negotiations and notify both successful and  unsuccessful applicants as quickly as possible.  To enable applicants to  efficiently and effectively manage their programs, it is important for  applicants to receive their notices in a timely manner.
HHS also provided technical comments, which we incorporated into the  report as appropriate.
We are sending copies of this report to the Secretary of HHS, relevant  congressional committees, and other interested parties. In addition, the  report will be made available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  Kay E. Brown at (202) 512-7215 or brownke@gao.gov. GAO staff who  made major contributions to this report are listed in appendix III.
section_title: Appendix I: Administration for Children and Families’ Notification Letters to Grant Applicants
section_title: Appendix II: Comments from the Department of Health and Human Services’ Administration for Children and Families
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Clarita Mrena (Assistant Director)  and Jacqueline Harpp (Analyst-in-Charge) managed all aspects of the  assignment up to report production. Anna Kelley managed the report  production and Vernette Shaw made significant contributions in all  aspects of the work. Lisa Fisher and Jennifer McDonald also made  significant contributions to this report. Additionally, Walter Vance and  Minette Richardson provided technical support in design and  methodology. James Rebbe provided legal support and Susannah Compton  assisted in message and report development. James Bennett assisted with  visual communications.