Education Grants: Promise Neighborhoods Promotes Collaboration but Needs National Evaluation Plan
Published: 2014-05-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Education's Promise Neighborhoods program is a competitive grant program with goals to improve educational and developmental outcomes for children in distressed neighborhoods. The grants fund community-based organizations' efforts to work with local partners to develop and evaluate a cradle-to-career continuum of services in a designated geographic footprint. As it is one of several federal programs using this model GAO was asked to review the program.
This report examines: (1) the extent to which Education's strategy for awarding grants aligns with program goals; (2) how Education aligns Promise Neighborhoods efforts with other related programs; (3) how Education evaluates grantees' efforts; and (4) the extent to which grants have enabled collaboration at the local level, and the results of such collaboration.
GAO reviewed Federal Register notices, applications, and guidance; surveyed all 48 grantees on the application process, coordination of resources, collaboration, and early results; visited 11 grantees selected based on geography and grant type; and interviewed Education officials and technical assistance providers.
section_title: What GAO Found
paragraphs: The Department of Education (Education) used a two-phase strategy for awarding Promise Neighborhoods (Promise) grants, and aligned grant activities with program goals. Education awarded 1-year planning grants to organizations with the potential to effectively align services for students in their respective neighborhoods. Planning grants were generally intended to enhance the grantees' capacity to plan a continuum of services. Through a separate competition, Education awarded 5-year implementation grants to organizations that demonstrated they were most ready to implement their plans. However, Education did not communicate clearly to grantees about its expectations for the planning grants and the likelihood of receiving implementation grants. As a result, some grantees experienced challenges sustaining momentum in the absence or delay of implementation grant funding.
The Promise program coordinates with related federal efforts primarily through a White House initiative that brings together neighborhood grant programs at five federal agencies. The Promise program's efforts are focused on ensuring that grants are mutually reinforcing by aligning goals, developing common performance measures, and sharing technical assistance resources. While Promise grantees incorporate a wide range of federal programs in their local strategies, Education coordinates with a more limited number of federal programs. Officials told us that they do this to avoid spreading program resources too thin. Further, Education did not develop an inventory of the federal programs that share Promise goals, a practice that could assist grantees; help officials make decisions about interagency coordination; and identify potential fragmentation, overlap, and duplication.
Education requires Promise grantees to develop information systems and collect extensive data, but it has not developed plans to evaluate the program. Specifically, implementation grantees must collect data on individuals they serve, services they provide, and related outcomes and report annually on multiple indicators. However, Education stated it must conduct a systematic examination of the reliability and validity of the data to determine whether it will be able to use the data for an evaluation. Absent an evaluation, Education cannot determine the viability and effectiveness of the Promise program's approach.
The Promise grant enabled grantees and their partners to collaborate in ways that align with leading practices GAO previously identified for enhancing collaboration among interagency groups including establishing common outcomes, leveraging resources, and tracking performance. For example, Education required grantees to work with partners to develop common goals and a plan to use existing and new resources to meet identified needs in target areas. Grantees were also required to leverage resources by committing funding from multiple sources. Implementation grantees were required to collect and use data to track performance. Some planning grantees used a leading collaborative strategy not required by Education that produced early benefits. For example, several grantees and partners told us they completed easily achievable projects during the planning year to help build momentum and trust. Grantees told us that collaboration yielded benefits, including deeper relationships with partners, such as schools, as well as the ability to attract additional funding. However, grantees also said they faced some challenges collaborating with partners, particularly in overcoming privacy concerns related to data collection.
section_title: What GAO Recommends
paragraphs: GAO recommends that Education communicate grant expectations more clearly, identify federal resources that can contribute to the program's goals, and develop a strategy for evaluation. In commenting on a draft of this report, Education outlined the steps it will take to respond to recommendations.

section_title: Letter
paragraphs: The Department of Education’s (Education) Promise Neighborhoods  (Promise) program aims to improve educational and developmental  outcomes for students in some of the country’s most distressed urban,  rural, and tribal neighborhoods by aligning a suite of cradle-to-career  services in a designated geographic footprint. Since 2010, Education has  competitively awarded Promise planning and implementation grants to 48  community-based organizations, including nonprofits, institutions of  higher education, and Indian tribes that work in partnership with several  other organizations, such as schools and social service agencies.  Promise Neighborhoods grants are one of several place-based strategies  under the White House Neighborhood Revitalization Initiative (NRI),  which was also launched in 2010. Place-based strategies focus on a  specific geographic location and provide flexibility to address the area’s  unique needs.
The purpose of the grants is to plan for, create, and evaluate a continuum  of services for children and youth residing in a target neighborhood. The  Promise program provides both planning and implementation grants. In  fiscal years 2010 through 2012, Education awarded a total of 46 1-year  planning grants of up to $500,000. In fiscal years 2011 and 2012,  Education awarded a total of 12 5-year implementation grants of up to $6  million per year. Almost all implementation grantees had received  planning grants. No new grants were awarded in fiscal year 2013, and  none have been awarded in fiscal year 2014, although Education has  continued to fund prior-year implementation grants.
The Promise Neighborhood program was established in fiscal year 2010,  with $10 million from Education’s discretionary Fund for the Improvement  of Education. Since 2011, the program has received an appropriation in  the annual appropriations acts, in amounts ranging from approximately  $30 million to $60 million. For fiscal year 2015, the President has  requested $100 million for the program and an additional $200 million  under the President’s Opportunity, Growth, and Security Initiative.
In light of the President’s requests for increased appropriations and  specific statutory authority for the Promise Neighborhoods program and  the increasing use of place-based strategies, you asked us to examine  the Promise Neighborhoods program. To do this, we examined (1) the  extent to which Education’s strategy for awarding grants aligns with  program goals, if at all; (2) how Education aligns Promise Neighborhoods  efforts with other related programs; (3) how Education evaluates  grantees’ efforts; and (4) the extent to which Promise Neighborhoods  grants have enabled collaboration at the local level, if at all, and the  results of such collaboration.
subsections: section_title: Scope and Methodology
paragraphs: To determine the extent to which the structure of the Promise  Neighborhoods program aligns with program goals and how Education  selected grantees, we reviewed relevant Federal Register notices,  application guidance, and agency information on applicants for fiscal year  2011 and 2012 implementation grants. To determine how Education  aligns Promise grant activities with other federal programs, we reviewed  documentation on Education’s alignment efforts. To assess Education’s  approach to evaluating the program, we reviewed its grant monitoring  reports, performance measures, and guidance for data collection. To  determine the extent to which Promise grants enabled collaboration at the  local level, we used GAO’s prior work on enhancing collaboration in  interagency groups as criteria. We compared the Promise grants’  collaboration approaches to certain successful approaches used by select  interagency groups and reviewed implementation grantees’ application  materials. To learn about grantees’ experiences with the program, we  conducted a web-based survey of all planning and implementation  grantees nationwide from late August to early November 2013. We  received responses from all 48 grantees. We asked grantees to provide  information on the application and peer review process, coordination of  federal resources, collaboration with local organizations, and results of  the planning grants. Because not all respondents answered every  question, the number of grantees responding to any particular question  will be noted throughout the report. In addition, we conducted site visits to  11 planning and implementation grantees. During these visits, we  interviewed five planning grantees and six implementation grantees. Sites  were selected based on several factors, such as the type of grant  awarded, the location of grantees, and whether they were urban or rural.  For all four objectives, we interviewed Education officials, technical  assistance providers, and subject matter specialists from the Promise  Neighborhoods Institute.(See appendix I for more detail on the scope  and methodology.)
We conducted this performance audit from February 2013 to May 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Background
paragraphs: The Promise Neighborhoods program is a place-based program that  attempts to address the problems of children and youth in a designated  geographic footprint. The program is designed to identify and address the  needs of children in low-performing schools in areas of concentrated  poverty by aligning a cradle-to-career continuum of services. The  program moves beyond a focus on low-performing schools by recognizing  the role an entire community plays in a child’s education (see fig. 1).  Place-based initiatives provide communities the flexibility to address their  unique needs and interrelated problems by taking into account the unique  circumstances, challenges, and resources in that particular geographic  area.
The Promise program is one of several place-based initiatives at the  federal level, but it is the only one focused on educational issues. In  addition to Education, the Departments of Justice (Justice), Housing and  Urban Development (HUD), and Health and Human Services (HHS) also  have grant programs aimed at impoverished neighborhoods. Together,  these four agencies and their grant programs form the core of the White  House Neighborhood Revitalization Initiative. This initiative coordinates  neighborhood grant programs at the federal level across agencies, and  identifies and shares best practices. Each agency’s grant program  focuses on its respective agency’s core mission, but together, they focus  on key components of neighborhood revitalization, education, housing,  crime prevention, and healthcare.
subsections: section_title: Promise Neighborhoods Model
paragraphs: Generally, the purpose of the Promise grants is to fund individual  grantees’ efforts to plan for and create a cradle-to-career pipeline of  services based on the specific needs of their communities. The grants are  focused on improving student outcomes on 15 performance indicators,  chosen by Education. Along with the grantee, partner organizations,  funded by federal, state, local, private, or nonprofit organizations, are  expected to collaborate to provide matching funds and services.
A number of nonprofits and foundations have worked on initiatives to  address complex problems in a similarly comprehensive way. Their  approach brings together a group of stakeholders from different sectors to  collaborate on a common agenda, align their efforts, and use common  measures of success. This approach has been described as the collective  impact model. The premise of the model is that better cross-sector  alignment and collaboration creates change more effectively than isolated  interventions by individual organizations. A number of organizations have  used this approach to address issues such as childhood obesity and  water pollution.
Several other cradle-to-career place-based collective impact programs  share key characteristics with the Promise program, including Cincinnati’s  Strive program and the Harlem Children’s Zone.   These collective  impact initiatives use a centralized infrastructure and a structured  process, including training, tools, and resources, intended to result in a  common agenda, shared measurement, and mutually-reinforcing  activities among all participants. This centralized infrastructure requires  staff to manage technology, communications support, data collection,  reporting, and administrative details. The Promise grantees’ role is to  create and provide this centralized infrastructure for their communities.
section_title: Types of Grants
paragraphs: The Promise program relies on a two-phase strategy for awarding grants,  which includes both one-year planning grants and three- to five-year  implementation grants. (See table 1.)
Among other things, planning grantees are required to conduct a  comprehensive needs assessment of children and youth in the  neighborhood and develop a plan to deliver a continuum of solutions with  the potential to achieve results. This effort involves building community  support for and involvement in developing the plan. Planning grantees are  also expected to establish effective partnerships with organizations for  purposes such as providing solutions along the continuum and obtaining  resources to sustain and scale up the activities that work. Finally,  planning grantees are required to plan to build, adapt, or expand a  longitudinal data system to provide information and use data for learning,  continuous improvement, and accountability.
The implementation grant provides funds to develop the administrative  capacity to implement the planned continuum of services. Education  expects implementation grantees to build and strengthen the partnerships  they developed to provide and sustain services and to continue to build  their longitudinal data systems.
Education awarded most of the 2010-2012 grants to non-profit  organizations (38 of 48), eight to institutions of higher education, and two  to tribal organizations. Almost all (10 of 12) implementation grantees  received planning grants, while two did not. (See fig. 2 for locations of  grantees.) (See appendix II for a list of grantees and year of grant award.)
section_title: Education’s Grant Strategy Aligns with Promise Goals, but Education Has Not Clearly Communicated Expectations for Planning Grants
subsections: section_title: Grant Requirements Enabled Education to Identify Organizations with Potential to Coordinate Student Services
paragraphs: The planning and implementation grant activities that Education  developed for the Promise program generally align with Education’s goal  of significantly improving the educational and developmental outcomes of  children and youth in the nation’s most distressed communities.  According to Education officials, the planning grant award process  enabled them to identify community-based organizations in distressed  neighborhoods with the potential to effectively coordinate the continuum  of services for students living in the neighborhood. The eligibility  requirements, which included matching funds or in-kind donations and an  established relationship with the community to be served, helped to  ensure that grantees had financial and organizational capacity and were  representative of the area to be served. Education developed criteria to  evaluate applications and select grantees based on the grantees’ ability  to describe the need for the project; the quality of the project design,  including the ability to leverage existing resources; the quality of the  project services; and the quality of the management plan.
Education’s Promise planning grants were intended to enhance the  capacity of identified organizations to create the cradle-to-career  continuum. The activities required of planning grantees enable grantees  and their partners to gain a depth of knowledge about their communities  and the communities’ needs, which can increase their capacity to focus  on improving educational and developmental outcomes for children and  youth throughout their neighborhood.
Through a separate competition, Education identified organizations that  application reviewers determined were most ready to implement their  plans. While acknowledging that the implementation grantees are best  positioned to determine the allocation of grant funds, Education expects  that grant funds will be used to develop the administrative capacity to  implement the planned continuum and that the majority of resources to  provide services to students and families will come from other public and  private funding sources rather than from the grant itself. This expectation  gives the Promise strategies a chance to extend beyond the 5-year life of  the grant. Further, the requirement that grantees build a longitudinal data  set allows Promise grantees and their partners to review and analyze  robust data in real time to make informed decisions about whether to  adjust their strategies. The data can also help the grantees and Education  learn about the impact of the program.
Education identified 10 desired results from implementation of the  program, which cover the cradle-to-career age span that Promise  Neighborhoods are expected to address. A technical assistance provider  stated that the list of desired results help grantees focus on improving  educational and developmental outcomes across the entire continuum.  (See table 2.) (The indicators that measure progress toward achieving  results are listed in Appendix III.)
section_title: Education’s Two-Phase Grant Strategy Created Challenges for Some Grantees
paragraphs: Education’s grantee selection process was generally clear and  transparent. However, Education did not communicate clearly to planning  grantees about the probability of receiving an implementation grant and  its expectations for grantees to continue their efforts without  implementation funding. This lack of clarity created challenges for some  grantees.
subsections: section_title: Education’s Selection Process
paragraphs: Education outlined its selection criteria and how grant applications would  be scored in its grant announcements and selected peer reviewers from  outside the organization. According to Education officials, the peer  reviewers had expertise in various related fields, including community  development and all levels of education. Education provided additional  training on the application review process. For the planning grant  selection, Education divided about 100 peer reviewers into panels of  three to review packages of about 10 applications. Afterward, peer  reviewers conferred about scores in a conference call.
For the first implementation grant selection, Education had a two-tiered  peer review process. During the first tier, peer reviewers were divided into  panels of three to review approximately seven applications. During the  second tier review of the 16 highest scoring applications, panels of  reviewers were adjusted so that different reviewers read and scored  different applications. For the second implementation grant selection,  there was only one round of reviews. Reviewers were asked to review the  applications and submit comments before meeting on-site to discuss  applications. Education posted the results online, including peer reviewer  comments for grantees and a list of applicants with scores above 80 out  of 100 points.
In our web-based survey of grantees, grantees had mixed views on the  clarity of application requirements and the helpfulness of peer reviewer  comments. Specifically, 13 of 18 planning grantees who applied  unsuccessfully for implementation grants and responded to the relevant  survey question said the application requirements were very clear or  extremely clear, while 8 of 19 grantees that responded said the same  about peer reviewer scores and comments (see fig. 3). The unsuccessful  applicants gave somewhat lower marks to the helpfulness of peer  reviewer comments in improving their future applications and  strengthening their current strategies (see fig. 4).
Some of the 11 planning and implementation grantees that we  interviewed raised concerns about specific application guidelines, such as  how the term “neighborhood” is defined and the length of the application.  Specifically, two rural grantees said that the grant application and  materials had a few areas that seemed to be more geared to urban or  suburban grantees. For example, the term “neighborhood” was somewhat  difficult for them to interpret in a rural context. In fact, two rural grantees  included multiple towns or counties in their neighborhood footprints.  Additionally, two grantees we spoke with had concerns about the  implementation grant application’s 50-page recommended maximum for  the project narrative. Both organizations limited their narratives to 50  pages, but said they later learned that most of the successful grant  recipients had exceeded this limit, often by a large amount.
The timing of the grant cycles created either an overlap or a long gap  between the two grants. Grantees who applied for the implementation  grant in the first cycle after receiving a planning grant had an overlap  between executing the first grant and applying for the second grant.  According to Education officials these grantees were unable to fully apply  the knowledge gained in the planning year to develop their  implementation applications. For example, one grantee said having to  apply for the implementation grant during the planning year made it  difficult to create opportunities for community input into the planning  process. On the other hand, one of the four grantees that received an  implementation grant 2 years after receiving a planning grant faced  challenges sustaining the momentum of its efforts without additional  funding. Another grantee in the same situation was able to sustain  momentum with a separate grant from a private foundation. Education  officials said they became aware of the problems with the timing of the  implementation applications a few months into the first planning grant  year. However, they said they did not have much flexibility in timing the  grant cycles. For example, they said that they needed to allow time for  public comment on the grant notification in the Federal Register. In  addition, they said that agency budget decisions were delayed that year  because the Department was operating under a continuing resolution for  over 6 months in fiscal year 2011—the first year implementation grants  were awarded.
Some grantees also said there was a disconnect between the planning  and the implementation grant application processes. Specifically, two  officials from the six implementation grantees we visited told us that a  high-quality planning year was not nearly as important for obtaining an  implementation grant as having someone who could write a high-quality  federal grant application. For example, one grantee noted that writing a  good implementation grant application was not heavily dependent on  information gleaned from the planning process. Another grantee said that  the implementation grant application was written by a completely different  person who was not involved in planning grant activities.
section_title: Challenges with the Two- Phase Strategy
paragraphs: Some grantees who received only planning grants reported in our survey  and in interviews that they experienced challenges continuing their work  without implementation funds. In addition, two of the five planning  grantees we interviewed had concerns with Education’s strategy of  awarding few implementation grants compared with the number of  planning grants.
Education informed grantees there was a possibility they would not  receive an implementation grant following the planning grant, but no  information was provided about the likelihood of whether this would occur.  We found indications that grantees did not fully appreciate that receiving  a planning grant would not necessarily result in receiving an  implementation grant. Three of the five planning grantees we interviewed  stated that they did not have contingency plans for continuing their  Promise Neighborhood efforts in the event that they did not receive  implementation funding. The lack of contingency planning raises  questions about the grantees’ understanding of the probability of  receiving an implementation grant. Internal control standards state that  management should ensure that effective external communications occur  with groups that can have a serious impact on programs, projects,  operations, and other activities, including budgeting and financing. To  date, Education has awarded 46 planning grants (21, 15, and 10 in 2010,  2011, and 2012, respectively) and 12 implementation grants. Even  though all but two implementation grants were awarded to planning  grantees, fewer than one-quarter of planning grantees received  implementation funding. (See table 3.)
Education officials provided several reasons for separating the planning  and implementation grants and for not awarding implementation grants to  all planning grantees who applied. Officials said that when they awarded  the first planning grants, they were not sure which neighborhoods had  potential grantees with the capacity to implement a Promise plan. In their  view, the planning grants allowed them to invest in the capacity of  communities to take on this work, while the implementation grants were  only awarded to those that demonstrated they were ready for  implementation. Education officials said it was important that grantees  demonstrate they have an implementation plan in place before receiving  such a large sum of money. In addition, after the first round of  implementation grants were awarded, they noted that some applicants did  not receive implementation grants because they were not yet  competitive—in part because they had applied for the implementation  grants before their planning efforts were complete. Finally, in commenting  on a draft of this report, Education officials said that in several years,  Congress appropriated less funds than were requested, which, they said,  affected the number of implementation grants Education awarded.
In 2010, both Education’s Federal Register Notice Inviting Applications for  planning grants and a related frequently asked questions document  informed organizations receiving planning grants that they should not  necessarily plan on automatically receiving implementation grants. The  frequently asked questions guidance noted that the two types of grants  could stand alone. For example, an applicant could receive just a  planning grant, consecutive planning and implementation grants, or—if  the applicant was further along in the planning process—just an  implementation grant. Education officials told us that they viewed the  planning grant activities as useful in themselves. For example, they told  us that the planning process offers rich data and begins the process of  bringing together partners and breaking down silos. They expected that  planning grantees that applied for but did not receive implementation  funding could continue their efforts without implementation grant funding,  using their partners’ pledged matching funds to implement their plans on  a smaller scale. They noted that the requirement to develop memoranda  of understanding with partners should have signaled that the obligations  of the partner organizations were not to be contingent upon receipt of an  implementation grant. However, Education did not require grantees to  have matching funds in-hand before submitting their applications.  Especially in light of the difficult fiscal climate that federal agencies will  likely continue to face in the future, we believe that it is important for  Education to clearly communicate to grantees regarding expectations for  planning and implementation grants. Clear communication and  expectations can also help promote more realistic expectations among  grantees about future funding opportunities given the fiscal realities of the  Promise program over the past 5 years.
Grantees who had not received implementation grants were trying to  continue their efforts and most reported significant challenges in  sustaining momentum. According to our survey, since the end of the  planning grant, most planning grantees who did not receive an  implementation grant (17 out of 29 that answered the related question)  found it very or extremely challenging to maintain funding, 12 out of 29  planning grantees felt that maintaining key leadership positions was very  or extremely challenging, and 13 out of 29 planning grantees found that  hiring staff was very or extremely challenging.
Four of the five planning grantees we interviewed who had not received  implementation grants told us that they need to determine how to  implement scaled-down versions of programs and services identified in  their implementation grant applications. They described challenges  continuing their work without implementation funding. For example, three  grantees noted that partners had pledged funding as a match for federal  dollars in their implementation grant proposal. Without the leverage of  implementation grant funds, it was difficult to maintain the proposed  funding streams. All of the five grantees we interviewed that had  received only planning grants said the planning process was very helpful  in building connections and trust and deepening communication among  partners, and between partners and the community. Four grantees were  concerned, however, that the trust and momentum they had built might  dissipate if they were not able to carry out their plans without an  implementation grant.
section_title: Promise Neighborhoods Program Coordinates with Some Programs with Related Goals but Has No Inventory of Related Programs
subsections: section_title: Promise Program Coordinates Closely with Some Related Federal Programs to Align Grants
paragraphs: In an effort to target its resources and align the Promise program goals  with those of other place-based initiatives, the Promise program  coordinates closely with a limited number of federal programs within  Education and with other federal programs as part of the White House  Neighborhood Revitalization Initiative (NRI).The NRI is an interagency  coordinating body that aligns place-based programs run by HUD, HHS,  Justice, and the Department of the Treasury (Treasury) (see fig. 5).  Coordination through NRI is more structured than internal coordination  within Education, which, according to Promise program officials, occurs  as needed. Liaisons from each grant program meet at biweekly and  monthly NRI meetings. They have formed a program integration  workgroup to coordinate program development, monitoring, and technical  assistance for the grant programs included. For example, they conducted  a joint monitoring trip to a neighborhood in San Antonio, Texas that has  Promise, HUD’s Choice Neighborhood, and Justice’s Byrne Criminal  Justice Innovation grants.
In coordinating within Education and with NRI, Education’s efforts are  focused on ensuring that grants are mutually reinforcing. These  coordination activities include aligning goals, developing common  performance measures where there are common purposes, and sharing  technical assistance resources in areas where programs address similar  issues or fund similar activities. (See table 4.)
The Promise program has also participated in another place-based  program led out of the White House Domestic Policy Council: the Strong  Cities, Strong Communities initiative. This program sends teams of  federal officials to work with distressed cities, providing them expertise to  more efficiently and effectively use the federal funds they already receive.  Education’s Promise program participates in initial on-site assessments of  communities. Education staff assisted two of the participating  communities by providing education expertise at their request.
Promise Zones had to meet a number of requirements, including meeting certain  poverty thresholds and having certain population levels. agencies and five other agencies in partnership with state and local  governments, businesses, and non-profit organizations. Only areas that  already had certain NRI grants or a similar rural or tribal grant were  eligible to apply in the first round. As of January 2014, three Promise  Neighborhoods implementation sites in San Antonio, Los Angeles, and  Southeastern Kentucky were located in designated Promise Zones, which  provide additional opportunity for coordination at the federal and local  level.
The Promise Neighborhoods program does, on occasion, coordinate with  other individual federal agencies and programs outside of the NRI, but  officials stated that the program is focused on deepening and broadening  the communication it has with the five named NRI programs and Promise  Zones. Promise Neighborhoods officials explained that they had concerns  about spreading their coordination efforts too thinly given the large  number of programs grantees may include in their strategies.
section_title: Education Does Not Have an Inventory of Federal Programs with Similar Goals
paragraphs: In addition to Promise grants from Education, individual Promise  Neighborhoods have access to a broad range of federal programs from  other agencies, including many programs that are not part of NRI.  However, Education has not developed an inventory of federal programs  that could contribute to Promise program goals that it could share with  planning and implementation grantees and use to make its own decisions  about coordination across agencies. In recent work examining  approaches used by interagency groups that successfully collaborated,  we found that an inventory of resources related to reaching interagency  goals can be used to promote an understanding of related  governmentwide programs. Such inventories are useful in making  decisions about coordinating related programs across agency lines and  between levels of government, according to officials. We have also found  that creating a comprehensive list of programs is a first step in identifying  potential fragmentation, overlap, or duplication among federal programs  or activities.
As shown in table 5, the 12 implementation grantees we surveyed stated  that they included a variety of federal resources in their Promise  Neighborhoods strategies. AmeriCorps was included in 9 out of 11  implementation grantees’ strategies, followed by Head Start (8 of 12)  and Education’s School Improvement Grants (6 of 11). None of these are  part of NRI. Few grantees said that NRI programs were part of their  Promise strategies. For example, four grantees said that a Choice  Neighborhood grant was part of their Promise strategy, and three  grantees stated that DOJ’s Byrne program was part of their strategy.  Education officials attributed the small number of grantees that use  HUD’s Choice program to the fact that few grantees have distressed  public housing within their footprint that is eligible for this funding.
Although Promise grantees conduct their own inventories of the existing  federal and other resources in their neighborhoods in order to develop  their strategies, two grantees we spoke with were unaware of some of the  other federal programs that could contribute towards their strategies. For  example, one implementation grantee we spoke to with concerns about  school safety was unaware of DOJ’s Byrne Criminal Justice Innovation  grant program. Another planning grantee who completed our survey  commented that a list of related federal programs like the one in our  survey would be especially useful to grantees who did not receive  implementation grants. Education officials with the Promise program told  us that sometimes grantees are unaware that the community is benefiting  from certain federal programs because programs are renamed as they  filter down through the state or local levels. Education officials said they  emphasize to grantees the importance of reaching out to key partners to  ensure they are aware of other federally funded programs in the  neighborhood because their partners may be more knowledgeable about  other sources of federal funding. While encouraging grantees to reach out  to key partners is helpful, Education, through its coordination with other  federal agencies, would likely have more knowledge about existing  federal resources.
Without a federal level inventory, Education is not well-positioned to  support grantee efforts to identify other federal programs that could  contribute to Promise program goals. Further, Education lacks complete  information to inform decisions about future federal coordination efforts  and identify potential fragmentation, overlap, and duplication.
section_title: Education Collects Extensive Data but Has Not Determined Whether or How to Evaluate the Program
paragraphs: While Education is collecting a large amount of data from Promise  grantees that was intended, in part, to be used to evaluate the program,  the Education offices responsible for program evaluation— the Institute  for Educational Sciences (IES) and Office of Planning, Evaluation, and  Policy Development (OPEPD)—have not yet determined whether or how  they will evaluate the program.
One of Education’s primary goals for the Promise program, as described  in the Federal Register, is to learn about the overall impact of the program  through a rigorous program evaluation. Applicants are required to  describe their commitment to work with a national evaluator for Promise  Neighborhoods to ensure that data collection and program design are  consistent with plans to conduct a rigorous national evaluation of the  program and the specific solutions and strategies pursued by individual  grantees. We have found that federal program evaluation studies provide  external accountability for the use of public resources. Evaluation can  help to determine the “value added” of the expenditure of federal  resources or to learn how to improve performance—or both. Evaluation  can play a key role in strategic planning and in program management,  informing both program design and execution.
Education requires implementation grantees to report annually on their  performance using 15 indicators. The indicators include graduation rates,  attendance, academic proficiency, student mobility, physical activity, and  perceptions of safety. (See table 11 in appendix III.) Education contracted  with the Urban Institute to provide guidance on how to collect data on the  indicators, including data sources and survey techniques. According to  Urban Institute officials, they used existing, validated measures whenever  possible to ensure comparability across programs. Seven of twelve  implementation grantees we surveyed said the guidance documents were  extremely or very helpful, while four found it moderately helpful and one  somewhat helpful.
The Urban Institute has analyzed the data on the indicators for the first  implementation year (the baseline), but Education has not decided  whether it will make the first year’s data public because it was not  collected in a consistent manner and not all grantees were able to collect  all of the necessary data. According to Promise program officials there  were inconsistencies in data collection because guidance was not  available until February 2013, 13 months after 2011 implementation  grants were awarded and over 1 month after 2012 implementation grants  were awarded.
Promise officials stated that they will use the performance data to target  their technical assistance. They are still working with grantees to develop  meaningful targets for the second implementation year. Urban Institute  officials noted that these 15 indicators help grantees focus their efforts on  the outcomes they are trying to achieve.
In addition, Promise grantees are required to develop a longitudinal data  system to collect information on the individuals served, services provided  in the cradle-to-career continuum, and the related outcomes.are expected to use the longitudinal data to evaluate their programs on an   Grantees  ongoing basis and make adjustments to their strategies and services, as  discussed later in this report.
Grantees are also required to provide the longitudinal data to Education,  which Education officials said they may use to create a restricted-use  data set. However, Education currently does not have a plan for analyzing  the data. In commenting on a draft of this report, Education stated it must  first conduct a systematic examination of the reliability and validity of the  data to determine whether it can be used for a descriptive study and a  restricted-use data set. Education further stated that the restricted-use  data set would only be made available to external researchers after  Education determines that the data quality is adequate and appropriate  for research; analyzes the data, taking into account privacy concerns; and  determines whether to release its own report. In addition, officials from  IES and OPEPD cited limitations and challenges to using the longitudinal  data for program evaluation.
An official from IES, the entity responsible for all impact evaluations  conducted by Education, told us that it is not feasible to conduct an  impact evaluation of individual program pieces or an overall  evaluation of the Promise approach. The official offered three options  for evaluation. IES’ preferred option is to conduct a rigorous impact  evaluation with a control group obtained through randomized  assignment to the program. However, Promise Neighborhoods are not  designed to create such a control group. Another option would be for  IES to use students or families who were not chosen to participate in  an oversubscribed program as a control group, but an informal poll  that IES took at a Promise Neighborhoods conference suggested that  there were not a sufficient number of oversubscribed programs. A  third option was to develop a comparison group of neighborhoods that  did not receive a Promise Neighborhood grant. However, IES officials  question whether such an approach would enable them to match  neighborhoods that were comparable to Promise neighborhoods at  the beginning of the grant period. Finally, IES noted that collecting  additional data for a control group could be expensive.
Education’s OPEPD is responsible for conducting other types of  program evaluations. According to Education officials, it could conduct  a more limited evaluation focused on outcomes without demonstrating  that they are a direct result of the Promise program, but they have no  specific plans to do so. An OPEPD official stated OPEPD is reluctant  to commit to a plan because they have not yet seen the data and do  not know how reliable or complete it will be. In addition, the official  said that OPEPD is unsure about funding and that any comprehensive  evaluations are expensive to carry out. By creating a restricted-use  data set, OPEPD hopes that other researchers may have the funding  to use the data to reach some conclusions about the program. The  OPEPD official further explained that no one has ever evaluated a  community-based approach like this one and that they hope  researchers may have some ideas about how to do so.
Researchers at the Urban Institute and within the Promise grantee  community have proposed other options for evaluating the program. A  researcher at the Urban Institute noted that random assignment is not  the right approach for evaluating place-based programs. Instead, the  researcher recommends a variety of other options for evaluating such  programs, including approaches that estimate a single site’s effect on  outcomes and aggregating those outcomes. This differs from the  traditional program evaluation approach, which IES has considered, of  isolating the effects of an intervention so that its effects can be measured  separately from other interventions.
While Education recognizes the importance of evaluating the Promise  program, they lack a plan to do so. If an evaluation is not conducted,  Education will have limited information about the Promise program’s  success or the viability of the program’s collaborative approach.
section_title: Grantees Collaborate Using Leading Practices, Resulting in Benefits and Challenges
subsections: section_title: Promise Grant Framework Encourages Collaboration
paragraphs: The Promise program generally requires grantees to use collaborative  approaches. We found that grantees are following approaches consistent  with those we have recognized as enhancing and sustaining collaboration  with partners. The approaches we have previously identified include:
Establishing common outcomes: Establishing common outcomes  helps collaborating agencies develop a clear and compelling rationale  to work together.
Addressing needs by leveraging resources: Leveraging the various  human, information technology, physical and financial resources  available from agencies in a collaborative group allows the group to  obtain benefits that would not be available if they worked separately.
Tracking performance and maintaining accountability: Tracking  performance and other mechanisms for maintaining accountability are  consistent with our prior work, which has shown that performance  information can be used to improve results by setting priorities and  allocating resources to take corrective actions to solve program  problems.
The approaches are discussed below and in Tables 6 through 8.
section_title: Grantees Collaborate to Deliver a Range of Services
paragraphs: Grantees and partners provided examples of how they have collaborated  through the Promise grant to deliver services and supports that are  intended to improve educational and developmental outcomes. Grantees  and their partners focused on delivering services at various steps along  the cradle-to-career pipeline, including:
Early learning supports: programs or services designed to improve  outcomes and ensure that young children enter kindergarten and  progress through early elementary school grades demonstrating age- appropriate functioning.
K-12 supports: programs, including policies and personnel, linked to  improving educational outcomes for children in pre-school through  12th grade. These include developing effective teachers and  principals, facilitating the use of data on student achievement and  student growth to inform decision-making, supporting a well-rounded  curriculum, and creating multiple pathways for students to earn high  school diplomas.
College and career supports: programs preparing students for college  and career success. These include partnering with higher education  institutions to help instill a college-going culture in the neighborhood,  providing dual-enrollment opportunities for students to gain college  credit while in high school, and providing access to career and  technical education programs.
Family and community supports: these include child and youth  physical, mental, behavioral and emotional health programs, safety  programs such as those to prevent or reduce gang activity and  programs that expand access to quality affordable housing.
For examples of the services delivered and outcomes reported by  grantees for each part of the cradle-to-career pipeline, see table 9 below.
section_title: Conclusions
paragraphs: The Promise program has energized the 48 planning and implementation  grantees and their partners to tackle the complex challenges facing  impoverished neighborhoods together. While grantees said they will  continue their efforts to build their Promise Neighborhoods, planning  grantees faced challenges in sustaining their work over the long term  without implementation grants. Planning grantees, especially those  concerned about building trust with their communities and partners, may  have been better served if Education had provided a more transparent,  realistic picture of the fiscal reality of the Promise program and its  potential impact on implementation grant funding. Lack of clear  communication about the expectations Education had for planning  grantees who did not receive implementation funding made it difficult for  these grantees to develop specific plans to continue their efforts without  future Promise funds. However, the reported small, yet tangible benefits  that some communities pursued during the planning year—such as a safe  place for children to play—increased momentum and built trust with  community members. Encouraging such “early wins” could help all  grantees and their partners build upon and improve their efforts,  especially since implementation funding has proven scarce.
Additionally, much of the information grantees use about what existing  federal, state, and local programs and resources to incorporate into their  strategies is gleaned through their needs assessment at the local level.  Education has not provided grantees with comprehensive information  about other federal resources that may be available to use in their  Promise strategies. Education is best positioned to develop and share  such an inventory of federal programs that relate to the goals of the  Promise program. Without such an inventory, Education may be missing  opportunities to better support grantees, find other federal programs for  future coordination efforts, and identify potential fragmentation, overlap,  and duplication at the federal level.
One of the Promise program’s primary goals is to identify the overall  impact of its approach and the relationship between particular strategies  and student outcomes. Grantees are investing significant time and  resources to collect data to assess the program, but Education lacks a  clear plan for using it. Without evaluating program, it will be difficult for  Education to determine whether it is successfully addressing the complex  problem of poor student outcomes in impoverished neighborhoods.  Finally, the Promise program is one of several place-based and collective  impact programs being implemented across many federal agencies.  Given the number of these initiatives, not evaluating the program limits  Education and other agencies from learning about the extent to which  model is effective and should be replicated.
section_title: Recommendations for Executive Action
paragraphs: In order to improve grantees’ planning and implementation efforts,  increase the effectiveness of grantee efforts to integrate and manage  resources, and learn more about the program’s impact, we recommend  that the Secretary of Education take the following three actions:  1.  Clarify program guidance about planning and implementation grants  to provide reasonable assurance that planning grantees are better  prepared to continue their efforts in the absence of implementation  funding. Additional guidance could include encouraging grantees to  set aside a small amount of the grant to identify and deliver early,  tangible benefits to their neighborhoods.  2.  Develop and disseminate to grantees on an ongoing basis an  inventory of federal programs and resources that can contribute to the  Promise Neighborhoods program’s goal to better support coordination  across agency lines.  3.  Develop a plan to use the data collected from grantees to conduct a  national evaluation of the program.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Education for  review and comment. Education’s comments are reproduced in appendix  IV and are summarized below. Education also provided technical  comments, which we incorporated into the final report as appropriate.  Education outlined the steps it would take to implement our three  recommendations, and provided its perspective on communicating  expectations to grantees regarding future funding. Education did not  explicitly agree or disagree with our findings.
Regarding our finding that Education did not communicate clearly to  planning grantees about its expectations for the grants, Education stated  that in any given year it does not know and therefore cannot  communicate the amount of funding available or the number of grant  awards anticipated in the following year. We agree, and have clarified our  finding in the report accordingly.
Education stated that an early assessment of planning grantees’  likelihood of receiving implementation funding would have been  premature. Education noted that although Congress has funded the  Promise program for the past 5 years, in 4 of those 5 years it  appropriated far less than the President requested, and for the last 3  years the program has essentially been level funded. Education further  stated that this underscores the limited control that the program had over  the number of implementation grants made. We recognize that federal  agencies have faced a difficult fiscal climate over the past few years,  particularly for discretionary programs. For that reason—and especially  given the level at which the Promise program has been funded for the  past 3 years—we believe it is even more important that Education be  clear and transparent with planning grantees about historical fiscal  realities of the Promise program and the implications this may have on  future implementation grants. We also believe this situation highlights the  need for planning grantees to have contingency plans, especially given  Education’s expectations that grantees continue their efforts even in the  absence of implementation funding. We further believe that this also  underscores the importance of “early wins” to demonstrate what can be  achieved when grantees and their partners work collaboratively, as such  demonstrations can encourage them to continue their efforts even without  implementation funding.
In discussing its perspective on communicating expectations to grantees  regarding future funding, Education stated that its Notifications Inviting  Applications indicated that future funding was contingent on the  availability of funds and that the program’s frequently asked questions  document noted that implementation funding was not guaranteed and that  planning grantees would have to compete for implementation grants. We  believe that our report adequately reflects these communication efforts.  However, as we reported, Education did not communicate to planning  grantees that it expected them to continue their efforts even in the  absence of implementation funding. Nor did Education communicate to  implementation grant applicants that it expected them to be able to use  their partners’ pledged matching funds even if they did not receive  implementation grants. This lack of communication was evidenced by  planning grantees’ lack of contingency plans and challenges they faced  accessing the pledged matching funds, according to the grantees we  interviewed.
In response to our first recommendation, Education stated that it would  continue to communicate to planning grant applicants that implementation  funding is contingent on the availability of funds, and that it would provide  more targeted technical assistance to planning grant recipients regarding  strategies for continuing grantees’ efforts absent implementation funding.  Education also stated that it would clarify to grantees that planning grant  funds could be used to achieve early, tangible benefits.
Regarding our second recommendation, Education stated that it would  work with its technical assistance providers to create a mechanism to  distribute a comprehensive list of external funding opportunities,  programs and resources on a regular basis to better support the grantees’  implementation efforts.
With regard to our final recommendation, Education stated that it will  consider options for how and whether it can use the data collected from  grantees to conduct a national evaluation. Education stated that as a first  step it will conduct a systematic evaluation of the reliability and validity of  the data, given issues that we and Education noted about inconsistencies  in data collection and privacy concerns. In addition, Education stated that  to date, it has not received sufficient funding to support a national  evaluation. We agree that conducting evaluations can be costly.  However, given that one of Education’s primary goals is to learn about the  overall impact of the program through a rigorous program evaluation, we  continue to believe that absent an evaluation, it will be difficult for  Education to determine whether it is successfully addressing the complex  problem of poor student outcomes in impoverished neighborhoods—one  of its stated goals. Further, developing an evaluation plan would provide  critical information about the resources required to conduct an evaluation,  and could better inform future funding requests for such an evaluation.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretary of Education and other interested congressional committees. In  addition, the report will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at 617-788-0580 or NowickiJ@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To better understand grantees’ experiences with the Promise  Neighborhoods program, we conducted a web-based survey of all 48  planning and implementation grantees. The survey was conducted from  August 23, 2013 through November 7, 2013. We received completed  surveys from all 48 grantees for a 100 percent response rate. The survey  included questions about the clarity and helpfulness of the application and  peer review process, challenges sustaining efforts after the end of the  planning grant, coordination of federal resources, collaboration with local  organizations and associated challenges, the extent to which local  coordination reduced duplication, overlap and fragmentation, if at all, the  mechanisms organizations use to track the results of their efforts, the  results of the grants, and the helpfulness of Education’s guidance and  resources for the program.
Because this was not a sample survey, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  nonsampling errors, such as variations in how respondents interpret  questions and their willingness to offer accurate responses. We took  steps to minimize nonsampling errors, including pretesting draft  instruments and using a web-based administration system. Specifically,  during survey development, we pretested draft instruments with five  grantees that received planning and/or implementation grants. In the  pretests, we were generally interested in the clarity, precision, and  objectivity of the questions, as well as the flow and layout of the survey.  For example, we wanted to ensure definitions used in the surveys were  clear and known to the respondents, categories provided in closed-ended  questions were complete and exclusive, and the ordering of survey  sections and the questions within each section was appropriate. We  revised the final survey based on pretest results. We took another step to  minimize nonsampling errors by using a web-based survey. This allowed  respondents to enter their responses directly into an electronic instrument  and created a record for each respondent in a data file—eliminating the  need for manual data entry and its associated errors. To further minimize  errors, programs used to analyze the survey data were independently  verified to ensure the accuracy of this work. Because not all respondents  answered every question, we reported the number of grantees  responding to particular questions throughout the report.
In addition, we conducted site visits to 11 Promise grantees. We selected  sites based on several factors, such as the type of grant awarded, the  location of the grantees, and whether the Promise Neighborhood was  urban or rural. The site visits provided opportunities to collect more in- depth information on the program and highlighted different types of  grantees and approaches. We visited six implementation grantees in  Boston, Massachusetts; Berea, Kentucky; Chula Vista, California;  Indianola, Mississippi; Los Angeles, California; and Washington, DC. We  visited five planning grantees in Campo, California; Lawrence,  Massachusetts; Los Angeles, California; Nashville, Tennessee; and  Worcester, Massachusetts. These include one tribal and two rural  grantees. We also interviewed Education officials and technical  assistance providers, as well as other experts who have worked with  Promise grant applicants, such as the Promise Neighborhoods Institute.
To determine how well the structure of Education’s Promise  Neighborhoods grant program aligns with program goals and how  Education selected grantees, using Education’s goals for the Promise  program as criteria, we reviewed Education reports on place-based  strategies; relevant Federal Register notices; and application guidance  and training materials, including both the guidance available to applicants  and to the peer reviewers regarding the technical evaluation/grant  selection process. We reviewed agency information on applicants for  implementation grants in the fiscal year 2011 and 2012 cycles, as fiscal  years 2011 and 2012 were the only years in which Education awarded  implementation grants. For both cycles, we analyzed application materials  and technical evaluation documentation for a subset of implementation  grant applicants—those that received planning grants in prior years. We  compared the scores in each component of the application for both  successful and unsuccessful applicants to identify criteria or factors that  accounted for significant variation in total scores. We conducted a limited  review of selected peer reviewer comments to gain more insight into the  reasons for any differences. We interviewed Education officials about the  process that the department used for the selection of both planning and  implementation grantees.
To determine how the Promise Neighborhoods program coordinated with  other Education programs and with the other federal agencies, including  those involved in the White House Neighborhood Revitalization Initiative  (NRI), we reviewed documentation of the NRI’s efforts and interviewed  agency officials participating in the NRI. We also interviewed cognizant  officials at other agencies participating in the NRI. To assess Education’s  approach to evaluating the success of the grants, we reviewed grant  monitoring reports, Education’s performance measures, and related  guidance for data collection for this program and interviewed agency  officials responsible for evaluation, including technical assistance  providers.
To determine the extent to which Promise grants enabled collaboration at  the local level, we used GAO’s prior work on implementing interagency  collaborative mechanisms as criteria. We compared the Promise grants’  collaboration mechanisms to certain successful approaches used by  select interagency groups and reviewed implementation grantees’  application materials. Our 11 site visits provided additional insight into  how selected grantees align services supported by multiple funding  streams and delivered by multiple providers. Using survey responses  from all planning grantees, we determined whether they have continued  their efforts, whether they have implemented any of their strategies, and  what, if any, interim results they have identified, regardless of whether  they received implementation grants. Site visits provided illustrative  examples of interim benefits and challenges.
We conducted this performance audit from February 2013 to May 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Promise Neighborhood Grantees
paragraphs: Location  New York , NY (Harlem)
Athens Clarke County Family Connection Inc.
Clay, Jackson, and Owsley Counties,  KY  Boys & Girls Club of the Northern Cheyenne Nation  Northern Cheyenne Reservation, MT  Community Day Care Center of Lawrence, Inc.
New York , NY (Brooklyn)
United Way of Central Massachusetts, Inc.
New York, NY (Brooklyn)
New York, NY (Queens)
New York, NY (Brooklyn)
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Sirois, Assistant  Director; Jacques Arsenault; Aimee Elivert; and Lara Laufer made key  contributions to this report. Also contributing to this report were James  Bennett, Deborah Bland, Mallory Barg Bulman, Holly Dye, Alex Galuten,  Jean McSween, Matthew Saradjian, and Sarah Veale.