Education Grants: Promise Neighborhoods Promotes Collaboration but Needs National Evaluation Plan
Released: 2014-06-04T00:00:00

Letter

Scope and Methodology

To determine the extent to which the structure of the Promise Neighborhoods program aligns with program goals and how Education selected grantees, we reviewed relevant Federal Register notices, application guidance, and agency information on applicants for fiscal year 2011 and 2012 implementation grants. To determine how Education aligns Promise grant activities with other federal programs, we reviewed documentation on Education’s alignment efforts. To assess Education’s approach to evaluating the program, we reviewed its grant monitoring reports, performance measures, and guidance for data collection. To determine the extent to which Promise grants enabled collaboration at the local level, we used GAO’s prior work on enhancing collaboration in interagency groups as criteria. We compared the Promise grants’ collaboration approaches to certain successful approaches used by select interagency groups and reviewed implementation grantees’ application materials. To learn about grantees’ experiences with the program, we conducted a web-based survey of all planning and implementation grantees nationwide from late August to early November 2013. We received responses from all 48 grantees. We asked grantees to provide information on the application and peer review process, coordination of federal resources, collaboration with local organizations, and results of the planning grants. Because not all respondents answered every question, the number of grantees responding to any particular question will be noted throughout the report. In addition, we conducted site visits to 11 planning and implementation grantees. During these visits, we interviewed five planning grantees and six implementation grantees. Sites were selected based on several factors, such as the type of grant awarded, the location of grantees, and whether they were urban or rural. For all four objectives, we interviewed Education officials, technical assistance providers, and subject matter specialists from the Promise Neighborhoods Institute.(See appendix I for more detail on the scope and methodology.)

We conducted this performance audit from February 2013 to May 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Background

The Promise Neighborhoods program is a place-based program that attempts to address the problems of children and youth in a designated geographic footprint. The program is designed to identify and address the needs of children in low-performing schools in areas of concentrated poverty by aligning a cradle-to-career continuum of services. The program moves beyond a focus on low-performing schools by recognizing the role an entire community plays in a child’s education (see fig. 1). Place-based initiatives provide communities the flexibility to address their unique needs and interrelated problems by taking into account the unique circumstances, challenges, and resources in that particular geographic area.

The Promise program is one of several place-based initiatives at the federal level, but it is the only one focused on educational issues. In addition to Education, the Departments of Justice (Justice), Housing and Urban Development (HUD), and Health and Human Services (HHS) also have grant programs aimed at impoverished neighborhoods. Together, these four agencies and their grant programs form the core of the White House Neighborhood Revitalization Initiative. This initiative coordinates neighborhood grant programs at the federal level across agencies, and identifies and shares best practices. Each agency’s grant program focuses on its respective agency’s core mission, but together, they focus on key components of neighborhood revitalization, education, housing, crime prevention, and healthcare.

Promise Neighborhoods Model

Generally, the purpose of the Promise grants is to fund individual grantees’ efforts to plan for and create a cradle-to-career pipeline of services based on the specific needs of their communities. The grants are focused on improving student outcomes on 15 performance indicators, chosen by Education. Along with the grantee, partner organizations, funded by federal, state, local, private, or nonprofit organizations, are expected to collaborate to provide matching funds and services.

A number of nonprofits and foundations have worked on initiatives to address complex problems in a similarly comprehensive way. Their approach brings together a group of stakeholders from different sectors to collaborate on a common agenda, align their efforts, and use common measures of success. This approach has been described as the collective impact model. The premise of the model is that better cross-sector alignment and collaboration creates change more effectively than isolated interventions by individual organizations. A number of organizations have used this approach to address issues such as childhood obesity and water pollution.

Several other cradle-to-career place-based collective impact programs share key characteristics with the Promise program, including Cincinnati’s Strive program and the Harlem Children’s Zone. These collective impact initiatives use a centralized infrastructure and a structured process, including training, tools, and resources, intended to result in a common agenda, shared measurement, and mutually-reinforcing activities among all participants. This centralized infrastructure requires staff to manage technology, communications support, data collection, reporting, and administrative details. The Promise grantees’ role is to create and provide this centralized infrastructure for their communities.

Types of Grants

The Promise program relies on a two-phase strategy for awarding grants, which includes both one-year planning grants and three- to five-year implementation grants. (See table 1.)

Among other things, planning grantees are required to conduct a comprehensive needs assessment of children and youth in the neighborhood and develop a plan to deliver a continuum of solutions with the potential to achieve results. This effort involves building community support for and involvement in developing the plan. Planning grantees are also expected to establish effective partnerships with organizations for purposes such as providing solutions along the continuum and obtaining resources to sustain and scale up the activities that work. Finally, planning grantees are required to plan to build, adapt, or expand a longitudinal data system to provide information and use data for learning, continuous improvement, and accountability.

The implementation grant provides funds to develop the administrative capacity to implement the planned continuum of services. Education expects implementation grantees to build and strengthen the partnerships they developed to provide and sustain services and to continue to build their longitudinal data systems.

Education awarded most of the 2010-2012 grants to non-profit organizations (38 of 48), eight to institutions of higher education, and two to tribal organizations. Almost all (10 of 12) implementation grantees received planning grants, while two did not. (See fig. 2 for locations of grantees.) (See appendix II for a list of grantees and year of grant award.)

Education’s Grant Strategy Aligns with Promise Goals, but Education Has Not Clearly Communicated Expectations for Planning Grants

Grant Requirements Enabled Education to Identify Organizations with Potential to Coordinate Student Services

The planning and implementation grant activities that Education developed for the Promise program generally align with Education’s goal of significantly improving the educational and developmental outcomes of children and youth in the nation’s most distressed communities. According to Education officials, the planning grant award process enabled them to identify community-based organizations in distressed neighborhoods with the potential to effectively coordinate the continuum of services for students living in the neighborhood. The eligibility requirements, which included matching funds or in-kind donations and an established relationship with the community to be served, helped to ensure that grantees had financial and organizational capacity and were representative of the area to be served. Education developed criteria to evaluate applications and select grantees based on the grantees’ ability to describe the need for the project; the quality of the project design, including the ability to leverage existing resources; the quality of the project services; and the quality of the management plan.

Education’s Promise planning grants were intended to enhance the capacity of identified organizations to create the cradle-to-career continuum. The activities required of planning grantees enable grantees and their partners to gain a depth of knowledge about their communities and the communities’ needs, which can increase their capacity to focus on improving educational and developmental outcomes for children and youth throughout their neighborhood.

Through a separate competition, Education identified organizations that application reviewers determined were most ready to implement their plans. While acknowledging that the implementation grantees are best positioned to determine the allocation of grant funds, Education expects that grant funds will be used to develop the administrative capacity to implement the planned continuum and that the majority of resources to provide services to students and families will come from other public and private funding sources rather than from the grant itself. This expectation gives the Promise strategies a chance to extend beyond the 5-year life of the grant. Further, the requirement that grantees build a longitudinal data set allows Promise grantees and their partners to review and analyze robust data in real time to make informed decisions about whether to adjust their strategies. The data can also help the grantees and Education learn about the impact of the program.

Education identified 10 desired results from implementation of the program, which cover the cradle-to-career age span that Promise Neighborhoods are expected to address. A technical assistance provider stated that the list of desired results help grantees focus on improving educational and developmental outcomes across the entire continuum. (See table 2.) (The indicators that measure progress toward achieving results are listed in Appendix III.)

Education’s Two-Phase Grant Strategy Created Challenges for Some Grantees

Education’s grantee selection process was generally clear and transparent. However, Education did not communicate clearly to planning grantees about the probability of receiving an implementation grant and its expectations for grantees to continue their efforts without implementation funding. This lack of clarity created challenges for some grantees.

Education’s Selection Process

Education outlined its selection criteria and how grant applications would be scored in its grant announcements and selected peer reviewers from outside the organization. According to Education officials, the peer reviewers had expertise in various related fields, including community development and all levels of education. Education provided additional training on the application review process. For the planning grant selection, Education divided about 100 peer reviewers into panels of three to review packages of about 10 applications. Afterward, peer reviewers conferred about scores in a conference call.

For the first implementation grant selection, Education had a two-tiered peer review process. During the first tier, peer reviewers were divided into panels of three to review approximately seven applications. During the second tier review of the 16 highest scoring applications, panels of reviewers were adjusted so that different reviewers read and scored different applications. For the second implementation grant selection, there was only one round of reviews. Reviewers were asked to review the applications and submit comments before meeting on-site to discuss applications. Education posted the results online, including peer reviewer comments for grantees and a list of applicants with scores above 80 out of 100 points.

In our web-based survey of grantees, grantees had mixed views on the clarity of application requirements and the helpfulness of peer reviewer comments. Specifically, 13 of 18 planning grantees who applied unsuccessfully for implementation grants and responded to the relevant survey question said the application requirements were very clear or extremely clear, while 8 of 19 grantees that responded said the same about peer reviewer scores and comments (see fig. 3). The unsuccessful applicants gave somewhat lower marks to the helpfulness of peer reviewer comments in improving their future applications and strengthening their current strategies (see fig. 4).

Some of the 11 planning and implementation grantees that we interviewed raised concerns about specific application guidelines, such as how the term “neighborhood” is defined and the length of the application. Specifically, two rural grantees said that the grant application and materials had a few areas that seemed to be more geared to urban or suburban grantees. For example, the term “neighborhood” was somewhat difficult for them to interpret in a rural context. In fact, two rural grantees included multiple towns or counties in their neighborhood footprints. Additionally, two grantees we spoke with had concerns about the implementation grant application’s 50-page recommended maximum for the project narrative. Both organizations limited their narratives to 50 pages, but said they later learned that most of the successful grant recipients had exceeded this limit, often by a large amount.

The timing of the grant cycles created either an overlap or a long gap between the two grants. Grantees who applied for the implementation grant in the first cycle after receiving a planning grant had an overlap between executing the first grant and applying for the second grant. According to Education officials these grantees were unable to fully apply the knowledge gained in the planning year to develop their implementation applications. For example, one grantee said having to apply for the implementation grant during the planning year made it difficult to create opportunities for community input into the planning process. On the other hand, one of the four grantees that received an implementation grant 2 years after receiving a planning grant faced challenges sustaining the momentum of its efforts without additional funding. Another grantee in the same situation was able to sustain momentum with a separate grant from a private foundation. Education officials said they became aware of the problems with the timing of the implementation applications a few months into the first planning grant year. However, they said they did not have much flexibility in timing the grant cycles. For example, they said that they needed to allow time for public comment on the grant notification in the Federal Register. In addition, they said that agency budget decisions were delayed that year because the Department was operating under a continuing resolution for over 6 months in fiscal year 2011—the first year implementation grants were awarded.

Some grantees also said there was a disconnect between the planning and the implementation grant application processes. Specifically, two officials from the six implementation grantees we visited told us that a high-quality planning year was not nearly as important for obtaining an implementation grant as having someone who could write a high-quality federal grant application. For example, one grantee noted that writing a good implementation grant application was not heavily dependent on information gleaned from the planning process. Another grantee said that the implementation grant application was written by a completely different person who was not involved in planning grant activities.

Challenges with the Two- Phase Strategy

Some grantees who received only planning grants reported in our survey and in interviews that they experienced challenges continuing their work without implementation funds. In addition, two of the five planning grantees we interviewed had concerns with Education’s strategy of awarding few implementation grants compared with the number of planning grants.

Education informed grantees there was a possibility they would not receive an implementation grant following the planning grant, but no information was provided about the likelihood of whether this would occur. We found indications that grantees did not fully appreciate that receiving a planning grant would not necessarily result in receiving an implementation grant. Three of the five planning grantees we interviewed stated that they did not have contingency plans for continuing their Promise Neighborhood efforts in the event that they did not receive implementation funding. The lack of contingency planning raises questions about the grantees’ understanding of the probability of receiving an implementation grant. Internal control standards state that management should ensure that effective external communications occur with groups that can have a serious impact on programs, projects, operations, and other activities, including budgeting and financing. To date, Education has awarded 46 planning grants (21, 15, and 10 in 2010, 2011, and 2012, respectively) and 12 implementation grants. Even though all but two implementation grants were awarded to planning grantees, fewer than one-quarter of planning grantees received implementation funding. (See table 3.)

Education officials provided several reasons for separating the planning and implementation grants and for not awarding implementation grants to all planning grantees who applied. Officials said that when they awarded the first planning grants, they were not sure which neighborhoods had potential grantees with the capacity to implement a Promise plan. In their view, the planning grants allowed them to invest in the capacity of communities to take on this work, while the implementation grants were only awarded to those that demonstrated they were ready for implementation. Education officials said it was important that grantees demonstrate they have an implementation plan in place before receiving such a large sum of money. In addition, after the first round of implementation grants were awarded, they noted that some applicants did not receive implementation grants because they were not yet competitive—in part because they had applied for the implementation grants before their planning efforts were complete. Finally, in commenting on a draft of this report, Education officials said that in several years, Congress appropriated less funds than were requested, which, they said, affected the number of implementation grants Education awarded.

In 2010, both Education’s Federal Register Notice Inviting Applications for planning grants and a related frequently asked questions document informed organizations receiving planning grants that they should not necessarily plan on automatically receiving implementation grants. The frequently asked questions guidance noted that the two types of grants could stand alone. For example, an applicant could receive just a planning grant, consecutive planning and implementation grants, or—if the applicant was further along in the planning process—just an implementation grant. Education officials told us that they viewed the planning grant activities as useful in themselves. For example, they told us that the planning process offers rich data and begins the process of bringing together partners and breaking down silos. They expected that planning grantees that applied for but did not receive implementation funding could continue their efforts without implementation grant funding, using their partners’ pledged matching funds to implement their plans on a smaller scale. They noted that the requirement to develop memoranda of understanding with partners should have signaled that the obligations of the partner organizations were not to be contingent upon receipt of an implementation grant. However, Education did not require grantees to have matching funds in-hand before submitting their applications. Especially in light of the difficult fiscal climate that federal agencies will likely continue to face in the future, we believe that it is important for Education to clearly communicate to grantees regarding expectations for planning and implementation grants. Clear communication and expectations can also help promote more realistic expectations among grantees about future funding opportunities given the fiscal realities of the Promise program over the past 5 years.

Grantees who had not received implementation grants were trying to continue their efforts and most reported significant challenges in sustaining momentum. According to our survey, since the end of the planning grant, most planning grantees who did not receive an implementation grant (17 out of 29 that answered the related question) found it very or extremely challenging to maintain funding, 12 out of 29 planning grantees felt that maintaining key leadership positions was very or extremely challenging, and 13 out of 29 planning grantees found that hiring staff was very or extremely challenging.

Four of the five planning grantees we interviewed who had not received implementation grants told us that they need to determine how to implement scaled-down versions of programs and services identified in their implementation grant applications. They described challenges continuing their work without implementation funding. For example, three grantees noted that partners had pledged funding as a match for federal dollars in their implementation grant proposal. Without the leverage of implementation grant funds, it was difficult to maintain the proposed funding streams. All of the five grantees we interviewed that had received only planning grants said the planning process was very helpful in building connections and trust and deepening communication among partners, and between partners and the community. Four grantees were concerned, however, that the trust and momentum they had built might dissipate if they were not able to carry out their plans without an implementation grant.

Promise Neighborhoods Program Coordinates with Some Programs with Related Goals but Has No Inventory of Related Programs

Promise Program Coordinates Closely with Some Related Federal Programs to Align Grants

In an effort to target its resources and align the Promise program goals with those of other place-based initiatives, the Promise program coordinates closely with a limited number of federal programs within Education and with other federal programs as part of the White House Neighborhood Revitalization Initiative (NRI).The NRI is an interagency coordinating body that aligns place-based programs run by HUD, HHS, Justice, and the Department of the Treasury (Treasury) (see fig. 5). Coordination through NRI is more structured than internal coordination within Education, which, according to Promise program officials, occurs as needed. Liaisons from each grant program meet at biweekly and monthly NRI meetings. They have formed a program integration workgroup to coordinate program development, monitoring, and technical assistance for the grant programs included. For example, they conducted a joint monitoring trip to a neighborhood in San Antonio, Texas that has Promise, HUD’s Choice Neighborhood, and Justice’s Byrne Criminal Justice Innovation grants.

In coordinating within Education and with NRI, Education’s efforts are focused on ensuring that grants are mutually reinforcing. These coordination activities include aligning goals, developing common performance measures where there are common purposes, and sharing technical assistance resources in areas where programs address similar issues or fund similar activities. (See table 4.)

The Promise program has also participated in another place-based program led out of the White House Domestic Policy Council: the Strong Cities, Strong Communities initiative. This program sends teams of federal officials to work with distressed cities, providing them expertise to more efficiently and effectively use the federal funds they already receive. Education’s Promise program participates in initial on-site assessments of communities. Education staff assisted two of the participating communities by providing education expertise at their request.

Promise Zones had to meet a number of requirements, including meeting certain poverty thresholds and having certain population levels. agencies and five other agencies in partnership with state and local governments, businesses, and non-profit organizations. Only areas that already had certain NRI grants or a similar rural or tribal grant were eligible to apply in the first round. As of January 2014, three Promise Neighborhoods implementation sites in San Antonio, Los Angeles, and Southeastern Kentucky were located in designated Promise Zones, which provide additional opportunity for coordination at the federal and local level.

The Promise Neighborhoods program does, on occasion, coordinate with other individual federal agencies and programs outside of the NRI, but officials stated that the program is focused on deepening and broadening the communication it has with the five named NRI programs and Promise Zones. Promise Neighborhoods officials explained that they had concerns about spreading their coordination efforts too thinly given the large number of programs grantees may include in their strategies.

Education Does Not Have an Inventory of Federal Programs with Similar Goals

In addition to Promise grants from Education, individual Promise Neighborhoods have access to a broad range of federal programs from other agencies, including many programs that are not part of NRI. However, Education has not developed an inventory of federal programs that could contribute to Promise program goals that it could share with planning and implementation grantees and use to make its own decisions about coordination across agencies. In recent work examining approaches used by interagency groups that successfully collaborated, we found that an inventory of resources related to reaching interagency goals can be used to promote an understanding of related governmentwide programs. Such inventories are useful in making decisions about coordinating related programs across agency lines and between levels of government, according to officials. We have also found that creating a comprehensive list of programs is a first step in identifying potential fragmentation, overlap, or duplication among federal programs or activities.

As shown in table 5, the 12 implementation grantees we surveyed stated that they included a variety of federal resources in their Promise Neighborhoods strategies. AmeriCorps was included in 9 out of 11 implementation grantees’ strategies, followed by Head Start (8 of 12) and Education’s School Improvement Grants (6 of 11). None of these are part of NRI. Few grantees said that NRI programs were part of their Promise strategies. For example, four grantees said that a Choice Neighborhood grant was part of their Promise strategy, and three grantees stated that DOJ’s Byrne program was part of their strategy. Education officials attributed the small number of grantees that use HUD’s Choice program to the fact that few grantees have distressed public housing within their footprint that is eligible for this funding.

Although Promise grantees conduct their own inventories of the existing federal and other resources in their neighborhoods in order to develop their strategies, two grantees we spoke with were unaware of some of the other federal programs that could contribute towards their strategies. For example, one implementation grantee we spoke to with concerns about school safety was unaware of DOJ’s Byrne Criminal Justice Innovation grant program. Another planning grantee who completed our survey commented that a list of related federal programs like the one in our survey would be especially useful to grantees who did not receive implementation grants. Education officials with the Promise program told us that sometimes grantees are unaware that the community is benefiting from certain federal programs because programs are renamed as they filter down through the state or local levels. Education officials said they emphasize to grantees the importance of reaching out to key partners to ensure they are aware of other federally funded programs in the neighborhood because their partners may be more knowledgeable about other sources of federal funding. While encouraging grantees to reach out to key partners is helpful, Education, through its coordination with other federal agencies, would likely have more knowledge about existing federal resources.

Without a federal level inventory, Education is not well-positioned to support grantee efforts to identify other federal programs that could contribute to Promise program goals. Further, Education lacks complete information to inform decisions about future federal coordination efforts and identify potential fragmentation, overlap, and duplication.

Education Collects Extensive Data but Has Not Determined Whether or How to Evaluate the Program

While Education is collecting a large amount of data from Promise grantees that was intended, in part, to be used to evaluate the program, the Education offices responsible for program evaluation— the Institute for Educational Sciences (IES) and Office of Planning, Evaluation, and Policy Development (OPEPD)—have not yet determined whether or how they will evaluate the program.

One of Education’s primary goals for the Promise program, as described in the Federal Register, is to learn about the overall impact of the program through a rigorous program evaluation. Applicants are required to describe their commitment to work with a national evaluator for Promise Neighborhoods to ensure that data collection and program design are consistent with plans to conduct a rigorous national evaluation of the program and the specific solutions and strategies pursued by individual grantees. We have found that federal program evaluation studies provide external accountability for the use of public resources. Evaluation can help to determine the “value added” of the expenditure of federal resources or to learn how to improve performance—or both. Evaluation can play a key role in strategic planning and in program management, informing both program design and execution.

Education requires implementation grantees to report annually on their performance using 15 indicators. The indicators include graduation rates, attendance, academic proficiency, student mobility, physical activity, and perceptions of safety. (See table 11 in appendix III.) Education contracted with the Urban Institute to provide guidance on how to collect data on the indicators, including data sources and survey techniques. According to Urban Institute officials, they used existing, validated measures whenever possible to ensure comparability across programs. Seven of twelve implementation grantees we surveyed said the guidance documents were extremely or very helpful, while four found it moderately helpful and one somewhat helpful.

The Urban Institute has analyzed the data on the indicators for the first implementation year (the baseline), but Education has not decided whether it will make the first year’s data public because it was not collected in a consistent manner and not all grantees were able to collect all of the necessary data. According to Promise program officials there were inconsistencies in data collection because guidance was not available until February 2013, 13 months after 2011 implementation grants were awarded and over 1 month after 2012 implementation grants were awarded.

Promise officials stated that they will use the performance data to target their technical assistance. They are still working with grantees to develop meaningful targets for the second implementation year. Urban Institute officials noted that these 15 indicators help grantees focus their efforts on the outcomes they are trying to achieve.

In addition, Promise grantees are required to develop a longitudinal data system to collect information on the individuals served, services provided in the cradle-to-career continuum, and the related outcomes.are expected to use the longitudinal data to evaluate their programs on an Grantees ongoing basis and make adjustments to their strategies and services, as discussed later in this report.

Grantees are also required to provide the longitudinal data to Education, which Education officials said they may use to create a restricted-use data set. However, Education currently does not have a plan for analyzing the data. In commenting on a draft of this report, Education stated it must first conduct a systematic examination of the reliability and validity of the data to determine whether it can be used for a descriptive study and a restricted-use data set. Education further stated that the restricted-use data set would only be made available to external researchers after Education determines that the data quality is adequate and appropriate for research; analyzes the data, taking into account privacy concerns; and determines whether to release its own report. In addition, officials from IES and OPEPD cited limitations and challenges to using the longitudinal data for program evaluation.

An official from IES, the entity responsible for all impact evaluations conducted by Education, told us that it is not feasible to conduct an impact evaluation of individual program pieces or an overall evaluation of the Promise approach. The official offered three options for evaluation. IES’ preferred option is to conduct a rigorous impact evaluation with a control group obtained through randomized assignment to the program. However, Promise Neighborhoods are not designed to create such a control group. Another option would be for IES to use students or families who were not chosen to participate in an oversubscribed program as a control group, but an informal poll that IES took at a Promise Neighborhoods conference suggested that there were not a sufficient number of oversubscribed programs. A third option was to develop a comparison group of neighborhoods that did not receive a Promise Neighborhood grant. However, IES officials question whether such an approach would enable them to match neighborhoods that were comparable to Promise neighborhoods at the beginning of the grant period. Finally, IES noted that collecting additional data for a control group could be expensive.

Education’s OPEPD is responsible for conducting other types of program evaluations. According to Education officials, it could conduct a more limited evaluation focused on outcomes without demonstrating that they are a direct result of the Promise program, but they have no specific plans to do so. An OPEPD official stated OPEPD is reluctant to commit to a plan because they have not yet seen the data and do not know how reliable or complete it will be. In addition, the official said that OPEPD is unsure about funding and that any comprehensive evaluations are expensive to carry out. By creating a restricted-use data set, OPEPD hopes that other researchers may have the funding to use the data to reach some conclusions about the program. The OPEPD official further explained that no one has ever evaluated a community-based approach like this one and that they hope researchers may have some ideas about how to do so.

Researchers at the Urban Institute and within the Promise grantee community have proposed other options for evaluating the program. A researcher at the Urban Institute noted that random assignment is not the right approach for evaluating place-based programs. Instead, the researcher recommends a variety of other options for evaluating such programs, including approaches that estimate a single site’s effect on outcomes and aggregating those outcomes. This differs from the traditional program evaluation approach, which IES has considered, of isolating the effects of an intervention so that its effects can be measured separately from other interventions.

While Education recognizes the importance of evaluating the Promise program, they lack a plan to do so. If an evaluation is not conducted, Education will have limited information about the Promise program’s success or the viability of the program’s collaborative approach.

Grantees Collaborate Using Leading Practices, Resulting in Benefits and Challenges

Promise Grant Framework Encourages Collaboration

The Promise program generally requires grantees to use collaborative approaches. We found that grantees are following approaches consistent with those we have recognized as enhancing and sustaining collaboration with partners. The approaches we have previously identified include:

Establishing common outcomes: Establishing common outcomes helps collaborating agencies develop a clear and compelling rationale to work together.

Addressing needs by leveraging resources: Leveraging the various human, information technology, physical and financial resources available from agencies in a collaborative group allows the group to obtain benefits that would not be available if they worked separately.

Tracking performance and maintaining accountability: Tracking performance and other mechanisms for maintaining accountability are consistent with our prior work, which has shown that performance information can be used to improve results by setting priorities and allocating resources to take corrective actions to solve program problems.

The approaches are discussed below and in Tables 6 through 8.

Grantees Collaborate to Deliver a Range of Services

Grantees and partners provided examples of how they have collaborated through the Promise grant to deliver services and supports that are intended to improve educational and developmental outcomes. Grantees and their partners focused on delivering services at various steps along the cradle-to-career pipeline, including:

Early learning supports: programs or services designed to improve outcomes and ensure that young children enter kindergarten and progress through early elementary school grades demonstrating age- appropriate functioning.

K-12 supports: programs, including policies and personnel, linked to improving educational outcomes for children in pre-school through 12th grade. These include developing effective teachers and principals, facilitating the use of data on student achievement and student growth to inform decision-making, supporting a well-rounded curriculum, and creating multiple pathways for students to earn high school diplomas.

College and career supports: programs preparing students for college and career success. These include partnering with higher education institutions to help instill a college-going culture in the neighborhood, providing dual-enrollment opportunities for students to gain college credit while in high school, and providing access to career and technical education programs.

Family and community supports: these include child and youth physical, mental, behavioral and emotional health programs, safety programs such as those to prevent or reduce gang activity and programs that expand access to quality affordable housing.

For examples of the services delivered and outcomes reported by grantees for each part of the cradle-to-career pipeline, see table 9 below.

Conclusions

The Promise program has energized the 48 planning and implementation grantees and their partners to tackle the complex challenges facing impoverished neighborhoods together. While grantees said they will continue their efforts to build their Promise Neighborhoods, planning grantees faced challenges in sustaining their work over the long term without implementation grants. Planning grantees, especially those concerned about building trust with their communities and partners, may have been better served if Education had provided a more transparent, realistic picture of the fiscal reality of the Promise program and its potential impact on implementation grant funding. Lack of clear communication about the expectations Education had for planning grantees who did not receive implementation funding made it difficult for these grantees to develop specific plans to continue their efforts without future Promise funds. However, the reported small, yet tangible benefits that some communities pursued during the planning year—such as a safe place for children to play—increased momentum and built trust with community members. Encouraging such “early wins” could help all grantees and their partners build upon and improve their efforts, especially since implementation funding has proven scarce.

Additionally, much of the information grantees use about what existing federal, state, and local programs and resources to incorporate into their strategies is gleaned through their needs assessment at the local level. Education has not provided grantees with comprehensive information about other federal resources that may be available to use in their Promise strategies. Education is best positioned to develop and share such an inventory of federal programs that relate to the goals of the Promise program. Without such an inventory, Education may be missing opportunities to better support grantees, find other federal programs for future coordination efforts, and identify potential fragmentation, overlap, and duplication at the federal level.

One of the Promise program’s primary goals is to identify the overall impact of its approach and the relationship between particular strategies and student outcomes. Grantees are investing significant time and resources to collect data to assess the program, but Education lacks a clear plan for using it. Without evaluating program, it will be difficult for Education to determine whether it is successfully addressing the complex problem of poor student outcomes in impoverished neighborhoods. Finally, the Promise program is one of several place-based and collective impact programs being implemented across many federal agencies. Given the number of these initiatives, not evaluating the program limits Education and other agencies from learning about the extent to which model is effective and should be replicated.

Recommendations for Executive Action

In order to improve grantees’ planning and implementation efforts, increase the effectiveness of grantee efforts to integrate and manage resources, and learn more about the program’s impact, we recommend that the Secretary of Education take the following three actions: 1. Clarify program guidance about planning and implementation grants to provide reasonable assurance that planning grantees are better prepared to continue their efforts in the absence of implementation funding. Additional guidance could include encouraging grantees to set aside a small amount of the grant to identify and deliver early, tangible benefits to their neighborhoods. 2. Develop and disseminate to grantees on an ongoing basis an inventory of federal programs and resources that can contribute to the Promise Neighborhoods program’s goal to better support coordination across agency lines. 3. Develop a plan to use the data collected from grantees to conduct a national evaluation of the program.

Agency Comments

We provided a draft of this report to the Department of Education for review and comment. Education’s comments are reproduced in appendix IV and are summarized below. Education also provided technical comments, which we incorporated into the final report as appropriate. Education outlined the steps it would take to implement our three recommendations, and provided its perspective on communicating expectations to grantees regarding future funding. Education did not explicitly agree or disagree with our findings.

Regarding our finding that Education did not communicate clearly to planning grantees about its expectations for the grants, Education stated that in any given year it does not know and therefore cannot communicate the amount of funding available or the number of grant awards anticipated in the following year. We agree, and have clarified our finding in the report accordingly.

Education stated that an early assessment of planning grantees’ likelihood of receiving implementation funding would have been premature. Education noted that although Congress has funded the Promise program for the past 5 years, in 4 of those 5 years it appropriated far less than the President requested, and for the last 3 years the program has essentially been level funded. Education further stated that this underscores the limited control that the program had over the number of implementation grants made. We recognize that federal agencies have faced a difficult fiscal climate over the past few years, particularly for discretionary programs. For that reason—and especially given the level at which the Promise program has been funded for the past 3 years—we believe it is even more important that Education be clear and transparent with planning grantees about historical fiscal realities of the Promise program and the implications this may have on future implementation grants. We also believe this situation highlights the need for planning grantees to have contingency plans, especially given Education’s expectations that grantees continue their efforts even in the absence of implementation funding. We further believe that this also underscores the importance of “early wins” to demonstrate what can be achieved when grantees and their partners work collaboratively, as such demonstrations can encourage them to continue their efforts even without implementation funding.

In discussing its perspective on communicating expectations to grantees regarding future funding, Education stated that its Notifications Inviting Applications indicated that future funding was contingent on the availability of funds and that the program’s frequently asked questions document noted that implementation funding was not guaranteed and that planning grantees would have to compete for implementation grants. We believe that our report adequately reflects these communication efforts. However, as we reported, Education did not communicate to planning grantees that it expected them to continue their efforts even in the absence of implementation funding. Nor did Education communicate to implementation grant applicants that it expected them to be able to use their partners’ pledged matching funds even if they did not receive implementation grants. This lack of communication was evidenced by planning grantees’ lack of contingency plans and challenges they faced accessing the pledged matching funds, according to the grantees we interviewed.

In response to our first recommendation, Education stated that it would continue to communicate to planning grant applicants that implementation funding is contingent on the availability of funds, and that it would provide more targeted technical assistance to planning grant recipients regarding strategies for continuing grantees’ efforts absent implementation funding. Education also stated that it would clarify to grantees that planning grant funds could be used to achieve early, tangible benefits.

Regarding our second recommendation, Education stated that it would work with its technical assistance providers to create a mechanism to distribute a comprehensive list of external funding opportunities, programs and resources on a regular basis to better support the grantees’ implementation efforts.

With regard to our final recommendation, Education stated that it will consider options for how and whether it can use the data collected from grantees to conduct a national evaluation. Education stated that as a first step it will conduct a systematic evaluation of the reliability and validity of the data, given issues that we and Education noted about inconsistencies in data collection and privacy concerns. In addition, Education stated that to date, it has not received sufficient funding to support a national evaluation. We agree that conducting evaluations can be costly. However, given that one of Education’s primary goals is to learn about the overall impact of the program through a rigorous program evaluation, we continue to believe that absent an evaluation, it will be difficult for Education to determine whether it is successfully addressing the complex problem of poor student outcomes in impoverished neighborhoods—one of its stated goals. Further, developing an evaluation plan would provide critical information about the resources required to conduct an evaluation, and could better inform future funding requests for such an evaluation.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Education and other interested congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at 617-788-0580 or NowickiJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are in appendix V.

Appendix I: Objectives, Scope, and Methodology

To better understand grantees’ experiences with the Promise Neighborhoods program, we conducted a web-based survey of all 48 planning and implementation grantees. The survey was conducted from August 23, 2013 through November 7, 2013. We received completed surveys from all 48 grantees for a 100 percent response rate. The survey included questions about the clarity and helpfulness of the application and peer review process, challenges sustaining efforts after the end of the planning grant, coordination of federal resources, collaboration with local organizations and associated challenges, the extent to which local coordination reduced duplication, overlap and fragmentation, if at all, the mechanisms organizations use to track the results of their efforts, the results of the grants, and the helpfulness of Education’s guidance and resources for the program.

Because this was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce nonsampling errors, such as variations in how respondents interpret questions and their willingness to offer accurate responses. We took steps to minimize nonsampling errors, including pretesting draft instruments and using a web-based administration system. Specifically, during survey development, we pretested draft instruments with five grantees that received planning and/or implementation grants. In the pretests, we were generally interested in the clarity, precision, and objectivity of the questions, as well as the flow and layout of the survey. For example, we wanted to ensure definitions used in the surveys were clear and known to the respondents, categories provided in closed-ended questions were complete and exclusive, and the ordering of survey sections and the questions within each section was appropriate. We revised the final survey based on pretest results. We took another step to minimize nonsampling errors by using a web-based survey. This allowed respondents to enter their responses directly into an electronic instrument and created a record for each respondent in a data file—eliminating the need for manual data entry and its associated errors. To further minimize errors, programs used to analyze the survey data were independently verified to ensure the accuracy of this work. Because not all respondents answered every question, we reported the number of grantees responding to particular questions throughout the report.

In addition, we conducted site visits to 11 Promise grantees. We selected sites based on several factors, such as the type of grant awarded, the location of the grantees, and whether the Promise Neighborhood was urban or rural. The site visits provided opportunities to collect more in- depth information on the program and highlighted different types of grantees and approaches. We visited six implementation grantees in Boston, Massachusetts; Berea, Kentucky; Chula Vista, California; Indianola, Mississippi; Los Angeles, California; and Washington, DC. We visited five planning grantees in Campo, California; Lawrence, Massachusetts; Los Angeles, California; Nashville, Tennessee; and Worcester, Massachusetts. These include one tribal and two rural grantees. We also interviewed Education officials and technical assistance providers, as well as other experts who have worked with Promise grant applicants, such as the Promise Neighborhoods Institute.

To determine how well the structure of Education’s Promise Neighborhoods grant program aligns with program goals and how Education selected grantees, using Education’s goals for the Promise program as criteria, we reviewed Education reports on place-based strategies; relevant Federal Register notices; and application guidance and training materials, including both the guidance available to applicants and to the peer reviewers regarding the technical evaluation/grant selection process. We reviewed agency information on applicants for implementation grants in the fiscal year 2011 and 2012 cycles, as fiscal years 2011 and 2012 were the only years in which Education awarded implementation grants. For both cycles, we analyzed application materials and technical evaluation documentation for a subset of implementation grant applicants—those that received planning grants in prior years. We compared the scores in each component of the application for both successful and unsuccessful applicants to identify criteria or factors that accounted for significant variation in total scores. We conducted a limited review of selected peer reviewer comments to gain more insight into the reasons for any differences. We interviewed Education officials about the process that the department used for the selection of both planning and implementation grantees.

To determine how the Promise Neighborhoods program coordinated with other Education programs and with the other federal agencies, including those involved in the White House Neighborhood Revitalization Initiative (NRI), we reviewed documentation of the NRI’s efforts and interviewed agency officials participating in the NRI. We also interviewed cognizant officials at other agencies participating in the NRI. To assess Education’s approach to evaluating the success of the grants, we reviewed grant monitoring reports, Education’s performance measures, and related guidance for data collection for this program and interviewed agency officials responsible for evaluation, including technical assistance providers.

To determine the extent to which Promise grants enabled collaboration at the local level, we used GAO’s prior work on implementing interagency collaborative mechanisms as criteria. We compared the Promise grants’ collaboration mechanisms to certain successful approaches used by select interagency groups and reviewed implementation grantees’ application materials. Our 11 site visits provided additional insight into how selected grantees align services supported by multiple funding streams and delivered by multiple providers. Using survey responses from all planning grantees, we determined whether they have continued their efforts, whether they have implemented any of their strategies, and what, if any, interim results they have identified, regardless of whether they received implementation grants. Site visits provided illustrative examples of interim benefits and challenges.

We conducted this performance audit from February 2013 to May 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Promise Neighborhood Grantees

Location New York , NY (Harlem)

Athens Clarke County Family Connection Inc.

Clay, Jackson, and Owsley Counties, KY Boys & Girls Club of the Northern Cheyenne Nation Northern Cheyenne Reservation, MT Community Day Care Center of Lawrence, Inc.

New York , NY (Brooklyn)

United Way of Central Massachusetts, Inc.

New York, NY (Brooklyn)

New York, NY (Queens)

New York, NY (Brooklyn)

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Elizabeth Sirois, Assistant Director; Jacques Arsenault; Aimee Elivert; and Lara Laufer made key contributions to this report. Also contributing to this report were James Bennett, Deborah Bland, Mallory Barg Bulman, Holly Dye, Alex Galuten, Jean McSween, Matthew Saradjian, and Sarah Veale.