K-12 Education: Education Needs to Improve Oversight of Its 21st Century Program
Released: 2017-04-26T00:00:00

Letter

Background

The 21st Century program is authorized to provide a wide range of activities for K-12 students and their families to: 1. provide opportunities for academic enrichment, including providing tutorial services to help students—particularly students who attend low-performing schools—meet state and local academic standards in core subjects such as reading and mathematics; 2. offer students a broad array of additional services, such as drug and violence prevention, counseling, art, music, recreation, and technology programs that are designed to reinforce and complement instruction in the regular school day; and 3. offer literacy and related educational development opportunities to families of students served.

Education has established performance objectives for the 21st Century program, including objectives related to student outcomes. These objectives state that 21st Century participants will demonstrate educational and social skills and exhibit positive behavioral changes; and they will improve in outcomes such as academic performance, school attendance, and rates of disciplinary incidents and other adverse behaviors (see fig. 1).

The 21st Century program is administered by state educational agencies. Education provides state agencies with annual formula grants, which in 2015 ranged from about $6 million to $132 million. The formula that determines the funding amount for a particular state is based in part on the percentage of total students from low-income families enrolled in K-12 public schools and how much that state spends per pupil on education. States, in turn, competitively award funds to sub-grantees, which may be school districts or community-based organizations, such as those that focus on youth development. By law, states must award sub-grants of a minimum $50,000 per year for periods of 3 to 5 years.

Sub-grantees oversee one or more physical locations, referred to as “centers,” where grant-funded services are provided to participating students and adult family members. Centers may be located in schools, churches, community centers’ or other spaces (see fig. 2). They must provide services during non-school hours or periods when school is not in session, such as before school, after school, on weekends, or during summer vacations and school breaks.

In making awards, states must give priority both to applications that propose to serve students who attend schools identified as needing improvement and that are submitted jointly by at least one school district receiving funds under Title I, Part A of the Elementary and Secondary Education Act of 1964 (ESEA), as amended, and at least one community- based organization or other public or private entity. In addition, states are authorized to include additional priorities in their sub-grant competitions so long as they are aligned with the statute’s requirements and priorities.

Program Monitoring and Evaluation

Education’s Office of Academic Improvement, under the Office of Elementary and Secondary Education, is responsible for overseeing states’ implementation of the 21st Century program. The Office of Academic Improvement is also responsible for providing ongoing technical assistance to states and monitoring state performance through on-site and desk monitoring visits and ongoing communications with state program officials. The Office of Academic Improvement conducts on-site monitoring for the 21st Century program for each state grantee every 3 years, or more frequently if needed.

The Office of Academic Improvement is also responsible for reviewing states’ grant applications for, among other things, compliance, including reviewing certain assurances that states are required to provide under ESEA. For example, Education must review states’ assurances that they will grant awards only to eligible entities that propose to serve students who primarily attend schools with at least 40 percent of students from low-income families. In addition, states must collect plans from applicants describing how activities funded by the sub-grant will continue after 21st Century funding ends, as well as plans for how centers will address participants’ transportation needs, among other things.

Education collects program data from states in order to report annually on whether 21st Century programs have met performance targets for their objectives. These data include information on sub-grantee characteristics, center activities, and participants’ demographics and outcomes. States monitor sub-grantee performance and program data submission to Education.

Under ESEA, states are required to conduct ongoing program evaluations to assess the effectiveness of their 21st Century programs and activities and disseminate the findings. Education reviews state evaluations as a part of its monitoring process.

States Use a Variety of Criteria to Award Sub-grants for 21st Century Program Centers, Which Use Funds to Provide a Broad Array of Services

States Use Varied Criteria, Funding Periods, and Amounts for Sub-grants to 21st Century Program Providers

Together states received more than 4,000 21st Century sub-grant applications and funded nearly 60 percent of them (about 2,400) in their most recent sub-grant competitions, according to our survey of all 50 states and the District of Columbia. States’ criteria for these competitions vary, and most states score applications on a point system based on a variety of criteria. To help determine whether applicants have the capacity to implement high-quality programs, over half of states award applicants additional points based on the quality of their program design (33 states) and use of evidence-based practices (28 states). To sustain these programs once grant funding ends, about half of the states reported awarding additional points based on the applicants’ level of support from schools and school districts (25 states) and other external organizations (23 states). (See fig. 3.)

Additionally, 10 states reported in our survey that they provided additional points to applicants serving schools identified as the lowest performing or with the largest gaps in student achievement based on comparisons by race or socio-economic levels.

To provide adequate time to implement successful programs, a majority of states reported that they offer sub-grantees 21st Century funding for 5 years, the maximum number of years allowed under law. Specifically, 39 states said in our survey that they provide 21st Century funding to sub- grantees for 5 years. Education’s 2003 program guidance states that research suggests it takes approximately 5 years of continual revision and improvement for a community to fully implement a successful 21st Century program. Further, existing 21st Century sub-grantees are also eligible to re-apply for funding outside the competition period if their grant is close to expiring. Of the applicants that received an award in the most recent competition, about 45 percent (nearly 1,100) had previously received a 21st Century grant award.

States have provided different levels of funding for sub-grantees and their centers. Fifteen states reported the minimum and maximum amounts sub-grantees allocated to their centers. Specifically, the amount that a sub-grantee’s center received ranged from a minimum of $100,000 to a maximum of about $660,000, with a median amount of $185,000 per center in school year 2015-2016. Officials in Idaho told us the amounts vary because each center has different needs and student numbers. They said their program’s average cost per student was $1,700 per year. They noted that they encourage centers to target students most in need of services instead of serving all students in a school, even if the school primarily serves students from low-income families. Officials in Texas told us that funding amounts to centers can vary based on other state requirements, such as a requirement to employ a full-time program director or coordinator at the sub-grantee or center level.

Sub-grantees Use 21st Century Funds to Provide Academic Enrichment and Recreational Activities

Sub-grantees have used 21st Century funds to provide a broad array of services to K-12 students. Education’s 2014 annual report on the characteristics of 21st Century programs found that the most commonly provided activities in school year 2012-2013 were those to enrich academics, provide tutoring and homework help, and offer recreational activities. According to Education’s 2003 guidance, academic enrichment can include tutoring in core academic subjects and providing extra learning opportunities so that students can practice their academic skills through hands-on activities. The 2014 annual report also found that many centers supported building students’ academic skills in reading and math as well as in the arts, science, technology, cultural activities, and health. Further, at least two states gathered information on the time spent on specific activities as part of their state 21st Century program evaluations. In New Jersey, for example, a 2015 state evaluation examined time spent by students on specific 21st Century activities, including academic improvement/remediation, tutoring, academic enrichment, community service, and recreation. Of these, academic improvement/remediation was the activity on which students spent the most time. Oregon’s 2012 state evaluation also found centers were most likely to offer weekly activities they categorized as enrichment, homework help, or recreational. In terms of subjects targeted by the program, centers were most likely to report in the state evaluation that their centers’ weekly activities focused on reading, math, and arts or music.

Officials in all 13 centers we visited in our four selected states said they offered a mix of academic support and enrichment activities. These activities generally focused on reading and math, science and technology, art and music, and fitness and nutrition. For example, centers we visited used computers and tablets to reinforce math skills or introduce new concepts, such as computer programming, for students who had limited access to technology. Officials at 7 of the 13 centers also said 21st Century activities can fill gaps in a school’s offerings such as providing music or computer instruction that schools may not have the opportunity to provide during the regular school day. (See fig. 4.) 21st Century centers serve different grade levels and use different staffing models. Education’s 2014 annual report and officials in the states we visited said that centers were primarily for elementary school students, but some also served middle and high school students. Further, Education’s report found that centers relied primarily on paid staff, but staff characteristics varied widely. For example, the report found that some centers used regular school-day teachers and non-school based staff, such as youth development workers or staff without college degrees. In the 13 centers we visited, officials told us their staff included a mix of certified teachers and paraprofessionals who work during the regular school day, as well as staff from outside the school who work in the program only part-time.

Sub-grantees Leverage Their 21st Century Funding to Obtain Additional Funding from Other Sources

Education’s guidance encourages sub-grantees to identify other sources of related funding, and Education’s monitoring protocols ask sub-grantees to describe how all of these resources will be combined or coordinated to offer a high-quality, sustainable program. Eight states in our survey reported that they require sub-grantees to match Education funds with funding from other sources. For example, one state required sub- grantees to match 30 percent of their 21st Century grant award, with at least 10 percent required from sources outside the participating school district. Even if a state does not require matching funding, 21st Century applicants for sub-grants must identify federal, state, and local programs they will combine or coordinate with on their 21st Century program to make the most effective use of public resources. In our survey, states reported that the two most frequent funding sources sub-grantees used to supplement their 21st Century funding were the local school district (23 states) and other federal government programs (21 states), such as the Department of Health and Human Services’ Child Care and Development Fund and the Department of Agriculture’s Child and Adult Care Food Program. (See fig. 5.) Officials at several centers we interviewed also said they provide snacks and meals during the afterschool programs using Department of Agriculture funds. Ten states reported in our survey that sub-grantees frequently use other Education funding, such as Title I, Part A of ESEA, as amended, which provides financial assistance to districts and schools with high numbers or percentages of children from low-income families. In addition, officials in Idaho told us their sub- grantees also use funds from Education’s Migrant Education program, which provides educational and support services to migrant children.

Officials from 46 states reported in our survey that they allow sub- grantees to charge student fees. These states, however, generally had a policy that no child would be turned away due to a family’s inability to pay. Of these states, 27 reported they require sub-grantees to have a sliding scale for fees based on family income.

To encourage sub-grantees to secure funding from alternate sources, most states reported that they gradually reduce 21st Century sub-grants over the funding cycle. Specifically, officials in 37 states reported in our survey that they reduce funding for sub-grantees after an initial period. For example, one state reported that it awards full funding in the first 2 years, but then reduces funding to 80 percent of that amount in the third year, 60 percent in the fourth year, and 40 percent in the fifth year. Further, 15 states reported that they reduce funding levels for sub- grantees that have received 21st Century grants in previous funding cycles (“repeat sub-grantees”). Officials in Texas told us they fund repeat sub-grantees for 3 years rather than the usual 5 to preserve funds for new recipients. In Massachusetts, repeat sub-grantees can only apply for 50 to 75 percent of their previous grant amount.

Research Suggests 21st Century Programs Improve Students’ Behavioral Outcomes More Often Than Academic Outcomes Research Indicates Improvements in Some Behavioral Outcomes for Participating Students

Few evaluations of the 21st Century program use methodologies that are appropriate for determining the effect of program participation on student outcomes. Of the 10 studies we included in our review of program effectiveness—4 state evaluations and 6 other studies—that do use such methodologies, 3 state evaluations found a positive relationship between program participation and school-day attendance and/or discipline. (See appendix I for more details on our process for selecting studies.) These state evaluations—from New Jersey, Texas, and Washington—examined behavioral outcomes such as school-day attendance and discipline. All three of these evaluations found there was a positive effect on school-day attendance, and two of them found there was also a positive effect on school-day discipline. For example, Washington’s state evaluation found that school-day attendance improved for students in grades 6 through 12 who participated in the program when compared to similar students who had not participated. Similarly, Texas’s state evaluation found participating students in grades 4 through 11 had improved school-day attendance. This effect was particularly strong for students who participated in the program 60 days or more, especially in grades 6 through 11. These students had an absentee rate that was more than 20 percent lower than non-participants. Additionally, Texas’s evaluation found that students in a statewide 21st Century pilot program with intensive academics showed improved school-day attendance. In New Jersey’s evaluation of students in grades 4 through 8, students in all grades had improved attendance; this improvement was generally larger in the higher grades, particularly 8th grade. Further, one of the three meta-analyses of afterschool programs in general also identified similar positive effects on school-day attendance. This meta-analysis, published by Education in 2014, synthesized the results of 30 studies, including 6 studies of 21st Century programs. It identified positive effects on students’ academic motivation, a broad category of outcomes which included measures such as school-day attendance and homework completion.

Two of the four state evaluations we included in our review also examined school-day disciplinary incidents—which include fighting, bullying, or disruptive conduct that result in a student’s removal from the classroom— and both found that 21st Century program participation may improve outcomes, particularly for students who participated in the program for 60 days or more. Specifically, Washington’s state evaluation found an association between program participation and fewer disciplinary incidents for students in grades 3 through 12 who attended 60 days or more. In Texas, student participation in the program for 60 days or more was also associated with fewer disciplinary incidents. Separately, Texas’s evaluation also found that the presence of certain program characteristics in the state’s 21st Century academic pilot program were associated with lower rates of disciplinary incidents. In particular, the evaluation showed that centers in the pilot that taught students face-to- face rather than via computer were associated with fewer disciplinary incidents. Moreover, this positive association was stronger for centers whose curricula focused on general learning strategies rather than on specific subject area skills.

Regarding the effect of the 21st Century program on reducing delinquency, a 2016 meta-analysis of 12 studies examining afterschool programs in general, including one on programs funded through the 21st Century program, found no significant effect. This meta-analysis identified levels of delinquency—incidents such as arrest rates and violent behavior—as an important measure for afterschool programs because many of them focus on reducing such behaviors by providing a safe, supervised environment where children spend less time with potentially delinquent peers. Nevertheless, when the researchers averaged the results of all afterschool programs in their review, they found no significant effect on delinquency rates.

Research Indicates Mixed Effects on Math and Reading Scores for Participating Students

Results of some studies we reviewed demonstrated a positive association between participation in 21st Century programs and improved academic outcomes for selected groups of students or for particular types of activities. For example, in Texas the program was associated with higher math scores for students who participated 60 days or more. However, none of the 10 studies in our review observed consistently better scores in either math or reading in program participants’ state assessments. (See appendix I for more details on our process for selecting studies.)

Program Effects on Math Scores

The 10 studies we reviewed identified differing effects of participation in 21st Century programs on students’ math scores. For example, evaluations for Virginia in grades 4 through 12 and for Washington in grades 4 through 8 found the program had no significant effect on math scores. On the other hand, two other state evaluations indicated that 21st Century programs were associated with improved math scores for certain grade levels. Specifically, New Jersey’s evaluation found a positive association between program participation and higher math scores, but it was not observable for students in 4th or 6th grade. Additionally, results from Texas’s evaluation found an association between program participation and increased math scores among middle school students.

Program Effects on Reading Scores

None of the state evaluations we reviewed showed a significant association between participation in a 21st Century program and increased reading scores. In fact, Texas’s evaluation showed lower reading scores among students who participated in the program compared to students who did not participate. In particular, participants in grades 4 and 5 had lower reading scores than non-participants. However, the Texas evaluation showed no observable decline in reading scores among students in middle school who attended 30 days or more. Evaluations for Washington, Virginia, and New Jersey found no significant relationship between program participation and increased reading scores for grades 4 through 8. Additionally, a 2014 academic study examined differences in program characteristics among 58 21st Century programs in New York City and found no association with improved reading scores for students.

Program Effects for Subgroups of Students

Afterschool programs’ inconsistent effects on math and reading scores may be the result of these programs serving students with different needs, according to recent research. In particular, two studies we reviewed indicated that the impact of specific types of activities in 21st Century programs may differ depending on the students being served. Specifically, they used statistical modeling to analyze the relationships between program features and the different educational needs among 21st Century program participants. One study found that programs focusing solely on academic content were associated with larger increases in reading scores for students with limited English proficiency than programs that mixed academic content with other activities; while students overall benefited from both solely academic programs as well as those that mixed academic with other activities. The other study found that activities with structured interactions with adults—including opportunities for collaboration and meaningful verbal exchanges—were associated with increased reading scores for middle school students. For elementary students, however, there was no association between these activities and improved reading scores.

Education Has Not Taken Important Steps to Ensure 21st Century Data Are Useful for Program Decision Making

Education’s Performance Measures and Data Do Not Address Some Key 21st Century Objectives

While Education has developed performance measures to align with some 21st Century program objectives—primarily student academic outcomes—it has not aligned its measures with other program objectives related to key student behavioral and socio-emotional outcomes. As previously noted, these objectives describe a goal of program participants demonstrating improvement in three areas—educational, social, and behavioral—with outcomes such as improved academic performance and school attendance, and lower rates of disciplinary incidents and other adverse behaviors.

Alignment of Performance Measures with Program Objectives

Education’s current performance measures were established in 1998. They address participating students’ English and math grades and state test scores as well as some behavioral outcomes, including homework completion, class participation, and classroom behavior. However, Education does not measure two other behavioral outcomes that are included in 21st Century program objectives: improved school-day attendance and a decrease in disciplinary incidents, although research has shown positive program effects for these two outcomes more often than for academic outcomes. Some states also recognize the importance of measuring behavioral outcomes associated with these programs, with about half of states (26 states) reporting in our survey that they choose to measure at least one of them. The remaining states, however, are not measuring either of these behavioral outcomes.

In addition, Education has not established any performance measures for socio-emotional outcomes, although social skills are also included in program objectives, and socio-emotional learning is an important component of 21st Century implementation across states we visited and surveyed. According to Education’s website, socio-emotional learning involves students’ knowledge and skills necessary to understand and manage emotions and establish positive relationships, among other things. Twenty-seven states reported in our survey that they currently measure or are developing measures for at least one socio-emotional outcome, including student relationships with adults or communication skills. Again, the remaining states are not measuring any of these socio- emotional outcomes. At the centers we visited, programs offered various activities geared toward socio-emotional learning, including activities that paired students with adult mentors and developed students’ problem- solving skills. Officials in two states we visited told us that socio-emotional learning is a major component of their 21st Century programs’ philosophy. (See figure 6 for a comparison of 21st Century program objectives and Education’s program performance measures).

Several factors contributed to Education’s decision to maintain its current 21st Century performance measures for over 15 years. In particular, Education officials told us they have not substantially revised performance measures since 1998 in part because the program’s authorization lapsed from 2008 through 2016, with ESSA providing funding authorization starting in fiscal year 2017. As a result, the officials said this created uncertainty about potential future program changes. In addition, they said they were reluctant to revise the measures given the costs to states of tracking and reporting on new measures, as well as concerns about maintaining the continuity of data collected over time. Education officials told us they may revise the program’s performance measures when they implement program changes pursuant to ESSA and that in doing so, they will consider including additional behavioral performance measures as well as socio-emotional measures.

Education’s lack of 21st Century performance measures for some key program objectives, however, has limited the usefulness of its performance data. Specifically, without measures for student behavioral and socio-emotional outcomes, Education lacks useful data on the extent to which the program is achieving its stated objectives, especially in areas where the program is likely to have positive effects on student outcomes. While about half of states reported they already measure at least one behavioral or socio-emotional outcome, absent measures for all states, Education will continue to report incomplete program results to Congress and the public. Further, leading practices in performance management call for agencies to use performance information as the basis for decision making. These practices also state that aligning performance measures with program objectives can enhance the use of information for management decision making.

Education Does Not Have Reasonable Assurance That 21st Century Program Data Submitted by States Are Accurate

While Education has taken steps to promote data quality, including data accuracy and completeness, it lacks reasonable assurance that the data submitted by states are accurate. Education officials told us they began using a new online data system in school year 2014-2015, called 21APR, in part to improve data quality. Among other changes, 21APR does not permit users to advance to the next page until the data on the current page are complete. Education officials said that this can help prevent missing data—a feature that was absent in the previous data system. In addition, in 2016 Education provided states with regional training sessions and instructional documents for data entry. However, Education has not independently assessed the accuracy of 21st Century data submitted in 21APR.

Education officials told us they require states to check their data submissions rather than Education conducting these checks because the agency does not have access to sub-grantees’ and centers’ source information. Education’s own Information Quality Guidelines state that data should be processed and edited to help ensure they are accurate. Performing accuracy checks would not necessarily require that Education have access to the source data. For example, it could perform basic logic checks between fields—such as verifying that the number of student participants in one grade is fewer than the total number of participants in all grades. Education officials told us they may explore additional checks for accuracy, but they did not provide a timeframe for doing so.

Education officials do review states’ procedures for data quality assurance, although some states have expressed concerns about the quality of data they receive from sub-grantees. Since 2015, when Education implemented new monitoring protocols that included a review of states’ data quality procedures, Education officials have monitored 20 states and have not found any states that are noncompliant. However, in our interviews and survey responses, officials from 10 states commented that they had concerns about the quality of program data they collect from sub-grantees. For example, states are unable to upload data to the 21APR system, unlike in Education’s previous data system, and several states expressed concerns that they must enter data manually. One state official commented that this can increase the risk of errors.

Without independently checking accuracy, Education may be unable to detect and address potential data errors in the 21APR system—data that Education eventually needs to use to conduct analyses that inform management decisions, identify sub-grantees needing technical assistance, and contribute to annual performance reports and budget justifications. Without reasonable assurance that data in the 21APR system are of sufficient quality, Education may not be able to effectively use the data for such decision making and reporting going forward, despite the fact that the new system was created, in part, to improve data quality. Further, absent better information on the quality of its data, Education will be unable to communicate any data limitations in its reports to Congress and the public. Our prior work has identified leading practices in performance management, which state that agencies should disclose limitations associated with data used in reporting and indicate what action they plan to take to address the limitations.

Education’s Technical Assistance Does Not Yet Adequately Address Many States’ Challenges in Evaluating and Sustaining Programs States Reported Mixed Reviews on Education’s Technical Assistance for Evaluating their 21st Century Programs

States in our survey gave mixed reviews on the usefulness of Education’s technical assistance for developing and conducting state-level evaluations. Twenty-nine states reported that they found Education’s technical assistance on conducting state evaluations to be very or moderately useful. On the other hand, 12 states reported they found it only slightly or not at all useful; 8 states reported they had not received any technical assistance on this topic; and one state said it was unaware that Education provided such technical assistance.

Further, 40 states commented in our survey that they face challenges in evaluating sub-grantee performance that may limit their capacity to conduct high-quality evaluations. For example, six states reported they have difficulty designing evaluations that can determine whether a change in student outcomes results from a student’s participation in a 21st Century program or from other factors, such as interventions during the school day. This difficulty is evidenced by the fact that we identified very few state evaluations whose evidence standards allowed us to include them in our review on program effectiveness, as noted above. In addition, four states reported that they face challenges in defining and establishing performance measures for students who participate in the program, such as measures for behavioral and academic outcomes. For example, one state commented that it is difficult to measure the benefits of enrichment activities on academic performance.

Education also found that states are having difficulty conducting their required evaluations when Education reviewed them as part of its monitoring process. For example, in recent monitoring findings, Education officials said they found that five states did not have plans in place to conduct these evaluations. In addition, according to technical assistance reports provided as a part of the monitoring process, three states had difficulty conducting comprehensive evaluations such as establishing the appropriate scope of work and timeframes for conducting evaluations and developing appropriate performance measurement tools.

According to Education officials, it has not provided written guidance to states on 21st Century program evaluations since 1999, when it provided states non-regulatory guidance in the form of a guide to help them evaluate their programs and use the results to make program improvements. However, since 1999, there have been significant changes in the 21st Century program. Specifically, the program was reauthorized twice, resulting in changes to requirements for measuring and evaluating program performance. First, the 2002 reauthorization added new requirements for states to describe in their funding applications how the state will evaluate the effectiveness of programs, including developing performance indicators and measures to evaluate programs and activities. It also required sub-grantees to conduct periodic local evaluations to assess progress toward achieving goals. Second, as a result of requirements under ESSA, which reauthorized the 21st Century program in 2015, additional changes are planned starting in school year 2017-2018. Specifically, states will have to evaluate their programs in conjunction with new data collection and evaluation planning requirements, including requirements to track student progress over time and to include state standardized test scores and other indicators of student success, such as improved school-day attendance.

Federal standards for internal control state that when significant changes occur in how an agency achieves its objectives it should periodically review policies and procedures for continued relevance and effectiveness in achieving its objectives or addressing related risks. Further, it states that written policies and procedures can help ensure that necessary actions are taken to address risks to achieving the entity’s objectives. In addition, our prior work has identified several leading practices in agencies’ use of evaluations, which state that agencies should promote capacity building to evaluate program activities as a key strategy for supporting objectives. Education officials told us they are developing a technical assistance plan to improve the support the agency provides to states to implement the 21st Century program. These officials told us they are considering the topics and areas of concern that they will address including developing and conducting effective evaluations and assessments, and they expect to finalize the list of topics in summer 2017. However, Education officials told us that the agency has not determined what type of technical assistance it will provide to states on conducting evaluations. Absent new written, non-regulatory guidance to states that addresses the areas in which they struggle most, Education may miss opportunities to help states improve their capacity to conduct high-quality evaluations and ultimately improve their programs.

Education’s Technical Assistance Does Not Effectively Address States’ Difficulty in Sustaining 21st Century Programs

Education’s technical assistance to states does not effectively address the challenges most states face in helping their 21st Century sub- grantees continue to operate their programs once grant funding ends, according to state and sub-grantee officials. In our survey, 35 states reported that their centers often face challenges providing the same level of services to students when the grant funding ends. Officials in 20 states reported that centers in their states generally reduce the level of services or cease to operate once 21st Century funding ends. Two states commented that less than 10 percent of centers had been able to continue operating after the 21st Century grants expired. These concerns were echoed by officials at 10 of the 13 21st Century centers we visited, who told us they had major concerns about sustaining their programs.

Twenty centers that commented they will be unable to sustain operations or maintain the same level of services after 21st Century funding ends had difficulty securing funds from other sources. For example, officials in one state commented that it is often difficult for centers to obtain funding from local school districts because district budgets are already strained by supporting school-day programs. Officials in three other states commented that there is no state funding dedicated exclusively to programs outside the regular school day.

By law, states must require sub-grant applicants to submit a preliminary plan for how their programs will continue to operate once grant funding ends. Education’s monitoring protocols call for Education officials to ask states if applicants have sustainability plans and how states monitor sub- grantees’ implementation of the plans. A majority of states (42 states) reported in our survey that they require sub-grantees to provide a written sustainability plan, while 6 states reported they do not. However, in its monitoring efforts, Education told us it did not identify any states that are currently out of compliance with this requirement.

Education may be missing opportunities in its monitoring efforts to collect information on states’ strategies and practices for program sustainability—information that could be useful for sharing promising practices across states. As a part of its monitoring process, Education officials told us they discuss with state officials how they ensure compliance with the requirement to have such sustainability plans in place, but they do examine the quality of the plans. Officials in 11 states in our survey commented they would benefit from opportunities to collaborate across states. States do appear to have valuable information on sustainability to share with each other. For example, 25 states—in our survey of all 50 states and the District of Columbia—reported providing additional points or credit based on applicants’ ability to leverage external funds and sustain programs when selecting sub-grantees. Further, in the states that collect information on what happens after 21st Century funding ends, most reported centers having some success in finding private or nonprofit funds to help replace Education’s grant funds. States reported that the most common sources of replacement funding were student fees charged to participants (17 states), private foundation funding (14 states), and non-profit funding (e.g., universities and community organizations) (13 states).

Education officials told us they provide technical assistance on 21st Century program sustainability through interactions such as an annual summer training conference on the program and an online learning portal. However, Education’s July 2016 training conference—a forum where 21st Century participants share experiences and best practices— was generally not focused on state practices and policies. The sessions were generally focused on centers’ collaboration with the school or community, centers’ activities (e.g., literacy, science, and math), or how to report performance data. Officials in about a third of states reported in our survey that they did not know about or had no basis to judge Education’s technical assistance on sustainability. Another third of states reported that Education’s technical assistance in this area was slightly or not at all useful. For example, two states reported that the content of Education’s professional development, training, and other assistance is heavily geared toward sub-grantees.

Education officials said they do have other interactions and forums, such as regional meetings, to share information with states. For example, Education started in-depth training in 2016 for state-level officials at four regional meetings for the Midwest, Northwest, South, and East, but the sessions did not cover program sustainability.

Federal standards for internal control state that information should be communicated in a form that enables an agency to achieve its objectives. In addition, our prior work has identified leading practices in federal collaboration that have shown that information sharing among grant participants, such as states, is important for effective grants management. Education’s efforts to help facilitate information sharing among states can help states identify practices that can be tailored to meet their individual needs and leverage their knowledge to address common challenges in continuing their programs over the long-term.

Unless Education undertakes such efforts for the 21st Century program, students and families who participate may be at greater risk of not receiving the full range program benefits for the longest amount of time.

Conclusions

The 21st Century program is designed to fund programs outside the regular school day to improve academic and behavioral outcomes for K- 12 students who are from low-income families or who attend low- performing schools. High-quality research on the effectiveness of this program is very limited due to several factors, including difficulty determining whether a change in students’ outcomes results from their participation in a 21st Century program or from other factors, such as interventions during the school day. Although existing research on effectiveness points to greater positive behavioral effects than academic effects, Education’s current performance measures do not address some key behavioral outcomes. The lack of performances for behavioral outcomes makes it difficult to determine whether the program is achieving some of its stated objectives—especially in areas where research has shown that the program is likely to have the greatest effect on student outcomes. Specifically, Education does not measure socio-emotional outcomes or two other behavioral outcomes included in program objectives: improved school-day attendance and discipline in the classroom. Absent these performance measures, Education is missing an opportunity to assess the full range of benefits of this program.

In addition, Education has not sufficiently assessed the quality of its program data, further limiting its ability to assess program effectiveness. Another factor hindering the agency’s ability to determine program effectiveness is the significant difficulty that states are experiencing in evaluating their programs. Although Education officials told us they are considering topics and areas of concern for additional technical assistance, Education has not yet provided states with sufficient guidance on developing rigorous evaluations. Unless Education takes steps to reasonably ensure the accuracy of data and provides written guidance to help states develop high-quality evaluations, it cannot ensure the program is effectively meeting its goals.

Lastly, states are experiencing substantial difficulty in sustaining their programs after 21st Century funding ends, and many states expressed interest in collaborating with other states to address such challenges. Education is uniquely situated to take the lead in sharing information with states to help them address their sustainability challenges by allowing them to identify state policies and practices that have had some success.

Sharing such information could help states address the challenges they face in continuing their programs over the long-term.

Recommendations for Executive Action

We recommend that the Secretary of Education direct the Office of Academic Improvement to: 1. Expand its performance measures for the 21st Century program to address all program objectives. Specifically, Education should establish performance measures related to key behavioral, including student attendance and disciplinary incidents, and socio-emotional outcomes. 2. Conduct federal-level data checks on the accuracy of 21st Century program data submitted by states. Such checks could test for logical relationships between fields. Education should also publicly disclose and address any data limitations it identifies, as appropriate. 3. Provide written, non-regulatory guidance to states on developing and conducting high-quality 21st Century state evaluations to help address the difficulties states face in measuring program performance and effectiveness. 4. Use the information it collects from its monitoring visits and ongoing interactions with states to share effective practices across states for sustaining their 21st Century programs once program funding ends. This information could be shared using existing mechanisms such as Education’s meetings with 21st Century state coordinators.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Education for comment, and its written comments are reproduced in appendix III. Education also provided technical comments that we incorporated in the report as appropriate. Education neither agreed nor disagreed with our recommendations; rather, it generally noted that it will keep our recommendations in mind as it continues to implement changes in the program as a result of ESSA, and outlined steps it will take to address our recommendations.

In response to our recommendation that it expand its performance measures for the 21st Century program to address all program objectives, including those related to behavioral and socio-emotional outcomes, Education said it will keep our recommendation in mind. Specifically, the department stated that it is in the process of re-examining whether additional or revised measures should be developed to align more significantly with the program’s statutory objectives under ESSA. Education also acknowledged the need to develop measures that provide sufficient information and data on program outcomes. As stated in our draft report, the Department currently measures some student behavioral outcomes; in its response, Education described one measure as: the percentage of all program participants with teacher-reported improvements in behavior. Therefore, we modified our recommendation to focus on measuring the program’s key behavioral outcomes. In its response, Education also expressed concern about collecting data on student attendance and disciplinary measures, noting that it will require effective collaboration between states, districts, and other eligible entities. However, as we stated in our report, about half of states already collect data on at least one of these two measures. Further, as we stated, research has shown that 21st Century programs more often have positive effects on student attendance and reducing disciplinary incidents than on improving students’ academic outcomes. Given these effects, we continue to believe that it is critical for Education to measure student attendance and disciplinary incidents to obtain more complete, accurate information on this program’s effect on student outcomes.

In response to our recommendation that it conduct federal-level data checks on the accuracy of 21st Century program data submitted by states, Education commented that it plans to build in additional data checks into the data system beyond its current checks on the data’s completeness. Specifically, Education anticipates that new technology enhancements in the data system will be designed to flag for inconsistencies in data reporting. For example, the system may send a “flag” that participation data is significantly lower or higher than previously reported participation data. Further, Education indicated that it will consider whether auditors performing audits under the Single Audit Act can be asked and guided to do more checks on the accuracy and reliability of 21st Century program data.

Regarding our recommendation to provide written, non-regulatory guidance to states on developing and conducting high-quality 21st Century state evaluations, Education outlined several steps it has taken to assist states in the past. For example, Education said that it provided six states with individualized technical assistance on strategies related to developing statewide evaluations and measures. Education also noted that, to date, it has conducted two webinars on state evaluations and is in the process of including presentations from those webinars on its online learning portal so that states will have easy access to the information. In addition, Education stated that it included presentations on evaluation strategies in the past during its Summer Institute. Education also said it would consider whether additional guidance for all states was needed. While these are important steps, we do not believe they are sufficient. Twenty-one of 51 states (41 percent of states) reported in our 2016 survey that they found Education’s technical assistance for developing and conducting state-level evaluations only slightly or not at all useful; they had not received any technical assistance on this topic; or they were unaware that Education provided such assistance. Therefore, we continue to believe that Education should prepare written guidance to assist all states in developing and conducting high-quality program evaluations.

Finally, regarding our recommendation to share effective practices across states for sustaining their 21st Century programs once program funding ends, Education stated that it hosts meetings twice a year for 21st Century state coordinators. At these meetings, Education officials share strategies with states related to program sustainability. Education stated that these meetings covered topics such as reducing the amounts of 21st Century grant awards by a percentage each year. However, Education officials told us in February 2017 that these meetings have not focused on topics on program sustainability for several years. We continue to believe that Education should take the lead in sharing information with states to help them address their sustainability challenges by sharing information on state policies and practices that have shown some success. In its comments, Education stated that it has not held regional meetings with states for several years; however, in 2016 it held four regional meetings with states on implementing its new 21APR data system. Therefore, we modified our recommendation to emphasize that such information be shared through these types of existing mechanisms.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff should have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations nowickij@gao.gov Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our study of the 21st Century program was framed around four objectives: (1) how 21st Century funds are awarded and used, (2) what is known about the effectiveness of 21st Century programs, (3) the extent to which Education has effectively managed and used program data to inform decisions about the 21st Century program, and (4) the extent to which Education’s technical assistance has helped states assess and sustain high-quality programs once 21st Century funding ends. We focused our review on 21st Century afterschool programs because centers have historically provided services more during this time as compared to before school and during the summer. To address our four objectives, we used a variety of methods, including a Web-based survey of 21st Century state coordinators; a review of selected state evaluations and academic literature; a review of federal laws, regulations, and agency documents such as annual performance reports and guidance; interviews with federal and other officials; and site visits to four states.

Survey of State Coordinators

To obtain information about the state sub-grantee award process, program sustainability and technical assistance of the 21st Century program, we conducted a Web-based survey of the 21st Century state coordinator at each state educational agency in all 50 grantee states and the District of Columbia. We conducted the survey from August 2016 through November 2016. The survey covered several topics, including state processes for awarding 21st Century sub-grants to local communities, state-level program evaluations, performance measures, and sub-grantees’ financial sustainability, among other things. We received responses from all 50 states and the District of Columbia for a 100 percent response rate. The survey included an introductory statement specifying that the survey focused on afterschool programs funded by the 21st Century program and collects information about federal-level guidance provided to state officials on this program.

The quality of survey data can be affected by nonsampling error, which includes variations in how respondents interpret questions, respondents’ willingness to offer accurate responses, and data collection and processing errors. To minimize such error, we included the following steps in developing the survey and in collecting and analyzing survey data: in developing the Web survey, we pre-tested draft versions of the instrument with 21st Century program officials in four states to check the clarity of the questions and the flow and layout of the survey. On the basis of the pretests, we made revisions to the survey. Further, using a Web- based survey and allowing 21st Century coordinators to enter their responses directly into an electronic instrument created an automatic record for each state in a data file and eliminated the errors associated with a manual data entry process. In addition, the program used to analyze the survey data was independently verified to ensure the accuracy of this work.

Review of State Evaluations and Academic Literature

In order to examine what is known about the effectiveness of 21st Century programs, we reviewed select state-level program evaluations and academic studies that reported outcomes for students who participated in the program. To identify relevant research, we took a two- pronged approach. First, we reviewed three state evaluations that Education determined in a 2012 report on 21st Century program grantee evaluation practices (the most recent available Education report on this topic) to have used quasi-experimental methods or statistical controls, such as a comparison group, to account for other plausible influences on the outcomes reported. We retrieved the most recent evaluations from those three states and through interviews with researchers in the field; we identified three other quasi-experimental state evaluations published since 2012. We reviewed each of these state evaluations for the soundness of its methodology and data. In four cases where the published evaluation did not include enough information to complete our review, we contacted the states and the researchers to obtain additional data from their samples. Ultimately, we determined that four of the six state evaluations were appropriate for purposes of our research objective about the 21st Century program’s effectiveness.

Second, we conducted a comprehensive literature review of 104 academic studies of afterschool programs, including the 21st Century program, and we ultimately determined that six of these academic studies were sufficiently rigorous and appropriately scoped to include in our review. To identify academic studies on the effectiveness of 21st Century programs, we conducted a literature search through ProQuest. Our initial search terms were adapted from a 2014 report produced by Education that looked at the effect of 21st Century programs, along with other programs that increased the amount of time that children spent in school. Because our review was focused on afterschool programs, rather than the broader range of out of school time or extended learning time programs covered in that study, we only searched for studies of afterschool programs and studies of 21st Century programs particularly.

We identified 104 papers initially and selected 25 of them on the basis of the following criteria:

Topic relevance–the studies covered 21st Century programs or afterschool programs that measured student outcomes and had some instructional component (i.e. programs were not only a study hall or a sports program).

Timeframe relevance–studies must have been published since 2012, in order to find studies that may not have been included in Education’s 2014 study, which examined studies published since 1998.

Publication status–studies were in their final form, not drafts.

Sample relevance–studies looked at afterschool programs in K-12 settings in the United States. Foreign school systems were excluded.

Design relevance–studies used experimental or quasi-experimental designs with well-formulated comparison groups or controls; or they used some statistical method to account for other plausible influences on the outcomes of the study.

Next, two analysts reviewed those studies a second time, retaining 13 studies that had a sample size above 100 students in the intervention group and evaluated more than one program or intervention site. The analysts further narrowed this sample to six studies, which we selected because they were focused on 21st Century programs. In addition to reviewing Education’s 2014 meta-analysis, in our initial search of 104 papers, the analysts also identified two additional meta-analyses focused on afterschool programs, which we included in our sample because they were of appropriate quality and scope to assess afterschool program effects. We then conducted detailed reviews of the studies. These reviews entailed an evaluation of each study’s research methodology, including its research design, sampling frame, selection of measures, data quality, limitations, and analytic techniques, as well as a summary of its major findings. We also assessed the extent to which each study was relevant to assessing what is known about the effectiveness of 21st Century programs. Three studies had major research design limitations resulting from the lack of a rigorously formed comparison group. As such, these studies were not able to demonstrate whether the programs were responsible for the effects being measured or whether other factors may have contributed. After eliminating the three studies with major research design limitations, six studies remained in our review.

Review of Agency Documents and Interviews with Agency Officials

For all four objectives, we reviewed agency documents and interviewed agency officials and researchers who had conducted work on the 21st Century program. To examine how states and other entities have used 21st Century funds, we reviewed Education’s 2014 annual characteristic report for the 21st Century program. The reports provide summaries and analyses over time, based on program data collected from states in school years 2001-2002 through 2012-2013 using data from the Profile and Performance Information Collection System. In order to assess the reliability of the data underlying the reports, we reviewed agency documents regarding this data system, spoke with agency officials, and conducted electronic tests of data from the system for select data elements. We determined that the underlying data summarized and analyzed in Education’s 2014 annual characteristic report was sufficiently reliable for our purpose of reporting on program characteristics.

To examine Education’s management of program data, we also reviewed Education documents including budget justifications and documents on the 21st Century data systems such as user guides, technical assistance documents for states, data dictionaries, and a 2013 Education Office of Inspector General report on federal and state oversight for the 21st Century program.

To examine Education’s technical assistance, we reviewed Education’s program guidance to states, monitoring protocols and tools, and annual grantee satisfaction survey results which generally includes Education’s largest grant programs including the 21st Century program. In order to compare the program to established criteria, we reviewed relevant legislation; Office of Management and Budget guidance; federal internal control standards; and past GAO work pertaining to leading practices in performance management, information quality, program evaluation, and federal collaboration for grants management.

We interviewed federal officials from Education’s Office of Academic Improvement, within the Office of Elementary and Secondary Education, regarding data management and use, research and evaluation, and program challenges. In addition, we interviewed stakeholders of the 21st Century program, including contractors and researchers. Finally, we observed Education’s annual Summer Institute, where Education provides technical assistance and training to state, sub-grantee, and center officials.

Site Visits

To learn about program activities and challenges, we conducted site visits to four states—Idaho, Massachusetts, Rhode Island, and Texas—to visit 21st Century centers and speak with program staff and state officials. We selected these states for diversity in geographic region, 21st Century formula grant funding levels to states, and student demographics. In each state, we interviewed the 21st Century state coordinator and other officials responsible for the program. In total, we visited 13 centers in the four states. During these visits, we interviewed center staff and observed afterschool program activities.

Information we gathered on our site visits represents only the conditions present in the states and local areas at the time of our visits. Furthermore, our fieldwork focused on in-depth analysis of only a few selected states. On the basis of our site visit information, we cannot generalize our findings beyond the states we visited.

We conducted this performance audit from April 2016 to April 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: 21st Century Grant Award Amount by States, Fiscal Year 2016

Appendix III: Comments from the U.S. Department of Education

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Elizabeth Sirois (Assistant Director), Sheranda Campbell (Analyst-in-Charge), Lucas Alvarez, Ashley Chaifetz, Jamila Kennedy, and Kelsey Kennedy made significant contributions to this report. Assistance, expertise, and guidance were provided by Susan Aschoff, Carl Barden, Deborah Bland, James Bennett, Angie Jacobs, Thomas James, Kirsten Lauber, Ben Licht, Edward Malone, John Mingus, Amy Moran Lowe, Sara Pelton and James Rebbe.