K-12 Education: Education Needs to Improve Oversight of Its 21st Century Program
Published: 2017-04-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Education's 21st Century program—funded about $1 billion annually since 2002—supports a broad array of activities outside the school day to improve student outcomes in high-poverty or low-performing K-12 schools. A statement accompanying the Consolidated and Further Continuing Appropriations Act of 2015 included a provision for GAO to review Education programs outside the regular school day.
GAO examined (1) how 21st Century funds are awarded and used, (2) what is known about the effectiveness of these programs, (3) how Education manages and uses program data to inform decision making, and (4) Education's technical assistance for evaluating and sustaining programs. GAO conducted a 50-state survey of program officials, obtaining a 100 percent response rate. GAO also reviewed selected state program evaluations and academic studies on student outcomes, and observed program activities and interviewed officials in four states representing a range of grant size and location.
section_title: What GAO Found
paragraphs: The Department of Education (Education) awards 21st Century Community Learning Centers (21st Century) grants to states, who in turn, competitively award funds to local organizations, which use them to offer academic enrichment and other activities to improve students' academic and behavioral outcomes. In their most recent grant competitions, states awarded 21st Century funds to nearly 2,400 organizations—including school districts and community-based organizations—based on a variety of criteria, such as the quality of their proposed program designs. Relevant research we reviewed that compared program participants to those of non-participants suggests that the 21st Century program is effective in improving students' behavioral outcomes, such as school-day attendance and reduced disciplinary incidents, more often than their academic outcomes. However, because Education's current 21st Century performance measures primarily focus on students' reading and math scores on state tests, Education lacks useful data about whether the program is achieving its objectives to improve students' behavioral outcomes such as attendance and discipline—the areas where the program most frequently has a positive effect. Education officials have not substantially revised the program's performance measures since 1998, in part because its authorization lapsed from fiscal years 2008 through 2016. Leading practices in performance measurement call for federal agencies to align performance measures with program objectives.
Education's technical assistance to states does not adequately address challenges states face in evaluating their 21st Century programs and sustaining them when program funding ends. About a third of states reported in GAO's 50-state survey that they face challenges in evaluating program performance, such as difficulty designing evaluations that shed light on program effects. Further, the 21st Century program was reauthorized twice, which resulted in significant changes to state requirements for evaluating programs. However, Education has not provided states written guidance on developing and conducting high-quality evaluations since 1999. Federal standards for internal control state that when significant changes occur agencies should periodically review policies for continued relevance and effectiveness in achieving their objectives. Absent written guidance to states on conducting high-quality evaluations, Education may miss opportunities to help states improve their capacity to conduct such evaluations to assure the program is meeting its goals.
section_title: What GAO Recommends
paragraphs: GAO is making four recommendations, including that Education expand its performance measures for behavioral outcomes and provide written guidance to states on conducting high-quality program evaluations. Education neither agreed nor disagreed with the recommendations, and outlined steps it is taking to address them.

section_title: Letter
paragraphs: The 21st Century Community Learning Centers (21st Century) program is  meant to support local communities in providing learning opportunities for  K-12 students outside the regular school day, such as before school, after  school, and during the summer. The program is designed to target  services primarily to students who are from low-income families or who  attend schools in need of academic assistance. From fiscal years 2002 to  2016, Congress appropriated a total of about $16 billion to the  Department of Education (Education) to administer 21st Century funds.  In 2015, the program was reauthorized by the Every Student Succeeds  Act (ESSA). According to Education, in school year 2013-2014 (the most  recent data available at the time we did our work) the program provided  funding for about 9,500 centers serving approximately 1.8 million students  and 430,000 adult family members.
A statement accompanying the Consolidated and Further Continuing  Appropriations Act of 2015 included a provision for GAO to review K-12  education programs that extend learning time beyond the regular school  day. This report examines (1) how 21st Century funds are awarded and  used, (2) what is known about the effectiveness of 21st Century  programs, (3) the extent to which Education has effectively managed and  used program data to inform decisions about the 21st Century program,  and (4) the extent to which Education’s technical assistance has helped  states assess and sustain high-quality programs once 21st Century  funding ends.
We focused our review on 21st Century afterschool programs because  centers have historically provided services more during this time as  compared to before school and during the summer. To answer all four  research questions, we reviewed Education documents, including annual  performance reports, studies, and budget justifications; and federal laws,  regulations, and policies. We also interviewed Education officials and  other researchers on the 21st Century program. To assess the reliability  of federal 21st Century program data underlying Education’s annual  performance reports and studies, we conducted electronic data tests, reviewed relevant documents, and interviewed agency officials. We  determined that the data were sufficiently reliable for our purposes. We  also conducted a Web-based survey of 21st Century state coordinators  from the 50 states and the District of Columbia and received responses  from 100 percent of the coordinators. We analyzed responses to survey  questions as well as additional comments state coordinators provided in  response to open-ended questions. In addition, we interviewed state  officials responsible for administering the 21st Century program in four  states: Idaho, Massachusetts, Rhode Island, and Texas; and we  observed activities and interviewed staff at two to four 21st Century  centers in each of those states. States were selected to represent a range  in the amount of 21st Century grant funding, student demographics such  as race and ethnicity, and geographic location. Information collected  about these four states is not generalizable but provides insight into 21st  Century program operations at the local level.
Additionally, to examine what is known about the effectiveness of 21st  Century programs, we reviewed select state-level program evaluations  and other studies that reported outcomes for students who participated in  the program. To identify relevant state-level evaluations, we took a two- pronged approach. First, we reviewed state evaluations that Education  determined in a 2012 report on 21st Century program state evaluation  practices (the most recent available Education report on this topic) to  have used quasi-experimental methods or statistical controls, such as a  comparison group, to account for other plausible influences on the  outcomes reported. We then interviewed researchers to identify other  quasi-experimental state evaluations published since 2012, and we  reviewed them for the soundness of their methodology and data.  Ultimately, we determined that four state evaluations were appropriate to  address our research objective about the 21st Century program’s  effectiveness. Second, we conducted a comprehensive literature review  of 104 other studies of afterschool programs, including the 21st Century  program. We ultimately determined that six studies were sufficiently  rigorous and appropriately scoped to include in our review, including three  meta-analyses that synthesized a broad array of studies on the outcomes  of afterschool programs in general, including the 21st Century program.  Additional information on our research objectives, scope, and  methodology is available in appendix I.
We conducted this performance audit from April 2016 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The 21st Century program is authorized to provide a wide range of  activities for K-12 students and their families to:  1.  provide opportunities for academic enrichment, including providing  tutorial services to help students—particularly students who attend  low-performing schools—meet state and local academic standards in  core subjects such as reading and mathematics;  2.  offer students a broad array of additional services, such as drug and  violence prevention, counseling, art, music, recreation, and  technology programs that are designed to reinforce and complement  instruction in the regular school day; and  3.  offer literacy and related educational development opportunities to  families of students served.
Education has established performance objectives for the 21st Century  program, including objectives related to student outcomes. These  objectives state that 21st Century participants will demonstrate  educational and social skills and exhibit positive behavioral changes; and  they will improve in outcomes such as academic performance, school  attendance, and rates of disciplinary incidents and other adverse  behaviors (see fig. 1).
The 21st Century program is administered by state educational agencies.  Education provides state agencies with annual formula grants, which in  2015 ranged from about $6 million to $132 million. The formula that  determines the funding amount for a particular state is based in part on  the percentage of total students from low-income families enrolled in K-12  public schools and how much that state spends per pupil on education.  States, in turn, competitively award funds to sub-grantees, which may be  school districts or community-based organizations, such as those that  focus on youth development. By law, states must award sub-grants of a  minimum $50,000 per year for periods of 3 to 5 years.
Sub-grantees oversee one or more physical locations, referred to as  “centers,” where grant-funded services are provided to participating  students and adult family members. Centers may be located in schools,  churches, community centers’ or other spaces (see fig. 2). They must  provide services during non-school hours or periods when school is not in  session, such as before school, after school, on weekends, or during  summer vacations and school breaks.
In making awards, states must give priority both to applications that  propose to serve students who attend schools identified as needing  improvement and that are submitted jointly by at least one school district  receiving funds under Title I, Part A of the Elementary and Secondary  Education Act of 1964 (ESEA), as amended, and at least one community- based organization or other public or private entity. In addition, states  are authorized to include additional priorities in their sub-grant  competitions so long as they are aligned with the statute’s requirements  and priorities.
subsections: section_title: Program Monitoring and Evaluation
paragraphs: Education’s Office of Academic Improvement, under the Office of  Elementary and Secondary Education, is responsible for overseeing  states’ implementation of the 21st Century program. The Office of  Academic Improvement is also responsible for providing ongoing  technical assistance to states and monitoring state performance through  on-site and desk monitoring visits and ongoing communications with state  program officials. The Office of Academic Improvement conducts on-site  monitoring for the 21st Century program for each state grantee every 3  years, or more frequently if needed.
The Office of Academic Improvement is also responsible for reviewing  states’ grant applications for, among other things, compliance, including  reviewing certain assurances that states are required to provide under  ESEA. For example, Education must review states’ assurances that they  will grant awards only to eligible entities that propose to serve students  who primarily attend schools with at least 40 percent of students from  low-income families. In addition, states must collect plans from  applicants describing how activities funded by the sub-grant will continue  after 21st Century funding ends, as well as plans for how centers will  address participants’ transportation needs, among other things.
Education collects program data from states in order to report annually on  whether 21st Century programs have met performance targets for their  objectives. These data include information on sub-grantee characteristics,  center activities, and participants’ demographics and outcomes. States  monitor sub-grantee performance and program data submission to  Education.
Under ESEA, states are required to conduct ongoing program evaluations  to assess the effectiveness of their 21st Century programs and activities  and disseminate the findings. Education reviews state evaluations as a  part of its monitoring process.
section_title: States Use a Variety of Criteria to Award Sub-grants for 21st Century Program Centers, Which Use Funds to Provide a Broad Array of Services
subsections: section_title: States Use Varied Criteria, Funding Periods, and Amounts for Sub-grants to 21st Century Program Providers
paragraphs: Together states received more than 4,000 21st Century sub-grant  applications and funded nearly 60 percent of them (about 2,400) in their  most recent sub-grant competitions, according to our survey of all 50  states and the District of Columbia. States’ criteria for these  competitions vary, and most states score applications on a point system  based on a variety of criteria. To help determine whether applicants have  the capacity to implement high-quality programs, over half of states award  applicants additional points based on the quality of their program design  (33 states) and use of evidence-based practices (28 states). To sustain  these programs once grant funding ends, about half of the states reported  awarding additional points based on the applicants’ level of support from  schools and school districts (25 states) and other external organizations  (23 states). (See fig. 3.)
Additionally, 10 states reported in our survey that they provided additional  points to applicants serving schools identified as the lowest performing or  with the largest gaps in student achievement based on comparisons by  race or socio-economic levels.
To provide adequate time to implement successful programs, a majority  of states reported that they offer sub-grantees 21st Century funding for 5  years, the maximum number of years allowed under law. Specifically,  39 states said in our survey that they provide 21st Century funding to sub- grantees for 5 years. Education’s 2003 program guidance states that  research suggests it takes approximately 5 years of continual revision  and improvement for a community to fully implement a successful 21st  Century program. Further, existing 21st Century sub-grantees are also  eligible to re-apply for funding outside the competition period if their grant  is close to expiring. Of the applicants that received an award in the most  recent competition, about 45 percent (nearly 1,100) had previously  received a 21st Century grant award.
States have provided different levels of funding for sub-grantees and their  centers. Fifteen states reported the minimum and maximum amounts  sub-grantees allocated to their centers. Specifically, the amount that a  sub-grantee’s center received ranged from a minimum of $100,000 to a  maximum of about $660,000, with a median amount of $185,000 per  center in school year 2015-2016. Officials in Idaho told us the amounts  vary because each center has different needs and student numbers. They  said their program’s average cost per student was $1,700 per year. They  noted that they encourage centers to target students most in need of  services instead of serving all students in a school, even if the school  primarily serves students from low-income families. Officials in Texas told  us that funding amounts to centers can vary based on other state  requirements, such as a requirement to employ a full-time program  director or coordinator at the sub-grantee or center level.
section_title: Sub-grantees Use 21st Century Funds to Provide Academic Enrichment and Recreational Activities
paragraphs: Sub-grantees have used 21st Century funds to provide a broad array of  services to K-12 students. Education’s 2014 annual report on the  characteristics of 21st Century programs found that the most commonly  provided activities in school year 2012-2013 were those to enrich  academics, provide tutoring and homework help, and offer recreational  activities. According to Education’s 2003 guidance, academic  enrichment can include tutoring in core academic subjects and providing  extra learning opportunities so that students can practice their academic  skills through hands-on activities. The 2014 annual report also found that  many centers supported building students’ academic skills in reading and  math as well as in the arts, science, technology, cultural activities, and  health. Further, at least two states gathered information on the time spent  on specific activities as part of their state 21st Century program  evaluations. In New Jersey, for example, a 2015 state evaluation  examined time spent by students on specific 21st Century activities,  including academic improvement/remediation, tutoring, academic  enrichment, community service, and recreation. Of these, academic  improvement/remediation was the activity on which students spent the  most time. Oregon’s 2012 state evaluation also found centers were  most likely to offer weekly activities they categorized as enrichment,  homework help, or recreational. In terms of subjects targeted by the  program, centers were most likely to report in the state evaluation that  their centers’ weekly activities focused on reading, math, and arts or  music.
Officials in all 13 centers we visited in our four selected states said they  offered a mix of academic support and enrichment activities. These  activities generally focused on reading and math, science and technology,  art and music, and fitness and nutrition. For example, centers we visited  used computers and tablets to reinforce math skills or introduce new  concepts, such as computer programming, for students who had limited  access to technology. Officials at 7 of the 13 centers also said 21st  Century activities can fill gaps in a school’s offerings such as providing  music or computer instruction that schools may not have the opportunity  to provide during the regular school day. (See fig. 4.)  21st Century centers serve different grade levels and use different  staffing models. Education’s 2014 annual report and officials in the states  we visited said that centers were primarily for elementary school students,  but some also served middle and high school students. Further,  Education’s report found that centers relied primarily on paid staff, but  staff characteristics varied widely. For example, the report found that  some centers used regular school-day teachers and non-school based  staff, such as youth development workers or staff without college  degrees. In the 13 centers we visited, officials told us their staff included a  mix of certified teachers and paraprofessionals who work during the  regular school day, as well as staff from outside the school who work in  the program only part-time.
section_title: Sub-grantees Leverage Their 21st Century Funding to Obtain Additional Funding from Other Sources
paragraphs: Education’s guidance encourages sub-grantees to identify other sources  of related funding, and Education’s monitoring protocols ask sub-grantees  to describe how all of these resources will be combined or coordinated to  offer a high-quality, sustainable program. Eight states in our survey  reported that they require sub-grantees to match Education funds with  funding from other sources. For example, one state required sub- grantees to match 30 percent of their 21st Century grant award, with at  least 10 percent required from sources outside the participating school  district. Even if a state does not require matching funding, 21st Century  applicants for sub-grants must identify federal, state, and local programs  they will combine or coordinate with on their 21st Century program to  make the most effective use of public resources. In our survey, states  reported that the two most frequent funding sources sub-grantees used to  supplement their 21st Century funding were the local school district (23  states) and other federal government programs (21 states), such as the  Department of Health and Human Services’ Child Care and Development  Fund and the Department of Agriculture’s Child and Adult Care Food  Program. (See fig. 5.) Officials at several centers we interviewed also  said they provide snacks and meals during the afterschool programs  using Department of Agriculture funds. Ten states reported in our survey  that sub-grantees frequently use other Education funding, such as Title I,  Part A of ESEA, as amended, which provides financial assistance to  districts and schools with high numbers or percentages of children from  low-income families. In addition, officials in Idaho told us their sub- grantees also use funds from Education’s Migrant Education program,  which provides educational and support services to migrant children.
Officials from 46 states reported in our survey that they allow sub- grantees to charge student fees. These states, however, generally had a  policy that no child would be turned away due to a family’s inability to pay.  Of these states, 27 reported they require sub-grantees to have a sliding  scale for fees based on family income.
To encourage sub-grantees to secure funding from alternate sources,  most states reported that they gradually reduce 21st Century sub-grants  over the funding cycle. Specifically, officials in 37 states reported in our  survey that they reduce funding for sub-grantees after an initial period.  For example, one state reported that it awards full funding in the first 2  years, but then reduces funding to 80 percent of that amount in the third  year, 60 percent in the fourth year, and 40 percent in the fifth year.  Further, 15 states reported that they reduce funding levels for sub- grantees that have received 21st Century grants in previous funding  cycles (“repeat sub-grantees”). Officials in Texas told us they fund repeat  sub-grantees for 3 years rather than the usual 5 to preserve funds for new  recipients. In Massachusetts, repeat sub-grantees can only apply for 50  to 75 percent of their previous grant amount.
section_title: Research Suggests 21st Century Programs Improve Students’ Behavioral Outcomes More Often Than Academic Outcomes Research Indicates Improvements in Some Behavioral Outcomes for Participating Students
paragraphs: Few evaluations of the 21st Century program use methodologies that are  appropriate for determining the effect of program participation on student  outcomes. Of the 10 studies we included in our review of program  effectiveness—4 state evaluations and 6 other studies—that do use such  methodologies, 3 state evaluations found a positive relationship between  program participation and school-day attendance and/or discipline. (See  appendix I for more details on our process for selecting studies.) These  state evaluations—from New Jersey, Texas, and Washington—examined  behavioral outcomes such as school-day attendance and discipline. All  three of these evaluations found there was a positive effect on school-day  attendance, and two of them found there was also a positive effect on  school-day discipline. For example, Washington’s state evaluation found  that school-day attendance improved for students in grades 6 through 12  who participated in the program when compared to similar students who  had not participated. Similarly, Texas’s state evaluation found  participating students in grades 4 through 11 had improved school-day  attendance. This effect was particularly strong for students who  participated in the program 60 days or more, especially in grades 6  through 11. These students had an absentee rate that was more than  20 percent lower than non-participants. Additionally, Texas’s evaluation  found that students in a statewide 21st Century pilot program with  intensive academics showed improved school-day attendance. In New  Jersey’s evaluation of students in grades 4 through 8, students in all  grades had improved attendance; this improvement was generally larger  in the higher grades, particularly 8th grade. Further, one of the three  meta-analyses of afterschool programs in general also identified similar  positive effects on school-day attendance. This meta-analysis,  published by Education in 2014, synthesized the results of 30 studies,  including 6 studies of 21st Century programs. It identified positive  effects on students’ academic motivation, a broad category of outcomes  which included measures such as school-day attendance and homework  completion.
Two of the four state evaluations we included in our review also examined  school-day disciplinary incidents—which include fighting, bullying, or  disruptive conduct that result in a student’s removal from the classroom— and both found that 21st Century program participation may improve  outcomes, particularly for students who participated in the program for 60  days or more. Specifically, Washington’s state evaluation found an association between program participation and fewer disciplinary  incidents for students in grades 3 through 12 who attended 60 days or  more. In Texas, student participation in the program for 60 days or more  was also associated with fewer disciplinary incidents. Separately,  Texas’s evaluation also found that the presence of certain program  characteristics in the state’s 21st Century academic pilot program were  associated with lower rates of disciplinary incidents. In particular, the  evaluation showed that centers in the pilot that taught students face-to- face rather than via computer were associated with fewer disciplinary  incidents. Moreover, this positive association was stronger for centers  whose curricula focused on general learning strategies rather than on  specific subject area skills.
Regarding the effect of the 21st Century program on reducing  delinquency, a 2016 meta-analysis of 12 studies examining afterschool  programs in general, including one on programs funded through the 21st  Century program, found no significant effect. This meta-analysis  identified levels of delinquency—incidents such as arrest rates and violent  behavior—as an important measure for afterschool programs because  many of them focus on reducing such behaviors by providing a safe,  supervised environment where children spend less time with potentially  delinquent peers. Nevertheless, when the researchers averaged the  results of all afterschool programs in their review, they found no  significant effect on delinquency rates.
subsections: section_title: Research Indicates Mixed Effects on Math and Reading Scores for Participating Students
paragraphs: Results of some studies we reviewed demonstrated a positive association  between participation in 21st Century programs and improved academic  outcomes for selected groups of students or for particular types of  activities. For example, in Texas the program was associated with higher  math scores for students who participated 60 days or more. However,  none of the 10 studies in our review observed consistently better scores  in either math or reading in program participants’ state assessments.  (See appendix I for more details on our process for selecting studies.)
subsections: section_title: Program Effects on Math Scores
paragraphs: The 10 studies we reviewed identified differing effects of participation in  21st Century programs on students’ math scores. For example,  evaluations for Virginia in grades 4 through 12 and for Washington in  grades 4 through 8 found the program had no significant effect on math  scores. On the other hand, two other state evaluations indicated that  21st Century programs were associated with improved math scores for  certain grade levels. Specifically, New Jersey’s evaluation found a  positive association between program participation and higher math  scores, but it was not observable for students in 4th or 6th grade.  Additionally, results from Texas’s evaluation found an association  between program participation and increased math scores among middle  school students.
section_title: Program Effects on Reading Scores
paragraphs: None of the state evaluations we reviewed showed a significant  association between participation in a 21st Century program and  increased reading scores. In fact, Texas’s evaluation showed lower  reading scores among students who participated in the program  compared to students who did not participate. In particular, participants  in grades 4 and 5 had lower reading scores than non-participants.  However, the Texas evaluation showed no observable decline in reading  scores among students in middle school who attended 30 days or more.  Evaluations for Washington, Virginia, and New Jersey found no significant  relationship between program participation and increased reading scores  for grades 4 through 8. Additionally, a 2014 academic study  examined differences in program characteristics among 58 21st Century  programs in New York City and found no association with improved  reading scores for students.
section_title: Program Effects for Subgroups of Students
paragraphs: Afterschool programs’ inconsistent effects on math and reading scores  may be the result of these programs serving students with different  needs, according to recent research. In particular, two studies we  reviewed indicated that the impact of specific types of activities in 21st  Century programs may differ depending on the students being served.  Specifically, they used statistical modeling to analyze the relationships  between program features and the different educational needs among  21st Century program participants. One study found that programs  focusing solely on academic content were associated with larger  increases in reading scores for students with limited English proficiency  than programs that mixed academic content with other activities; while  students overall benefited from both solely academic programs as well as  those that mixed academic with other activities. The other study found  that activities with structured interactions with adults—including  opportunities for collaboration and meaningful verbal exchanges—were  associated with increased reading scores for middle school students. For  elementary students, however, there was no association between these  activities and improved reading scores.
section_title: Education Has Not Taken Important Steps to Ensure 21st Century Data Are Useful for Program Decision Making
subsections: section_title: Education’s Performance Measures and Data Do Not Address Some Key 21st Century Objectives
paragraphs: While Education has developed performance measures to align with  some 21st Century program objectives—primarily student academic  outcomes—it has not aligned its measures with other program objectives  related to key student behavioral and socio-emotional outcomes. As  previously noted, these objectives describe a goal of program participants  demonstrating improvement in three areas—educational, social, and  behavioral—with outcomes such as improved academic performance and  school attendance, and lower rates of disciplinary incidents and other  adverse behaviors.
subsections: section_title: Alignment of Performance Measures with Program Objectives
paragraphs: Education’s current performance measures were established in 1998.  They address participating students’ English and math grades and state  test scores as well as some behavioral outcomes, including homework  completion, class participation, and classroom behavior. However,  Education does not measure two other behavioral outcomes that are  included in 21st Century program objectives: improved school-day  attendance and a decrease in disciplinary incidents, although research  has shown positive program effects for these two outcomes more often  than for academic outcomes. Some states also recognize the importance  of measuring behavioral outcomes associated with these programs, with  about half of states (26 states) reporting in our survey that they choose to  measure at least one of them. The remaining states, however, are not  measuring either of these behavioral outcomes.
In addition, Education has not established any performance measures for  socio-emotional outcomes, although social skills are also included in  program objectives, and socio-emotional learning is an important  component of 21st Century implementation across states we visited and  surveyed. According to Education’s website, socio-emotional learning  involves students’ knowledge and skills necessary to understand and  manage emotions and establish positive relationships, among other  things. Twenty-seven states reported in our survey that they currently  measure or are developing measures for at least one socio-emotional  outcome, including student relationships with adults or communication  skills. Again, the remaining states are not measuring any of these socio- emotional outcomes. At the centers we visited, programs offered various  activities geared toward socio-emotional learning, including activities that  paired students with adult mentors and developed students’ problem- solving skills. Officials in two states we visited told us that socio-emotional  learning is a major component of their 21st Century programs’  philosophy. (See figure 6 for a comparison of 21st Century program  objectives and Education’s program performance measures).
Several factors contributed to Education’s decision to maintain its current  21st Century performance measures for over 15 years. In particular,  Education officials told us they have not substantially revised  performance measures since 1998 in part because the program’s  authorization lapsed from 2008 through 2016, with ESSA providing  funding authorization starting in fiscal year 2017. As a result, the officials  said this created uncertainty about potential future program changes. In  addition, they said they were reluctant to revise the measures given the  costs to states of tracking and reporting on new measures, as well as  concerns about maintaining the continuity of data collected over time.  Education officials told us they may revise the program’s performance  measures when they implement program changes pursuant to ESSA and  that in doing so, they will consider including additional behavioral  performance measures as well as socio-emotional measures.
Education’s lack of 21st Century performance measures for some key  program objectives, however, has limited the usefulness of its  performance data. Specifically, without measures for student behavioral  and socio-emotional outcomes, Education lacks useful data on the extent  to which the program is achieving its stated objectives, especially in areas  where the program is likely to have positive effects on student outcomes.  While about half of states reported they already measure at least one  behavioral or socio-emotional outcome, absent measures for all states,  Education will continue to report incomplete program results to Congress  and the public. Further, leading practices in performance management  call for agencies to use performance information as the basis for decision  making. These practices also state that aligning performance measures  with program objectives can enhance the use of information for  management decision making.
section_title: Education Does Not Have Reasonable Assurance That 21st Century Program Data Submitted by States Are Accurate
paragraphs: While Education has taken steps to promote data quality, including data  accuracy and completeness, it lacks reasonable assurance that the data  submitted by states are accurate. Education officials told us they began  using a new online data system in school year 2014-2015, called 21APR,  in part to improve data quality. Among other changes, 21APR does not  permit users to advance to the next page until the data on the current  page are complete. Education officials said that this can help prevent  missing data—a feature that was absent in the previous data system. In  addition, in 2016 Education provided states with regional training  sessions and instructional documents for data entry. However, Education  has not independently assessed the accuracy of 21st Century data  submitted in 21APR.
Education officials told us they require states to check their data  submissions rather than Education conducting these checks because the  agency does not have access to sub-grantees’ and centers’ source  information. Education’s own Information Quality Guidelines state that  data should be processed and edited to help ensure they are accurate.  Performing accuracy checks would not necessarily require that Education  have access to the source data. For example, it could perform basic logic  checks between fields—such as verifying that the number of student  participants in one grade is fewer than the total number of participants in  all grades. Education officials told us they may explore additional checks  for accuracy, but they did not provide a timeframe for doing so.
Education officials do review states’ procedures for data quality  assurance, although some states have expressed concerns about the  quality of data they receive from sub-grantees. Since 2015, when  Education implemented new monitoring protocols that included a review  of states’ data quality procedures, Education officials have monitored 20  states and have not found any states that are noncompliant. However, in  our interviews and survey responses, officials from 10 states commented  that they had concerns about the quality of program data they collect from  sub-grantees. For example, states are unable to upload data to the  21APR system, unlike in Education’s previous data system, and several  states expressed concerns that they must enter data manually. One state  official commented that this can increase the risk of errors.
Without independently checking accuracy, Education may be unable to  detect and address potential data errors in the 21APR system—data that  Education eventually needs to use to conduct analyses that inform  management decisions, identify sub-grantees needing technical  assistance, and contribute to annual performance reports and budget  justifications. Without reasonable assurance that data in the 21APR  system are of sufficient quality, Education may not be able to effectively  use the data for such decision making and reporting going forward,  despite the fact that the new system was created, in part, to improve data  quality. Further, absent better information on the quality of its data,  Education will be unable to communicate any data limitations in its reports  to Congress and the public. Our prior work has identified leading  practices in performance management, which state that agencies should  disclose limitations associated with data used in reporting and indicate  what action they plan to take to address the limitations.
section_title: Education’s Technical Assistance Does Not Yet Adequately Address Many States’ Challenges in Evaluating and Sustaining Programs States Reported Mixed Reviews on Education’s Technical Assistance for Evaluating their 21st Century Programs
paragraphs: States in our survey gave mixed reviews on the usefulness of Education’s  technical assistance for developing and conducting state-level  evaluations. Twenty-nine states reported that they found Education’s  technical assistance on conducting state evaluations to be very or  moderately useful. On the other hand, 12 states reported they found it  only slightly or not at all useful; 8 states reported they had not received  any technical assistance on this topic; and one state said it was unaware  that Education provided such technical assistance.
Further, 40 states commented in our survey that they face challenges in  evaluating sub-grantee performance that may limit their capacity to  conduct high-quality evaluations. For example, six states reported they  have difficulty designing evaluations that can determine whether a  change in student outcomes results from a student’s participation in a  21st Century program or from other factors, such as interventions during  the school day. This difficulty is evidenced by the fact that we identified  very few state evaluations whose evidence standards allowed us to  include them in our review on program effectiveness, as noted above. In  addition, four states reported that they face challenges in defining and  establishing performance measures for students who participate in the  program, such as measures for behavioral and academic outcomes. For  example, one state commented that it is difficult to measure the benefits  of enrichment activities on academic performance.
Education also found that states are having difficulty conducting their  required evaluations when Education reviewed them as part of its  monitoring process. For example, in recent monitoring findings, Education  officials said they found that five states did not have plans in place to  conduct these evaluations. In addition, according to technical assistance  reports provided as a part of the monitoring process, three states had  difficulty conducting comprehensive evaluations such as establishing the  appropriate scope of work and timeframes for conducting evaluations and  developing appropriate performance measurement tools.
According to Education officials, it has not provided written guidance to  states on 21st Century program evaluations since 1999, when it provided  states non-regulatory guidance in the form of a guide to help them  evaluate their programs and use the results to make program  improvements. However, since 1999, there have been significant  changes in the 21st Century program. Specifically, the program was  reauthorized twice, resulting in changes to requirements for measuring  and evaluating program performance. First, the 2002 reauthorization  added new requirements for states to describe in their funding  applications how the state will evaluate the effectiveness of programs,  including developing performance indicators and measures to evaluate  programs and activities. It also required sub-grantees to conduct  periodic local evaluations to assess progress toward achieving goals.  Second, as a result of requirements under ESSA, which reauthorized the  21st Century program in 2015, additional changes are planned starting in  school year 2017-2018. Specifically, states will have to evaluate their  programs in conjunction with new data collection and evaluation planning  requirements, including requirements to track student progress over time  and to include state standardized test scores and other indicators of  student success, such as improved school-day attendance.
Federal standards for internal control state that when significant changes  occur in how an agency achieves its objectives it should periodically  review policies and procedures for continued relevance and effectiveness  in achieving its objectives or addressing related risks. Further, it states  that written policies and procedures can help ensure that necessary  actions are taken to address risks to achieving the entity’s objectives. In  addition, our prior work has identified several leading practices in  agencies’ use of evaluations, which state that agencies should promote  capacity building to evaluate program activities as a key strategy for  supporting objectives. Education officials told us they are developing a  technical assistance plan to improve the support the agency provides to  states to implement the 21st Century program. These officials told us they  are considering the topics and areas of concern that they will address  including developing and conducting effective evaluations and  assessments, and they expect to finalize the list of topics in summer  2017. However, Education officials told us that the agency has not  determined what type of technical assistance it will provide to states on  conducting evaluations. Absent new written, non-regulatory guidance to  states that addresses the areas in which they struggle most, Education  may miss opportunities to help states improve their capacity to conduct  high-quality evaluations and ultimately improve their programs.
subsections: section_title: Education’s Technical Assistance Does Not Effectively Address States’ Difficulty in Sustaining 21st Century Programs
paragraphs: Education’s technical assistance to states does not effectively address  the challenges most states face in helping their 21st Century sub- grantees continue to operate their programs once grant funding ends,  according to state and sub-grantee officials. In our survey, 35 states  reported that their centers often face challenges providing the same level  of services to students when the grant funding ends. Officials in 20 states  reported that centers in their states generally reduce the level of services  or cease to operate once 21st Century funding ends. Two states  commented that less than 10 percent of centers had been able to  continue operating after the 21st Century grants expired. These concerns  were echoed by officials at 10 of the 13 21st Century centers we visited,  who told us they had major concerns about sustaining their programs.
Twenty centers that commented they will be unable to sustain operations  or maintain the same level of services after 21st Century funding ends  had difficulty securing funds from other sources. For example, officials in one state commented that it is often difficult for centers to obtain funding  from local school districts because district budgets are already strained by  supporting school-day programs. Officials in three other states  commented that there is no state funding dedicated exclusively to  programs outside the regular school day.
By law, states must require sub-grant applicants to submit a preliminary  plan for how their programs will continue to operate once grant funding  ends. Education’s monitoring protocols call for Education officials to ask  states if applicants have sustainability plans and how states monitor sub- grantees’ implementation of the plans. A majority of states (42 states)  reported in our survey that they require sub-grantees to provide a written  sustainability plan, while 6 states reported they do not. However, in its  monitoring efforts, Education told us it did not identify any states that are  currently out of compliance with this requirement.
Education may be missing opportunities in its monitoring efforts to collect  information on states’ strategies and practices for program  sustainability—information that could be useful for sharing promising  practices across states. As a part of its monitoring process, Education  officials told us they discuss with state officials how they ensure  compliance with the requirement to have such sustainability plans in  place, but they do examine the quality of the plans. Officials in 11 states  in our survey commented they would benefit from opportunities to  collaborate across states. States do appear to have valuable information  on sustainability to share with each other. For example, 25 states—in our  survey of all 50 states and the District of Columbia—reported providing  additional points or credit based on applicants’ ability to leverage external  funds and sustain programs when selecting sub-grantees. Further, in the  states that collect information on what happens after 21st Century funding  ends, most reported centers having some success in finding private or  nonprofit funds to help replace Education’s grant funds. States reported  that the most common sources of replacement funding were student fees  charged to participants (17 states), private foundation funding (14 states),  and non-profit funding (e.g., universities and community organizations)  (13 states).
Education officials told us they provide technical assistance on 21st  Century program sustainability through interactions such as an annual  summer training conference on the program and an online learning  portal. However, Education’s July 2016 training conference—a forum  where 21st Century participants share experiences and best practices— was generally not focused on state practices and policies. The sessions  were generally focused on centers’ collaboration with the school or  community, centers’ activities (e.g., literacy, science, and math), or how to  report performance data. Officials in about a third of states reported in our  survey that they did not know about or had no basis to judge Education’s  technical assistance on sustainability. Another third of states reported that  Education’s technical assistance in this area was slightly or not at all  useful. For example, two states reported that the content of Education’s  professional development, training, and other assistance is heavily  geared toward sub-grantees.
Education officials said they do have other interactions and forums, such  as regional meetings, to share information with states. For example,  Education started in-depth training in 2016 for state-level officials at four  regional meetings for the Midwest, Northwest, South, and East, but the  sessions did not cover program sustainability.
Federal standards for internal control state that information should be  communicated in a form that enables an agency to achieve its  objectives. In addition, our prior work has identified leading practices in  federal collaboration that have shown that information sharing among  grant participants, such as states, is important for effective grants  management. Education’s efforts to help facilitate information sharing  among states can help states identify practices that can be tailored to  meet their individual needs and leverage their knowledge to address  common challenges in continuing their programs over the long-term.
Unless Education undertakes such efforts for the 21st Century program,  students and families who participate may be at greater risk of not  receiving the full range program benefits for the longest amount of time.
section_title: Conclusions
paragraphs: The 21st Century program is designed to fund programs outside the  regular school day to improve academic and behavioral outcomes for K- 12 students who are from low-income families or who attend low- performing schools. High-quality research on the effectiveness of this  program is very limited due to several factors, including difficulty  determining whether a change in students’ outcomes results from their  participation in a 21st Century program or from other factors, such as  interventions during the school day. Although existing research on  effectiveness points to greater positive behavioral effects than academic  effects, Education’s current performance measures do not address some  key behavioral outcomes. The lack of performances for behavioral  outcomes makes it difficult to determine whether the program is achieving  some of its stated objectives—especially in areas where research has  shown that the program is likely to have the greatest effect on student  outcomes. Specifically, Education does not measure socio-emotional  outcomes or two other behavioral outcomes included in program  objectives: improved school-day attendance and discipline in the  classroom. Absent these performance measures, Education is missing an  opportunity to assess the full range of benefits of this program.
In addition, Education has not sufficiently assessed the quality of its  program data, further limiting its ability to assess program effectiveness.  Another factor hindering the agency’s ability to determine program  effectiveness is the significant difficulty that states are experiencing in  evaluating their programs. Although Education officials told us they are  considering topics and areas of concern for additional technical  assistance, Education has not yet provided states with sufficient guidance  on developing rigorous evaluations. Unless Education takes steps to  reasonably ensure the accuracy of data and provides written guidance to  help states develop high-quality evaluations, it cannot ensure the program  is effectively meeting its goals.
Lastly, states are experiencing substantial difficulty in sustaining their  programs after 21st Century funding ends, and many states expressed  interest in collaborating with other states to address such challenges.  Education is uniquely situated to take the lead in sharing information with  states to help them address their sustainability challenges by allowing  them to identify state policies and practices that have had some success.
Sharing such information could help states address the challenges they  face in continuing their programs over the long-term.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Education direct the Office of  Academic Improvement to:  1.  Expand its performance measures for the 21st Century program to  address all program objectives. Specifically, Education should  establish performance measures related to key behavioral, including  student attendance and disciplinary incidents, and socio-emotional  outcomes.  2.  Conduct federal-level data checks on the accuracy of 21st Century  program data submitted by states. Such checks could test for logical  relationships between fields. Education should also publicly disclose  and address any data limitations it identifies, as appropriate.  3.  Provide written, non-regulatory guidance to states on developing and  conducting high-quality 21st Century state evaluations to help address  the difficulties states face in measuring program performance and  effectiveness.  4.  Use the information it collects from its monitoring visits and ongoing  interactions with states to share effective practices across states for  sustaining their 21st Century programs once program funding ends.  This information could be shared using existing mechanisms such as  Education’s meetings with 21st Century state coordinators.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Education for  comment, and its written comments are reproduced in appendix III.   Education also provided technical comments that we incorporated in the  report as appropriate. Education neither agreed nor disagreed with our  recommendations; rather, it generally noted that it will keep our  recommendations in mind as it continues to implement changes in the  program as a result of ESSA, and outlined steps it will take to address our  recommendations.
In response to our recommendation that it expand its performance  measures for the 21st Century program to address all program objectives,  including those related to behavioral and socio-emotional outcomes,  Education said it will keep our recommendation in mind. Specifically, the  department stated that it is in the process of re-examining whether  additional or revised measures should be developed to align more  significantly with the program’s statutory objectives under ESSA.   Education also acknowledged the need to develop measures that provide  sufficient information and data on program outcomes. As stated in our  draft report, the Department currently measures some student behavioral  outcomes; in its response, Education described one measure as: the  percentage of all program participants with teacher-reported  improvements in behavior. Therefore, we modified our recommendation  to focus on measuring the program’s key behavioral outcomes. In its  response, Education also expressed concern about collecting data on  student attendance and disciplinary measures, noting that it will require  effective collaboration between states, districts, and other eligible entities.  However, as we stated in our report, about half of states already collect  data on at least one of these two measures. Further, as we stated,  research has shown that 21st Century programs more often have positive  effects on student attendance and reducing disciplinary incidents than on  improving students’ academic outcomes. Given these effects, we  continue to believe that it is critical for Education to measure student  attendance and disciplinary incidents to obtain more complete, accurate  information on this program’s effect on student outcomes.
In response to our recommendation that it conduct federal-level data  checks on the accuracy of 21st Century program data submitted by  states, Education commented that it plans to build in additional data  checks into the data system beyond its current checks on the data’s  completeness. Specifically, Education anticipates that new technology  enhancements in the data system will be designed to flag for  inconsistencies in data reporting. For example, the system may send a  “flag” that participation data is significantly lower or higher than previously  reported participation data. Further, Education indicated that it will  consider whether auditors performing audits under the Single Audit Act  can be asked and guided to do more checks on the accuracy and  reliability of 21st Century program data.
Regarding our recommendation to provide written, non-regulatory  guidance to states on developing and conducting high-quality 21st  Century state evaluations, Education outlined several steps it has taken  to assist states in the past. For example, Education said that it provided  six states with individualized technical assistance on strategies related to  developing statewide evaluations and measures. Education also noted  that, to date, it has conducted two webinars on state evaluations and is in  the process of including presentations from those webinars on its online  learning portal so that states will have easy access to the information. In  addition, Education stated that it included presentations on evaluation  strategies in the past during its Summer Institute. Education also said it  would consider whether additional guidance for all states was needed.  While these are important steps, we do not believe they are sufficient.  Twenty-one of 51 states (41 percent of states) reported in our 2016  survey that they found Education’s technical assistance for developing  and conducting state-level evaluations only slightly or not at all useful;  they had not received any technical assistance on this topic; or they were  unaware that Education provided such assistance. Therefore, we  continue to believe that Education should prepare written guidance to  assist all states in developing and conducting high-quality program  evaluations.
Finally, regarding our recommendation to share effective practices across  states for sustaining their 21st Century programs once program funding  ends, Education stated that it hosts meetings twice a year for 21st  Century state coordinators. At these meetings, Education officials share  strategies with states related to program sustainability. Education stated  that these meetings covered topics such as reducing the amounts of 21st  Century grant awards by a percentage each year. However, Education  officials told us in February 2017 that these meetings have not focused on  topics on program sustainability for several years. We continue to believe  that Education should take the lead in sharing information with states to  help them address their sustainability challenges by sharing information  on state policies and practices that have shown some success. In its  comments, Education stated that it has not held regional meetings with  states for several years; however, in 2016 it held four regional meetings  with states on implementing its new 21APR data system. Therefore, we  modified our recommendation to emphasize that such information be  shared through these types of existing mechanisms.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Education, and other interested parties. In  addition, the report will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff should have any questions about this report, please  contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for  our Offices of Congressional Relations nowickij@gao.gov Public Affairs  may be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Our study of the 21st Century program was framed around four  objectives: (1) how 21st Century funds are awarded and used, (2) what is  known about the effectiveness of 21st Century programs, (3) the extent to  which Education has effectively managed and used program data to  inform decisions about the 21st Century program, and (4) the extent to  which Education’s technical assistance has helped states assess and  sustain high-quality programs once 21st Century funding ends. We  focused our review on 21st Century afterschool programs because  centers have historically provided services more during this time as  compared to before school and during the summer. To address our four  objectives, we used a variety of methods, including a Web-based survey  of 21st Century state coordinators; a review of selected state evaluations  and academic literature; a review of federal laws, regulations, and agency  documents such as annual performance reports and guidance; interviews  with federal and other officials; and site visits to four states.
subsections: section_title: Survey of State Coordinators
paragraphs: To obtain information about the state sub-grantee award process,  program sustainability and technical assistance of the 21st Century  program, we conducted a Web-based survey of the 21st Century state  coordinator at each state educational agency in all 50 grantee states and  the District of Columbia. We conducted the survey from August 2016  through November 2016. The survey covered several topics, including  state processes for awarding 21st Century sub-grants to local  communities, state-level program evaluations, performance measures,  and sub-grantees’ financial sustainability, among other things. We  received responses from all 50 states and the District of Columbia for a  100 percent response rate. The survey included an introductory  statement specifying that the survey focused on afterschool programs  funded by the 21st Century program and collects information about  federal-level guidance provided to state officials on this program.
The quality of survey data can be affected by nonsampling error, which  includes variations in how respondents interpret questions, respondents’  willingness to offer accurate responses, and data collection and  processing errors. To minimize such error, we included the following  steps in developing the survey and in collecting and analyzing survey  data: in developing the Web survey, we pre-tested draft versions of the  instrument with 21st Century program officials in four states to check the  clarity of the questions and the flow and layout of the survey. On the basis  of the pretests, we made revisions to the survey. Further, using a Web- based survey and allowing 21st Century coordinators to enter their  responses directly into an electronic instrument created an automatic  record for each state in a data file and eliminated the errors associated  with a manual data entry process. In addition, the program used to  analyze the survey data was independently verified to ensure the  accuracy of this work.
section_title: Review of State Evaluations and Academic Literature
paragraphs: In order to examine what is known about the effectiveness of 21st  Century programs, we reviewed select state-level program evaluations  and academic studies that reported outcomes for students who  participated in the program. To identify relevant research, we took a two- pronged approach. First, we reviewed three state evaluations that  Education determined in a 2012 report on 21st Century program grantee  evaluation practices (the most recent available Education report on this  topic) to have used quasi-experimental methods or statistical controls,  such as a comparison group, to account for other plausible influences on  the outcomes reported. We retrieved the most recent evaluations from  those three states and through interviews with researchers in the field; we  identified three other quasi-experimental state evaluations published  since 2012. We reviewed each of these state evaluations for the  soundness of its methodology and data. In four cases where the  published evaluation did not include enough information to complete our  review, we contacted the states and the researchers to obtain additional  data from their samples. Ultimately, we determined that four of the six  state evaluations were appropriate for purposes of our research objective  about the 21st Century program’s effectiveness.
Second, we conducted a comprehensive literature review of 104  academic studies of afterschool programs, including the 21st Century  program, and we ultimately determined that six of these academic studies  were sufficiently rigorous and appropriately scoped to include in our  review. To identify academic studies on the effectiveness of 21st Century  programs, we conducted a literature search through ProQuest. Our initial  search terms were adapted from a 2014 report produced by Education  that looked at the effect of 21st Century programs, along with other  programs that increased the amount of time that children spent in school.  Because our review was focused on afterschool programs, rather than  the broader range of out of school time or extended learning time  programs covered in that study, we only searched for studies of  afterschool programs and studies of 21st Century programs particularly.
We identified 104 papers initially and selected 25 of them on the basis of  the following criteria:
Topic relevance–the studies covered 21st Century programs or  afterschool programs that measured student outcomes and had some  instructional component (i.e. programs were not only a study hall or a  sports program).
Timeframe relevance–studies must have been published since 2012,  in order to find studies that may not have been included in Education’s  2014 study, which examined studies published since 1998.
Publication status–studies were in their final form, not drafts.
Sample relevance–studies looked at afterschool programs in K-12  settings in the United States. Foreign school systems were excluded.
Design relevance–studies used experimental or quasi-experimental  designs with well-formulated comparison groups or controls; or they  used some statistical method to account for other plausible influences  on the outcomes of the study.
Next, two analysts reviewed those studies a second time, retaining 13  studies that had a sample size above 100 students in the intervention  group and evaluated more than one program or intervention site. The  analysts further narrowed this sample to six studies, which we selected  because they were focused on 21st Century programs. In addition to  reviewing Education’s 2014 meta-analysis, in our initial search of 104  papers, the analysts also identified two additional meta-analyses focused  on afterschool programs, which we included in our sample because they  were of appropriate quality and scope to assess afterschool program  effects. We then conducted detailed reviews of the studies. These  reviews entailed an evaluation of each study’s research methodology,  including its research design, sampling frame, selection of measures,  data quality, limitations, and analytic techniques, as well as a summary of  its major findings. We also assessed the extent to which each study was  relevant to assessing what is known about the effectiveness of 21st  Century programs. Three studies had major research design limitations  resulting from the lack of a rigorously formed comparison group. As such,  these studies were not able to demonstrate whether the programs were  responsible for the effects being measured or whether other factors may  have contributed. After eliminating the three studies with major research  design limitations, six studies remained in our review.
section_title: Review of Agency Documents and Interviews with Agency Officials
paragraphs: For all four objectives, we reviewed agency documents and interviewed  agency officials and researchers who had conducted work on the 21st  Century program. To examine how states and other entities have used  21st Century funds, we reviewed Education’s 2014 annual characteristic  report for the 21st Century program. The reports provide summaries and  analyses over time, based on program data collected from states in  school years 2001-2002 through 2012-2013 using data from the Profile  and Performance Information Collection System. In order to assess the  reliability of the data underlying the reports, we reviewed agency  documents regarding this data system, spoke with agency officials, and  conducted electronic tests of data from the system for select data  elements. We determined that the underlying data summarized and  analyzed in Education’s 2014 annual characteristic report was sufficiently  reliable for our purpose of reporting on program characteristics.
To examine Education’s management of program data, we also reviewed  Education documents including budget justifications and documents on  the 21st Century data systems such as user guides, technical assistance  documents for states, data dictionaries, and a 2013 Education Office of  Inspector General report on federal and state oversight for the 21st  Century program.
To examine Education’s technical assistance, we reviewed Education’s  program guidance to states, monitoring protocols and tools, and annual  grantee satisfaction survey results which generally includes Education’s  largest grant programs including the 21st Century program. In order to  compare the program to established criteria, we reviewed relevant  legislation; Office of Management and Budget guidance; federal internal  control standards; and past GAO work pertaining to leading practices in  performance management, information quality, program evaluation, and  federal collaboration for grants management.
We interviewed federal officials from Education’s Office of Academic  Improvement, within the Office of Elementary and Secondary Education,  regarding data management and use, research and evaluation, and  program challenges. In addition, we interviewed stakeholders of the 21st  Century program, including contractors and researchers. Finally, we  observed Education’s annual Summer Institute, where Education  provides technical assistance and training to state, sub-grantee, and  center officials.
section_title: Site Visits
paragraphs: To learn about program activities and challenges, we conducted site visits  to four states—Idaho, Massachusetts, Rhode Island, and Texas—to visit  21st Century centers and speak with program staff and state officials. We  selected these states for diversity in geographic region, 21st Century  formula grant funding levels to states, and student demographics. In each  state, we interviewed the 21st Century state coordinator and other  officials responsible for the program. In total, we visited 13 centers in the  four states. During these visits, we interviewed center staff and observed  afterschool program activities.
Information we gathered on our site visits represents only the conditions  present in the states and local areas at the time of our visits. Furthermore,  our fieldwork focused on in-depth analysis of only a few selected states.  On the basis of our site visit information, we cannot generalize our  findings beyond the states we visited.
We conducted this performance audit from April 2016 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: 21st Century Grant Award Amount by States, Fiscal Year 2016
section_title: Appendix III: Comments from the U.S. Department of Education
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Sirois (Assistant  Director), Sheranda Campbell (Analyst-in-Charge), Lucas Alvarez, Ashley  Chaifetz, Jamila Kennedy, and Kelsey Kennedy made significant  contributions to this report. Assistance, expertise, and guidance were  provided by Susan Aschoff, Carl Barden, Deborah Bland, James Bennett,  Angie Jacobs, Thomas James, Kirsten Lauber, Ben Licht, Edward  Malone, John Mingus, Amy Moran Lowe, Sara Pelton and James Rebbe.