Science, Technology, Engineering, and Mathematics Education: Governmentwide Strategy Needed to Better Manage Overlapping Programs
Released: 2013-04-10T00:00:00

Letter

Background

Since 2005, there have been several efforts to inventory federal STEM education programs and reports that call for the need to better coordinate and evaluate STEM education programs. In 2005, for example, GAO identified a multitude of agencies that administer such programs. The primary missions of these agencies vary, but most often, they are to promote and enhance an area that is related to a STEM field or enhance general education. In addition, the National Science and Technology Council (NSTC) was established in 1993 and is the principal means for the administration to coordinate science and technology with the federal government’s larger research and development effort. The America COMPETES Reauthorization Act of 2010 sought to address coordination and oversight issues, including those associated with the coordination and potential duplication of federal STEM education efforts.the law required the Director of the Office for Science and Technology Policy (OSTP) to establish a committee under the NSTC to inventory, review, and coordinate federal STEM education programs. The law also directed this NSTC committee to develop a 5-year governmentwide STEM education strategic plan, which must specify and prioritize annual and long-term objectives for STEM education. Moreover, the Director of OSTP is required to send a report to Congress annually on this strategic plan, which must include, among other things, an evaluation of the levels of duplication and fragmentation of STEM programs and activities.

In our January 2012 report on STEM education, we defined a federally funded STEM education program as a program funded in fiscal year 2010 by congressional appropriation or allocation that included one or more of the following as a primary objective: attract or prepare students to pursue classes or coursework in STEM areas through formal or informal education activities, attract students to pursue degrees (2-year, 4-year, graduate, or doctoral degrees) in STEM fields through formal or informal education activities, provide training opportunities for undergraduate or graduate students attract graduates to pursue careers in STEM fields, improve teacher (preservice or in-service) education in STEM areas, improve or expand the capacity of K-12 schools or postsecondary institutions to promote or foster education in STEM fields, or conduct research to enhance the quality of STEM education programs provided to students.

In addition, a program was defined as an organized set of activities supported by a congressional appropriation or allocation. Further, we defined a program as a single program even when its funds were allocated to other programs as well. We asked agency officials to provide a list of programs that received funds in fiscal year 2010.

In our January 2012 report, we examined the extent to which federal STEM education programs were fragmented, overlapping, and duplicative. overlap, and duplication work, key terms were defined as follows: Using our framework established in previous fragmentation,

Fragmentation occurs when more than one federal agency (or more than one organization within an agency) is involved in the same broad area of national need.

Overlap occurs when multiple programs offer similar services to similar target groups in similar STEM fields to achieve similar objectives.

Duplication occurs when multiple programs offer the same services to the same target beneficiaries in the same STEM fields.

GAO-12-108.

Thirteen Federal Agencies Administered over 200 Programs with Over $3 Billion in Obligated Funds

As we reported in 2012, 13 agencies administered 209 STEM education programs in fiscal year 2010. Agencies reported that they developed the majority (130) of these programs through their general statutory authority and that Congress specifically directed agencies to create 59 of these programs. The number of programs each agency administered ranged from 3 to 46 with three agencies—the Department of Health and Human Services, the Department of Energy, and the National Science Foundation (NSF)—administering more than half of all programs—112 of 209. (See fig. 1) Agencies obligated over $3 billion to STEM education programs in fiscal year 2010, ranging from $15,000 to hundreds of millions of dollars per program. NSF and the Department of Education programs accounted for over half of this funding. Almost a third of the programs had obligations of $1 million or less, with five programs having obligations of more than $100 million each.

Beyond the 209 programs identified in our review, federal agencies carried out other activities that contribute to the overall federal STEM education effort. Having multiple agencies, with varying expertise, involved in delivering STEM education has both advantages and disadvantages. On the one hand, it could allow agencies to tailor programs to suit their specific missions and needs to attract new employees to their workforce. On the other hand, it could make it challenging to develop a coherent federal approach to educating STEM students and creating a workforce with STEM skills. Further, it could make it difficult to identify gaps and allocate resources across the federal government.

Most STEM Programs Overlapped to Some Degree

As we reported in 2012, and as figure 2 illustrates, in fiscal year 2010, 83 percent of STEM education programs overlapped to some degree with another program in that they offered at least one similar service to at least one similar target group in at least one similar STEM field to achieve at least one similar objective. These programs ranged from being narrowly focused on a specific group or field of study to offering a range of services to students and teachers across STEM fields. This complicated patchwork of overlapping programs has largely resulted from federal efforts to both create and expand programs across many agencies in an effort to improve STEM education and increase the number of students going into STEM fields. Program officials reported that approximately one-third of STEM education programs funded in fiscal year 2010 were first funded between 2005 and 2010. We believe the creation of new programs during that time frame may have contributed to overlap and, ultimately, to inefficiencies in how STEM programs across the federal government are focused and delivered. Overlap among STEM education programs is not new. In 2007, the Academic Competitiveness Council (ACC) identified extensive overlap among STEM education programs, and, in 2009, we identified overlap among teacher quality programs, which include several programs focused on STEM education. Overlapping programs can lead to individuals and institutions being eligible for similar services in similar STEM fields offered through multiple programs and, without information sharing, could lead to the same service being provided to the same individual or institution.

Our analysis found that many programs provided services to similar target groups, such as K-12 students, postsecondary students, K-12 teachers, and college faculty and staff. The vast majority of programs (170) served postsecondary students. Ninety-five programs served college faculty and staff, 75 programs served K-12 students, and 70 programs served K-12 teachers. In addition, many programs served multiple target groups. In fact, 177 programs primarily served two or more target groups.

We also found many STEM programs providing similar services.

To support students, 167 different programs provided research opportunities, internships, mentorships, or career guidance. In addition, 144 programs provided short-term experiential learning opportunities and 127 long-term experiential learning opportunities. Short-term experiential learning activities included field trips, guest speakers, workshops, and summer camps. Long-term experiential learning activities last a semester in length or longer. Furthermore, 137 programs provided outreach and recognition to generate student interest, 124 provided classroom instruction, and 75 provided student scholarships or fellowships.

To support teachers, 115 programs provided curriculum development, 83 programs provided teacher in-service, professional development, or retention activities, and 52 programs provided preservice or recruitment activities.

To support STEM research, 68 programs reported conducting research to enhance the quality of STEM education.

To support institutions, 65 programs provided institutional support to management and administrative activities, and 46 programs provided support for expanding the facilities, classrooms, and other physical infrastructure of institutions.

In addition to serving multiple target groups, our analysis found that most programs also provided services in multiple STEM fields. Twenty-three programs targeted one specific STEM field, while 121 programs targeted four or more specific STEM fields. In addition, 26 programs indicated not focusing on any specific STEM field, and instead provided services eligible for use in any STEM field. Five different STEM fields had over 100 programs that provided services. Biological sciences and technology were the most selected STEM fields that programs focused on. Agricultural sciences, which was the least commonly selected, still had 27 programs that provided services specifically to that STEM field.

While our 2011 survey data also show that many programs overlapped, it is important to compare programs’ target groups and STEM fields of focus to get a better picture of the potential target beneficiaries that could be served within a given STEM discipline. For example, both the National Oceanic and Atmospheric Administration’s National Environmental Satellite, Data, and Information Service (NESDIS) Education program and the Department of Energy’s Graduate Automotive Technology Education program provided scholarships or fellowships to postsecondary students, but NEDSIS focused on students in earth sciences programs, and the other on engineering; therefore, the target beneficiaries served by these two similar programs are quite different. Nevertheless, we found that 76 programs served postsecondary students in physics. As table 1 illustrates, many programs offered services to similar target groups in similar STEM fields of focus.

We also found that many STEM education programs had similar objectives. In response to our 2011 survey, the vast majority (87 percent) of STEM education program officials indicated that attracting and preparing students throughout their academic careers in STEM areas was a primary objective. In addition to attracting and preparing students throughout their academic careers in STEM areas, officials also indicated the following primary program objectives: improving teacher education in STEM areas (teacher development)— 26 percent, improving or expanding the capacity of K-12 schools or postsecondary institutions to promote or foster education in STEM fields (institution capacity building)—24 percent, and conducting research to enhance the quality of STEM education provided to students (STEM education research)—18 percent.

Many programs also reported having multiple primary objectives. While 107 programs focused solely on student education, 82 others indicated having multiple primary objectives, and 9 programs reported having 4 or more primary objectives. Few programs reported focusing solely on teacher development, institution capacity building, or STEM education research. Most of these objectives were part of a larger program that also focused on attracting and preparing students in STEM education.

However, even when programs overlapped, we found that the services they provided and the populations they served may differ in meaningful ways and would therefore not necessarily be duplicative. There may be important differences between the specific field(s) of focus and a program’s stated goals. For example, both Commerce’s National Estuarine Research Reserve System Education Program and the Nuclear Regulatory Commission’s Integrated University Program provided scholarships or fellowships to doctoral students in the field of physics; however, the Commerce program focuses on increasing environmental literacy related to estuaries and coastal watersheds, while the Nuclear Regulatory Commission program focuses on supporting education in nuclear science, engineering, and related fields with the goal of developing a workforce capable of designing, constructing, operating, and regulating nuclear facilities and capable of handling nuclear materials safely. In addition, programs may be primarily intended to serve different specific populations within a given target group. For example, of the 34 programs that we surveyed in 2011 that provided services to K-12 students in the field of technology, 10 were primarily intended to serve specific underrepresented, minority, or disadvantaged groups, and 2 were limited geographically to individual cities or universities. Furthermore, individuals may receive assistance from different programs at different points throughout their academic careers that provide services that complement or build upon each other, simultaneously supporting a common goal rather than serving cross purposes.

Limited Use of Performance Measures and Evaluations Hamper Ability to Assess Effectiveness

In 2012, we reported that in addition to the fragmented and overlapping nature of federal STEM education programs, agencies’ limited use of performance measures and evaluations may hamper their ability to assess the effectiveness of their individual programs as well as the overall STEM education effort. Understanding program performance and effectiveness is key in determining where to strategically invest limited federal funds to achieve the greatest impact in developing a pipeline of future workers in STEM fields. Program officials varied in their ability to provide reliable output measures—for example, the number of students, teachers, or institutions directly served by their program. In some cases, the program’s agency did not maintain databases or contracts that would track the number of students served by the program. In other cases, programs may not have been able to provide information on the numbers of institutions they served because they provided grants to secondary recipients. In 2012, we reported that the inconsistent collection of output measures across programs makes it challenging to aggregate the number of students, teachers, and institutions served and to assess the effectiveness of the overall federal effort.

In addition, most agencies did not use outcome measures in a way that is clearly reflected in their performance plans and reports—publicly available documents they use for performance planning. These documents typically lay out agency performance goals that establish the level of performance to be achieved by program activities during a given fiscal year, the measures developed to track progress, and what progress has been made toward meeting those performance goals. The lack of performance outcome measures may hinder decision makers’ ability to assess how agencies’ STEM education efforts contribute to agencywide performance goals and the overall federal STEM effort. For our 2012 report, we reviewed fiscal year 2010 annual performance plans and reports of the 13 agencies with STEM programs and found that most agencies did not connect STEM education activities to agency goals or measure and report on the progress of those activities.

We define “evaluation” as an individual systematic study conducted periodically or on an ad hoc basis to assess how well a program is working, typically relative to its program objectives.

Implementation of Our Recommendations Would Better Align Federal Efforts

In our January 2012 report, we made four recommendations to the Director of OSTP to direct the NSTC to: 1. Work with agencies, through its strategic-planning process, to identify programs that might be candidates for consolidation or elimination, which could be identified through an analysis that includes information on program overlap and program effectiveness. As part of this effort, OSTP should work with agency officials to identify and report any changes in statutory authority necessary to execute each specific program consolidation identified by NSTC’s strategic plan. 2. Develop guidance to help agencies determine the types of evaluations that may be feasible and appropriate for different types of STEM education programs and develop a mechanism for sharing this information across agencies. This step could include guidance and sharing of information that outlines practices for evaluating similar types of programs. 3. Develop guidance for how agencies can better incorporate each agency’s STEM education efforts and the goals from NSTC’s 5-year STEM education strategic plan into each agency’s own performance plans and reports. 4. Develop a framework for how agencies will be monitored to ensure that they are collecting and reporting on NSTC strategic plan goals. This framework should include alternatives for a sustained focus on monitoring coordination of STEM programs if the NSTC Committee on STEM terminates in 2015 as called for in its charter.

OSTP agreed with our conclusions and, as figure 4 shows, NSTC has made some progress in addressing recommendations from our January 2012 report.

Subsequently, OSTP has stated that NSTC’s 5-Year Federal STEM Education Strategic Plan, originally scheduled to be released in spring 2012, would address our recommendations; however, the release of NSTC’s Strategic Plan has been delayed. In February 2012, NSTC published Coordinating Federal Science, Technology, Engineering, and Mathematics (STEM) Education Investments: Progress Report, which identified a number of programs that could be eliminated in fiscal year 2013. By identifying programs for consolidation, elimination, and other actions, the administration could increase the efficient use of scarce government resources to achieve the greatest impact in developing a pipeline of future workers in STEM fields. Although NSTC said it planned to create a small working group to develop guidance on the appropriateness of different types of evaluations for different types of STEM education programs, OSTP has not released the findings of this working group. Agency and program officials would benefit from guidance and information sharing within and across agencies about what is working and how to best evaluate programs. This could help improve individual program performance and also inform agency and governmentwide decisions about which programs should continue to be funded. We continue to believe that without an understanding of what is working in some programs, it will be difficult to develop a clear strategy for how to spend limited federal funds. In addition, STEM education was named as an interim crosscutting priority goal in the President’s 2013 budget submission; however, it will be important for NSTC to finalize its strategic plan, which should include guidance for how agencies can better align their performance plans and reports to new governmentwide goals. Although OSTP agreed to develop milestones and metrics to track the implementation of NSTC strategic goals by each agency, it has not taken action to develop a framework for how agencies will be monitored to ensure that they are collecting and reporting on NSTC strategic plan goals. A framework for monitoring agency progress towards NSTC’s strategic plan is necessary to improve transparency and strengthen accountability of NSTC’s strategic planning and coordination efforts. In conclusion, if NSTC’s 5-year strategic plan is not developed in a way that aligns agencies’ efforts to achieve governmentwide goals, enhances the federal government’s ability to assess what works, and concentrates resources on those programs that advance the strategy, the federal government may spend limited funds in an inefficient and ineffective manner that does not best help to improve the nation’s global competitiveness.

Chairman Rokita, Ranking Member McCarthy, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to answer any questions that you may have at this time.

Contacts and Staff Acknowledgments

For further information about this testimony, please contact George A. Scott at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. Other key contributors to this testimony include: Bill Keller, Assistant Director; Susan Baxter; James Bennett; Karen Brown; David Chrisinger; Melinda Cordero; Elizabeth Curda; Karen Febey; Jill Lacey; Ben Licht; Dan Meyer; Amy Radovich; James Rebbe; Nyree Ryder Tee; Martin Scire; Ryan Siegel; and Walter Vance.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.