America COMPETES Acts: Overall Appropriations Have Increased and Have Mainly Funded Existing Federal Research Entities
Published: 2013-07-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Scientific and technological innovation and a workforce educated in STEM fields are critical to long-term U.S. economic competitiveness. Leaders in government, business, and education have expressed concern about the nation's ability to compete with other technologically advanced countries in these fields. In this context, Congress passed COMPETES 2007 and reauthorized the act with COMPETES 2010, each with the overall goal of investing in research and development to improve U.S. competitiveness. Among other things, the acts specifically authorized funding for certain programs.
COMPETES 2010 mandated GAO to evaluate the status of authorized programs. GAO examined (1) the extent to which funding was appropriated under the authorization of COMPETES 2007 and COMPETES 2010 and (2) what recent evaluations suggest about how programs for which the acts specifically authorized funding are working. To answer these questions, GAO reviewed relevant federal laws, interviewed agency officials, and reviewed program evaluations for quality and content.
This report contains no recommendations.
section_title: What GAO Found
paragraphs: In fiscal years 2008-2012, $52.4 billion was appropriated out of the $62.2 billion authorized under the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act of 2007 (COMPETES 2007) and the America COMPETES Reauthorization Act of 2010 (COMPETES 2010). Almost all of these funds went to the entire budgets of three existing research entities--the National Science Foundation (NSF), the National Institute of Standards and Technology (NIST), and the Department of Energy's (DOE) Office of Science (Science)--including all of the programs and activities the entities carry out. Appropriations for NSF, NIST, and Science generally increased under the acts but did not reach levels authorized by the acts. In addition to authorizing the budgets of these entities, COMPETES 2007 and COMPETES 2010 specifically authorized funding for 40 individual programs, including some programs within and some outside of these entities. Among those 40 programs, the 12 programs that existed before COMPETES 2007 received appropriations and continue to operate. Six of 28 newly authorized programs were also funded. Of these 6 programs, 1--DOE's Advanced Research Projects Agency-Energy, set up to develop new energy technologies--is continuing operations, 3 were not funded in fiscal year 2012, and 2 were not fully implemented as of May 2013. For the 22 programs that were not funded, agency officials generally said that they did not request funding in their budget submissions; most often this was because agencies had similar programs under way or could pursue similar objectives within current programs. For example, Science said it did not request funding for the Discovery Science and Engineering Innovation Institutes because it would have duplicated other Science programs.
For the fully implemented programs for which the COMPETES Acts specifically authorized funding, recent evaluations generally reported positive results, and some evaluations provided suggestions for improvements. Recent evaluations have been conducted for almost all of the programs that were implemented, or for aspects of those programs. For example, studies of the Robert Noyce Teacher Scholarship Program found that the program has increased the number of qualified science, technology, engineering, and mathematics (STEM) teachers, but also suggested that retention of teachers in high-need schools could be improved.

section_title: Letter
paragraphs: Scientific and technological innovation and a workforce educated in the  fields of science, technology, engineering, and mathematics (STEM) are  critical to long-term U.S. economic competitiveness. In recent years,  leaders in government, business, and education have expressed concern  that declines in federal funding for scientific research, coupled with a  shortage of qualified students and future workers in STEM fields, threaten  the nation’s ability to compete with other technologically advanced  countries. For example, according to a group of leaders gathered by the  National Academies in 2005 and 2010 to examine U.S. competitiveness  in the twenty-first century, scientific research and education are  necessary investments that will drive the economy and allow the United  States to maintain the economic strength to provide services such as  health care and national security in the current economic and fiscal  climate.
For decades, the federal government has invested in scientific research  at a number of agencies and departments, including the National Science  Foundation (NSF), the Department of Energy (DOE), and the Department  of Commerce’s (Commerce) National Institute of Standards and  Technology (NIST). The federal government also invests in STEM  education through the Department of Education (Education), NSF, DOE,  and other departments.
Congress passed the America Creating Opportunities to Meaningfully  Promote Excellence in Technology, Education, and Science Act of 2007  (COMPETES 2007) with the overall goal of investing in research and  development to improve U.S. competitiveness. The act also authorized  investments in education in STEM fields. In 2011, Congress passed the  COMPETES Reauthorization Act of 2010 (COMPETES 2010). Together,  these acts authorized $62.2 billion in funding from fiscal year 2008  through 2012. For fiscal year 2013, COMPETES 2010 authorized an  additional $16 billion, bringing the total amount authorized under the acts  to $78.2 billion. Both COMPETES 2007 and COMPETES 2010  (COMPETES Acts) authorized the entire budgets of three previously  existing federal research entities: NSF, DOE’s Office of Science  (Science), and NIST, including all activities and the many programs within  these entities. In addition, the acts specifically authorized funding for 40  individual programs, including some programs within and some outside of  NSF, Science, and NIST.
COMPETES 2010 directed us to evaluate the status of programs  authorized under the act, including the extent to which those programs  have been funded, implemented, and are contributing to achieving the  goals of the act. In response, we (1) determined the extent to which  funding was appropriated under the authorization of the COMPETES Acts  and (2) examined what recent evaluations suggest about how  implemented programs for which the acts specifically authorized funding  are working.
To conduct this work, we reviewed relevant laws and program  evaluations, and we interviewed agency officials from Commerce,  including NIST and the Economic Development Administration;  Education; DOE, including Science and the Advanced Research Projects  Agency–Energy;which the COMPETES Acts specifically authorized funding, we reviewed  the COMPETES Acts. To determine the extent to which funding was  appropriated to such entities and programs, we reviewed appropriations  data in Congressional Research Service reports and agency budget  justification documents, and we confirmed these data with agency  officials. Agency officials were not able to provide complete  appropriations data at the program level or final appropriations data for  fiscal year 2013. We also interviewed agency officials to learn about  program implementation. To examine, for the implemented programs for  which the COMPETES Acts specifically authorized funding, what recent  evaluations suggest about how the programs are working, we reviewed  evaluations published from 2008 through 2012 that were identified by  agencies and through a literature review; in some cases, these studies  included data on program activities that occurred before 2008. We  reviewed the methodology of the identified evaluations, and we reported  on the results of those we determined to be methodologically sound. (See  app. I for further details on our scope and methodology.)   and NSF. To identify the entities and programs for  We conducted this performance audit from October 2012 to July 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In response to a congressional request in 2005, the National Academies  gathered a group of business, government, and academic leaders to  identify steps the leaders thought would ensure that the United States is a  leader in science and engineering and can compete, prosper, and be  secure in the twenty-first century. The resulting 2007 report, entitled  Rising above the Gathering Storm: Energizing and Employing America for  a Brighter Economic Future, recommended a number of specific actions  to address these goals.increasing federal investment in long-term basic and cross-disciplinary  scientific research, creating an agency within DOE to support  transformational energy research that might be high risk but could also  provide dramatic benefits for the nation, increasing the number and skills  of science and mathematics teachers in primary and secondary schools,  and investing in higher education with the goal of increasing the number  of undergraduate and graduate students with degrees in science,  engineering, and mathematics fields. The COMPETES Acts addressed  some of the actions in these areas. For example, COMPETES 2007  authorized creation of the Advanced Research Projects Agency-Energy in  DOE to overcome long-term and high-risk technological barriers in  developing energy technologies, and it authorized programs in Education  and NSF to train teachers in STEM fields.
Among other things, the report advocated  Investments in scientific research have led to significant advances such  as the development of the Internet, satellites, aircraft, and the mapping of  the human genome, while investments in STEM education have provided  multiple forms of support for developing a highly qualified STEM  workforce. However, evaluations of such investments face inherent  challenges, such as those related to the long-term nature of many  scientific research projects, an inability to predict certain outcomes, and  difficulty tying specific investments to direct outcomes. As we reported in  2012, evaluations of STEM education programs may be hindered by  inconsistent collection of output data, such as the number of institutions  or students directly served by programs.effectiveness of investments in scientific research and STEM education in  improving U.S. competitiveness—the overall goal of the COMPETES  Acts—are complicated by a number of factors. For example, it is difficult  to measure competitiveness. The Council on Competitiveness, a group of  business, academic, and labor leaders focused on ensuring U.S.  prosperity, reported that traditional measures of competitiveness, such as  trade balances, levels of foreign direct investment, employment, or wages   Further, efforts to evaluate the  may not fully capture a nation’s competitiveness because of the  complexities brought about by multinational corporations competing in  constantly shifting global networks. Further, complications arise from  ambiguities surrounding the term competitiveness, which has multiple  definitions.
Efforts are under way to address some of these challenges. For example,  STAR Metrics—Science and Technology for America’s Reinvestment:  Measuring the Effect of Research on Innovation, Competitiveness, and  Science—is a partnership between science agencies and research  institutions to consistently document the outcomes of federally funded  science investments. In addition, the National Science and Technology  Committee on STEM Education within the Office of Science and  Technology Policy collects and maintains information on how investments  in STEM education are distributed across agencies, programs, and target  groups. The committee compiled a comprehensive inventory of STEM  education programs across the federal government in 2011 and released  a 5-year federal STEM education plan in May 2013 to establish a strategy  for focusing federal STEM education investments so they have the most  significant impact possible on national priorities.
As we reported in 2010 when we reviewed COMPETES 2007, agencies  collect data and use different approaches to evaluate their progress   NSF, DOE, Commerce, and Education use  toward long-term outcomes.several tools—which may either broadly assess agency-wide activities or  focus on program-level activities—to evaluate their scientific research and  STEM education activities. Such tools include the following: advisory committees, such as NSF’s committees that exist for each  directorate—for example, the Education and Human Resources  Advisory Committee provides advice, guidance, and  recommendations concerning NSF’s science and engineering  education programs; performance reviews, such as Commerce’s Annual Performance and  Accountability Reports, which provide data on performance measures  for NIST including the number of publications produced, or the  Advanced Research Projects Agency–Energy’s (ARPA-E) quarterly  technical milestone reviews of funded projects, which help reviewers  decide if project funding should be continued;
Committees of Visitors, such as those used by NSF, Science and  NIST, which are groups of external experts that assess the overall  quality of program operations and, in some cases, program outcomes;  and formal program evaluations, which are systematic, empirical studies  used by agencies to assess how well a particular program or  component of a program is working.
section_title: Appropriations Generally Increased but Fell Short of Authorizations, and Most Newly Authorized Programs Were Not Funded
paragraphs: Of the $62.2 billion authorized under the COMPETES Acts in fiscal years  2008 through 2012, $52.4 billion was appropriated, including $51.9 billion  for the entire budgets of NSF, Science, and NIST. Funding for these three  entities accounts for more than 99 percent of the funding appropriated  under the COMPETES Acts during this period. (See fig. 1.)
Appropriations for NSF, Science, and NIST generally increased in fiscal  years 2008 through 2012 but did not reach authorized levels. For  example, NSF’s appropriations increased from about $5.9 billion in fiscal  year 2007—the last year before its appropriation was authorized under  the COMPETES Acts—to about $7 billion in fiscal year 2012, when it was  authorized to receive $7.8 billion. Appropriations for Science, which were  authorized under the COMPETES Acts starting in fiscal year 2010,  increased from $4.8 billion in fiscal year 2009 to about $4.9 billion in fiscal  year 2012, when Science was authorized to receive $5.6 billion. Likewise,  funding for NIST increased but not to authorized levels: in fiscal year  2007, before its appropriation was authorized under the COMPETES  Acts, it received about $680 million, compared with $750 million in fiscal  year 2012, when NIST was authorized to receive about $970 million. In  addition, the American Recovery and Reinvestment Act of 2009  (Recovery Act) appropriated about $5.2 billion to these entities in fiscal  year 2009. The majority of the Recovery Act appropriations—over $3  billion—went to NSF. Figure 2 shows the amounts authorized under the  COMPETES Acts for NSF, Science, and NIST, as compared with the  amounts appropriated, including Recovery Act appropriations.
The COMPETES Acts also specifically authorized funding for 40  individual programs, including some programs within and some outside of  NSF, Science, and NIST. (See app. II.) For example, in addition to  authorizing $22.1 billion for the entire budget of NSF in fiscal years 2008  through 2010, COMPETES 2007 specifically authorized $345 million of  that total for NSF’s Robert Noyce Teacher Scholarship Program in fiscal  years 2008 through 2010. The programs not within NSF, Science, or  NIST fell elsewhere within the Departments of Commerce and Energy, or  in Education.
Among the 40 programs for which the COMPETES Acts specifically  authorized funding, the 12 programs that existed before the acts all  received appropriations and continue to operate. Six of the 28 newly  authorized programs also received appropriations. Of these 6 programs,  1—ARPA-E—is continuing operations, 3 did not receive appropriations in  fiscal year 2012, and 2 are in the process of being implemented as of  May 2013. More specifically, according to officials NSF’s Science  Master’s Program, NIST’s Technology Innovation Program, and  Education’s Teachers for a Competitive Tomorrow did not receive  appropriations in fiscal year 2012. Officials told us the Science Master’s  Program and the Technology Innovation Program are in the process of  shutting down, and Teachers for a Competitive Tomorrow has not  awarded new grants since fiscal year 2010. Further, Commerce’s Loan  Guarantees for Science Park Infrastructure and Federal Loan Guarantees  for Innovative Technologies in Manufacturing are in the process of being  implemented, according to Commerce officials; these programs first  received appropriations in fiscal year 2012. Twenty-two of the 28 newly  authorized programs did not receive appropriations, including 9 programs  that were newly authorized in COMPETES 2007 but repealed in  COMPETES 2010.
In total, 16 of the 40 programs for which the COMPETES Acts specifically  authorized funding have been implemented, including the 12 previously  existing programs, ARPA-E, and the 3 newly authorized programs that  did not receive appropriations in fiscal year 2012. As noted previously,  two other newly authorized programs that received appropriations are in  the process of being implemented. As shown in figure 3, the implemented  programs generally focus on five areas: (1) research and development  programs focus on activities aimed at enhancing scientific research and  development,(2) manufacturing performance programs focus on  supporting innovation among U.S. manufacturers and other  organizations,(3) STEM teacher training programs focus on education  and professional development for prospective or existing STEM teachers,  (4) STEM undergraduate programs focus on encouraging or improving  undergraduate STEM education, and (5) STEM graduate programs focus  on supporting graduate students training for careers in research or  education in STEM disciplines.
With few exceptions, agency officials told us they did not include funding  requests in their budget submissions for the programs that did not receive  funding. Agencies did include funding requests in their budget  submissions for 4 of the 22 newly authorized programs that did not  receive funding. Specifically, Education requested appropriations for the  Math Now program in fiscal years 2008 and 2009 and for the Foreign  Language Partnership and Advanced Placement and International  Baccalaureate programs in fiscal year 2009, while Commerce requested  appropriations in fiscal year 2012 for the Regional Innovation Program.  For the other 18 programs, officials most often told us they did not  request appropriations because their agencies already had similar  programs under way or could work within current programs to carry out  similar activities. For example, Science officials told us the office did not  request appropriations for the Discovery Science and Engineering  Innovation Institutes because the program’s implementation would have  duplicated existing activities. Commerce officials told us that the  department’s Economic Development Administration implemented certain  aspects of the Regional Innovation Program (RIP) through the existing  Economic Adjustment Assistance program when RIP did not receive  appropriations. However, the officials also noted that by implementing  aspects of RIP in this manner they have reduced funding available to  other worthwhile aspects of the Economic Adjustment Assistance  program. In other cases, officials told us they did not request  appropriations because the programs did not fit into the agency mission  or because the agencies prioritized other programs in light of limited  resources or other factors.
section_title: Recent Evaluations Generally Suggest Positive Results and Some Recommend Improvements for Implemented Programs
paragraphs: Recent evaluations that we identified have generally suggested positive  results for the fully implemented programs for which the COMPETES Acts  specifically authorized funding; some of the evaluations have also  recommended ways to improve the programs. Recent evaluations have  been conducted for 15 of 16 fully implemented programs. According to  NSF officials, no evaluation of the Science Master’s Program was  published during 2008-2012, which is the time frame of evaluations  included in our review. These evaluations have provided information on  how well programs—or aspects of programs—were working in each of  the five areas on which the programs focus. Many studies have also  made recommendations for program improvement. Following are  examples of selected evaluations and key findings for programs in the  five focus areas:
Research and development. Studies for these programs found that  they were generally producing positive results and noted areas for  continued improvement. For example, one study used econometric  modeling to examine the effects of the Experimental Program to  Stimulate Competitive Research, which aims to improve the research  and development capacity of participating states. The results  suggested participating states have been effective in growing federal  financial support for science and engineering at a faster rate  compared with nonparticipating states. However, the authors noted  that while the effect they found was statistically significant, it was a  small effect. They concluded that more enhanced and innovative  efforts are needed to sustainably build states’ research and  development capacity. In another example, our 2012 review of ARPA- E found that it successfully funded projects that would not have been  funded solely by private investors, in keeping with its goals.
Manufacturing Extension Partnership, Delivering Measurable Results to Manufacturing  Clients: Fiscal Year 2009 Results (Washington, D.C.: March, 2011). These findings were  based on MEP surveys of program clients; fewer than 50 percent of respondents reported  that MEP had an impact on sales, investment levels, jobs created, or jobs retained. A  majority of client respondents did report cost savings in areas such as labor, materials,  inventory, and energy.
Jennifer Carney et al., Evaluation of the National Science Foundation’s Integrative  Graduate Education and Research Traineeship Program (IGERT): Follow-Up Study of  IGERT Graduates (Arlington, VA: February 2011). experience positively contributed to their ability to finish their PhDs. A  majority of respondents (94 percent) also reported that the IGERT  experience helped them obtain their current work positions. An  exploratory analysis comparing IGERT graduates with non-IGERT  graduates from similar academic departments found no significant  difference between the graduates in securing employment, but they  did find that IGERT graduates reported a greater interest in  interdisciplinary education or research training.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to Commerce, DOE, Education, and  NSF for comment. Commerce’s Economic Development Administration  did not have any comments on the draft. Commerce’s NIST provided  written comments, which are reproduced in appendix III, along with our  response. On behalf of NIST, the Secretary of Commerce stated that the  draft, as scoped, does not fully capture the significant positive impact that  the COMPETES Acts have had on NIST. As noted in our response in  appendix III, the scope of our review is based on a mandate in  COMPETES 2010 that calls for GAO to evaluate the extent to which  programs authorized under the law have been funded, implemented, and  are contributing to achieving the goals of the act. Our report addresses  the total appropriations to NIST in fiscal years 2008 through 2012 in  objective 1. To satisfy the needs of our congressional clients, we focused  our review on programs for which the COMPETES Acts specifically  authorized funding. DOE provided technical comments, which we  incorporated as appropriate. Education did not have any comments on  the draft. NSF provided technical comments, which we incorporated, as  appropriate.
We are sending copies of this report to the Secretaries of Commerce,  Education, and Energy; the Director of NSF; the appropriate  congressional committees; and other interested parties. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To inform our objectives, we reviewed our October 2010 report on agency  obligations under the America Creating Opportunities to Meaningfully  Promote Excellence in Technology, Education, and Science Act of 2007  (COMPETES 2007) and the steps agencies were taking to evaluate  implemented programs under COMPETES 2007. We also reviewed  relevant laws, including COMPETES 2007 and the COMPETES  Reauthorization Act of 2010 (COMPETES 2010), and we interviewed  agency officials from the National Science Foundation (NSF); the  Department of Energy (DOE), including the Office of Science (Science)  and the Advanced Research Projects Agency–Energy (ARPA-E); the  Department of Commerce (Commerce), including the National Institute of  Standards and Technology (NIST) and the Economic Development  Administration; and the Department of Education (Education).
To determine the extent to which funding was appropriated under the  authorization of the COMPETES Acts, we reviewed the laws and  identified the entities and programs for which the acts specifically  authorized funding. To determine the extent to which funding was actually  provided to such entities and programs, we reviewed annual  appropriations data in Congressional Research Service reports and  agency budget justification documents, and confirmed these data with  agency officials. Agency officials were not able to provide complete  appropriations data at the program level. Complete final appropriations  data for fiscal year 2013 were also not available. We interviewed agency  officials to learn about the status of programs that received funding. We  also asked agency officials which programs were included in annual  budget requests and why other programs were not included in those  requests.
To examine what the results of evaluations suggest about how the  programs for which the COMPETES Acts specifically authorized funding  are working, we identified and reviewed a selection of recent studies that  evaluated these programs. To identify these evaluations, we conducted a  literature review and interviewed agency officials. Specifically, we  conducted a literature search in databases such as ProQuest, SciSearch,  and Academic One to search for recent reports or publications that  evaluated programs authorized by the COMPETES Acts. We conducted  an initial review of the summaries of the reports or publications returned  in our literature search and provided by agencies, and we excluded those  that did not appear to evaluate how well programs were working and were  not published in 2008-2012. We included studies published from 2008  through 2012; in some cases, these studies included data on program  activities that occurred before 2008. We reviewed the methodology of the  identified studies and reported on the results of those we determined to  be methodologically sound and reliable for the purposes of our report. We  included the results of 21 studies covering 13 programs in our review. We  reported findings on a select number of programs and evaluations based  on the following criteria: (1) we included programs that received federal  funding in fiscal year 2012 and are continuing operations as of May 2013  and (2) for each focus area, we chose programs with the most recent  evaluations and included up to two of the most recent studies for those  programs. Some agencies provided us with performance reviews and  other reports containing performance measurements that we did not  include in our review of evaluations. When drafting our report, we  provided agency officials with information on the studies to be included.  We determined focus areas for implemented programs based on agency  information about the purposes and goals of those programs. We also  interviewed officials from departments, agencies, and program offices  with authorized funding under the COMPETES Acts, including  representatives from NSF’s Education and Human Resources  Directorate, Engineering Directorate, and Office of Integrative Activities;  Energy’s Science and ARPA-E; Commerce’s NIST and Economic  Development Administration; and Education’s Office of Postsecondary  Education.
We conducted this performance audit from October 2012 to July 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Programs for Which the COMPETES Acts Specifically Authorized Funding
paragraphs: Figure 4 below shows the 40 programs for which the America Creating  Opportunities to Meaningfully Promote Excellence in Technology,  Education, and Science (COMPETES) Act of 2007 or the COMPETES  Reauthorization Act of 2010 specifically authorized funding.
section_title: Appendix III: Comments from the Department of Commerce
subsections: section_title: GAO Comments
paragraphs: 1.  The scope of our review is based on a mandate in the America  COMPETES Reauthorization Act of 2010 (COMPETES 2010) that  calls for GAO to evaluate the extent to which programs authorized  under the law have been funded, implemented, and are contributing to  achieving the goals of the act. Our report addresses the total  appropriations to NIST in fiscal years 2008 through 2012 in our first  objective. To satisfy the needs of our congressional clients, we  focused our review on programs for which COMPETES 2010 and the  America Creating Opportunities to Meaningfully Promote Excellence  in Technology, Education, and Science Act of 2007 (COMPETES  2007) specifically authorized funding.   2.  Our draft report notes that the Baldrige Performance Excellence  Program is currently operating using private funds, according to  agency officials. We clarified in our footnote that the program remains  authorized to receive funding. We did not include information about  funding for fiscal year 2013 because that is beyond the scope of our  review.   3.  The scope of our review includes program evaluations published from  2008 through 2012. The draft report includes the findings from the  Manufacturing Extension Partnership’s (MEP) fiscal year 2009 client  impact survey, which was the most recent survey published in our  time frame. The results of the MEP 2011 client impact survey, which  was published in March 2013, were outside the scope of our review.  4.  To identify the evaluations we included in our analysis, as described  in appendix I, we conducted a literature search and asked the  agencies to provide evaluations of the programs that were published  from 2008 through 2012. We conducted an initial review of the  summaries of the reports or publications identified and excluded those  that did not appear to evaluate how well programs were working.  Based on the summary of the publication cited in NIST’s letter, we  concluded it was not in our scope, and it was not included in the  selection of studies we reported on. When drafting our report, we  provided agency officials with information on the studies to be  included, and Commerce officials did not bring this publication to our  attention at that time. However, upon receiving NIST’s comments, we  reviewed our criteria for identifying studies, and we continue to believe  that our approach to identifying and selecting studies was appropriate.  5.  We believe figure 4 clearly communicates the authorization of funding  for the Technology Innovation Program (TIP) under COMPETES  2007. Figure 4 also shows that TIP did not receive appropriations in  fiscal year 2012, and that the program is not currently operating.  Further, in the body of our report, we say that according to agency  officials TIP is in the process of shutting down. We do not provide  information about fiscal year 2014 because that is outside the scope  of our review.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Karla Springer (Assistant  Director), Nicole Dery, Cindy Gilbert, Michael Kendix, Cynthia Norris,  Marietta Mayfield Revesz, and Barbara Timmerman made key  contributions to this report.