```
Precollege Math and Science Education: Department of Energy's Precollege
Program Managed Ineffectively (Letter Report, 09/13/94, GAO/HEHS-94-208).

Public policymakers and U.S. industrial leaders have expressed grave
concern about precollege students in other industrialized countries
significantly outperforming American students on recent international
mathematics and science tests. In recognition of the Energy Department's
(DOE) world-class scientists and engineers as well as its
state-of-the-art research facilities, Congress made mathematics and
science education a major mission for DOE in fiscal year 1991. DOE's
precollege mathematics and science budget has grown about 1,250
percent--reaching about $27 million in fiscal year 1993. In the early
1990s, DOE did not effectively manage this program. First, DOE
jeopardized the program's success by not using  a risk management
strategy to run the program's projects. Second, DOE forfeited an
invaluable management tool by taking a lax approach to program
evaluation. Third, DOE greatly reduced its chances of helping achieve
National Education Goal 5--"By the year 2000, U.S. students will be
first in the world in mathematics and science achievement"--by launching
a variety of projects that did not clearly seek to improve student
achievement. DOE has announced steps to substantially improve the
program's management and evaluation functions; however, the depth of
executive support for these measures and their subsequent staying power
are uncertain.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-94-208
     TITLE:  Precollege Math and Science Education: Department of 
             Energy's Precollege Program Managed Ineffectively
      DATE:  09/13/94
   SUBJECT:  Secondary education
             Agency missions
             Energy research
             Education program evaluation
             Mathematics
             Scientific research
             Elementary education
             Teacher education
             Mission budgeting
             Students
IDENTIFIER:  National Education Goals
             National Education Longitudinal Study of 1988
             Goals 2000
             
**************************************************************************
* This file contains an ASCII representation of the text of a GAO        *
* report.  Delineations within the text indicating chapter titles,       *
* headings, and bullets are preserved.  Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are           *
* identified by double and single lines.  The numbers on the right end   *
* of these lines indicate the position of each of the subsections in the *
* document outline.  These numbers do NOT correspond with the page       *
* numbers of the printed product.                                        *
*                                                                        *
* No attempt has been made to display graphic images, although figure    *
* captions are reproduced. Tables are included, but may not resemble     *
* those in the printed version.                                          *
*                                                                        *
* A printed copy of this report may be obtained from the GAO Document    *
* Distribution Facility by calling (202) 512-6000, by faxing your        *
* request to (301) 258-4066, or by writing to P.O. Box 6015,             *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time.                                    *
**************************************************************************

Cover
================================================================ COVER

Report to the Chairman, Committee on Governmental Affairs, U.S. 
Senate

September 1994

PRECOLLEGE MATH AND SCIENCE
EDUCATION - DEPARTMENT OF ENERGY'S
PRECOLLEGE PROGRAM MANAGED
INEFFECTIVELY

GAO/HEHS-94-208

Precollege Math and Science Education

Abbreviations
=============================================================== ABBREV

  AMES - Ames Laboratory
  ANL - Argonne National Laboratory
  AWU - Associated Western Universities
  BNL - Brookhaven National Laboratory
  FCCSET - Federal Coordinating Council for Science, Engineering, and
     Technology
  CEHR - Committee on Education and Human Resources
  LANL - Los Alamos National Laboratory
  LBL - Lawrence Berkeley Laboratory
  LLNL - Lawrence Livermore National Laboratory
  NSF - National Science Foundation
  ORISE - Oak Ridge Institute for Science and Education
  ORNL - Oak Ridge National Laboratory
  PNL - Pacific Northwest Laboratory
  SNL-A - Sandia National Laboratory-Albuquerque
  SNL-L - Sandia National Laboratory-Livermore

Letter
=============================================================== LETTER

B-251474

September 13, 1994

The Honorable John Glenn
Chairman, Committee on
 Governmental Affairs
United States Senate

Dear Mr.  Chairman: 

Our nation's ability to remain the economic world leader depends on
its citizens' strong mathematics and science skills.  Understandably,
public policymakers and industrial leaders have expressed grave
concern about precollege students in other industrialized countries
significantly outperforming American students on recent international
mathematics and science tests.  In response to educational reform and
competency concerns, former President Bush and the nation's governors
developed six National Education Goals to be achieved by the year
2000.\1

In recognition of the Department of Energy's world-class scientists,
engineers, and technicians, as well as its state-of-the-art
laboratories and research facilities, the Congress made mathematics
and science education a major mission for the Department in fiscal
year 1991.  Consequently, Energy's precollege mathematics and science
program budget has grown approximately 1,250 percent--from
approximately $2 million in fiscal year 1990 to approximately $27
million in fiscal year 1993 (see app.  I). 

This report responds to your questions about how effectively Energy
manages its precollege mathematics and science program.  On the basis
of discussions with your office, we agreed to determine (1) the
appropriateness of Energy's precollege program implementation
priorities, (2) the role of project evaluations in ensuring rational
budget decisions, and (3) whether its precollege program helps
achieve National Education Goal 5--"By the year 2000, U.S.  students
will be first in the world in mathematics and science achievement."

--------------------
\1 In 1994, the Goals 2000:  Educate America Act expanded the
National Education Goals from six to eight.  The goals address (1)
school readiness; (2) school completion; (3) student achievement and
citizenship; (4) teacher education and professional development; (5)
mathematics and science achievement;(6) adult literacy; (7) safe,
disciplined, and alcohol- and drug-free schools; and (8) parental
participation.  The National Education Goals were originally
developed in 1989. 

   BACKGROUND
------------------------------------------------------------ Letter :1

Over the last 10 years, numerous reports have charged that many U.S. 
students complete high school scientifically and technologically
illiterate.  According to these reports, not only are U.S.  students
less well educated than their predecessors, they are also less well
trained in mathematics and science than their peers in other
industrialized countries.  Reported decreasing student enrollments in
science courses, declining achievement test scores, and the
continuing decline in the number of high-quality mathematics and
science teachers have highlighted problems in precollege mathematics
and science instruction.\2

To remedy this perceived crisis in education, the Congress conferred
mathematics and science education responsibilities on the Department
of Energy and 13 other federal agencies--in addition to the
Department of Education and the National Science Foundation (NSF). 
To improve mathematics and science education, the Federal
Coordinating Council for Science, Engineering, and Technology
(FCCSET)\3 Committee on Education and Human Resources (CEHR) was
charged with coordinating the efforts of these 16 agencies.  FCCSET
developed broad implementation priorities for precollege education,
including (1) standards for curriculum, teaching, and assessment; (2)
curriculum, course, and instructional materials; (3) systemic reform;
and (4) teacher preparation and enhancement.  These priorities were
formalized in January 1993 as part of a 5-year strategic plan. 

--------------------
\2 Science, Engineering, and Mathematics Education, Congressional
Research Service Issue Brief, (Washington, D.C.:  1992), p.  1. 

\3 In November 1993, President Clinton established by executive order
a cabinet level National Science and Technology Council (NSTC) to
coordinate science, space, and technology policies throughout the
federal government.  The establishment of NSTC consolidated the
responsibilities previously carried out by a number of agencies,
including FCCSET. 

      ENERGY'S PRECOLLEGE
      MATHEMATICS AND SCIENCE
      PROGRAM
---------------------------------------------------------- Letter :1.1

The Department of Energy Science Education Enhancement Act authorized
Energy to undertake a wide range of precollege education and training
activities.  Some of Energy's activities include making loans of
equipment and staff to schools; allowing Energy employees to provide
education-oriented community services; and participating in joint
programs with schools, businesses, museums, and other community
partners.  Energy manages its precollege program with a decentralized
organizational structure:  Energy's nine national laboratories and 22
of its research facilities were given the flexibility to design and
implement projects using the broad implementation priorities provided
by FCCSET.  In designing projects, each Energy facility considered
FCCSET's broad implementation priorities, as well as its individual
areas of specialization and local needs.  Although the projects vary,
most emphasize hands-on experiences and fall within three
implementation categories: 

  Teacher enhancement--These projects attempt to further the content
     knowledge, skills, and experiences of teachers already in the
     workforce.  Teacher enhancement projects typically provide
     teachers with opportunities to (1) work on a variety of
     scientific and technical subjects as a member of an Energy
     laboratory research team, (2) train with mentors who assist them
     with in-school science experiments, or (3) obtain sophisticated
     computer training. 

  Student support--These projects seek to reward students for
     outstanding achievement, afford them enrichment experiences, and
     furnish them with supplementary educational services such as
     tutoring and mentoring.  Student support projects typically
     allow students opportunities to (1) participate in cutting-edge
     research at an Energy laboratory, (2) attend scientific lectures
     and demonstrations, and (3) study emerging topics such as
     environmental energy. 

  Systemic reform--These projects aim to improve education by
     changing all aspects of an educational system.  Systemic reform
     projects typically involve key education stakeholders--students,
     teachers, administrators, policymakers, and parents--in
     collaborative efforts to (1) create goals and standards for all
     students, (2) develop related curricula and instructional
     materials, and (3) provide professional development for
     teachers. 

   RESULTS IN BRIEF
------------------------------------------------------------ Letter :2

Although Energy invested more than $50 million in precollege
education in fiscal years 1990-93, it did not effectively oversee or
direct the program.  For example, although research findings indicate
that systemic reform may have the greatest potential to improve
student learning, projects in this implementation category
constituted the smallest share of Energy's precollege budget, about
11 percent.  In contrast, Energy used about 70 percent of its
precollege budget to finance teacher enhancement projects, even
though research suggests that these projects may be ineffective at
increasing student achievement. 

To compound problems, Energy did not link budget decisions to project
evaluation results.  As a result, Energy had not evaluated almost
half of its 17 most resource-intensive projects at the time of our
review; for those projects with evaluation reports, all were
inadequate.  Nonetheless, Energy substantially increased funding for
most of these projects--in one case by over 1,700 percent. 

In addition, it is doubtful whether Energy's precollege program will
help achieve National Education Goal 5.  In this regard, Energy's
projects typically do not focus on student achievement, which is
central to achieving this goal.  In fact, more than half of Energy's
most resource-intensive projects did not directly include improving
student achievement as an objective. 

During our review, Energy indicated recognition of the need to pay
closer attention to managing its precollege program.  To correct this
situation, Energy recently drafted an agency-specific strategic plan. 
In addition, officials said they plan to restructure their program to
ensure that all projects are evaluated and linked more clearly to
National Education Goal 5. 

   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3

Our review of Energy's precollege mathematics and science program
focused primarily on its 288 fiscal year 1992 projects.  To
accomplish our objectives, we conducted site visits at Energy's
headquarters in Washington, D.C., and eight laboratories and research
facilities in California, Illinois, New Mexico, Tennessee, and Utah,
which administered the most resource-intensive precollege projects.\4
During these site visits, we interviewed laboratory personnel as well
as school administrators, teachers, and students involved in the
precollege program. 

To obtain an overview of the program, we collected general
information on Energy's entire array of fiscal year 1992 precollege
mathematics and science projects and reviewed the literature on the
relationship between teacher quality and subsequent student
achievement.  For the 17 most resource-intensive projects, we
collected and analyzed budget data for fiscal years 1990-93 (see app. 
III) and, when available, evaluation reports.  Although constituting
less than 6 percent of Energy's program portfolio, these 17 projects
accounted for 50 percent ($11 million) of total program dollars in
fiscal year 1992.  To determine whether the evaluation reports
provided reliable information on project effectiveness, we also
conducted a technical review of the nine project evaluation reports
Energy officials submitted to our staff (see app.  IV). 

We conducted our work between November 1992 and July 1994 in
accordance with generally accepted government auditing standards. 

--------------------
\4 To identify the most resource-intensive projects, we examined the
extent to which both financial and personnel resources were used.  We
selected 16 projects with the largest budgets and 1 project that
involved considerably more Energy personnel than other projects. 

   FOCUS ON TEACHER ENHANCEMENT
   PROJECTS IS QUESTIONABLE
------------------------------------------------------------ Letter :4

Energy's decision to concentrate its precollege program resources on
teacher enhancement projects, which account for more than two-thirds
of the program budget, is a questionable implementation strategy.  In
an earlier report, we found no evidence that training programs to
upgrade existing science and mathematics teachers' skills will
improve teaching effectiveness.\5 In reviewing more recent studies,
we found mixed results:  in some instances, researchers found small
yet statistically significant positive correlations between teacher
knowledge and student achievement; in others, researchers failed to
demonstrate any significant correlations.  Conversely, current
literature suggests that systemic reform measures, such as
high-quality curriculum development, may hold the most promise for
improving academic achievement and realizing the national math and
science goals. 

--------------------
\5 New Directions for Federal Programs to Aid Mathematics and Science
Teaching (GAO/PEMD-84-5, Mar.  6, 1984). 

      PRECOLLEGE PROGRAM HEAVILY
      WEIGHTED TOWARD TEACHER
      ENHANCEMENT PROJECTS
---------------------------------------------------------- Letter :4.1

Both in budget dollars and project numbers, Energy devoted most of
its precollege program resources to teacher enhancement projects
during fiscal year 1992.  Regarding the program budget, Energy spent
about 70 percent ($15.4 million) of all precollege program dollars to
upgrade teachers' mathematics and science skills--45 percent
exclusively for teacher enhancement and another 25 percent for
teacher enhancement combined with a student support component.  For
the residual, about 19 percent ($4.2 million) of Energy's precollege
budget focused exclusively on student support; systemic reform
efforts constituted just 11 percent ($2.4 million).  Regarding the
total number of projects, 157 of 288 (55 percent) had a substantial
teacher enhancement component, another 113 projects (39 percent)
focused exclusively on student support, 14 projects (5 percent)
involved systemic reform efforts, and 4 projects (1 percent) included
activities that did not specifically involve teachers and students
(see table 1). 

                           Table 1
           
             Precollege Program Heavily Weighted
             Toward Teacher Enhancement Projects

                              Budget
                                 (in
                              thousa  Percen          Percen
Implementation priorities       nds)       t  Number       t
----------------------------  ------  ------  ------  ------
Teacher enhancement           $9,911      45     100      35
Teacher enhancement and        5,524      25      57      20
 student support
Student support                4,187      19     113      39
Systemic reform                2,448      11      14       5
Other\a                           95      \b       4       1
============================================================
Total                         $22,16     100     288     100
                                   5
------------------------------------------------------------
\a "Other" includes activities that do not specifically involve
students and teachers, such as efforts by Energy's staff to develop a
catalog listing precollege physics projects. 

\b Less than 1 percent. 

      NO STRONG RELATIONSHIP
      BETWEEN TEACHER ENHANCEMENT
      AND STUDENT ACHIEVEMENT
---------------------------------------------------------- Letter :4.2

Research has failed to show conclusively a relationship between
teacher enhancement and student achievement.\6 We based our finding
on the results of a 1984 GAO report and a review of recent studies
that examined the effect of teacher quality on student achievement
(see bibliography, p.  25).  In our earlier report, we found no
evidence that training programs to upgrade mathematics and science
teachers' skills improved student achievement.\7 That analysis was
based first on an NSF study that compared the achievement of eighth
and eleventh grade students whose teachers participated in NSF
institutes to those who did not.  The study showed that teacher
participation in NSF institutes had a positive effect on eleventh
grade students' science and mathematics achievement.  However,
institute participation did not have a statistically significant
effect on eighth grade student achievement in either science or
mathematics.  Second, several general studies from the 1960s and
1970s as a group failed to show a consistent relationship between
teacher knowledge and student achievement. 

More recent studies have also failed to demonstrate a strong
relationship between teacher enhancement/knowledge and student
achievement.  Two of the most prominent studies conducted between
1984 and 1994 that examined this relationship reported mixed results: 

  A 1992 NSF study, which analyzed teacher transcript and student
     test data from the National Education Longitudinal Study of 1988
     (NELS:88),\8 showed a statistically significant relationship
     between eighth grade students' mathematics achievement and their
     teachers' preparation.\9 Specifically, students whose teachers
     had majored in mathematics performed slightly better than those
     whose teachers had majored in education only.  However, no
     statistically significant relationship existed for science. 

  A 1994 Chicago Academy of Sciences study, which analyzed the effect
     of teachers' participation in 4-week summer science workshops,
     reported a small, but statistically significant, increase in the
     level of science achievement for seventh grade students of
     participants.\10 For these students, the average science
     achievement score increased from 47.1 to 49.6.  However, no
     statistically significant change occurred for either sixth or
     eighth grade science students. 

At least four explanations for the weak relationship between
teachers' knowledge or participation in training programs and student
achievement exist.  First, most training programs generally involve
one-time, relatively short events with little or no follow-up.  For
example, a single 4- to 8-week summer research experience probably
would not produce a dramatic change in a teacher's effectiveness. 
Second, teacher training programs are subject to self-selection bias;
that is, in-service training often attracts exemplary or highly
motivated teachers.  Exposing such teachers to short-term workshop
training may not significantly add to their teaching effectiveness. 
Third, the most knowledgeable teachers may not be the best teachers. 
Other factors besides teacher knowledge--such as enthusiasm,
confidence, and organization of class time--may determine student
achievement.  Fourth, some researchers suggest that current student
assessments inadequately measure the higher order, problem-solving
skills, which could be affected by teacher training and knowledge. 

--------------------
\6 Although research in this area has been limited, the studies we
identified continue to be cited in current research, and the findings
remain unchallenged. 

\7 New Directions for Federal Programs to Aid Mathematics and Science
Teaching. 

\8 NELS:88 is a nationally representative sample of 26,435
eighth-grade students clustered within 1,052 schools. 

\9 Senta Raizen and Theodore Britton, "Science and Mathematics
Teachers," Indicators of Science and Mathematics Education in 1992,
National Science Foundation (Washington, D.C.:  1993), pp.  85-113. 

\10 Jon D.  Miller, Enriching Middle School Science:  A Final
Evaluation of the 1991-92 Columbia College Workshops Utilizing an
Innovative Approach to the Teaching of Science.  Chicago Academy of
Sciences (NSF grant TPE 89-55128), (Chicago:  1994), pp.  30-31. 

      SYSTEMIC REFORM CONSIDERED
      PROMISING APPROACH FOR
      INCREASING STUDENT
      ACHIEVEMENT
---------------------------------------------------------- Letter :4.3

Energy has implemented few systemic reform projects, even though
current educational literature suggests systemic reform may have the
greatest potential for improving student learning.\11 Systemic reform
is promising because it (1) attempts to stimulate change in many or
all components of the educational system simultaneously; (2)
establishes clear standards for what students should know and be able
to do; and (3) involves key educational stakeholders--students,
teachers, administrators, teacher educators, textbook publishers,
policymakers, and parents--at all levels of the education
system--national, state, district, and school. 

Under systemic reform, teacher training ideally takes place in an
environment that supports new curricula or teaching techniques
learned during training.  However, many educators believe that
retraining individual teachers will have little measurable impact on
student achievement if the education system is not prepared to absorb
improvements.  For example, although teacher enhancement programs may
provide teachers the knowledge, skills, and enthusiasm essential to
implement new curricula, teachers can rarely implement and sustain a
new program if their school support systems and attitudes of
administrators, colleagues, and parents have not changed. 

--------------------
\11 Much of this literature is cited in Marshall Smith and Jennifer
O'Day, "Systemic School Reform," Politics of Education Association
Yearbook, (1990), pp.  233-267. 

      ADDITIONAL PROGRAM DIVERSITY
      COULD REDUCE PROGRAM RISK
---------------------------------------------------------- Letter :4.4

Given the evidence cited, Energy's heavy investment in teacher
enhancement projects substantially increases its risk of not
improving student achievement in mathematics and science.  However,
Energy could reduce this risk by changing its mix of projects to
balance the program, much like financial advisers do by diversifying
investment portfolios.  In managing uncertainty, financial advisers
minimize the risk of loss by acquiring a variety of investment
vehicles; thus, good returns from one investment counterbalance poor
returns from another.  Building on this analogy, Energy could view
its precollege program as a collection of investment vehicles
assembled to meet an investment goal--improved student achievement in
mathematics and science.  Thus, given the uncertainty of its
projects' educational payoffs, a more diverse program portfolio would
enhance the likelihood of Energy's achieving its program goal. 

To address these concerns, Energy developed a strategic plan, which
identifies agency-specific precollege goals and objectives in March
1994.  Energy envisions using the strategic plan, which should be in
place by November 1994, to help it create a program strategy that
supports the best mix of projects and minimizes risk.  In addition,
Energy plans to look at the feasibility of eliminating projects that
do not support its strategic plan and restructuring all teacher
enhancement projects to include systemic reform elements, such as
follow-up support beginning in fiscal year 1995. 

   BUDGET DECISIONS NOT LINKED TO
   PROJECT EVALUATIONS
------------------------------------------------------------ Letter :5

Energy did not link budget decisions to project evaluations.  Until
recently, Energy neither required project evaluations nor ensured
their adequacy when its research facilities conducted them.  For
example, Energy elected not to evaluate eight of its 17 most
resource-intensive projects.  When conducted, evaluations were of
poor quality.  For instance, all projects with evaluations contained
discrediting technical flaws--such as insufficient sample sizes, the
absence of statistical tests, and insufficient supporting data--that
potentially invalidated any evaluation findings (see app.  IV). 

Energy's limited use of program evaluation reflected its management
priorities.  According to Energy officials, the Department did not
emphasize effectiveness evaluations because program expansion was its
primary objective.  These officials also said that Energy lacked the
capacity--staff, funds, and expertise--to design and monitor
evaluations for such a vast array of projects.  Consequently, in the
absence of sufficient evaluation results, Energy substantially
increased project budgets on the basis of self-reported data, such as
customer satisfaction surveys, or anecdotal data such as participant
testimonials, requests to participate in particular projects, or
other popularity indicators.  Consider the following examples: 

  Energy increased the budgets for four projects with no evaluations,
     with increases ranging from 62 to 912 percent from fiscal year
     1991 to 1992 (see fig.  1).\12

   Figure 1:  Project Budgets
   Increased Substantially Without
   Evaluations

   (See figure in printed
   edition.)

Notes: 

\a The Science/Math Carnival (SNL-L) project budget was increased
from approximately $3,000 to $26,000 or by 912 percent. 

\b The Bay Area Science and Technology Education Collaboration (LBL)
project budget was increased from approximately $288,000 to $466,000
or by 62 percent. 

\c The Environmental Management Precollege Analytical Chemistry (AWU)
project budget was increased from approximately $225,000 to $615,000
or by 173 percent. 

\d The National Science Explorers (ANL) project budget was increased
from approximately $318,000 to $693,000 or by 118 percent. 

  Energy increased the budgets for five projects with inadequate
     evaluations, with increases ranging from 113 to 1,731 percent
     from fiscal year 1990 to 1992 (see fig.  2). 

   Figure 2:  Project Budgets
   Increased Substantially Despite
   Inadequate Evaluations

   (See figure in printed
   edition.)

Notes: 

\a The Environmental Education Outreach for Minorities (BNL) project
budget was increased from approximately $93,000 to $199,000 or by 113
percent. 

\b The Teacher Research Associates (ORISE) project budget was
increased from approximately $49,000 to $231,000 or by 372 percent. 

\c The OPTIONS (PNL) project budget was increased from approximately
$50,000 to $915,000 or by 1,731 percent. 

\d The Teacher Research Associates (AWU) project budget was increased
from approximately $526,000 to $1,779,000 or by 238 percent. 

\e The Science Advisors (SNL-A) project budget was increased from
approximately $545,000 to $2,500,000 or by 359 percent. 

On the basis of our findings, Energy needs to change the way it views
the relationship between project implementation and evaluation. 
Energy should view program evaluation as an integral part of program
management; rather than perceiving the two as mutually exclusive. 
Because quality program evaluation is expensive, Energy must accept
the unavoidable trade-off that it must fund fewer projects with
stronger evaluation components. 

To address its program evaluation shortcomings, Energy began a
partnership with the National Center for Improving Science Education
to jointly develop a system for ongoing evaluation of its precollege
program in May 1992.  In addition, Energy established an evaluation
guidance committee responsible for developing a long-range
implementation plan for project evaluation.  Energy officials also
said that beginning in fiscal year 1995, the Department will require
each precollege project to have an evaluation component as a
prerequisite for funding. 

--------------------
\12 Projects shown in figure 1 began in fiscal year 1991. 

   PROGRAM UNLIKELY TO CONTRIBUTE
   TO ACHIEVING NATIONAL EDUCATION
   GOAL 5
------------------------------------------------------------ Letter :6

Energy has greatly diminished the prospect of its program's helping
to achieve National Education Goal 5--making American students first
in the world in mathematics and science--by not emphasizing student
achievement.  Although it is the essence of Goal 5, increasing
student achievement is the key objective in only 7 of the 17 most
resource-intensive program projects.  The remaining 10 projects
generally seek to improve students' attitudes toward mathematics and
science and motivate them to eventually pursue science careers by
improving their perceptions of scientists. 

Moreover, Energy's evaluation process did not focus on student
achievement:  only one of the nine projects that were evaluated tried
to measure student achievement.  In fact, Energy's projects are
seldom clearly linked to National Education Goal 5.  For example, two
of seven program managers we interviewed were unfamiliar with the
National Education Goals.  In addition, none of the program managers
could demonstrate how their projects helped improve student
achievement.  Generally, these managers told us their projects aim to
increase mathematics and science literacy and promote science as a
career, not improve student achievement. 

   CONCLUSION
------------------------------------------------------------ Letter :7

In the early 1990s, Energy did not effectively manage its precollege
mathematics and science program.  First, the Department jeopardized
the program's success by not using a risk management strategy to
administer the program's projects.  Second, Energy forfeited an
invaluable management tool by taking a lax approach to program
evaluation.  Third, Energy greatly reduced its probability of helping
achieve National Education Goal 5 by implementing a variety of
projects that did not clearly seek to improve student achievement. 

In response to concerns raised during our review, Energy officials
announced plans to undertake several initiatives to substantially
improve this program's management and evaluation functions.  These
initiatives constitute an important step toward effective program
management.  However, the depth of executive support for these
initiatives and their subsequent staying power were uncertain at the
time our review was completed.  If ongoing changes in Energy's
management philosophy are fully implemented, needed program
improvements could ultimately result. 

   RECOMMENDATIONS TO THE
   SECRETARY OF ENERGY
------------------------------------------------------------ Letter :8

In continuing the Department's efforts to improve management of the
precollege mathematics and science education program, we recommend
that the Secretary of Energy strengthen its management role. 
Specifically, the Secretary should

  place greater emphasis on balancing the program by increasing the
     proportion of systemic reform projects;

  strengthen its evaluation component so that it serves as a basis
     for (1) improving projects; (2) making informed budget decisions
     about terminating, retaining, and expanding projects; and (3)
     measuring gains in student achievement; and

  restructure or discontinue all projects that do not clearly support
     National Education Goal 5--increasing students' mathematics and
     science achievement. 

---------------------------------------------------------- Letter :8.1

Department of Energy officials who reviewed a draft of this report
generally agreed with our findings.  As requested, unless you
publicly announce its contents earlier, we plan no further
distribution of this report until 30 days from its issue date.  At
that time, we will send copies of this report to the appropriate
House and Senate Committees, the Secretary of Energy, and other
interested parties.  If you or your staff have any questions about
this report, please call me on (202) 512-7014 or Cornelia Blanchette,
Associate Director, on (202) 512-8403.  The major contributors to
this report are listed in appendix V. 

Sincerely yours,

Linda G.  Morra
Director, Education and
 Employment Issues

PRECOLLEGE PROGRAM BUDGET
INCREASED SIGNIFICANTLY (FISCAL
YEARS 1990-93)
=========================================================== Appendix I

   (See figure in printed
   edition.)

Note:  Actual budget totals shown for fiscal years 1990-92. 
Appropriated budget totals shown for fiscal year 1993. 

OVERVIEW OF 17 MOST
RESOURCE-INTENSIVE PROJECTS
(FISCAL YEAR 1992)
========================================================== Appendix II

                                                          Fiscal
                                                            year
Energy                                                      1992  Implementation
facility        Project name       Project objective      budget  priority
--------------  -----------------  -----------------  ----------  --------------
Projects with no evaluation (eight projects)
--------------------------------------------------------------------------------
Ames            Adventures in      Introduce            $315,888  Teacher
Laboratory      Supercomputing     teachers to the                enhancement
(Iowa)                             use of high-
                                   performance
                                   computers in
                                   mathematics and
                                   science
                                   instruction.

Argonne         National Science   Provide teachers      693,000  Teacher
National        Explorers          with science                   enhancement
Laboratory                         videos,
(Illinois)                         instructional
                                   guides, and
                                   training
                                   workshops to
                                   improve science
                                   teaching.

Associated      Environmental      Provide students      615,014  Teacher
Western         Management         with an                        enhancement
Universities    Precollege         opportunity to                 and student
(Utah)          Analytical         take a college-                support
                Chemistry          level analytical
                                   chemistry course.

Lawrence        Bay Area Science   Provide hands-on      466,000  Systemic
Berkeley        & Technology       activities,                    reform
Laboratory      Education          curriculum
(California)    Collaboration      development,
                                   instructional
                                   materials, and
                                   training
                                   districtwide to
                                   enhance classroom
                                   teaching and
                                   learning of
                                   science,
                                   mathematics, and
                                   technology.

Oak Ridge       Science &          Provide teachers      244,000  Systemic
Institute for   Mathematics        and students with              reform
Science and     Action for         research
Education       Revitalized        opportunities,
(Tennessee)     Teaching           instructional
                                   materials, and
                                   technical support
                                   to increase the
                                   effectiveness of
                                   mathematics and
                                   science education
                                   districtwide.

Oak Ridge       Adventures in      Provide teachers      398,000  Teacher
National        Supercomputing     and students with              enhancement
Laboratory                         access to and                  and student
(Tennessee)                        training on high-              support
                                   performance
                                   computers to
                                   improve
                                   mathematics and
                                   science
                                   instruction.

Oak Ridge       Preparation &      Provide teachers      260,000  Teacher
National        Enhancement        with summer                    enhancement
Laboratory                         research
(Tennessee)                        opportunities and
                                   training to
                                   improve
                                   mathematics and
                                   science
                                   instruction.

Sandia          Science/Math       Provide                26,306  Teacher
National        Carnival           scientific                     enhancement
Laboratory-                        demonstrations to
Livermore                          improve
(California)                       mathematics and
                                   science teaching.

Subtotal                                              $3,018,208

Projects with evaluation (nine projects)
--------------------------------------------------------------------------------
Argonne         Chicago Science    Provide teachers     $531,000  Teacher
National        Explorers          and students with              enhancement
Laboratory                         hands-on                       and student
(Illinois)                         activities, field              support
                                   trips, and videos
                                   to improve
                                   science
                                   instruction and
                                   learning.

Associated      Teacher Research   Provide summer      1,779,269  Teacher
Western         Associates         research                       enhancement
Universities                       opportunities to
(Utah)                             teachers.

Brookhaven      Environmental      Provide students      199,211  Student
National        Education          with an                        support
Laboratory      Outreach for       opportunity to
(New York)      Minorities         take a college-
                                   level
                                   environmental
                                   science course.

Brookhaven      Northeast          Provide students      229,381  Student
National        Consortium for     with an                        support
Laboratory      Minorities         opportunity to
(New York)                         take college-
                                   level science
                                   courses.

Los Alamos      Students Watching  Provide             1,016,900  Teacher
National        Over Our Planet    materials,                     enhancement
Laboratory      Earth              teacher training,              and student
(New Mexico)                       and student                    support
                                   instruction on
                                   environmental
                                   concerns.

Lawrence        National           Provide access to     726,000  Teacher
Livermore       Education          and training on                enhancement
National        Supercomputer      high-performance               and student
Laboratory      Program            computers to                   support
(California)                       students and
                                   teachers.

Oak Ridge       Teacher Research   Provide summer        230,600  Teacher
Institute for   Associates         research                       enhancement
Science and                        opportunities for
Education                          teachers.
(Tennessee)

Pacific         OPTIONS in         Provide students      915,370  Systemic
Northwest       Science            with high-                     reform
Laboratory                         quality
(Washington)                       mathematics and
                                   science education
                                   by enhancing
                                   teachers'
                                   instructional
                                   strategies and
                                   ability to
                                   develop
                                   curriculum
                                   through a
                                   statewide
                                   systemic reform
                                   effort.

Sandia          Science Advisors   Provide a           2,500,000  Teacher
National                           scientist in the               enhancement
Laboratory-                        school who offers              and student
Albuquerque                        technical                      support
(New Mexico)                       assistance to
                                   teachers and
                                   students and
                                   participates in
                                   activities to
                                   support science
                                   (i.e., science
                                   fairs).

Subtotal                                              $8,127,731

Total                                                 $11,145,93
                                                               9
--------------------------------------------------------------------------------

PERCENT CHANGE IN BUDGET FOR
ENERGY'S
17 MOST RESOURCE-INTENSIVE
PROJECTS
(FISCAL YEARS 1990-93)
========================================================= Appendix III

                                                       Per
                                                       cen                Perc
                                                        t                 ent                 Percen
                                      Bud              cha                chan                     t
Energy facility     Project name      get      Budget  nge        Budget   ge         Budget  change
------------------  ----------------  ---  ----------  ---  ------------  ----  ------------  ------
Projects with no evaluation (eight projects)
----------------------------------------------------------------------------------------------------
Ames Laboratory     Adventures in                               $315,888  N/A       $650,000     106
 (Iowa)              Supercomputing
Argonne National    National Science         $318,000  N/        693,000  118        507,000     -27
 Laboratory          Explorers                          A
 (Illinois)
Associated Western  Environmental             225,117  N/        615,014  173        601,870      -2
 Universities        Management                         A
 (Utah)              Precollege
                     Analytical
                     Chemistry
Lawrence Berkeley   Bay Area Science          288,300  N/        466,000   62        515,000      11
 Laboratory          & Technology                       A
 (California)        Education
                     Collaboration
Oak Ridge           Science &         24,     279,100  1,0       244,000  -13        260,000       7
 Institute for       Mathematics      100              58
 Science and         Action for
 Education           Revitalized
 (Tennessee)         Teaching
Oak Ridge National  Adventures in                                398,000  N/A        920,000     131
 Laboratory          Supercomputing
 (Tennessee)
Oak Ridge National  Teacher                                      260,000  N/A        260,000     N/A
 Laboratory          Preparation &
 (Tennessee)         Enhancement
Sandia National     Science/Math                2,600  N/         26,306  912         29,000      10
 Laboratory -        Carnival                           A
 Livermore
 (California)
Subtotal                              $24  $1,113,117  4,5    $3,018,208  171     $3,742,870      24
                                      ,10              19
                                       0

Projects with evaluation (nine projects)
----------------------------------------------------------------------------------------------------
Argonne National    Chicago Science   $61    $570,000  -7       $531,000   -7       $750,000      41
 Laboratory          Explorers        3,0
 (Chicago)                            00
Associated Western  Teacher Research  525   1,105,519  110     1,779,269   61      1,731,326      -3
 Universities        Associates       ,79
 (Utah)                                1
Brookhaven          Environmental     93,     125,358  34        199,211   59        168,500     -15
 National            Education        351
 Laboratory (New     Outreach for
 York)               Minorities
Brookhaven          Northeast                 122,503  N/        229,381   87        249,700       9
 National            Consortium for                     A
 Laboratory (New     Minorities
 York)
Los Alamos          Students          85,     750,000  782     1,016,900   36        289,000     -72
 National            Watching Over    000
 Laboratory (New     Our Planet
 Mexico)             Earth
Lawrence Livermore  National                  340,000  N/        726,000  114        403,000     -45
 National            Education                          A
 Laboratory          Supercomputer
 (California)
Oak Ridge           Teacher Research  48,     139,700  186       230,600   65        193,000     -16
 Institute for       Associates       900
 Science and
 Education
 (Tennessee)
Pacific Northwest   OPTIONS in        50,     629,762  1,1       915,370   45        374,033     -59
 Laboratory          Science          000              60
 (Washington)
Sandia National     Science Advisors  545   1,400,000  157     2,500,000   79      2,200,000     -12
 Laboratory -                         ,00
 Albuquerque (New                      0
 Mexico)
====================================================================================================
Subtotal                              $1,  $5,182,842  164    $8,127,731   57      6,358,559     -22
                                      961
                                      ,04
                                       2
====================================================================================================
Total                                 $1,  $6,295,959  217   $11,145,939   77    $10,101,429      -9
                                      985
                                      ,14
                                       2
----------------------------------------------------------------------------------------------------
Note:  N/A represents not applicable. 

EVALUATION REPORT RESULTS FOR NINE
PROJECTS WITH EVALUATIONS
========================================================== Appendix IV

   (See figure in printed
   edition.)

MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V

Wayne B.  Upshaw, Assistant Director
Valerie Giles-Reynolds, Assignment Manager, (313) 256-8000
Lemuel Jackson, Evaluator-in-Charge
Ella Cleveland
Joel Grossman
Revae E.  Steinman
Yelena K.  Thompson

BIBLIOGRAPHY
=========================================================== Appendix 0

Begle, Edward.  Teacher Knowledge and Student Achievement in Algebra. 
School Mathematics Study Group.  (Palo Alto, Calif.:1972). 

Darling-Hammond, Linda.  "Enlarging the Knowledge Base--Are Our
Teachers Ready To Teach?" National Council for Accreditation of
Teacher Education, 1991. 

Darling-Hammond, Linda, and Lisa Hudson.  Indicators for Monitoring
Mathematics and Science Education:  A Sourcebook.  ed.  Shavelson,
McDonnell, and Oakes.  RAND and the National Science Foundation,
1989. 

Druva, Cynthia A., and Ronald D.  Anderson, "Science Teacher
Characteristics by Teacher Behavior and by Student Outcome:  A
Meta-Analysis of Research." Journal of Research in Science Teaching,
Vol.  20, No.  5 (1983), pp.  467-479. 

Fitzsimmons, Stephen J.  A Study of NSF Teacher Enhancement Program
(TEP) Participants and Principal Investigators:  1984-1989 (DRAFT)
Vol.  I:  Summary Report.  Abt Associates, Inc.  for the National
Science Foundation.  Cambridge, Mass.:  1993. 

Juster, Thomas, F.  Precollege Science and Mathematics Teachers: 
Monitoring Supply, Demand and Quality.  National Research Council,
Committee on National Statistics, Commission on Behavioral and Social
Sciences and Education, 1990. 

Lawrenz, Frances.  "The Relationship Between Science Teacher
Characteristics and Student Achievement and Attitude," Journal of
Research in Science Teaching, Vol.  12, No.  4 (1975), pp.  433-37. 

McKinney, Kay.  Improving Math and Science Teaching.  U.S. 
Department of Education, Office of Educational Research and
Improvement.  Washington, D.C.:  1992. 

Miller, Jon.  Enriching Middle School Science:  A Final Evaluation of
the 1991-92 Columbia College Workshops Utilizing an Innovative
Approach to the Teaching of Science.  Chicago Academy of Sciences. 
Chicago:  1994. 

Murnane, Richard J., and Senta A.  Raizen, Improving Indicators of
the Quality of Science and Mathematics Education in Grades K-12. 
National Research Council, Committee on Indicators of Precollege
Science and Mathematics Education, Commission on Behavioral and
Social Sciences and Education, 1988. 

Raizen, Senta, and Theodore Britton, "Science and Mathematics
Teachers," Indicators of Science and Mathematics Education--1992. 
National Science Foundation.  (NSF 93-95).  Washington, D.C.:  1993. 

Rothman, Arthur I.  "Teacher Characteristics and Student Learning,"
Journal of Research in Science Teaching, Vol.  6 (1969), pp.  340-48. 

Smith, Marshall, S., and Jennifer O'Day, "Systemic School Reform."
Politics of Education Association Yearbook, 1990, pp.  233-267. 

U.S.  General Accounting Office.  New Directions for Federal Programs
to Aid Mathematics and Science Teaching, GAO/PEMD-84-5.  Washington,
D.C.:  1984. 

U.S.  General Accounting Office.  Systemwide Education Reform: 
Federal Leadership Could Facilitate District-Level Efforts,
GAO/HRD-93-97.  Washington, D.C.:  1993. 


```