Evaluation of the 

Appalachian Regional Commission 

Oak Ridge National Laboratory 

Summer Institute for Math/Science/Technology 

VMarch 2006 

Evaluation of the 

Appalachian Regional Commission 

Oak Ridge National Laboratory 

Summer Institute for 

Math/Science/Technology 

March 2006 
Evaluation of the 

Appalachian Regional Commission 

Oak Ridge National Laboratory 

Summer Institute for Math/Science/Technology 

March 2006 

Prepared by 

Linda S. Simkin, M.Phil. 

Valerie Futch, B.S. 

 

 

Prepared for 

Appalachian Regional Commission 

1666 Connecticut Avenue, NW 

Suite 700 

Washington DC 20009

Evaluation of the 

Appalachian Regional Commission 

Oak Ridge National Laboratory 

Summer Institute for 

Math/Science/Technology 

March 2006 
 

Acknowledgements 

Many people contributed to this evaluation and report. We would like to begin by thanking 

Catherine Saldutti, president of EduChange in New York City and valued expert consultant for 

this project. Ms. Saldutti provided knowledgeable guidance about STEM programs for students 

and teachers as well as about the current state-of-the-art in evaluating these programs. Alexandra 

Weinbaum, vice president and director of AED’s Center for School and Community Services 

also provided expert guidance throughout the evaluation and valuable insights about national 

efforts to improve college-access and success for underrepresented populations. In addition, we 

would like to thank members of AED’s staff for the technical expertise they provided on this 

evaluation. Eliana Orellana and Wanda Dallas provided assistance with database development 

and data entry, and Adria Gallup-Black conducted the statistical analyses. Elayne Archer, who 

was responsible for editing this report, brought to it both her deep knowledge of educational 

assessment and her keen attention to language. We would also like to acknowledge the ARCORNL participants who took the time to complete surveys and provide thoughtful feedback in 

interviews at the time of the year when holidays and end-of-semester assignments competed for 

their attention. 

Finally, we thank Jeffrey Schwartz for his ongoing and knowledgeable assistance. The site visit 

to the ARC-ORNL Summer Institute in 2005 that he arranged for the AED evaluator was 

especially valuable. The background he provided, as well as meetings with staff of the ORNL 

and the Oak Ridge Institute for Science and Education he arranged, informed the evaluation. We 

appreciated being able to have informal conversations with the 2005 participants and were 

impressed by the palpable enthusiasm of students, teachers, chaperones and mentors. 

Linda Simkin, M.Phil. 

Valerie Futch, B.S. 

Academy for Educational Development 

New York City 

March 2006 
 

Highlights 

This report describes some of the key immediate and long-term outcomes achieved by the 

Appalachian Regional Commission (ARC)–Oak Ridge National Laboratory (ORNL) Summer 

Institute for Math/Science/Technology for student and teacher participants. This two-week 

summer program provides high school students and teachers from the Appalachian region the 

opportunity to work with mentor scientists from ORNL on inquiry-based, applied projects in 

science, math, and computer technology. The institute culminates with group, student and 

teacher, presentations about their projects. Other planned activities are designed to promote 

teamwork, expose students to college opportunities, and promote pride in the cultural richness 

and historical importance of the Appalachian region. 

The goals of the Summer Institute, in operation since 1990, are to 1) encourage more high school 

students to continue their studies beyond high school, 2) encourage more students to pursue 

careers in the projected shortage areas of science, technology, engineering, and math (STEM), 

and 3) raise the level of math, science, and technology instruction in high schools throughout the 

region to facilitate the first two goals. 

The findings in this report are the result of an evaluation conducted by staff of the Academy for 

Educational Development, in which data were collected from eight cohorts of participants 

attending the Summer Institute between 1997 and 2004. The evaluation took place in 2005-06 

and is based on surveys from 89 students and 67 teachers, as well as interviews with a small 

number of participants. The evaluation has several limitations, including absence of a 

comparison group that would have made it possible to attribute outcomes solely to participation 

at the institute. Also, this evaluation was able to collect data only after participation in the 

institute, whereas with a carefully constructed pre-post comparison, the evaluation could have 

measured changes in student/teacher knowledge, attitudes, and intentions. 

Key findings for students and teachers follow. 

Students 

• Male and female students from each of the 13 states in the region participated in the Summer 

Institutes, and 31% of students came from counties designated as economically distressed by 

ARC. 

• Seventy-four percent of students said they felt more confident in their STEM abilities as a 

result of the institute. 

• Participation in the Summer Institute influenced 24% of students to take more science classes 

and 22% to take more math classes when they returned to high school. Somewhat more than 

half the students reported that their experience at ORNL reinforced prior decisions about the 

science and math courses they had already chosen to take (56% and 52%, respectively). 

• All the student-survey respondents reported that, even prior to attending the Summer 

Institute, they planned to attend college. Even so, the students reported many ways that the 

institute reinforced their intentions to go to college and reduced some of the barriers. More 

than 50% of students reported that the program positively influenced their intentions to go to 

college. 
 

• Fully 96% of student participants who had graduated from high school at the time of the 

survey had continued their formal education. Of the 83 respondents who provided data, a 

great majority (79%) went to four-year institutions; 15 % went to community colleges or 

technical centers; 2% to military academies; and 3% did not continue their education beyond 

high school. 

• Of the 23 students who attended the institute in summer 1997 and 1998, all reported having 

attained higher education—26% had some college, 39% had earned a bachelor’s degree, and 

35% went on to graduate work. 

• After the Summer Institute, 36 students went on to earn a total of 46 degrees from higher 

education institutions. Overall, 54% of degrees earned were in STEM—38% of associate 

degrees, 52% of bachelor degrees, and 86% of graduate degrees. 

• In total, 51 student-survey respondents indicated that they were currently pursuing post-high 

school education; 82% were pursuing degrees in STEM fields, including core subjects, 

health, and computer sciences. 

• Fifty-five percent of participants who were employed full-time reported that their jobs 

required proficiency in one of the STEM fields “to a great extent.” 

• The highlight of the experience named by students most often was interacting with people, 

especially their peers, from different regions of the country and meeting the ORNL scientists. 

• Many students reported that the Summer Institute experience helped them gain more mature 

perspectives socially and academically, an observation that was corroborated by teachers. 

According to many students, it was particularly significant to become aware that “there were 

others like me.” 

• Fifty-nine percent of students currently reside in the Appalachian region, and 53% 

anticipated residing in the region five years hence. There was no difference in intention to 

remain in the region among those who had majored in STEM or were currently majoring in 

STEM. 

Teachers 

• Teachers from each of the 13 states in the Appalachian region participated; 47% of teacher 

participants were from ARC-designated distressed counties. 

• Many teachers reported that they incorporated activities and approaches learned at the 

Summer Institute into their classrooms. When they returned to their classrooms, 77% drew 

on their experience at the institute for explanations and examples, 52% did so for classroom 

demonstrations, and 50% incorporated new knowledge in their lab experiments. 

• Eighty-six percent of teachers said that, following the institute, they encouraged students to 

continue their education in science, math or technology. 
 

• Sixty-nine percent reported talking to students in their classrooms about research applications 

in the STEM areas. 

• Seventy-five percent reported that they shared their experience and things they had learned 

with other teachers through informal conversations. Four teachers who gave formal 

presentations to colleagues estimated reaching a total of 95 teachers. 

• Fifty-seven percent of teachers said the institute encouraged them to seek more professional 

development in STEM. 

• Of the benefits of attending the Summer Institute named by teachers, they most often 

described the opportunity of meeting and networking with other teachers from the region. 

• Many interviewed teacher participants desired a more effective professional development 

component that would provide them with lesson plans and resources to better integrate their 

ORNL experience into their own classrooms. 

• Ninety-one percent of teachers currently reside in the Appalachian region, and 93% said they 

planned to reside in the region five years hence. 

Based on the findings, the evaluation report offers recommendations to help the institute fulfill 

its purpose more deliberately and effectively. The first recommendations listed below address the 

program overall, and then address specific student and teacher components of the program. 

Overall Program 

• Clarify recruitment criteria and process for students and teachers and make these criteria 

known to participants as well as the groups that recruit them. 

• Increase the number of youth from underrepresented groups in STEM. 

• Create connections with college-access programs in the region to expand the applicant pool 

and support student success over the long term. 

• Continue to conduct a mixed-method evaluation using a design that incorporates data 

collection before, and at two points following, participation in the Summer Institute. This 

will necessitate better student tracking.

Student Components 

• Match student interests with projects or, if this is not possible, provide students with the 

opportunity to sample many projects. 

• Provide opportunities for students to meet or work with graduate students and young 

professionals. 

• Foster peer communication during and after the institute, both within and among different 

cohorts. 
 

• Create connections between the Summer Institute, sending school, and parents to support 

participants’ progress when they return home. 

Teacher Components 

• Give more attention to curricular issues by providing more structured time for teachers to 

discuss both specific curricular implications as well as ways to continue to promote interest 

in STEM education and careers among their students. 

• Foster networking among teachers from multiple years of the Summer Institute. 

• Make explicit expectations that teachers share their experience with other teachers in their 

schools and districts. 
 

Table of Contents 

1. Introduction............................................................................................................................... 1

1.1 Background........................................................................................................................... 1 

1.2 Description of the Appalachian Region................................................................................ 2 

1.3 Description of the Summer Institute..................................................................................... 3 

2. Evaluation of Other High School STEM Enrichment Programs......................................... 5

2.1 Evaluation Approaches ......................................................................................................... 5 

2.2 STEM Evaluation Results..................................................................................................... 6 

2.3 Resources .............................................................................................................................. 8 

3. Evaluation Methodology .......................................................................................................... 9

3.1 Approach and limitations...................................................................................................... 9 

3.2 Student and Teacher Surveys.............................................................................................. 10

3.3 Student and Teacher Interviews.......................................................................................... 10 

3.4 Study Population................................................................................................................. 10 

3.5 Response Bias ..................................................................................................................... 11 

4. Student Findings ..................................................................................................................... 14

4.1 Description of Student Respondents................................................................................... 14 

4.2 Overall Assessment of the Summer Institute...................................................................... 14 

4.3 Influence of Peer Support ................................................................................................... 16 

4.4 Influence of Mentors, Chaperones, and Teachers............................................................... 17 

4.5 Changes in Attitudes about School and Subjects Studied .................................................. 17 

4.6 Pursuit of STEM Education and Employment.................................................................... 19 

4.7 College Aspirations and Planning....................................................................................... 21 

4.8 College Enrollment ............................................................................................................. 23 

4.9 Workforce Participation...................................................................................................... 26 

4.10 Student Recommendations................................................................................................ 28 

5. Teacher Findings..................................................................................................................... 30

5.1 Teacher Respondents .......................................................................................................... 30 

5.2 Overall Assessment............................................................................................................. 32 

5.3 Impact on Teaching............................................................................................................. 33 

5.4 Barriers to Bringing ORNL Experiences to Classrooms .................................................... 36 

5.5 Pursuit of Professional Development.................................................................................. 37 

5.6 Impact on Other Teachers................................................................................................... 37 

5.7 Appalachian Region Employment ...................................................................................... 37 

5.8 Teacher Recommendations................................................................................................. 38 

6. Recruitment............................................................................................................................. 40

6.1 Teacher Recruitment........................................................................................................... 41 

6.2 Student Recruitment............................................................................................................ 42 

7. Review of Selected Pre-College STEM Programs................................................................ 43
 

8. Conclusions and Recommendations...................................................................................... 48

8.1 Recommendations about Program Objectives and Recruitment ........................................ 48 

8.2 Student-Specific Recommendations ................................................................................... 51 

8.3 Teacher-Specific Recommendations .................................................................................. 53 

8.4 Recommendations for Future Evaluation ........................................................................... 54 

Appendices...................................................................................................................................... i
 

Tables 

Table 1—Percentage of students who became science majors post-program 

 in selected STEM programs................................................................................................... 7 

Table 2—Comparison of respondents with all attendees and with attendees 

 who were located but who did not respond to the survey–students..................................... 12 

Table 3—Comparison of respondents with all attendees and with attendees 

who were located but who did not respond to the survey–teachers...................................... 13 

Table 4—Selected characteristics of student survey respondents ................................................ 15 

Table 5—Parents’ highest level of formal education ................................................................... 15 

Table 6—Students’ ratings of the extent to which the Summer Institute 

influenced awareness, confidence and interest in STEM ..................................................... 19 

Table 7—Students’ ratings of the extent to which various aspects of the 

Summer Institute positively influenced their thinking about going to college..................... 22 

Table 8—Mean scores for students’ ratings of the extent to which 

they received encouragement to attend college from the following:.................................... 23 

Table 9—Students’ highest educational attainment ..................................................................... 24 

Table 10—Major fields of study................................................................................................... 24 

Table 11—Degrees pursued or earned in STEM and other fields of study.................................. 25 

Table 12—Workforce status of student participants .................................................................... 26 

Table 13—Students’ opinions about STEM proficiency requirements of 

current jobs and future career ............................................................................................... 27

Table 14—Students who currently live in the Appalachian Region (AR) 

and those who plan on living there in 5 years, by cohort ..................................................... 27 

Table 15—Selected Characteristics of Teacher Survey Respondents .......................................... 31 

Table 16—Educational attainment of teachers............................................................................. 32 

Table 17—Research experience prior to attending the Summer Institute .................................... 32 

Table 18—Teachers who reported incorporating Summer Institute experiences 

in their classrooms. ............................................................................................................... 34 

Table 19—Teachers who gave high ratings to selected outcomes ............................................... 36 

Table 20—Percentage of teachers who encountered various obstacles........................................ 36 

Table 21—Current teaching status and place of employment ...................................................... 38 
 

Figure 

Figure 1: Employer ........................................................................................................................26 

Appendix Tables

Table A1—Percentage of students and teachers residing in Appalachian 

 states when they attended the Summer Institute........................................................................... iii 

Table A2—Selected characteristics of student interviewees......................................................... iv 

Table A3—Selected characteristics of teacher interviewees......................................................... iv 

Table A4—First educational institution attended after high school ................................................v 

Table A5—Highest level of education attained...............................................................................v 
 

1

1. Introduction 

1.1 Background 

The Appalachian Regional Commission (ARC)–Oak Ridge National Laboratory (ORNL) 

Summer Institute for Math/Science/Technology is a partnership administered by the Oak Ridge 

Institute for Science and Education (ORISE), a program of the Oak Ridge Associated 

Universities. 

The Summer Institute was designed to address the declining availability of a scientifically 

literate workforce in the U.S., as well as to improve professional development for teachers in 

order to increase student achievement in these areas. Census data described below indicate that 

many counties within the Appalachian region lag behind the national average in college 

completion rates. ARC, seeking to promote economic development in the region, has funded this 

and other programs to increase the “intellectual capital” of the area. 

The institute’s three main goals are 1) encourage more high school students to continue their 

studies beyond high school, 2) encourage more students to pursue careers in the projected 

shortage areas of science, technology, engineering, and math (STEM),1 and 3) raise the level of 

STEM instruction in high schools throughout the region to facilitate the first two goals. An 

unwritten but underlying goal of the program is to encourage students who obtain higher 

education, especially in STEM fields, and teachers who have achieved higher competencies in 

STEM, to remain in the Appalachian Region. To assess the results of its more than $2 million 

investment in the Summer Institute since its inception 16 years ago in 1990, ARC funded this 

evaluation to determine the extent to which these goals have been achieved. The evaluation 

sought to measure short- and longer-term outcomes aligned with the Summer Institute’s goals, 

including the following for students and teachers: 

Students

• Participation in STEM courses while in high school 

• High school completion 

• College-going 

• Degrees pursued and earned in STEM and non-STEM fields 

• Employment in jobs requiring STEM proficiency 

Teachers 

• Incorporating activities and approaches learned at the Summer Institute into the classroom 

and sharing new knowledge with other teachers 

• Encouraging students to pursue higher education in STEM fields 

 

1

 Throughout this report, for simplicity, we use the acronym STEM (science, technology, engineering and 

math) for all programs addressing at least three of these fields. 
 

2

• Pursuing professional development 

This evaluation was designed to illuminate how the Summer Institute influenced these outcomes. 

Because it was not possible to measure longer-term outcomes for students who were still in 

college, the evaluation sought to measure attitudes and intentions that might mediate these 

outcomes. ARC also asked that the evaluation compare findings with outcomes obtained by 

similar programs. As will be discussed in section 2 below, such comparisons were generally 

unavailable, both because it is difficult to find comparable programs (e.g., in terms of target 

population, program duration and approach), and because few evaluations, if conducted, publish 

their findings. Finally ARC asked AED evaluators to make recommendations for establishing an 

ongoing evaluation capability for the ARC-ORNL Summer Institute. These recommendations 

can be found in the appendix to this report. 

1.2 Description of the Appalachian Region 

As a context for this report, it is important to understand the Appalachian region, which was 

established in federal law and comprises 410 counties in West Virginia and in parts of 12 other 

states extending from the southern tier of New York State to northeast Mississippi. The 

economy, once dependent on heavy industry, agriculture, and resource-extraction, is now more 

diversified and reliant on service-sector employment. The population of 22.8 million is 

distributed across the region, with the largest percentage in metropolitan counties (56%), 27% in 

counties adjacent to metropolitan areas, and 17% in rural counties. The regional poverty rate of 

13.6 is 1.2 percentage points above the national average.2

 

According to ARC, the region faces educational challenges, including low rates of educational 

attainment and college completion: 

Recent research on educational attainment has revealed a widening gap between 

the nation’s and Appalachia’s educational attainment rates. Detailed census data 

on young adults tell two stories. The high school educational attainment gap has 

closed even further across Appalachia, but particularly in the north, as well as 

Northern Alabama. The college completion rate, with the exception of Northern 

Alabama has not narrowed, even after controlling for age. 

Looking at county-level changes, only 18 of the 410 counties have higher college 

attainment rates than the nation, and most are in college towns. Intrastate 

differences show that Appalachian counties have lower college attainment rates 

than every state except Alabama and South Carolina. With respect to gender, 

Appalachian trends parallel the nation, with women improving their educational 

attainment rates compared to men, especially in completing Associate’s degrees. 

Because Appalachia’s college-age population is forecast to grow little over the 

next decade, the window of opportunity for the region is to improve the collegegoing rates and to provide further educational opportunity among those with no 

college or some college to attain a degree.3

 

2

 Appalachian Regional Commission. 2005. Request for proposals for An Evaluation of Outcomes from 

the Appalachian Regional Commission–Oak Ridge National Laboratory, Summer 

Math/Science/Technology Institutes. Washington DC. 

3

 Ibid. 
 

3

1.3 Description of the Summer Institute 

The ARC–ORNL Summer Institute for Math/Science/Technology is one of many programs 

funded by the Appalachian Regional Commission in pursuit of its mandate to improve education, 

economic conditions, and leadership within the region. The institute is held each July at the Oak 

Ridge National Laboratory in Oak Ridge, TN. The institute, initially managed by the ORNL 

education office, has been administered by staff of the Oak Ridge Institute for Science and 

Education since 2001. Participation is open to students and teachers from all designated counties 

in the region. The governor of each state proposes nominees and the final selection of 

participants is made by ARC staff. 

Held during two weeks in July, this program does not undertake any formal follow-up with 

schools or participants. The following description of the program, edited in places in line with 

the purpose of this report, appeared in the ARC request for proposals for this evaluation in 2005. 

The institute has openings for 52 participants each year—26 students and 26 teachers, or two 

students and two teachers from each state in the Appalachian region. An additional eight 

participants may attend if states cover their costs. Teachers and students are divided into separate 

research groups, and the teachers have no formal responsibility for the students. These small 

groups work with one or more of the ORNL scientists, who are designated as “mentors.” 

Scientists may be current or past ORNL employees. For two weeks, the scientists guide assigned 

groups of students or teachers on projects that are either designed as learning experiences or are 

part of ongoing research needed by ORNL or one of its clients. Topics in 2003 and 2004 

included: 

• Robotic systems and engineering development 

• Simulations of combined cooling, heating, and power systems 

• Building envelope technologies 

• Properties of a magnetic ion beam-steering device 

• Fusion energy 

• Calibration and testing of neutron focusing mirrors 

• Stream ecology 

• Global climate change 

• Nano-biosensor technology and human genome bioinformatics 

• Collaborative research and its implications 

• Website design and development 

Four resident teachers employed by ORISE supervise students during non-institute hours. These 

chaperones work with students in the evenings to help them understand the applied research they 

have been conducting and help them design the final presentation all groups make about their 
 

4

projects. Mentor teachers work with teacher participants to strategize about ways they can bring 

their experience back to the classroom. 

The ORNL experience is enriched by activities outside the lab. To build teamwork, an essential 

part of the research environment, participants are encouraged to complete a ropes course and 

participate in social events, such as a trip to a Smokies baseball game and Dollywood. There are 

also several science/education field trips each year to such places as the American Museum of 

Science and Technology and Tennessee Aquarium and IMAX theater, and the University of 

Tennessee and Maryville College campuses. The Summer Institute experience also includes a 

cultural activity designed to develop pride in Appalachia. For example, in previous years, Bill 

Landry, host of The Heartland Series, has lectured to participants. Institute staff have also 

arranged trips to the Museum of Appalachia in Norris, TN. 

The teachers and students attend at the same time, stay at the same hotel, and have meals and 

attend cultural and social functions together. Students and teachers have their travel, lodging, 

meal and related expenses paid. Teachers receive a small stipend. 
 

5

2. Evaluation of Other High School STEM Enrichment 

Programs 

This section of the report summarizes ways that STEM programs have approached evaluation, 

and some of the pitfalls that prevent the field from capturing truly reliable and cross-cutting data. 

Where possible, we have included literature on the successes of these types of programs. It is 

important to state upfront that published results that indicate participant progress in the sciences 

or overall program impact are difficult to find and, when found, are difficult to compare with 

results of other programs. Many individual STEM programs are either not required to publish 

their evaluation data or cannot publish evaluations because they have not received the 

appropriate human subjects permissions. However, some well-documented evaluations of precollege STEM programs do exist, and some resources cited in section 7 of the report use 

anecdotal data to identify “what works.” 

2.1 Evaluation Approaches 

Most programs conduct evaluations because they are required by a funder, because the program 

directors are interested in learning how their efforts are being received by the participants, and/or 

because the directors wish to assess long-term impacts. The National Science Foundation (NSF) 

usually requires that programs devote 10-15% of their overall budget to evaluation activities. 

Increasingly, even small funding organizations are requiring evaluations. This research is often 

conducted by third parties, mainly to enhance objectivity, but also because program directors 

often have competing demands for their time and lack expertise in program evaluation. 

STEM programs engage in two main types of evaluations—summative and formative

evaluations. Summative evaluations assess the impact of specific, measurable program goals, 

some of which may have been jointly set with the funder. Results are generally quantitative, but 

often include qualitative data that have been collected over the duration of the program. Highquality summative evaluations are generally expensive and have limitations, for example, the 

absence of comparison groups that would allow attribution of results to a specific intervention. 

However, if the studies use standard evaluation methods and a funder is willing to pay for 

dissemination, the results may be published. Compared with summative evaluation, formative 

evaluation of STEM programs is conducted with higher frequency. The insights gathered during 

a formative evaluation are meant to inform future iterations of the program rather than inform the 

field. In order to assist STEM programs in understanding both the value and process of program 

evaluation, NSF released an evaluation handbook for program directors in 1993.4

The Howard Hughes Medical Institute (HHMI) also funds several programs nationwide, 

including some of the ones included in this review. In July 2005, the first study on the evaluation 

of these programs was archived.5

 HHMI wanted to evaluate evaluation; literally, it wanted to 

determine if there were any consistencies across a set of pre-college STEM programs with highly 

diverse purposes, target populations, and operations. HHMI also wished to learn how evaluations 

were conducted. Participants in 35 pre-college STEM programs for both teachers and students 

 

4

 Frechtling, J.A. (1993) User-Friendly Handbook for Project Evaluation: Science Mathematics, 

Engineering and Technology Education. VA: The National Science Foundation.

5 http://www.nahsep.org/study.html 
 

6

and a control group of programs, which were selected as finalists in the proposal review process 

but ultimately were not awarded a grant, completed surveys and responded to questions about 

their evaluations. This particular study of the evaluation of STEM programs is the most recent 

and most comprehensive study of its kind. Below are some notable highlights, although the 

entire study is worthy of examination: 

Almost all of the sites (n=31) were utilizing surveys/questionnaires. Several sites 

(n=27) were conducting observations (one project director indicated that the site 

had conducted informal observations following the implementation of a 

neuroscience curriculum in participating teachers' classrooms). Twenty-one sites 

conducted interviews; some of these were informal, representing a way to revise 

project components to be more effective in the classroom. Nineteen sites were 

using performance measures/participant portfolios, and 12 conducted focus 

groups. 

[At one site] project staff are still grappling with what key elements of their 

program most influence an increase in student achievement at the 15 schools with 

which they are working.6

General indicators of program success were shared by the NSF subset of STEM programs known 

as the Research Experiences for Teachers (RET) programs, where teachers are invited to work 

alongside scientists in the lab or in the field.7 According to participants, success seems to be 

centered on two main indicators—collaboration with university scientists and/or fellow educators 

and increased confidence in, and awareness of, the scientific enterprise. 

2.2 STEM Evaluation Results 

Published results that demonstrate participant progress in the sciences or overall program impact 

are difficult to find and, when found, are difficult to compare with results of other programs, due 

to limitations such as the following: 

• Almost no baseline data are gathered by programs across the literature. Baseline 

demographic, attitudinal and other pre-participant data, including data for the entire applicant 

pool could answer questions such as: Who applies? Who was chosen? Why? What changes 

in attitudes and intentions occur over time? 

• The impact of STEM programs on the attitudes, aptitudes, and behaviors of participants may 

not manifest during the program itself, yet that is when most evaluation activities are 

conducted in order to gain the highest response rate for surveys, focus groups, and 

interviews.8

 

 

6

 Ibid. 

7 Emily Driscoll, RET at Northeastern University (2004) http://www.ret.neu.edu/NSTADallas/Carousel.pdf

8

 Frechtling, J.A. (1993) User-Friendly Handbook for Project Evaluation: Science Mathematics, 

Engineering and Technology Education. VA: The National Science Foundation. 
 

7

• Participants’ self-reported data often exaggerate the positive effects of a given program 

because they have invested time and energy to participate.9

 Although this does not negate the 

value of probing participants about their perceptions of certain tactics, strategies or 

presentations, results need to be interpreted with caution. 

• When data are collected over a considerable length of time, consideration may not be given 

to collecting the data with standardized instruments and interviews that will allow results to 

be pooled over time and allow correlations to be computed. 

Nonetheless, some evaluation literature has been published despite these limitations. The 

following results from several programs are particularly worthy of mention. The HHMI study 

that queried its own program pool about evaluation practices also collected data on student 

motivation to study sciences. (See Table 1 for findings and a comparison with the national 

averages). Even though the motivating factors for students to pursue science in later educational 

levels are numerous, and sometimes even undetectable, HHMI did make an effort to include “the 

data only if they included a control group of similar participants or if results on the same students 

were collected before and after the intervention to demonstrate the change.”10 

Table 1—Percentage of students who became science majors post-program 

in selected STEM programs

Grantee % 

University of Cincinnati College of Medicine 83 

University of Nevada School of Medicine 63 

Robert C. Byrd Health Sciences Center of 

West Virginia University 

59 

University of Mississippi Medical Center 59 

Cleveland Clinic 53 

National average 32 

National average for underrepresented minorities 

(most program participants) 

5 

It is interesting that many of these programs accept participants through a competitive 

recruitment process, tapping those who already have demonstrated aptitude or interest in the 

sciences. It is unclear how far along in their science studies the students were when each survey 

was taken. 

 

9

 Cameron, J. & Pierce, D. P. (1994). “Reinforcement, Reward, and Intrinsic Motivation: A Metaanalysis.” Review of Educational Research, 64 (3), 363-423. 

10 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=520842 
 

8

Evaluation results from a 1993 study of the Dartmouth Thayer School of Engineering program 

showed that 75% of the 1993 teacher participants conducted an engineering experience or parts 

of that experience in their own classrooms and schools, using the pedagogy presented in the 

summer program. Further, 75% of participants made presentations to their colleagues about their 

program and classroom experiences within 16 months of the summer session, reaching an 

additional 974 teachers. The evaluation also found that 75% of student participants implemented 

the Dartmouth/Thayer problem-solving methods upon return to their high schools.11 These 

findings show that because teachers were able to replicate this specific pedagogical practice, the 

goals and intent of this program were achieved. 

In 1998, a multsite study of Scientific Work Experience Programs for Teachers (SWEPT) was 

funded with a four-year $1.6 million grant by NSF. The goals of this study were to measure the 

effect of SWEPT on students in the classroom, in a way that helps identify key variables, 

regardless of geographic location or particular facilities/personnel. The College of Physicians 

and Surgeons at Columbia University, coordinated this eight-site effort, summarized briefly 

below: 

Data has been collected on the more than 30,000 students who have been in the 

classes of participating teachers since 1993 and on approximately 600,000 

students in the science classes of nonparticipating teachers in the same schools 

and science departments.

The researchers found a three-fold increase in the number of students of 

participating teachers who undertake a competitive science project. The number 

of students participating in after-school science programs has grown from about 

10 percent to about 13 percent in the classes of participating teachers, while the 

average in classes of non-participating teachers remained about the same at 3.5 

percent. They also found a significant increase in the number of students of 

participating teachers who passed the science Regents exams. The researchers 

plan to submit their findings for publication.12

2.3 Resources 

Program directors from the aforementioned HHMI study were asked to recommend print and 

online resources they found useful for planning and conducting their evaluation efforts, and these 

are footnoted here.13 Another set of references, compiled by program directors and funders of 

SWEPT programs is also provided as a footnote.14 A bibliography relevant to goal-setting, 

evaluation and effective professional development for science educators may also be useful.15

 

11 http://fie.engrng.pitt.edu/fie95/4b1/4b14/4b14.htm 

12 http://www.cumc.columbia.edu/news/in-vivo/Vol2_Iss13_aug18_03/science-outreach.html 

13 http://www.nahsep.org/study_results#sites and http://www.nahsep.org/study_results#assets 

14 http://www-ed.fnal.gov/trc/program_docs/biblio_trp.html 

15 http://demeter.hampshire.edu/~manual/back.html#Gibson,%20Helen%20L.%201998. 
 

9

3. Evaluation Methodology 

3.1 Approach and limitations 

The purpose of this evaluation was to assess the extent to which the ARC-ORNL Summer 

Math/Science/Technology Institutes met the program’s stated goals. ARC also asked that AED 

staff compare the evaluation findings with similar programs. In addition, ARC asked AED staff 

to make recommendations for establishing an ongoing evaluation capability for the ARC-ORNL 

Summer Institute. 

This evaluation employed a mixed-methods approach, including surveys that collected 

quantitative data through questions with fixed-choice responses. Qualitative data were collected 

through open-ended survey questions and through interviews with former student and teacher 

participants. In this way quantitative data are illuminated by more in-depth perspectives offered 

by participants. 

This evaluation is a first step for ARC and offers an objective assessment by an outside 

evaluator. It is important to keep in mind that the Summer Institute program is relatively modest 

in scope, in that it is a one-time, two-week program without connection to the sending schools or 

the institutions of higher education to which the students may apply. While we have explored 

outcomes that ARC hopes to achieve, these outcomes are incredibly ambitious given the scope of 

the intervention. The real strength of this evaluation lies in its attention to the program’s impacts 

as perceived by the participants. 

It is nonetheless important to note some of the limitations of this evaluation. 

Absence of a comparison group. One of the major limitations is the absence of a comparison 

group that would have made it possible to say with a high degree of certainty that outcomes in 

this report can be attributed solely to the Summer Institute. Also, because baseline data were not 

available, we were unable to measure, over time, changes in participants’ knowledge, attitudes, 

intentions, and behaviors. 

Small sample size. Each year the group that attends the Summer Institute is small—no more than 

60 participants. In order to conduct quantitative analysis and analysis involving subgroups (e.g. 

gender or length-of-time teaching), it is necessary to have a large sample. Because we were 

unable to locate many participants despite intense efforts, the sample size remained small 

permitting only a few subgroup analyses. 

Time needed to measure long-term outcomes. Outcomes such as completing higher education 

and beginning a career may take many years. Thus it made sense to look at college completion 

outcomes and employment for students who had attended ORNL Summer Institutes in 1997 and 

1998. For those in later cohorts, it was necessary to explore shorter-term, mediating outcomes, 

such as high school completion and college enrollment. 

Uncertain reliability of self-report. The survey findings are entirely based on self-report which 

may not be completely reliable. Evaluation staff were unable to confirm these responses either 

through record review or observation. 
 

10

3.2 Student and Teacher Surveys 

Survey questions were adapted, where possible, from questions used in evaluations of other precollege STEM programs. New survey items were added when needed. The questionnaires were 

reviewed by ARC staff and pilot-tested on 2005 student and teacher participants. Student survey 

questions were designed to obtain basic demographic information, as well as information on 

educational attainment, career and employment choices, and the perceived influences of the 

Summer Institute on student attitudes and college-going. Teacher surveys collected demographic 

data and included questions pertaining to teaching experience and how the Summer Institute 

directly influenced teaching practices. Both surveys had open-ended questions to allow 

participants to describe in greater depth the influence of the Summer Institute on them and the 

aspects of their experience (e.g., people, projects, cultural programming) that they considered to 

be most influential. 

Survey data collection commenced October 2005 and concluded in December 2005. Participants 

were given the option of completing hard copies of the survey and returning it in a preaddressed, stamped envelope or completing the survey on-line. 

3.3 Student and Teacher Interviews 

During December 2005 and January 2006, staff conducted semistructured, 15-30 minute 

telephone interviews to explore selected responses to the survey in greater depth. The student 

interviews covered recruitment, in terms of how the student heard about the institute; the 

application process and his/her decision to attend; the overall experience of the Summer 

Institute; how the institute influenced college decisions; perceptions of college-going attitudes in 

their school and community, as well as the perceived presence or absence of support structures; 

and recommendations for improving the effectiveness of the institute. The teacher interviews 

covered teacher recruitment, how the institute influenced their approaches to teaching and 

interacting with students, their perspectives on whether the institute had an impact on their 

career, and recommendations for improvement. 

3.4 Study Population 

All 254 student and 132 teacher participants who attended the Summer Institute during the eight 

years spanning 1997-2004 were eligible to complete the surveys. One of the greatest challenges 

to conducting the evaluation was locating the participants, many of whom, especially students, 

may have moved away from home. We expected that some young women had married and 

changed names. Because the survey was conducted during the fall, we also needed to be able to 

contact students who were away at college. Accordingly, we employed a variety of techniques to 

ensure the highest possible response rate. These included searching the Web to confirm or revise 

contact information; e-mailing to addresses provided to ARC by participants (in the 2003-04 

cohorts) or found on the Web; phoning the participants or their families; or enlisting the 

assistance of the sending school. In some cases, we asked participants we located to help find 

others who had attended with them. 

Through the exhaustive use of these techniques, we were able to successfully "find" 63% of the 

students and 80% of the teachers on the lists ARC provided AED. AED staff sent surveys to all 

participants but sought to obtain the highest possible response rate from the “found” participants. 

Two to three weeks after the initial survey mailing, we sent reminder postcards and emails to the 

participants. In addition, staff made follow-up phone calls to all the teachers and students who 
 

11

had not yet returned the surveys (and for whom we had correct contact information). We 

received surveys from 92 students and 71 teachers. Using a denominator of those with confirmed 

contact information, the response rates were 58% and 67%, respectively. Of the surveys 

returned, there were 89 usable student surveys and 67 useable teacher surveys. Response rates 

varied by cohort.16 (See Tables 2 and 3.) 

Interview samples were drawn from participants who responded positively to a question on the 

survey asking if they would agree to be interviewed. From these, we selected individuals who 

together would represent the diversity of participants on characteristics such as gender, year they 

attended the institute, and race/ethnicity. The student sample included students who had attended 

or were attending two- and four-year colleges, as well as those who were employed. The teacher 

sample included some who were early in their careers and others with many years of experience. 

(See Tables A2 and A3 in the appendix for a more detailed description of the interview samples.) 

3.5 Response Bias 

In order to determine whether there was response bias in our survey findings, we compared 

respondents with the “found” participants who did not complete the survey and with the entire 

group of participants on key variables, including gender, year attended, and the economic status 

of the county in which the sending school was located according to a classification system used 

by ARC.17 A variety of statistical methods were used to determine whether there was any bias.18

For students, there were no significant differences among groups by gender, but the more recent 

their entry into the program, the greater likelihood of their being found. However, cohort had no 

significant correlation with likelihood of responding to the survey. There were no significant 

differences among groups for teachers or students with regard to location of the sending school 

in a distressed county. With regard to teachers, there were no significant differences between the 

full group and those found or between found participants and respondents. (See Table 2 for data 

on students and Table 3 for data on teachers.) 

 

16 Our response rate based on the total number of attendees was 36% for students and 54% for teachers. 

As comparison, a response rate of 48% was achieved in a follow-up study reported in 1996 of 1985 

participants of the Student Research Program, a science and engineering summer program for 

undergraduate students sponsored by the U.S. Department of Education. As reported in a U.S. 

Department of Energy Working Paper, “Impacts on Participants of DOE Research Participation 

Programs,” prepared by Argonne National Laboratory and Oak Ridge Institute for Science Education 

(1996). 

For another study, the STRIVE Teacher Research Associates Program, 1986-1991, a response rate of 

83% of teachers was achieved one year following their participation in an eight-week program. By 

comparison, our response rate for 2004 teacher participants was 75%. Oak Ridge Institute for Science and 

Education (1992). Assessment Summary, STRIVE Teacher Research Associates Program, 1986-1991. 

17 ARC designates counties as economically distressed on the basis of low per-capita income and high 

rates of poverty and unemployment. The number of distressed counties changes from year to year, 

depending on conditions. For this evaluation, we used the county’s ARC designation the year the 

participant attended the program. Between 1997 and 2004 the number of counties designated as distressed 

ranged from 90 to 120. The average number of counties in the region designated as distressed was 26%. 

18 Statistical methods included one-way ANOVA, bivariate Pearson correlations, two-tailed t-tests, and 

binary logistic regression (the latter because some of the variables were dichotomous). 
 

12

Table 2—Comparison of respondents with all attendees and with attendees who were 

located but who did not respond to the survey–students 

 All Attendees 

(n=254) 

Located Attendees who 

were Nonrespondents 

(n=67) 

Respondents 

(n=92)* 

 N % N % N %

Gender 

 Male 123 48% 30 45% 46 50%

 Female 131 52% 37 55% 46 50%

Year 

 1997 32 13% 5 7% 10 11%

 1998 37 15% 10 15% 14 15%

 1999 43 17% 7 10% 11 12%

 2000 34 13% 10 15% 12 13%

 2001 18 7% 6 9% 5 6%

 2002 36 14% 10 15% 14 15%

 2003 22 9% 7 10% 10 11%

 2004 32 13% 12 18% 16 18%

Sending 

School in 

Distressed 

County**

 

 Yes 94 37% 26 39% 28 31%

 No 160 63% 41 61% 63 69%

* Of the 92 respondents, 89 were used in analysis. Two students said they did not attend, and one 

returned home after attending for only a day or so. 

** We could not identify sending school for one survey respondent who did not give a name. 

Note: Some percentages in this table do not equal 100% due to rounding. 
 

13

Table 3—Comparison of respondents with all attendees and with attendees who were 

located but who did not respond to the survey–teachers 

 All Attendees 

(n=132) 

Located Attendees who 

were Nonrespondents 

(n=35) 

Respondents 

(n=71) * 

 # % # % # %

Gender 

 Male 48 36% 12 34% 28 40%

 Female 84 64% 23 66% 43 60%

Year 

 1997 17 13% 3 9% 10 14%

 1998 22 17% 7 20% 9 13%

 1999 10 8% 3 9% 5 7%

 2000 18 14% 8 23% 3 4%

 2001 15 11% 5 14% 8 11%

 2002 16 12% 6 17% 9 13%

 2003 14 11% 1 3% 12 17%

 2004 20 15% 2 6% 15 21%

Sending 

School in 

Distressed 

County

 

 Yes 59 45% 15 43% 33 47%

 No 73 55% 20 57% 38 53%

*The survey analysis in this evaluation uses data from 67 respondents. Four of the 71 respondents 

were listed in the database of participants more than once because they attended more than one 

institute between 1997 and 2004. 

Note: Some percentages in this table do not equal 100% due to rounding. 
 

14

4. Student Findings 

4.1 Description of Student Respondents 

Survey respondents were fairly evenly divided by gender. On average they were 16.5 years of 

age when they attended, and most were going into their junior or senior years in high school. The 

majority of respondents (88%) were white and 11% were minority. (See Table 4.) These 

proportions mirror the diversity in the region.19 Thirty-one percent attended schools in counties 

designated by the ARC as “distressed.” (See Table 2.); and 47% of respondents were from three 

states—Georgia, New York, and Ohio. (See appendix Table A1.) While quite a few of them 

(27%) had attended other STEM programs outside of school, most (73%) had not. 

In general, student respondents lived in households with adults who had some college education 

(see Table 5). Almost three-quarters of the students (73%) reported that their mothers, or another 

female adult with whom they lived during high school, had some college education or more. 

Similarly, 60% of students lived with a father, or other male adult, who had at least some college 

education. 

Approximately half the students (46; 53%) lived in households where one or both 

adults/caregivers had earned a bachelor’s or graduate degree; 15% of students (13) lived in 

households where both the mothers and fathers had a high school diploma or GED or less. 

4.2 Overall Assessment of the Summer Institute 

Overall, student participants were enthusiastic about the Summer Institute, and fully 90% 

reported that they recommended the Summer Institute to other students once they returned home. 

“Motivating,” “refreshing,” and “liberating,” were words used to describe their two-weeks at 

ORNL. 

Survey responses and interviews revealed that participants benefited from the program in 

significant ways. They became more self-confident and approached their high school studies 

with renewed interest. Their aspirations for college were reinforced, and many pursued STEM 

majors in their post-high school studies. For once, one student reported, “I wasn’t penalized for 

being smart.” In short, students expressed a collective sigh of relief in a comfortable 

environment. 

A large majority of students reported that they were challenged by the activities in which they 

participated: 29% reported activities at the Summer Institute as “very challenging” and 61% said 

they were “somewhat challenging;” 3% said they were overwhelmed by the activities and felt 

unprepared; and, 7% found the activities to be “not at all challenging.” 

 

19 According to the U.S. Census for 2000 figures, the population was 88% white, 8% black, 2% Hispanic 

and 2% other. Kelvin Pollard (2004). A “New Diversity”: Race and Ethnicity in the Appalachian Region.

Washington DC: Population Reference Bureau. 
 

15

Table 4—Selected characteristics of student survey 

respondents 

Mean age in years 16.5 

 N %

Gender 

 Male 46 52% 

 Female 43 48% 

Highest grade in school completed prior to 

attending the Summer Institute*

 

 9th 2 2% 

 10th 34 38% 

 11th 51 57% 

 12th or above** 2 2% 

Ethnicity*** 

 White 78 88% 

 Black or African American 6 7% 

 Hispanic or Latino 0 0% 

 Asian or Pacific Islander 3 3% 

 American Indian or Alaska Native 1 1% 

* Does not total to 100% due to rounding. 

**One respondent completed a community college course. 

***Multiple answers were allowed. Only one respondent 

identified as more than one ethnicity (White and African 

American). 

Table 5—Parents’ highest level of formal education 

Mother Father 

 N % N % 

Less than high school graduate 2 2% 8 9% 

High school or GED 17 19% 17 19% 

Post high school trade school 4 5% 9 10% 

Some college 19 21% 11 12% 

Associates degree 12 13% 8 9% 

Bachelor’s degree 19 21% 21 24% 

Graduate degree 16 18% 13 15% 
 

16

The next sections of this report outline the influence of the Summer Institutes on students when 

they returned to school, and with regard to college-going and to considering and pursuing STEM 

careers. The quotes from students scattered throughout this section illuminate the aspects of the 

Summer Institute that students found most influential with regard to these outcomes. 

4.3 Influence of Peer Support 

According to a report on strategies to increase postsecondary access for underrepresented youth 

by the National Postsecondary Education Cooperative, one of the most effective program 

practices is “providing a peer group that supports students’ academic aspirations as well as 

giving them social and emotional support.”20 It is therefore not surprising that one of the major 

findings of this evaluation is that, for students, the most important and influential feature of the 

Summer Institute was their interactions with other students. These interactions also made their 

time at the institute enjoyable. The value of peer-to-peer learning and support echoed through 

responses to multiple questions on the survey and in the interviews. For example, one student 

said: 

The interesting things you learn about science were all very fascinating, but for 

me, I will always remember the people I met. When you live for two weeks with 

other people, you learn a lot. 

Students also deeply valued the social experience of the institute. This was the first time many 

students had ventured outside their hometowns, traveled alone, and stayed with people they 

didn’t know. Students’ comments revealed that they felt challenged, learned more about 

themselves, and discovered that there were more opportunities and possibilities for further 

education and jobs than they had realized. Some characterized their two-weeks as an “eyeopening” experience: 

The friendship and social enrichment that I received by interacting with the other 

students at the Summer Institute was priceless. It was very valuable to me to 

reach outside my home area, my friend group that I had known since childhood, 

to see what new people from different backgrounds had to say and teach me. 

In the course of student interviews, many students expressed satisfaction and relief when they 

realized they were not “the only one” their age with interests in science or plans to pursue their 

education. They were relieved and pleased to discover that outside their circle of acquaintances 

there were students who had similar interests and goals. 

It made me realize that others my age truly cared about their futures and a global 

world rather than just the valleys they, and myself, grew up in. It made me feel 

like I didn’t stick out . . . I fit in for a change. 

The students I met were terrific. What stood out about them was their high goals 

and expectations to succeed. I felt that I could relate with them, that I was one of 

them, and that all of us would be successful in education and in our careers. This 

confidence continued with me during high school and college. 

 

20 U.S. Department of Education, National Center for Education Statistics. Paving the Way to 

Postsecondary Education: K-12 Intervention Programs for Underrepresented Youth, NCES 2001-205, 

prepared by Patricia Gándara with the assistance of Deborah Bial for the National Postsecondary 

Cooperative Access Working Group. Washington DC: 2001. 
 

17

When asked to describe the people at the institute who were influential, respondents mentioned 

students more frequently than teachers, and almost as often as mentors. While participants 

gained a great deal from seeing mentors and teachers in action, exposure to the decisions and 

goals of their peers had a profound impact. 

There was another student there . . .with big dreams about a degree at WVU 

[West Virginia University] . . . [she] had a very positive attitude and made me feel 

more comfortable. 

Many of the students made me realize how many different opportunities there 

were and how much I still had to discover outside of the small town that I grew up 

in. They all gave me confidence and excitement about my future––to leave the 

familiar for the unknown! 

4.4 Influence of Mentors, Chaperones, and Teachers 

Students also highly valued their relationships with mentors, chaperones and teachers. In fact, 

46% of survey respondents answered “yes” in response to a question on the survey that asked: 

“Was there a particular person, or experience/activity that had an impact on your academic or 

professional development?” Approximately half of them mentioned someone from the institute 

(teacher, student, mentor, or chaperone). Responses indicated that the person influenced their 

career choice, increased their enthusiasm for scholarship and learning, and increased their level 

of self-confidence. While students did not always articulate what it was about these people that 

inspired their academic and professional development, many noted that the person(s) provided 

advice and/or encouragement. 

There were several people who had an influence on me after the institute; 

however, one in particular stands out in my mind. He was a teacher, and I have 

remained in contact with him ever since the institute. He frequently speaks with 

me about what I am doing with my life. He gives me advice and encouragement as 

well. He has seriously been a huge aspect as to why I aspire to go into the 

medical field. 

One student expressed the importance of the people she met on her future plans: 

The greatest highlight of all was meeting so many new people. Not only did I 

make friends that I still remain in contact with, but I also met many 

professors/mentors/chaperones that could give me great insight as to what it takes 

to be successful in a science career. 

4.5 Changes in Attitudes about School and Subjects Studied 

Students reported that the Summer Institute had a reinvigorating effect on their attitudes and 

behaviors when they returned to high school that fostered their desire for academic success. A 

total of 57% of them answered “yes” to the question, “Did the Summer Institute have an effect 

on you when you returned to school, for example, grades and attendance; attitude toward school, 

teacher or courses; or involvement in math or science clubs or activities?” 

Over half of these respondents stated that the experience increased their enthusiasm, drive, 

motivation, effort, involvement, and attitude toward school. Slightly less than half of respondents 

stated that they were more enthusiastic and interested in STEM subject areas. The following 

quotes are illustrative: 
 

18

I cared more. Bottom line. I never would admit that or show it then, but for the 

rest of high school, I actually didn’t sleep through class. I read non-fiction books 

to learn from outside the curriculum for the first time in 16 years. The following 

school year, I actually spent hours trying to solve geometry problems from a 

teachers’ master’s course book. 

The Summer Institute gave me a different outlook on life. It made me even more 

excited to learn and go to college and meet new people. It helped me to 

experience the world at such a young age which in turn helped prepare me for 

college and independence. 

I was always active in high school, but after I returned from the Summer Institute, 

I was so much more enthusiastic about it all, including school in general. I spent 

my senior year being very active with a teacher at my school, who also attended 

the institute, in many clubs such as the science club. Also, I buckled down on my 

studies and graduated valedictorian of my class. 

The Summer Institute definitely had an effect on the way I acted around other 

students and teachers. It reassured me that I can speak my mind and people will 

listen to what I have to say. The Summer Institute didn’t give me a voice, but it 

helped me find mine. 

We asked students whether their experience at the Summer Institute influenced the science 

courses they took when they returned to school. Almost a quarter of the students (24%) reported 

that they took more science classes when they returned to high school. More than half (56%) 

indicated that the institute reinforced their prior decisions about the science courses they were 

scheduled to take and 20% said the institute had no influence. Of the 21 students who reported 

taking more science classes, six took an advanced or honors-level course, and approximately half 

(12) took two or more science classes. Students took science classes in a variety of disciplines—

chemistry (13 students), physics (10), biology (7), anatomy and physiology (3), and physical 

science (1), environment (1), nutrition (1), and computer science (1). 

Fewer students, 22% reported taking more math classes as a result of attending the institute; 52% 

reported that the institute reinforced prior decisions, and 26% said that the institute made no 

difference on the courses they took. Of the 19 students who said they took more math classes, 

four took advanced placement-level courses. The number of students taking various math course 

were as follows—pre-calculus (9), calculus (12), algebra I and/or II (4), trigonometry (3), 

geometry (2), and statistics (3). 

Teachers also noticed that students profited from the institute in terms of their attitudes and 

behaviors. Of the 34 teachers who commented about changes they noticed in students either 

during the institute or when students returned to school, the highest percentage (29%) mentioned 

students’ increased interpersonal or social skills, noting that they seemed more “personable,” 

“self-confident,” “proud,” and “outgoing.” A quarter of the teachers mentioned that, as a result of 

the institute, students were more dedicated to their studies and more serious about their future 

and motivated. The following quote illustrates the changes noted by teachers: 

It seems that all five students from my school returned with a much more mature 

outlook on life and their future goals. I think the experiences in the workplace, the 

visits to different schools/businesses and the relationships with students different 

from them were really beneficial. 
 

19

A quarter of the interviewed teachers also noted that students gained a deeper knowledge of the 

subject area they studied and a greater interest in math and science: 

I think both students gained a greater appreciation for science. They have both 

gone on to major in some aspect of science. 

4.6 Pursuit of STEM Education and Employment 

Findings displayed in Table 6, show that students reported that participating in the Summer 

Institute greatly influenced their views of STEM. More than two-thirds of students gave the two 

highest ratings to statements reflecting increased awareness of and improved attitudes toward 

STEM. Fully 88% reported that the institute positively increased their awareness of ways STEM 

can be applied. They strongly agreed with statements on the survey that said the Summer 

Institute had increased their interest in STEM and increased their confidence in their abilities in 

these fields. A small number of students (14) reported that the Summer Institute increased their 

interest in a field outside of STEM; however, when we asked them to specify these other fields, 

they named occupations that were in the sciences or computer fields, such as medicine, 

pharmacy, nursing, and computer software design and Web design. 

Males and females held fairly similar opinions about the influence of the Summer Institute on 

them. However, females were significantly more likely than males to indicate that the program 

increased their awareness of job opportunities in STEM (p=0.63).21

Table 6—Students’ ratings of the extent to which the Summer Institute 

influenced awareness, confidence and interest in STEM 

Percentage Distribution 

Great 

extent 

 Not at 

all 

5 4 3 2 1 

Increased their awareness of 

ways STEM can be applied. 

48 40 9 1 2 

Increased their interest in STEM. 34 35 24 5 2 

Increased their confidence in 

their ability in STEM. 

27 47 19 5 2 

Increased their awareness of job 

opportunities in STEM. 

33 40 16 7 5 

Increased their interest in a 

career in STEM. 

33 34 20 7 6 

Increased their interest in 

another career or field. 

9 9 23 13 45 

Notes: 

Students rated statements on a 5-point scale where the endpoints only were 

labeled. 

Percentages in this table may not total 100% due to rounding. 

 

21 Two tailed t-tests were conducted to determine the significance of means. Because of the small 

respondent pool (n=88), a relaxed standard for significance was used (p≤ 0.10). 
 

20

 

In addition, some students commented that the institute broadened their choices in the careers 

they were considering after high school. For example: 

Growing up in a small town, many students marry and go into labor jobs 

immediately after high school. Meeting with students from around the 

Appalachian region, who all had interest in furthering their education challenged 

me to look past the typical career choices in my area. I wanted to challenge 

myself to do more, and now, that is exactly what I am doing. 

Many of the students’ responses related to the project they worked on and their experience 

working in that topic area. Students enjoyed the “real world” research and recognized its 

applicability outside the laboratory setting. In addition, students commented on the value of 

gaining hands-on experience, and many cited their specific project assignment as the highlight of 

their stay at ORNL. One student commented on how a project directly influenced her career 

choice: 

The field components of our study–collecting snails, fish, water quality and 

stream data–were my favorite part of the Summer Institute. Thanks to the time I 

spent with Mike and Art, I became certain that biology, with a strong field 

component, was the path that I wanted to pursue. 

Another prevalent theme of students’ comments was the experience of working in the ORNL 

facility and touring the labs. Students enjoyed learning more about science, technology, 

engineering, and mathematics. Some students felt privileged to have access to equipment that 

“not just anyone” could use: 

Scanning electron microscopes are multi-million dollar pieces of equipment, so, 

as a student, I was shocked by the amount of trust our supervisors at ORNL gave 

us when allowing us their SEM. That experience was the highlight of my trip 

there. I could have spent hours in that room studying specimens beneath the 

microscope. 

The specific project that the students were assigned to was a key determining factor in the 

lessons they took away from the Summer Institute. For example, one student selected a project 

that involved fieldwork and she enjoyed it so much that it confirmed her desire to be a biologist 

and conduct field work; this student is now leading outdoor tours at her university and will 

graduate in May 2006 with a degree in biology. Not being assigned to a chosen project caused 

some disgruntlement. One student said that he did not find his experience very influential on his 

thinking because he was not assigned the project he requested. Another expressed relief that she 

was on the project she desired and knew she would have been unhappy if she had not been on 

this project. The students reported different processes for selecting the projects that they could 

work on. Some remember ranking their choices whereas another, from 2001, said students in his 

cohort were assigned to projects without making any selections. Students who were interviewed 

felt that overall the selection process should be more tailored to the students’ interests. 
 

21

4.7 College Aspirations and Planning 

Preparation for College 

In general, in order to attend college, students and their families must mount a complex series of 

steps, beginning with identifying college as a goal. This usually starts with recognizing the 

importance of higher education for future careers, followed by taking and doing well in the 

requisite courses, selecting and applying to one or more colleges, and applying for financial 

assistance. 

Ohio’s Appalachian Research and Success Project identified several significant barriers to 

college-access in Appalachian Ohio. These included lack of information or misinformation 

among students. Other barriers were lack of guidance, assistance, academic preparation, and 

encouragement to help overcome low self-esteem. The research also found that students and their 

families lacked information about financial opportunities and the process for applying for aid.22

As has been found in other surveys administered in school settings, the current evaluation 

revealed that all of the student survey respondents reported that, even prior to attending the 

Summer Institute, they planned to attend college. Even so, students reported many ways that the 

Summer Institute reinforced their intentions to go to college and reduced some of the barriers. 

For example, more than 50% of Summer Institute students indicated that the program positively 

influenced their intentions to go to college. As shown in Table 7, they rated highly (i.e., a rating 

of 4 or 5) the contributions of meeting other college-bound students, as well as the high 

expectation on the part of mentors, chaperones, and other staff. Information about financial aid 

was seen as helpful by 37% of students, but, on average, played a lesser role in influencing their 

thinking about college. Because we had no way to measure students’ families’ economic status, 

we cannot ascertain whether this information might have been crucial for students from 

economically disadvantaged households. What we do know from interviews is that several 

students complained that their guidance counselors had not provided sufficient information about 

college-going. 

Interviewed students described how the ORNL experience reinforced their decisions to attend 

college and helped guide them in a general direction of study. They reported feeling encouraged 

and inspired to pursue higher education. For example, one student reported that he was certain he 

was going to go to college “no matter what,” but his experience at the Summer Institute helped 

him decide to go into mechanical engineering. Another student who was considering pharmacy 

training was assigned to a project that actually dealt with pharmaceuticals. This confirmed her 

decision and she is now a practicing pharmacist after recently completing her schooling and 

residency. 

 

22 Crowther T., Lykins D., and Spohn K., (1992). Report of the Appalachian Access and Success Project 

to the Ohio Board of Regents. Athens/Portsmouth: Institute for Local Government Administration and 

Rural Development, Ohio University/Shawnee State University. 
 

22

Table 7—Students’ ratings of the extent to which various aspects of the Summer 

Institute positively influenced their thinking about going to college. 

Percentage Distribution 

Great 

extent 

 Not 

at all 

5 4 3 2 1

Meeting other students with college plans 44 27 17 6 6

High expectations of mentors and other staff 32 40 18 3 7

Interaction with mentors 33 36 17 7 7

Interaction with other staff at ORNL 26 38 22 6 8

Learning about educational requirements for 

careers in science 

27 35 22 7 9

Interactions with chaperones 24 29 28 7 11

Trips to local colleges 27 23 23 16 11

Information about financial aid 15 22 25 17 21

Notes: 

Students rated statements on a 5-point scale where the endpoints only were labeled. 

Percentages in this table may not total 100% due to rounding. 

As already mentioned, many students found it especially beneficial to hear about other students’ 

college plans. Another student who described the ORNL experience as a “big confidencebuilder,” said it pushed him to look to the future and consider schools that were a bit farther 

away from home. One student appreciated that the institute gave him a taste of college life. 

Several students reported that the institute positively influenced their thinking about college. One 

stated that the institute was extremely influential and helped her clarify what she “wanted to do 

for the rest of her life.” 

The evaluation explored steps students had taken to achieve their college goals. We found that 

although all planned to continue their education, approximately half the students (51%) had 

never toured a college campus before the institute. It is not surprising, then, that 50% of students 

reported that visiting the college campuses as part of the Summer Institute positively influenced 

their thinking about going to college. Further, when it came to applying to college, 85% of 

students who applied mentioned their experience at ORNL either in essays or interviews. 

While most of the college-going process is out of the control of the Summer Institute, the 

evaluation asked students questions about other supports students needed to translate a desire to 

go to college into a reality. These findings have implications for the advisability of building 

school connections that ARC–ORNL might want to consider. It appears from student responses 

that schools could improve in terms of helping students apply, and gain admission to, college. 

Although 64% of students thought that their school gave them sufficient information regarding 

college choices, 32% said schools had not, and 4% were unsure. According to students, schools 

were less helpful providing what students considered sufficient information about college costs 

and financial aid. Half of students (50%) reported that their school provided enough information, 

40% said schools did not, and 10% were unsure. 
 

23

Table 8 shows that students consider parents and teachers as providing the most encouragement 

for their college plans. 

Table 8—Mean scores for students’ ratings of the 

extent to which they received encouragement to 

attend college from the following: 

Parents or guardians 4.73 

Teachers 4.56 

Siblings or friends 3.95 

Guidance counselor 3.92 

Community or religious organizations 3.14 

Note: Students rated each on a 5-point scale where 1= 

Not at all and 5= To a great extent. 

4.8 College Enrollment 

Fully 96% of student participants who had graduated from high school said that they continued 

their formal education. Of the 83 respondents who provided data: 

• a great majority (79%) went to four-year institutions. 

• fifteen percent went to community colleges or technical centers. 

• two percent to military academies. 

• three percent did not continue their education beyond high school.23 

By comparison, these rates are much higher than rates for the U.S. population overall, where 

enrollment in four-year institutions is 37% for the white population, 26% for the black 

population, and 15% for the Hispanic population.24 (As shown in Table 4, Summer Institute 

student-survey respondents were 88% white and 12% minority.) Among our participants, boys 

were more likely than girls to stop at high school and go to community colleges or technical 

centers. Girls were significantly more likely than boys to go on to four-year colleges (p=0.51). 

(See appendix Table A4.) 

Because many student participants were still pursuing their educations, it is not possible to 

determine how many of them will ultimately complete their degrees and at what level. Table 9 

shows their educational attainment at the time of the survey. Appendix Table A5 displays 

findings on the highest education that students had completed to date by cohort. As would be 

expected the earliest cohorts have achieved the most college and graduate degrees and the later 

cohorts appear to be in the process of completing their undergraduate degrees. 

 

23 Two participants still in high school were excluded from the analysis. 

24 Forster, Greg (2006). Opinion. The Embarrassing Good News on College Access. The Chronicle of 

Higher Education. http://chronicle.com/free/v52/i27/27b05001.htm (accessed March 2006). 
 

24

Table 9—Students’ highest educational attainment 

N % 

Some high school or high school graduate 18 20% 

One or more years of college but no degree 32 36% 

Associate’s degree, certificate or technical diploma 7 8% 

Bachelor’s degree 19 21% 

Some graduate work or advanced degree 12 14% 

Note: Percentages in this table do not total 100% due to rounding. 

Considering only the 23 students in the two earliest cohorts––i.e. those out of high school long 

enough to have obtained a college degree––all reported having some higher education, with 26% 

having some college but no degree, 39% earning a bachelor’s degree, and 35% going on to 

graduate work. 

All students from households with at least one parent with a college degree continued their 

formal education after high school. Of the nine students who lived in families where both adults 

had no more than a high school education, seven (78%) continued their formal education after 

high school. At the time of the survey, the highest level of education attained to date, for two was 

an associate’s and for five a bachelor’s degree. One student who did not go to college entered the 

military service. 

We estimate that approximately two thirds of students pursued STEM majors in college. 

Findings in Table 10 show the majors reported in the survey. Unfortunately, the survey did not 

ask students to name their first declared major at the postsecondary level. Rather, it asked 

students to list all of the educational institutions they were attending/had attended and their 

majors. Those who graduated gave the major for which they had fulfilled requirements. Those 

who were early in their undergraduate education gave current majors, which could change. 

Therefore, the data should be interpreted with caution. 

Table 10—Major fields of study 

N %

STEM 

Biological sciences 20 28% 

Engineering 10 14% 

Computer & technology 6 8% 

Mathematics 3 4% 

Environment or ecology 3 4% 

Chemistry 2 3% 

Non-STEM 

Social Science 10 14% 

Other 18 25% 
 

25

Looking more closely at degrees earned from higher education institutions, we learned that 36 

students had earned a total of 46 degrees: 54% of degrees earned were in STEM—38% of 

associate’s degrees, 52% of bachelor’s, and 86% of graduate degrees. (See Table 11.) 

By comparison, 82% of students currently pursuing degrees at the time of the survey were 

majoring in STEM fields. Table 11 below also shows the distribution of STEM and other degrees 

attained at the associate’s, bachelor’s, and graduate levels.25 

• Thirty-four students were pursuing four-year degrees, and of the 32 that indicated their field 

of study, 27 were majoring in STEM subjects. 

• Of the six students attending a community college or technical center, five were pursuing 

STEM subjects. 

• Of 10 students pursuing graduate degrees, six provided information about their field of study. 

Four of the six were in STEM fields. 

Table 11—Degrees pursued or earned in STEM and other fields of study 

STEM Other 

 N % N % 

Completed degrees (n=46) 25 54% 21 45% 

Students pursuing higher education (n=44) 36 82% 8 18% 

Associates and technical degrees (n=8) 3 38% 5 62% 

Bachelor’s degrees (n=31) 16 52% 15 48% 

Master’s and doctorates (n=7) 6 86% 1 14% 

 

25 One respondent provided no information about the level of education or field of study. 
 

26

4.9 Workforce Participation 

Overall, 38% of student participants were employed full-time, 35% were part-time, 15% were 

unemployed but looking for work, and 12% were out of the workforce. Of the student 

respondents reporting full-time employment, the largest proportion (37%) worked in business 

and industry (see Figure 1). Asked whether proficiency in one of the STEM fields is a necessary 

requirement of their jobs, 55% of those working full-time responded “to a great extent.” 

Figure 1 

Since some students were still in school at the time of the survey, we looked at workforce 

participation by cohort. As would be expected, those in the earlier cohorts had higher rates of 

employment. As shown in Table 12, approximately 60% of student participants in the 1997-2000 

cohorts were working full-time, and most of the others were working part-time. Fewer students 

in the later cohorts were working full time; more were working part-time, and a sizeable 

percentage was unemployed and looking for work. 

Table 12—Workforce status of student participants 

Cohort Working 

Full-time 

Working 

Part-time 

Not 

workinglooking for 

work 

Not 

working-not 

looking for 

work 

Total 

 N % N % N % N % N %

1997-1998 14 61% 7 30% 1 4% 1 4% 23 99%

1999-2000 13 59% 8 36% 1 5% 0 0% 22 100%

2001-2002 3 19% 4 25% 5 31% 4 25% 16 100%

2003-2004 3 12% 11 44% 6 24% 5 20% 25 100%

Total 33 38% 30 35% 13 15% 10 12% 86 100%

Note: Percentages in this table may not total 100% due to rounding. 

Asked if their current jobs (full- or part-time) required proficiency in STEM, close to half of 

student participants said a “great extent” and 31% said “somewhat.” Asked to project ahead to 

their future careers, most participants indicated that proficiency in STEM would probably be a 

requirement. (See Table 13.) 

Employer (n=33)

Health Care

7

Business 

and Industry

12

U.S. Military

5

University or

College

5 Public 

school

1

Government

3
 

27

Table 13—Students’ opinions about STEM proficiency 

requirements of current jobs and future career 

STEM proficiency 

is a requirement of 

current job 

STEM proficiency 

will be a 

requirement in 

future career 

 N % N % 

To a great extent 29 46% 65 74% 

Somewhat 19 31% 21 24% 

Not at all 14 23% 2 2% 

Total 62 100% 88 100% 

To assess the extent to which the investment in students is likely to directly benefit the 

Appalachian region, we compared the percentage of student participants who were currently 

living in the region with the percentage who projected they would still be there in five years. At 

the time they attended the Summer Institute, all students lived in the Appalachian region. At the 

time of the survey, 59% lived in the region, and 53% of the total group of student participants 

thought they would be living in some part of the region five years hence. There was virtually no 

difference in the percentage of respondents who planned to remain in the region between those 

who majored in STEM and those who majored in non-STEM fields (66% vs. 65%, respectively). 

As shown in Table 14, earlier (1997-2000), and later (2001-04) cohorts did not differ appreciably 

in terms of the proportion currently residing in the Appalachian region. However, compared with 

the early cohorts, a higher percentage of participants from the later cohorts thought they would 

be living in the region in five years. Because the number of participants who answered the 

question about future residence was low (N=49), findings should be interpreted with caution. 

Table 14—Students who currently live in the Appalachian Region 

(AR) and those who plan on living there in 5 years, by cohort 

Cohort 

Currently lives in 

AR 

Plans to live in AR 

5 years from now 

 N % N % 

1997-2000 25 59% 13 46% 

2001-2004 26 62% 13 62% 
 

28

4.10 Student Recommendations 

While many students indicated that there should be no changes or that they had no 

recommendations for increasing the Summer Institute’s effectiveness, others pointed out some 

specific things that they felt would have made their positive experience even better. The student 

recommendations can be broadly grouped into two major categories: recommendations based on 

program activities and projects and recommendations related to program logistics. 

In terms of the activities and projects, the students overwhelmingly reported a desire to select the 

project they were assigned to for the two weeks. Although in some years students were allowed 

to rank their project choices, one student stated that he did not really understand what they would 

actually be doing and wished he had had more information in order to make a more appropriate 

choice. Some other recommendations from students that relate to projects and activities included: 

• Offer more medical and math-related projects. 

• Offer more hands-on projects. 

• Increase communication between students working on different projects. 

• Obtain participant feedback about the strengths and weakness of specific projects. 

• Increase interaction between students and teachers. 

• Continue access to the newest technology, especially computer technology. 

The second group of recommendations focuses on the logistics and design of the entire program. 

Student recommendations for strengthening the program included the following: 

• Operate the program for longer than two weeks. 

• Invite more students to attend the program. 

• Focus on low-income students with less support and expand the age range. 

• Increase publicity to make students more aware of the program. Ensure that the person 

disseminating information at the school can explain the recruitment and application process 

as well as the target population. (For instance one student recommended targeting students 

“with potential who may not, themselves, see it.”) 

• Involve graduate student scientists or young professionals in the program as role models. 

This last recommendation, which was suggested in various ways by many students, was an 

innovative recommendation to help “bridge the gap” between the professional scientists and high 

school participants. Some students recommended that graduate students could be invited to speak 

to the group or play a mentoring role. As one student pointed out, this would help students “see 

science as something that could be in their future,” providing them the opportunity to “meet 

[college or graduate] students who were enthusiastic and interested in science.” A few other 
 

29

students recommended bringing past attendees back to the institute as junior counselors who 

could serve this “bridging” role. 

Regardless of whether some students are invited back as junior counselors or not, many students 

expressed the desire to have a reunion with others in their cohort or at least a more effective way 

of maintaining contact with each other. As one student stated, this could really “keep the 

excitement and energy going.” 
 

30

5. Teacher Findings 

The Summer Institute offers a professional development component for teachers because 

teachers are essential to motivating students and fostering their academic achievement. At the 

Summer Institute, teachers participate in collaborative learning in groups with other teachers and 

ORNL mentors. Inquiry-based learning is modeled in group settings similar to the ones to which 

students are assigned. Unlike some other STEM enrichment programs for underrepresented 

populations of high school students, there is no set curriculum that teachers are expected to bring 

back to the classroom, nor lesson-planning activities or formal attention to strategies for 

enhancing students interest and achievement in STEM. 

5.1 Teacher Respondents 

Of the 67 teacher respondents, 64% were female and 36% were male. The percentage of female 

teachers at the institute was generally higher than that of high school science teachers in the 

U.S.26 Nine of the teachers had attended the Summer Institute more than once. Teachers reported 

that they ranged in age from 23-70 years when they attended the Summer Institute; their mean 

age was 42 years. (See Table 15.) A total of 47% worked at schools in distressed counties. (See 

Table 3 on page 12.) Almost half (48%) of teachers were from three states—Ohio, West Virginia, 

and New York. (See Appendix Table A1). 

Most teacher attendees (64%) were high school science teachers. Somewhat more than a quarter 

(27%) were high school math teachers, and 5% taught computer courses (information 

technology, introduction to computers, personal computers, and computer applications for 

business). Teachers often taught several grade levels, e.g., 9-12 or 11-12. 

Teachers attending the institute appear to have been experienced, with 63% having taught for 

more than 10 years. Slightly more than a fifth of teachers (21%) were new to the field, having 

taught for five or fewer years. Sixty-eight percent reported having attended other 

math/science/technology enrichment programs. Overall, 40% earned professional development 

credits for attending the Summer Institute. 

As shown in Table 16, 19% had a bachelor’s, 67% had a master’s degree, and 13% had 

educational specialist degrees or doctorates; 20 teachers were in the process of pursuing 

advanced degrees. 

 

26 In 2002, the percentage of female high school teachers was as follows: biology–52%, chemistry–47%, 

physics–28%. National Science Teachers Association (2004). K-12 Science and Mathematics is Critical 

to our Future Workforce. Talking Points. Arlington, VA. 

http://www.nsta.org/main/pdfs/TalkingPoints15pp.pdf 
 

31

Table 15—Selected Characteristics of Teacher Survey Respondents 

Mean age in years (n=64) 42 

 N % 

Gender (n=67) 

Male 24 36% 

 Female 43 64% 

Ethnicity (n=65)* 

 White 59 89% 

Black or African American 4 6% 

 Hispanic or Latino 1 1% 

 Asian or Pacific Islander 1 1% 

 American Indian or Alaska Native 0 0% 

Years of teaching experience (n=65) 

 1-5 14 21% 

 6-10 10 15% 

 11-15 7 11% 

 16-20 17 26% 

 >20 17 26% 

Teaching position (n=68) 

 High school science 42 64% 

 High school math 18 27% 

 Computer science/technology 3 5% 

 Other 5 7% 

* Multiple answers were allowed. No respondents identified as more 

than one ethnicity. 

Note: Percentages in this table may not total 100% due to rounding. 

 
 

32

Table 16—Educational attainment of teachers 

 Education 

Attained 

Education 

Pursing 

 N % N % 

B.S./B.A./B.Ed. 13 19% 0 0% 

M.S./M.A./M.Ed. 45 67% 6 60% 

Ed.S.* 7 10% 4 40% 

Ed.D./Ph.D. or other doctorate 2 3% 0 0% 

Total 67 99% 10 100% 

* Education Specialist 

Note: Percentages in this table may not total 100% due to rounding. 

Many teachers reported having prior research experience; 75% had attended prior professional 

development for teachers, and a good number also had some research experience as part of their 

graduate or undergraduate education. Respondents also reported having research experience 

related to part- and full-time employment (19% and 6%, respectively). (See Table 17.) 

Table 17—Research experience prior to attending the 

Summer Institute 

 N % 

Professional development for teachers 49 75% 

College coursework 58 89% 

Research/graduate assistance 16 25% 

Independent research 16 25% 

Part-time or summer employment 12 19% 

Full-time employment 4 6% 

Other 1 1% 

Note: Multiple answers were allowed. 

5.2 Overall Assessment 

Overall teachers highly valued their experience at the Summer Institute. Asked “What stands out 

as a highlight of that two-week experience,” 63 teachers named a particular highlight or 

indicated that the experience, overall, was a highlight. The most common response, named by 

40% of these 63 teachers, was networking with and learning from the other teacher participants. 

For 30%, a highlight was working with an ORNL research scientist, and 30% mentioned learning 

about a specific topic (e.g., electron microscopy, the history of atomic research, or hay bale 

walls). One teacher especially appreciated the opportunity of being exposed to other ORNL 

projects. For 21%, a highlight was their lab experience and contributing to a current research 
 

33

project. Five of the teachers (8%) mentioned something about the pedagogical process, such as 

sharing what they learned with the other participants, the presentations, and getting in touch with 

what it is like to be a student. Some teachers mentioned their interactions with students, the 

opportunity afforded students, and the field trips. The following quotes illustrate these points and 

capture teachers’ enthusiasm: 

Meeting other teachers and discussing teaching and learning situations. Working 

with students and observing them working with others and viewing their learning 

processes. 

Working in the physics lab on a project that would really be used to help design a 

new piece of equipment. Sharing ideas with teachers from other states. 

The ability to work with members of the Oak Ridge Laboratory staff and the 

knowledge I received. 

The ability to work at a government facility and do hands-on research was 

remarkable. I have worked in research previously and the opportunity afforded 

the students to have this experience was remarkable. 

Working on a variety of real projects with other teachers and scientists using 

“state-of-the-art equipment.” 

I enjoyed working with the Robotics Division at ORNL. It was definitely a positive 

experience and one that I could take back to my classroom and share with my 

students. 

Evidence of teacher satisfaction with their experience was demonstrated by the fact that 92% 

recommended the Summer Institute to teachers in their school district and 62% recommended it 

to teachers in other school districts. Teachers also recommended the Summer Institute to other 

students. Fully 86% of teachers recommended it to students in their home school district and 

25% recommended it to students in other school districts. 

5.3 Impact on Teaching 

Teachers were asked whether they drew on their Summer Institute experience for various 

classroom activities. As shown in Table 18, many teachers integrated what they had learned at 

ORNL into their classrooms.27 Some reported they were better able to emphasize the importance 

of science in our world and describe how science is applied to “real-life situations” and how it 

can affect students’ lives. Other teachers incorporated new topics, laboratory experiments, and 

demonstrations, either as a result of specific knowledge gained while conducting their projects or 

in conversation with other teachers. Some teachers used materials they obtained from ORNL, 

(for example, the Oak Ridge history video and “Probeware”), Web-based resources they learned 

about, as well as their own photos and presentations. The following quotes from teachers 

 

27 By comparison, the National Teacher Enhancement Project found that at one-year follow-up, 99% said 

“yes” to the question, “I have drawn on my program experiences for explanations and examples in my 

teaching, class demonstrations, or laboratory exercises.” Participants were K-8 teachers who attended a 

three-week summer program at one of five Department of Energy Laboratories. Oak Ridge Institute for 

Science and Education (1993) U.S. Department of Energy, National Teacher Enhancement project: Final 

report on 1990-1992 Teacher Participants. Oak Ridge, TN. 
 

34

describe specific ways they applied information learned at the Summer Institute in their 

classrooms: 

Part of what I teach is the nature of science including methods, philosophy and 

underlying fundamental beliefs. I convey these ideas through labs, demonstrations 

and sharing my experiences. 

I used several of the demonstrations (particularly from our teacher group 

sessions) in my physics class. 

In teaching genetics I use the “mouse house” video and can explain more about 

mutations and how “we” as scientists further the knowledge of each other. I do a 

lab on protein synthesis and can add information on the genome and advances in 

technology I learned at ORNL. 

When I was there they were building a particle accelerator at the site. I was able 

to discuss that with my chemistry students. Also I learned how to navigate some 

internet sites that uncode the human genome and I have been able to share that 

knowledge and have the AP Biology students navigate them as well. 

I use the tour of the spallation neutron facility to introduce students to the idea of 

how new technology could change what we know. I use the research we did in 

building technology to develop an extra credit project for my college prep 

students. 

Table 18—Teachers who reported incorporating 

Summer Institute experiences in their classrooms. 

 N % 

Explanations and examples in teaching 50 77% 

Classroom demonstrations 32 52% 

Laboratory exercises 31 50% 

Other* 18 29% 

Note: Multiple responses were allowed. 

* Other included adding field work, sharing Appalachian 

heritage, describing a national lab to students, and 

explaining the nature of science. 

Interviewed teachers gave specific examples of ways their approach to teaching had changed as a 

result of the institute. In one case, a teacher involved low-achieving students in research, setting 

up a competition between a science class with the lowest achieving students and an applied 

physics class of high-achieving students. Students were challenged to conduct a research project 

and write up results. Through this process, she discovered that both groups had strengths. 

Students in the science class got to work immediately but had difficulty presenting findings 

while the physics class spent a great deal of time deciding what to do, but easily wrote up their 

results. An added benefit she noted was that the week of the competition, attendance in the lowachieving science class rose from its usual low level to 100%. 
 

35

Table 19 displays some other short-term teacher outcomes.28 More than two-thirds of teachers 

said that the institute influenced them to discuss applications of STEM with students and 

encourage them to continue their education in STEM subjects. The following examples from 

interview notes illustrate how teachers felt they changed their approach to teaching. 

One teacher explained that as a result of her experience of finding it a challenge to 

learn about electron microscopes, UNIX and LUnix, she better understood her 

students’ frustration when they don’t comprehend classroom material. Now she 

breaks things down into “simpler bites,” goes more slowly, and has more patience 

when they have trouble. 

Another teacher, who said he was now more likely to try hands-on learning with 

his students, talked about teaching some of the techniques for taking a census of 

vegetation learned in his assigned ORNL group, which was “tasked” with 

exploring invasive species. 

A teacher who described attending ORNL as a “transformative experience” said 

that as a result of his interactions with student participants, his current 

relationships with students are less hierarchical, and now, when students present 

problems, rather than talking down to them, he tells them that he sees his job as 

helping them succeed. 

Two interviewed teachers noted that a major weakness of their experience at the Summer 

Institute was the absence of formal discussions concerning ways to apply what they were 

learning to classroom settings. While the experience had been “awesome,” they felt teachers 

“didn’t get much to bring back to the classroom.” 

As shown in Table 19, 86% of teachers reported they specifically encouraged students to 

continue their education in STEM. Three-quarters of them shared what they had learned with 

colleagues, and almost 70% reported that they talked to students about specific applications of 

STEM research. A majority of teachers said the institute motivated them to seek professional 

development in STEM. 

One teacher said that although she had always pushed her students to achieve, her 

experience at ORNL resulted in her telling students “with conviction” that there 

are jobs in science at multiple levels––that in addition to the PhDs working at 

ORNL, there were many with few years of higher education who were doing 

important work. She also described telling students about the many opportunities 

in applied science in other work places. 

Another teacher who also maintained she had always encouraged her students 

academically, reported that, as a result of being in a computer group at ORNL, 

she became more aware of the need for computer literacy. Now she emphasizes to 

her students that “every business has computers” and when students help her 

 

28 An evaluation of the STRIVE Teacher Research Associates Program 1986-1991, found that one-year 

after the program, 94% of teachers reported using program experiences for explanations and examples in 

their teaching, demonstrations, or lab activities; 91% discussed science/math applications with students; 

and 99% shared experiences and knowledge with colleagues informally. STRIVE is an eight-week 

program sponsored by NSF and the U.S. Department of Energy for secondary math and science teachers. 
 

36

figure out something on the computer, she says: “If you get a degree, you can get 

paid for doing this.” 

Table 19—Teachers who gave high ratings to selected outcomes 

N % 

I encouraged students to continue their education in science, math or technology. 56 86% 

I talked to students about research applications in the areas of math, science, or 

technology. 

45 69% 

I used Summer Institute materials or resources I learned about in classes or when 

working with student groups (e.g., science clubs). 

25 38% 

I became involved in science/math competitions or other related extra curricular or 

out-of-school time activities. 

7 11% 

Notes: 

The percentages represent teachers who gave ratings of 4 or 5 on a 5-point scale from 1= “not at all” 

to 5= “to a great extent.” 

Multiple answers were allowed. 

5.4 Barriers to Bringing ORNL Experiences to Classrooms 

However, it was not always easy for teachers to incorporate their experiences to the extent they 

wanted because of school-based constraints. Approximately half of teachers (51%) reported that 

they encountered obstacles to implementing changes to their teaching methods, curriculum, or 

course content upon return to their classrooms. The most common obstacles mentioned on the 

survey can be found in Table 20. During interviews teachers explained that in light of state 

standards, they could not “stray from the core ‘cookbook’ curriculum.” One teacher explained 

that teaching students with a range of abilities usually meant there was little time for the types of 

enriched curriculum components she would have liked to implement. In this regard, she noted 

that programs like the Summer Institute were important enrichment opportunities for gifted 

students. Some teachers reported that their schools did not allow field trips, primarily because of 

funding cutbacks. Another teacher did not go on field trips because she said there were no places 

within a reasonable distance to take students. 

Table 20—Percentage of teachers who encountered various obstacles 

(n=65) 

 N %

Pressure to cover the standard curriculum 21 32%

Inadequate lab equipment of other resources 21 32%

No resources for field trips 19 29%

Students lack sufficient skills or maturity 15 23%

Experience at ORNL was unrelated to current teaching assignment 7 11%

Inadequate support from the school administration 3 5%

Note: Multiple answers were allowed.
 

37

5.5 Pursuit of Professional Development 

Of teachers responding to the survey, 57% indicated that the institute encouraged them to pursue 

further STEM professional development. Three of the 13 teachers interviewed offered examples. 

One said she was exploring certification in physics and had already completed two classes. 

Another, who characterized herself as shy, admitted that she had been quite nervous before the 

trip and somewhat fearful of technology. She credited the institute with having given her the 

impetus to subsequently take two additional weeklong training sessions in technology. Another 

teacher reported that he was now more “more aggressive about seeking out and talking to people 

in science.” 

Interviews with teachers also shed light on a reason that it is difficult to recruit more teachers to 

professional development opportunities. It seems that those who had been teaching for a long 

time already had advanced degrees. Young teachers with young families are reluctant to spend a 

week apart from them. It also emerged that some of those who applied to the institute continually 

seek professional development. They inferred that they were high achievers, and while the 

institute may have reinforced their commitment to professional development, it did not 

necessarily increase their, already high, motivation. 

5.6 Impact on Other Teachers 

A majority of teachers (75%) reported that they shared their experience and information learned 

informally with other teachers.29 However, only a few teachers conducted formal workshops or 

in-service activities for other teachers or school administrators. The four teachers who conducted 

such events estimated reaching 95 others––one reached a group of 50 and the other three reached 

groups of about a dozen participants.30

5.7 Appalachian Region Employment 

At the time of the survey, most teachers were still teaching at the middle or high school levels. 

Five teachers had retired, three of whom were still employed. Of the non-retired teachers, 77% 

were currently high school teachers and 10% were teaching middle school including two of the 

respondents teaching both middle and high school subjects. As shown in Table 21, the large 

majority was working in the Appalachian region; 91% were living in the Appalachian region and 

all of these planned to be there five years hence. Of the six currently living outside of the region, 

two planned to return within five years. 

 

29 The 1992 evaluation of the eight-week STRIVE/TRAC reported that 99% of teachers shared 

experiences and knowledge from the program with colleagues in informal conversations. 

30 Other programs may have formal expectations that teachers share program information and materials. 

In the year following the program, the 173 teachers from the STRIVE Teacher Research Associates 

program estimated that they reached close to 20,000 students and more than 5,500 educators. 
 

38

Table 21—Current teaching status and place of 

employment 

 Current 

teaching 

status 

Percent 

working 

in the 

AR 

 N % % 

Middle school 5 7% 100% 

High school 51 77% 92% 

Both middle & high 2 3% 100% 

Other 3 5% 33% 

Retired 5 7% 100%* 

Total 66 99%**

*Of the 5 retirees, three were currently employed, all 

of these in the Appalachian region. 

**Percentage does not total 100% due to rounding.

5.8 Teacher Recommendations 

Teachers provided many ideas and recommendations to increase the effectiveness of the Summer 

Institute through their answers to both open-ended survey and interview questions. 

Several teachers suggested more activities and resources for lesson plan development in response 

to some frustration that they could not see how to incorporate what they learned into their 

classrooms. To remedy this problem, a teacher suggested that the schedule include time set aside 

for the teachers to work in groups to develop lesson plans, activities, and modules for classroom 

use. It was recommended that the groups could then combine all their materials in a notebook so 

all teachers could take notebooks back to their classrooms. Another teacher suggested offering 

graduate credit in exchange for creating a unit incorporating the ORNL experience. 

Teachers also stated that they would have liked more exposure to the other projects to obtain a 

broader understanding of ORNL projects to bring back to their classrooms. In response to this 

perceived need, one teacher suggested that the groups of teachers meet every few days for an 

“exchange” of what they were learning and doing in their groups. Another suggested that 

teachers rotate jobs and/or groups throughout the two-week experience. 

Another major area of concern that teachers expressed was that their Summer Institute project 

assignment did not always relate to the content areas they regularly taught. Some teachers 

expressed frustration over not being assigned to a relevant topic, while others were concerned 

that there were no relevant topics because available projects were too heavily focused on 

biology, environmental science, or computers. Specific suggestions were made to include more 

math, physics and chemistry projects. 

The above suggestions were the most prevalent in the qualitative data. However, other 

suggestions were made by teachers and may be equally useful to improving the Summer 

Institute: 
 

39

• Provide more information on job training and careers in the region to offer to their students. 

• Address innovative ways for working in classrooms with limited technology resources. 

• Allow students and teachers to work together cooperatively. 

• Increase follow-up communication or have a reunion. 

• Expand the program to include more teachers. 

From the above suggestions it is very clear that the teachers’ desire is to return to their schools 

with the tools necessary to help their students both understand STEM subjects and become 

interested in higher education. Some teachers left ORNL feeling more prepared to do this than 

others. 
 

40

6. Recruitment 

This evaluation explored the way students and teachers were recruited for the program because 

of its implications for who ultimately attends the program and the outcomes that can be expected. 

Questions about recruitment were included on both the student and teacher interviews. The 

analysis also considered the counties from which participants were drawn. 

According to recruitment materials, ARC-ORNL mandatory criteria for selection are as follows: 

• Student participants must be at least 16 years of age by the date the program begins. 

• All participants must be U.S. citizens. 

• Student participants must attend public school in a designated Appalachian county and plan 

to enroll in an eligible public school for the following school year. 

• Student participants must have a letter of reference from a teacher, school counselor, or 

administrator. 

• Teacher participants must plan to teach math, science and/or technology (e.g. Web design, 

computer programming) the following year in the ninth grade or higher in a public school in 

a designated Appalachian county. 

Other considerations for selection include the following: 

• Location in a distressed county or a distressed area in other Appalachian counties. 

• Teachers and students do not have to be from the same school. 

• Students and teachers who have not had an opportunity to participate in previous Oak Ridge 

or similar math and science programs are given priority. 

• ARC has not established income levels for participation. However, ARC requests that 

recruitment focus on those students from families without the financial means to send their 

children to such a program. 

• ARC has not established academic criteria requirements for participation. ARC requests that 

recruitment focus on “middle tier” students who show potential in math and/or science and 

who, with some encouragement, may improve their academic standing and consider college. 

• ARC would like those who recruit applicants to target “middle tier” students who have 

potential for success in math and science and who may be encouraged to take higher-level 

courses and/or to improve their academic standing by participating in the program. 

On the basis of the ARC designations of counties’ economic status 1997-2004, we determined 

that 31% of students and 47% of teachers were from schools in distressed counties. According to 

teachers, schools were selected either by the state department of education, the school district, 
 

41

and, in one case, a Local Development District in the Appalachian region.31 Within a school, 

teachers themselves choose whether to attend. In our small, and possibly unrepresentative 

sample, it appears that about one-third of teachers filled in when another teacher, either in their 

school or another, was unable to attend; 60% of teachers reported attending the Summer Institute 

with students from their schools. 

6.1 Teacher Recruitment 

The interviews with teachers revealed a range of ways of learning about the institute. Although 

all teachers completed applications, teachers who were invited or filled vacancies were sure they 

would be accepted, while the others assumed the process was competitive. The specific ways that 

teachers were recruited bulleted below account for 19 recruitment events for the 13 interviewed 

teachers because four of the interviewed teachers had attended more than one year. 

• In five cases, staff at the state department of education invited the school to send a teacher 

and students. 

• In four cases, the teacher said the school had been contacted directly by the school district. 

• In another four, the principal contacted the teacher. 

• In one case each, the invitation came directly to the school from a Local Development 

District office to the school; a student-teacher pair applied and learned that the school could 

send another teacher; and one teacher was contacted by a friend who had heard there was an 

opening. 

• One respondent recalled that he filled an unexpected vacancy prior to 1997 and was invited 

back the following year. 

The interview with teachers explored obstacles to recruitment and strategies for overcoming 

them. A few teachers who were involved with other summer teacher development programs 

confirmed that teacher recruitment was a common problem. Those who were interviewed posited 

numerous reasons that more teachers did not apply to the Summer Institute. Lack of information 

was by far the most common reason and mentioned by all interviewed teachers. One teacher who 

had received a program announcement felt that the projects to which teachers would be assigned 

were not clearly described. Other reasons included: 

• scheduling conflicts and competition from other professional development options 

• necessity for summer employment 

• desire to spend the summer with family or pursue other interests 

• pessimism—students and teachers do not believe a competitive program would select anyone 

from a “school like theirs” 

• disinterest in professional development, particularly for teachers at the end of their careers. 

 

31 Local Development Districts are multicounty planning and development organizations. 
 

42

Teachers recommended a variety of actions that could be taken. 

• Chief among these was more personalized outreach by past participants––for example, 

through conference presentations at meetings of the National Council of Teachers of Math 

and of the National Science Teachers Association. 

• Other teachers thought it would help if teachers considering applying could contact a past 

participant. 

• Other teachers suggested sending program announcements directly to teachers through 

listservs and mailings as well as to others (e.g., district personnel, principals, guidance 

counselors). They underscored the importance of timeliness of this outreach. 

Seeking insights about the things that promotional materials might highlight, we asked teachers 

why they decided to attend the Summer Institute. Some attended to accompany promising 

students whom they were anxious to expose to this experience. Others mentioned that they 

enjoyed professional development because of the growth opportunity and because summer 

programs “rejuvenate” and “revitalize” them. Three of the teachers mentioned specifically being 

interested in working at ORNL. One suggested that flyers should emphasize that the institute is 

“fun,” that teachers will have something to bring back to their classrooms and that there is a 

stipend. 

6.2 Student Recruitment 

The ways in which students were made aware of the Summer Institute varied: 

• Of the nine students interviewed, seven were personally approached by a teacher or school 

staff member and asked to attend the institute. 

• One student learned of the opportunity after a guidance counselor spoke to her sophomore 

class. 

• One student received no information from his school but heard about the institute from his 

grandfather, a physics teacher, who had attended the Summer Institute a few times before. 

Although this is a small sample of students, it appears that information regarding the institute is 

not made widely available to students but is targeted to a few students on the “radar” of teachers 

or guidance counselors. 

Students’ recollections of the application process confirm their impressions of the recruitment 

process. One student attending the 1999 Summer Institute reported that she did not have to apply 

because her science teacher was able to select two students. Five of the students didn’t recall 

much about the application process but did not remember it as stressful. Another student, 

however, recalled that this was the first competitive program to which she had ever applied and 

recalled being anxious about whether she would be accepted. Another student stated that some 

students at the institute “were kind of wondering how they got there.” In other words, for some 

students this process was competitive, whereas for others it was unclear to them how they were 

chosen to attend. 
 

43

7. Review of Selected Pre-College STEM Programs 

AED staff reviewed a wide range of pre-college STEM programs in the United States in an effort 

to identify attributes of successful pre-college programs and to provide descriptions that might 

benefit the continuing improvement of ARC-ORNL Math/Science/Technology Summer 

Institute. The programs described here do not comprise an exhaustive review, but rather a 

representative sample of programs that are compelling to participants, attractive to funders, and 

sustainable for the organizations that develop and deliver them. 

The review revealed a veritable smorgasbord of approaches, goals, resources, academic scope, 

target population, and program duration. Because evaluations of these programs have either not 

been undertaken or are unpublished, it is difficult to make claims about “what works” across all 

such programs. Nonetheless, this review offers an opportunity to compare approaches used by 

the Summer Institute with those of similar programs. The programs described below were chosen 

because they demonstrate the following criteria: 

• They have garnered funding through multiple sources over time, perhaps indicating favorable 

reviews by participants and past funders, though such data are unpublished. 

• They have strong program leadership, as indicated by the growth, longevity or general 

reception of the program. 

• They represent a range of geographic locations across the United States. 

Programs are grouped according to similar theories of action or modes of operation that strive to 

meet goals similar to ORNL goals. Within the groupings, programs are discussed in alphabetical 

order. The grouping titles may indicate a specific target population. 

College Bridge Programs for At-Risk Youth Who Show Promise 

College and universities that offer scholarships for minority students in the sciences often 

provide summer programs that bridge high school and undergraduate STEM experiences, in an 

effort to decrease attrition rates in college STEM majors among this target population. Such 

programs are often successful because they target students who show promise, because these 

same students often matriculate to the institution, and because program goals align with the goals 

of the parent institution/department. The AMP/Pre-College Summer Program at Fisk University 

was a three-week summer program targeted minority students with GPAs of 2.5 or higher, 

competitive ACT/SAT scores, and teacher recommendation. In other words, this is an example 

of a program that seeks to increase the success of minority students who already show academic 

motivation and promise. Academic instruction in science and mathematics, laboratory work, 

communications classes, field trips, and seminars on learning strategies were offered. Students 

enrolling in Fisk University who successfully completed this program were eligible for future 

stipends and undergraduate research posts. The program’s evaluation included an assessment of 

students’ academic progress via pre- and post-program written and oral exams. Participants also 

completed surveys to offer feedback on each aspect of the program. No results were available.32 

 

32 http://www.fisk.edu/index.asp?cat=49&parent=384&pid=385 
 

44

Another program targeting minority youth for a two-week summer program is the Build a 

Human Project at the Creighton University School of Medicine in Nebraska.33 The University of 

Cincinnati offers competitive summer programs for local high school students interested in 

health careers and the biomedical sciences.34 

Programs in Informal Science Institutions 

The Arizona Bioengineering Collaboration (ABC) is run by the Arizona Science Center and 

offers middle school science teachers the opportunity to enroll in each of four six-hour courses 

on various topics and applications in biotechnology over the course of the school year. Teachers 

also receive curriculum materials.35 The Center provides similar opportunities in biotechnology 

targeting students and parents in the form of afterschool and weekend programs. This approach 

is used at several other informal science institutions around the country, mainly because it is 

easiest to bring teachers to the institution where resources are readily available and because 

teachers and students appreciate experiencing something new at a prestigious or cutting-edge 

science institution. Receiving curriculum materials is often appealing to teachers in schools 

where curricular resources are limited or not well-coordinated throughout the county or school 

district. No evaluative data was available for this program. The California Academy of Sciences 

in San Francisco has run BioForum, a similar series.36 Other sites of similar programs include the 

Chicago Botanic Garden,37 the Boston Museum of Science,38 and the Fairchild Tropical Botanic 

Garden.39 

Institution-School Partnerships 

Many current grant programs require partnerships between organizations and school districts. 

This direct link with schools provides assistance with participant recruitment and follow-up 

communications, and allows program staff to gain insight into the immediate and long-term 

needs of local teachers. The Baylor Science Leadership Program/HU-LINC partners with the 

Houston Independent School District (HISD) to reform the science education practices of 177 

elementary schools. In addition, the program works to involve parents, scientists, school 

administrators, and other local community organizations, acting as the coordinator of these 

efforts. This institution is well known for its high-quality and ever-expanding outreach programs, 

mainly because staff form strong partnerships directly with school districts and maintain 

relationships over time. The program currently serves elementary, middle and high school 

science teachers, students, and their parents.40 

 

33 http://www.biomedsci.creighton.edu/education/outreach.html 

34 http://www.med.uc.edu/admissions/summerenrich.cfm 

35 http://www.azscience.org/investigating_biotech.php 

36 http://www.calacademy.org/education/course_catalog/spring_summer_2002/teacher_workshops.html 

37 http://www.chicagobotanic.org/explorations/ 

38 http://www.mos.org/doc/1812?id=666 

39 http://www.fairchildgarden.org/education/n_education.html 

40 http://www.cc.bcm.tmc.edu/ceo/ 
 

45

The Baylor programs have been extensively evaluated; however, results are unavailable to the 

public. Other programs with similar theories of action are located at the Cleveland Clinic 

Foundation in Florida and Ohio,41 and the Duke University School of Medicine.42 BioEd Online 

is a web-based resource designed to enhance the skills of high school biology teachers. This 

resource provides state-of-the-art information through streaming videos, slide libraries, and 

nature science updates.43 

Programs in Organizations Focusing on Single Issues or Specialized Areas 

Such programs have the advantage of a single issue-based focus that aligns naturally with the 

missions of their organizations. Further, they have ready access to experts who can demonstrate 

fieldwork techniques and science “in action,” thus providing a science immersion experience for 

students and teachers. 

For example, the Boston Waterfront Learning Project at the Children’s Museum joins forces with 

Save the Harbor, Save the Bay and the Urban Harbor Institute at UMASS Boston to provide 

educational programs highlighting the Boston harbor and local wharves.44 This project is 

strengthened through a partnership with Boston Public Schools, through which a regular rotation 

of classrooms is arranged. Similarly, the Chesapeake Bay Foundation, whose mission is to 

improve local water quality, invites classes of students to collect samples and observe marine 

life.45 The Children’s Discovery Museum of San Jose has created BioSITE, a program for 

students in grades 4 through middle and high school, to build awareness of the environmental 

issues of the local Guadalupe River.46 

The Centers for Ocean Science Education Excellence (COSEE) offer week-long teacher 

professional development programs in the summer where marine fieldwork methods are 

practiced. Approximately 20 teachers from North Carolina, South Carolina and Georgia are 

selected.47 The DNA Learning Center at Cold Spring Harbor Laboratories, New York provides 

field trips to local schools, where students can use state-of-the-art equipment to perform 

biotechnology techniques. The Center draws on its long history of DNA science, hosting 

exhibitions as well as online resources for teachers.48 A program with a similar biotechnology 

focus is CityLab at Boston University’s School of Medicine.49 

Programs Promoting a Specific Method or Practice 

A report covering 1990-95 describes how the Dartmouth Thayer School of Engineering created a 

summer course for K-12 science and math teachers that “represents a distinctive pedagogical 

 

41 http://cms.clevelandclinic.org/body.cfm?id=204 

42 http://www.duke.edu/~dbc4/boost/teachers/sci-immersion.html 

43 http://www.bioedonline.org/site/about.cfm

44 http://www.waterfrontlearning.org/ 

45 http://www.cbf.org/site/PageServer?pagename=edu_home 

46 http://www.cdm.org/biosite/about.htm 

47 http://www.scseagrant.org/se-cosee/teacher/06_leadership.htm 

48 http://www.dnalc.org/home.html 

49 http://www.bumc.bu.edu/Dept/Home.aspx?DepartmentID=285 
 

46

strategy which mimics the actual practice of engineering.” Teachers spent seven days at this 

institute, first experiencing immersion in this approach, and then determining ways to adapt this 

approach to their classrooms. At the time of the report, 155 teachers in 35 states had participated 

in the program.50 This was the only example we found of a program that demonstrates a 

particular academic strategy that teachers are expected to reproduce in the classroom. Translation 

to the classroom works well because there is a concrete, successful pedagogical mode that can be 

presented effectively by the institution and replicated by participating teachers. It suggests that 

when teachers have time during the session to consider issues of classroom adaptation, it is more 

likely that they will follow through once school begins. 

Collaborative Programs with Local, Rural and Native Populations 

Scientists at the Dakota Science Center are making connections with local Native American 

tribes and rural populations to bring science and technology to students in grades 6 to 9 in the 

Science Circle of Life Program. A unique aspect of this program is its efforts to staff the program 

with tribal elders and Native American counselors, as well as with members of the Dakota 

Science Center. Though funding limitations are not allowing an expansion of this program, it has 

served 36 Native American and rural students.51 

Several other programs work directly with Native American/Alaskan populations. A wellregarded program with a mission to serve Native American students is located at the Fred 

Hutchinson Cancer Center in the state of Washington; the Science Education Partnership at the 

center has expanded to include the provision of curriculum kits to local schools.52 Another 

program, the Imaginarium, bridges native communities in Alaska through cultural and science 

events.53 By working directly with community members to appreciate the cultural and lifestyle 

differences of Native American and rural populations, these programs report long-term 

relationships with not only participants, but with those connected to them (e.g., family members, 

community leaders, teachers, etc.). 

Scientific Work Experience Programs for Teachers (SWEPTs) 

As of 2000, there were approximately 72 SWEPT programs serving upwards of 1,300 teachers 

each summer around the country.54 Pioneer institutions include the Columbia University College 

of Physicians and Surgeons,55 offering laboratory experiences for teachers only, and Rockefeller 

University,56 offering laboratory experiences for high school students and teachers of all grade 

levels. Several students from Rockefeller University have developed their research projects 

through the Westinghouse and Intel science competitions, earning awards in the finalist rounds. 

 

50 http://fie.engrng.pitt.edu/fie95/4b1/4b14/4b14.htm 

51 http://www.dakota-science.org/ 

52 http://www.fhcrc.org/science/education/educators/sep/ 

53 http://www.imaginarium.org/types%20of%20programs.html 

54Bacon, W. S., ed. (2000). Bringing the Excitement of Science to the Classroom. Tucson, AZ: Research 

Corporation. 

55 http://www.scienceteacherprogram.org/indexorig.html 

56 http://www.rockefeller.edu/outreach/ 
 

47

The FoxChase Cancer Center in Philadelphia is another example of an institution that provides 

teachers and students an opportunity to conduct authentic research.57 The opportunity for 

teachers and students to spend between four and nine weeks working in a research laboratory 

under the supervision of practicing scientists is considered by participants to be invaluable in 

helping students and teachers understand the nature of research and the scientific enterprise.58 

Summary 

A search of the literature for evaluations of pre-college STEM programs for comparison with 

findings of the current evaluation revealed that either evaluation is not conducted or programs 

have chosen not to publish evaluation findings. The few evaluations we reviewed and referenced 

earlier in this report were conducted by ORISE for Department of Energy projects. Findings of 

the Summer Institute with regard to variables, such as using the experience at Oak Ridge in 

teaching and satisfaction with the program, compare favorably. 

The programs described in this section of the report can be considered best practices on the basis 

of their longevity and perceived effectiveness by administrators, funders and participants. Like 

the ARC-ORNL Summer Institute, many of the programs reviewed offer immersion experiences 

for both students and teachers. Programs are administered by higher education institutions and 

medical programs as well as informal science institutions, such as science museums and botanic 

gardens. 

A positive attribute of these programs, like that of the ARC-ORNL Summer Institute, is their 

location at a prestigious institution where participants can use state-of-the-art equipment and 

work on meaningful, relevant projects. Unlike ARC’s wide geographic reach, other programs 

have chosen to target populations in cities. When this occurs, local programs facilitate the 

development of ongoing partnerships with local school systems, which assists recruitment, 

ongoing assessment of participant needs and interests, and parent involvement. While some 

programs reviewed were longer, others were of similar duration to the Summer Institute. The 

programs that target student populations underrepresented in STEM appear to demonstrate a 

more sustained commitment to participants. 

 

57 http://www.fccc.edu/research/education/index.html 

58 For more insights into recently funded programs (2005), refer to the compendium of Math Science 

Partnerships funded by NSF and available online at 

http://www.ed.gov/programs/mathsci/nsfabstracts.doc
 

48

8. Conclusions and Recommendations 

The Appalachian Regional Commission–Oak Ridge National Laboratory Summer Institute for 

Math/Science/Technology is a program that addresses national goals to meet the needs for a 

skilled, highly scientific and educated workforce, particularly in science, technology, engineering 

and math. Building human capital through education is one of ARC’s core economic 

development strategies. 

The three main goals for the Summer Institute are: 

1. Encourage more high school students to continue their studies beyond high school. 

2. Encourage more students to pursue careers in the projected shortage areas of math, 

engineering, science, and technology. 

3. Raise the level of math, science, and technology instruction in high schools throughout 

the region to facilitate the first two goals. 

This evaluation was commissioned to assess the extent to which the three main goals for the 

institute are being achieved. In addition to measuring long-term outcomes, the evaluation also 

focused on short-term results and sought to examine participants’ perspectives on the aspects of 

the Summer Institute experience that were particularly meaningful in relation to the program’s 

overall goals. 

Evaluation findings are based on data collected through self-administered questionnaires and 

interviews conducted in 2005 with students and teachers who attended the Summer Institute 

between 1997 and 2004. One of the key limitations of this evaluation, as stated in the beginning 

of this report, is the absence of a comparison group, which would have allowed for the 

attribution of outcomes to the program, and of data which would have provided pre-post 

comparison of knowledge, attitudes and intentions. 

Recommendations below begin with the issue of program definition and recruitment and then 

separately address student- and teacher-specific issues. Three key policy issues emerge from 

these findings: 1) clarify and widely disseminate recruitment objectives, criteria and process, 2) 

build on the current strengths of the Summer Institute and consider increasing the intensity of the 

program, and 3) strengthen the evaluation of the institute for the purpose of ongoing 

improvement. 

 8.1 Recommendations about Program Objectives and Recruitment 

The purposes of the Summer Institute program are laudable and address a critical need. The 

recommendations below are offered to help the institute fulfill these purposes more deliberately 

and effectively. Recommendations include: 

Review specific program objectives and target population. One of the most important 

recommendations, and one that drives several of the recommendations that follow, is that 

Summer Institute revisit and formalize its specific objectives regarding program purpose and the 

target population. In defining objectives, the program must take into account what can 

reasonably be accomplished in two weeks or consider what steps could be taken to increase 

program intensity. 
 

49

Make the recruitment criteria and process more transparent. A clearer definition of the 

program’s specific objectives would help clarify the target populations of students and 

teachers who should be recruited. Formalized criteria would help generate a uniform and 

transparent process for recruitment and dissemination to those recruited would clarify the 

selection process. Because virtually all students reported plans to attend college prior to the 

institute, and the majority of students were from families where parents/guardians were college 

educated, it appears that student recruitment is targeting highly motivated students from 

households likely to be providing support for college-going. Similarly, teachers who were 

recruited tended to be highly motivated and experienced. If the program wishes to continue to 

recruit teachers with such characteristics, expectations regarding their leadership role upon 

returning to school might be made more explicit. The involvement of newer teachers, on the 

other hand, might increase the possibility that they would reach a larger number of students over 

the course of their careers and continue their professional development in STEM. Additionally, 

clearer recruitment criteria would address teacher uncertainty, given the limited number of 

spaces in the program, about whether the institute seeks to involve students and teachers from 

different schools each year or whether they should encourage others from their schools to apply. 

Recruit teachers directly. Teachers, noting the vagaries of receiving announcements sent to 

principals or district administrators, suggested that recruitment strategies be broadened and more 

directed at them. 

The number of students and teachers who can be reached by this program is relatively small, 

given that the program has openings for 52—26 students and 26 teachers each year—plus an 

additional eight openings for states that will pick up the costs. From an analysis of the data from 

all participants 1997-2004, all 13 states in the Appalachian region sent participants, but three 

states—Georgia, New York, and Ohio—were overrepresented. The gender distribution of 

student participants is fairly evenly divided between females (52%) and males (48%). 

Approximately 31% of students and 47% of teachers came from schools in ARC-designated 

distressed counties, which comprised on average 26% of counties in the region between 1997 

and 2004. While ARC does not collect data on the race/ethnicity of program participants, 

diversity of the student and teacher survey respondents reflected the racial and ethnic diversity in 

the region, where approximately 12% of the population in 2000 were racial and ethnic 

minorities.59

Increase the number of youth from underrepresented groups in STEM. If the Summer 

Institute chooses to more closely align its objectives for its target population with federal goals to 

increase the number of individuals from underrepresented racial, ethnic, and socioeconomic 

groups in STEM, the Summer Institute should make its purposes explicit in recruitment 

materials and outreach to agencies and organizations that play a role in recruitment.60

 

59 http://www.arc.gov/index.do?nodeId=1842 

60 National Science Board (2003). The Science and Engineering Workforce, Realizing America's 

Potential. Washington DC: National Science Foundation, 

http://www.nsf.gov/nsb/documents/2003/nsb0369/nsb0369.pdf

NSF publications relating to STEM and: 

Gender equity: http://www.nsf.gov/pubs/2002/nsf02107/nsf02107.pdf , 
 

50

Another outreach strategy, addressed again below, would involve partnerships with other 

college-access programs already reaching underrepresented populations. 

In a two-week, one-shot program, an argument can be made that it is acceptable to target 

students who may “make it” without the program but who can still benefit from the “extra push” 

in terms of their self-confidence and interest in STEM majors and careers that the Summer 

Institute provides. Certainly student surveys and interviews suggest that the institute does play a 

role in giving students this push. However, if the institute seeks to target a more diverse 

population, improving access to college and STEM careers for students from racial, ethnic, and 

socioeconomic groups not traditionally represented in these areas, then this goal will have to be 

made an explicit part of the recruitment process. Further, recruiting a more diverse population 

may mean making changes in the institute itself—the programming and staff, the approaches, 

and the supports, such as tutoring and mentoring, available during the two weeks. 

Another recruitment issue is the assignment of students and teachers to specific ORNL projects. 

A number of participants from both groups noted that there was a mismatch between their 

interests and their assignment. This seemed to be a particular problem for non-science teachers. 

According to a conversation with the program coordinator, we understand that it is difficult for 

ORNL to commit to mentors/projects well in advance of the time recruitment announcements go 

out to prospective participants. Nevertheless, it seems important to obtain commitments from 

ORNL staff earlier in the year, even if it means broadening the number of ORNL staff invited, 

for example, graduate students and young professionals. 

Recruit fewer non-science teachers. Another possible solution is for the program to recruit and 

accept fewer nonscience teachers unless it is known for certain that there will be an appropriate 

project for them. 

Match student interests with projects, even if it means rejecting a student’s application and 

substituting someone else whose interests will be better served by the projects available that 

year. Another possibility would be to limit the institute to two or three subject areas per year but 

rotate the subject areas, for example, physics and mathematics one year, biology and computers 

the next. 

Constitute a “general” group. Another possibility, if students or teachers feel the match 

between their interests and the available projects is not good, is to constitute a group that spends 

a day or more visiting and working with all the projects to give participants a broad exposure to 

ORNL activities. If members of each project group were asked to describe their work to the 

members of the visiting “general group,” it would reinforce participants’ understanding of the 

significance of their work and give them the opportunity to practice explaining their activities to 

others when they returned home. 

 

Native Americans: http://www.nsf.gov/pubs/2002/nsf02072/nsf02072.pdf , 

Black populations: http://www.nsf.gov/pubs/2002/nsf02160/nsf02160.pdf . 
 

51

8.2 Student-Specific Recommendations 

Overall, students who attended the Summer Institute appear to have been highly motivated and 

interested in STEM prior to attending the program. Nonetheless, the institute appears to have had 

an important influence on reinforcing students’ decisions about college and inspiring their 

interests in STEM. Students and teachers interviewed attributed students’ gains in confidence 

and maturity to the institute. Upon returning to school, many were more serious about their 

studies and motivated to achieve. About a quarter of them reported taking more science and math 

courses in high school than planned. 

Fully 96% of student participants went on to higher education. Somewhat more than half of 

degrees earned by student participants at the time of the survey have been in STEM fields and 

82% of those currently pursuing degrees are majoring in STEM fields. 

While the program appears to be beneficial and certainly highly appreciated by participants, 

there are ways programs can be strengthened. Possible avenues are found in the appraisal of 

program models reviewed in this report. Of particular relevance are programs that create 

partnerships with school districts and develop sustained relationships with communities, 

teachers, and parents. Moreover, it is constructive to consider the elements of college-access 

programs believed to be most effective in increasing college-going rates, particularly if the 

Summer Institute wants to reach out to more underrepresented students. The recommendations 

that follow the list below, derive from survey and interview findings, but they specifically 

address the first four items on the list. Effective college-access programs have been found to: 

• Provide a peer group that supports students’ academic aspirations and provides them social 

and emotional support. 

• Provide a key person who reviews students’ progress and guides them over a long period of 

time. 

• Make long-term investments in students rather than short-term interventions. 

• Pay attention to the cultural background of students. 

• Provide high-quality instruction. 

• Provide financial assistance and incentives.61 

Student-related recommendations include: 

Fostering peer communication: Students described multiple ways in which their peer group 

exerted a positive influence. Some interviewed students reported being in contact with others in 

their cohort after the institute. Computer networks and Internet-based communications make this 

easier each year. However, cross-cohort communication does not appear to be occurring. To 

strengthen the connections that students stated were so valuable, Summer Institute administrators 

might consider using on-line groups to create a bulletin board or other more formal website 

to encourage communication among past participants. Such a site, for example, could link 

 

61 U.S. Department of Education, National Center for Education Statistics. See footnote 20. 
 

52

students exploring college options with past participants at colleges which current or recent 

participants are considering. Students working on science projects or interested in knowing 

requirements for specific types of jobs might be able to find help from former institute 

participants. An ORNL–ARC Summer Institute website could also facilitate connections 

between those students thinking of applying to the Summer Institute with past participants who 

identified themselves on the survey as willing to provide information about their experience. 

Bridging the age/experience gap: Another peer-related suggestion made by students was 

bridging the age/experience gap between mentors and students. Students recommended that at 

some time during the institute, students should have opportunities to meet graduate students 

or past program participants to talk to them about education and career choices. Graduate 

students and young professionals in STEM can also be recruited to assist mentors with the 

groups throughout the two-weeks. 

Creating connections between the Summer Institute, sending school and parents: The 

Summer Institute might consider ensuring that staff members from the sending school (e.g., 

current STEM teachers and guidance counselors) are aware that the student attended the institute 

so that they can continue to encourage participants’ progress. If students give the names of these 

school staff to the program, a follow-up letter or phone call to the school from a Summer 

Institute staff member could be used to describe the student’s experience and offer suggestions 

about ways to reinforce his/her college-going ambitions. Because students indicated on the 

survey that parents were most influential in students’ college-going decisions, outreach to 

parents that reinforces connections between parents and school or district resources could also 

benefit students. 

Creating connections with college-access programs in the region. Finally, acknowledging that 

the Summer Institute may not have the resources to create a program that establishes a long-term 

relationship with students, it may be useful to create connections with college- access programs 

in the region that do. Such collaboration could provide an applicant pool of underrepresented 

students and follow-up and support for students over a longer period. Another possibility would 

be to build relationships with specific high schools in the region. High school-STEM program 

partnerships such as the Baylor Science Leadership Program, reviewed in section 7 of this report, 

are considered promising practices. 
 

53

8.3 Teacher-Specific Recommendations 

The Summer Institute appears to be attracting highly motivated and experienced teachers who 

find it to be a valuable experience, and several have taken the opportunity to attend more than 

one year. Teachers stated that they felt revitalized by the program, which both increased their 

STEM knowledge and their awareness of opportunities in applied science to help them prepare 

students for careers in STEM fields. Teachers reported that the institute gave them concrete 

examples of science applications to share with students and the majority reported they integrated 

aspects of their Summer Institute projects into their classes. While many interviewed teachers 

said that they had always encouraged their students to pursue education beyond high school, 

their experience at ORNL increased their awareness of career opportunities. In this regard, 

several indicated that they were better able to instruct lower-achieving students and encourage 

them to pursue careers in STEM fields because they now were more aware of the variety of jobs 

requiring different levels of STEM education. 

Like the students, teachers mentioned the important influence of their peer group during their 

time at Oak Ridge. Specific recommendations reflecting the value teachers place on their peer 

group and addressing teachers’ concerns about ways to integrate their lab experience in the 

classroom include: 

Give more attention to curricular issues: The program might provide more structured time for 

teachers to discuss both specific curricular implications as well as ways to continue to promote 

interest in STEM education and careers among their students. Ultimately teachers need to be 

involved designing any improvements to this component. The following is a suggestion that 

could be raised in a planning session. As part of their assignment at the lab, teachers might create 

a three-part notebook: the first would contain one lesson plan related to each project; the 

second part would contain concrete suggestions about encouraging student interests in STEM; 

and the third part could contain references and descriptions for resources that teachers learned 

about at the Summer Institute or that they have found useful. Each year’s notebook could be 

posted on a Summer Institute Website. This notebook would be a way to share information with 

colleagues in the sending schools and districts and could be the centerpiece of a districtwide 

workshop for teachers that would expand the impact of the Summer Institute program. Provision 

of a small amount of funding to duplicate the notebook and provide refreshments, might increase 

the number of such workshops teachers conduct. 

Foster networking among teachers. Enlist the assistance of teachers to suggest ways to 

enhance communications between teacher participants from multiple years of the Summer 

Institute. The website would be most useful if it addressed curricular issues. As was 

recommended with regard to students, the website could contain contact information for teachers 

willing to tell new applicants about the program. Another way to foster networking would be to 

host a mid-year conference call for teachers to talk about their experience with strategies they 

have tried to foster college-going and STEM studies among students or other professional 

development opportunities they have learned about.

Make explicit expectations that teachers share their experience with other teachers. 

Teachers reported that they shared their Summer Institute experience with other teachers, but few 

did so in a formal way that would reach significant numbers. If more than one teacher per school 

or school district were selected to attend the institute, they could jointly develop a presentation 

for their colleagues. Small grants to cover refreshments and other meeting costs would be 
 

54

incentives. Another approach would be to encourage selected teachers from each cohort to 

describe their experience at statewide or national professional association meetings. Again, 

reimbursement for travel or conference registration would help.

Summary 

Findings of this evaluation indicate that ARC-ORNL Summer Institute Program has been of 

benefit to student and teacher participants and has helped ARC achieve its goal of increasing 

college-access and graduation in STEM fields in the Appalachian region. More rigorous 

evaluation, as proposed in the appendix to this report, will provide the program’s stakeholders 

with more scientifically accurate evidence of program strengths and weaknesses to inform 

decisions about program improvement and continuation. 

8.4 Recommendations for Future Evaluation 

The following are recommendations for ongoing evaluation of the Appalachian Regional 

Commission (ARC)–Oak Ridge National Laboratory (ORNL) Summer Institute for 

Math/Science/Technology. These recommendations were requested as part of the Request for 

Proposals (2005) for the evaluation of the Summer Institute. 

AED’s proposed evaluation is grounded in 10 principles of effective evaluation included in the 

appendix. Following these principles, the first step in designing an ongoing evaluation plan is 

reaching agreement about the purpose of the evaluation and the use to which it will be put. This 

plan assumes that ARC is seeking to determine the extent to which the program is reaching its 

objectives, both for the purposes of quality improvement and to obtain a sense of whether the 

government’s continued investment in the program appears to be worthwhile. With this in mind, 

we would recommend an evaluation that explores how the program is implemented (e.g. one that 

assesses the program’s recruitment process and activities) as well as the program’s outcomes. 

Agreement about the purpose of the evaluation would need to be decided before the evaluation 

design was finalized. 

The second step for the evaluation is establishing realistic objectives (and associated outcomes) 

in light of the current scope of the program. While a small number of the student participants we 

interviewed claimed that the Summer Institute was a life-changing experience, it is unrealistic to 

expect that a two-week intervention will have an impact on college-going and STEM careers for 

the group as a whole. Nonetheless, it is reasonable to expect that the institute contribute, along 

with other influences, to encouraging youth to have higher academic expectations for themselves 

and to steer them toward college. Rather than measuring outcomes such as college persistence 

and pursuit of STEM careers, the evaluation should measure attitudes, behaviors, and intentions 

that mediate the achievement these long-term outcomes. 

 AED recommends ongoing evaluation to address questions such as the following, many 

of which were studied in the current evaluation. The final evaluation questions will need 

to be aligned with the objectives: 

Is the Summer Institute recruiting/serving the type of students and teachers it 

seeks to reach? 

Are targeted populations recruited and attending the institute? 
 

55

Is the application process reaching students who are promising but not bound for 

a four-year college? 

Are teachers who are early in their careers attending the institute? 

If more experienced teachers are being recruited, do they influence other teachers 

once they return to their school districts? 

 To what extent does the Summer Institute have an impact on students’ attitudes, 

intentions, and behaviors related to college-going and careers in STEM? 

Upon return to high school, do students take the STEM classes that will prepare 

them for college? 

Are students more confident of themselves and their abilities in STEM? 

Do students prepare for college in terms of taking college entrance exams (SAT 

and ACT) and visiting colleges? 

Are students more likely to apply to a four-year college rather than a community 

college? 

 To what extent do teachers incorporate their experiences into the classroom and to what 

extent and in what ways do they encourage students to pursue their education in STEM 

fields? 

Do teachers draw on the Summer Institute experience for explanations and 

examples, classroom demonstrations or laboratory exercises? 

Do teachers change their teaching methods, or use materials and other resources 

in their classrooms that they obtained at the Summer Institute? 

Do teachers encourage more students to pursue higher education and STEM 

studies after they attend the institute and do they do so in different ways? 

Do teachers take part in more professional development opportunities in STEM 

after the Summer Institute? 

 What are the aspects of the program that participants, mentors, and chaperones believe 

are the most influential for achieving the program’s objectives? 

What are the strengths and weaknesses of the program from the perspectives of 

participants, mentors, and chaperones? 

What are participants’ and other staff members’ recommendations for 

improvement? 
 

56

Design 

We propose an evaluation in which data are collected from participants, before the institute and 

at two times post-participation, once immediately after and one nine-month follow-up; which 

will lessen sample attrition and is sufficient time to observe change. The post-test at the end of 

the institute will provide information about immediate outcomes and allow evaluators to obtain 

feedback about any new improvements to the program. A nine-month follow-up will also allow 

evaluators to assess ways the program may influence students in light of decisions they will be 

making when they return to school, as well as to determine the extent to which effects persist 

over time. If students enter their senior year in high school when they return, at the nine-month 

follow up (April), most will know where they will be going to college, and juniors will have 

selected their courses for the following year. Teachers will have had the opportunity to schedule 

and possibly conduct workshops for other teachers and to have planned further professional 

development. If resources permit, students who are juniors when they return to school after the 

institute can be surveyed in April of their senior year. 

We propose that the same evaluation instruments be used with new Summer Institute cohorts so 

that responses for two or three years can be grouped to provide a larger sample size and to permit 

subgroup analyses. 

A pre-post design will allow evaluators to observe change in the participants but it will not allow 

us to attribute any of this change to the Summer Institute. If possible, a comparison group should 

be used to allow for comparison between student participants who attend the program and those 

who do not. One possibility would be to create a larger applicant pool. Youth who are selected 

but are unable to attend or those not selected as participants, but who are similar to those who do 

attend, would be asked to complete both follow-up surveys. In order to gain their cooperation, 

the comparison group should receive some mild intervention such as information about STEM 

career opportunities in the region or resources about college-going. Also, members of the 

comparison group should receive a small monetary incentive for completing both surveys. 

AED also recommends two qualitative methods for the evaluation. The first is a site visit by an 

objective third party during the second week of the institute to observe each activity group and 

interview staff. The second method is personal (telephone) interviews or group discussions 

(conference call), with a small number of participants each year. These discussions can offer 

invaluable insights about how the Summer Institute influences participants and can elicit useful 

suggestions for program improvement. 

The Study Population 

Because the Summer Institute serves approximately 26 students and 26 teachers each year (and 

an additional eight participants if states pick up the costs), any evaluation will need to include the 

entire population of participants rather than a sample. In addition, it would be useful to include 

mentors, chaperones, and any other staff closely associated with the program, such as the ORISE 

project director. 

One of the difficulties encountered in AED’s evaluation was finding program participants. The 

time-frame for the recommended ongoing evaluation will obviate the need for intense tracking. 

However, these strategies are recommended in order to ensure that participants can be found: 

 To obtain accurate contact data from students and teachers accepted into the program, 

have participants enter street and email addresses directly into an electronic database and 
 

57

then have them check the information on hard copy. This will minimize difficulties in 

interpreting handwriting or data-entry errors. 

 Obtain contact information (names, addresses, phone numbers) for a variety of 

individuals including parents’ names and addresses (students only); sending school; and 

one additional contact who would know where to find them should they move. 

 Send postcards to participants 3 months after the program. Any returned postcards will 

have forwarding information up to 60 days of a move. Ask participants to update contact 

information by email or by return postcard. Participants can also enter updated 

information on a Web page. 

Consent for Participation in the Evaluation and Confidentiality Protections 

During the application process, students and teachers should be advised that ARC conducts an 

evaluation as part of the Summer Institute for the purpose of program improvement. Applicants 

should be advised that participation in the evaluation is not considered a prerequisite for 

selection into the program. Applicants should be asked for their informed consent to take part in 

the evaluation, assured of confidentiality of their data, and advised of the possibility that findings 

will be published. 

Consent from student participants will need to be obtained a second time if the evaluators track 

college access and persistence forward, for example using the National Student Clearinghouse 

(NSC) database. Social security numbers, collected for security clearance, can be used with 

consent of the participants 18 years of age or older. Because NSC data will be accessed after the 

participant has turned 18, consent will have to be obtained subsequent to the summer of their 

participation. It is recommended that the evaluators obtain consent from as many students as 

possible who will have turned 18 years old at the 9-month survey and from others at a later point 

(which can also afford an opportunity to update contact information). 

Dissemination of Evaluation Findings 

The full report should be shared with program stakeholders and a summary of the evaluation 

should be shared with participants from whom data were collected. 
Appendices 

1. Appendix Tables 

2. Ten Principles of Effective Evaluation 
 

Appendix Tables 
 iii

Table A1—Percentage of students and teachers 

residing in Appalachian states when they attended the 

Summer Institute 

Students Teachers 

 N % N % 

AL 4 5% 5 8% 

GA 10 11% 5 8% 

KY 2 2% 1 1% 

MD 8 9% 2 3% 

MS 4 5% 5 8% 

NC 2 2% 1 1% 

NY 15 17% 7 11% 

OH 17 19% 13 20% 

PA 7 8% 3 5% 

SC 4 5% 4 6% 

TN 5 6% 6 9% 

VA 4 5% 3 5% 

WV 6 7% 11 17% 

Total 88 98% 66 102% 

Note: Percentages in this table do not total 100% due to 

rounding
 iv

Table A2—Selected characteristics of 

student interviewees 

(n=9) 

 N % 

Gender 

 Male 3 33% 

 Female 6 67% 

Year Attended 

 1997-1998 2 22% 

 1999-2000 3 33% 

 2001-2002 3 33% 

 2003-2004 1 11% 

Use of STEM in current 

occupation

 

 To a great extent 4 67% 

 Somewhat 2 33% 

 Unemployed 3 N/A 

Highest level of 

education completed 

since attending the 

Summer Institute 

 

 One or more years of 

college but no degree 

6 67% 

 Associate’s degree 1 11% 

 Bachelor’s degree 0 0% 

 Some graduate work 1 11% 

 Advanced degree 

(Master’s or Ph.D.) 

1 11% 

Ethnicity 

 White 8 88% 

White and African 

American 

1 11% 

Table A3—Selected characteristics of 

teacher interviewees 

 (n=13) 

N % 

Gender 

 Male 6 46% 

 Female 7 54% 

Year attended 

 Multiple years 4 31% 

 1997-1998 1 8% 

 1999-2000 2 15% 

 2001-2002 2 15% 

 2003-2004 4 31% 

Number of years 

teaching experience at 

time of the institute 

 

1-5 3 23% 

 6-15 2 15% 

 16+ 7 54% 

 Missing data 1 8% 

Ethnicity 

 White 9 69% 

 Black or African 

American 

1 8% 

Hispanic or Latino 1 8% 

 Missing data 2 15% 
 v

Table A4—First educational institution attended after high 

school 

Male Female 

 N % N % 

No post HS education 2 5% 1 2% 

Community college or 

Technical Center 

8 19% 4 9% 

Military Academy 2 5% 0 0% 

4-year school 29 71% 37 88% 

Total 41 100% 42 99% 

Table A5—Highest level of education attained 

High school or 

less 

Some college 

but no degree 

Associate’s 

degree or 

technical 

diploma/ 

certificate 

Bachelor’s 

degree Graduate work 

 N % N % N % N % N % 

1997& 

1998 

(n=23) 

 0 0% 5 22% 1 4% 9 39% 8 35% 

1999& 

2000 

(n=23) 

 1 4% 5 22% 3 13% 10 43% 4 17% 

2001& 

2002 

(n=16) 

 3 19% 11 69% 2 12% 0 0% 0 0% 

2003& 

2004 

(n=26) 

14 54% 11 42% 1 4% 0 0% 0 0% 

Total 

(n=88) 18 20% 32 36% 7 8% 19 22% 12 14% 

 
 vi

Ten Principles of Effective Evaluation 

1. Evaluators ideally should be involved in systematic inquiry with the reform leaders in the 

early stages of project conceptualization; in assessing current conditions and capacities and 

needs for improvement; in identifying the performance gaps and other problems, and in 

envisioning program designs that seek to close the gaps, solve problems, and meet identified 

needs. 

2. Evaluators should work with project designers and proposal writers to develop a project logic 

model to ensure that an internally consistent program is designed to respond to the needs and 

problems identified, with a set of project activities or interventions likely to impact the 

original conditions, problems, and performance gaps favorably. 

3. Evaluators should combine quantitative and qualitative methods and employ a comparative 

evaluation design where feasible. 

4. Evaluators should seek a comprehensive understanding of the important contextual elements 

of the evaluation. Contextual factors that may influence the results of a study include 

geographic location, timing, political and social climate, economic conditions, and other 

relevant activities in progress at the same time. 

5. Evaluators should involve all stakeholders in a participatory process that builds future 

internal evaluation capacity and also communicate their values, assumptions, theories, 

approaches and analytic methods accurately and in sufficient detail to allow the stakeholders 

to understand, interpret, and critique their work and evaluation findings. 

6. Evaluators should conduct the evaluation and communicate its results in a way that respects 

the stakeholders' dignity and self-worth. 

7. Evaluators should seek to ensure that those who bear the burden of collecting data have full 

knowledge of, and opportunity to use for program improvement, the evaluators’ findings, 

analyses, and recommendations. 

8. Evaluators should make explicit their own interests, their clients’ interests, and other 

stakeholders’ interests concerning the conduct and outcomes of an evaluation. 

9. Evaluators should allow all relevant stakeholders access to evaluative information in forms 

that respect people and honor promises of confidentiality. 

10. Evaluators should make clear to clients when client interests and requests conflict with the 

obligation of evaluators for objective inquiry, competence, integrity, and respect for people. 

In these cases, evaluators should discuss the conflicts with the client and relevant 

stakeholders, resolve them when possible, determine whether continued work on the 

evaluation is advisable if the conflicts cannot be resolved, and state clearly any significant 

limitations on the evaluation that might result if the conflict is not resolved. The public 

interest in professional, objective inquiry and evaluation for the welfare of society should be 

upheld as a higher value than a particular client’s or other stakeholder’s interests. 
he Academy for Educational Development (AED) is an independent, nonprofit 

organization committed to addressing human development needs in the United States and 

throughout the world. As one of the world's foremost human and social development 

organizations, AED works in five major program areas: U.S. Education and Workforce 

Development; Global Learning; Global Health, Population and Nutrition; Leadership and 

Institutional Development; and Social Change. At the heart of all our programs is an emphasis on 

building skills and knowledge to improve people's lives. 

The AED Center for School and Community Services is part of AED’s U.S. Education and 

Workforce Development Group. The Center uses multidisciplinary approaches to address critical 

issues in education, health, and youth development. To achieve its goals, the center provides 

technical assistance to strengthen schools, school districts, and community-based organizations. 

It conducts evaluations of school and community programs while striving to provide the skills 

and impetus for practitioners to undertake ongoing assessment and improvement. The center also 

manages large-scale initiatives to strengthen practitioner networks and accelerate systems change 

and uses the knowledge gained from this work to advocate for effective policies and practices 

and disseminate information through publications, presentations, and on the World Wide Web. 

In the past 27 years, the Center has undertaken over 125 evaluation, technical assistance, and 

dissemination projects in 90 cities and 40 states. 

In 2005, the Educational Equity Center at AED (EEC) was formed.. The Center is an outgrowth 

of Educational Equity Concepts, a national nonprofit organization with a 22-year history of 

addressing educational excellence for all children regardless of gender, race/ethnicity, disability, 

or level of family income. EEC’s goal is to ensure that equity is a key focus within national 

reform efforts to ensure equality of opportunity on in schools and afterschool settings, starting in 

early childhood. 

AED is headquartered in Washington, DC, and has offices in 167 countries and cities around the 

world and throughout the United States. The Center for School and Community Services is in 

AED’s office in New York City. For more information about the Center’s work, go to the 

Center’s website at www.aed.org/scs or contact Patrick Montesano or Alexandra Weinbaum, codirectors, at 212-243-1110, or e-mail sweinbau or pmontesa@aed.org. 

Principal Offices 

1825 Connecticut Avenue 

Washington DC 20009-5721 

Tel: 202-884-8000 

Fax: 202-884-8400 

www.aed.org 

100 Fifth Avenue 

New York, NY 10011 

Tel: 212-243-1110 

Fax: 212-627-0407 

www.aed.org/scs 

T